2024-09-17T04:34:33.055 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T04:34:33.080 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T04:34:33.177 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553 branch: wip-pdonnell-testing-20240916.200549-debug description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7908553' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-pdonnell-testing-20240916.200549-debug ceph: cephfs: fs: - name: a - name: b session_timeout: 300 cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: client mount timeout: 30 debug client: 20 debug mgr: 20 debug ms: 1 mon warn on pool no app: false mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: client: client_shutdown_timeout: 120 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - Replacing daemon mds - SLOW_OPS - slow request sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: cephfs_name: a client.1: cephfs_name: b install: ceph: flavor: default sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf kclient: client.0: cephfs_name: a client.1: cephfs_name: b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240916.200549-debug sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf owner: scheduled_pdonnell@teuthology priority: 90 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 476 sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf sleep_before_teardown: 0 subset: 320/512 suite: fs suite_branch: wip-pdonnell-testing-20240916.200549-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGqHXZJ+NevIFOUtaSgOQC+9oVBsa6O9Rg5eTIQLLfUvetEwd3tfyVlJOjEfbxaaZK40s7VE9lTyEb96I5o7e1Q= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNv7q2LI+oN0FRGKoTHLVcqcqEuvW0iqhHW+hk9sWEZKXSSNzGV5hYVcEEbrSgu5znAL9Z/DQQDemv84tZo3HAY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_02:09:51 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T04:34:33.177 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa; will attempt to use it 2024-09-17T04:34:33.178 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa/tasks 2024-09-17T04:34:33.178 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T04:34:33.180 INFO:teuthology.task.internal:Checking packages... 2024-09-17T04:34:33.203 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf' 2024-09-17T04:34:33.203 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T04:34:33.203 INFO:teuthology.packaging:ref: None 2024-09-17T04:34:33.203 INFO:teuthology.packaging:tag: None 2024-09-17T04:34:33.203 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240916.200549-debug 2024-09-17T04:34:33.203 INFO:teuthology.packaging:sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T04:34:33.203 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-pdonnell-testing-20240916.200549-debug 2024-09-17T04:34:33.347 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4989-gce1b342b-1jammy 2024-09-17T04:34:33.348 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T04:34:33.355 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T04:34:33.355 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T04:34:33.372 INFO:teuthology.task.internal:Saving configuration 2024-09-17T04:34:33.392 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T04:34:33.399 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T04:34:33.422 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 04:28:49.574113', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ4DMP/NjMeSkqIRs0JCUVKEDZY0mV1DyD4Xfk+sPUNw'} 2024-09-17T04:34:33.439 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 04:28:49.572484', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID6/QhkFgGVk90gVURHyNP8g3oDu2uYZs43XP+EzIOmO'} 2024-09-17T04:34:33.440 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T04:34:33.446 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-17T04:34:33.446 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-09-17T04:34:33.446 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T04:34:33.531 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f638720b490>, signals=[15]) 2024-09-17T04:34:33.531 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T04:34:33.537 INFO:teuthology.task.internal:Opening connections... 2024-09-17T04:34:33.537 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-09-17T04:34:33.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T04:34:33.617 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-09-17T04:34:33.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T04:34:33.698 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T04:34:33.704 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-09-17T04:34:33.709 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-09-17T04:34:33.710 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-09-17T04:34:33.757 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-17T04:34:33.757 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="Ubuntu" 2024-09-17T04:34:33.757 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="22.04" 2024-09-17T04:34:33.757 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-17T04:34:33.757 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_CODENAME=jammy 2024-09-17T04:34:33.757 INFO:teuthology.orchestra.run.smithi150.stdout:ID=ubuntu 2024-09-17T04:34:33.757 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE=debian 2024-09-17T04:34:33.757 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-17T04:34:33.757 INFO:teuthology.orchestra.run.smithi150.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-17T04:34:33.757 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-17T04:34:33.758 INFO:teuthology.orchestra.run.smithi150.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-17T04:34:33.758 INFO:teuthology.orchestra.run.smithi150.stdout:UBUNTU_CODENAME=jammy 2024-09-17T04:34:33.758 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-09-17T04:34:33.770 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-09-17T04:34:33.785 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-09-17T04:34:33.786 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-09-17T04:34:33.838 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-17T04:34:33.838 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="Ubuntu" 2024-09-17T04:34:33.838 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="22.04" 2024-09-17T04:34:33.838 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-17T04:34:33.838 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_CODENAME=jammy 2024-09-17T04:34:33.838 INFO:teuthology.orchestra.run.smithi176.stdout:ID=ubuntu 2024-09-17T04:34:33.838 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE=debian 2024-09-17T04:34:33.838 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-17T04:34:33.838 INFO:teuthology.orchestra.run.smithi176.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-17T04:34:33.838 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-17T04:34:33.838 INFO:teuthology.orchestra.run.smithi176.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-17T04:34:33.838 INFO:teuthology.orchestra.run.smithi176.stdout:UBUNTU_CODENAME=jammy 2024-09-17T04:34:33.839 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-09-17T04:34:33.862 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T04:34:33.871 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T04:34:33.885 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T04:34:33.885 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-09-17T04:34:33.888 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-09-17T04:34:33.892 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T04:34:33.899 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T04:34:33.899 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-09-17T04:34:33.935 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-09-17T04:34:33.951 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T04:34:34.004 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T04:34:34.018 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T04:34:34.018 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T04:34:34.018 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T04:34:34.019 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T04:34:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:34:34.023 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-09-17T04:34:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:34:34.024 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-09-17T04:34:34.070 INFO:teuthology.orchestra.run.smithi150.stdout:5.15.0-56-generic 2024-09-17T04:34:34.070 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.15.0-56-generic 2024-09-17T04:34:34.070 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get clean 2024-09-17T04:34:34.072 INFO:teuthology.orchestra.run.smithi176.stdout:5.15.0-56-generic 2024-09-17T04:34:34.072 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.15.0-56-generic 2024-09-17T04:34:34.072 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get clean 2024-09-17T04:34:34.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2024-09-17T04:34:34.204 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2024-09-17T04:34:34.473 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T04:34:34.474 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-17T04:34:34.484 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T04:34:34.484 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-17T04:34:34.525 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-17T04:34:34.535 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-17T04:34:34.882 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-17T04:34:34.889 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-17T04:34:34.897 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-17T04:34:34.909 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-17T04:34:35.022 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [701 kB] 2024-09-17T04:34:35.027 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [701 kB] 2024-09-17T04:34:35.203 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-17T04:34:35.212 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-17T04:34:35.217 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,058 kB] 2024-09-17T04:34:35.222 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,058 kB] 2024-09-17T04:34:35.377 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-17T04:34:35.378 INFO:teuthology.orchestra.run.smithi150.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [355 kB] 2024-09-17T04:34:35.385 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [355 kB] 2024-09-17T04:34:35.385 INFO:teuthology.orchestra.run.smithi150.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-17T04:34:35.390 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-17T04:34:35.391 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-17T04:34:35.391 INFO:teuthology.orchestra.run.smithi150.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-17T04:34:35.392 INFO:teuthology.orchestra.run.smithi150.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-17T04:34:35.393 INFO:teuthology.orchestra.run.smithi150.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,495 kB] 2024-09-17T04:34:35.399 INFO:teuthology.orchestra.run.smithi150.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-17T04:34:35.399 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-17T04:34:35.399 INFO:teuthology.orchestra.run.smithi176.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,495 kB] 2024-09-17T04:34:35.399 INFO:teuthology.orchestra.run.smithi150.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-17T04:34:35.466 INFO:teuthology.orchestra.run.smithi150.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [429 kB] 2024-09-17T04:34:35.469 INFO:teuthology.orchestra.run.smithi176.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-17T04:34:35.469 INFO:teuthology.orchestra.run.smithi176.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [429 kB] 2024-09-17T04:34:35.491 INFO:teuthology.orchestra.run.smithi150.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-17T04:34:35.495 INFO:teuthology.orchestra.run.smithi176.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-17T04:34:35.495 INFO:teuthology.orchestra.run.smithi176.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-17T04:34:35.495 INFO:teuthology.orchestra.run.smithi176.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-17T04:34:35.496 INFO:teuthology.orchestra.run.smithi176.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-17T04:34:35.498 INFO:teuthology.orchestra.run.smithi150.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-17T04:34:35.504 INFO:teuthology.orchestra.run.smithi176.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-17T04:34:35.504 INFO:teuthology.orchestra.run.smithi176.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-17T04:34:35.508 INFO:teuthology.orchestra.run.smithi176.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-17T04:34:35.521 INFO:teuthology.orchestra.run.smithi176.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-17T04:34:35.535 INFO:teuthology.orchestra.run.smithi176.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-17T04:34:35.536 INFO:teuthology.orchestra.run.smithi150.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-17T04:34:35.536 INFO:teuthology.orchestra.run.smithi150.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-17T04:34:35.538 INFO:teuthology.orchestra.run.smithi176.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-17T04:34:35.540 INFO:teuthology.orchestra.run.smithi176.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-17T04:34:35.541 INFO:teuthology.orchestra.run.smithi150.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-17T04:34:35.541 INFO:teuthology.orchestra.run.smithi176.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-17T04:34:35.542 INFO:teuthology.orchestra.run.smithi176.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-17T04:34:35.543 INFO:teuthology.orchestra.run.smithi176.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-17T04:34:35.548 INFO:teuthology.orchestra.run.smithi176.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-17T04:34:35.549 INFO:teuthology.orchestra.run.smithi176.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-17T04:34:35.549 INFO:teuthology.orchestra.run.smithi150.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-17T04:34:35.550 INFO:teuthology.orchestra.run.smithi150.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-17T04:34:35.559 INFO:teuthology.orchestra.run.smithi150.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-17T04:34:35.559 INFO:teuthology.orchestra.run.smithi150.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-17T04:34:35.572 INFO:teuthology.orchestra.run.smithi150.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-17T04:34:35.573 INFO:teuthology.orchestra.run.smithi150.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-17T04:34:35.575 INFO:teuthology.orchestra.run.smithi150.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-17T04:34:35.577 INFO:teuthology.orchestra.run.smithi150.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-17T04:34:35.589 INFO:teuthology.orchestra.run.smithi150.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-17T04:34:35.611 INFO:teuthology.orchestra.run.smithi150.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-17T04:34:35.613 INFO:teuthology.orchestra.run.smithi150.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-17T04:34:35.617 INFO:teuthology.orchestra.run.smithi150.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-17T04:34:35.623 INFO:teuthology.orchestra.run.smithi176.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-17T04:34:35.623 INFO:teuthology.orchestra.run.smithi150.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-17T04:34:35.623 INFO:teuthology.orchestra.run.smithi176.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-17T04:34:35.624 INFO:teuthology.orchestra.run.smithi176.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-17T04:34:35.624 INFO:teuthology.orchestra.run.smithi176.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-17T04:34:35.624 INFO:teuthology.orchestra.run.smithi150.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-17T04:34:35.625 INFO:teuthology.orchestra.run.smithi176.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-17T04:34:35.625 INFO:teuthology.orchestra.run.smithi176.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-17T04:34:35.627 INFO:teuthology.orchestra.run.smithi176.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-17T04:34:35.629 INFO:teuthology.orchestra.run.smithi150.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-17T04:34:35.629 INFO:teuthology.orchestra.run.smithi176.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-17T04:34:35.630 INFO:teuthology.orchestra.run.smithi176.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-17T04:34:35.631 INFO:teuthology.orchestra.run.smithi176.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-17T04:34:35.631 INFO:teuthology.orchestra.run.smithi150.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-17T04:34:35.632 INFO:teuthology.orchestra.run.smithi150.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-17T04:34:35.634 INFO:teuthology.orchestra.run.smithi150.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-17T04:34:35.705 INFO:teuthology.orchestra.run.smithi150.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-17T04:34:35.705 INFO:teuthology.orchestra.run.smithi150.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-17T04:34:35.706 INFO:teuthology.orchestra.run.smithi150.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-17T04:34:35.706 INFO:teuthology.orchestra.run.smithi150.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-17T04:34:35.707 INFO:teuthology.orchestra.run.smithi150.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-17T04:34:35.711 INFO:teuthology.orchestra.run.smithi176.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-17T04:34:35.712 INFO:teuthology.orchestra.run.smithi176.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-17T04:34:41.631 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 16.1 MB in 3s (5,849 kB/s) 2024-09-17T04:34:41.635 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 16.1 MB in 3s (5,815 kB/s) 2024-09-17T04:34:42.965 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-17T04:34:42.984 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-17T04:34:43.051 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-17T04:34:43.127 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-17T04:34:43.145 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-17T04:34:43.210 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-17T04:34:43.261 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2024-09-17T04:34:43.261 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2024-09-17T04:34:43.422 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2024-09-17T04:34:43.422 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2024-09-17T04:34:43.450 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T04:34:43.450 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T04:34:43.450 INFO:teuthology.orchestra.run.smithi176.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T04:34:43.450 INFO:teuthology.orchestra.run.smithi176.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T04:34:43.450 INFO:teuthology.orchestra.run.smithi176.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-17T04:34:43.451 INFO:teuthology.orchestra.run.smithi176.stdout: python2.7-minimal 2024-09-17T04:34:43.451 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T04:34:43.452 INFO:teuthology.orchestra.run.smithi176.stdout:The following additional packages will be installed: 2024-09-17T04:34:43.452 INFO:teuthology.orchestra.run.smithi176.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-17T04:34:43.452 INFO:teuthology.orchestra.run.smithi176.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-17T04:34:43.453 INFO:teuthology.orchestra.run.smithi176.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-17T04:34:43.454 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2024-09-17T04:34:43.454 INFO:teuthology.orchestra.run.smithi176.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-17T04:34:43.454 INFO:teuthology.orchestra.run.smithi176.stdout:Recommended packages: 2024-09-17T04:34:43.454 INFO:teuthology.orchestra.run.smithi176.stdout: thermald 2024-09-17T04:34:43.492 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2024-09-17T04:34:43.492 INFO:teuthology.orchestra.run.smithi176.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-17T04:34:43.492 INFO:teuthology.orchestra.run.smithi176.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-17T04:34:43.493 INFO:teuthology.orchestra.run.smithi176.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-17T04:34:43.494 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages will be upgraded: 2024-09-17T04:34:43.495 INFO:teuthology.orchestra.run.smithi176.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-17T04:34:43.547 INFO:teuthology.orchestra.run.smithi176.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-17T04:34:43.547 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 113 MB of archives. 2024-09-17T04:34:43.547 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-17T04:34:43.547 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-17T04:34:43.613 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T04:34:43.613 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T04:34:43.613 INFO:teuthology.orchestra.run.smithi150.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T04:34:43.614 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T04:34:43.614 INFO:teuthology.orchestra.run.smithi150.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-17T04:34:43.614 INFO:teuthology.orchestra.run.smithi150.stdout: python2.7-minimal 2024-09-17T04:34:43.614 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T04:34:43.615 INFO:teuthology.orchestra.run.smithi150.stdout:The following additional packages will be installed: 2024-09-17T04:34:43.616 INFO:teuthology.orchestra.run.smithi150.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-17T04:34:43.616 INFO:teuthology.orchestra.run.smithi150.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-17T04:34:43.616 INFO:teuthology.orchestra.run.smithi150.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-17T04:34:43.617 INFO:teuthology.orchestra.run.smithi150.stdout:Suggested packages: 2024-09-17T04:34:43.617 INFO:teuthology.orchestra.run.smithi150.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-17T04:34:43.618 INFO:teuthology.orchestra.run.smithi150.stdout:Recommended packages: 2024-09-17T04:34:43.618 INFO:teuthology.orchestra.run.smithi150.stdout: thermald 2024-09-17T04:34:43.665 INFO:teuthology.orchestra.run.smithi150.stdout:The following NEW packages will be installed: 2024-09-17T04:34:43.666 INFO:teuthology.orchestra.run.smithi150.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-17T04:34:43.666 INFO:teuthology.orchestra.run.smithi150.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-17T04:34:43.666 INFO:teuthology.orchestra.run.smithi150.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-17T04:34:43.667 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages will be upgraded: 2024-09-17T04:34:43.668 INFO:teuthology.orchestra.run.smithi150.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-17T04:34:43.848 INFO:teuthology.orchestra.run.smithi150.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-17T04:34:43.848 INFO:teuthology.orchestra.run.smithi150.stdout:Need to get 113 MB of archives. 2024-09-17T04:34:43.849 INFO:teuthology.orchestra.run.smithi150.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-17T04:34:43.849 INFO:teuthology.orchestra.run.smithi150.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-17T04:34:43.980 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-17T04:34:44.068 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-17T04:34:44.866 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-17T04:34:44.866 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-17T04:34:44.866 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-17T04:34:44.988 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-17T04:34:45.008 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-17T04:34:45.292 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-17T04:34:45.475 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 113 MB in 2s (72.2 MB/s) 2024-09-17T04:34:45.642 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-17T04:34:45.668 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-17T04:34:47.646 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-17T04:34:47.651 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T04:34:47.856 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:34:47.917 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-17T04:34:47.917 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-17T04:34:47.918 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-17T04:34:48.354 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-17T04:34:48.452 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-17T04:34:48.852 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 113 MB in 5s (23.7 MB/s) 2024-09-17T04:34:49.026 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-17T04:34:50.190 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-17T04:34:50.211 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T04:34:50.293 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:34:50.760 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-17T04:34:50.780 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T04:34:50.818 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:34:51.107 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-17T04:34:51.111 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T04:34:51.215 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:34:53.742 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-17T04:34:53.763 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T04:34:53.849 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:34:54.370 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-17T04:34:54.391 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T04:34:54.428 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:34:57.600 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T04:34:57.725 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T04:34:58.099 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T04:34:58.237 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T04:34:58.540 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-17T04:34:58.562 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-17T04:34:58.606 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-17T04:35:01.056 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T04:35:01.184 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T04:35:01.514 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T04:35:01.626 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T04:35:01.963 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-17T04:35:01.985 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-17T04:35:02.046 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-17T04:35:04.817 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-17T04:35:04.854 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T04:35:04.891 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:35:07.484 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T04:35:07.610 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T04:35:07.960 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-17T04:35:08.081 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:35:08.134 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-17T04:35:08.171 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T04:35:08.207 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-17T04:35:08.209 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:35:08.336 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:35:09.646 INFO:teuthology.orchestra.run.smithi176.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-17T04:35:09.647 INFO:teuthology.orchestra.run.smithi176.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-17T04:35:09.647 INFO:teuthology.orchestra.run.smithi176.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-17T04:35:09.647 INFO:teuthology.orchestra.run.smithi176.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-17T04:35:09.812 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:35:11.339 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-17T04:35:11.465 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-17T04:35:11.590 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:35:12.124 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T04:35:12.247 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T04:35:12.622 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-17T04:35:12.735 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:35:12.861 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-17T04:35:12.959 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:35:12.967 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:35:13.014 INFO:teuthology.orchestra.run.smithi176.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-17T04:35:13.015 INFO:teuthology.orchestra.run.smithi176.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-17T04:35:14.291 INFO:teuthology.orchestra.run.smithi150.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-17T04:35:14.291 INFO:teuthology.orchestra.run.smithi150.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-17T04:35:14.292 INFO:teuthology.orchestra.run.smithi150.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-17T04:35:14.292 INFO:teuthology.orchestra.run.smithi150.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-17T04:35:14.449 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:35:15.884 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-17T04:35:16.878 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-17T04:35:16.986 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:35:18.347 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T04:35:18.423 INFO:teuthology.orchestra.run.smithi150.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-17T04:35:18.423 INFO:teuthology.orchestra.run.smithi150.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-17T04:35:26.995 INFO:teuthology.orchestra.run.smithi176.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-17T04:35:26.995 INFO:teuthology.orchestra.run.smithi176.stdout:Sourcing file `/etc/default/grub' 2024-09-17T04:35:27.010 INFO:teuthology.orchestra.run.smithi176.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-17T04:35:27.030 INFO:teuthology.orchestra.run.smithi176.stdout:Generating grub configuration file ... 2024-09-17T04:35:27.307 INFO:teuthology.orchestra.run.smithi176.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-17T04:35:27.327 INFO:teuthology.orchestra.run.smithi176.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-17T04:35:27.612 INFO:teuthology.orchestra.run.smithi176.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-17T04:35:27.617 INFO:teuthology.orchestra.run.smithi176.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-17T04:35:27.685 INFO:teuthology.orchestra.run.smithi176.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-17T04:35:27.690 INFO:teuthology.orchestra.run.smithi176.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-17T04:35:27.882 INFO:teuthology.orchestra.run.smithi176.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-17T04:35:27.882 INFO:teuthology.orchestra.run.smithi176.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-17T04:35:27.882 INFO:teuthology.orchestra.run.smithi176.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-17T04:35:27.930 INFO:teuthology.orchestra.run.smithi176.stdout:done 2024-09-17T04:35:28.406 DEBUG:teuthology.orchestra.run.smithi176:> dpkg -s linux-image-generic 2024-09-17T04:35:28.423 INFO:teuthology.orchestra.run.smithi176.stdout:Package: linux-image-generic 2024-09-17T04:35:28.423 INFO:teuthology.orchestra.run.smithi176.stdout:Status: install ok installed 2024-09-17T04:35:28.423 INFO:teuthology.orchestra.run.smithi176.stdout:Priority: optional 2024-09-17T04:35:28.423 INFO:teuthology.orchestra.run.smithi176.stdout:Section: kernel 2024-09-17T04:35:28.423 INFO:teuthology.orchestra.run.smithi176.stdout:Installed-Size: 21 2024-09-17T04:35:28.423 INFO:teuthology.orchestra.run.smithi176.stdout:Maintainer: Ubuntu Kernel Team 2024-09-17T04:35:28.423 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: amd64 2024-09-17T04:35:28.424 INFO:teuthology.orchestra.run.smithi176.stdout:Source: linux-meta 2024-09-17T04:35:28.424 INFO:teuthology.orchestra.run.smithi176.stdout:Version: 5.15.0.122.122 2024-09-17T04:35:28.424 INFO:teuthology.orchestra.run.smithi176.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-17T04:35:28.424 INFO:teuthology.orchestra.run.smithi176.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-17T04:35:28.424 INFO:teuthology.orchestra.run.smithi176.stdout:Recommends: thermald 2024-09-17T04:35:28.424 INFO:teuthology.orchestra.run.smithi176.stdout:Description: Generic Linux kernel image 2024-09-17T04:35:28.424 INFO:teuthology.orchestra.run.smithi176.stdout: This package will always depend on the latest generic kernel image 2024-09-17T04:35:28.424 INFO:teuthology.orchestra.run.smithi176.stdout: available. 2024-09-17T04:35:28.424 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-17T04:35:28.424 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T04:35:28.424 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T04:35:28.425 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-17T04:35:28.425 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T04:35:28.425 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get clean 2024-09-17T04:35:28.611 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2024-09-17T04:35:28.813 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T04:35:28.813 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-17T04:35:28.817 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-17T04:35:28.834 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-17T04:35:30.313 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-17T04:35:30.333 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-17T04:35:30.401 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-17T04:35:30.613 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2024-09-17T04:35:30.613 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2024-09-17T04:35:30.803 INFO:teuthology.orchestra.run.smithi176.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-17T04:35:30.803 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T04:35:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T04:35:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T04:35:30.803 INFO:teuthology.orchestra.run.smithi176.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T04:35:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-17T04:35:30.804 INFO:teuthology.orchestra.run.smithi176.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-17T04:35:30.804 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T04:35:30.847 INFO:teuthology.orchestra.run.smithi176.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-17T04:35:30.849 DEBUG:teuthology.orchestra.run.smithi176:> dpkg -s linux-image-generic 2024-09-17T04:35:30.864 INFO:teuthology.orchestra.run.smithi176.stdout:Package: linux-image-generic 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout:Status: install ok installed 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout:Priority: optional 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout:Section: kernel 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout:Installed-Size: 21 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout:Maintainer: Ubuntu Kernel Team 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: amd64 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout:Source: linux-meta 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout:Version: 5.15.0.122.122 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout:Recommends: thermald 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout:Description: Generic Linux kernel image 2024-09-17T04:35:30.865 INFO:teuthology.orchestra.run.smithi176.stdout: This package will always depend on the latest generic kernel image 2024-09-17T04:35:30.866 INFO:teuthology.orchestra.run.smithi176.stdout: available. 2024-09-17T04:35:30.866 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-09-17T04:35:30.910 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.fHwcJilJoq 2024-09-17T04:35:30.910 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub/grub.cfg /tmp/tmp.fHwcJilJoq 2024-09-17T04:35:30.966 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.fHwcJilJoq 2024-09-17T04:35:31.097 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.fHwcJilJoq is 10KB 2024-09-17T04:35:31.110 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.fHwcJilJoq 2024-09-17T04:35:31.116 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-17T04:35:31.170 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:35:31.170 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-17T04:35:31.170 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-17T04:35:31.237 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-17T04:35:31.237 DEBUG:teuthology.orchestra.run.smithi176:> sudo update-grub 2024-09-17T04:35:32.434 INFO:teuthology.orchestra.run.smithi176.stderr:Sourcing file `/etc/default/grub' 2024-09-17T04:35:32.435 INFO:teuthology.orchestra.run.smithi176.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-17T04:35:32.437 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-09-17T04:35:32.521 INFO:teuthology.orchestra.run.smithi150.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-17T04:35:32.521 INFO:teuthology.orchestra.run.smithi150.stdout:Sourcing file `/etc/default/grub' 2024-09-17T04:35:32.549 INFO:teuthology.orchestra.run.smithi150.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-17T04:35:32.592 INFO:teuthology.orchestra.run.smithi150.stdout:Generating grub configuration file ... 2024-09-17T04:35:32.620 INFO:teuthology.orchestra.run.smithi176.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-17T04:35:32.626 INFO:teuthology.orchestra.run.smithi176.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-17T04:35:32.851 INFO:teuthology.orchestra.run.smithi176.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-17T04:35:32.856 INFO:teuthology.orchestra.run.smithi176.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-17T04:35:32.866 INFO:teuthology.orchestra.run.smithi150.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-17T04:35:32.893 INFO:teuthology.orchestra.run.smithi150.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-17T04:35:32.896 INFO:teuthology.orchestra.run.smithi176.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-17T04:35:32.901 INFO:teuthology.orchestra.run.smithi176.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-17T04:35:33.037 INFO:teuthology.orchestra.run.smithi176.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-17T04:35:33.037 INFO:teuthology.orchestra.run.smithi176.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-17T04:35:33.037 INFO:teuthology.orchestra.run.smithi176.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-17T04:35:33.053 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-09-17T04:35:33.061 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-09-17T04:35:33.247 INFO:teuthology.orchestra.run.smithi150.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-17T04:35:33.252 INFO:teuthology.orchestra.run.smithi150.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-17T04:35:33.319 INFO:teuthology.orchestra.run.smithi150.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-17T04:35:33.325 INFO:teuthology.orchestra.run.smithi150.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-17T04:35:33.493 INFO:teuthology.orchestra.run.smithi150.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-17T04:35:33.493 INFO:teuthology.orchestra.run.smithi150.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-17T04:35:33.493 INFO:teuthology.orchestra.run.smithi150.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-17T04:35:33.538 INFO:teuthology.orchestra.run.smithi150.stdout:done 2024-09-17T04:35:34.074 DEBUG:teuthology.orchestra.run.smithi150:> dpkg -s linux-image-generic 2024-09-17T04:35:34.097 INFO:teuthology.orchestra.run.smithi150.stdout:Package: linux-image-generic 2024-09-17T04:35:34.097 INFO:teuthology.orchestra.run.smithi150.stdout:Status: install ok installed 2024-09-17T04:35:34.097 INFO:teuthology.orchestra.run.smithi150.stdout:Priority: optional 2024-09-17T04:35:34.097 INFO:teuthology.orchestra.run.smithi150.stdout:Section: kernel 2024-09-17T04:35:34.097 INFO:teuthology.orchestra.run.smithi150.stdout:Installed-Size: 21 2024-09-17T04:35:34.097 INFO:teuthology.orchestra.run.smithi150.stdout:Maintainer: Ubuntu Kernel Team 2024-09-17T04:35:34.097 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: amd64 2024-09-17T04:35:34.097 INFO:teuthology.orchestra.run.smithi150.stdout:Source: linux-meta 2024-09-17T04:35:34.097 INFO:teuthology.orchestra.run.smithi150.stdout:Version: 5.15.0.122.122 2024-09-17T04:35:34.097 INFO:teuthology.orchestra.run.smithi150.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-17T04:35:34.098 INFO:teuthology.orchestra.run.smithi150.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-17T04:35:34.098 INFO:teuthology.orchestra.run.smithi150.stdout:Recommends: thermald 2024-09-17T04:35:34.098 INFO:teuthology.orchestra.run.smithi150.stdout:Description: Generic Linux kernel image 2024-09-17T04:35:34.098 INFO:teuthology.orchestra.run.smithi150.stdout: This package will always depend on the latest generic kernel image 2024-09-17T04:35:34.098 INFO:teuthology.orchestra.run.smithi150.stdout: available. 2024-09-17T04:35:34.098 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-17T04:35:34.098 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T04:35:34.098 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T04:35:34.098 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T04:35:34.098 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T04:35:34.099 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get clean 2024-09-17T04:35:34.276 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2024-09-17T04:35:34.394 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-17T04:35:34.516 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T04:35:34.604 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-17T04:35:34.678 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-17T04:35:35.879 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-17T04:35:35.897 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-17T04:35:35.951 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-17T04:35:36.146 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2024-09-17T04:35:36.146 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2024-09-17T04:35:36.316 INFO:teuthology.orchestra.run.smithi150.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-17T04:35:36.317 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T04:35:36.317 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T04:35:36.317 INFO:teuthology.orchestra.run.smithi150.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T04:35:36.317 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T04:35:36.318 INFO:teuthology.orchestra.run.smithi150.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-17T04:35:36.318 INFO:teuthology.orchestra.run.smithi150.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-17T04:35:36.318 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T04:35:36.358 INFO:teuthology.orchestra.run.smithi150.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-17T04:35:36.360 DEBUG:teuthology.orchestra.run.smithi150:> dpkg -s linux-image-generic 2024-09-17T04:35:36.415 INFO:teuthology.orchestra.run.smithi150.stdout:Package: linux-image-generic 2024-09-17T04:35:36.415 INFO:teuthology.orchestra.run.smithi150.stdout:Status: install ok installed 2024-09-17T04:35:36.415 INFO:teuthology.orchestra.run.smithi150.stdout:Priority: optional 2024-09-17T04:35:36.415 INFO:teuthology.orchestra.run.smithi150.stdout:Section: kernel 2024-09-17T04:35:36.416 INFO:teuthology.orchestra.run.smithi150.stdout:Installed-Size: 21 2024-09-17T04:35:36.416 INFO:teuthology.orchestra.run.smithi150.stdout:Maintainer: Ubuntu Kernel Team 2024-09-17T04:35:36.416 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: amd64 2024-09-17T04:35:36.416 INFO:teuthology.orchestra.run.smithi150.stdout:Source: linux-meta 2024-09-17T04:35:36.416 INFO:teuthology.orchestra.run.smithi150.stdout:Version: 5.15.0.122.122 2024-09-17T04:35:36.416 INFO:teuthology.orchestra.run.smithi150.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-17T04:35:36.416 INFO:teuthology.orchestra.run.smithi150.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-17T04:35:36.416 INFO:teuthology.orchestra.run.smithi150.stdout:Recommends: thermald 2024-09-17T04:35:36.416 INFO:teuthology.orchestra.run.smithi150.stdout:Description: Generic Linux kernel image 2024-09-17T04:35:36.416 INFO:teuthology.orchestra.run.smithi150.stdout: This package will always depend on the latest generic kernel image 2024-09-17T04:35:36.416 INFO:teuthology.orchestra.run.smithi150.stdout: available. 2024-09-17T04:35:36.416 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-09-17T04:35:36.461 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.6W0on5Bxk3 2024-09-17T04:35:36.461 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub/grub.cfg /tmp/tmp.6W0on5Bxk3 2024-09-17T04:35:36.516 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.6W0on5Bxk3 2024-09-17T04:35:36.629 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.6W0on5Bxk3 is 10KB 2024-09-17T04:35:36.641 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.6W0on5Bxk3 2024-09-17T04:35:36.647 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-17T04:35:36.701 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:35:36.701 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-17T04:35:36.701 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-17T04:35:36.768 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-17T04:35:36.768 DEBUG:teuthology.orchestra.run.smithi150:> sudo update-grub 2024-09-17T04:35:38.125 INFO:teuthology.orchestra.run.smithi150.stderr:Sourcing file `/etc/default/grub' 2024-09-17T04:35:38.126 INFO:teuthology.orchestra.run.smithi150.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-17T04:35:38.130 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2024-09-17T04:35:38.313 INFO:teuthology.orchestra.run.smithi150.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-17T04:35:38.318 INFO:teuthology.orchestra.run.smithi150.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-17T04:35:38.615 INFO:teuthology.orchestra.run.smithi150.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-17T04:35:38.620 INFO:teuthology.orchestra.run.smithi150.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-17T04:35:38.659 INFO:teuthology.orchestra.run.smithi150.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-17T04:35:38.665 INFO:teuthology.orchestra.run.smithi150.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-17T04:35:38.785 INFO:teuthology.orchestra.run.smithi150.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-17T04:35:38.785 INFO:teuthology.orchestra.run.smithi150.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-17T04:35:38.785 INFO:teuthology.orchestra.run.smithi150.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-17T04:35:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:done 2024-09-17T04:35:38.816 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2024-09-17T04:36:03.085 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-17T04:36:03.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-17T04:36:03.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T04:36:08.825 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T04:36:08.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-17T04:36:08.826 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T04:36:21.437 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-09-17T04:36:27.329 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-09-17T04:36:30.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-17T04:36:30.442 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T04:36:33.501 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-09-17T04:36:36.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-17T04:36:36.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T04:36:39.393 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-09-17T04:36:45.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-17T04:36:45.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T04:36:51.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-17T04:36:51.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T04:37:16.994 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-09-17T04:37:32.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-17T04:37:32.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T04:37:32.273 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-09-17T04:37:32.893 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-17T04:37:32.893 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-122-generic"... 2024-09-17T04:37:32.893 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-09-17T04:37:32.941 INFO:teuthology.orchestra.run.smithi176.stdout:5.15.0-122-generic 2024-09-17T04:37:32.941 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-17T04:37:32.941 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T04:37:32.941 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-17T04:37:33.942 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-17T04:37:33.942 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T04:37:34.012 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-09-17T04:37:34.034 DEBUG:teuthology.parallel:result is None 2024-09-17T04:37:51.401 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T04:38:06.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-17T04:38:06.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T04:38:06.632 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-09-17T04:38:07.301 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-17T04:38:07.301 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-122-generic"... 2024-09-17T04:38:07.302 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-09-17T04:38:07.349 INFO:teuthology.orchestra.run.smithi150.stdout:5.15.0-122-generic 2024-09-17T04:38:07.349 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-17T04:38:07.349 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T04:38:07.349 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-17T04:38:08.350 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T04:38:08.351 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T04:38:08.721 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2024-09-17T04:38:08.742 DEBUG:teuthology.parallel:result is None 2024-09-17T04:38:08.743 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T04:38:08.750 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T04:38:08.750 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T04:38:08.752 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T04:38:08.759 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T04:38:08.786 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T04:38:08.797 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T04:38:08.797 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T04:38:08.799 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T04:38:08.831 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T04:38:08.837 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T04:38:08.838 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T04:38:08.858 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T04:38:08.876 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T04:38:08.884 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T04:38:08.889 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T04:38:08.896 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T04:38:08.898 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T04:38:08.903 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T04:38:08.904 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T04:38:08.934 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T04:38:08.961 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T04:38:08.970 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T04:38:08.970 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T04:38:08.994 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T04:38:09.008 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T04:38:09.045 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T04:38:09.093 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:38:09.093 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T04:38:09.163 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T04:38:09.169 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T04:38:09.217 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:38:09.217 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T04:38:09.275 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-09-17T04:38:09.277 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-09-17T04:38:09.348 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T04:38:09.355 INFO:teuthology.task.internal:Starting timer... 2024-09-17T04:38:09.355 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T04:38:09.400 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T04:38:09.409 DEBUG:teuthology.task.selinux:Excluding smithi150: OS 'ubuntu' does not support SELinux 2024-09-17T04:38:09.409 DEBUG:teuthology.task.selinux:Excluding smithi176: OS 'ubuntu' does not support SELinux 2024-09-17T04:38:09.409 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T04:38:09.409 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-17T04:38:09.409 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T04:38:09.409 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T04:38:09.417 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T04:38:09.428 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T04:38:09.429 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi150.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T04:45:39.760 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2024-09-17T04:45:39.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-17T04:45:39.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T04:45:39.848 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-09-17T04:45:39.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-09-17T04:45:39.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-17T04:45:39.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T04:45:40.014 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-09-17T04:45:40.092 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-17T04:45:40.093 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T04:45:40.103 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T04:45:40.103 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T04:45:40.104 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T04:45:40.109 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T04:45:40.109 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T04:45:40.131 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-17T04:45:40.131 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: Command line: ntpd -gq 2024-09-17T04:45:40.131 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: ---------------------------------------------------- 2024-09-17T04:45:40.131 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: ntp-4 is maintained by Network Time Foundation, 2024-09-17T04:45:40.132 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-17T04:45:40.132 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: corporation. Support and training for ntp-4 are 2024-09-17T04:45:40.132 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: available at https://www.nwtime.org/support 2024-09-17T04:45:40.132 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: ---------------------------------------------------- 2024-09-17T04:45:40.132 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: proto: precision = 0.043 usec (-24) 2024-09-17T04:45:40.132 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: basedate set to 2022-02-04 2024-09-17T04:45:40.133 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: gps base set to 2022-02-06 (week 2196) 2024-09-17T04:45:40.133 INFO:teuthology.orchestra.run.smithi150.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-17T04:45:40.133 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-17T04:45:40.133 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: restrict ::: KOD does nothing without LIMITED. 2024-09-17T04:45:40.134 INFO:teuthology.orchestra.run.smithi150.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-17T04:45:40.134 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: Listen and drop on 0 v6wildcard [::]:123 2024-09-17T04:45:40.134 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-17T04:45:40.134 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: Listen normally on 2 lo 127.0.0.1:123 2024-09-17T04:45:40.135 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: Listen normally on 3 enp3s0f1 172.21.15.150:123 2024-09-17T04:45:40.135 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: Listen normally on 4 lo [::1]:123 2024-09-17T04:45:40.135 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e35%5]:123 2024-09-17T04:45:40.135 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:40 ntpd[18465]: Listening on routing socket on fd #22 for interface updates 2024-09-17T04:45:40.165 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-17T04:45:40.165 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: Command line: ntpd -gq 2024-09-17T04:45:40.165 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: ---------------------------------------------------- 2024-09-17T04:45:40.165 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: ntp-4 is maintained by Network Time Foundation, 2024-09-17T04:45:40.165 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-17T04:45:40.165 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: corporation. Support and training for ntp-4 are 2024-09-17T04:45:40.165 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: available at https://www.nwtime.org/support 2024-09-17T04:45:40.165 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: ---------------------------------------------------- 2024-09-17T04:45:40.166 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: proto: precision = 0.046 usec (-24) 2024-09-17T04:45:40.166 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: basedate set to 2022-02-04 2024-09-17T04:45:40.166 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: gps base set to 2022-02-06 (week 2196) 2024-09-17T04:45:40.166 INFO:teuthology.orchestra.run.smithi176.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-17T04:45:40.167 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-17T04:45:40.167 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: restrict ::: KOD does nothing without LIMITED. 2024-09-17T04:45:40.167 INFO:teuthology.orchestra.run.smithi176.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-17T04:45:40.168 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: Listen and drop on 0 v6wildcard [::]:123 2024-09-17T04:45:40.168 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-17T04:45:40.168 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: Listen normally on 2 lo 127.0.0.1:123 2024-09-17T04:45:40.169 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: Listen normally on 3 enp3s0f1 172.21.15.176:123 2024-09-17T04:45:40.169 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: Listen normally on 4 lo [::1]:123 2024-09-17T04:45:40.169 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f61%5]:123 2024-09-17T04:45:40.169 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:40 ntpd[18466]: Listening on routing socket on fd #22 for interface updates 2024-09-17T04:45:41.132 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:41 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:41.132 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:41 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:41.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:41 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:41.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:41 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:41.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:41 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:41.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:41 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:41.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:41 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:41.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:41 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:41.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:41 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:41.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:41 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:41.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:41 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:41.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:41 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:41.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:41 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:41.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:41 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:41.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:41 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:41.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:41 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:41.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:41 ntpd[18466]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:41.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:41 ntpd[18466]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:42.132 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:42 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:42.132 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:42 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:42.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:42 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:42.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:42 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:42.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:42 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:42.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:42 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:42.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:42 ntpd[18466]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:42.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:42 ntpd[18466]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:43.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:43 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:43.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:43 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:43.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:43 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:43.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:43 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:43.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:43 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:43.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:43 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:43.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:43 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:43.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:43 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:43.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:43 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:43.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:43 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:43.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:43 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:43.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:43 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:43.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:43 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:43.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:43 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:43.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:43 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:43.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:43 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:43.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:43 ntpd[18466]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:43.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:43 ntpd[18466]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:44.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:44 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:44.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:44 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:44.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:44 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:44.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:44 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:44.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:44 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:44.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:44 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:44.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:44 ntpd[18466]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:44.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:44 ntpd[18466]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:44.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:44 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:44.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:44 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:44.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:44 ntpd[18466]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:44.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:44 ntpd[18466]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:45.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:45.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:45.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:45.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:45.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:45.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:45 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:45.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:45 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:45.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:45 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:45.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:45 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:45.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:45 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:45.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:45 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:45.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:45 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:45.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:45 ntpd[18466]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:45.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:45 ntpd[18466]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:46.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:46 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:46.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:46 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:46.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:46 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:46.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:46 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:46.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:46 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:46.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:46 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:46.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:46 ntpd[18466]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:46.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:46 ntpd[18466]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:46.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:46 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:46.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:46 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:46.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:46 ntpd[18466]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:46.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:46 ntpd[18466]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:47.132 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:47 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:47.132 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:47 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:47.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:47 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:47.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:47 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:47.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:47 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:47.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:47 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:47.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:47 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:47.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:47 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:47.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:47 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:47.133 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:47 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:47.133 INFO:teuthology.orchestra.run.smithi150.stdout:17 Sep 04:45:47 ntpd[18465]: ntpd: time slew +0.002470 s 2024-09-17T04:45:47.133 INFO:teuthology.orchestra.run.smithi150.stdout:ntpd: time slew +0.002470s 2024-09-17T04:45:47.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:47 ntpd[18465]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-17T04:45:47.134 INFO:teuthology.orchestra.run.smithi150.stderr:17 Sep 04:45:47 ntpd[18465]: can't open /var/log/ntpstats/loopstats.20240917: Permission denied 2024-09-17T04:45:47.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:47 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:47.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:47 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:47.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:47 ntpd[18466]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T04:45:47.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:47 ntpd[18466]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T04:45:47.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:47 ntpd[18466]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T04:45:47.166 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:47 ntpd[18466]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T04:45:47.167 INFO:teuthology.orchestra.run.smithi176.stdout:17 Sep 04:45:47 ntpd[18466]: ntpd: time slew -0.000438 s 2024-09-17T04:45:47.167 INFO:teuthology.orchestra.run.smithi176.stdout:ntpd: time slew -0.000438s 2024-09-17T04:45:47.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:47 ntpd[18466]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-17T04:45:47.167 INFO:teuthology.orchestra.run.smithi176.stderr:17 Sep 04:45:47 ntpd[18466]: can't open /var/log/ntpstats/loopstats.20240917: Permission denied 2024-09-17T04:45:47.189 INFO:teuthology.orchestra.run.smithi150.stdout: remote refid st t when poll reach delay offset jitter 2024-09-17T04:45:47.189 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================== 2024-09-17T04:45:47.189 INFO:teuthology.orchestra.run.smithi150.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T04:45:47.189 INFO:teuthology.orchestra.run.smithi150.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T04:45:47.189 INFO:teuthology.orchestra.run.smithi150.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T04:45:47.189 INFO:teuthology.orchestra.run.smithi150.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T04:45:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: remote refid st t when poll reach delay offset jitter 2024-09-17T04:45:47.220 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================== 2024-09-17T04:45:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T04:45:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T04:45:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T04:45:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T04:45:47.221 INFO:teuthology.run_tasks:Running task install... 2024-09-17T04:45:47.229 DEBUG:teuthology.task.install:project ceph 2024-09-17T04:45:47.229 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}} 2024-09-17T04:45:47.229 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'} 2024-09-17T04:45:47.229 INFO:teuthology.task.install:Using flavor: default 2024-09-17T04:45:47.235 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T04:45:47.235 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-17T04:45:47.235 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-key list | grep Ceph 2024-09-17T04:45:47.236 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-key list | grep Ceph 2024-09-17T04:45:47.292 INFO:teuthology.orchestra.run.smithi150.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-17T04:45:47.321 INFO:teuthology.orchestra.run.smithi150.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-17T04:45:47.321 INFO:teuthology.orchestra.run.smithi150.stdout:uid [ unknown] Ceph.com (release key) 2024-09-17T04:45:47.322 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-09-17T04:45:47.322 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-09-17T04:45:47.322 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T04:45:47.327 INFO:teuthology.orchestra.run.smithi176.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-17T04:45:47.357 INFO:teuthology.orchestra.run.smithi176.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-17T04:45:47.357 INFO:teuthology.orchestra.run.smithi176.stdout:uid [ unknown] Ceph.com (release key) 2024-09-17T04:45:47.357 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-09-17T04:45:47.358 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-09-17T04:45:47.358 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T04:45:47.464 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default/ 2024-09-17T04:45:47.464 INFO:teuthology.task.install.deb:Package version is 19.3.0-4989-gce1b342b-1jammy 2024-09-17T04:45:47.501 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default/ 2024-09-17T04:45:47.502 INFO:teuthology.task.install.deb:Package version is 19.3.0-4989-gce1b342b-1jammy 2024-09-17T04:45:47.576 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:45:47.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-17T04:45:47.591 DEBUG:teuthology.orchestra.run.smithi150:> sudo apt-get update 2024-09-17T04:45:47.613 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:45:47.614 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-17T04:45:47.628 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2024-09-17T04:45:47.794 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T04:45:47.795 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-17T04:45:47.805 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-17T04:45:47.812 INFO:teuthology.orchestra.run.smithi150.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy InRelease 2024-09-17T04:45:47.822 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-17T04:45:47.833 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-17T04:45:47.850 INFO:teuthology.orchestra.run.smithi176.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy InRelease 2024-09-17T04:45:47.856 INFO:teuthology.orchestra.run.smithi150.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-17T04:45:47.875 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-17T04:45:47.877 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,420 B] 2024-09-17T04:45:47.896 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T04:45:47.896 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-17T04:45:47.897 INFO:teuthology.orchestra.run.smithi176.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-17T04:45:47.917 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-17T04:45:47.918 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,420 B] 2024-09-17T04:45:47.946 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-17T04:45:47.959 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-17T04:45:48.041 INFO:teuthology.orchestra.run.smithi150.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-17T04:45:48.146 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 29.2 kB in 0s (86.4 kB/s) 2024-09-17T04:45:48.248 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 29.2 kB in 0s (60.7 kB/s) 2024-09-17T04:45:49.169 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-17T04:45:49.184 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4989-gce1b342b-1jammy cephadm=19.3.0-4989-gce1b342b-1jammy ceph-mds=19.3.0-4989-gce1b342b-1jammy ceph-mgr=19.3.0-4989-gce1b342b-1jammy ceph-common=19.3.0-4989-gce1b342b-1jammy ceph-fuse=19.3.0-4989-gce1b342b-1jammy ceph-test=19.3.0-4989-gce1b342b-1jammy ceph-volume=19.3.0-4989-gce1b342b-1jammy radosgw=19.3.0-4989-gce1b342b-1jammy python3-rados=19.3.0-4989-gce1b342b-1jammy python3-rgw=19.3.0-4989-gce1b342b-1jammy python3-cephfs=19.3.0-4989-gce1b342b-1jammy python3-rbd=19.3.0-4989-gce1b342b-1jammy libcephfs2=19.3.0-4989-gce1b342b-1jammy libcephfs-dev=19.3.0-4989-gce1b342b-1jammy librados2=19.3.0-4989-gce1b342b-1jammy librbd1=19.3.0-4989-gce1b342b-1jammy rbd-fuse=19.3.0-4989-gce1b342b-1jammy python3-cephfs=19.3.0-4989-gce1b342b-1jammy cephfs-shell=19.3.0-4989-gce1b342b-1jammy cephfs-top=19.3.0-4989-gce1b342b-1jammy cephfs-mirror=19.3.0-4989-gce1b342b-1jammy 2024-09-17T04:45:49.215 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-17T04:45:49.230 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4989-gce1b342b-1jammy cephadm=19.3.0-4989-gce1b342b-1jammy ceph-mds=19.3.0-4989-gce1b342b-1jammy ceph-mgr=19.3.0-4989-gce1b342b-1jammy ceph-common=19.3.0-4989-gce1b342b-1jammy ceph-fuse=19.3.0-4989-gce1b342b-1jammy ceph-test=19.3.0-4989-gce1b342b-1jammy ceph-volume=19.3.0-4989-gce1b342b-1jammy radosgw=19.3.0-4989-gce1b342b-1jammy python3-rados=19.3.0-4989-gce1b342b-1jammy python3-rgw=19.3.0-4989-gce1b342b-1jammy python3-cephfs=19.3.0-4989-gce1b342b-1jammy python3-rbd=19.3.0-4989-gce1b342b-1jammy libcephfs2=19.3.0-4989-gce1b342b-1jammy libcephfs-dev=19.3.0-4989-gce1b342b-1jammy librados2=19.3.0-4989-gce1b342b-1jammy librbd1=19.3.0-4989-gce1b342b-1jammy rbd-fuse=19.3.0-4989-gce1b342b-1jammy python3-cephfs=19.3.0-4989-gce1b342b-1jammy cephfs-shell=19.3.0-4989-gce1b342b-1jammy cephfs-top=19.3.0-4989-gce1b342b-1jammy cephfs-mirror=19.3.0-4989-gce1b342b-1jammy 2024-09-17T04:45:49.247 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-17T04:45:49.284 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-17T04:45:49.451 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2024-09-17T04:45:49.452 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2024-09-17T04:45:49.470 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2024-09-17T04:45:49.471 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2024-09-17T04:45:49.649 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T04:45:49.650 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T04:45:49.650 INFO:teuthology.orchestra.run.smithi176.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T04:45:49.650 INFO:teuthology.orchestra.run.smithi176.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T04:45:49.650 INFO:teuthology.orchestra.run.smithi176.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-17T04:45:49.650 INFO:teuthology.orchestra.run.smithi176.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-17T04:45:49.650 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T04:45:49.651 INFO:teuthology.orchestra.run.smithi176.stdout:The following additional packages will be installed: 2024-09-17T04:45:49.651 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-17T04:45:49.652 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-17T04:45:49.652 INFO:teuthology.orchestra.run.smithi176.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-17T04:45:49.652 INFO:teuthology.orchestra.run.smithi176.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-17T04:45:49.652 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-17T04:45:49.653 INFO:teuthology.orchestra.run.smithi176.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-17T04:45:49.653 INFO:teuthology.orchestra.run.smithi176.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-17T04:45:49.653 INFO:teuthology.orchestra.run.smithi176.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-17T04:45:49.653 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-17T04:45:49.653 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot python3-cherrypy3 python3-cmd2 python3-colorama 2024-09-17T04:45:49.653 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth python3-idna python3-iniconfig python3-jaraco.classes 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-09-17T04:45:49.654 INFO:teuthology.orchestra.run.smithi176.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-09-17T04:45:49.655 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2024-09-17T04:45:49.655 INFO:teuthology.orchestra.run.smithi176.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-17T04:45:49.655 INFO:teuthology.orchestra.run.smithi176.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-17T04:45:49.655 INFO:teuthology.orchestra.run.smithi176.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-17T04:45:49.656 INFO:teuthology.orchestra.run.smithi176.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-17T04:45:49.656 INFO:teuthology.orchestra.run.smithi176.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-17T04:45:49.656 INFO:teuthology.orchestra.run.smithi176.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-17T04:45:49.656 INFO:teuthology.orchestra.run.smithi176.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-17T04:45:49.656 INFO:teuthology.orchestra.run.smithi176.stdout:Recommended packages: 2024-09-17T04:45:49.656 INFO:teuthology.orchestra.run.smithi176.stdout: btrfs-tools 2024-09-17T04:45:49.658 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T04:45:49.658 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T04:45:49.658 INFO:teuthology.orchestra.run.smithi150.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T04:45:49.658 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T04:45:49.658 INFO:teuthology.orchestra.run.smithi150.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-17T04:45:49.658 INFO:teuthology.orchestra.run.smithi150.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-17T04:45:49.658 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T04:45:49.659 INFO:teuthology.orchestra.run.smithi150.stdout:The following additional packages will be installed: 2024-09-17T04:45:49.659 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-17T04:45:49.659 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-17T04:45:49.659 INFO:teuthology.orchestra.run.smithi150.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-17T04:45:49.659 INFO:teuthology.orchestra.run.smithi150.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-17T04:45:49.659 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-17T04:45:49.659 INFO:teuthology.orchestra.run.smithi150.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-17T04:45:49.659 INFO:teuthology.orchestra.run.smithi150.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-17T04:45:49.659 INFO:teuthology.orchestra.run.smithi150.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-17T04:45:49.659 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-17T04:45:49.660 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot python3-cherrypy3 python3-cmd2 python3-colorama 2024-09-17T04:45:49.660 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth python3-idna python3-iniconfig python3-jaraco.classes 2024-09-17T04:45:49.660 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-17T04:45:49.660 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-17T04:45:49.660 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-17T04:45:49.660 INFO:teuthology.orchestra.run.smithi150.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-17T04:45:49.660 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-17T04:45:49.660 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-17T04:45:49.660 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-09-17T04:45:49.660 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-09-17T04:45:49.660 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-09-17T04:45:49.660 INFO:teuthology.orchestra.run.smithi150.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-09-17T04:45:49.661 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-09-17T04:45:49.661 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-09-17T04:45:49.661 INFO:teuthology.orchestra.run.smithi150.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-09-17T04:45:49.661 INFO:teuthology.orchestra.run.smithi150.stdout:Suggested packages: 2024-09-17T04:45:49.661 INFO:teuthology.orchestra.run.smithi150.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-17T04:45:49.661 INFO:teuthology.orchestra.run.smithi150.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-17T04:45:49.661 INFO:teuthology.orchestra.run.smithi150.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-17T04:45:49.661 INFO:teuthology.orchestra.run.smithi150.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-17T04:45:49.662 INFO:teuthology.orchestra.run.smithi150.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-17T04:45:49.662 INFO:teuthology.orchestra.run.smithi150.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-17T04:45:49.662 INFO:teuthology.orchestra.run.smithi150.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-17T04:45:49.662 INFO:teuthology.orchestra.run.smithi150.stdout:Recommended packages: 2024-09-17T04:45:49.662 INFO:teuthology.orchestra.run.smithi150.stdout: btrfs-tools 2024-09-17T04:45:49.764 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2024-09-17T04:45:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-17T04:45:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-17T04:45:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-17T04:45:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-09-17T04:45:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-17T04:45:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-09-17T04:45:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-09-17T04:45:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-09-17T04:45:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-09-17T04:45:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-17T04:45:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-17T04:45:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-09-17T04:45:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-09-17T04:45:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-17T04:45:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-17T04:45:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-17T04:45:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-17T04:45:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-17T04:45:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-17T04:45:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-09-17T04:45:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-17T04:45:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-17T04:45:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-17T04:45:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-17T04:45:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-17T04:45:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-17T04:45:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-17T04:45:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: socat unzip xmlstarlet zip 2024-09-17T04:45:49.769 INFO:teuthology.orchestra.run.smithi150.stdout:The following NEW packages will be installed: 2024-09-17T04:45:49.769 INFO:teuthology.orchestra.run.smithi150.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-17T04:45:49.769 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-17T04:45:49.769 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-17T04:45:49.769 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-09-17T04:45:49.770 INFO:teuthology.orchestra.run.smithi150.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-17T04:45:49.770 INFO:teuthology.orchestra.run.smithi150.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-09-17T04:45:49.770 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-09-17T04:45:49.770 INFO:teuthology.orchestra.run.smithi150.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-09-17T04:45:49.770 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-09-17T04:45:49.770 INFO:teuthology.orchestra.run.smithi150.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-17T04:45:49.770 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-17T04:45:49.770 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-09-17T04:45:49.770 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-09-17T04:45:49.770 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-17T04:45:49.770 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-17T04:45:49.771 INFO:teuthology.orchestra.run.smithi150.stdout: socat unzip xmlstarlet zip 2024-09-17T04:45:49.825 INFO:teuthology.orchestra.run.smithi176.stdout:0 upgraded, 125 newly installed, 0 to remove and 331 not upgraded. 2024-09-17T04:45:49.825 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 239 MB of archives. 2024-09-17T04:45:49.825 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 986 MB of additional disk space will be used. 2024-09-17T04:45:49.825 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-09-17T04:45:49.843 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-17T04:45:49.852 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-09-17T04:45:49.893 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-17T04:45:49.910 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4989-gce1b342b-1jammy [3,591 kB] 2024-09-17T04:45:49.914 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-17T04:45:49.917 INFO:teuthology.orchestra.run.smithi150.stdout:0 upgraded, 125 newly installed, 0 to remove and 331 not upgraded. 2024-09-17T04:45:49.917 INFO:teuthology.orchestra.run.smithi150.stdout:Need to get 239 MB of archives. 2024-09-17T04:45:49.917 INFO:teuthology.orchestra.run.smithi150.stdout:After this operation, 986 MB of additional disk space will be used. 2024-09-17T04:45:49.917 INFO:teuthology.orchestra.run.smithi150.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4989-gce1b342b-1jammy [3,591 kB] 2024-09-17T04:45:49.917 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-17T04:45:49.928 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-17T04:45:49.957 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-09-17T04:45:49.966 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-17T04:45:49.969 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-17T04:45:49.977 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-17T04:45:49.980 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-17T04:45:49.981 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-17T04:45:49.982 INFO:teuthology.orchestra.run.smithi176.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-17T04:45:49.990 INFO:teuthology.orchestra.run.smithi176.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-17T04:45:49.992 INFO:teuthology.orchestra.run.smithi176.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-17T04:45:49.994 INFO:teuthology.orchestra.run.smithi176.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-17T04:45:49.995 INFO:teuthology.orchestra.run.smithi176.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-17T04:45:50.003 INFO:teuthology.orchestra.run.smithi176.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-17T04:45:50.006 INFO:teuthology.orchestra.run.smithi176.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-17T04:45:50.009 INFO:teuthology.orchestra.run.smithi176.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-17T04:45:50.012 INFO:teuthology.orchestra.run.smithi176.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-17T04:45:50.013 INFO:teuthology.orchestra.run.smithi176.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-17T04:45:50.020 INFO:teuthology.orchestra.run.smithi176.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-17T04:45:50.022 INFO:teuthology.orchestra.run.smithi176.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-17T04:45:50.023 INFO:teuthology.orchestra.run.smithi176.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-17T04:45:50.023 INFO:teuthology.orchestra.run.smithi176.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-17T04:45:50.024 INFO:teuthology.orchestra.run.smithi176.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-17T04:45:50.037 INFO:teuthology.orchestra.run.smithi176.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-17T04:45:50.038 INFO:teuthology.orchestra.run.smithi176.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-17T04:45:50.038 INFO:teuthology.orchestra.run.smithi176.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-17T04:45:50.039 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-17T04:45:50.042 INFO:teuthology.orchestra.run.smithi176.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-17T04:45:50.043 INFO:teuthology.orchestra.run.smithi176.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-17T04:45:50.054 INFO:teuthology.orchestra.run.smithi176.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-17T04:45:50.054 INFO:teuthology.orchestra.run.smithi176.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-17T04:45:50.056 INFO:teuthology.orchestra.run.smithi176.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-17T04:45:50.057 INFO:teuthology.orchestra.run.smithi176.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-17T04:45:50.057 INFO:teuthology.orchestra.run.smithi176.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-17T04:45:50.071 INFO:teuthology.orchestra.run.smithi176.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-17T04:45:50.072 INFO:teuthology.orchestra.run.smithi176.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-17T04:45:50.073 INFO:teuthology.orchestra.run.smithi176.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-17T04:45:50.080 INFO:teuthology.orchestra.run.smithi176.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-17T04:45:50.080 INFO:teuthology.orchestra.run.smithi176.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-17T04:45:50.085 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-09-17T04:45:50.088 INFO:teuthology.orchestra.run.smithi176.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-17T04:45:50.089 INFO:teuthology.orchestra.run.smithi176.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-17T04:45:50.090 INFO:teuthology.orchestra.run.smithi176.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-17T04:45:50.092 INFO:teuthology.orchestra.run.smithi176.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-17T04:45:50.092 INFO:teuthology.orchestra.run.smithi176.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-17T04:45:50.105 INFO:teuthology.orchestra.run.smithi176.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-17T04:45:50.106 INFO:teuthology.orchestra.run.smithi176.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-17T04:45:50.108 INFO:teuthology.orchestra.run.smithi176.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-17T04:45:50.109 INFO:teuthology.orchestra.run.smithi176.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-17T04:45:50.109 INFO:teuthology.orchestra.run.smithi176.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-17T04:45:50.120 INFO:teuthology.orchestra.run.smithi176.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4989-gce1b342b-1jammy [3,368 kB] 2024-09-17T04:45:50.122 INFO:teuthology.orchestra.run.smithi176.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-17T04:45:50.129 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4989-gce1b342b-1jammy [3,368 kB] 2024-09-17T04:45:50.161 INFO:teuthology.orchestra.run.smithi176.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-17T04:45:50.162 INFO:teuthology.orchestra.run.smithi176.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-17T04:45:50.163 INFO:teuthology.orchestra.run.smithi176.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-17T04:45:50.163 INFO:teuthology.orchestra.run.smithi176.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-17T04:45:50.164 INFO:teuthology.orchestra.run.smithi176.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-17T04:45:50.180 INFO:teuthology.orchestra.run.smithi176.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-17T04:45:50.182 INFO:teuthology.orchestra.run.smithi176.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-17T04:45:50.182 INFO:teuthology.orchestra.run.smithi176.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-17T04:45:50.196 INFO:teuthology.orchestra.run.smithi176.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-17T04:45:50.196 INFO:teuthology.orchestra.run.smithi176.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-17T04:45:50.197 INFO:teuthology.orchestra.run.smithi176.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-17T04:45:50.197 INFO:teuthology.orchestra.run.smithi176.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-17T04:45:50.198 INFO:teuthology.orchestra.run.smithi176.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-17T04:45:50.198 INFO:teuthology.orchestra.run.smithi176.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-17T04:45:50.208 INFO:teuthology.orchestra.run.smithi176.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4989-gce1b342b-1jammy [733 kB] 2024-09-17T04:45:50.214 INFO:teuthology.orchestra.run.smithi176.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-17T04:45:50.214 INFO:teuthology.orchestra.run.smithi176.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-17T04:45:50.218 INFO:teuthology.orchestra.run.smithi176.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-17T04:45:50.219 INFO:teuthology.orchestra.run.smithi176.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-17T04:45:50.220 INFO:teuthology.orchestra.run.smithi176.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-17T04:45:50.229 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4989-gce1b342b-1jammy [733 kB] 2024-09-17T04:45:50.231 INFO:teuthology.orchestra.run.smithi176.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-17T04:45:50.234 INFO:teuthology.orchestra.run.smithi176.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-17T04:45:50.236 INFO:teuthology.orchestra.run.smithi176.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-09-17T04:45:50.236 INFO:teuthology.orchestra.run.smithi176.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-17T04:45:50.237 INFO:teuthology.orchestra.run.smithi176.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-17T04:45:50.238 INFO:teuthology.orchestra.run.smithi176.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4989-gce1b342b-1jammy [356 kB] 2024-09-17T04:45:50.245 INFO:teuthology.orchestra.run.smithi176.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4989-gce1b342b-1jammy [32.8 kB] 2024-09-17T04:45:50.246 INFO:teuthology.orchestra.run.smithi176.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4989-gce1b342b-1jammy [180 kB] 2024-09-17T04:45:50.248 INFO:teuthology.orchestra.run.smithi176.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-17T04:45:50.250 INFO:teuthology.orchestra.run.smithi176.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4989-gce1b342b-1jammy [71.3 kB] 2024-09-17T04:45:50.252 INFO:teuthology.orchestra.run.smithi176.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-17T04:45:50.252 INFO:teuthology.orchestra.run.smithi176.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4989-gce1b342b-1jammy [337 kB] 2024-09-17T04:45:50.252 INFO:teuthology.orchestra.run.smithi176.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-17T04:45:50.255 INFO:teuthology.orchestra.run.smithi176.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-17T04:45:50.255 INFO:teuthology.orchestra.run.smithi176.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-17T04:45:50.259 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4989-gce1b342b-1jammy [356 kB] 2024-09-17T04:45:50.265 INFO:teuthology.orchestra.run.smithi176.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-17T04:45:50.266 INFO:teuthology.orchestra.run.smithi176.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-17T04:45:50.266 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4989-gce1b342b-1jammy [32.8 kB] 2024-09-17T04:45:50.267 INFO:teuthology.orchestra.run.smithi150.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4989-gce1b342b-1jammy [180 kB] 2024-09-17T04:45:50.268 INFO:teuthology.orchestra.run.smithi176.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-17T04:45:50.268 INFO:teuthology.orchestra.run.smithi176.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-17T04:45:50.270 INFO:teuthology.orchestra.run.smithi150.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4989-gce1b342b-1jammy [71.3 kB] 2024-09-17T04:45:50.272 INFO:teuthology.orchestra.run.smithi150.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4989-gce1b342b-1jammy [337 kB] 2024-09-17T04:45:50.273 INFO:teuthology.orchestra.run.smithi176.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-17T04:45:50.274 INFO:teuthology.orchestra.run.smithi176.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4989-gce1b342b-1jammy [8,847 kB] 2024-09-17T04:45:50.280 INFO:teuthology.orchestra.run.smithi150.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-17T04:45:50.282 INFO:teuthology.orchestra.run.smithi176.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-17T04:45:50.282 INFO:teuthology.orchestra.run.smithi176.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-17T04:45:50.284 INFO:teuthology.orchestra.run.smithi176.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-17T04:45:50.285 INFO:teuthology.orchestra.run.smithi176.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-17T04:45:50.297 INFO:teuthology.orchestra.run.smithi150.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4989-gce1b342b-1jammy [8,847 kB] 2024-09-17T04:45:50.372 INFO:teuthology.orchestra.run.smithi150.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-17T04:45:50.385 INFO:teuthology.orchestra.run.smithi150.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-17T04:45:50.434 INFO:teuthology.orchestra.run.smithi150.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-17T04:45:50.548 INFO:teuthology.orchestra.run.smithi176.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4989-gce1b342b-1jammy [112 kB] 2024-09-17T04:45:50.550 INFO:teuthology.orchestra.run.smithi176.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4989-gce1b342b-1jammy [478 kB] 2024-09-17T04:45:50.577 INFO:teuthology.orchestra.run.smithi150.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4989-gce1b342b-1jammy [112 kB] 2024-09-17T04:45:50.578 INFO:teuthology.orchestra.run.smithi150.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4989-gce1b342b-1jammy [478 kB] 2024-09-17T04:45:50.582 INFO:teuthology.orchestra.run.smithi176.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4989-gce1b342b-1jammy [29.1 MB] 2024-09-17T04:45:50.606 INFO:teuthology.orchestra.run.smithi150.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-17T04:45:50.612 INFO:teuthology.orchestra.run.smithi150.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4989-gce1b342b-1jammy [29.1 MB] 2024-09-17T04:45:50.615 INFO:teuthology.orchestra.run.smithi150.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-17T04:45:50.642 INFO:teuthology.orchestra.run.smithi150.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-17T04:45:50.651 INFO:teuthology.orchestra.run.smithi150.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-17T04:45:50.653 INFO:teuthology.orchestra.run.smithi150.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-17T04:45:50.654 INFO:teuthology.orchestra.run.smithi150.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-17T04:45:50.672 INFO:teuthology.orchestra.run.smithi150.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-17T04:45:50.676 INFO:teuthology.orchestra.run.smithi150.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-17T04:45:50.681 INFO:teuthology.orchestra.run.smithi150.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-17T04:45:50.689 INFO:teuthology.orchestra.run.smithi150.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-17T04:45:50.766 INFO:teuthology.orchestra.run.smithi150.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-17T04:45:50.770 INFO:teuthology.orchestra.run.smithi150.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-17T04:45:50.774 INFO:teuthology.orchestra.run.smithi150.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-17T04:45:50.778 INFO:teuthology.orchestra.run.smithi150.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-17T04:45:50.779 INFO:teuthology.orchestra.run.smithi150.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-17T04:45:50.779 INFO:teuthology.orchestra.run.smithi150.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-17T04:45:50.782 INFO:teuthology.orchestra.run.smithi150.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-17T04:45:50.782 INFO:teuthology.orchestra.run.smithi150.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-17T04:45:50.783 INFO:teuthology.orchestra.run.smithi150.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-17T04:45:50.852 INFO:teuthology.orchestra.run.smithi150.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-17T04:45:50.865 INFO:teuthology.orchestra.run.smithi150.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-17T04:45:50.866 INFO:teuthology.orchestra.run.smithi150.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-17T04:45:50.866 INFO:teuthology.orchestra.run.smithi150.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-17T04:45:50.872 INFO:teuthology.orchestra.run.smithi150.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-17T04:45:50.873 INFO:teuthology.orchestra.run.smithi150.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-17T04:45:50.933 INFO:teuthology.orchestra.run.smithi150.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-17T04:45:50.934 INFO:teuthology.orchestra.run.smithi150.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-17T04:45:50.936 INFO:teuthology.orchestra.run.smithi150.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-17T04:45:50.936 INFO:teuthology.orchestra.run.smithi150.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-17T04:45:51.014 INFO:teuthology.orchestra.run.smithi150.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-17T04:45:51.016 INFO:teuthology.orchestra.run.smithi150.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-17T04:45:51.017 INFO:teuthology.orchestra.run.smithi150.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-17T04:45:51.019 INFO:teuthology.orchestra.run.smithi150.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-17T04:45:51.029 INFO:teuthology.orchestra.run.smithi150.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-17T04:45:51.095 INFO:teuthology.orchestra.run.smithi150.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-17T04:45:51.096 INFO:teuthology.orchestra.run.smithi150.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-17T04:45:51.097 INFO:teuthology.orchestra.run.smithi150.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-17T04:45:51.100 INFO:teuthology.orchestra.run.smithi150.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-17T04:45:51.104 INFO:teuthology.orchestra.run.smithi150.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-17T04:45:51.177 INFO:teuthology.orchestra.run.smithi150.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-17T04:45:51.179 INFO:teuthology.orchestra.run.smithi150.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-17T04:45:51.181 INFO:teuthology.orchestra.run.smithi150.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-17T04:45:51.185 INFO:teuthology.orchestra.run.smithi150.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-17T04:45:51.186 INFO:teuthology.orchestra.run.smithi150.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-17T04:45:51.258 INFO:teuthology.orchestra.run.smithi150.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-17T04:45:51.262 INFO:teuthology.orchestra.run.smithi150.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-17T04:45:51.427 INFO:teuthology.orchestra.run.smithi150.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-17T04:45:51.429 INFO:teuthology.orchestra.run.smithi150.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-17T04:45:51.431 INFO:teuthology.orchestra.run.smithi150.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-17T04:45:51.432 INFO:teuthology.orchestra.run.smithi150.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-17T04:45:51.434 INFO:teuthology.orchestra.run.smithi150.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-17T04:45:51.470 INFO:teuthology.orchestra.run.smithi150.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-17T04:45:51.473 INFO:teuthology.orchestra.run.smithi150.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-17T04:45:51.473 INFO:teuthology.orchestra.run.smithi150.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-17T04:45:51.532 INFO:teuthology.orchestra.run.smithi150.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-17T04:45:51.532 INFO:teuthology.orchestra.run.smithi150.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-17T04:45:51.593 INFO:teuthology.orchestra.run.smithi150.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-17T04:45:51.594 INFO:teuthology.orchestra.run.smithi150.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-17T04:45:51.595 INFO:teuthology.orchestra.run.smithi150.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-17T04:45:51.595 INFO:teuthology.orchestra.run.smithi176.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4989-gce1b342b-1jammy [5,020 kB] 2024-09-17T04:45:51.596 INFO:teuthology.orchestra.run.smithi150.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-17T04:45:51.597 INFO:teuthology.orchestra.run.smithi150.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-17T04:45:51.597 INFO:teuthology.orchestra.run.smithi150.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-17T04:45:51.659 INFO:teuthology.orchestra.run.smithi150.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-17T04:45:51.672 INFO:teuthology.orchestra.run.smithi176.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4989-gce1b342b-1jammy [251 kB] 2024-09-17T04:45:51.674 INFO:teuthology.orchestra.run.smithi176.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4989-gce1b342b-1jammy [126 kB] 2024-09-17T04:45:51.677 INFO:teuthology.orchestra.run.smithi176.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4989-gce1b342b-1jammy [1,551 kB] 2024-09-17T04:45:51.703 INFO:teuthology.orchestra.run.smithi176.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4989-gce1b342b-1jammy [6,200 kB] 2024-09-17T04:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-17T04:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4989-gce1b342b-1jammy [5,020 kB] 2024-09-17T04:45:51.753 INFO:teuthology.orchestra.run.smithi150.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-17T04:45:51.754 INFO:teuthology.orchestra.run.smithi150.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-17T04:45:51.763 INFO:teuthology.orchestra.run.smithi150.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-17T04:45:51.772 INFO:teuthology.orchestra.run.smithi150.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-09-17T04:45:51.772 INFO:teuthology.orchestra.run.smithi150.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-17T04:45:51.773 INFO:teuthology.orchestra.run.smithi150.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-17T04:45:51.787 INFO:teuthology.orchestra.run.smithi150.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-17T04:45:51.798 INFO:teuthology.orchestra.run.smithi150.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-17T04:45:51.831 INFO:teuthology.orchestra.run.smithi150.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4989-gce1b342b-1jammy [251 kB] 2024-09-17T04:45:51.834 INFO:teuthology.orchestra.run.smithi150.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4989-gce1b342b-1jammy [126 kB] 2024-09-17T04:45:51.837 INFO:teuthology.orchestra.run.smithi150.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-17T04:45:51.847 INFO:teuthology.orchestra.run.smithi150.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4989-gce1b342b-1jammy [1,551 kB] 2024-09-17T04:45:51.875 INFO:teuthology.orchestra.run.smithi176.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4989-gce1b342b-1jammy [22.9 MB] 2024-09-17T04:45:51.881 INFO:teuthology.orchestra.run.smithi150.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-17T04:45:51.881 INFO:teuthology.orchestra.run.smithi150.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-17T04:45:51.883 INFO:teuthology.orchestra.run.smithi150.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-17T04:45:51.884 INFO:teuthology.orchestra.run.smithi150.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-17T04:45:51.887 INFO:teuthology.orchestra.run.smithi150.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-17T04:45:51.896 INFO:teuthology.orchestra.run.smithi150.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4989-gce1b342b-1jammy [6,200 kB] 2024-09-17T04:45:51.920 INFO:teuthology.orchestra.run.smithi150.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-17T04:45:51.937 INFO:teuthology.orchestra.run.smithi150.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-17T04:45:51.937 INFO:teuthology.orchestra.run.smithi150.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-17T04:45:51.938 INFO:teuthology.orchestra.run.smithi150.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-17T04:45:52.000 INFO:teuthology.orchestra.run.smithi150.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-17T04:45:52.001 INFO:teuthology.orchestra.run.smithi150.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-17T04:45:52.116 INFO:teuthology.orchestra.run.smithi150.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4989-gce1b342b-1jammy [22.9 MB] 2024-09-17T04:45:52.644 INFO:teuthology.orchestra.run.smithi176.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4989-gce1b342b-1jammy [14.0 kB] 2024-09-17T04:45:52.645 INFO:teuthology.orchestra.run.smithi176.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4989-gce1b342b-1jammy [865 kB] 2024-09-17T04:45:52.660 INFO:teuthology.orchestra.run.smithi176.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4989-gce1b342b-1jammy [2,386 kB] 2024-09-17T04:45:52.698 INFO:teuthology.orchestra.run.smithi176.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4989-gce1b342b-1jammy [814 kB] 2024-09-17T04:45:52.720 INFO:teuthology.orchestra.run.smithi176.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4989-gce1b342b-1jammy [160 kB] 2024-09-17T04:45:52.722 INFO:teuthology.orchestra.run.smithi176.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4989-gce1b342b-1jammy [61.5 MB] 2024-09-17T04:45:53.005 INFO:teuthology.orchestra.run.smithi150.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4989-gce1b342b-1jammy [14.0 kB] 2024-09-17T04:45:53.005 INFO:teuthology.orchestra.run.smithi150.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4989-gce1b342b-1jammy [865 kB] 2024-09-17T04:45:53.022 INFO:teuthology.orchestra.run.smithi150.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4989-gce1b342b-1jammy [2,386 kB] 2024-09-17T04:45:53.064 INFO:teuthology.orchestra.run.smithi150.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4989-gce1b342b-1jammy [814 kB] 2024-09-17T04:45:53.074 INFO:teuthology.orchestra.run.smithi150.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4989-gce1b342b-1jammy [160 kB] 2024-09-17T04:45:53.076 INFO:teuthology.orchestra.run.smithi150.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4989-gce1b342b-1jammy [61.5 MB] 2024-09-17T04:45:54.877 INFO:teuthology.orchestra.run.smithi176.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4989-gce1b342b-1jammy [8,625 kB] 2024-09-17T04:45:55.063 INFO:teuthology.orchestra.run.smithi176.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4989-gce1b342b-1jammy [14.2 kB] 2024-09-17T04:45:55.063 INFO:teuthology.orchestra.run.smithi176.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4989-gce1b342b-1jammy [40.9 MB] 2024-09-17T04:45:55.319 INFO:teuthology.orchestra.run.smithi150.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4989-gce1b342b-1jammy [8,625 kB] 2024-09-17T04:45:55.518 INFO:teuthology.orchestra.run.smithi150.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4989-gce1b342b-1jammy [14.2 kB] 2024-09-17T04:45:55.519 INFO:teuthology.orchestra.run.smithi150.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4989-gce1b342b-1jammy [40.9 MB] 2024-09-17T04:45:56.324 INFO:teuthology.orchestra.run.smithi176.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4989-gce1b342b-1jammy [130 kB] 2024-09-17T04:45:56.326 INFO:teuthology.orchestra.run.smithi176.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.3.0-4989-gce1b342b-1jammy [237 kB] 2024-09-17T04:45:56.331 INFO:teuthology.orchestra.run.smithi176.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.3.0-4989-gce1b342b-1jammy [33.9 kB] 2024-09-17T04:45:56.332 INFO:teuthology.orchestra.run.smithi176.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.3.0-4989-gce1b342b-1jammy [23.4 kB] 2024-09-17T04:45:56.333 INFO:teuthology.orchestra.run.smithi176.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4989-gce1b342b-1jammy [39.0 kB] 2024-09-17T04:45:56.334 INFO:teuthology.orchestra.run.smithi176.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4989-gce1b342b-1jammy [17.6 MB] 2024-09-17T04:45:56.894 INFO:teuthology.orchestra.run.smithi176.stdout:Get:125 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4989-gce1b342b-1jammy [93.1 kB] 2024-09-17T04:45:56.972 INFO:teuthology.orchestra.run.smithi150.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4989-gce1b342b-1jammy [130 kB] 2024-09-17T04:45:56.974 INFO:teuthology.orchestra.run.smithi150.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.3.0-4989-gce1b342b-1jammy [237 kB] 2024-09-17T04:45:56.979 INFO:teuthology.orchestra.run.smithi150.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.3.0-4989-gce1b342b-1jammy [33.9 kB] 2024-09-17T04:45:56.980 INFO:teuthology.orchestra.run.smithi150.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.3.0-4989-gce1b342b-1jammy [23.4 kB] 2024-09-17T04:45:56.981 INFO:teuthology.orchestra.run.smithi150.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4989-gce1b342b-1jammy [39.0 kB] 2024-09-17T04:45:56.982 INFO:teuthology.orchestra.run.smithi150.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4989-gce1b342b-1jammy [17.6 MB] 2024-09-17T04:45:57.316 INFO:teuthology.orchestra.run.smithi150.stdout:Get:125 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4989-gce1b342b-1jammy [93.1 kB] 2024-09-17T04:45:57.559 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 239 MB in 7s (33.6 MB/s) 2024-09-17T04:45:57.668 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyperclip. 2024-09-17T04:45:57.759 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-17T04:45:57.764 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-09-17T04:45:57.799 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-09-17T04:45:57.956 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 239 MB in 8s (31.8 MB/s) 2024-09-17T04:45:58.069 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-17T04:45:58.095 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-17T04:45:58.098 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyperclip. 2024-09-17T04:45:58.135 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-17T04:45:58.198 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-17T04:45:58.204 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-09-17T04:45:58.246 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-09-17T04:45:58.463 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cmd2. 2024-09-17T04:45:58.490 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-09-17T04:45:58.530 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-09-17T04:45:58.549 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-17T04:45:58.575 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-17T04:45:58.616 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-17T04:45:58.894 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-17T04:45:58.921 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-17T04:45:58.943 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cmd2. 2024-09-17T04:45:58.970 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-09-17T04:45:59.008 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-17T04:45:59.010 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-09-17T04:45:59.363 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-17T04:45:59.369 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-17T04:45:59.390 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-17T04:45:59.396 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-17T04:45:59.436 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-17T04:45:59.472 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-17T04:45:59.714 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-17T04:45:59.729 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-17T04:45:59.772 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-17T04:45:59.842 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-17T04:45:59.868 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-17T04:45:59.908 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-17T04:46:00.203 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-17T04:46:00.226 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-17T04:46:00.229 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-17T04:46:00.252 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T04:46:00.270 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-17T04:46:00.300 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T04:46:00.687 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-17T04:46:00.713 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T04:46:00.723 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-17T04:46:00.749 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T04:46:00.753 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T04:46:00.798 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T04:46:01.040 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-17T04:46:01.066 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T04:46:01.098 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T04:46:01.202 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-17T04:46:01.227 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T04:46:01.268 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T04:46:01.443 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-17T04:46:01.469 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-17T04:46:01.518 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-17T04:46:01.596 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-17T04:46:01.622 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T04:46:01.662 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T04:46:01.829 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librados2. 2024-09-17T04:46:01.855 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../010-librados2_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:01.895 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librados2 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:02.032 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-17T04:46:02.058 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-17T04:46:02.099 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-17T04:46:02.373 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libnbd0. 2024-09-17T04:46:02.399 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../011-libnbd0_1.10.5-1_amd64.deb ... 2024-09-17T04:46:02.431 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-17T04:46:02.435 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librados2. 2024-09-17T04:46:02.461 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../010-librados2_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:02.501 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librados2 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:02.726 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librbd1. 2024-09-17T04:46:02.752 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../012-librbd1_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:02.792 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librbd1 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:02.980 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libnbd0. 2024-09-17T04:46:03.006 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../011-libnbd0_1.10.5-1_amd64.deb ... 2024-09-17T04:46:03.038 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-17T04:46:03.262 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libcephfs2. 2024-09-17T04:46:03.288 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../013-libcephfs2_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:03.329 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libcephfs2 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:03.340 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librbd1. 2024-09-17T04:46:03.367 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../012-librbd1_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:03.407 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librbd1 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:03.606 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rados. 2024-09-17T04:46:03.633 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../014-python3-rados_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:03.673 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rados (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:03.893 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libcephfs2. 2024-09-17T04:46:03.920 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../013-libcephfs2_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:03.934 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-17T04:46:03.960 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libcephfs2 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:03.960 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../015-python3-ceph-argparse_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:03.992 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-ceph-argparse (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:04.245 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cephfs. 2024-09-17T04:46:04.263 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rados. 2024-09-17T04:46:04.271 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../016-python3-cephfs_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:04.289 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../014-python3-rados_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:04.311 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cephfs (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:04.329 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rados (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:04.607 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-17T04:46:04.622 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-17T04:46:04.633 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../015-python3-ceph-argparse_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:04.649 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../017-python3-ceph-common_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:04.673 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-ceph-argparse (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:04.688 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-ceph-common (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:04.926 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cephfs. 2024-09-17T04:46:04.952 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../016-python3-cephfs_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:04.992 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cephfs (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:05.008 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-prettytable. 2024-09-17T04:46:05.034 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2024-09-17T04:46:05.075 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-17T04:46:05.320 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-17T04:46:05.327 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rbd. 2024-09-17T04:46:05.342 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../019-python3-rbd_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:05.346 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../017-python3-ceph-common_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:05.377 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rbd (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:05.387 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-ceph-common (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:05.707 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-prettytable. 2024-09-17T04:46:05.722 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-17T04:46:05.733 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2024-09-17T04:46:05.748 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-17T04:46:05.773 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-17T04:46:05.788 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-17T04:46:06.042 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rbd. 2024-09-17T04:46:06.057 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../019-python3-rbd_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:06.066 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-17T04:46:06.092 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../021-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-17T04:46:06.100 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rbd (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:06.132 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-17T04:46:06.436 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-17T04:46:06.445 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-17T04:46:06.465 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../022-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-17T04:46:06.472 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-17T04:46:06.502 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-17T04:46:06.503 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-17T04:46:06.764 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-17T04:46:06.780 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../021-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-17T04:46:06.814 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package lua5.1. 2024-09-17T04:46:06.822 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-17T04:46:06.840 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../023-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-17T04:46:06.872 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-17T04:46:07.134 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-17T04:46:07.149 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../022-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-17T04:46:07.183 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package lua-any. 2024-09-17T04:46:07.192 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-17T04:46:07.209 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../024-lua-any_27ubuntu1_all.deb ... 2024-09-17T04:46:07.249 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-17T04:46:07.462 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package zip. 2024-09-17T04:46:07.477 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../025-zip_3.0-12build2_amd64.deb ... 2024-09-17T04:46:07.521 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking zip (3.0-12build2) ... 2024-09-17T04:46:07.562 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package lua5.1. 2024-09-17T04:46:07.588 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../023-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-17T04:46:07.629 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-17T04:46:07.884 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package unzip. 2024-09-17T04:46:07.898 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package lua-any. 2024-09-17T04:46:07.910 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../026-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-17T04:46:07.913 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../024-lua-any_27ubuntu1_all.deb ... 2024-09-17T04:46:07.948 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-17T04:46:07.949 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-17T04:46:08.192 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package zip. 2024-09-17T04:46:08.219 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../025-zip_3.0-12build2_amd64.deb ... 2024-09-17T04:46:08.259 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking zip (3.0-12build2) ... 2024-09-17T04:46:08.352 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package luarocks. 2024-09-17T04:46:08.379 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../027-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-17T04:46:08.419 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-17T04:46:08.785 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librgw2. 2024-09-17T04:46:08.811 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../028-librgw2_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:08.851 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librgw2 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:08.971 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package unzip. 2024-09-17T04:46:08.997 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../026-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-17T04:46:09.037 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-17T04:46:09.416 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package luarocks. 2024-09-17T04:46:09.442 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../027-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-17T04:46:09.483 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-17T04:46:09.537 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rgw. 2024-09-17T04:46:09.552 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../029-python3-rgw_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:09.603 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rgw (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:09.832 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package librgw2. 2024-09-17T04:46:09.858 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../028-librgw2_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:09.899 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking librgw2 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:09.931 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-17T04:46:09.957 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../030-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-17T04:46:09.998 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-17T04:46:10.309 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libradosstriper1. 2024-09-17T04:46:10.335 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../031-libradosstriper1_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:10.375 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libradosstriper1 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:10.634 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rgw. 2024-09-17T04:46:10.650 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../029-python3-rgw_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:10.692 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rgw (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:10.787 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-common. 2024-09-17T04:46:10.911 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../032-ceph-common_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:10.911 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-common (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:11.054 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-17T04:46:11.081 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../030-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-17T04:46:11.121 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-17T04:46:11.415 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libradosstriper1. 2024-09-17T04:46:11.441 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../031-libradosstriper1_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:11.473 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libradosstriper1 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:11.902 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-common. 2024-09-17T04:46:11.928 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../032-ceph-common_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:11.968 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-common (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:12.550 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-base. 2024-09-17T04:46:12.577 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../033-ceph-base_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:12.626 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-base (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:13.513 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-17T04:46:13.540 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../034-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-17T04:46:13.579 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-17T04:46:13.840 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cheroot. 2024-09-17T04:46:13.867 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../035-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-17T04:46:13.907 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-17T04:46:13.998 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-base. 2024-09-17T04:46:14.025 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../033-ceph-base_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:14.075 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-base (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:14.193 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-17T04:46:14.220 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../036-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-17T04:46:14.260 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-17T04:46:14.562 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-17T04:46:14.589 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../037-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-17T04:46:14.629 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-17T04:46:14.636 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-17T04:46:14.662 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../034-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-17T04:46:14.694 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-17T04:46:14.932 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-17T04:46:14.959 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../038-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-17T04:46:14.998 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-17T04:46:15.022 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cheroot. 2024-09-17T04:46:15.048 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../035-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-17T04:46:15.088 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-17T04:46:15.268 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-tempora. 2024-09-17T04:46:15.295 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../039-python3-tempora_4.1.2-1_all.deb ... 2024-09-17T04:46:15.326 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-17T04:46:15.416 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-17T04:46:15.443 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../036-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-17T04:46:15.483 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-17T04:46:15.646 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-portend. 2024-09-17T04:46:15.673 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../040-python3-portend_3.0.0-1_all.deb ... 2024-09-17T04:46:15.712 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-17T04:46:15.769 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-17T04:46:15.795 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../037-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-17T04:46:15.835 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-17T04:46:16.015 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-17T04:46:16.042 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../041-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-17T04:46:16.082 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-17T04:46:16.122 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-17T04:46:16.141 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../038-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-17T04:46:16.172 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-17T04:46:16.385 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-17T04:46:16.412 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../042-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-17T04:46:16.441 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-tempora. 2024-09-17T04:46:16.451 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-17T04:46:16.468 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../039-python3-tempora_4.1.2-1_all.deb ... 2024-09-17T04:46:16.508 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-17T04:46:16.804 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-natsort. 2024-09-17T04:46:16.811 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-portend. 2024-09-17T04:46:16.831 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../043-python3-natsort_8.0.2-1_all.deb ... 2024-09-17T04:46:16.837 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../040-python3-portend_3.0.0-1_all.deb ... 2024-09-17T04:46:16.871 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-17T04:46:16.877 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-17T04:46:17.166 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-logutils. 2024-09-17T04:46:17.188 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-17T04:46:17.192 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../044-python3-logutils_0.3.3-8_all.deb ... 2024-09-17T04:46:17.215 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../041-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-17T04:46:17.232 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-17T04:46:17.255 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-17T04:46:17.516 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-17T04:46:17.531 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../042-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-17T04:46:17.543 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-17T04:46:17.566 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-17T04:46:17.569 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../045-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-17T04:46:17.601 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-17T04:46:17.887 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-mako. 2024-09-17T04:46:17.914 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../046-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-17T04:46:17.920 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-natsort. 2024-09-17T04:46:17.946 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../043-python3-natsort_8.0.2-1_all.deb ... 2024-09-17T04:46:17.954 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-17T04:46:17.986 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-17T04:46:18.306 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-logutils. 2024-09-17T04:46:18.307 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-17T04:46:18.321 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../044-python3-logutils_0.3.3-8_all.deb ... 2024-09-17T04:46:18.334 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../047-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-17T04:46:18.356 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-17T04:46:18.373 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-17T04:46:18.625 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-17T04:46:18.652 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../045-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-17T04:46:18.676 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-17T04:46:18.692 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-17T04:46:18.703 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../048-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-17T04:46:18.743 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-17T04:46:19.020 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-mako. 2024-09-17T04:46:19.020 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-webob. 2024-09-17T04:46:19.036 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../049-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-17T04:46:19.046 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../046-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-17T04:46:19.079 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-17T04:46:19.086 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-17T04:46:19.398 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-waitress. 2024-09-17T04:46:19.415 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-17T04:46:19.425 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../050-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-17T04:46:19.441 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../047-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-17T04:46:19.460 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-17T04:46:19.481 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-17T04:46:19.742 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-17T04:46:19.751 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-tempita. 2024-09-17T04:46:19.757 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../048-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-17T04:46:19.778 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../051-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-17T04:46:19.792 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-17T04:46:19.817 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-17T04:46:20.061 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-webob. 2024-09-17T04:46:20.088 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../049-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-17T04:46:20.120 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-paste. 2024-09-17T04:46:20.128 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-17T04:46:20.148 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../052-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-17T04:46:20.187 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-17T04:46:20.481 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-waitress. 2024-09-17T04:46:20.482 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-17T04:46:20.497 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../053-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-17T04:46:20.508 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../050-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-17T04:46:20.540 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-17T04:46:20.551 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-17T04:46:20.826 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-17T04:46:20.853 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../054-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-17T04:46:20.859 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-tempita. 2024-09-17T04:46:20.886 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../051-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-17T04:46:20.892 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-17T04:46:20.925 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-17T04:46:21.195 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-webtest. 2024-09-17T04:46:21.220 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-paste. 2024-09-17T04:46:21.222 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../055-python3-webtest_2.0.35-1_all.deb ... 2024-09-17T04:46:21.247 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../052-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-17T04:46:21.262 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-17T04:46:21.286 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-17T04:46:21.573 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pecan. 2024-09-17T04:46:21.600 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../056-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-17T04:46:21.615 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-17T04:46:21.639 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-17T04:46:21.642 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../053-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-17T04:46:21.681 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-17T04:46:21.976 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-17T04:46:22.003 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../054-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-17T04:46:22.003 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-certifi. 2024-09-17T04:46:22.030 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../057-python3-certifi_2020.6.20-1_all.deb ... 2024-09-17T04:46:22.042 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-17T04:46:22.069 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-17T04:46:22.287 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-webtest. 2024-09-17T04:46:22.314 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../055-python3-webtest_2.0.35-1_all.deb ... 2024-09-17T04:46:22.353 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-17T04:46:22.389 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-idna. 2024-09-17T04:46:22.416 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../058-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-17T04:46:22.455 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-17T04:46:22.698 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pecan. 2024-09-17T04:46:22.725 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../056-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-17T04:46:22.765 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-17T04:46:22.766 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-urllib3. 2024-09-17T04:46:22.793 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../059-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-17T04:46:22.833 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-17T04:46:23.136 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-certifi. 2024-09-17T04:46:23.152 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-requests. 2024-09-17T04:46:23.163 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../057-python3-certifi_2020.6.20-1_all.deb ... 2024-09-17T04:46:23.179 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../060-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-17T04:46:23.461 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-17T04:46:23.586 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-17T04:46:24.239 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-17T04:46:24.266 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../061-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-17T04:46:24.297 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-17T04:46:24.323 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-idna. 2024-09-17T04:46:24.349 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../058-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-17T04:46:24.389 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-17T04:46:24.575 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-17T04:46:24.591 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../062-ceph-mgr-modules-core_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:24.625 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:24.709 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-urllib3. 2024-09-17T04:46:24.735 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../059-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-17T04:46:24.775 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-17T04:46:24.971 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-17T04:46:24.998 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../063-libsqlite3-mod-ceph_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:25.046 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:25.095 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-requests. 2024-09-17T04:46:25.122 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../060-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-17T04:46:25.161 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-17T04:46:25.374 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-17T04:46:25.401 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../064-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-17T04:46:25.440 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-17T04:46:25.481 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-17T04:46:25.508 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../061-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-17T04:46:25.548 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-17T04:46:25.785 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-openssl. 2024-09-17T04:46:25.812 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../065-python3-openssl_21.0.0-1_all.deb ... 2024-09-17T04:46:25.842 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-17T04:46:25.851 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-17T04:46:25.858 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../062-ceph-mgr-modules-core_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:25.900 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:26.179 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr. 2024-09-17T04:46:26.206 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../066-ceph-mgr_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:26.237 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:26.264 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-17T04:46:26.290 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../063-libsqlite3-mod-ceph_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:26.330 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:26.624 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mon. 2024-09-17T04:46:26.651 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../067-ceph-mon_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:26.675 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-17T04:46:26.690 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mon (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:26.701 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../064-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-17T04:46:26.741 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-17T04:46:27.044 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-openssl. 2024-09-17T04:46:27.071 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../065-python3-openssl_21.0.0-1_all.deb ... 2024-09-17T04:46:27.110 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-17T04:46:27.294 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-osd. 2024-09-17T04:46:27.321 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../068-ceph-osd_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:27.360 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-osd (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:27.431 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr. 2024-09-17T04:46:27.457 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../066-ceph-mgr_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:27.488 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:27.875 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mon. 2024-09-17T04:46:27.902 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../067-ceph-mon_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:27.942 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mon (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:28.413 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph. 2024-09-17T04:46:28.429 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../069-ceph_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:28.463 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:28.537 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-osd. 2024-09-17T04:46:28.563 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../068-ceph-osd_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:28.605 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-osd (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:28.949 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-fuse. 2024-09-17T04:46:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../070-ceph-fuse_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:29.015 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-fuse (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:29.385 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mds. 2024-09-17T04:46:29.412 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../071-ceph-mds_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:29.451 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mds (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:29.798 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph. 2024-09-17T04:46:29.813 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../069-ceph_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:29.848 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:29.854 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cephadm. 2024-09-17T04:46:29.881 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../072-cephadm_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:29.929 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cephadm (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:30.101 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-fuse. 2024-09-17T04:46:30.127 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../070-ceph-fuse_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:30.167 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-fuse (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:30.232 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-17T04:46:30.259 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../073-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-17T04:46:30.290 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-17T04:46:30.545 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mds. 2024-09-17T04:46:30.572 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../071-ceph-mds_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:30.609 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-17T04:46:30.611 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mds (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:30.636 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../074-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-17T04:46:30.675 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-17T04:46:31.023 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package cephadm. 2024-09-17T04:46:31.049 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../072-cephadm_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:31.089 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking cephadm (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:31.367 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-17T04:46:31.393 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../073-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-17T04:46:31.425 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-17T04:46:31.697 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-babel. 2024-09-17T04:46:31.724 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../075-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-17T04:46:31.752 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-17T04:46:31.755 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-17T04:46:31.768 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../074-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-17T04:46:31.811 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-17T04:46:32.050 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jinja2. 2024-09-17T04:46:32.077 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../076-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-17T04:46:32.108 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-17T04:46:32.427 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-17T04:46:32.443 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../077-ceph-mgr-cephadm_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:32.477 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:32.822 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-17T04:46:32.850 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../078-python3-repoze.lru_0.7-2_all.deb ... 2024-09-17T04:46:32.859 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-babel. 2024-09-17T04:46:32.880 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-17T04:46:32.883 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../075-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-17T04:46:32.918 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-17T04:46:33.167 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-routes. 2024-09-17T04:46:33.194 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../079-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-17T04:46:33.233 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-17T04:46:33.246 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-jinja2. 2024-09-17T04:46:33.273 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../076-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-17T04:46:33.312 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-17T04:46:33.536 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-17T04:46:33.563 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../080-ceph-mgr-dashboard_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:33.602 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:33.665 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-17T04:46:33.692 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../077-ceph-mgr-cephadm_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:33.731 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:34.077 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-17T04:46:34.104 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../078-python3-repoze.lru_0.7-2_all.deb ... 2024-09-17T04:46:34.143 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-17T04:46:34.471 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-routes. 2024-09-17T04:46:34.498 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../079-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-17T04:46:34.537 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-17T04:46:34.882 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-17T04:46:34.909 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../080-ceph-mgr-dashboard_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:34.949 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:36.190 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-17T04:46:36.207 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../081-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-17T04:46:36.249 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-17T04:46:36.735 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-joblib. 2024-09-17T04:46:36.762 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../082-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-17T04:46:36.879 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-17T04:46:37.241 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-17T04:46:37.257 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../083-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-17T04:46:37.299 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-17T04:46:37.602 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-sklearn. 2024-09-17T04:46:37.629 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../084-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-17T04:46:37.668 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-17T04:46:37.712 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-17T04:46:37.729 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../081-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-17T04:46:37.770 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-17T04:46:38.265 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-joblib. 2024-09-17T04:46:38.293 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../082-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-17T04:46:38.410 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-17T04:46:38.417 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-17T04:46:38.438 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../085-ceph-mgr-diskprediction-local_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:38.476 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:38.788 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-17T04:46:38.815 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../083-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-17T04:46:38.854 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-17T04:46:39.149 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-sklearn. 2024-09-17T04:46:39.176 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../084-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-17T04:46:39.215 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-17T04:46:39.447 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cachetools. 2024-09-17T04:46:39.475 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../086-python3-cachetools_5.0.0-1_all.deb ... 2024-09-17T04:46:39.513 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-17T04:46:39.782 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-17T04:46:39.810 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../087-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-17T04:46:39.841 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-17T04:46:40.136 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-17T04:46:40.163 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../088-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-17T04:46:40.174 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-17T04:46:40.202 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-17T04:46:40.203 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../085-ceph-mgr-diskprediction-local_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:40.241 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:40.505 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rsa. 2024-09-17T04:46:40.532 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../089-python3-rsa_4.8-1_all.deb ... 2024-09-17T04:46:40.571 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-17T04:46:40.892 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-google-auth. 2024-09-17T04:46:40.919 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../090-python3-google-auth_1.5.1-3_all.deb ... 2024-09-17T04:46:40.958 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-17T04:46:41.244 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-cachetools. 2024-09-17T04:46:41.261 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-17T04:46:41.271 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../086-python3-cachetools_5.0.0-1_all.deb ... 2024-09-17T04:46:41.289 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../091-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-17T04:46:41.310 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-17T04:46:41.328 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-17T04:46:41.613 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-17T04:46:41.631 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-websocket. 2024-09-17T04:46:41.632 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../087-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-17T04:46:41.658 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../092-python3-websocket_1.2.3-1_all.deb ... 2024-09-17T04:46:41.680 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-17T04:46:41.697 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-17T04:46:42.009 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-17T04:46:42.016 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-17T04:46:42.033 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../088-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-17T04:46:42.036 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../093-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-17T04:46:42.074 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-17T04:46:42.075 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-17T04:46:42.370 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-rsa. 2024-09-17T04:46:42.397 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../089-python3-rsa_4.8-1_all.deb ... 2024-09-17T04:46:42.427 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-17T04:46:42.731 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-17T04:46:42.731 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-google-auth. 2024-09-17T04:46:42.759 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../090-python3-google-auth_1.5.1-3_all.deb ... 2024-09-17T04:46:42.760 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../094-ceph-mgr-k8sevents_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:42.797 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:42.798 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-17T04:46:43.117 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-17T04:46:43.126 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-17T04:46:43.145 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../095-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-17T04:46:43.154 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../091-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-17T04:46:43.183 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-17T04:46:43.192 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-17T04:46:43.478 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-17T04:46:43.504 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-websocket. 2024-09-17T04:46:43.506 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../096-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-17T04:46:43.531 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../092-python3-websocket_1.2.3-1_all.deb ... 2024-09-17T04:46:43.536 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-17T04:46:43.570 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-17T04:46:43.797 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package jq. 2024-09-17T04:46:43.825 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../097-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-17T04:46:43.863 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-17T04:46:43.865 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-17T04:46:43.893 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../093-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-17T04:46:43.932 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-17T04:46:44.333 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package socat. 2024-09-17T04:46:44.349 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../098-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-17T04:46:44.391 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-17T04:46:44.545 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-17T04:46:44.562 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../094-ceph-mgr-k8sevents_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:44.604 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:44.836 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package xmlstarlet. 2024-09-17T04:46:44.852 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../099-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-17T04:46:44.894 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-17T04:46:44.940 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-17T04:46:44.968 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../095-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-17T04:46:45.006 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-17T04:46:45.265 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-test. 2024-09-17T04:46:45.281 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../100-ceph-test_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:45.323 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-test (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:45.360 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-17T04:46:45.388 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../096-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-17T04:46:45.426 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-17T04:46:45.662 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package jq. 2024-09-17T04:46:45.678 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../097-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-17T04:46:45.737 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-17T04:46:45.990 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package socat. 2024-09-17T04:46:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../098-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-17T04:46:46.048 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-17T04:46:46.376 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package xmlstarlet. 2024-09-17T04:46:46.405 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../099-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-17T04:46:46.443 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-17T04:46:46.805 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-test. 2024-09-17T04:46:46.833 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../100-ceph-test_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:46.872 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-test (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:47.311 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-volume. 2024-09-17T04:46:47.338 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../101-ceph-volume_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:47.377 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-volume (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:47.756 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cephfs-mirror. 2024-09-17T04:46:47.784 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../102-cephfs-mirror_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:47.822 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cephfs-mirror (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:48.134 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-colorama. 2024-09-17T04:46:48.162 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../103-python3-colorama_0.4.4-1_all.deb ... 2024-09-17T04:46:48.200 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-09-17T04:46:48.444 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cephfs-shell. 2024-09-17T04:46:48.461 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../104-cephfs-shell_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:48.494 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cephfs-shell (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:48.730 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cephfs-top. 2024-09-17T04:46:48.746 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../105-cephfs-top_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:48.790 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cephfs-top (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:49.059 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-17T04:46:49.060 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ceph-volume. 2024-09-17T04:46:49.087 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../101-ceph-volume_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:49.088 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../106-libcephfs-dev_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:49.125 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libcephfs-dev (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:49.126 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ceph-volume (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:49.428 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-17T04:46:49.457 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../107-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-17T04:46:49.495 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-17T04:46:49.513 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package cephfs-mirror. 2024-09-17T04:46:49.541 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../102-cephfs-mirror_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:49.588 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking cephfs-mirror (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:49.799 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-17T04:46:49.828 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../108-lua-sec_1.0.2-1_amd64.deb ... 2024-09-17T04:46:49.857 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-17T04:46:49.908 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-colorama. 2024-09-17T04:46:49.936 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../103-python3-colorama_0.4.4-1_all.deb ... 2024-09-17T04:46:49.974 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-09-17T04:46:50.218 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package cephfs-shell. 2024-09-17T04:46:50.219 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package nvme-cli. 2024-09-17T04:46:50.235 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../104-cephfs-shell_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:50.247 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../109-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-17T04:46:50.286 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-17T04:46:50.293 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking cephfs-shell (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:50.554 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package cephfs-top. 2024-09-17T04:46:50.582 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../105-cephfs-top_19.3.0-4989-gce1b342b-1jammy_all.deb ... 2024-09-17T04:46:50.621 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking cephfs-top (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:50.708 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package pkg-config. 2024-09-17T04:46:50.735 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../110-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-17T04:46:50.774 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-17T04:46:50.873 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-17T04:46:50.902 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../106-libcephfs-dev_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:50.939 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libcephfs-dev (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:51.035 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-17T04:46:51.063 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../111-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-17T04:46:51.101 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-17T04:46:51.259 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-17T04:46:51.288 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../107-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-17T04:46:51.326 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-17T04:46:51.513 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-17T04:46:51.540 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../112-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-17T04:46:51.579 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-17T04:46:51.655 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-17T04:46:51.683 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../108-lua-sec_1.0.2-1_amd64.deb ... 2024-09-17T04:46:51.721 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-17T04:46:51.882 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pastescript. 2024-09-17T04:46:51.910 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../113-python3-pastescript_2.0.2-4_all.deb ... 2024-09-17T04:46:51.948 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-17T04:46:52.117 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package nvme-cli. 2024-09-17T04:46:52.149 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../109-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-17T04:46:52.184 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-17T04:46:52.302 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pluggy. 2024-09-17T04:46:52.329 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../114-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-17T04:46:52.368 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-17T04:46:52.606 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package pkg-config. 2024-09-17T04:46:52.634 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../110-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-17T04:46:52.672 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-17T04:46:52.696 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-psutil. 2024-09-17T04:46:52.724 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../115-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-17T04:46:52.762 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-17T04:46:52.916 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-17T04:46:52.933 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../111-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-17T04:46:52.975 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-17T04:46:53.107 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-py. 2024-09-17T04:46:53.134 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../116-python3-py_1.10.0-1_all.deb ... 2024-09-17T04:46:53.181 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-17T04:46:53.436 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-17T04:46:53.465 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../112-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-17T04:46:53.485 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pygments. 2024-09-17T04:46:53.503 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-17T04:46:53.513 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../117-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-17T04:46:53.551 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-17T04:46:53.806 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pastescript. 2024-09-17T04:46:53.834 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../113-python3-pastescript_2.0.2-4_all.deb ... 2024-09-17T04:46:53.872 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-17T04:46:53.963 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-17T04:46:53.993 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../118-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-17T04:46:54.021 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-17T04:46:54.201 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pluggy. 2024-09-17T04:46:54.229 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../114-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-17T04:46:54.267 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-17T04:46:54.324 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-toml. 2024-09-17T04:46:54.352 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../119-python3-toml_0.10.2-1_all.deb ... 2024-09-17T04:46:54.391 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-17T04:46:54.545 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-psutil. 2024-09-17T04:46:54.573 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../115-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-17T04:46:54.605 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-17T04:46:54.677 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pytest. 2024-09-17T04:46:54.705 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../120-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-17T04:46:54.743 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-17T04:46:54.939 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-py. 2024-09-17T04:46:54.968 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../116-python3-py_1.10.0-1_all.deb ... 2024-09-17T04:46:55.006 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-17T04:46:55.080 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-simplejson. 2024-09-17T04:46:55.108 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../121-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-17T04:46:55.146 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-17T04:46:55.343 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pygments. 2024-09-17T04:46:55.371 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../117-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-17T04:46:55.399 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-17T04:46:55.409 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-17T04:46:55.415 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../122-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-17T04:46:55.482 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-17T04:46:56.138 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-17T04:46:56.167 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../118-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-17T04:46:56.203 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package radosgw. 2024-09-17T04:46:56.204 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-17T04:46:56.231 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../123-radosgw_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:56.269 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking radosgw (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:56.524 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-toml. 2024-09-17T04:46:56.553 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../119-python3-toml_0.10.2-1_all.deb ... 2024-09-17T04:46:56.590 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-17T04:46:56.894 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-pytest. 2024-09-17T04:46:56.922 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../120-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-17T04:46:56.960 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-17T04:46:57.314 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package python3-simplejson. 2024-09-17T04:46:57.343 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../121-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-17T04:46:57.380 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-17T04:46:57.431 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package rbd-fuse. 2024-09-17T04:46:57.459 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../124-rbd-fuse_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:57.497 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking rbd-fuse (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:57.649 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-17T04:46:57.670 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../122-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-17T04:46:57.707 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-17T04:46:57.809 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-17T04:46:58.068 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-17T04:46:58.186 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-17T04:46:58.421 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-17T04:46:58.462 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package radosgw. 2024-09-17T04:46:58.490 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../123-radosgw_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:58.528 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking radosgw (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:58.757 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-17T04:46:59.174 INFO:teuthology.orchestra.run.smithi176.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-17T04:46:59.229 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cephadm (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:46:59.622 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package rbd-fuse. 2024-09-17T04:46:59.636 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../124-rbd-fuse_19.3.0-4989-gce1b342b-1jammy_amd64.deb ... 2024-09-17T04:46:59.680 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking rbd-fuse (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:00.003 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-17T04:47:00.037 INFO:teuthology.orchestra.run.smithi176.stdout:Adding system user cephadm....done 2024-09-17T04:47:00.193 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-17T04:47:00.261 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-17T04:47:00.387 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-17T04:47:00.461 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-17T04:47:00.648 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-17T04:47:00.687 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-17T04:47:00.813 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-17T04:47:00.953 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-17T04:47:01.064 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-17T04:47:01.324 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-17T04:47:01.404 INFO:teuthology.orchestra.run.smithi150.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-17T04:47:01.450 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-17T04:47:01.480 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up cephadm (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:01.751 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-17T04:47:02.030 INFO:teuthology.orchestra.run.smithi150.stdout:Adding system user cephadm....done 2024-09-17T04:47:02.122 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-17T04:47:02.186 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-17T04:47:02.380 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-09-17T04:47:02.446 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-17T04:47:02.631 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-17T04:47:02.714 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-17T04:47:02.840 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-17T04:47:02.907 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-17T04:47:03.099 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-17T04:47:03.200 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-17T04:47:03.367 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-17T04:47:03.468 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-17T04:47:03.485 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-17T04:47:03.727 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-17T04:47:03.795 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-17T04:47:03.962 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-ceph-argparse (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:04.155 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-17T04:47:04.221 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-17T04:47:04.338 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-17T04:47:04.431 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-09-17T04:47:04.490 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-17T04:47:04.624 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-09-17T04:47:04.699 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-17T04:47:04.883 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-17T04:47:04.976 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-17T04:47:05.009 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-17T04:47:05.227 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-17T04:47:05.269 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-17T04:47:05.478 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-17T04:47:05.734 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-17T04:47:05.738 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-17T04:47:05.783 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-17T04:47:05.830 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-17T04:47:05.947 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-17T04:47:05.997 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-ceph-argparse (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:06.256 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-17T04:47:06.299 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-17T04:47:06.383 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-17T04:47:06.425 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-17T04:47:06.517 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-17T04:47:06.626 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-09-17T04:47:06.701 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-17T04:47:06.861 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-17T04:47:06.952 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-17T04:47:06.962 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-17T04:47:07.221 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-17T04:47:07.236 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up zip (3.0-12build2) ... 2024-09-17T04:47:07.337 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-17T04:47:07.606 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-17T04:47:07.636 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-17T04:47:07.699 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-17T04:47:07.834 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-17T04:47:07.972 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-17T04:47:08.177 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-17T04:47:08.240 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-17T04:47:08.277 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-17T04:47:08.366 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-17T04:47:08.492 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-17T04:47:08.529 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-17T04:47:08.796 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-17T04:47:08.810 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-17T04:47:09.097 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up zip (3.0-12build2) ... 2024-09-17T04:47:09.211 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-17T04:47:09.223 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-17T04:47:09.437 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-17T04:47:09.822 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-17T04:47:09.876 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-17T04:47:10.131 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-09-17T04:47:10.194 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-17T04:47:10.294 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-17T04:47:10.404 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-17T04:47:10.432 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-17T04:47:10.705 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-17T04:47:10.775 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-17T04:47:11.018 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-17T04:47:11.144 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-ceph-common (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:11.173 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-17T04:47:11.428 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-17T04:47:11.432 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-17T04:47:11.721 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-17T04:47:11.792 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-17T04:47:12.060 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-09-17T04:47:12.336 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-17T04:47:12.679 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-17T04:47:12.877 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-17T04:47:13.086 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-17T04:47:13.087 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-ceph-common (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:13.120 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T04:47:13.229 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-17T04:47:13.399 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-17T04:47:13.480 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-17T04:47:13.606 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-17T04:47:13.734 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-17T04:47:13.740 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-17T04:47:14.025 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-17T04:47:14.284 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-17T04:47:14.536 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T04:47:14.653 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-17T04:47:14.840 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-17T04:47:14.938 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-17T04:47:15.091 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T04:47:15.226 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-17T04:47:15.292 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-17T04:47:15.485 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-17T04:47:15.586 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-17T04:47:15.687 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-17T04:47:15.706 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-17T04:47:15.966 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-17T04:47:15.972 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-17T04:47:16.217 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-17T04:47:16.223 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-17T04:47:16.474 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T04:47:16.476 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-17T04:47:16.585 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-17T04:47:16.600 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-17T04:47:16.861 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-17T04:47:16.902 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-17T04:47:16.987 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-17T04:47:17.153 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-17T04:47:17.271 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-17T04:47:17.404 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-17T04:47:17.564 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-17T04:47:17.638 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-17T04:47:17.824 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-17T04:47:17.881 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-17T04:47:17.950 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-17T04:47:18.067 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-17T04:47:18.148 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-17T04:47:18.156 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-17T04:47:18.274 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-17T04:47:18.402 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-17T04:47:18.528 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-17T04:47:18.561 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-17T04:47:18.687 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-17T04:47:18.913 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-17T04:47:18.972 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-17T04:47:19.172 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-17T04:47:19.273 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-17T04:47:19.307 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-17T04:47:19.532 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-17T04:47:19.566 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T04:47:19.658 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-17T04:47:19.692 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-17T04:47:19.776 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-17T04:47:19.848 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-17T04:47:20.018 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-17T04:47:20.086 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-17T04:47:20.212 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-17T04:47:20.319 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-17T04:47:20.622 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-17T04:47:20.647 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-17T04:47:20.781 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-17T04:47:20.898 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-17T04:47:21.024 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-17T04:47:21.058 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-17T04:47:21.275 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T04:47:21.325 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-17T04:47:21.384 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-17T04:47:21.577 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-17T04:47:21.685 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-17T04:47:21.953 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-17T04:47:21.987 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-17T04:47:22.238 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-17T04:47:22.332 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-17T04:47:22.458 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-17T04:47:22.582 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-17T04:47:22.708 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librados2 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:22.734 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-17T04:47:22.834 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:22.952 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-17T04:47:22.994 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-17T04:47:23.262 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-17T04:47:23.271 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libcephfs2 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:23.397 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libradosstriper1 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:23.523 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-17T04:47:23.632 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-17T04:47:23.808 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librbd1 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:23.907 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-17T04:47:23.934 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-modules-core (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:24.060 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-fuse (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:24.243 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-17T04:47:24.264 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-17T04:47:24.264 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-17T04:47:24.352 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librados2 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:24.478 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:24.604 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-17T04:47:24.694 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libcephfs-dev (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:24.796 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rados (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:24.905 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librgw2 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:24.940 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libcephfs2 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:25.032 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cephfs-top (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:25.066 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libradosstriper1 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:25.174 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-17T04:47:25.192 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-17T04:47:25.468 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librbd1 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:25.603 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-modules-core (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:25.714 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-fuse (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:25.905 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-17T04:47:25.905 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-17T04:47:26.377 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rbd (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:26.431 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libcephfs-dev (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:26.478 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up rbd-fuse (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:26.556 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rados (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:26.579 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rgw (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:26.682 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up librgw2 (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:26.688 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cephfs (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:26.808 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up cephfs-top (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:26.814 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cephfs-shell (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:26.936 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-17T04:47:26.949 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-common (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:27.491 INFO:teuthology.orchestra.run.smithi176.stdout:Adding group ceph....done 2024-09-17T04:47:27.834 INFO:teuthology.orchestra.run.smithi176.stdout:Adding system user ceph....done 2024-09-17T04:47:28.019 INFO:teuthology.orchestra.run.smithi176.stdout:Setting system user ceph properties....done 2024-09-17T04:47:28.039 INFO:teuthology.orchestra.run.smithi176.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-17T04:47:28.163 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rbd (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:28.175 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-17T04:47:28.289 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up rbd-fuse (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:28.415 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-rgw (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:28.489 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-17T04:47:28.541 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up python3-cephfs (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:28.659 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up cephfs-shell (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:28.768 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-common (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:28.949 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cephfs-mirror (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:29.151 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-17T04:47:29.151 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-17T04:47:29.294 INFO:teuthology.orchestra.run.smithi150.stdout:Adding group ceph....done 2024-09-17T04:47:29.596 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-test (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:29.621 INFO:teuthology.orchestra.run.smithi150.stdout:Adding system user ceph....done 2024-09-17T04:47:29.773 INFO:teuthology.orchestra.run.smithi150.stdout:Setting system user ceph properties....done 2024-09-17T04:47:29.786 INFO:teuthology.orchestra.run.smithi150.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-17T04:47:29.791 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up radosgw (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:29.923 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-17T04:47:30.253 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-17T04:47:30.341 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-17T04:47:30.342 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-17T04:47:30.745 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up cephfs-mirror (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:30.779 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-base (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:30.952 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-17T04:47:30.952 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-17T04:47:31.130 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-17T04:47:31.396 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-test (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:31.581 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mds (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:31.652 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up radosgw (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:31.790 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-17T04:47:31.790 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-17T04:47:32.224 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:32.258 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-17T04:47:32.258 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-17T04:47:32.443 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-17T04:47:32.443 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-17T04:47:32.716 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-base (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:32.892 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-osd (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:33.036 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-17T04:47:33.164 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-17T04:47:33.164 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-17T04:47:33.484 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mds (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:33.611 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:33.710 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-17T04:47:33.710 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-17T04:47:33.728 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:33.863 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mon (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:34.058 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-17T04:47:34.058 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-17T04:47:34.195 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:34.411 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-17T04:47:34.411 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-17T04:47:34.522 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-cephadm (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:34.656 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:34.782 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-dashboard (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:34.904 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-osd (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:34.925 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-volume (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:35.182 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-17T04:47:35.182 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-17T04:47:35.261 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-17T04:47:35.378 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-17T04:47:35.539 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-17T04:47:35.681 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:35.807 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:35.958 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mon (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:36.171 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-17T04:47:36.171 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-17T04:47:36.341 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-17T04:47:36.667 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-cephadm (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:36.818 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:36.980 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-mgr-dashboard (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:37.012 INFO:teuthology.orchestra.run.smithi176.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-17T04:47:37.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-09-17T04:47:37.074 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-17T04:47:37.122 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ceph-volume (19.3.0-4989-gce1b342b-1jammy) ... 2024-09-17T04:47:37.287 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2024-09-17T04:47:37.288 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2024-09-17T04:47:37.484 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-17T04:47:37.486 INFO:teuthology.orchestra.run.smithi176.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-09-17T04:47:37.486 INFO:teuthology.orchestra.run.smithi176.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-09-17T04:47:37.486 INFO:teuthology.orchestra.run.smithi176.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-09-17T04:47:37.486 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T04:47:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T04:47:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T04:47:37.486 INFO:teuthology.orchestra.run.smithi176.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T04:47:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-17T04:47:37.487 INFO:teuthology.orchestra.run.smithi176.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-17T04:47:37.487 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T04:47:37.488 INFO:teuthology.orchestra.run.smithi176.stdout:The following additional packages will be installed: 2024-09-17T04:47:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-09-17T04:47:37.488 INFO:teuthology.orchestra.run.smithi176.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-09-17T04:47:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-09-17T04:47:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-09-17T04:47:37.489 INFO:teuthology.orchestra.run.smithi176.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-17T04:47:37.490 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2024-09-17T04:47:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-09-17T04:47:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-09-17T04:47:37.490 INFO:teuthology.orchestra.run.smithi176.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-09-17T04:47:37.533 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2024-09-17T04:47:37.533 INFO:teuthology.orchestra.run.smithi176.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-09-17T04:47:37.533 INFO:teuthology.orchestra.run.smithi176.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-09-17T04:47:37.534 INFO:teuthology.orchestra.run.smithi176.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-09-17T04:47:37.534 INFO:teuthology.orchestra.run.smithi176.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-09-17T04:47:37.534 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-09-17T04:47:37.534 INFO:teuthology.orchestra.run.smithi176.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-09-17T04:47:37.534 INFO:teuthology.orchestra.run.smithi176.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-17T04:47:37.535 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages will be upgraded: 2024-09-17T04:47:37.536 INFO:teuthology.orchestra.run.smithi176.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-09-17T04:47:37.584 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-17T04:47:37.716 INFO:teuthology.orchestra.run.smithi176.stdout:4 upgraded, 39 newly installed, 0 to remove and 327 not upgraded. 2024-09-17T04:47:37.716 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 55.8 MB of archives. 2024-09-17T04:47:37.716 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 203 MB of additional disk space will be used. 2024-09-17T04:47:37.716 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-09-17T04:47:37.737 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-17T04:47:38.335 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-09-17T04:47:38.351 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.18 [1,905 kB] 2024-09-17T04:47:38.475 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-09-17T04:47:38.476 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-09-17T04:47:38.477 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-09-17T04:47:38.505 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-09-17T04:47:38.507 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-09-17T04:47:38.511 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-09-17T04:47:38.514 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-09-17T04:47:38.515 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.3 [86.9 kB] 2024-09-17T04:47:38.518 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-09-17T04:47:38.578 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-09-17T04:47:38.579 INFO:teuthology.orchestra.run.smithi176.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-09-17T04:47:38.579 INFO:teuthology.orchestra.run.smithi176.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-09-17T04:47:38.659 INFO:teuthology.orchestra.run.smithi176.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-09-17T04:47:38.863 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-17T04:47:39.482 INFO:teuthology.orchestra.run.smithi176.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-09-17T04:47:39.487 INFO:teuthology.orchestra.run.smithi176.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-09-17T04:47:39.487 INFO:teuthology.orchestra.run.smithi176.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-09-17T04:47:39.494 INFO:teuthology.orchestra.run.smithi176.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libndp0 amd64 1.8-0ubuntu3.1 [12.4 kB] 2024-09-17T04:47:39.494 INFO:teuthology.orchestra.run.smithi176.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-09-17T04:47:39.505 INFO:teuthology.orchestra.run.smithi176.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.13-0ubuntu0.22.04.1 [149 kB] 2024-09-17T04:47:39.509 INFO:teuthology.orchestra.run.smithi176.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-09-17T04:47:39.524 INFO:teuthology.orchestra.run.smithi150.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-17T04:47:39.527 DEBUG:teuthology.orchestra.run.smithi150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-09-17T04:47:39.528 INFO:teuthology.orchestra.run.smithi176.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-09-17T04:47:39.528 INFO:teuthology.orchestra.run.smithi176.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.18 [2,375 kB] 2024-09-17T04:47:39.599 INFO:teuthology.orchestra.run.smithi176.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-09-17T04:47:39.613 INFO:teuthology.orchestra.run.smithi176.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-09-17T04:47:39.625 INFO:teuthology.orchestra.run.smithi150.stdout:Reading package lists... 2024-09-17T04:47:39.631 INFO:teuthology.orchestra.run.smithi176.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-09-17T04:47:39.703 INFO:teuthology.orchestra.run.smithi176.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-09-17T04:47:39.710 INFO:teuthology.orchestra.run.smithi176.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-09-17T04:47:39.711 INFO:teuthology.orchestra.run.smithi176.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-09-17T04:47:39.712 INFO:teuthology.orchestra.run.smithi176.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-09-17T04:47:39.713 INFO:teuthology.orchestra.run.smithi176.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.13-0ubuntu0.22.04.1 [1,225 kB] 2024-09-17T04:47:39.743 INFO:teuthology.orchestra.run.smithi176.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-09-17T04:47:39.746 INFO:teuthology.orchestra.run.smithi176.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.13-0ubuntu0.22.04.1 [16.2 MB] 2024-09-17T04:47:39.838 INFO:teuthology.orchestra.run.smithi150.stdout:Building dependency tree... 2024-09-17T04:47:39.839 INFO:teuthology.orchestra.run.smithi150.stdout:Reading state information... 2024-09-17T04:47:40.038 INFO:teuthology.orchestra.run.smithi150.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-09-17T04:47:40.038 INFO:teuthology.orchestra.run.smithi150.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-09-17T04:47:40.038 INFO:teuthology.orchestra.run.smithi150.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-09-17T04:47:40.038 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T04:47:40.038 INFO:teuthology.orchestra.run.smithi150.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T04:47:40.038 INFO:teuthology.orchestra.run.smithi150.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T04:47:40.038 INFO:teuthology.orchestra.run.smithi150.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T04:47:40.039 INFO:teuthology.orchestra.run.smithi150.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-17T04:47:40.039 INFO:teuthology.orchestra.run.smithi150.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-17T04:47:40.039 INFO:teuthology.orchestra.run.smithi150.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T04:47:40.039 INFO:teuthology.orchestra.run.smithi150.stdout:The following additional packages will be installed: 2024-09-17T04:47:40.039 INFO:teuthology.orchestra.run.smithi150.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-09-17T04:47:40.039 INFO:teuthology.orchestra.run.smithi150.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-09-17T04:47:40.039 INFO:teuthology.orchestra.run.smithi150.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-09-17T04:47:40.039 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-09-17T04:47:40.039 INFO:teuthology.orchestra.run.smithi150.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-17T04:47:40.039 INFO:teuthology.orchestra.run.smithi150.stdout:Suggested packages: 2024-09-17T04:47:40.040 INFO:teuthology.orchestra.run.smithi150.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-09-17T04:47:40.040 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-09-17T04:47:40.040 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-09-17T04:47:40.080 INFO:teuthology.orchestra.run.smithi150.stdout:The following NEW packages will be installed: 2024-09-17T04:47:40.080 INFO:teuthology.orchestra.run.smithi150.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-09-17T04:47:40.081 INFO:teuthology.orchestra.run.smithi150.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-09-17T04:47:40.081 INFO:teuthology.orchestra.run.smithi150.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-09-17T04:47:40.081 INFO:teuthology.orchestra.run.smithi150.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-09-17T04:47:40.081 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-09-17T04:47:40.081 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-09-17T04:47:40.082 INFO:teuthology.orchestra.run.smithi150.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-17T04:47:40.083 INFO:teuthology.orchestra.run.smithi150.stdout:The following packages will be upgraded: 2024-09-17T04:47:40.083 INFO:teuthology.orchestra.run.smithi150.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-09-17T04:47:40.134 INFO:teuthology.orchestra.run.smithi150.stdout:4 upgraded, 39 newly installed, 0 to remove and 327 not upgraded. 2024-09-17T04:47:40.134 INFO:teuthology.orchestra.run.smithi150.stdout:Need to get 55.8 MB of archives. 2024-09-17T04:47:40.134 INFO:teuthology.orchestra.run.smithi150.stdout:After this operation, 203 MB of additional disk space will be used. 2024-09-17T04:47:40.135 INFO:teuthology.orchestra.run.smithi150.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-09-17T04:47:40.268 INFO:teuthology.orchestra.run.smithi150.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-09-17T04:47:40.276 INFO:teuthology.orchestra.run.smithi150.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.18 [1,905 kB] 2024-09-17T04:47:40.287 INFO:teuthology.orchestra.run.smithi176.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-09-17T04:47:40.288 INFO:teuthology.orchestra.run.smithi176.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-09-17T04:47:40.288 INFO:teuthology.orchestra.run.smithi176.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-09-17T04:47:40.288 INFO:teuthology.orchestra.run.smithi176.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-09-17T04:47:40.289 INFO:teuthology.orchestra.run.smithi176.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-09-17T04:47:40.289 INFO:teuthology.orchestra.run.smithi176.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2.1 [1,482 kB] 2024-09-17T04:47:40.316 INFO:teuthology.orchestra.run.smithi150.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-09-17T04:47:40.317 INFO:teuthology.orchestra.run.smithi150.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-09-17T04:47:40.317 INFO:teuthology.orchestra.run.smithi176.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-09-17T04:47:40.318 INFO:teuthology.orchestra.run.smithi150.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-09-17T04:47:40.318 INFO:teuthology.orchestra.run.smithi176.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-09-17T04:47:40.328 INFO:teuthology.orchestra.run.smithi150.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-09-17T04:47:40.329 INFO:teuthology.orchestra.run.smithi150.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-09-17T04:47:40.331 INFO:teuthology.orchestra.run.smithi150.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-09-17T04:47:40.332 INFO:teuthology.orchestra.run.smithi150.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-09-17T04:47:40.333 INFO:teuthology.orchestra.run.smithi150.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.3 [86.9 kB] 2024-09-17T04:47:40.334 INFO:teuthology.orchestra.run.smithi150.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-09-17T04:47:40.335 INFO:teuthology.orchestra.run.smithi150.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-09-17T04:47:40.352 INFO:teuthology.orchestra.run.smithi150.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-09-17T04:47:40.352 INFO:teuthology.orchestra.run.smithi150.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-09-17T04:47:40.354 INFO:teuthology.orchestra.run.smithi150.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-09-17T04:47:40.649 INFO:teuthology.orchestra.run.smithi150.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-09-17T04:47:40.650 INFO:teuthology.orchestra.run.smithi150.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-09-17T04:47:40.650 INFO:teuthology.orchestra.run.smithi150.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-09-17T04:47:40.652 INFO:teuthology.orchestra.run.smithi150.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libndp0 amd64 1.8-0ubuntu3.1 [12.4 kB] 2024-09-17T04:47:40.697 INFO:teuthology.orchestra.run.smithi150.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-09-17T04:47:40.701 INFO:teuthology.orchestra.run.smithi150.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.13-0ubuntu0.22.04.1 [149 kB] 2024-09-17T04:47:40.703 INFO:teuthology.orchestra.run.smithi150.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-09-17T04:47:40.710 INFO:teuthology.orchestra.run.smithi150.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-09-17T04:47:40.710 INFO:teuthology.orchestra.run.smithi150.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.18 [2,375 kB] 2024-09-17T04:47:40.732 INFO:teuthology.orchestra.run.smithi150.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-09-17T04:47:40.732 INFO:teuthology.orchestra.run.smithi150.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-09-17T04:47:40.732 INFO:teuthology.orchestra.run.smithi176.stdout:Preconfiguring packages ... 2024-09-17T04:47:40.743 INFO:teuthology.orchestra.run.smithi150.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-09-17T04:47:40.763 INFO:teuthology.orchestra.run.smithi150.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-09-17T04:47:40.765 INFO:teuthology.orchestra.run.smithi150.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-09-17T04:47:40.766 INFO:teuthology.orchestra.run.smithi150.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-09-17T04:47:40.767 INFO:teuthology.orchestra.run.smithi150.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-09-17T04:47:40.767 INFO:teuthology.orchestra.run.smithi150.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.13-0ubuntu0.22.04.1 [1,225 kB] 2024-09-17T04:47:40.778 INFO:teuthology.orchestra.run.smithi150.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-09-17T04:47:40.779 INFO:teuthology.orchestra.run.smithi150.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.13-0ubuntu0.22.04.1 [16.2 MB] 2024-09-17T04:47:40.903 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 55.8 MB in 3s (20.1 MB/s) 2024-09-17T04:47:40.985 INFO:teuthology.orchestra.run.smithi150.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-09-17T04:47:40.985 INFO:teuthology.orchestra.run.smithi150.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-09-17T04:47:40.985 INFO:teuthology.orchestra.run.smithi150.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-09-17T04:47:40.985 INFO:teuthology.orchestra.run.smithi150.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-09-17T04:47:40.985 INFO:teuthology.orchestra.run.smithi150.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-09-17T04:47:40.985 INFO:teuthology.orchestra.run.smithi150.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2.1 [1,482 kB] 2024-09-17T04:47:40.997 INFO:teuthology.orchestra.run.smithi150.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-09-17T04:47:40.997 INFO:teuthology.orchestra.run.smithi150.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-09-17T04:47:41.273 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176353 files and directories currently installed.) 2024-09-17T04:47:41.279 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-17T04:47:41.405 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-17T04:47:41.406 INFO:teuthology.orchestra.run.smithi150.stdout:Preconfiguring packages ... 2024-09-17T04:47:41.571 INFO:teuthology.orchestra.run.smithi150.stdout:Fetched 55.8 MB in 1s (61.5 MB/s) 2024-09-17T04:47:41.851 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176353 files and directories currently installed.) 2024-09-17T04:47:41.856 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-17T04:47:42.010 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-17T04:47:42.028 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-09-17T04:47:42.750 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-09-17T04:47:42.958 INFO:teuthology.orchestra.run.smithi176.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-09-17T04:47:43.210 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package flex. 2024-09-17T04:47:43.304 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176353 files and directories currently installed.) 2024-09-17T04:47:43.309 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-09-17T04:47:43.350 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking flex (2.6.4-8build2) ... 2024-09-17T04:47:43.699 INFO:teuthology.orchestra.run.smithi150.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-09-17T04:47:43.859 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.18_amd64.deb ... 2024-09-17T04:47:43.966 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package flex. 2024-09-17T04:47:44.039 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.18) over (3.0.2-0ubuntu1.7) ... 2024-09-17T04:47:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176353 files and directories currently installed.) 2024-09-17T04:47:44.065 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-09-17T04:47:44.106 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking flex (2.6.4-8build2) ... 2024-09-17T04:47:44.425 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-17T04:47:44.631 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.18_amd64.deb ... 2024-09-17T04:47:44.812 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.18) over (3.0.2-0ubuntu1.7) ... 2024-09-17T04:47:44.953 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176426 files and directories currently installed.) 2024-09-17T04:47:44.958 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-17T04:47:45.079 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-17T04:47:45.232 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-17T04:47:45.514 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-17T04:47:45.644 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-17T04:47:45.749 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176426 files and directories currently installed.) 2024-09-17T04:47:45.755 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-17T04:47:45.886 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-17T04:47:45.971 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-09-17T04:47:46.259 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-09-17T04:47:46.368 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-09-17T04:47:46.712 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-09-17T04:47:46.821 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package bison. 2024-09-17T04:47:46.929 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176426 files and directories currently installed.) 2024-09-17T04:47:46.935 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-09-17T04:47:46.981 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-09-17T04:47:47.077 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package bison. 2024-09-17T04:47:47.183 INFO:teuthology.orchestra.run.smithi150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 176426 files and directories currently installed.) 2024-09-17T04:47:47.196 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-09-17T04:47:47.306 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-09-17T04:47:47.390 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-09-17T04:47:47.418 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-09-17T04:47:47.457 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-09-17T04:47:47.715 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-09-17T04:47:47.742 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-09-17T04:47:47.781 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-09-17T04:47:47.785 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package dump. 2024-09-17T04:47:47.813 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-09-17T04:47:47.851 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking dump (0.4b47-3) ... 2024-09-17T04:47:48.113 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package indent. 2024-09-17T04:47:48.117 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package dump. 2024-09-17T04:47:48.127 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-09-17T04:47:48.146 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-09-17T04:47:48.171 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-09-17T04:47:48.184 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking dump (0.4b47-3) ... 2024-09-17T04:47:48.424 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-09-17T04:47:48.440 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-09-17T04:47:48.463 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package indent. 2024-09-17T04:47:48.479 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-09-17T04:47:48.482 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-09-17T04:47:48.521 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-09-17T04:47:48.782 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-09-17T04:47:48.798 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-09-17T04:47:48.828 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-09-17T04:47:48.840 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-09-17T04:47:48.856 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.3_amd64.deb ... 2024-09-17T04:47:48.894 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.3) ... 2024-09-17T04:47:49.133 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-09-17T04:47:49.147 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-09-17T04:47:49.178 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-09-17T04:47:49.185 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-09-17T04:47:49.207 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.3_amd64.deb ... 2024-09-17T04:47:49.245 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.3) ... 2024-09-17T04:47:49.517 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libfl2:amd64. 2024-09-17T04:47:49.545 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-09-17T04:47:49.547 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-09-17T04:47:49.575 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-09-17T04:47:49.584 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-09-17T04:47:49.622 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-09-17T04:47:49.836 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-09-17T04:47:49.853 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-09-17T04:47:49.894 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-09-17T04:47:50.026 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libfl2:amd64. 2024-09-17T04:47:50.054 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-09-17T04:47:50.101 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-09-17T04:47:50.139 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libjson-perl. 2024-09-17T04:47:50.157 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-09-17T04:47:50.197 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-09-17T04:47:50.370 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-09-17T04:47:50.398 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-09-17T04:47:50.445 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-09-17T04:47:50.483 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libllvm14:amd64. 2024-09-17T04:47:50.511 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-09-17T04:47:50.550 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-09-17T04:47:50.748 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libjson-perl. 2024-09-17T04:47:50.776 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-09-17T04:47:50.823 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-09-17T04:47:51.176 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libllvm14:amd64. 2024-09-17T04:47:51.204 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-09-17T04:47:51.242 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-09-17T04:47:52.304 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-09-17T04:47:52.333 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-09-17T04:47:52.371 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-09-17T04:47:52.607 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libmbim-proxy. 2024-09-17T04:47:52.623 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-09-17T04:47:52.657 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-09-17T04:47:52.976 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-09-17T04:47:53.004 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-09-17T04:47:53.043 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-09-17T04:47:53.172 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-09-17T04:47:53.200 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-09-17T04:47:53.263 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-09-17T04:47:53.387 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libndp0:amd64. 2024-09-17T04:47:53.415 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3.1_amd64.deb ... 2024-09-17T04:47:53.454 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3.1) ... 2024-09-17T04:47:53.574 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libmbim-proxy. 2024-09-17T04:47:53.602 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-09-17T04:47:53.649 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-09-17T04:47:53.740 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libnm0:amd64. 2024-09-17T04:47:53.769 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-09-17T04:47:53.806 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-09-17T04:47:53.969 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-09-17T04:47:53.997 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-09-17T04:47:54.035 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-09-17T04:47:54.134 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libpq5:amd64. 2024-09-17T04:47:54.163 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../16-libpq5_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-17T04:47:54.209 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libpq5:amd64 (14.13-0ubuntu0.22.04.1) ... 2024-09-17T04:47:54.422 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libndp0:amd64. 2024-09-17T04:47:54.450 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3.1_amd64.deb ... 2024-09-17T04:47:54.497 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3.1) ... 2024-09-17T04:47:54.537 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-09-17T04:47:54.565 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-09-17T04:47:54.603 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-09-17T04:47:54.898 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqmi-proxy. 2024-09-17T04:47:54.914 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-09-17T04:47:54.916 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libnm0:amd64. 2024-09-17T04:47:54.944 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-09-17T04:47:54.956 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-09-17T04:47:54.991 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-09-17T04:47:55.208 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-09-17T04:47:55.225 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.18_amd64.deb ... 2024-09-17T04:47:55.267 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-17T04:47:55.361 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libpq5:amd64. 2024-09-17T04:47:55.389 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../16-libpq5_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-17T04:47:55.427 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libpq5:amd64 (14.13-0ubuntu0.22.04.1) ... 2024-09-17T04:47:55.796 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-09-17T04:47:55.814 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-09-17T04:47:55.824 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-09-17T04:47:55.842 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-09-17T04:47:55.862 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-09-17T04:47:55.889 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-09-17T04:47:56.149 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package modemmanager. 2024-09-17T04:47:56.176 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-09-17T04:47:56.215 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-09-17T04:47:56.250 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libqmi-proxy. 2024-09-17T04:47:56.278 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-09-17T04:47:56.325 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-09-17T04:47:56.627 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-09-17T04:47:56.644 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.18_amd64.deb ... 2024-09-17T04:47:56.686 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-17T04:47:56.788 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package network-manager. 2024-09-17T04:47:56.816 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-09-17T04:47:56.855 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-09-17T04:47:57.257 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-09-17T04:47:57.285 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-09-17T04:47:57.323 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-09-17T04:47:57.680 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ppp. 2024-09-17T04:47:57.709 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-09-17T04:47:57.727 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package modemmanager. 2024-09-17T04:47:57.755 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-09-17T04:47:57.794 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-09-17T04:47:57.979 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-09-17T04:47:58.293 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package pptp-linux. 2024-09-17T04:47:58.322 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-09-17T04:47:58.360 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-09-17T04:47:58.484 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package network-manager. 2024-09-17T04:47:58.513 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-09-17T04:47:58.559 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-09-17T04:47:58.613 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package network-manager-pptp. 2024-09-17T04:47:58.629 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-09-17T04:47:58.671 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-09-17T04:47:58.990 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql-client-common. 2024-09-17T04:47:59.019 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-09-17T04:47:59.057 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql-client-common (238) ... 2024-09-17T04:47:59.142 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package ppp. 2024-09-17T04:47:59.170 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-09-17T04:47:59.327 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql-client-14. 2024-09-17T04:47:59.356 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../27-postgresql-client-14_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-17T04:47:59.394 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql-client-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-17T04:47:59.426 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-09-17T04:47:59.722 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package pptp-linux. 2024-09-17T04:47:59.750 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-09-17T04:47:59.797 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-09-17T04:48:00.058 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package network-manager-pptp. 2024-09-17T04:48:00.065 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql-common. 2024-09-17T04:48:00.074 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-09-17T04:48:00.094 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-09-17T04:48:00.116 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-09-17T04:48:00.159 INFO:teuthology.orchestra.run.smithi176.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-09-17T04:48:00.187 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql-common (238) ... 2024-09-17T04:48:00.419 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql-client-common. 2024-09-17T04:48:00.447 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-09-17T04:48:00.485 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql-client-common (238) ... 2024-09-17T04:48:00.647 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql-14. 2024-09-17T04:48:00.675 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../29-postgresql-14_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-17T04:48:00.717 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-17T04:48:00.748 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql-client-14. 2024-09-17T04:48:00.776 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../27-postgresql-client-14_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-17T04:48:00.814 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql-client-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-17T04:48:01.444 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql-common. 2024-09-17T04:48:01.473 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-09-17T04:48:01.538 INFO:teuthology.orchestra.run.smithi150.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-09-17T04:48:01.567 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql-common (238) ... 2024-09-17T04:48:01.908 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql. 2024-09-17T04:48:01.925 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-09-17T04:48:01.966 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql (14+238) ... 2024-09-17T04:48:02.162 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql-14. 2024-09-17T04:48:02.191 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../29-postgresql-14_14.13-0ubuntu0.22.04.1_amd64.deb ... 2024-09-17T04:48:02.231 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-17T04:48:02.235 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql-client. 2024-09-17T04:48:02.251 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-09-17T04:48:02.301 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql-client (14+238) ... 2024-09-17T04:48:02.554 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package postgresql-contrib. 2024-09-17T04:48:02.582 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-09-17T04:48:02.620 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking postgresql-contrib (14+238) ... 2024-09-17T04:48:02.939 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package tcl8.6. 2024-09-17T04:48:02.969 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-09-17T04:48:03.006 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-09-17T04:48:03.267 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package tcl. 2024-09-17T04:48:03.284 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-09-17T04:48:03.321 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-09-17T04:48:03.686 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package wpasupplicant. 2024-09-17T04:48:03.715 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2.1_amd64.deb ... 2024-09-17T04:48:03.753 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2.1) ... 2024-09-17T04:48:04.124 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-09-17T04:48:04.152 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-09-17T04:48:04.185 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql. 2024-09-17T04:48:04.190 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-09-17T04:48:04.200 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-09-17T04:48:04.249 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql (14+238) ... 2024-09-17T04:48:04.493 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql-client. 2024-09-17T04:48:04.522 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-09-17T04:48:04.526 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package usb-modeswitch. 2024-09-17T04:48:04.555 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-09-17T04:48:04.559 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql-client (14+238) ... 2024-09-17T04:48:04.594 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-09-17T04:48:04.778 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package postgresql-contrib. 2024-09-17T04:48:04.804 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-09-17T04:48:04.836 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking postgresql-contrib (14+238) ... 2024-09-17T04:48:04.873 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql-client-common (238) ... 2024-09-17T04:48:05.082 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up flex (2.6.4-8build2) ... 2024-09-17T04:48:05.156 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package tcl8.6. 2024-09-17T04:48:05.185 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-09-17T04:48:05.209 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-09-17T04:48:05.222 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-09-17T04:48:05.334 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libpq5:amd64 (14.13-0ubuntu0.22.04.1) ... 2024-09-17T04:48:05.485 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-09-17T04:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package tcl. 2024-09-17T04:48:05.546 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-09-17T04:48:05.588 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-09-17T04:48:05.603 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-09-17T04:48:05.704 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql-client-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-17T04:48:05.894 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package wpasupplicant. 2024-09-17T04:48:05.920 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2.1_amd64.deb ... 2024-09-17T04:48:05.961 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2.1) ... 2024-09-17T04:48:06.193 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-09-17T04:48:06.292 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2.1) ... 2024-09-17T04:48:06.390 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-09-17T04:48:06.419 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-09-17T04:48:06.465 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-09-17T04:48:06.682 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-17T04:48:06.682 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-17T04:48:06.877 INFO:teuthology.orchestra.run.smithi150.stdout:Selecting previously unselected package usb-modeswitch. 2024-09-17T04:48:06.905 INFO:teuthology.orchestra.run.smithi150.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-09-17T04:48:06.952 INFO:teuthology.orchestra.run.smithi150.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-09-17T04:48:07.145 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-09-17T04:48:07.223 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql-client-common (238) ... 2024-09-17T04:48:07.254 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-09-17T04:48:07.364 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-17T04:48:07.433 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up flex (2.6.4-8build2) ... 2024-09-17T04:48:07.523 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-09-17T04:48:07.559 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-09-17T04:48:07.685 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libpq5:amd64 (14.13-0ubuntu0.22.04.1) ... 2024-09-17T04:48:07.744 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-09-17T04:48:07.811 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-09-17T04:48:07.954 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-09-17T04:48:08.055 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql-client-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-17T04:48:08.150 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-09-17T04:48:08.285 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-09-17T04:48:08.411 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-09-17T04:48:08.579 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-09-17T04:48:08.677 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2.1) ... 2024-09-17T04:48:09.049 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-09-17T04:48:09.057 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-17T04:48:09.057 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-17T04:48:09.175 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up dump (0.4b47-3) ... 2024-09-17T04:48:09.268 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-09-17T04:48:09.326 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-09-17T04:48:09.436 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.3) ... 2024-09-17T04:48:09.522 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-09-17T04:48:09.562 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up tcl (8.6.11+1build2) ... 2024-09-17T04:48:09.647 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-09-17T04:48:09.688 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libjson-perl (4.04000-1) ... 2024-09-17T04:48:09.773 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.18) ... 2024-09-17T04:48:09.814 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-09-17T04:48:09.908 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-09-17T04:48:09.911 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-09-17T04:48:09.982 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-09-17T04:48:10.053 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-09-17T04:48:10.092 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-09-17T04:48:10.160 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-09-17T04:48:10.267 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3.1) ... 2024-09-17T04:48:10.311 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-09-17T04:48:10.402 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-09-17T04:48:10.428 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-09-17T04:48:10.519 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-09-17T04:48:10.662 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql-client (14+238) ... 2024-09-17T04:48:10.788 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-09-17T04:48:11.033 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-09-17T04:48:11.159 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up dump (0.4b47-3) ... 2024-09-17T04:48:11.257 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:11.257 INFO:teuthology.orchestra.run.smithi176.stdout:The following network interfaces were found in /etc/network/interfaces 2024-09-17T04:48:11.257 INFO:teuthology.orchestra.run.smithi176.stdout:which means they are currently configured by ifupdown: 2024-09-17T04:48:11.257 INFO:teuthology.orchestra.run.smithi176.stdout:- enp3s0f1 2024-09-17T04:48:11.257 INFO:teuthology.orchestra.run.smithi176.stdout:If you want to manage those interfaces with NetworkManager instead 2024-09-17T04:48:11.257 INFO:teuthology.orchestra.run.smithi176.stdout:remove their configuration from /etc/network/interfaces. 2024-09-17T04:48:11.257 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:11.270 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-09-17T04:48:11.344 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-09-17T04:48:11.390 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-09-17T04:48:11.462 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.3) ... 2024-09-17T04:48:11.580 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up tcl (8.6.11+1build2) ... 2024-09-17T04:48:11.680 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libjson-perl (4.04000-1) ... 2024-09-17T04:48:11.723 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-09-17T04:48:11.806 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-09-17T04:48:11.897 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-09-17T04:48:11.967 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-09-17T04:48:12.039 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-09-17T04:48:12.092 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-09-17T04:48:12.269 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3.1) ... 2024-09-17T04:48:12.395 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-09-17T04:48:12.512 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-09-17T04:48:12.532 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-09-17T04:48:12.655 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql-client (14+238) ... 2024-09-17T04:48:12.667 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-09-17T04:48:12.765 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-09-17T04:48:12.785 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-09-17T04:48:12.886 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql-common (238) ... 2024-09-17T04:48:13.394 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:13.395 INFO:teuthology.orchestra.run.smithi150.stdout:The following network interfaces were found in /etc/network/interfaces 2024-09-17T04:48:13.395 INFO:teuthology.orchestra.run.smithi150.stdout:which means they are currently configured by ifupdown: 2024-09-17T04:48:13.395 INFO:teuthology.orchestra.run.smithi150.stdout:- enp3s0f1 2024-09-17T04:48:13.395 INFO:teuthology.orchestra.run.smithi150.stdout:If you want to manage those interfaces with NetworkManager instead 2024-09-17T04:48:13.395 INFO:teuthology.orchestra.run.smithi150.stdout:remove their configuration from /etc/network/interfaces. 2024-09-17T04:48:13.395 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:13.529 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-09-17T04:48:13.819 INFO:teuthology.orchestra.run.smithi176.stdout:Adding user postgres to group ssl-cert 2024-09-17T04:48:13.849 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-09-17T04:48:14.046 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:14.046 INFO:teuthology.orchestra.run.smithi176.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-09-17T04:48:14.180 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-09-17T04:48:14.241 INFO:teuthology.orchestra.run.smithi176.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-09-17T04:48:14.241 INFO:teuthology.orchestra.run.smithi176.stdout:Removing obsolete dictionary files: 2024-09-17T04:48:14.684 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-09-17T04:48:14.778 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-09-17T04:48:14.828 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-09-17T04:48:14.953 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-09-17T04:48:15.079 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql-common (238) ... 2024-09-17T04:48:15.147 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-17T04:48:15.469 INFO:teuthology.orchestra.run.smithi176.stdout:Creating new PostgreSQL cluster 14/main ... 2024-09-17T04:48:15.486 INFO:teuthology.orchestra.run.smithi176.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-09-17T04:48:15.524 INFO:teuthology.orchestra.run.smithi176.stdout:The files belonging to this database system will be owned by user "postgres". 2024-09-17T04:48:15.524 INFO:teuthology.orchestra.run.smithi176.stdout:This user must also own the server process. 2024-09-17T04:48:15.524 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:15.524 INFO:teuthology.orchestra.run.smithi176.stdout:The database cluster will be initialized with locale "en_US". 2024-09-17T04:48:15.524 INFO:teuthology.orchestra.run.smithi176.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-09-17T04:48:15.524 INFO:teuthology.orchestra.run.smithi176.stdout:The default text search configuration will be set to "english". 2024-09-17T04:48:15.524 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:15.524 INFO:teuthology.orchestra.run.smithi176.stdout:Data page checksums are disabled. 2024-09-17T04:48:15.524 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:15.525 INFO:teuthology.orchestra.run.smithi176.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-09-17T04:48:15.525 INFO:teuthology.orchestra.run.smithi176.stdout:creating subdirectories ... ok 2024-09-17T04:48:15.525 INFO:teuthology.orchestra.run.smithi176.stdout:selecting dynamic shared memory implementation ... posix 2024-09-17T04:48:15.540 INFO:teuthology.orchestra.run.smithi176.stdout:selecting default max_connections ... 100 2024-09-17T04:48:15.560 INFO:teuthology.orchestra.run.smithi176.stdout:selecting default shared_buffers ... 128MB 2024-09-17T04:48:15.570 INFO:teuthology.orchestra.run.smithi176.stdout:selecting default time zone ... Etc/UTC 2024-09-17T04:48:15.571 INFO:teuthology.orchestra.run.smithi176.stdout:creating configuration files ... ok 2024-09-17T04:48:15.682 INFO:teuthology.orchestra.run.smithi176.stdout:running bootstrap script ... ok 2024-09-17T04:48:16.065 INFO:teuthology.orchestra.run.smithi150.stdout:Adding user postgres to group ssl-cert 2024-09-17T04:48:16.261 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:16.261 INFO:teuthology.orchestra.run.smithi150.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-09-17T04:48:16.325 INFO:teuthology.orchestra.run.smithi176.stdout:performing post-bootstrap initialization ... ok 2024-09-17T04:48:16.451 INFO:teuthology.orchestra.run.smithi150.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-09-17T04:48:16.451 INFO:teuthology.orchestra.run.smithi150.stdout:Removing obsolete dictionary files: 2024-09-17T04:48:17.022 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-09-17T04:48:17.151 INFO:teuthology.orchestra.run.smithi176.stdout:syncing data to disk ... ok 2024-09-17T04:48:17.414 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql-14 (14.13-0ubuntu0.22.04.1) ... 2024-09-17T04:48:17.518 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-09-17T04:48:17.744 INFO:teuthology.orchestra.run.smithi150.stdout:Creating new PostgreSQL cluster 14/main ... 2024-09-17T04:48:17.769 INFO:teuthology.orchestra.run.smithi150.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-09-17T04:48:17.810 INFO:teuthology.orchestra.run.smithi150.stdout:The files belonging to this database system will be owned by user "postgres". 2024-09-17T04:48:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:This user must also own the server process. 2024-09-17T04:48:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:The database cluster will be initialized with locale "en_US". 2024-09-17T04:48:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-09-17T04:48:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:The default text search configuration will be set to "english". 2024-09-17T04:48:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:Data page checksums are disabled. 2024-09-17T04:48:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-09-17T04:48:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:creating subdirectories ... ok 2024-09-17T04:48:17.811 INFO:teuthology.orchestra.run.smithi150.stdout:selecting dynamic shared memory implementation ... posix 2024-09-17T04:48:17.824 INFO:teuthology.orchestra.run.smithi150.stdout:selecting default max_connections ... 100 2024-09-17T04:48:17.841 INFO:teuthology.orchestra.run.smithi150.stdout:selecting default shared_buffers ... 128MB 2024-09-17T04:48:17.850 INFO:teuthology.orchestra.run.smithi150.stdout:selecting default time zone ... Etc/UTC 2024-09-17T04:48:17.851 INFO:teuthology.orchestra.run.smithi150.stdout:creating configuration files ... ok 2024-09-17T04:48:17.964 INFO:teuthology.orchestra.run.smithi150.stdout:running bootstrap script ... ok 2024-09-17T04:48:18.642 INFO:teuthology.orchestra.run.smithi150.stdout:performing post-bootstrap initialization ... ok 2024-09-17T04:48:19.610 INFO:teuthology.orchestra.run.smithi150.stdout:syncing data to disk ... ok 2024-09-17T04:48:20.004 INFO:teuthology.orchestra.run.smithi150.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-09-17T04:48:20.237 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-09-17T04:48:20.364 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql-contrib (14+238) ... 2024-09-17T04:48:20.489 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up postgresql (14+238) ... 2024-09-17T04:48:20.615 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-09-17T04:48:20.725 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-09-17T04:48:20.953 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-09-17T04:48:20.953 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-09-17T04:48:21.477 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-17T04:48:21.561 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-09-17T04:48:21.863 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-17T04:48:22.040 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-17T04:48:22.738 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-09-17T04:48:22.865 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql-contrib (14+238) ... 2024-09-17T04:48:22.990 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up postgresql (14+238) ... 2024-09-17T04:48:23.117 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-09-17T04:48:23.243 INFO:teuthology.orchestra.run.smithi150.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-09-17T04:48:23.499 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-09-17T04:48:23.499 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-09-17T04:48:24.035 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-09-17T04:48:24.095 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-17T04:48:24.126 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-17T04:48:24.187 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-09-17T04:48:24.490 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-17T04:48:24.608 INFO:teuthology.orchestra.run.smithi176.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-17T04:48:24.614 DEBUG:teuthology.parallel:result is None 2024-09-17T04:48:24.642 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-17T04:48:25.995 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-09-17T04:48:26.095 INFO:teuthology.orchestra.run.smithi150.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-17T04:48:26.625 INFO:teuthology.orchestra.run.smithi150.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-17T04:48:26.630 DEBUG:teuthology.parallel:result is None 2024-09-17T04:48:26.631 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T04:48:26.780 DEBUG:teuthology.orchestra.run.smithi150:> dpkg-query -W -f '${Version}' ceph 2024-09-17T04:48:26.799 INFO:teuthology.orchestra.run.smithi150.stdout:19.3.0-4989-gce1b342b-1jammy 2024-09-17T04:48:26.799 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4989-gce1b342b-1jammy 2024-09-17T04:48:26.800 INFO:teuthology.task.install:The correct ceph version 19.3.0-4989-gce1b342b-1jammy is installed. 2024-09-17T04:48:26.801 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T04:48:26.939 DEBUG:teuthology.orchestra.run.smithi176:> dpkg-query -W -f '${Version}' ceph 2024-09-17T04:48:26.959 INFO:teuthology.orchestra.run.smithi176.stdout:19.3.0-4989-gce1b342b-1jammy 2024-09-17T04:48:26.959 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4989-gce1b342b-1jammy 2024-09-17T04:48:26.959 INFO:teuthology.task.install:The correct ceph version 19.3.0-4989-gce1b342b-1jammy is installed. 2024-09-17T04:48:26.961 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T04:48:26.962 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:48:26.962 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T04:48:26.976 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:48:26.976 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T04:48:27.017 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T04:48:27.017 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:48:27.017 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T04:48:27.030 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T04:48:27.085 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:48:27.085 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T04:48:27.099 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T04:48:27.154 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T04:48:27.154 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:48:27.154 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T04:48:27.167 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T04:48:27.221 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:48:27.221 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T04:48:27.234 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T04:48:27.290 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T04:48:27.290 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:48:27.291 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T04:48:27.303 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T04:48:27.356 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:48:27.356 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T04:48:27.369 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T04:48:27.423 INFO:teuthology.run_tasks:Running task ceph... 2024-09-17T04:48:27.500 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-17T04:48:27.500 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /var/log/ceph 2024-09-17T04:48:27.502 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /var/log/ceph 2024-09-17T04:48:27.515 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-17T04:48:27.515 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T04:48:27.559 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T04:48:27.571 INFO:tasks.ceph:Creating extra log directories... 2024-09-17T04:48:27.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T04:48:27.615 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T04:48:27.634 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-17T04:48:27.634 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-17T04:48:27.634 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-17T04:48:27.634 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:48:27.634 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-17T04:48:27.674 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-17T04:48:27.750 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:48:27.750 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-17T04:48:27.755 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-17T04:48:27.835 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-17T04:48:27.836 INFO:tasks.ceph:config {'conf': {'client': {'client_shutdown_timeout': 120}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-17T04:48:27.836 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553', 'branch': 'wip-pdonnell-testing-20240916.200549-debug', 'description': 'fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7908553', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240916.200549-debug'}, 'ceph': {'cephfs': {'fs': [{'name': 'a'}, {'name': 'b'}], 'session_timeout': 300}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'client_shutdown_timeout': 120}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'ceph-fuse': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}}, 'kclient': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240916.200549-debug', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 90, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'seed': 476, 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'sleep_before_teardown': 0, 'subset': '320/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240916.200549-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGqHXZJ+NevIFOUtaSgOQC+9oVBsa6O9Rg5eTIQLLfUvetEwd3tfyVlJOjEfbxaaZK40s7VE9lTyEb96I5o7e1Q=', 'smithi176.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNv7q2LI+oN0FRGKoTHLVcqcqEuvW0iqhHW+hk9sWEZKXSSNzGV5hYVcEEbrSgu5znAL9Z/DQQDemv84tZo3HAY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'fs': [{'name': 'a'}, {'name': 'b'}], 'session_timeout': 300}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'client_shutdown_timeout': 120}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_02:09:51', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-17T04:48:27.836 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T04:48:27.839 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T04:48:27.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T04:48:27.888 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T04:48:27.936 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:48:27.936 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T04:48:27.946 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T04:48:27.946 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-09-17T04:48:27.994 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T04:48:27.995 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T04:48:27.995 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 874 Links: 1 2024-09-17T04:48:27.995 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T04:48:27.995 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-17 04:42:10.931853296 +0000 2024-09-17T04:48:27.995 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-17 04:42:10.675855583 +0000 2024-09-17T04:48:27.995 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-17 04:42:10.675855583 +0000 2024-09-17T04:48:27.995 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2024-09-17T04:48:27.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T04:48:28.048 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-17T04:48:28.048 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-17T04:48:28.049 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000439913 s, 1.2 MB/s 2024-09-17T04:48:28.050 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T04:48:28.099 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-09-17T04:48:28.146 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T04:48:28.146 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T04:48:28.146 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 894 Links: 1 2024-09-17T04:48:28.146 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T04:48:28.146 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-17 04:42:11.423848901 +0000 2024-09-17T04:48:28.146 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-17 04:42:11.167851188 +0000 2024-09-17T04:48:28.146 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-17 04:42:11.167851188 +0000 2024-09-17T04:48:28.147 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2024-09-17T04:48:28.147 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T04:48:28.200 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-17T04:48:28.200 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-17T04:48:28.200 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.00044482 s, 1.2 MB/s 2024-09-17T04:48:28.201 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T04:48:28.255 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-09-17T04:48:28.302 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T04:48:28.302 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T04:48:28.302 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-09-17T04:48:28.302 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T04:48:28.302 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-17 04:42:11.931844364 +0000 2024-09-17T04:48:28.302 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-17 04:42:11.655846829 +0000 2024-09-17T04:48:28.302 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-17 04:42:11.655846829 +0000 2024-09-17T04:48:28.302 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2024-09-17T04:48:28.303 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T04:48:28.356 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-17T04:48:28.356 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-17T04:48:28.356 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.00041579 s, 1.2 MB/s 2024-09-17T04:48:28.358 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T04:48:28.407 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-09-17T04:48:28.454 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T04:48:28.454 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T04:48:28.454 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-09-17T04:48:28.454 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T04:48:28.454 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-09-17 04:42:12.395840218 +0000 2024-09-17T04:48:28.454 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-17 04:42:12.163842290 +0000 2024-09-17T04:48:28.454 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-17 04:42:12.163842290 +0000 2024-09-17T04:48:28.454 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2024-09-17T04:48:28.454 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T04:48:28.508 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-09-17T04:48:28.508 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-09-17T04:48:28.508 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.00040477 s, 1.3 MB/s 2024-09-17T04:48:28.509 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T04:48:28.559 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T04:48:28.559 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:48:28.559 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T04:48:28.566 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T04:48:28.566 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-09-17T04:48:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T04:48:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T04:48:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 875 Links: 1 2024-09-17T04:48:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T04:48:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-09-17 04:42:06.697648363 +0000 2024-09-17T04:48:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-09-17 04:42:06.441650651 +0000 2024-09-17T04:48:28.612 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-09-17 04:42:06.441650651 +0000 2024-09-17T04:48:28.612 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-09-17T04:48:28.613 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T04:48:28.666 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-09-17T04:48:28.667 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-09-17T04:48:28.667 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000374144 s, 1.4 MB/s 2024-09-17T04:48:28.668 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T04:48:28.717 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-09-17T04:48:28.763 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T04:48:28.763 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T04:48:28.764 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 893 Links: 1 2024-09-17T04:48:28.764 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T04:48:28.764 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-09-17 04:42:07.185644005 +0000 2024-09-17T04:48:28.764 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-09-17 04:42:06.913646435 +0000 2024-09-17T04:48:28.764 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-09-17 04:42:06.913646435 +0000 2024-09-17T04:48:28.764 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-09-17T04:48:28.764 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T04:48:28.817 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-09-17T04:48:28.818 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-09-17T04:48:28.818 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000354435 s, 1.4 MB/s 2024-09-17T04:48:28.819 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T04:48:28.869 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-09-17T04:48:28.915 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T04:48:28.915 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T04:48:28.916 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-09-17T04:48:28.916 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T04:48:28.916 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-09-17 04:42:07.705639360 +0000 2024-09-17T04:48:28.916 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-09-17 04:42:07.437641754 +0000 2024-09-17T04:48:28.916 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-09-17 04:42:07.437641754 +0000 2024-09-17T04:48:28.916 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-09-17T04:48:28.916 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T04:48:28.970 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-09-17T04:48:28.970 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-09-17T04:48:28.970 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000301116 s, 1.7 MB/s 2024-09-17T04:48:28.971 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T04:48:29.021 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-09-17T04:48:29.067 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T04:48:29.067 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T04:48:29.068 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 928 Links: 1 2024-09-17T04:48:29.068 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T04:48:29.068 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-09-17 04:42:08.177635145 +0000 2024-09-17T04:48:29.068 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-09-17 04:42:07.929637360 +0000 2024-09-17T04:48:29.068 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-09-17 04:42:07.929637360 +0000 2024-09-17T04:48:29.068 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-09-17T04:48:29.068 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T04:48:29.121 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-09-17T04:48:29.122 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-09-17T04:48:29.122 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000265607 s, 1.9 MB/s 2024-09-17T04:48:29.123 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T04:48:29.173 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T04:48:29.173 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-17T04:48:29.173 INFO:tasks.ceph:Generating config... 2024-09-17T04:48:29.174 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-09-17T04:48:29.174 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-17T04:48:29.175 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-17T04:48:29.175 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-17T04:48:29.175 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-17T04:48:29.175 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-17T04:48:29.175 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-09-17T04:48:29.175 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-17T04:48:29.175 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-17T04:48:29.175 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-17T04:48:29.175 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-17T04:48:29.175 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-17T04:48:29.175 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-17T04:48:29.175 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-17T04:48:29.175 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-17T04:48:29.175 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-17T04:48:29.176 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-17T04:48:29.176 INFO:tasks.ceph:[osd] filestore flush min = 0 2024-09-17T04:48:29.176 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-17T04:48:29.176 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-17T04:48:29.176 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-17T04:48:29.176 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-17T04:48:29.176 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-17T04:48:29.176 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-17T04:48:29.176 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-17T04:48:29.176 INFO:tasks.ceph:Setting up mon.a... 2024-09-17T04:48:29.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-17T04:48:29.240 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.keyring 2024-09-17T04:48:29.261 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-17T04:48:29.330 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T04:48:29.340 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.150'), ('mon.b', '172.21.15.176'), ('mon.c', '[v2:172.21.15.176:3301,v1:172.21.15.176:6790]')] 2024-09-17T04:48:29.341 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.150,172.21.15.176,[v2:172.21.15.176:3301,v1:172.21.15.176:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client_shutdown_timeout': 120}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}} 2024-09-17T04:48:29.342 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:48:29.342 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T04:48:29.385 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.150 --add b 172.21.15.176 --addv c '[v2:172.21.15.176:3301,v1:172.21.15.176:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:48:29.491 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:48:29.491 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: generated fsid 172984ad-e98f-497a-a83c-2299d42da6ea 2024-09-17T04:48:29.491 INFO:teuthology.orchestra.run.smithi150.stdout:setting min_mon_release = quincy 2024-09-17T04:48:29.491 INFO:teuthology.orchestra.run.smithi150.stdout:epoch 0 2024-09-17T04:48:29.491 INFO:teuthology.orchestra.run.smithi150.stdout:fsid 172984ad-e98f-497a-a83c-2299d42da6ea 2024-09-17T04:48:29.491 INFO:teuthology.orchestra.run.smithi150.stdout:last_changed 2024-09-17T04:48:29.496218+0000 2024-09-17T04:48:29.492 INFO:teuthology.orchestra.run.smithi150.stdout:created 2024-09-17T04:48:29.496218+0000 2024-09-17T04:48:29.492 INFO:teuthology.orchestra.run.smithi150.stdout:min_mon_release 17 (quincy) 2024-09-17T04:48:29.492 INFO:teuthology.orchestra.run.smithi150.stdout:election_strategy: 1 2024-09-17T04:48:29.492 INFO:teuthology.orchestra.run.smithi150.stdout:0: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.a 2024-09-17T04:48:29.492 INFO:teuthology.orchestra.run.smithi150.stdout:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-17T04:48:29.492 INFO:teuthology.orchestra.run.smithi150.stdout:2: [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] mon.c 2024-09-17T04:48:29.493 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-17T04:48:29.497 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T04:48:29.502 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 172984ad-e98f-497a-a83c-2299d42da6ea... 2024-09-17T04:48:29.503 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:48:29.547 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:48:29.573 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-09-17T04:48:29.573 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2024-09-17T04:48:29.573 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:48:29.573 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2024-09-17T04:48:29.573 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.573 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2024-09-17T04:48:29.573 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.573 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2024-09-17T04:48:29.573 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.573 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2024-09-17T04:48:29.574 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2024-09-17T04:48:29.574 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.574 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2024-09-17T04:48:29.574 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2024-09-17T04:48:29.574 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.574 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:48:29.574 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2024-09-17T04:48:29.574 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.574 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:48:29.574 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:48:29.574 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.575 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2024-09-17T04:48:29.575 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.575 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2024-09-17T04:48:29.575 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2024-09-17T04:48:29.575 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:48:29.575 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:48:29.575 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2024-09-17T04:48:29.575 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:48:29.575 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2024-09-17T04:48:29.575 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:48:29.575 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:48:29.575 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2024-09-17T04:48:29.576 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.576 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:48:29.576 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.576 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2024-09-17T04:48:29.576 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.576 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2024-09-17T04:48:29.576 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.576 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2024-09-17T04:48:29.576 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2024-09-17T04:48:29.576 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2024-09-17T04:48:29.576 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = "172.21.15.150,172.21.15.176,[v2:172.21.15.176:3301,v1:172.21.15.176:6790]" 2024-09-17T04:48:29.577 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = 172984ad-e98f-497a-a83c-2299d42da6ea 2024-09-17T04:48:29.577 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.577 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2024-09-17T04:48:29.577 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2024-09-17T04:48:29.577 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.577 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2024-09-17T04:48:29.577 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2024-09-17T04:48:29.577 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:48:29.577 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.577 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2024-09-17T04:48:29.577 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2024-09-17T04:48:29.578 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.578 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2024-09-17T04:48:29.578 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2024-09-17T04:48:29.578 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2024-09-17T04:48:29.578 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug trim objects = true 2024-09-17T04:48:29.578 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.578 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2024-09-17T04:48:29.578 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2024-09-17T04:48:29.578 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.578 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:48:29.578 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.579 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2024-09-17T04:48:29.579 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.579 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2024-09-17T04:48:29.579 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.579 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:48:29.579 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2024-09-17T04:48:29.579 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.579 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2024-09-17T04:48:29.579 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2024-09-17T04:48:29.579 INFO:teuthology.orchestra.run.smithi150.stdout: bdev async discard = True 2024-09-17T04:48:29.579 INFO:teuthology.orchestra.run.smithi150.stdout: bdev enable discard = True 2024-09-17T04:48:29.579 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore allocator = bitmap 2024-09-17T04:48:29.580 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2024-09-17T04:48:29.580 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2024-09-17T04:48:29.580 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 1/20 2024-09-17T04:48:29.580 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 1/20 2024-09-17T04:48:29.580 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-09-17T04:48:29.580 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2024-09-17T04:48:29.580 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 4/10 2024-09-17T04:48:29.580 INFO:teuthology.orchestra.run.smithi150.stdout: filestore flush min = 0 2024-09-17T04:48:29.580 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:48:29.580 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2024-09-17T04:48:29.580 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:48:29.581 INFO:teuthology.orchestra.run.smithi150.stdout: osd blocked scrub grace period = 3600 2024-09-17T04:48:29.581 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:48:29.581 INFO:teuthology.orchestra.run.smithi150.stdout: osd max markdown count = 1000 2024-09-17T04:48:29.581 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2024-09-17T04:48:29.581 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.581 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2024-09-17T04:48:29.581 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-09-17T04:48:29.581 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2024-09-17T04:48:29.581 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-09-17T04:48:29.581 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-09-17T04:48:29.581 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:48:29.582 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:48:29.582 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2024-09-17T04:48:29.582 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.582 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2024-09-17T04:48:29.582 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-09-17T04:48:29.582 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-09-17T04:48:29.582 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2024-09-17T04:48:29.582 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-09-17T04:48:29.582 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2024-09-17T04:48:29.582 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2024-09-17T04:48:29.582 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:48:29.583 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2024-09-17T04:48:29.583 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2024-09-17T04:48:29.583 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:48:29.583 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.583 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:48:29.583 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:48:29.583 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:48:29.583 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.583 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:48:29.583 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:48:29.583 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout: mon down mkfs grace = 300 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:48:29.584 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:48:29.585 INFO:teuthology.orchestra.run.smithi150.stdout: client_shutdown_timeout = 120 2024-09-17T04:48:29.585 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2024-09-17T04:48:29.585 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.b] 2024-09-17T04:48:29.585 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.c] 2024-09-17T04:48:29.585 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2024-09-17T04:48:29.585 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2024-09-17T04:48:29.585 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2024-09-17T04:48:29.585 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2024-09-17T04:48:29.586 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2024-09-17T04:48:29.586 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2024-09-17T04:48:29.586 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:48:29.586 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2024-09-17T04:48:29.586 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.586 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2024-09-17T04:48:29.586 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.586 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2024-09-17T04:48:29.587 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.587 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2024-09-17T04:48:29.587 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2024-09-17T04:48:29.587 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.587 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2024-09-17T04:48:29.587 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2024-09-17T04:48:29.587 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.587 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:48:29.587 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2024-09-17T04:48:29.587 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.587 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:48:29.588 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:48:29.588 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.588 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2024-09-17T04:48:29.588 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.588 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2024-09-17T04:48:29.588 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2024-09-17T04:48:29.588 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:48:29.588 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:48:29.588 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2024-09-17T04:48:29.588 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:48:29.588 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2024-09-17T04:48:29.589 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:48:29.589 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:48:29.589 INFO:teuthology.orchestra.run.smithi176.stdout: mon_allow_pool_size_one = true 2024-09-17T04:48:29.589 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.589 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:48:29.589 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.589 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2024-09-17T04:48:29.589 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.589 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2024-09-17T04:48:29.589 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.589 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2024-09-17T04:48:29.589 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2024-09-17T04:48:29.590 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2024-09-17T04:48:29.590 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = "172.21.15.150,172.21.15.176,[v2:172.21.15.176:3301,v1:172.21.15.176:6790]" 2024-09-17T04:48:29.590 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = 172984ad-e98f-497a-a83c-2299d42da6ea 2024-09-17T04:48:29.590 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.590 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2024-09-17T04:48:29.590 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2024-09-17T04:48:29.590 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.590 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2024-09-17T04:48:29.590 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2024-09-17T04:48:29.590 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:48:29.590 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.590 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2024-09-17T04:48:29.591 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2024-09-17T04:48:29.591 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.591 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2024-09-17T04:48:29.591 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2024-09-17T04:48:29.591 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2024-09-17T04:48:29.591 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug trim objects = true 2024-09-17T04:48:29.591 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.591 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2024-09-17T04:48:29.591 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2024-09-17T04:48:29.591 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.592 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:48:29.592 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.592 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2024-09-17T04:48:29.592 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.592 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2024-09-17T04:48:29.592 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.592 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:48:29.592 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2024-09-17T04:48:29.592 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.592 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2024-09-17T04:48:29.593 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2024-09-17T04:48:29.593 INFO:teuthology.orchestra.run.smithi176.stdout: bdev async discard = True 2024-09-17T04:48:29.593 INFO:teuthology.orchestra.run.smithi176.stdout: bdev enable discard = True 2024-09-17T04:48:29.593 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore allocator = bitmap 2024-09-17T04:48:29.593 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2024-09-17T04:48:29.593 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2024-09-17T04:48:29.593 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 1/20 2024-09-17T04:48:29.593 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 1/20 2024-09-17T04:48:29.593 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-09-17T04:48:29.593 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2024-09-17T04:48:29.593 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 4/10 2024-09-17T04:48:29.594 INFO:teuthology.orchestra.run.smithi176.stdout: filestore flush min = 0 2024-09-17T04:48:29.594 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:48:29.594 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2024-09-17T04:48:29.594 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:48:29.594 INFO:teuthology.orchestra.run.smithi176.stdout: osd blocked scrub grace period = 3600 2024-09-17T04:48:29.594 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:48:29.594 INFO:teuthology.orchestra.run.smithi176.stdout: osd max markdown count = 1000 2024-09-17T04:48:29.594 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2024-09-17T04:48:29.594 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.594 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2024-09-17T04:48:29.594 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-09-17T04:48:29.595 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2024-09-17T04:48:29.595 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-09-17T04:48:29.595 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-09-17T04:48:29.595 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:48:29.595 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:48:29.595 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2024-09-17T04:48:29.595 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.595 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2024-09-17T04:48:29.595 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-09-17T04:48:29.595 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-09-17T04:48:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2024-09-17T04:48:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-09-17T04:48:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2024-09-17T04:48:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2024-09-17T04:48:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:48:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2024-09-17T04:48:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2024-09-17T04:48:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:48:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:48:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:48:29.596 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: # 1m isn't quite enough 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: mon down mkfs grace = 300 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2024-09-17T04:48:29.597 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2024-09-17T04:48:29.598 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2024-09-17T04:48:29.598 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2024-09-17T04:48:29.598 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:48:29.598 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:48:29.598 INFO:teuthology.orchestra.run.smithi176.stdout: client_shutdown_timeout = 120 2024-09-17T04:48:29.598 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2024-09-17T04:48:29.598 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.b] 2024-09-17T04:48:29.598 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.c] 2024-09-17T04:48:29.598 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.a] 2024-09-17T04:48:29.599 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.c] 2024-09-17T04:48:29.599 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.b] 2024-09-17T04:48:29.599 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.d] 2024-09-17T04:48:29.602 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-17T04:48:29.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-17T04:48:29.722 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-17T04:48:29.722 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:48:29.723 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-17T04:48:29.727 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:48:29.727 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-17T04:48:29.774 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi150.front.sepia.ceph.com 2024-09-17T04:48:29.774 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:48:29.774 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T04:48:29.774 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T04:48:29.836 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:48:29.836 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:48:29.882 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2024-09-17T04:48:29.882 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:48:29.882 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T04:48:29.882 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T04:48:29.904 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:48:29.904 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:48:29.955 INFO:tasks.ceph:Setting up mon nodes... 2024-09-17T04:48:29.956 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-17T04:48:29.956 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-17T04:48:30.030 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-17T04:48:30.037 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T04:48:30.111 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T04:48:30.134 INFO:tasks.ceph:Setting up mds nodes... 2024-09-17T04:48:30.134 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-17T04:48:30.206 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-17T04:48:30.212 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-17T04:48:30.223 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-17T04:48:30.332 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-17T04:48:30.339 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-17T04:48:30.351 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-17T04:48:30.426 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-17T04:48:30.432 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-17T04:48:30.444 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-17T04:48:30.550 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-17T04:48:30.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-17T04:48:30.567 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-17T04:48:30.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T04:48:30.634 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-17T04:48:30.648 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-17T04:48:30.711 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-17T04:48:30.725 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-17T04:48:30.725 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-17T04:48:30.725 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-17T04:48:30.738 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T04:48:30.738 INFO:tasks.ceph:role: osd.0 2024-09-17T04:48:30.738 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-17T04:48:30.739 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T04:48:30.801 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T04:48:30.802 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T04:48:30.802 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T04:48:30.802 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T04:48:30.802 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T04:48:30.802 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-09-17T04:48:30.802 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T04:48:30.802 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T04:48:30.802 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T04:48:30.802 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T04:48:31.267 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-09-17T04:48:31.269 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-09-17T04:48:31.269 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-17T04:48:31.421 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-17T04:48:31.433 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T04:48:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:48:31.434 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-17T04:48:31.489 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T04:48:31.489 INFO:tasks.ceph:role: osd.1 2024-09-17T04:48:31.489 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-17T04:48:31.490 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T04:48:31.555 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T04:48:31.555 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T04:48:31.555 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T04:48:31.555 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T04:48:31.555 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T04:48:31.555 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-09-17T04:48:31.555 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T04:48:31.556 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T04:48:31.556 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T04:48:31.556 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T04:48:32.009 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-09-17T04:48:32.009 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-09-17T04:48:32.009 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-17T04:48:32.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-17T04:48:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:48:32.110 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T04:48:32.111 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-17T04:48:32.165 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T04:48:32.165 INFO:tasks.ceph:role: osd.2 2024-09-17T04:48:32.165 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-17T04:48:32.166 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T04:48:32.229 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T04:48:32.229 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T04:48:32.229 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T04:48:32.229 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T04:48:32.229 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T04:48:32.229 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-09-17T04:48:32.229 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T04:48:32.229 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T04:48:32.229 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T04:48:32.229 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T04:48:32.689 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-09-17T04:48:32.691 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-09-17T04:48:32.691 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-17T04:48:32.776 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-17T04:48:32.827 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T04:48:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:48:32.829 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-17T04:48:32.882 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T04:48:32.882 INFO:tasks.ceph:role: osd.3 2024-09-17T04:48:32.882 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-17T04:48:32.882 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-17T04:48:32.948 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T04:48:32.948 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T04:48:32.948 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T04:48:32.949 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T04:48:32.949 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T04:48:32.949 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-09-17T04:48:32.949 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T04:48:32.949 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T04:48:32.949 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T04:48:32.949 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T04:48:33.449 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-09-17T04:48:33.450 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-09-17T04:48:33.450 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-17T04:48:33.597 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-17T04:48:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:48:33.609 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T04:48:33.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:48:33.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:33.765+0000 7fea98cd57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-17T04:48:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:33.765+0000 7fea98cd57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-17T04:48:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:33.765+0000 7fea98cd57c0 -1 bdev(0x55b6d8890a00 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-17T04:48:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:33.765+0000 7fea98cd57c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-17T04:48:36.918 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T04:48:36.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:48:37.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:37.049+0000 7f2b6d1427c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-17T04:48:37.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:37.049+0000 7f2b6d1427c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-17T04:48:37.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:37.049+0000 7f2b6d1427c0 -1 bdev(0x55b5d2dbca00 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-17T04:48:37.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:37.049+0000 7f2b6d1427c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-17T04:48:40.198 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T04:48:40.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:48:40.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:40.329+0000 7fde10b807c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-17T04:48:40.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:40.329+0000 7fde10b807c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-17T04:48:40.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:40.329+0000 7fde10b807c0 -1 bdev(0x55de1497aa00 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-17T04:48:40.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:40.329+0000 7fde10b807c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-17T04:48:43.457 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T04:48:43.471 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:48:43.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:43.585+0000 7f4c954b27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-17T04:48:43.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:43.585+0000 7f4c954b27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-17T04:48:43.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:43.585+0000 7f4c954b27c0 -1 bdev(0x556cfb116a00 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-17T04:48:43.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:48:43.585+0000 7f4c954b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-17T04:48:46.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T04:48:46.979 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-17T04:48:46.995 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T04:48:46.995 INFO:tasks.ceph:role: osd.4 2024-09-17T04:48:46.995 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-17T04:48:46.995 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T04:48:47.059 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T04:48:47.059 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T04:48:47.059 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T04:48:47.059 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T04:48:47.059 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T04:48:47.059 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-09-17T04:48:47.059 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T04:48:47.060 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T04:48:47.060 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T04:48:47.060 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T04:48:47.536 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-09-17T04:48:47.537 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-09-17T04:48:47.538 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-17T04:48:47.694 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-17T04:48:47.705 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T04:48:47.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:48:47.707 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-17T04:48:47.762 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T04:48:47.762 INFO:tasks.ceph:role: osd.5 2024-09-17T04:48:47.762 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-17T04:48:47.763 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T04:48:47.830 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T04:48:47.831 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T04:48:47.831 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T04:48:47.831 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T04:48:47.831 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T04:48:47.831 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-09-17T04:48:47.831 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T04:48:47.831 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T04:48:47.831 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T04:48:47.831 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T04:48:48.288 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-09-17T04:48:48.290 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-09-17T04:48:48.290 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-17T04:48:48.374 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-17T04:48:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:48:48.426 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T04:48:48.426 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-17T04:48:48.478 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T04:48:48.478 INFO:tasks.ceph:role: osd.6 2024-09-17T04:48:48.479 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-17T04:48:48.479 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T04:48:48.544 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T04:48:48.544 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T04:48:48.544 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T04:48:48.544 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T04:48:48.544 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T04:48:48.544 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-09-17T04:48:48.544 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T04:48:48.544 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T04:48:48.544 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T04:48:48.545 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T04:48:49.002 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-09-17T04:48:49.004 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-09-17T04:48:49.004 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-17T04:48:49.148 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-17T04:48:49.159 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T04:48:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:48:49.160 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-17T04:48:49.219 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T04:48:49.219 INFO:tasks.ceph:role: osd.7 2024-09-17T04:48:49.219 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-17T04:48:49.219 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-17T04:48:49.282 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T04:48:49.282 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T04:48:49.282 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T04:48:49.282 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T04:48:49.282 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T04:48:49.282 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-09-17T04:48:49.282 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T04:48:49.282 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T04:48:49.282 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T04:48:49.282 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T04:48:49.771 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-09-17T04:48:49.774 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-09-17T04:48:49.774 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-17T04:48:49.911 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-17T04:48:49.921 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T04:48:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:48:49.922 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:48:50.068 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:50.066+0000 7f9f92c577c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-17T04:48:50.069 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:50.066+0000 7f9f92c577c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-17T04:48:50.069 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:50.066+0000 7f9f92c577c0 -1 bdev(0x556726d84a00 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-17T04:48:50.069 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:50.066+0000 7f9f92c577c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-17T04:48:53.230 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T04:48:53.247 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:48:53.360 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:53.358+0000 7f17f76137c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-17T04:48:53.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:53.358+0000 7f17f76137c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-17T04:48:53.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:53.358+0000 7f17f76137c0 -1 bdev(0x558bd842ca00 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-17T04:48:53.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:53.358+0000 7f17f76137c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-17T04:48:56.537 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T04:48:56.551 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:48:56.661 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:56.658+0000 7f30117d97c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-17T04:48:56.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:56.658+0000 7f30117d97c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-17T04:48:56.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:56.658+0000 7f30117d97c0 -1 bdev(0x55d7e1dc8a00 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-17T04:48:56.662 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:56.658+0000 7f30117d97c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-17T04:48:57.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:48:57.841 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:48:59.794 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T04:48:59.807 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:48:59.921 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:59.918+0000 7fbf400f37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-17T04:48:59.922 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:59.918+0000 7fbf400f37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-17T04:48:59.922 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:59.918+0000 7fbf400f37c0 -1 bdev(0x556c7cf80a00 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-17T04:48:59.922 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:48:59.918+0000 7fbf400f37c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-17T04:49:03.104 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T04:49:03.121 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-17T04:49:03.122 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:49:03.122 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-17T04:49:03.136 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:49:03.136 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-17T04:49:03.189 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:49:03.189 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-17T04:49:03.244 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:49:03.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-17T04:49:03.297 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:49:03.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-17T04:49:03.353 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:49:03.353 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-17T04:49:03.409 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:49:03.409 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-17T04:49:03.464 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:49:03.465 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-17T04:49:03.478 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:49:03.479 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-17T04:49:03.535 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:49:03.535 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-17T04:49:03.590 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:49:03.590 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-17T04:49:03.642 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:49:03.642 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-17T04:49:03.694 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:49:03.694 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-17T04:49:03.747 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:49:03.747 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-17T04:49:03.802 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:49:03.802 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-17T04:49:03.808 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:49:03.808 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-17T04:49:03.847 INFO:tasks.ceph:Adding keys to all mons... 2024-09-17T04:49:03.848 DEBUG:teuthology.orchestra.run.smithi150:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T04:49:03.850 DEBUG:teuthology.orchestra.run.smithi176:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T04:49:03.893 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.y] 2024-09-17T04:49:03.893 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAeCulm/zogAhAA/fBwzhNR2wmmQTA7n5tktA== 2024-09-17T04:49:03.893 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAeCulmIf6WDBAAVq/xrS1iNzTvvp6FM7HHQw== 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAeCulmmtAjFBAAPt7jgB6PcsrW2eCnTTCVlg== 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.0] 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAhCulmMV7TLRAA1er//zpaze6NnrHBpklcug== 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.1] 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAlCulmMOYqAxAANDccDRJAh2x+MsQtw/k6pg== 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.2] 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAoCulmxr2/ExAAkiNa8GJ9nN2PEZTdY5WOzA== 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.3] 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQArCulmjCchIxAA7nuoaEEtqxgvM6tUocHSuw== 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.x] 2024-09-17T04:49:03.894 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAeCulmy/OzBhAAqzX61wh5HWEEW4/6wFi3OA== 2024-09-17T04:49:03.895 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2024-09-17T04:49:03.895 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAeCulmmT9xGRAAffhmwHH+otUsKHw6o3uoGQ== 2024-09-17T04:49:03.895 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2024-09-17T04:49:03.895 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAeCulmjBTWIBAAwFgdGTAN1aPj3MwTYYs0zA== 2024-09-17T04:49:03.895 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.4] 2024-09-17T04:49:03.895 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAyCulmINEaBBAASu+IdEcUiBuqxs6zj5x9cA== 2024-09-17T04:49:03.895 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.5] 2024-09-17T04:49:03.895 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQA1CulmenqJFRAAWiF36LGizcTLNrqNioFEoA== 2024-09-17T04:49:03.896 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.6] 2024-09-17T04:49:03.896 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQA4CulmsW1/JxAAPqVZM+zKDW5tIm74SYdSyA== 2024-09-17T04:49:03.896 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.7] 2024-09-17T04:49:03.896 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQA7Culmmkv0NhAAUDhI8M+5H0PQZh/OJ7vKDw== 2024-09-17T04:49:03.896 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-09-17T04:49:03.896 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAeCulm8uMUJhAAzTjpzFYGlfymWWvN9AxLlQ== 2024-09-17T04:49:03.896 INFO:teuthology.orchestra.run.smithi150.stdout:[client.1] 2024-09-17T04:49:03.896 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAeCulmkwdtKhAAEAa3src5wBZTPCv6/ScTyw== 2024-09-17T04:49:03.902 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.y] 2024-09-17T04:49:03.902 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAeCulm/zogAhAA/fBwzhNR2wmmQTA7n5tktA== 2024-09-17T04:49:03.902 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.a] 2024-09-17T04:49:03.902 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAeCulmIf6WDBAAVq/xrS1iNzTvvp6FM7HHQw== 2024-09-17T04:49:03.902 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.c] 2024-09-17T04:49:03.902 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAeCulmmtAjFBAAPt7jgB6PcsrW2eCnTTCVlg== 2024-09-17T04:49:03.902 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.0] 2024-09-17T04:49:03.902 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAhCulmMV7TLRAA1er//zpaze6NnrHBpklcug== 2024-09-17T04:49:03.902 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.1] 2024-09-17T04:49:03.902 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAlCulmMOYqAxAANDccDRJAh2x+MsQtw/k6pg== 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.2] 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAoCulmxr2/ExAAkiNa8GJ9nN2PEZTdY5WOzA== 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.3] 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQArCulmjCchIxAA7nuoaEEtqxgvM6tUocHSuw== 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.x] 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAeCulmy/OzBhAAqzX61wh5HWEEW4/6wFi3OA== 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.b] 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAeCulmmT9xGRAAffhmwHH+otUsKHw6o3uoGQ== 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout:[mds.d] 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAeCulmjBTWIBAAwFgdGTAN1aPj3MwTYYs0zA== 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.4] 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAyCulmINEaBBAASu+IdEcUiBuqxs6zj5x9cA== 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.5] 2024-09-17T04:49:03.903 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQA1CulmenqJFRAAWiF36LGizcTLNrqNioFEoA== 2024-09-17T04:49:03.904 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.6] 2024-09-17T04:49:03.904 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQA4CulmsW1/JxAAPqVZM+zKDW5tIm74SYdSyA== 2024-09-17T04:49:03.904 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.7] 2024-09-17T04:49:03.904 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQA7Culmmkv0NhAAUDhI8M+5H0PQZh/OJ7vKDw== 2024-09-17T04:49:03.904 INFO:teuthology.orchestra.run.smithi176.stdout:[client.0] 2024-09-17T04:49:03.904 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAeCulm8uMUJhAAzTjpzFYGlfymWWvN9AxLlQ== 2024-09-17T04:49:03.904 INFO:teuthology.orchestra.run.smithi176.stdout:[client.1] 2024-09-17T04:49:03.904 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAeCulmkwdtKhAAEAa3src5wBZTPCv6/ScTyw== 2024-09-17T04:49:03.905 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T04:49:03.943 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T04:49:04.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T04:49:04.028 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T04:49:04.104 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T04:49:04.107 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T04:49:04.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.182 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.253 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.255 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.325 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.328 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.399 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.401 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.474 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T04:49:04.476 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T04:49:04.544 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T04:49:04.547 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T04:49:04.617 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T04:49:04.620 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T04:49:04.691 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.693 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.761 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.764 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.836 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.838 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.909 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T04:49:04.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T04:49:04.979 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T04:49:05.050 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T04:49:05.052 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T04:49:05.124 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-17T04:49:05.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-17T04:49:05.134 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T04:49:05.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-17T04:49:05.265 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-17T04:49:05.279 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T04:49:05.410 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-17T04:49:05.420 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-17T04:49:05.472 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T04:49:05.597 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-17T04:49:05.609 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:49:05.611 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T04:49:05.655 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-17T04:49:05.655 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T04:49:05.655 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T04:49:05.658 INFO:tasks.ceph.mon.a:Started 2024-09-17T04:49:05.659 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T04:49:05.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T04:49:05.700 INFO:tasks.ceph.mon.b:Started 2024-09-17T04:49:05.700 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T04:49:05.700 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T04:49:05.702 INFO:tasks.ceph.mon.c:Started 2024-09-17T04:49:05.702 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-17T04:49:05.703 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-17T04:49:05.703 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-17T04:49:05.705 INFO:tasks.ceph.mgr.y:Started 2024-09-17T04:49:05.706 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-17T04:49:05.706 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-17T04:49:05.708 INFO:tasks.ceph.mgr.x:Started 2024-09-17T04:49:05.708 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-09-17T04:49:05.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-09-17T04:49:05.710 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-09-17T04:49:05.710 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set client debug_client 20 2024-09-17T04:49:05.758 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-09-17T04:49:05.758 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set client debug_ms 1 2024-09-17T04:49:05.802 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-09-17T04:49:05.802 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-09-17T04:49:05.846 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-09-17T04:49:05.846 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-09-17T04:49:05.890 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-09-17T04:49:05.890 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-09-17T04:49:05.934 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-09-17T04:49:05.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-09-17T04:49:05.978 INFO:tasks.ceph:setting config [mds] debug_ms = 2 2024-09-17T04:49:05.978 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds debug_ms 2 2024-09-17T04:49:06.022 INFO:tasks.ceph:setting config [mds] mds_bal_fragment_size_max = 10000 2024-09-17T04:49:06.022 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds mds_bal_fragment_size_max 10000 2024-09-17T04:49:06.066 INFO:tasks.ceph:setting config [mds] mds_bal_merge_size = 5 2024-09-17T04:49:06.066 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds mds_bal_merge_size 5 2024-09-17T04:49:06.110 INFO:tasks.ceph:setting config [mds] mds_bal_split_bits = 3 2024-09-17T04:49:06.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds mds_bal_split_bits 3 2024-09-17T04:49:06.154 INFO:tasks.ceph:setting config [mds] mds_bal_split_size = 100 2024-09-17T04:49:06.154 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds mds_bal_split_size 100 2024-09-17T04:49:06.198 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-09-17T04:49:06.198 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-09-17T04:49:06.242 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-09-17T04:49:06.242 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-09-17T04:49:06.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.280+0000 7fc7877fe640 1 -- 172.21.15.150:0/3149123229 <== mon.2 v2:172.21.15.176:3301/0 9 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v3)=0 v3) ==== 109+0+0 (secure 0 0 0) 0x7fc7a014daf0 con 0x7fc7a00ae920 2024-09-17T04:49:06.279 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-09-17T04:49:06.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-09-17T04:49:06.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.284+0000 7ffae9ffb640 1 -- 172.21.15.150:0/1543334455 <== mon.1 v2:172.21.15.176:3300/0 9 ==== config(3 keys) ==== 79+0+0 (secure 0 0 0) 0x7ffae40b7430 con 0x7ffae0003a10 2024-09-17T04:49:06.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.284+0000 7fe38cff9640 1 -- 172.21.15.150:0/1208350574 <== mon.0 v2:172.21.15.150:3300/0 9 ==== config(3 keys) ==== 79+0+0 (secure 0 0 0) 0x7fe3840049a0 con 0x7fe3900734d0 2024-09-17T04:49:06.285 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-09-17T04:49:06.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-09-17T04:49:06.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.288+0000 7f7514ff9640 1 -- 172.21.15.150:0/1726958560 <== mon.2 v2:172.21.15.176:3301/0 9 ==== config(3 keys) ==== 79+0+0 (secure 0 0 0) 0x7f7508011dd0 con 0x7f7518156720 2024-09-17T04:49:06.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.288+0000 7fc7877fe640 1 -- 172.21.15.150:0/3149123229 <== mon.2 v2:172.21.15.176:3301/0 10 ==== config(3 keys) ==== 79+0+0 (secure 0 0 0) 0x7fc7940067f0 con 0x7fc7a00ae920 2024-09-17T04:49:06.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.288+0000 7f1931ffb640 1 -- 172.21.15.150:0/2453242476 <== mon.2 v2:172.21.15.176:3301/0 9 ==== config(3 keys) ==== 79+0+0 (secure 0 0 0) 0x7f19380099f0 con 0x7f1928004810 2024-09-17T04:49:06.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.288+0000 7f539affd640 1 -- 172.21.15.150:0/2579171431 <== mon.2 v2:172.21.15.176:3301/0 9 ==== config(3 keys) ==== 79+0+0 (secure 0 0 0) 0x7f539c004bd0 con 0x7f53ac152860 2024-09-17T04:49:06.289 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-09-17T04:49:06.289 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-09-17T04:49:06.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.292+0000 7f1931ffb640 1 -- 172.21.15.150:0/2453242476 <== mon.2 v2:172.21.15.176:3301/0 10 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v4)=0 v4) ==== 123+0+0 (secure 0 0 0) 0x7f19380106b0 con 0x7f1928004810 2024-09-17T04:49:06.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.292+0000 7fc7a56a7640 1 -- 172.21.15.150:0/3149123229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7a00ae920 msgr2=0x7fc7a01f3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.292+0000 7fc7a56a7640 1 --2- 172.21.15.150:0/3149123229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7a00ae920 0x7fc7a01f3db0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc79400f8b0 tx=0x7fc79400fcb0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.294 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-09-17T04:49:06.294 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-09-17T04:49:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.296+0000 7f4e4dc21640 1 -- 172.21.15.150:0/4106023482 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e481643b0 con 0x7f4e480c31f0 2024-09-17T04:49:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.296+0000 7f4e4dc21640 1 -- 172.21.15.150:0/4106023482 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e480c6440 con 0x7f4e480c31f0 2024-09-17T04:49:06.296 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-09-17T04:49:06.296 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-09-17T04:49:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.296+0000 7f539affd640 1 -- 172.21.15.150:0/2579171431 <== mon.2 v2:172.21.15.176:3301/0 10 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v5)=0 v5) ==== 108+0+0 (secure 0 0 0) 0x7f53ac0ae9b0 con 0x7f53ac152860 2024-09-17T04:49:06.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.300+0000 7f4e2ffff640 1 -- 172.21.15.150:0/4106023482 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4e3c00e440 con 0x7f4e480c31f0 2024-09-17T04:49:06.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.300+0000 7f4e2ffff640 1 -- 172.21.15.150:0/4106023482 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4e3c017c60 con 0x7f4e480c31f0 2024-09-17T04:49:06.298 INFO:tasks.ceph:setting config [mgr] client_mount_timeout = 30 2024-09-17T04:49:06.298 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mgr client_mount_timeout 30 2024-09-17T04:49:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.304+0000 7fc7a56a7640 1 -- 172.21.15.150:0/3149123229 shutdown_connections 2024-09-17T04:49:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.304+0000 7fc7a56a7640 1 --2- 172.21.15.150:0/3149123229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7a01f42f0 0x7fc7a01e29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.304+0000 7fc7a56a7640 1 --2- 172.21.15.150:0/3149123229 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7a00ae920 0x7fc7a01f3db0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.304+0000 7fc7a56a7640 1 --2- 172.21.15.150:0/3149123229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc788003c40 0x7fc7a00adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.304+0000 7fc7a56a7640 1 -- 172.21.15.150:0/3149123229 >> 172.21.15.150:0/3149123229 conn(0x7fc7a006e360 msgr2=0x7fc7a0070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.302 INFO:tasks.ceph:setting config [mgr] debug_client = 20 2024-09-17T04:49:06.302 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mgr debug_client 20 2024-09-17T04:49:06.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.304+0000 7fc7a56a7640 1 -- 172.21.15.150:0/3149123229 shutdown_connections 2024-09-17T04:49:06.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.304+0000 7fc7a56a7640 1 -- 172.21.15.150:0/3149123229 wait complete. 2024-09-17T04:49:06.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.304+0000 7f4e4dc21640 1 -- 172.21.15.150:0/4106023482 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e280052a0 con 0x7f4e480c31f0 2024-09-17T04:49:06.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.312+0000 7f4e2ffff640 1 -- 172.21.15.150:0/4106023482 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f4e3c013030 con 0x7f4e480c31f0 2024-09-17T04:49:06.311 INFO:tasks.ceph:setting config [mgr] debug_mgr = 20 2024-09-17T04:49:06.311 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mgr debug_mgr 20 2024-09-17T04:49:06.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.316+0000 7f53b0d33640 1 -- 172.21.15.150:0/2579171431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53ac152860 msgr2=0x7f53ac1562b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.316+0000 7f53b0d33640 1 --2- 172.21.15.150:0/2579171431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53ac152860 0x7f53ac1562b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f539c00e7b0 tx=0x7f539c00ec70 comp rx=0 tx=0).stop 2024-09-17T04:49:06.314 INFO:tasks.ceph:setting config [mgr] debug_ms = 1 2024-09-17T04:49:06.314 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mgr debug_ms 1 2024-09-17T04:49:06.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f1943966640 1 -- 172.21.15.150:0/2453242476 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1928004810 msgr2=0x7f193c149c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f1943966640 1 --2- 172.21.15.150:0/2453242476 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1928004810 0x7f193c149c30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f193800d600 tx=0x7f193800dac0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.318 INFO:tasks.ceph:setting config [mgr] mon_warn_on_pool_no_app = False 2024-09-17T04:49:06.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mgr mon_warn_on_pool_no_app False 2024-09-17T04:49:06.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f53b0d33640 1 -- 172.21.15.150:0/2579171431 shutdown_connections 2024-09-17T04:49:06.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f53b0d33640 1 --2- 172.21.15.150:0/2579171431 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53ac1567f0 0x7f53ac155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f53b0d33640 1 --2- 172.21.15.150:0/2579171431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53ac152c70 0x7f53ac0ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f53b0d33640 1 --2- 172.21.15.150:0/2579171431 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53ac152860 0x7f53ac1562b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f53b0d33640 1 -- 172.21.15.150:0/2579171431 >> 172.21.15.150:0/2579171431 conn(0x7f53ac06e360 msgr2=0x7f53ac157e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f53b0d33640 1 -- 172.21.15.150:0/2579171431 shutdown_connections 2024-09-17T04:49:06.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f53b0d33640 1 -- 172.21.15.150:0/2579171431 wait complete. 2024-09-17T04:49:06.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7ffafba21640 1 -- 172.21.15.150:0/1543334455 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7ffacc005320 con 0x7ffae0003a10 2024-09-17T04:49:06.322 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-09-17T04:49:06.322 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-09-17T04:49:06.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f1943966640 1 -- 172.21.15.150:0/2453242476 shutdown_connections 2024-09-17T04:49:06.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f1943966640 1 --2- 172.21.15.150:0/2453242476 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f193c146340 0x7f193c154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f1943966640 1 --2- 172.21.15.150:0/2453242476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193c152b40 0x7f193c145e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.320+0000 7f1943966640 1 --2- 172.21.15.150:0/2453242476 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1928004810 0x7f193c149c30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.324+0000 7f1943966640 1 -- 172.21.15.150:0/2453242476 >> 172.21.15.150:0/2453242476 conn(0x7f193c0c5ae0 msgr2=0x7f193c151050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.324+0000 7f751e2e4640 1 -- 172.21.15.150:0/1726958560 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7f74f0005320 con 0x7f7518156720 2024-09-17T04:49:06.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.324+0000 7f1943966640 1 -- 172.21.15.150:0/2453242476 shutdown_connections 2024-09-17T04:49:06.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.324+0000 7f1943966640 1 -- 172.21.15.150:0/2453242476 wait complete. 2024-09-17T04:49:06.326 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-09-17T04:49:06.326 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-09-17T04:49:06.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.328+0000 7fe38cff9640 1 -- 172.21.15.150:0/1208350574 <== mon.0 v2:172.21.15.150:3300/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7fe384010750 con 0x7fe3900734d0 2024-09-17T04:49:06.330 DEBUG:tasks.ceph:set 26 configs 2024-09-17T04:49:06.330 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi150') 2024-09-17T04:49:06.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.332+0000 7ffae9ffb640 1 -- 172.21.15.150:0/1543334455 <== mon.1 v2:172.21.15.176:3300/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7ffae40b7730 con 0x7ffae0003a10 2024-09-17T04:49:06.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.332+0000 7ffae9ffb640 1 -- 172.21.15.150:0/1543334455 <== mon.1 v2:172.21.15.176:3300/0 11 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7ffae40b79d0 con 0x7ffae0003a10 2024-09-17T04:49:06.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.332+0000 7f7514ff9640 1 -- 172.21.15.150:0/1726958560 <== mon.2 v2:172.21.15.176:3301/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f75080036e0 con 0x7f7518156720 2024-09-17T04:49:06.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.332+0000 7f4e2ffff640 1 -- 172.21.15.150:0/4106023482 <== mon.2 v2:172.21.15.176:3301/0 7 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f4e3c00e820 con 0x7f4e480c31f0 2024-09-17T04:49:06.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.336+0000 7fe38cff9640 1 -- 172.21.15.150:0/1208350574 <== mon.0 v2:172.21.15.150:3300/0 11 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe384010b50 con 0x7fe3900734d0 2024-09-17T04:49:06.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.340+0000 7ffae9ffb640 1 -- 172.21.15.150:0/1543334455 <== mon.1 v2:172.21.15.176:3300/0 12 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffae40050a0 con 0x7ffae0003a10 2024-09-17T04:49:06.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.344+0000 7f7514ff9640 1 -- 172.21.15.150:0/1726958560 <== mon.2 v2:172.21.15.176:3301/0 11 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7508003a70 con 0x7f7518156720 2024-09-17T04:49:06.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.344+0000 7f4e2ffff640 1 -- 172.21.15.150:0/4106023482 <== mon.2 v2:172.21.15.176:3301/0 8 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e3c00ea30 con 0x7f4e480c31f0 2024-09-17T04:49:06.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.344+0000 7f7514ff9640 1 -- 172.21.15.150:0/1726958560 <== mon.2 v2:172.21.15.176:3301/0 12 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v7)=0 v7) ==== 123+0+0 (secure 0 0 0) 0x7f7508023430 con 0x7f7518156720 2024-09-17T04:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.352+0000 7ffafba21640 1 -- 172.21.15.150:0/1543334455 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffae0003a10 msgr2=0x7ffaf40aba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.352+0000 7ffafba21640 1 --2- 172.21.15.150:0/1543334455 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffae0003a10 0x7ffaf40aba60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ffae4098880 tx=0x7ffae0005840 comp rx=0 tx=0).stop 2024-09-17T04:49:06.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.352+0000 7ffafba21640 1 -- 172.21.15.150:0/1543334455 shutdown_connections 2024-09-17T04:49:06.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.352+0000 7ffafba21640 1 --2- 172.21.15.150:0/1543334455 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffaf41f91f0 0x7ffaf41f9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.352+0000 7ffafba21640 1 --2- 172.21.15.150:0/1543334455 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffaf414ce10 0x7ffaf414d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.352+0000 7ffafba21640 1 --2- 172.21.15.150:0/1543334455 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffae0003a10 0x7ffaf40aba60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.352+0000 7ffafba21640 1 -- 172.21.15.150:0/1543334455 >> 172.21.15.150:0/1543334455 conn(0x7ffaf40c5ae0 msgr2=0x7ffaf40c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.360+0000 7ffafba21640 1 -- 172.21.15.150:0/1543334455 shutdown_connections 2024-09-17T04:49:06.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.360+0000 7ffafba21640 1 -- 172.21.15.150:0/1543334455 wait complete. 2024-09-17T04:49:06.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.364+0000 7f751e2e4640 1 -- 172.21.15.150:0/1726958560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7518156720 msgr2=0x7f75180c0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.364+0000 7f751e2e4640 1 --2- 172.21.15.150:0/1726958560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7518156720 0x7f75180c0190 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f750800e930 tx=0x7f750800edf0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.364+0000 7f751e2e4640 1 -- 172.21.15.150:0/1726958560 shutdown_connections 2024-09-17T04:49:06.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.364+0000 7f751e2e4640 1 --2- 172.21.15.150:0/1726958560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75180c19e0 0x7f75180bf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.364+0000 7f751e2e4640 1 --2- 172.21.15.150:0/1726958560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75180c15d0 0x7f75180bef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.364+0000 7f751e2e4640 1 --2- 172.21.15.150:0/1726958560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7518156720 0x7f75180c0190 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.364+0000 7f751e2e4640 1 -- 172.21.15.150:0/1726958560 >> 172.21.15.150:0/1726958560 conn(0x7f751806e360 msgr2=0x7f751806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.364+0000 7f751e2e4640 1 -- 172.21.15.150:0/1726958560 shutdown_connections 2024-09-17T04:49:06.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.364+0000 7f751e2e4640 1 -- 172.21.15.150:0/1726958560 wait complete. 2024-09-17T04:49:06.443 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi150') 2024-09-17T04:49:06.443 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi150') 2024-09-17T04:49:06.444 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi150') 2024-09-17T04:49:06.446 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi150') 2024-09-17T04:49:06.446 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi150') 2024-09-17T04:49:06.446 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi150') 2024-09-17T04:49:06.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.492+0000 7fe3963fa640 1 -- 172.21.15.150:0/1208350574 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7fe370005300 con 0x7fe3900734d0 2024-09-17T04:49:06.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.500+0000 7fe38cff9640 1 -- 172.21.15.150:0/1208350574 <== mon.0 v2:172.21.15.150:3300/0 12 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v8)=0 v8) ==== 117+0+0 (secure 0 0 0) 0x7fe3901f1b00 con 0x7fe3900734d0 2024-09-17T04:49:06.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.516+0000 7fe3963fa640 1 -- 172.21.15.150:0/1208350574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3900734d0 msgr2=0x7fe390147a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.516+0000 7fe3963fa640 1 --2- 172.21.15.150:0/1208350574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3900734d0 0x7fe390147a90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe384005d10 tx=0x7fe38400dbb0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.516+0000 7fe3963fa640 1 -- 172.21.15.150:0/1208350574 shutdown_connections 2024-09-17T04:49:06.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.516+0000 7fe3963fa640 1 --2- 172.21.15.150:0/1208350574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe390074f70 0x7fe390147fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.516+0000 7fe3963fa640 1 --2- 172.21.15.150:0/1208350574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3900734d0 0x7fe390147a90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.516+0000 7fe3963fa640 1 --2- 172.21.15.150:0/1208350574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe39006daa0 0x7fe39014f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.516+0000 7fe3963fa640 1 -- 172.21.15.150:0/1208350574 >> 172.21.15.150:0/1208350574 conn(0x7fe3900c4e30 msgr2=0x7fe390070490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.516+0000 7fe3963fa640 1 -- 172.21.15.150:0/1208350574 shutdown_connections 2024-09-17T04:49:06.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.516+0000 7fe3963fa640 1 -- 172.21.15.150:0/1208350574 wait complete. 2024-09-17T04:49:06.595 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi150') 2024-09-17T04:49:06.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.616+0000 7fddee41e640 1 -- 172.21.15.150:0/3299722562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde80751a0 msgr2=0x7fdde80aac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.616+0000 7fddee41e640 1 --2- 172.21.15.150:0/3299722562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde80751a0 0x7fdde80aac90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009900 tx=0x7fdddc01c7f0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.616+0000 7fddee41e640 1 -- 172.21.15.150:0/3299722562 shutdown_connections 2024-09-17T04:49:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.616+0000 7fddee41e640 1 --2- 172.21.15.150:0/3299722562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde80751a0 0x7fdde80aac90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.616+0000 7fddee41e640 1 --2- 172.21.15.150:0/3299722562 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde80a72c0 0x7fdde8156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.616+0000 7fddee41e640 1 -- 172.21.15.150:0/3299722562 >> 172.21.15.150:0/3299722562 conn(0x7fdde80c7be0 msgr2=0x7fdde80c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.616+0000 7fddee41e640 1 -- 172.21.15.150:0/3299722562 shutdown_connections 2024-09-17T04:49:06.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.616+0000 7fddee41e640 1 -- 172.21.15.150:0/3299722562 wait complete. 2024-09-17T04:49:06.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7fddee41e640 1 Processor -- start 2024-09-17T04:49:06.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7fddee41e640 1 -- start start 2024-09-17T04:49:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7f7237fff640 1 -- 172.21.15.150:0/4213660000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7238070d30 msgr2=0x7f72380c4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7f7237fff640 1 --2- 172.21.15.150:0/4213660000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7238070d30 0x7f72380c4d50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7220009840 tx=0x7f722001c560 comp rx=0 tx=0).stop 2024-09-17T04:49:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7f7237fff640 1 -- 172.21.15.150:0/4213660000 shutdown_connections 2024-09-17T04:49:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7f7237fff640 1 --2- 172.21.15.150:0/4213660000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7238070d30 0x7f72380c4d50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7f7237fff640 1 -- 172.21.15.150:0/4213660000 >> 172.21.15.150:0/4213660000 conn(0x7f723806d250 msgr2=0x7f723806d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7f7237fff640 1 -- 172.21.15.150:0/4213660000 shutdown_connections 2024-09-17T04:49:06.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7fddee41e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde8074ce0 0x7fdde8153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7fddee41e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde80751a0 0x7fdde814c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7fddee41e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde80a72c0 0x7fdde814aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7fddee41e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdde81607e0 con 0x7fdde80751a0 2024-09-17T04:49:06.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fddee41e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdde8160660 con 0x7fdde80a72c0 2024-09-17T04:49:06.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fdde7fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde8074ce0 0x7fdde8153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fdde7fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde8074ce0 0x7fdde8153e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:36364/0 (socket says 172.21.15.150:36364) 2024-09-17T04:49:06.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fdde7fff640 1 -- 172.21.15.150:0/3259473869 learned_addr learned my addr 172.21.15.150:0/3259473869 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fddec9bb640 1 --2- 172.21.15.150:0/3259473869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde80a72c0 0x7fdde814aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fddee41e640 1 -- 172.21.15.150:0/3259473869 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdde8160960 con 0x7fdde8074ce0 2024-09-17T04:49:06.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fdde77fe640 1 --2- 172.21.15.150:0/3259473869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde80751a0 0x7fdde814c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fdde77fe640 1 -- 172.21.15.150:0/3259473869 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde8074ce0 msgr2=0x7fdde8153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fdde77fe640 1 --2- 172.21.15.150:0/3259473869 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde8074ce0 0x7fdde8153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fdde77fe640 1 -- 172.21.15.150:0/3259473869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde80a72c0 msgr2=0x7fdde814aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fdde77fe640 1 --2- 172.21.15.150:0/3259473869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde80a72c0 0x7fdde814aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fdde77fe640 1 -- 172.21.15.150:0/3259473869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde80a8370 con 0x7fdde80751a0 2024-09-17T04:49:06.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.620+0000 7f7237fff640 1 -- 172.21.15.150:0/4213660000 wait complete. 2024-09-17T04:49:06.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f7237fff640 1 Processor -- start 2024-09-17T04:49:06.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7fdde77fe640 1 --2- 172.21.15.150:0/3259473869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde80751a0 0x7fdde814c120 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c3c0 tx=0x7fddd800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f7237fff640 1 -- start start 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f7237fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7238070d30 0x7f72380c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f7237fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7238156c70 0x7f72380c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f7237fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7238157080 0x7f72381fcd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f7237fff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7238154260 con 0x7f7238156c70 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f7237fff640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7238071bf0 con 0x7f7238070d30 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f72367fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7238156c70 0x7f72380c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f7237fff640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7238070320 con 0x7f7238157080 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f72367fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7238156c70 0x7f72380c7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54536/0 (socket says 172.21.15.150:54536) 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f72367fc640 1 -- 172.21.15.150:0/3723138255 learned_addr learned my addr 172.21.15.150:0/3723138255 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f7236ffd640 1 --2- 172.21.15.150:0/3723138255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7238070d30 0x7f72380c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.624+0000 7f72377fe640 1 --2- 172.21.15.150:0/3723138255 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7238157080 0x7f72381fcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.628+0000 7f72367fc640 1 -- 172.21.15.150:0/3723138255 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7238157080 msgr2=0x7f72381fcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.628+0000 7f72367fc640 1 --2- 172.21.15.150:0/3723138255 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7238157080 0x7f72381fcd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.628+0000 7f72367fc640 1 -- 172.21.15.150:0/3723138255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7238070d30 msgr2=0x7f72380c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.628+0000 7f72367fc640 1 --2- 172.21.15.150:0/3723138255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7238070d30 0x7f72380c6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.628+0000 7f72367fc640 1 -- 172.21.15.150:0/3723138255 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f723806dba0 con 0x7f7238156c70 2024-09-17T04:49:06.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.628+0000 7f7236ffd640 1 --2- 172.21.15.150:0/3723138255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7238070d30 0x7f72380c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:06.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.628+0000 7f72367fc640 1 --2- 172.21.15.150:0/3723138255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7238156c70 0x7f72380c7f30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f722800fcb0 tx=0x7f722800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.644+0000 7f722f7fe640 1 -- 172.21.15.150:0/3723138255 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722801b070 con 0x7f7238156c70 2024-09-17T04:49:06.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.644+0000 7f7237fff640 1 -- 172.21.15.150:0/3723138255 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72381ed620 con 0x7f7238156c70 2024-09-17T04:49:06.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.644+0000 7f7237fff640 1 -- 172.21.15.150:0/3723138255 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f723814f850 con 0x7f7238156c70 2024-09-17T04:49:06.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.644+0000 7f722f7fe640 1 -- 172.21.15.150:0/3723138255 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7228006aa0 con 0x7f7238156c70 2024-09-17T04:49:06.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.644+0000 7f722f7fe640 1 -- 172.21.15.150:0/3723138255 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7228008070 con 0x7f7238156c70 2024-09-17T04:49:06.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.648+0000 7fdde4ff9640 1 -- 172.21.15.150:0/3259473869 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8017070 con 0x7fdde80751a0 2024-09-17T04:49:06.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.648+0000 7fdde4ff9640 1 -- 172.21.15.150:0/3259473869 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fddd80053c0 con 0x7fdde80751a0 2024-09-17T04:49:06.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.648+0000 7fddee41e640 1 -- 172.21.15.150:0/3259473869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde80a8610 con 0x7fdde80751a0 2024-09-17T04:49:06.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.648+0000 7fddee41e640 1 -- 172.21.15.150:0/3259473869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde80a8bd0 con 0x7fdde80751a0 2024-09-17T04:49:06.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.644+0000 7f722f7fe640 1 -- 172.21.15.150:0/3723138255 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f722801e070 con 0x7f7238156c70 2024-09-17T04:49:06.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.644+0000 7f722f7fe640 1 -- 172.21.15.150:0/3723138255 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7228020c60 con 0x7f7238156c70 2024-09-17T04:49:06.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.652+0000 7fdde4ff9640 1 -- 172.21.15.150:0/3259473869 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8003860 con 0x7fdde80751a0 2024-09-17T04:49:06.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.652+0000 7fdde4ff9640 1 -- 172.21.15.150:0/3259473869 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fddd801a070 con 0x7fdde80751a0 2024-09-17T04:49:06.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.652+0000 7fdde4ff9640 1 -- 172.21.15.150:0/3259473869 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fddd801c780 con 0x7fdde80751a0 2024-09-17T04:49:06.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.664+0000 7f7237fff640 1 -- 172.21.15.150:0/3723138255 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f721c0052a0 con 0x7f7238156c70 2024-09-17T04:49:06.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.668+0000 7f722f7fe640 1 -- 172.21.15.150:0/3723138255 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f7228025020 con 0x7f7238156c70 2024-09-17T04:49:06.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.672+0000 7fddee41e640 1 -- 172.21.15.150:0/3259473869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddc80052a0 con 0x7fdde80751a0 2024-09-17T04:49:06.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.676+0000 7fdde4ff9640 1 -- 172.21.15.150:0/3259473869 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fddd8021080 con 0x7fdde80751a0 2024-09-17T04:49:06.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.716+0000 7f4e4dc21640 1 -- 172.21.15.150:0/4106023482 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f4e28005590 con 0x7f4e480c31f0 2024-09-17T04:49:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.728+0000 7f4e2ffff640 1 -- 172.21.15.150:0/4106023482 <== mon.2 v2:172.21.15.176:3301/0 9 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v9)=0 v9) ==== 106+0+0 (secure 0 0 0) 0x7f4e3c00ec50 con 0x7f4e480c31f0 2024-09-17T04:49:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.748+0000 7f4e2dffb640 1 -- 172.21.15.150:0/4106023482 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e480c31f0 msgr2=0x7f4e480c5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.748+0000 7f4e2dffb640 1 --2- 172.21.15.150:0/4106023482 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e480c31f0 0x7f4e480c5f00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00b560 tx=0x7f4e3c00ba20 comp rx=0 tx=0).stop 2024-09-17T04:49:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.748+0000 7f4e2dffb640 1 -- 172.21.15.150:0/4106023482 shutdown_connections 2024-09-17T04:49:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.748+0000 7f4e2dffb640 1 --2- 172.21.15.150:0/4106023482 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e480c3890 0x7f4e481f4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.748+0000 7f4e2dffb640 1 --2- 172.21.15.150:0/4106023482 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e480c31f0 0x7f4e480c5f00 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.748+0000 7f4e2dffb640 1 --2- 172.21.15.150:0/4106023482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e480a7430 0x7f4e481636c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.748+0000 7f4e2dffb640 1 -- 172.21.15.150:0/4106023482 >> 172.21.15.150:0/4106023482 conn(0x7f4e480c7be0 msgr2=0x7f4e48068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.748+0000 7f4e2dffb640 1 -- 172.21.15.150:0/4106023482 shutdown_connections 2024-09-17T04:49:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.748+0000 7f4e2dffb640 1 -- 172.21.15.150:0/4106023482 wait complete. 2024-09-17T04:49:06.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.756+0000 7f3c2e119640 1 -- 172.21.15.150:0/675915254 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c280c40c0 msgr2=0x7f3c2806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.756+0000 7f3c2e119640 1 --2- 172.21.15.150:0/675915254 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c280c40c0 0x7f3c2806f840 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009870 tx=0x7f3c1c01c610 comp rx=0 tx=0).stop 2024-09-17T04:49:06.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.756+0000 7f3c2e119640 1 -- 172.21.15.150:0/675915254 shutdown_connections 2024-09-17T04:49:06.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.756+0000 7f3c2e119640 1 --2- 172.21.15.150:0/675915254 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c280c40c0 0x7f3c2806f840 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.756+0000 7f3c2e119640 1 -- 172.21.15.150:0/675915254 >> 172.21.15.150:0/675915254 conn(0x7f3c280c7be0 msgr2=0x7f3c280c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.760+0000 7f6f70bd3640 1 -- 172.21.15.150:0/4166361179 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f6c06f840 msgr2=0x7f6f6c155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.760+0000 7f6f70bd3640 1 --2- 172.21.15.150:0/4166361179 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f6c06f840 0x7f6f6c155df0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c0c3760 tx=0x7f6f6401c9e0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.764+0000 7f3c2e119640 1 -- 172.21.15.150:0/675915254 shutdown_connections 2024-09-17T04:49:06.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.764+0000 7f3c2e119640 1 -- 172.21.15.150:0/675915254 wait complete. 2024-09-17T04:49:06.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.764+0000 7f3c2e119640 1 Processor -- start 2024-09-17T04:49:06.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f6f70bd3640 1 -- 172.21.15.150:0/4166361179 shutdown_connections 2024-09-17T04:49:06.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f6f70bd3640 1 --2- 172.21.15.150:0/4166361179 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f6c06f840 0x7f6f6c155df0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f6f70bd3640 1 -- 172.21.15.150:0/4166361179 >> 172.21.15.150:0/4166361179 conn(0x7f6f6c0c7be0 msgr2=0x7f6f6c0c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f6f70bd3640 1 -- 172.21.15.150:0/4166361179 shutdown_connections 2024-09-17T04:49:06.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.764+0000 7f3c2e119640 1 -- start start 2024-09-17T04:49:06.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c2e119640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c280c2e80 0x7f3c280be370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c2e119640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c280c40c0 0x7f3c281ffec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c2e119640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c280c7700 0x7f3c28151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c2e119640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3c28157010 con 0x7f3c280c2e80 2024-09-17T04:49:06.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c26ffd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c280c40c0 0x7f3c281ffec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c277fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c280c2e80 0x7f3c280be370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c2e119640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3c28070dc0 con 0x7f3c280c40c0 2024-09-17T04:49:06.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c2e119640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3c28157190 con 0x7f3c280c7700 2024-09-17T04:49:06.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c277fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c280c2e80 0x7f3c280be370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54548/0 (socket says 172.21.15.150:54548) 2024-09-17T04:49:06.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c277fe640 1 -- 172.21.15.150:0/3263321310 learned_addr learned my addr 172.21.15.150:0/3263321310 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c27fff640 1 --2- 172.21.15.150:0/3263321310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c280c7700 0x7f3c28151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c277fe640 1 -- 172.21.15.150:0/3263321310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c280c7700 msgr2=0x7f3c28151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c277fe640 1 --2- 172.21.15.150:0/3263321310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c280c7700 0x7f3c28151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c277fe640 1 -- 172.21.15.150:0/3263321310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c280c40c0 msgr2=0x7f3c281ffec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c277fe640 1 --2- 172.21.15.150:0/3263321310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c280c40c0 0x7f3c281ffec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c277fe640 1 -- 172.21.15.150:0/3263321310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c280bcda0 con 0x7f3c280c2e80 2024-09-17T04:49:06.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c277fe640 1 --2- 172.21.15.150:0/3263321310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c280c2e80 0x7f3c280be370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3c1800c760 tx=0x7f3c1800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f6f70bd3640 1 -- 172.21.15.150:0/4166361179 wait complete. 2024-09-17T04:49:06.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f6f70bd3640 1 Processor -- start 2024-09-17T04:49:06.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c0ffff640 1 -- 172.21.15.150:0/3263321310 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c18005480 con 0x7f3c280c2e80 2024-09-17T04:49:06.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c2e119640 1 -- 172.21.15.150:0/3263321310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c282009e0 con 0x7f3c280c2e80 2024-09-17T04:49:06.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c2e119640 1 -- 172.21.15.150:0/3263321310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c280c1d50 con 0x7f3c280c2e80 2024-09-17T04:49:06.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c0ffff640 1 -- 172.21.15.150:0/3263321310 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c1800a640 con 0x7f3c280c2e80 2024-09-17T04:49:06.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f3c0ffff640 1 -- 172.21.15.150:0/3263321310 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1800aa90 con 0x7f3c280c2e80 2024-09-17T04:49:06.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.768+0000 7f6f70bd3640 1 -- start start 2024-09-17T04:49:06.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f3c0ffff640 1 -- 172.21.15.150:0/3263321310 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3c1800acf0 con 0x7f3c280c2e80 2024-09-17T04:49:06.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f3c0ffff640 1 -- 172.21.15.150:0/3263321310 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3c1801b920 con 0x7f3c280c2e80 2024-09-17T04:49:06.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f70bd3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f6c06f840 0x7f6f6c0aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f70bd3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f6c0c40c0 0x7f6f6c074570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f70bd3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f6c0c7700 0x7f6f6c151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f70bd3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c156b40 con 0x7f6f6c0c7700 2024-09-17T04:49:06.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f70bd3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6f6c06fce0 con 0x7f6f6c06f840 2024-09-17T04:49:06.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f70bd3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6f6c156cc0 con 0x7f6f6c0c40c0 2024-09-17T04:49:06.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f2ebef2a640 1 -- 172.21.15.150:0/3240288535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb80c40b0 msgr2=0x7f2eb8149cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f2ebef2a640 1 --2- 172.21.15.150:0/3240288535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb80c40b0 0x7f2eb8149cf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8072ab0 tx=0x7f2eb001d000 comp rx=0 tx=0).stop 2024-09-17T04:49:06.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f69d9b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f6c0c40c0 0x7f6f6c074570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f6a59c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f6c06f840 0x7f6f6c0aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f6ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f6c0c7700 0x7f6f6c151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f6ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f6c0c7700 0x7f6f6c151210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54552/0 (socket says 172.21.15.150:54552) 2024-09-17T04:49:06.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f6ad9d640 1 -- 172.21.15.150:0/1562326491 learned_addr learned my addr 172.21.15.150:0/1562326491 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f69d9b640 1 -- 172.21.15.150:0/1562326491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f6c06f840 msgr2=0x7f6f6c0aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f69d9b640 1 --2- 172.21.15.150:0/1562326491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f6c06f840 0x7f6f6c0aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f69d9b640 1 -- 172.21.15.150:0/1562326491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f6c0c7700 msgr2=0x7f6f6c151210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f69d9b640 1 --2- 172.21.15.150:0/1562326491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f6c0c7700 0x7f6f6c151210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f69d9b640 1 -- 172.21.15.150:0/1562326491 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f6c0aaf90 con 0x7f6f6c0c40c0 2024-09-17T04:49:06.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.772+0000 7f6f69d9b640 1 --2- 172.21.15.150:0/1562326491 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f6c0c40c0 0x7f6f6c074570 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000c3f0 tx=0x7f6f6000c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.780+0000 7f6f5affd640 1 -- 172.21.15.150:0/1562326491 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f600052d0 con 0x7f6f6c0c40c0 2024-09-17T04:49:06.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.780+0000 7f6f5affd640 1 -- 172.21.15.150:0/1562326491 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f60007020 con 0x7f6f6c0c40c0 2024-09-17T04:49:06.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.780+0000 7f6f5affd640 1 -- 172.21.15.150:0/1562326491 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60007470 con 0x7f6f6c0c40c0 2024-09-17T04:49:06.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.780+0000 7f6f70bd3640 1 -- 172.21.15.150:0/1562326491 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f6c1f3f40 con 0x7f6f6c0c40c0 2024-09-17T04:49:06.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.780+0000 7f6f70bd3640 1 -- 172.21.15.150:0/1562326491 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f6c0ac240 con 0x7f6f6c0c40c0 2024-09-17T04:49:06.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.780+0000 7f3c2e119640 1 -- 172.21.15.150:0/3263321310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c080052a0 con 0x7f3c280c2e80 2024-09-17T04:49:06.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.784+0000 7f3c0ffff640 1 -- 172.21.15.150:0/3263321310 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f3c18003600 con 0x7f3c280c2e80 2024-09-17T04:49:06.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.788+0000 7f6f5affd640 1 -- 172.21.15.150:0/1562326491 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6f600067e0 con 0x7f6f6c0c40c0 2024-09-17T04:49:06.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.788+0000 7f6f5affd640 1 -- 172.21.15.150:0/1562326491 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6f60003b40 con 0x7f6f6c0c40c0 2024-09-17T04:49:06.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.788+0000 7f2ebef2a640 1 -- 172.21.15.150:0/3240288535 shutdown_connections 2024-09-17T04:49:06.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.788+0000 7f2ebef2a640 1 --2- 172.21.15.150:0/3240288535 >> v2:172.21.15.176:3300/0 conn(0x7f2eb80ad660 0x7f2eb8071430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.788+0000 7f2ebef2a640 1 --2- 172.21.15.150:0/3240288535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb80c40b0 0x7f2eb8149cf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.788+0000 7f2ebef2a640 1 -- 172.21.15.150:0/3240288535 >> 172.21.15.150:0/3240288535 conn(0x7f2eb806d250 msgr2=0x7f2eb806d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.788+0000 7f2ebef2a640 1 -- 172.21.15.150:0/3240288535 shutdown_connections 2024-09-17T04:49:06.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.788+0000 7f2ebef2a640 1 -- 172.21.15.150:0/3240288535 wait complete. 2024-09-17T04:49:06.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.788+0000 7f2ebef2a640 1 Processor -- start 2024-09-17T04:49:06.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f6f70bd3640 1 -- 172.21.15.150:0/1562326491 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f500052a0 con 0x7f6f6c0c40c0 2024-09-17T04:49:06.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.788+0000 7f2ebef2a640 1 -- start start 2024-09-17T04:49:06.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebef2a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eb8071970 0x7f2eb81ffb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebef2a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb80ad660 0x7f2eb814ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebef2a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2eb80c40b0 0x7f2eb80a9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebef2a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb80c3210 con 0x7f2eb80ad660 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebef2a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb80c3090 con 0x7f2eb8071970 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebe729640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2eb80c40b0 0x7f2eb80a9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebe729640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2eb80c40b0 0x7f2eb80a9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:36424/0 (socket says 172.21.15.150:36424) 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebe729640 1 -- 172.21.15.150:0/2489937224 learned_addr learned my addr 172.21.15.150:0/2489937224 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebef2a640 1 -- 172.21.15.150:0/2489937224 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb8160470 con 0x7f2eb80c40b0 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebe729640 1 -- 172.21.15.150:0/2489937224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eb8071970 msgr2=0x7f2eb81ffb70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebe729640 1 --2- 172.21.15.150:0/2489937224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eb8071970 0x7f2eb81ffb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebe729640 1 -- 172.21.15.150:0/2489937224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb80ad660 msgr2=0x7f2eb814ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebe729640 1 --2- 172.21.15.150:0/2489937224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb80ad660 0x7f2eb814ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebe729640 1 -- 172.21.15.150:0/2489937224 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb814fe80 con 0x7f2eb80c40b0 2024-09-17T04:49:06.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.792+0000 7f2ebe729640 1 --2- 172.21.15.150:0/2489937224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2eb80c40b0 0x7f2eb80a9880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800b410 tx=0x7f2ea800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.800+0000 7f6f5affd640 1 -- 172.21.15.150:0/1562326491 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f6f60007610 con 0x7f6f6c0c40c0 2024-09-17T04:49:06.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.800+0000 7f2eae7fc640 1 -- 172.21.15.150:0/2489937224 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8015020 con 0x7f2eb80c40b0 2024-09-17T04:49:06.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.800+0000 7f2eae7fc640 1 -- 172.21.15.150:0/2489937224 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ea8011920 con 0x7f2eb80c40b0 2024-09-17T04:49:06.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.800+0000 7f2ebef2a640 1 -- 172.21.15.150:0/2489937224 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb8150160 con 0x7f2eb80c40b0 2024-09-17T04:49:06.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.800+0000 7f2ebef2a640 1 -- 172.21.15.150:0/2489937224 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb81fd7e0 con 0x7f2eb80c40b0 2024-09-17T04:49:06.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.804+0000 7f03e11c2640 1 -- 172.21.15.150:0/1510953376 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03c4004cf0 msgr2=0x7f03c4005140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.804+0000 7f03e11c2640 1 --2- 172.21.15.150:0/1510953376 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03c4004cf0 0x7f03c4005140 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f03d00018f0 tx=0x7f03d001e750 comp rx=0 tx=0).stop 2024-09-17T04:49:06.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.804+0000 7f03e11c2640 1 -- 172.21.15.150:0/1510953376 shutdown_connections 2024-09-17T04:49:06.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.804+0000 7f03e11c2640 1 --2- 172.21.15.150:0/1510953376 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03c4004cf0 0x7f03c4005140 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f03d00018f0 tx=0x7f03d001e750 comp rx=0 tx=0).stop 2024-09-17T04:49:06.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.804+0000 7f03e11c2640 1 -- 172.21.15.150:0/1510953376 >> 172.21.15.150:0/1510953376 conn(0x7f03dc0c7be0 msgr2=0x7f03dc0c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.804+0000 7f2eae7fc640 1 -- 172.21.15.150:0/2489937224 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8011d70 con 0x7f2eb80c40b0 2024-09-17T04:49:06.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.804+0000 7f03e11c2640 1 -- 172.21.15.150:0/1510953376 shutdown_connections 2024-09-17T04:49:06.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.804+0000 7f2eae7fc640 1 -- 172.21.15.150:0/2489937224 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2ea8018020 con 0x7f2eb80c40b0 2024-09-17T04:49:06.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.804+0000 7f2eae7fc640 1 -- 172.21.15.150:0/2489937224 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2ea8019930 con 0x7f2eb80c40b0 2024-09-17T04:49:06.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.808+0000 7f03e11c2640 1 -- 172.21.15.150:0/1510953376 wait complete. 2024-09-17T04:49:06.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.808+0000 7f03e11c2640 1 Processor -- start 2024-09-17T04:49:06.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.808+0000 7f03e11c2640 1 -- start start 2024-09-17T04:49:06.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03e11c2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03dc0bdda0 0x7f03dc0bedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03e11c2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03dc0be1d0 0x7f03dc1fd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03e11c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03dc156100 0x7f03dc1f9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03e11c2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc152a70 con 0x7f03dc156100 2024-09-17T04:49:06.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03e11c2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f03dc1528f0 con 0x7f03dc0be1d0 2024-09-17T04:49:06.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03e11c2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f03dc153030 con 0x7f03dc0bdda0 2024-09-17T04:49:06.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03dad9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03dc0bdda0 0x7f03dc0bedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03db59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03dc156100 0x7f03dc1f9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03db59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03dc156100 0x7f03dc1f9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54578/0 (socket says 172.21.15.150:54578) 2024-09-17T04:49:06.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03dad9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03dc0bdda0 0x7f03dc0bedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:36450/0 (socket says 172.21.15.150:36450) 2024-09-17T04:49:06.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03dad9d640 1 -- 172.21.15.150:0/4210187339 learned_addr learned my addr 172.21.15.150:0/4210187339 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03da59c640 1 --2- 172.21.15.150:0/4210187339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03dc0be1d0 0x7f03dc1fd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03dad9d640 1 -- 172.21.15.150:0/4210187339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03dc0be1d0 msgr2=0x7f03dc1fd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03dad9d640 1 --2- 172.21.15.150:0/4210187339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03dc0be1d0 0x7f03dc1fd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03dad9d640 1 -- 172.21.15.150:0/4210187339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03dc156100 msgr2=0x7f03dc1f9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03dad9d640 1 --2- 172.21.15.150:0/4210187339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03dc156100 0x7f03dc1f9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03dad9d640 1 -- 172.21.15.150:0/4210187339 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03dc0bf330 con 0x7f03dc0bdda0 2024-09-17T04:49:06.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f2ebef2a640 1 -- 172.21.15.150:0/2489937224 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea00052a0 con 0x7f2eb80c40b0 2024-09-17T04:49:06.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03db59e640 1 --2- 172.21.15.150:0/4210187339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03dc156100 0x7f03dc1f9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:06.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.812+0000 7f03dad9d640 1 --2- 172.21.15.150:0/4210187339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03dc0bdda0 0x7f03dc0bedf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f03d00ad040 tx=0x7f03d0003c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.820+0000 7f2eae7fc640 1 -- 172.21.15.150:0/2489937224 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f2ea801f020 con 0x7f2eb80c40b0 2024-09-17T04:49:06.829 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi150') 2024-09-17T04:49:06.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.828+0000 7f03c37fe640 1 -- 172.21.15.150:0/4210187339 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d00ba070 con 0x7f03dc0bdda0 2024-09-17T04:49:06.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.832+0000 7f03e11c2640 1 -- 172.21.15.150:0/4210187339 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03dc156a60 con 0x7f03dc0bdda0 2024-09-17T04:49:06.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.832+0000 7f03e11c2640 1 -- 172.21.15.150:0/4210187339 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f03dc201120 con 0x7f03dc0bdda0 2024-09-17T04:49:06.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.832+0000 7f03c37fe640 1 -- 172.21.15.150:0/4210187339 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03d0003e50 con 0x7f03dc0bdda0 2024-09-17T04:49:06.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.832+0000 7f03c37fe640 1 -- 172.21.15.150:0/4210187339 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d00af040 con 0x7f03dc0bdda0 2024-09-17T04:49:06.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.836+0000 7f03c17fa640 1 -- 172.21.15.150:0/4210187339 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03dc153540 con 0x7f03dc0bdda0 2024-09-17T04:49:06.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.840+0000 7f03c37fe640 1 -- 172.21.15.150:0/4210187339 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f03d00b5470 con 0x7f03dc0bdda0 2024-09-17T04:49:06.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.840+0000 7f03c37fe640 1 -- 172.21.15.150:0/4210187339 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f03d00aa540 con 0x7f03dc0bdda0 2024-09-17T04:49:06.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.844+0000 7f03c37fe640 1 -- 172.21.15.150:0/4210187339 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f03d00c3070 con 0x7f03dc0bdda0 2024-09-17T04:49:06.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.848+0000 7f1112a61640 1 -- 172.21.15.150:0/1147723485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110c0c40c0 msgr2=0x7f110c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.848+0000 7f1112a61640 1 --2- 172.21.15.150:0/1147723485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110c0c40c0 0x7f110c06f840 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f10fc009840 tx=0x7f10fc02d7e0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.852+0000 7f1112a61640 1 -- 172.21.15.150:0/1147723485 shutdown_connections 2024-09-17T04:49:06.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.852+0000 7f1112a61640 1 --2- 172.21.15.150:0/1147723485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110c0c40c0 0x7f110c06f840 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.852+0000 7f1112a61640 1 -- 172.21.15.150:0/1147723485 >> 172.21.15.150:0/1147723485 conn(0x7f110c0c7be0 msgr2=0x7f110c0c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.852+0000 7f1112a61640 1 -- 172.21.15.150:0/1147723485 shutdown_connections 2024-09-17T04:49:06.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.852+0000 7f1112a61640 1 -- 172.21.15.150:0/1147723485 wait complete. 2024-09-17T04:49:06.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.856+0000 7f1112a61640 1 Processor -- start 2024-09-17T04:49:06.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.856+0000 7f1112a61640 1 -- start start 2024-09-17T04:49:06.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f1112a61640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f110c0c2e80 0x7f110c149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f1112a61640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f110c0c40c0 0x7f110c158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f110bfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f110c0c2e80 0x7f110c149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f110bfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f110c0c2e80 0x7f110c149a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:33192/0 (socket says 172.21.15.150:33192) 2024-09-17T04:49:06.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f110b7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f110c0c40c0 0x7f110c158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f110b7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f110c0c40c0 0x7f110c158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:36460/0 (socket says 172.21.15.150:36460) 2024-09-17T04:49:06.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f110bfff640 1 -- 172.21.15.150:0/3022401607 learned_addr learned my addr 172.21.15.150:0/3022401607 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f1112a61640 1 --2- 172.21.15.150:0/3022401607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110c0c7700 0x7f110c1eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f1112a61640 1 -- 172.21.15.150:0/3022401607 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f110c156f80 con 0x7f110c0c7700 2024-09-17T04:49:06.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f1112a61640 1 -- 172.21.15.150:0/3022401607 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f110c156e00 con 0x7f110c0c2e80 2024-09-17T04:49:06.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f1110ffe640 1 --2- 172.21.15.150:0/3022401607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110c0c7700 0x7f110c1eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f2a8128b640 1 -- 172.21.15.150:0/526634519 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c06f840 msgr2=0x7f2a7c155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f2a8128b640 1 --2- 172.21.15.150:0/526634519 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c06f840 0x7f2a7c155df0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c009840 tx=0x7f2a6c01c640 comp rx=0 tx=0).stop 2024-09-17T04:49:06.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f1112a61640 1 -- 172.21.15.150:0/3022401607 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f110c157100 con 0x7f110c0c40c0 2024-09-17T04:49:06.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f110bfff640 1 -- 172.21.15.150:0/3022401607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f110c0c40c0 msgr2=0x7f110c158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f110bfff640 1 --2- 172.21.15.150:0/3022401607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f110c0c40c0 0x7f110c158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f110bfff640 1 -- 172.21.15.150:0/3022401607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110c0c7700 msgr2=0x7f110c1eeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f110bfff640 1 --2- 172.21.15.150:0/3022401607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110c0c7700 0x7f110c1eeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f110bfff640 1 -- 172.21.15.150:0/3022401607 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110c1585e0 con 0x7f110c0c2e80 2024-09-17T04:49:06.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f110bfff640 1 --2- 172.21.15.150:0/3022401607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f110c0c2e80 0x7f110c149a80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f110400cbc0 tx=0x7f110400ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f2a8128b640 1 -- 172.21.15.150:0/526634519 shutdown_connections 2024-09-17T04:49:06.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f2a8128b640 1 --2- 172.21.15.150:0/526634519 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c06f840 0x7f2a7c155df0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f2a8128b640 1 -- 172.21.15.150:0/526634519 >> 172.21.15.150:0/526634519 conn(0x7f2a7c0c7be0 msgr2=0x7f2a7c0c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f2a8128b640 1 -- 172.21.15.150:0/526634519 shutdown_connections 2024-09-17T04:49:06.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.864+0000 7f532dc21640 1 -- 172.21.15.150:0/1858572059 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53280ad310 msgr2=0x7f532814bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.864+0000 7f532dc21640 1 --2- 172.21.15.150:0/1858572059 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53280ad310 0x7f532814bb10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5320009bc0 tx=0x7f532001cbe0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.864+0000 7f532dc21640 1 -- 172.21.15.150:0/1858572059 shutdown_connections 2024-09-17T04:49:06.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.864+0000 7f532dc21640 1 --2- 172.21.15.150:0/1858572059 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f53280ad310 0x7f532814bb10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.864+0000 7f532dc21640 1 --2- 172.21.15.150:0/1858572059 >> v2:172.21.15.176:3300/0 conn(0x7f5328071740 0x7f53280c3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.864+0000 7f532dc21640 1 -- 172.21.15.150:0/1858572059 >> 172.21.15.150:0/1858572059 conn(0x7f532806d250 msgr2=0x7f532806d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.864+0000 7f532dc21640 1 -- 172.21.15.150:0/1858572059 shutdown_connections 2024-09-17T04:49:06.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.864+0000 7f532dc21640 1 -- 172.21.15.150:0/1858572059 wait complete. 2024-09-17T04:49:06.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.864+0000 7f532dc21640 1 Processor -- start 2024-09-17T04:49:06.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.860+0000 7f2a8128b640 1 -- 172.21.15.150:0/526634519 wait complete. 2024-09-17T04:49:06.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.864+0000 7f2a8128b640 1 Processor -- start 2024-09-17T04:49:06.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.864+0000 7f2a8128b640 1 -- start start 2024-09-17T04:49:06.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a8128b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2a7c06f840 0x7f2a7c158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a8128b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c0c40c0 0x7f2a7c165380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a8128b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a7c0c7700 0x7f2a7c14bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a7a7fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c0c40c0 0x7f2a7c165380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a7a7fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c0c40c0 0x7f2a7c165380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:33194/0 (socket says 172.21.15.150:33194) 2024-09-17T04:49:06.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a7a7fc640 1 -- 172.21.15.150:0/4073369974 learned_addr learned my addr 172.21.15.150:0/4073369974 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a7b7fe640 1 --2- 172.21.15.150:0/4073369974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a7c0c7700 0x7f2a7c14bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a8128b640 1 -- 172.21.15.150:0/4073369974 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c156b40 con 0x7f2a7c0c7700 2024-09-17T04:49:06.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a8128b640 1 -- 172.21.15.150:0/4073369974 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2a7c070cc0 con 0x7f2a7c0c40c0 2024-09-17T04:49:06.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a8128b640 1 -- 172.21.15.150:0/4073369974 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2a7c156cc0 con 0x7f2a7c06f840 2024-09-17T04:49:06.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.864+0000 7f532dc21640 1 -- start start 2024-09-17T04:49:06.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532dc21640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5328070d60 0x7f53280a9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7fcbb329a640 1 -- 172.21.15.150:0/322991288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbac0c2e80 msgr2=0x7fcbac152b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7fcbb329a640 1 --2- 172.21.15.150:0/322991288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbac0c2e80 0x7fcbac152b10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009840 tx=0x7fcba801c640 comp rx=0 tx=0).stop 2024-09-17T04:49:06.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532dc21640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5328071740 0x7f5328165b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a7affd640 1 --2- 172.21.15.150:0/4073369974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2a7c06f840 0x7f2a7c158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532dc21640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53280ad310 0x7f5328146af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532cc1f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5328070d60 0x7f53280a9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532cc1f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5328070d60 0x7f53280a9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54608/0 (socket says 172.21.15.150:54608) 2024-09-17T04:49:06.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532cc1f640 1 -- 172.21.15.150:0/1900322876 learned_addr learned my addr 172.21.15.150:0/1900322876 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532dc21640 1 -- 172.21.15.150:0/1900322876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f53280ad8a0 con 0x7f5328070d60 2024-09-17T04:49:06.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a7a7fc640 1 -- 172.21.15.150:0/4073369974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2a7c06f840 msgr2=0x7f2a7c158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a7a7fc640 1 --2- 172.21.15.150:0/4073369974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2a7c06f840 0x7f2a7c158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a7a7fc640 1 -- 172.21.15.150:0/4073369974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a7c0c7700 msgr2=0x7f2a7c14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a7a7fc640 1 --2- 172.21.15.150:0/4073369974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a7c0c7700 0x7f2a7c14bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a7a7fc640 1 -- 172.21.15.150:0/4073369974 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7c158000 con 0x7f2a7c0c40c0 2024-09-17T04:49:06.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a7b7fe640 1 --2- 172.21.15.150:0/4073369974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a7c0c7700 0x7f2a7c14bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:06.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532dc21640 1 -- 172.21.15.150:0/1900322876 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f53280ad720 con 0x7f53280ad310 2024-09-17T04:49:06.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532dc21640 1 -- 172.21.15.150:0/1900322876 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f532814edf0 con 0x7f5328071740 2024-09-17T04:49:06.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f2a7a7fc640 1 --2- 172.21.15.150:0/4073369974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c0c40c0 0x7f2a7c165380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2a7400cb00 tx=0x7f2a74002120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532cc1f640 1 -- 172.21.15.150:0/1900322876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5328071740 msgr2=0x7f5328165b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532cc1f640 1 --2- 172.21.15.150:0/1900322876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5328071740 0x7f5328165b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532cc1f640 1 -- 172.21.15.150:0/1900322876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53280ad310 msgr2=0x7f5328146af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532cc1f640 1 --2- 172.21.15.150:0/1900322876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53280ad310 0x7f5328146af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532cc1f640 1 -- 172.21.15.150:0/1900322876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f532815aa10 con 0x7f5328070d60 2024-09-17T04:49:06.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.868+0000 7f532cc1f640 1 --2- 172.21.15.150:0/1900322876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5328070d60 0x7f53280a9ad0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f532400e770 tx=0x7f532400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.872+0000 7fddee41e640 1 -- 172.21.15.150:0/3259473869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7fddc8005590 con 0x7fdde80751a0 2024-09-17T04:49:06.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.872+0000 7fcbb329a640 1 -- 172.21.15.150:0/322991288 shutdown_connections 2024-09-17T04:49:06.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.872+0000 7fcbb329a640 1 --2- 172.21.15.150:0/322991288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbac0c2e80 0x7fcbac152b10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.872+0000 7fcbb329a640 1 --2- 172.21.15.150:0/322991288 >> v2:172.21.15.176:3300/0 conn(0x7fcbac0c40c0 0x7fcbac06f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.872+0000 7fcbb329a640 1 -- 172.21.15.150:0/322991288 >> 172.21.15.150:0/322991288 conn(0x7fcbac0c7be0 msgr2=0x7fcbac0c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.872+0000 7fcbb329a640 1 -- 172.21.15.150:0/322991288 shutdown_connections 2024-09-17T04:49:06.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.872+0000 7fcbb329a640 1 -- 172.21.15.150:0/322991288 wait complete. 2024-09-17T04:49:06.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f531d7fa640 1 -- 172.21.15.150:0/1900322876 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532400ccc0 con 0x7f5328070d60 2024-09-17T04:49:06.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f2a6b7fe640 1 -- 172.21.15.150:0/4073369974 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74017070 con 0x7f2a7c0c40c0 2024-09-17T04:49:06.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f531d7fa640 1 -- 172.21.15.150:0/1900322876 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f532400ee80 con 0x7f5328070d60 2024-09-17T04:49:06.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f532dc21640 1 -- 172.21.15.150:0/1900322876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53280a8fc0 con 0x7f5328070d60 2024-09-17T04:49:06.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f531d7fa640 1 -- 172.21.15.150:0/1900322876 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532400ce60 con 0x7f5328070d60 2024-09-17T04:49:06.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f532dc21640 1 -- 172.21.15.150:0/1900322876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f532814aaa0 con 0x7f5328070d60 2024-09-17T04:49:06.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f531d7fa640 1 -- 172.21.15.150:0/1900322876 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f53240047f0 con 0x7f5328070d60 2024-09-17T04:49:06.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f531d7fa640 1 -- 172.21.15.150:0/1900322876 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5324003a60 con 0x7f5328070d60 2024-09-17T04:49:06.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7fcbb329a640 1 Processor -- start 2024-09-17T04:49:06.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f2a8128b640 1 -- 172.21.15.150:0/4073369974 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a7c0c6470 con 0x7f2a7c0c40c0 2024-09-17T04:49:06.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f2a8128b640 1 -- 172.21.15.150:0/4073369974 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a7c166630 con 0x7f2a7c0c40c0 2024-09-17T04:49:06.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f2a6b7fe640 1 -- 172.21.15.150:0/4073369974 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a7400a760 con 0x7f2a7c0c40c0 2024-09-17T04:49:06.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f2a6b7fe640 1 -- 172.21.15.150:0/4073369974 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7400ab70 con 0x7f2a7c0c40c0 2024-09-17T04:49:06.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7fcbb329a640 1 -- start start 2024-09-17T04:49:06.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7fcbb329a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbac0c2e80 0x7fcbac1664d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7fcbb329a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbac0c40c0 0x7fcbac151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7fcbb329a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbac0c7700 0x7fcbac158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f2a6b7fe640 1 -- 172.21.15.150:0/4073369974 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2a7401a070 con 0x7f2a7c0c40c0 2024-09-17T04:49:06.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.876+0000 7f2a6b7fe640 1 -- 172.21.15.150:0/4073369974 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2a7401c8c0 con 0x7f2a7c0c40c0 2024-09-17T04:49:06.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb329a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac155f70 con 0x7fcbac0c2e80 2024-09-17T04:49:06.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb329a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcbac155df0 con 0x7fcbac0c7700 2024-09-17T04:49:06.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb0835640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbac0c40c0 0x7fcbac151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb0835640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbac0c40c0 0x7fcbac151630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:36486/0 (socket says 172.21.15.150:36486) 2024-09-17T04:49:06.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb0835640 1 -- 172.21.15.150:0/622736408 learned_addr learned my addr 172.21.15.150:0/622736408 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb1837640 1 --2- 172.21.15.150:0/622736408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbac0c7700 0x7fcbac158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb329a640 1 -- 172.21.15.150:0/622736408 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcbac1560f0 con 0x7fcbac0c40c0 2024-09-17T04:49:06.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fdde4ff9640 1 -- 172.21.15.150:0/3259473869 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v10)=0 v10) ==== 127+0+0 (secure 0 0 0) 0x7fddd8011db0 con 0x7fdde80751a0 2024-09-17T04:49:06.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb1036640 1 --2- 172.21.15.150:0/622736408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbac0c2e80 0x7fcbac1664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb1837640 1 -- 172.21.15.150:0/622736408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbac0c40c0 msgr2=0x7fcbac151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb1837640 1 --2- 172.21.15.150:0/622736408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbac0c40c0 0x7fcbac151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb1837640 1 -- 172.21.15.150:0/622736408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbac0c2e80 msgr2=0x7fcbac1664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb1837640 1 --2- 172.21.15.150:0/622736408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbac0c2e80 0x7fcbac1664d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb1837640 1 -- 172.21.15.150:0/622736408 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbac14d230 con 0x7fcbac0c7700 2024-09-17T04:49:06.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7fcbb1837640 1 --2- 172.21.15.150:0/622736408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbac0c7700 0x7fcbac158a90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcbac071380 tx=0x7fcba80a9460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7f1108ff9640 1 -- 172.21.15.150:0/3022401607 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104016020 con 0x7f110c0c2e80 2024-09-17T04:49:06.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7f1108ff9640 1 -- 172.21.15.150:0/3022401607 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f110400aae0 con 0x7f110c0c2e80 2024-09-17T04:49:06.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7f1112a61640 1 -- 172.21.15.150:0/3022401607 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f110c0ad0b0 con 0x7f110c0c2e80 2024-09-17T04:49:06.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.880+0000 7f1112a61640 1 -- 172.21.15.150:0/3022401607 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f110c1efb90 con 0x7f110c0c2e80 2024-09-17T04:49:06.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.884+0000 7f1108ff9640 1 -- 172.21.15.150:0/3022401607 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104012420 con 0x7f110c0c2e80 2024-09-17T04:49:06.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.884+0000 7f1112a61640 1 -- 172.21.15.150:0/3022401607 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10ec0052a0 con 0x7f110c0c2e80 2024-09-17T04:49:06.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.888+0000 7f532dc21640 1 -- 172.21.15.150:0/1900322876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53080052a0 con 0x7f5328070d60 2024-09-17T04:49:06.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.888+0000 7f1108ff9640 1 -- 172.21.15.150:0/3022401607 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1104019020 con 0x7f110c0c2e80 2024-09-17T04:49:06.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.888+0000 7f1108ff9640 1 -- 172.21.15.150:0/3022401607 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f110401ba40 con 0x7f110c0c2e80 2024-09-17T04:49:06.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.888+0000 7fddce7fc640 1 -- 172.21.15.150:0/3259473869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde80751a0 msgr2=0x7fdde814c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.888+0000 7fddce7fc640 1 --2- 172.21.15.150:0/3259473869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde80751a0 0x7fdde814c120 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c3c0 tx=0x7fddd800c880 comp rx=0 tx=0).stop 2024-09-17T04:49:06.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.888+0000 7f2a8128b640 1 -- 172.21.15.150:0/4073369974 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a580052a0 con 0x7f2a7c0c40c0 2024-09-17T04:49:06.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.892+0000 7f531d7fa640 1 -- 172.21.15.150:0/1900322876 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f532815aa10 con 0x7f5328070d60 2024-09-17T04:49:06.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.892+0000 7f1108ff9640 1 -- 172.21.15.150:0/3022401607 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f1104020020 con 0x7f110c0c2e80 2024-09-17T04:49:06.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7fddce7fc640 1 -- 172.21.15.150:0/3259473869 shutdown_connections 2024-09-17T04:49:06.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7fddce7fc640 1 --2- 172.21.15.150:0/3259473869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde80a72c0 0x7fdde814aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7fddce7fc640 1 --2- 172.21.15.150:0/3259473869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde80751a0 0x7fdde814c120 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7fddce7fc640 1 --2- 172.21.15.150:0/3259473869 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde8074ce0 0x7fdde8153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7fddce7fc640 1 -- 172.21.15.150:0/3259473869 >> 172.21.15.150:0/3259473869 conn(0x7fdde80c7be0 msgr2=0x7fdde8155540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7fcba1ffb640 1 -- 172.21.15.150:0/622736408 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba80ad050 con 0x7fcbac0c7700 2024-09-17T04:49:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7fcba1ffb640 1 -- 172.21.15.150:0/622736408 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcba80043e0 con 0x7fcbac0c7700 2024-09-17T04:49:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7fcba1ffb640 1 -- 172.21.15.150:0/622736408 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba8004930 con 0x7fcbac0c7700 2024-09-17T04:49:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7f2a6b7fe640 1 -- 172.21.15.150:0/4073369974 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f2a74021020 con 0x7f2a7c0c40c0 2024-09-17T04:49:06.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7fcbb329a640 1 -- 172.21.15.150:0/622736408 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbac0749c0 con 0x7fcbac0c7700 2024-09-17T04:49:06.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7fcbb329a640 1 -- 172.21.15.150:0/622736408 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbac14b000 con 0x7fcbac0c7700 2024-09-17T04:49:06.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7fddce7fc640 1 -- 172.21.15.150:0/3259473869 shutdown_connections 2024-09-17T04:49:06.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.896+0000 7fddce7fc640 1 -- 172.21.15.150:0/3259473869 wait complete. 2024-09-17T04:49:06.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.900+0000 7fcba1ffb640 1 -- 172.21.15.150:0/622736408 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcba80b0070 con 0x7fcbac0c7700 2024-09-17T04:49:06.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.900+0000 7fcba1ffb640 1 -- 172.21.15.150:0/622736408 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcba8003940 con 0x7fcbac0c7700 2024-09-17T04:49:06.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.900+0000 7fe72d41a640 1 -- 172.21.15.150:0/1912824478 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7281609e0 msgr2=0x7fe728070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.900+0000 7fe72d41a640 1 --2- 172.21.15.150:0/1912824478 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7281609e0 0x7fe728070800 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe72000b2e0 tx=0x7fe72001dc50 comp rx=0 tx=0).stop 2024-09-17T04:49:06.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.908+0000 7fcbb329a640 1 -- 172.21.15.150:0/622736408 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbac152b10 con 0x7fcbac0c7700 2024-09-17T04:49:06.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.912+0000 7fcba1ffb640 1 -- 172.21.15.150:0/622736408 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fcba80c1060 con 0x7fcbac0c7700 2024-09-17T04:49:06.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.924+0000 7fe72d41a640 1 -- 172.21.15.150:0/1912824478 shutdown_connections 2024-09-17T04:49:06.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.924+0000 7fe72d41a640 1 --2- 172.21.15.150:0/1912824478 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe7281609e0 0x7fe728070800 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.924+0000 7fe72d41a640 1 -- 172.21.15.150:0/1912824478 >> 172.21.15.150:0/1912824478 conn(0x7fe72806ce50 msgr2=0x7fe72806d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.924+0000 7fe72d41a640 1 -- 172.21.15.150:0/1912824478 shutdown_connections 2024-09-17T04:49:06.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.924+0000 7fe72d41a640 1 -- 172.21.15.150:0/1912824478 wait complete. 2024-09-17T04:49:06.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe72d41a640 1 Processor -- start 2024-09-17T04:49:06.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe72d41a640 1 -- start start 2024-09-17T04:49:06.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe72d41a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72808b880 0x7fe728152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe72d41a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72808bdb0 0x7fe728153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe60b4a8640 1 -- 172.21.15.150:0/1922022623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6041535c0 msgr2=0x7fe6040c2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe60b4a8640 1 --2- 172.21.15.150:0/1922022623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6041535c0 0x7fe6040c2f80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009c80 tx=0x7fe5fc01cd00 comp rx=0 tx=0).stop 2024-09-17T04:49:06.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe727fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72808b880 0x7fe728152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe727fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72808b880 0x7fe728152eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:36532/0 (socket says 172.21.15.150:36532) 2024-09-17T04:49:06.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe7277fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72808bdb0 0x7fe728153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe7277fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72808bdb0 0x7fe728153780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54638/0 (socket says 172.21.15.150:54638) 2024-09-17T04:49:06.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe60b4a8640 1 -- 172.21.15.150:0/1922022623 shutdown_connections 2024-09-17T04:49:06.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe60b4a8640 1 --2- 172.21.15.150:0/1922022623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6041535c0 0x7fe6040c2f80 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009c80 tx=0x7fe5fc01cd00 comp rx=0 tx=0).stop 2024-09-17T04:49:06.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe60b4a8640 1 --2- 172.21.15.150:0/1922022623 >> v2:172.21.15.150:3300/0 conn(0x7fe604073150 0x7fe604149600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe60b4a8640 1 -- 172.21.15.150:0/1922022623 >> 172.21.15.150:0/1922022623 conn(0x7fe60406ce50 msgr2=0x7fe60406d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe727fff640 1 -- 172.21.15.150:0/908947671 learned_addr learned my addr 172.21.15.150:0/908947671 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe72d41a640 1 --2- 172.21.15.150:0/908947671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7281609e0 0x7fe72815b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe72d41a640 1 -- 172.21.15.150:0/908947671 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe728070d40 con 0x7fe72808bdb0 2024-09-17T04:49:06.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe72d41a640 1 -- 172.21.15.150:0/908947671 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe728163e80 con 0x7fe7281609e0 2024-09-17T04:49:06.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe72d41a640 1 -- 172.21.15.150:0/908947671 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe728160df0 con 0x7fe72808b880 2024-09-17T04:49:06.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.928+0000 7fe72cc19640 1 --2- 172.21.15.150:0/908947671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7281609e0 0x7fe72815b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.932+0000 7fe7277fe640 1 -- 172.21.15.150:0/908947671 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72808b880 msgr2=0x7fe728152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.932+0000 7fe7277fe640 1 --2- 172.21.15.150:0/908947671 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72808b880 0x7fe728152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.932+0000 7fe7277fe640 1 -- 172.21.15.150:0/908947671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7281609e0 msgr2=0x7fe72815b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.932+0000 7fe7277fe640 1 --2- 172.21.15.150:0/908947671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7281609e0 0x7fe72815b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.932+0000 7fe7277fe640 1 -- 172.21.15.150:0/908947671 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72815af90 con 0x7fe72808bdb0 2024-09-17T04:49:06.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.932+0000 7fe7277fe640 1 --2- 172.21.15.150:0/908947671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72808bdb0 0x7fe728153780 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe71c00c3c0 tx=0x7fe71c00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.936+0000 7fe60b4a8640 1 -- 172.21.15.150:0/1922022623 shutdown_connections 2024-09-17T04:49:06.937 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi150') 2024-09-17T04:49:06.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.936+0000 7fe60b4a8640 1 -- 172.21.15.150:0/1922022623 wait complete. 2024-09-17T04:49:06.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.940+0000 7fe60b4a8640 1 Processor -- start 2024-09-17T04:49:06.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.940+0000 7fe60b4a8640 1 -- start start 2024-09-17T04:49:06.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe60b4a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe604073150 0x7fe60414ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe60b4a8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6040c34c0 0x7fe604200480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe60b4a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6042009c0 0x7fe6040c0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe60a4a6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe604073150 0x7fe60414ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe60a4a6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe604073150 0x7fe60414ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:33246/0 (socket says 172.21.15.150:33246) 2024-09-17T04:49:06.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe60a4a6640 1 -- 172.21.15.150:0/4260560417 learned_addr learned my addr 172.21.15.150:0/4260560417 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe609ca5640 1 --2- 172.21.15.150:0/4260560417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6040c34c0 0x7fe604200480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe60aca7640 1 --2- 172.21.15.150:0/4260560417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6042009c0 0x7fe6040c0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe60b4a8640 1 -- 172.21.15.150:0/4260560417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe60415fce0 con 0x7fe6040c34c0 2024-09-17T04:49:06.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe60b4a8640 1 -- 172.21.15.150:0/4260560417 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe6040c3d40 con 0x7fe604073150 2024-09-17T04:49:06.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7f0f03026640 1 -- 172.21.15.150:0/1033285454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0efc156c50 msgr2=0x7f0efc1659d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7f0f03026640 1 --2- 172.21.15.150:0/1033285454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0efc156c50 0x7f0efc1659d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009c80 tx=0x7f0ef401cb60 comp rx=0 tx=0).stop 2024-09-17T04:49:06.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe60b4a8640 1 -- 172.21.15.150:0/4260560417 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe60415fe60 con 0x7fe6042009c0 2024-09-17T04:49:06.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7f0f03026640 1 -- 172.21.15.150:0/1033285454 shutdown_connections 2024-09-17T04:49:06.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7f0f03026640 1 --2- 172.21.15.150:0/1033285454 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0efc156c50 0x7f0efc1659d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7f0f03026640 1 --2- 172.21.15.150:0/1033285454 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0efc074f70 0x7f0efc0734d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7f0f03026640 1 -- 172.21.15.150:0/1033285454 >> 172.21.15.150:0/1033285454 conn(0x7f0efc0c4e30 msgr2=0x7f0efc06ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7f0f03026640 1 -- 172.21.15.150:0/1033285454 shutdown_connections 2024-09-17T04:49:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe609ca5640 1 -- 172.21.15.150:0/4260560417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6042009c0 msgr2=0x7fe6040c0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe609ca5640 1 --2- 172.21.15.150:0/4260560417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6042009c0 0x7fe6040c0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe609ca5640 1 -- 172.21.15.150:0/4260560417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe604073150 msgr2=0x7fe60414ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe609ca5640 1 --2- 172.21.15.150:0/4260560417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe604073150 0x7fe60414ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe609ca5640 1 -- 172.21.15.150:0/4260560417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60414a680 con 0x7fe6040c34c0 2024-09-17T04:49:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe609ca5640 1 --2- 172.21.15.150:0/4260560417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6040c34c0 0x7fe604200480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe604153bc0 tx=0x7fe5fc004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe5faffd640 1 -- 172.21.15.150:0/4260560417 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0b2040 con 0x7fe6040c34c0 2024-09-17T04:49:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe5faffd640 1 -- 172.21.15.150:0/4260560417 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe5fc003a70 con 0x7fe6040c34c0 2024-09-17T04:49:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7fe5faffd640 1 -- 172.21.15.150:0/4260560417 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0ad900 con 0x7fe6040c34c0 2024-09-17T04:49:06.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.944+0000 7f0f03026640 1 -- 172.21.15.150:0/1033285454 wait complete. 2024-09-17T04:49:06.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0f03026640 1 Processor -- start 2024-09-17T04:49:06.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7fe60b4a8640 1 -- 172.21.15.150:0/4260560417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe60414c490 con 0x7fe6040c34c0 2024-09-17T04:49:06.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7fe60b4a8640 1 -- 172.21.15.150:0/4260560417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe60414ca50 con 0x7fe6040c34c0 2024-09-17T04:49:06.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7fe5faffd640 1 -- 172.21.15.150:0/4260560417 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe5fc0bd020 con 0x7fe6040c34c0 2024-09-17T04:49:06.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7fe5faffd640 1 -- 172.21.15.150:0/4260560417 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe5fc0b64d0 con 0x7fe6040c34c0 2024-09-17T04:49:06.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0f03026640 1 -- start start 2024-09-17T04:49:06.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0f03026640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc074f70 0x7f0efc153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0f00dc2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc074f70 0x7f0efc153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0f00dc2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc074f70 0x7f0efc153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54668/0 (socket says 172.21.15.150:54668) 2024-09-17T04:49:06.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0f00dc2640 1 -- 172.21.15.150:0/3853123604 learned_addr learned my addr 172.21.15.150:0/3853123604 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0f03026640 1 --2- 172.21.15.150:0/3853123604 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0efc156840 0x7f0efc155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0f03026640 1 --2- 172.21.15.150:0/3853123604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0efc156010 0x7f0efc0bd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0f03026640 1 -- 172.21.15.150:0/3853123604 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc06ddf0 con 0x7f0efc074f70 2024-09-17T04:49:06.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0f03026640 1 -- 172.21.15.150:0/3853123604 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc06dc70 con 0x7f0efc156010 2024-09-17T04:49:06.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0f015c3640 1 --2- 172.21.15.150:0/3853123604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0efc156010 0x7f0efc0bd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0f03026640 1 -- 172.21.15.150:0/3853123604 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0efc071c50 con 0x7f0efc156840 2024-09-17T04:49:06.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7f0efbfff640 1 --2- 172.21.15.150:0/3853123604 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0efc156840 0x7f0efc155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7fcdfc52b640 1 -- 172.21.15.150:0/3358979855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcdf40c5dc0 msgr2=0x7fcdf40c61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7fcdfc52b640 1 --2- 172.21.15.150:0/3358979855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcdf40c5dc0 0x7fcdf40c61d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0008360 tx=0x7fcdf001e2f0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.948+0000 7fcdfc52b640 1 -- 172.21.15.150:0/3358979855 shutdown_connections 2024-09-17T04:49:06.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7fcdfc52b640 1 --2- 172.21.15.150:0/3358979855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcdf40c5dc0 0x7fcdf40c61d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7fcdfc52b640 1 -- 172.21.15.150:0/3358979855 >> 172.21.15.150:0/3358979855 conn(0x7fcdf40c7be0 msgr2=0x7fcdf40c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7fcdfc52b640 1 -- 172.21.15.150:0/3358979855 shutdown_connections 2024-09-17T04:49:06.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7f0f00dc2640 1 -- 172.21.15.150:0/3853123604 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0efc156840 msgr2=0x7f0efc155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7f0f00dc2640 1 --2- 172.21.15.150:0/3853123604 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0efc156840 0x7f0efc155ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7f0f00dc2640 1 -- 172.21.15.150:0/3853123604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0efc156010 msgr2=0x7f0efc0bd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7f0f00dc2640 1 --2- 172.21.15.150:0/3853123604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0efc156010 0x7f0efc0bd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7f0f00dc2640 1 -- 172.21.15.150:0/3853123604 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc0accc0 con 0x7f0efc074f70 2024-09-17T04:49:06.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7f0f00dc2640 1 --2- 172.21.15.150:0/3853123604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc074f70 0x7f0efc153270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000d6d0 tx=0x7f0ef000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7fcdfc52b640 1 -- 172.21.15.150:0/3358979855 wait complete. 2024-09-17T04:49:06.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7fcdfc52b640 1 Processor -- start 2024-09-17T04:49:06.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7fcdfc52b640 1 -- start start 2024-09-17T04:49:06.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7f0ef97fa640 1 -- 172.21.15.150:0/3853123604 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0017070 con 0x7f0efc074f70 2024-09-17T04:49:06.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7f0f03026640 1 -- 172.21.15.150:0/3853123604 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0efc0bf8c0 con 0x7f0efc074f70 2024-09-17T04:49:06.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.952+0000 7f0f03026640 1 -- 172.21.15.150:0/3853123604 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0efc0ac420 con 0x7f0efc074f70 2024-09-17T04:49:06.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fa18645a640 1 -- 172.21.15.150:0/2662791136 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa18008b5a0 msgr2=0x7fa1800c7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fa18645a640 1 --2- 172.21.15.150:0/2662791136 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa18008b5a0 0x7fa1800c7130 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa1800c1f70 tx=0x7fa17801cc80 comp rx=0 tx=0).stop 2024-09-17T04:49:06.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fe724ff9640 1 -- 172.21.15.150:0/908947671 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c0052d0 con 0x7fe72808bdb0 2024-09-17T04:49:06.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f5ab850b640 1 -- 172.21.15.150:0/877412777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ab00c40c0 msgr2=0x7f5ab00c4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f5ab850b640 1 --2- 172.21.15.150:0/877412777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ab00c40c0 0x7f5ab00c4840 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00722a0 tx=0x7f5aa802daa0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f0ef97fa640 1 -- 172.21.15.150:0/3853123604 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ef0009b10 con 0x7f0efc074f70 2024-09-17T04:49:06.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f0ef97fa640 1 -- 172.21.15.150:0/3853123604 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0010460 con 0x7f0efc074f70 2024-09-17T04:49:06.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f5ab850b640 1 -- 172.21.15.150:0/877412777 shutdown_connections 2024-09-17T04:49:06.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f5ab850b640 1 --2- 172.21.15.150:0/877412777 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ab00c2e80 0x7f5ab0159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f5ab850b640 1 --2- 172.21.15.150:0/877412777 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ab00c40c0 0x7f5ab00c4840 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f5ab850b640 1 --2- 172.21.15.150:0/877412777 >> v2:172.21.15.176:3300/0 conn(0x7f5ab00c7700 0x7f5ab006f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f5ab850b640 1 -- 172.21.15.150:0/877412777 >> 172.21.15.150:0/877412777 conn(0x7f5ab00c7be0 msgr2=0x7f5ab00c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdfc52b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcdf4075250 0x7fcdf40c0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f0ef97fa640 1 -- 172.21.15.150:0/3853123604 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0ef001a070 con 0x7f0efc074f70 2024-09-17T04:49:06.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdfc52b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdf408b5a0 0x7fcdf40c09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdfc52b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdf40c5dc0 0x7fcdf4201f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f0ef97fa640 1 -- 172.21.15.150:0/3853123604 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0ef001cb50 con 0x7f0efc074f70 2024-09-17T04:49:06.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdfc52b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf4176520 con 0x7fcdf408b5a0 2024-09-17T04:49:06.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f5ab850b640 1 -- 172.21.15.150:0/877412777 shutdown_connections 2024-09-17T04:49:06.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7f5ab850b640 1 -- 172.21.15.150:0/877412777 wait complete. 2024-09-17T04:49:06.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fe5f8ff9640 1 -- 172.21.15.150:0/4260560417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6040c2f80 con 0x7fe6040c34c0 2024-09-17T04:49:06.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fe72d41a640 1 -- 172.21.15.150:0/908947671 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe728170a60 con 0x7fe72808bdb0 2024-09-17T04:49:06.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fe72d41a640 1 -- 172.21.15.150:0/908947671 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe72815acb0 con 0x7fe72808bdb0 2024-09-17T04:49:06.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdfc52b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf41763a0 con 0x7fcdf40c5dc0 2024-09-17T04:49:06.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdfa2c7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcdf4075250 0x7fcdf40c0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdfc52b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcdf41766a0 con 0x7fcdf4075250 2024-09-17T04:49:06.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdfaac8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdf40c5dc0 0x7fcdf4201f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdfa2c7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcdf4075250 0x7fcdf40c0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:36588/0 (socket says 172.21.15.150:36588) 2024-09-17T04:49:06.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdfa2c7640 1 -- 172.21.15.150:0/1864869572 learned_addr learned my addr 172.21.15.150:0/1864869572 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdf9ac6640 1 --2- 172.21.15.150:0/1864869572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdf408b5a0 0x7fcdf40c09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdf9ac6640 1 -- 172.21.15.150:0/1864869572 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcdf4075250 msgr2=0x7fcdf40c0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdf9ac6640 1 --2- 172.21.15.150:0/1864869572 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcdf4075250 0x7fcdf40c0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdf9ac6640 1 -- 172.21.15.150:0/1864869572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdf40c5dc0 msgr2=0x7fcdf4201f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdf9ac6640 1 --2- 172.21.15.150:0/1864869572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdf40c5dc0 0x7fcdf4201f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdf9ac6640 1 -- 172.21.15.150:0/1864869572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf40bf430 con 0x7fcdf408b5a0 2024-09-17T04:49:06.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fcdf9ac6640 1 --2- 172.21.15.150:0/1864869572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdf408b5a0 0x7fcdf40c09a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00c8a0 tx=0x7fcdec00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab850b640 1 Processor -- start 2024-09-17T04:49:06.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab850b640 1 -- start start 2024-09-17T04:49:06.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fe72d41a640 1 -- 172.21.15.150:0/908947671 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7280c4470 con 0x7fe72808bdb0 2024-09-17T04:49:06.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab850b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ab00c2e80 0x7f5ab00a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7fe5faffd640 1 -- 172.21.15.150:0/4260560417 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fe5fc0c4020 con 0x7fe6040c34c0 2024-09-17T04:49:06.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab62a7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ab00c2e80 0x7f5ab00a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab62a7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ab00c2e80 0x7f5ab00a7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54680/0 (socket says 172.21.15.150:54680) 2024-09-17T04:49:06.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab62a7640 1 -- 172.21.15.150:0/612241646 learned_addr learned my addr 172.21.15.150:0/612241646 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab850b640 1 --2- 172.21.15.150:0/612241646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ab00c40c0 0x7f5ab0156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab850b640 1 --2- 172.21.15.150:0/612241646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ab00c7700 0x7f5ab0156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab850b640 1 -- 172.21.15.150:0/612241646 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab0152b10 con 0x7f5ab00c2e80 2024-09-17T04:49:06.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab850b640 1 -- 172.21.15.150:0/612241646 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab0070d90 con 0x7f5ab00c7700 2024-09-17T04:49:06.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab850b640 1 -- 172.21.15.150:0/612241646 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab0152c90 con 0x7f5ab00c40c0 2024-09-17T04:49:06.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab5aa6640 1 --2- 172.21.15.150:0/612241646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ab00c40c0 0x7f5ab0156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab6aa8640 1 --2- 172.21.15.150:0/612241646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ab00c7700 0x7f5ab0156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab5aa6640 1 -- 172.21.15.150:0/612241646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ab00c7700 msgr2=0x7f5ab0156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab5aa6640 1 --2- 172.21.15.150:0/612241646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ab00c7700 0x7f5ab0156aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab5aa6640 1 -- 172.21.15.150:0/612241646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ab00c2e80 msgr2=0x7f5ab00a7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab5aa6640 1 --2- 172.21.15.150:0/612241646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ab00c2e80 0x7f5ab00a7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.960+0000 7f5ab5aa6640 1 -- 172.21.15.150:0/612241646 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab01effd0 con 0x7f5ab00c40c0 2024-09-17T04:49:06.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.964+0000 7f5ab5aa6640 1 --2- 172.21.15.150:0/612241646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ab00c40c0 0x7f5ab0156560 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00c8b20 tx=0x7f5aa801a860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.956+0000 7fe724ff9640 1 -- 172.21.15.150:0/908947671 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe71c007020 con 0x7fe72808bdb0 2024-09-17T04:49:06.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.968+0000 7f0f03026640 1 -- 172.21.15.150:0/3853123604 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0edc0052a0 con 0x7f0efc074f70 2024-09-17T04:49:06.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.968+0000 7f5aa6ffd640 1 -- 172.21.15.150:0/612241646 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa80c3040 con 0x7f5ab00c40c0 2024-09-17T04:49:06.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.968+0000 7f5ab850b640 1 -- 172.21.15.150:0/612241646 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab01efcf0 con 0x7f5ab00c40c0 2024-09-17T04:49:06.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.968+0000 7f5ab850b640 1 -- 172.21.15.150:0/612241646 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab00a8f10 con 0x7f5ab00c40c0 2024-09-17T04:49:06.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.968+0000 7fe724ff9640 1 -- 172.21.15.150:0/908947671 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71c007470 con 0x7fe72808bdb0 2024-09-17T04:49:06.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.968+0000 7fe724ff9640 1 -- 172.21.15.150:0/908947671 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe71c0067e0 con 0x7fe72808bdb0 2024-09-17T04:49:06.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.968+0000 7f5aa6ffd640 1 -- 172.21.15.150:0/612241646 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5aa80b6d20 con 0x7f5ab00c40c0 2024-09-17T04:49:06.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.968+0000 7fe724ff9640 1 -- 172.21.15.150:0/908947671 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe71c003cd0 con 0x7fe72808bdb0 2024-09-17T04:49:06.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.968+0000 7f5aa6ffd640 1 -- 172.21.15.150:0/612241646 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8003be0 con 0x7f5ab00c40c0 2024-09-17T04:49:06.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.968+0000 7fe724ff9640 1 -- 172.21.15.150:0/908947671 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fe72815af90 con 0x7fe72808bdb0 2024-09-17T04:49:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fa18645a640 1 -- 172.21.15.150:0/2662791136 shutdown_connections 2024-09-17T04:49:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fa18645a640 1 --2- 172.21.15.150:0/2662791136 >> v2:172.21.15.150:3300/0 conn(0x7fa1800c7670 0x7fa1800c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fa18645a640 1 --2- 172.21.15.150:0/2662791136 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa18008b5a0 0x7fa1800c7130 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fa18645a640 1 --2- 172.21.15.150:0/2662791136 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa180075250 0x7fa1800c3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fa18645a640 1 -- 172.21.15.150:0/2662791136 >> 172.21.15.150:0/2662791136 conn(0x7fa1800c7be0 msgr2=0x7fa1800c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7f0ef97fa640 1 -- 172.21.15.150:0/3853123604 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f0ef0010650 con 0x7f0efc074f70 2024-09-17T04:49:06.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fcdeaffd640 1 -- 172.21.15.150:0/1864869572 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec017070 con 0x7fcdf408b5a0 2024-09-17T04:49:06.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fcdfc52b640 1 -- 172.21.15.150:0/1864869572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf40bf9b0 con 0x7fcdf408b5a0 2024-09-17T04:49:06.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fcdfc52b640 1 -- 172.21.15.150:0/1864869572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf40be220 con 0x7fcdf408b5a0 2024-09-17T04:49:06.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fcdeaffd640 1 -- 172.21.15.150:0/1864869572 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcdec005070 con 0x7fcdf408b5a0 2024-09-17T04:49:06.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fcdeaffd640 1 -- 172.21.15.150:0/1864869572 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec005480 con 0x7fcdf408b5a0 2024-09-17T04:49:06.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7f5aa6ffd640 1 -- 172.21.15.150:0/612241646 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5aa80043c0 con 0x7f5ab00c40c0 2024-09-17T04:49:06.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7f5aa6ffd640 1 -- 172.21.15.150:0/612241646 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5aa8002020 con 0x7f5ab00c40c0 2024-09-17T04:49:06.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fa18645a640 1 -- 172.21.15.150:0/2662791136 shutdown_connections 2024-09-17T04:49:06.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fa18645a640 1 -- 172.21.15.150:0/2662791136 wait complete. 2024-09-17T04:49:06.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fa18645a640 1 Processor -- start 2024-09-17T04:49:06.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.972+0000 7fa18645a640 1 -- start start 2024-09-17T04:49:06.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa18645a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa180075250 0x7fa180169cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa18645a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa18008b5a0 0x7fa18020d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa18645a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1800c7670 0x7fa180167af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:06.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa17ffff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa180075250 0x7fa180169cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa17ffff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa180075250 0x7fa180169cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:33296/0 (socket says 172.21.15.150:33296) 2024-09-17T04:49:06.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa17ffff640 1 -- 172.21.15.150:0/1501746750 learned_addr learned my addr 172.21.15.150:0/1501746750 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:06.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa1849f7640 1 --2- 172.21.15.150:0/1501746750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1800c7670 0x7fa180167af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:06.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa18645a640 1 -- 172.21.15.150:0/1501746750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa180166b80 con 0x7fa1800c7670 2024-09-17T04:49:06.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa18645a640 1 -- 172.21.15.150:0/1501746750 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa180166a00 con 0x7fa180075250 2024-09-17T04:49:06.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa18645a640 1 -- 172.21.15.150:0/1501746750 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa180166d00 con 0x7fa18008b5a0 2024-09-17T04:49:06.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fcdeaffd640 1 -- 172.21.15.150:0/1864869572 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcdec01a070 con 0x7fcdf408b5a0 2024-09-17T04:49:06.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa1849f7640 1 -- 172.21.15.150:0/1501746750 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa18008b5a0 msgr2=0x7fa18020d5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T04:49:06.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa1849f7640 1 --2- 172.21.15.150:0/1501746750 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa18008b5a0 0x7fa18020d5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa1849f7640 1 -- 172.21.15.150:0/1501746750 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa180075250 msgr2=0x7fa180169cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa1849f7640 1 --2- 172.21.15.150:0/1501746750 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa180075250 0x7fa180169cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa1849f7640 1 -- 172.21.15.150:0/1501746750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa180166660 con 0x7fa1800c7670 2024-09-17T04:49:06.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fa1849f7640 1 --2- 172.21.15.150:0/1501746750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1800c7670 0x7fa180167af0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa17400bb20 tx=0x7fa1740071c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:06.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.976+0000 7fcdeaffd640 1 -- 172.21.15.150:0/1864869572 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcdec01c780 con 0x7fcdf408b5a0 2024-09-17T04:49:06.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.980+0000 7fa17cff9640 1 -- 172.21.15.150:0/1501746750 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa174004220 con 0x7fa1800c7670 2024-09-17T04:49:06.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.980+0000 7fa18645a640 1 -- 172.21.15.150:0/1501746750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa18016cb30 con 0x7fa1800c7670 2024-09-17T04:49:06.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.980+0000 7fa18645a640 1 -- 172.21.15.150:0/1501746750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa180165d60 con 0x7fa1800c7670 2024-09-17T04:49:06.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.980+0000 7fa17cff9640 1 -- 172.21.15.150:0/1501746750 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa174014070 con 0x7fa1800c7670 2024-09-17T04:49:06.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.980+0000 7fa17cff9640 1 -- 172.21.15.150:0/1501746750 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1740078d0 con 0x7fa1800c7670 2024-09-17T04:49:06.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.980+0000 7f5ab850b640 1 -- 172.21.15.150:0/612241646 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a980052a0 con 0x7f5ab00c40c0 2024-09-17T04:49:06.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.980+0000 7fa17cff9640 1 -- 172.21.15.150:0/1501746750 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa174016030 con 0x7fa1800c7670 2024-09-17T04:49:06.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.980+0000 7fa17cff9640 1 -- 172.21.15.150:0/1501746750 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa174018a60 con 0x7fa1800c7670 2024-09-17T04:49:06.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.988+0000 7f5aa6ffd640 1 -- 172.21.15.150:0/612241646 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f5aa80ce060 con 0x7f5ab00c40c0 2024-09-17T04:49:06.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.992+0000 7fcdfc52b640 1 -- 172.21.15.150:0/1864869572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcddc0052a0 con 0x7fcdf408b5a0 2024-09-17T04:49:06.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.992+0000 7fa18645a640 1 -- 172.21.15.150:0/1501746750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa15c0052a0 con 0x7fa1800c7670 2024-09-17T04:49:06.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.996+0000 7f0f1d43f640 1 -- 172.21.15.150:0/2756485854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f181501c0 msgr2=0x7f0f18152fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:06.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:06.996+0000 7f0f1d43f640 1 --2- 172.21.15.150:0/2756485854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f181501c0 0x7f0f18152fd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0f1000afd0 tx=0x7f0f1001d650 comp rx=0 tx=0).stop 2024-09-17T04:49:06.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.000+0000 7fa17cff9640 1 -- 172.21.15.150:0/1501746750 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fa174007e00 con 0x7fa1800c7670 2024-09-17T04:49:06.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.000+0000 7fcdeaffd640 1 -- 172.21.15.150:0/1864869572 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fcdec021020 con 0x7fcdf408b5a0 2024-09-17T04:49:06.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.000+0000 7f0f1d43f640 1 -- 172.21.15.150:0/2756485854 shutdown_connections 2024-09-17T04:49:06.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.000+0000 7f0f1d43f640 1 --2- 172.21.15.150:0/2756485854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f181501c0 0x7f0f18152fd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:06.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.000+0000 7f0f1d43f640 1 -- 172.21.15.150:0/2756485854 >> 172.21.15.150:0/2756485854 conn(0x7f0f180c4a00 msgr2=0x7f0f180c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:06.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.000+0000 7f7237fff640 1 -- 172.21.15.150:0/3723138255 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f721c005590 con 0x7f7238156c70 2024-09-17T04:49:06.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.000+0000 7f0f1d43f640 1 -- 172.21.15.150:0/2756485854 shutdown_connections 2024-09-17T04:49:07.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.000+0000 7f0f1d43f640 1 -- 172.21.15.150:0/2756485854 wait complete. 2024-09-17T04:49:07.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.000+0000 7f0f1d43f640 1 Processor -- start 2024-09-17T04:49:07.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.000+0000 7f0f1d43f640 1 -- start start 2024-09-17T04:49:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f1d43f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f18071310 0x7f0f1814d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f1d43f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f180c3ac0 0x7f0f180a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f167fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f180c3ac0 0x7f0f180a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f167fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f180c3ac0 0x7f0f180a5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54694/0 (socket says 172.21.15.150:54694) 2024-09-17T04:49:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f1d43f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f181501c0 0x7f0f181f7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f16ffd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f18071310 0x7f0f1814d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f16ffd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f18071310 0x7f0f1814d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:33312/0 (socket says 172.21.15.150:33312) 2024-09-17T04:49:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f167fc640 1 -- 172.21.15.150:0/1177860568 learned_addr learned my addr 172.21.15.150:0/1177860568 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:07.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f1d43f640 1 -- 172.21.15.150:0/1177860568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1806f3e0 con 0x7f0f180c3ac0 2024-09-17T04:49:07.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f1d43f640 1 -- 172.21.15.150:0/1177860568 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0f180710d0 con 0x7f0f18071310 2024-09-17T04:49:07.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f1d43f640 1 -- 172.21.15.150:0/1177860568 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0f18162c00 con 0x7f0f181501c0 2024-09-17T04:49:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f177fe640 1 --2- 172.21.15.150:0/1177860568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f181501c0 0x7f0f181f7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f167fc640 1 -- 172.21.15.150:0/1177860568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f181501c0 msgr2=0x7f0f181f7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f167fc640 1 --2- 172.21.15.150:0/1177860568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f181501c0 0x7f0f181f7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f167fc640 1 -- 172.21.15.150:0/1177860568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f18071310 msgr2=0x7f0f1814d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f167fc640 1 --2- 172.21.15.150:0/1177860568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f18071310 0x7f0f1814d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f167fc640 1 -- 172.21.15.150:0/1177860568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1814dec0 con 0x7f0f180c3ac0 2024-09-17T04:49:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f177fe640 1 --2- 172.21.15.150:0/1177860568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f181501c0 0x7f0f181f7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:07.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.004+0000 7f0f167fc640 1 --2- 172.21.15.150:0/1177860568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f180c3ac0 0x7f0f180a5000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0f0800c430 tx=0x7f0f0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:07.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.008+0000 7f0f077fe640 1 -- 172.21.15.150:0/1177860568 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f080052d0 con 0x7f0f180c3ac0 2024-09-17T04:49:07.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.008+0000 7f0f077fe640 1 -- 172.21.15.150:0/1177860568 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f08007020 con 0x7f0f180c3ac0 2024-09-17T04:49:07.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.008+0000 7f0f077fe640 1 -- 172.21.15.150:0/1177860568 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f08007470 con 0x7f0f180c3ac0 2024-09-17T04:49:07.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.008+0000 7f0f1d43f640 1 -- 172.21.15.150:0/1177860568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f180a3650 con 0x7f0f180c3ac0 2024-09-17T04:49:07.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.008+0000 7f0f1d43f640 1 -- 172.21.15.150:0/1177860568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f180c1fb0 con 0x7f0f180c3ac0 2024-09-17T04:49:07.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.008+0000 7f3c2e119640 1 -- 172.21.15.150:0/3263321310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f3c08005590 con 0x7f3c280c2e80 2024-09-17T04:49:07.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.008+0000 7f722f7fe640 1 -- 172.21.15.150:0/3723138255 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v11)=0 v11) ==== 116+0+0 (secure 0 0 0) 0x7f7228020440 con 0x7f7238156c70 2024-09-17T04:49:07.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.008+0000 7f0f077fe640 1 -- 172.21.15.150:0/1177860568 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0f080067e0 con 0x7f0f180c3ac0 2024-09-17T04:49:07.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.008+0000 7f0f077fe640 1 -- 172.21.15.150:0/1177860568 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0f08003b80 con 0x7f0f180c3ac0 2024-09-17T04:49:07.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.016+0000 7f0f1d43f640 1 -- 172.21.15.150:0/1177860568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0efc0052a0 con 0x7f0f180c3ac0 2024-09-17T04:49:07.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.016+0000 7f3c0ffff640 1 -- 172.21.15.150:0/3263321310 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v12)=0 v12) ==== 120+0+0 (secure 0 0 0) 0x7f3c1801bb70 con 0x7f3c280c2e80 2024-09-17T04:49:07.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.020+0000 7f7237fff640 1 -- 172.21.15.150:0/3723138255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7238156c70 msgr2=0x7f72380c7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.020+0000 7f7237fff640 1 --2- 172.21.15.150:0/3723138255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7238156c70 0x7f72380c7f30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f722800fcb0 tx=0x7f722800d430 comp rx=0 tx=0).stop 2024-09-17T04:49:07.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.020+0000 7f7237fff640 1 -- 172.21.15.150:0/3723138255 shutdown_connections 2024-09-17T04:49:07.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.020+0000 7f7237fff640 1 --2- 172.21.15.150:0/3723138255 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7238157080 0x7f72381fcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.020+0000 7f7237fff640 1 --2- 172.21.15.150:0/3723138255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7238156c70 0x7f72380c7f30 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f722800fcb0 tx=0x7f722800d430 comp rx=0 tx=0).stop 2024-09-17T04:49:07.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.020+0000 7f7237fff640 1 --2- 172.21.15.150:0/3723138255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7238070d30 0x7f72380c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.020+0000 7f7237fff640 1 -- 172.21.15.150:0/3723138255 >> 172.21.15.150:0/3723138255 conn(0x7f723806d250 msgr2=0x7f7238150040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.020+0000 7f0f077fe640 1 -- 172.21.15.150:0/1177860568 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f0f08007610 con 0x7f0f180c3ac0 2024-09-17T04:49:07.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.032+0000 7f7237fff640 1 -- 172.21.15.150:0/3723138255 shutdown_connections 2024-09-17T04:49:07.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.032+0000 7f7237fff640 1 -- 172.21.15.150:0/3723138255 wait complete. 2024-09-17T04:49:07.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.044+0000 7f3c0dffb640 1 -- 172.21.15.150:0/3263321310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c280c2e80 msgr2=0x7f3c280be370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.044+0000 7f3c0dffb640 1 --2- 172.21.15.150:0/3263321310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c280c2e80 0x7f3c280be370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3c1800c760 tx=0x7f3c1800cc20 comp rx=0 tx=0).stop 2024-09-17T04:49:07.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.044+0000 7f3c0dffb640 1 -- 172.21.15.150:0/3263321310 shutdown_connections 2024-09-17T04:49:07.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.044+0000 7f3c0dffb640 1 --2- 172.21.15.150:0/3263321310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c280c40c0 0x7f3c281ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.044+0000 7f3c0dffb640 1 --2- 172.21.15.150:0/3263321310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c280c7700 0x7f3c28151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.044+0000 7f3c0dffb640 1 --2- 172.21.15.150:0/3263321310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c280c2e80 0x7f3c280be370 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.044+0000 7f3c0dffb640 1 -- 172.21.15.150:0/3263321310 >> 172.21.15.150:0/3263321310 conn(0x7f3c280c7be0 msgr2=0x7f3c280741e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.044+0000 7f3c0dffb640 1 -- 172.21.15.150:0/3263321310 shutdown_connections 2024-09-17T04:49:07.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.044+0000 7f3c0dffb640 1 -- 172.21.15.150:0/3263321310 wait complete. 2024-09-17T04:49:07.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.056+0000 7f03c17fa640 1 -- 172.21.15.150:0/4210187339 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7f03dc0c7520 con 0x7f03dc0bdda0 2024-09-17T04:49:07.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.068+0000 7f03c37fe640 1 -- 172.21.15.150:0/4210187339 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v13)=0 v13) ==== 119+0+0 (secure 0 0 0) 0x7f03dc156a60 con 0x7f03dc0bdda0 2024-09-17T04:49:07.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.072+0000 7f56b6e92640 1 -- 172.21.15.150:0/2446342126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b00c4480 msgr2=0x7f56b00747b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.072+0000 7f56b6e92640 1 --2- 172.21.15.150:0/2446342126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b00c4480 0x7f56b00747b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f56a8009c80 tx=0x7f56a801cd30 comp rx=0 tx=0).stop 2024-09-17T04:49:07.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.076+0000 7f56b6e92640 1 -- 172.21.15.150:0/2446342126 shutdown_connections 2024-09-17T04:49:07.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.076+0000 7f56b6e92640 1 --2- 172.21.15.150:0/2446342126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b00c4480 0x7f56b00747b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.076+0000 7f56b6e92640 1 -- 172.21.15.150:0/2446342126 >> 172.21.15.150:0/2446342126 conn(0x7f56b006ce50 msgr2=0x7f56b006d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.080+0000 7f56b6e92640 1 -- 172.21.15.150:0/2446342126 shutdown_connections 2024-09-17T04:49:07.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.080+0000 7f56b6e92640 1 -- 172.21.15.150:0/2446342126 wait complete. 2024-09-17T04:49:07.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.080+0000 7f56b6e92640 1 Processor -- start 2024-09-17T04:49:07.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.080+0000 7f56b6e92640 1 -- start start 2024-09-17T04:49:07.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b6e92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56b00c4480 0x7f56b00c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:07.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b6e92640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b00c4890 0x7f56b00c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:07.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b6e92640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56b0156680 0x7f56b00c5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:07.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b6e92640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f56b013f480 con 0x7f56b0156680 2024-09-17T04:49:07.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b6e92640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f56b013f300 con 0x7f56b00c4890 2024-09-17T04:49:07.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b6e92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f56b013f600 con 0x7f56b00c4480 2024-09-17T04:49:07.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b5e90640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56b00c4480 0x7f56b00c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:07.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b5e90640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56b00c4480 0x7f56b00c75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:36628/0 (socket says 172.21.15.150:36628) 2024-09-17T04:49:07.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b5e90640 1 -- 172.21.15.150:0/56425817 learned_addr learned my addr 172.21.15.150:0/56425817 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:07.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b568f640 1 --2- 172.21.15.150:0/56425817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b00c4890 0x7f56b00c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:07.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b6691640 1 --2- 172.21.15.150:0/56425817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56b0156680 0x7f56b00c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:07.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b6691640 1 -- 172.21.15.150:0/56425817 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56b00c4480 msgr2=0x7f56b00c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b6691640 1 --2- 172.21.15.150:0/56425817 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56b00c4480 0x7f56b00c75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b6691640 1 -- 172.21.15.150:0/56425817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b00c4890 msgr2=0x7f56b00c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b6691640 1 --2- 172.21.15.150:0/56425817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b00c4890 0x7f56b00c8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b6691640 1 -- 172.21.15.150:0/56425817 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b00ab2b0 con 0x7f56b0156680 2024-09-17T04:49:07.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b5e90640 1 --2- 172.21.15.150:0/56425817 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56b00c4480 0x7f56b00c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:07.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f56b568f640 1 --2- 172.21.15.150:0/56425817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b00c4890 0x7f56b00c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:07.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f2ebef2a640 1 -- 172.21.15.150:0/2489937224 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7f2ea0005590 con 0x7f2eb80c40b0 2024-09-17T04:49:07.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.084+0000 7f6f70bd3640 1 -- 172.21.15.150:0/1562326491 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f6f50005590 con 0x7f6f6c0c40c0 2024-09-17T04:49:07.086 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.088+0000 7f56b6691640 1 --2- 172.21.15.150:0/56425817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56b0156680 0x7f56b00c5820 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f56b00738c0 tx=0x7f56a000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:07.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.088+0000 7f03e11c2640 1 -- 172.21.15.150:0/4210187339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03dc0bdda0 msgr2=0x7f03dc0bedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.088+0000 7f03e11c2640 1 --2- 172.21.15.150:0/4210187339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03dc0bdda0 0x7f03dc0bedf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f03d00ad040 tx=0x7f03d0003c60 comp rx=0 tx=0).stop 2024-09-17T04:49:07.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.088+0000 7f56a67fc640 1 -- 172.21.15.150:0/56425817 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a0015020 con 0x7f56b0156680 2024-09-17T04:49:07.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.088+0000 7f56b6e92640 1 -- 172.21.15.150:0/56425817 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b00c6320 con 0x7f56b0156680 2024-09-17T04:49:07.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.088+0000 7f56b6e92640 1 -- 172.21.15.150:0/56425817 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b00acb90 con 0x7f56b0156680 2024-09-17T04:49:07.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.088+0000 7f56a67fc640 1 -- 172.21.15.150:0/56425817 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56a0011920 con 0x7f56b0156680 2024-09-17T04:49:07.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.088+0000 7f56a67fc640 1 -- 172.21.15.150:0/56425817 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a0011d70 con 0x7f56b0156680 2024-09-17T04:49:07.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.092+0000 7f03e11c2640 1 -- 172.21.15.150:0/4210187339 shutdown_connections 2024-09-17T04:49:07.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.092+0000 7f03e11c2640 1 --2- 172.21.15.150:0/4210187339 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03dc0be1d0 0x7f03dc1fd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.092+0000 7f03e11c2640 1 --2- 172.21.15.150:0/4210187339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03dc156100 0x7f03dc1f9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.092+0000 7f03e11c2640 1 --2- 172.21.15.150:0/4210187339 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03dc0bdda0 0x7f03dc0bedf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.092+0000 7f03e11c2640 1 -- 172.21.15.150:0/4210187339 >> 172.21.15.150:0/4210187339 conn(0x7f03dc0c7be0 msgr2=0x7f03dc1e5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.092+0000 7f56a67fc640 1 -- 172.21.15.150:0/56425817 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f56a0018020 con 0x7f56b0156680 2024-09-17T04:49:07.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.092+0000 7f56a67fc640 1 -- 172.21.15.150:0/56425817 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f56a0019860 con 0x7f56b0156680 2024-09-17T04:49:07.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.096+0000 7f2eae7fc640 1 -- 172.21.15.150:0/2489937224 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v14)=0 v14) ==== 116+0+0 (secure 0 0 0) 0x7f2ea8019490 con 0x7f2eb80c40b0 2024-09-17T04:49:07.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.100+0000 7f03e11c2640 1 -- 172.21.15.150:0/4210187339 shutdown_connections 2024-09-17T04:49:07.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.100+0000 7f03e11c2640 1 -- 172.21.15.150:0/4210187339 wait complete. 2024-09-17T04:49:07.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.100+0000 7f56b6e92640 1 -- 172.21.15.150:0/56425817 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56980052a0 con 0x7f56b0156680 2024-09-17T04:49:07.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.104+0000 7f6f5affd640 1 -- 172.21.15.150:0/1562326491 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v15)=0 v15) ==== 121+0+0 (secure 0 0 0) 0x7f6f60003d90 con 0x7f6f6c0c40c0 2024-09-17T04:49:07.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.104+0000 7f56a67fc640 1 -- 172.21.15.150:0/56425817 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f56a001f020 con 0x7f56b0156680 2024-09-17T04:49:07.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f2ebef2a640 1 -- 172.21.15.150:0/2489937224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2eb80c40b0 msgr2=0x7f2eb80a9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f2ebef2a640 1 --2- 172.21.15.150:0/2489937224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2eb80c40b0 0x7f2eb80a9880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800b410 tx=0x7f2ea800b8d0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f6f70bd3640 1 -- 172.21.15.150:0/1562326491 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f6c0c40c0 msgr2=0x7f6f6c074570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f6f70bd3640 1 --2- 172.21.15.150:0/1562326491 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f6c0c40c0 0x7f6f6c074570 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000c3f0 tx=0x7f6f6000c7f0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f2ebef2a640 1 -- 172.21.15.150:0/2489937224 shutdown_connections 2024-09-17T04:49:07.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f2ebef2a640 1 --2- 172.21.15.150:0/2489937224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2eb80c40b0 0x7f2eb80a9880 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f2ebef2a640 1 --2- 172.21.15.150:0/2489937224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb80ad660 0x7f2eb814ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f2ebef2a640 1 --2- 172.21.15.150:0/2489937224 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eb8071970 0x7f2eb81ffb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f2ebef2a640 1 -- 172.21.15.150:0/2489937224 >> 172.21.15.150:0/2489937224 conn(0x7f2eb806d250 msgr2=0x7f2eb806d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f6f70bd3640 1 -- 172.21.15.150:0/1562326491 shutdown_connections 2024-09-17T04:49:07.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f6f70bd3640 1 --2- 172.21.15.150:0/1562326491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f6c0c7700 0x7f6f6c151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f6f70bd3640 1 --2- 172.21.15.150:0/1562326491 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f6c0c40c0 0x7f6f6c074570 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f6f70bd3640 1 --2- 172.21.15.150:0/1562326491 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f6c06f840 0x7f6f6c0aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f6f70bd3640 1 -- 172.21.15.150:0/1562326491 >> 172.21.15.150:0/1562326491 conn(0x7f6f6c0c7be0 msgr2=0x7f6f6c068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f2ebef2a640 1 -- 172.21.15.150:0/2489937224 shutdown_connections 2024-09-17T04:49:07.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f6f70bd3640 1 -- 172.21.15.150:0/1562326491 shutdown_connections 2024-09-17T04:49:07.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f2ebef2a640 1 -- 172.21.15.150:0/2489937224 wait complete. 2024-09-17T04:49:07.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.132+0000 7f6f70bd3640 1 -- 172.21.15.150:0/1562326491 wait complete. 2024-09-17T04:49:07.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.156+0000 7fe72d41a640 1 -- 172.21.15.150:0/908947671 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fe7280ab460 con 0x7fe72808bdb0 2024-09-17T04:49:07.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.160+0000 7fe724ff9640 1 -- 172.21.15.150:0/908947671 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v16)=0 v16) ==== 106+0+0 (secure 0 0 0) 0x7fe71c007610 con 0x7fe72808bdb0 2024-09-17T04:49:07.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.176+0000 7fe7167fc640 1 -- 172.21.15.150:0/908947671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72808bdb0 msgr2=0x7fe728153780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.176+0000 7fe7167fc640 1 --2- 172.21.15.150:0/908947671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72808bdb0 0x7fe728153780 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe71c00c3c0 tx=0x7fe71c00c7c0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.176+0000 7f2a8128b640 1 -- 172.21.15.150:0/4073369974 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f2a58005590 con 0x7f2a7c0c40c0 2024-09-17T04:49:07.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.176+0000 7fe7167fc640 1 -- 172.21.15.150:0/908947671 shutdown_connections 2024-09-17T04:49:07.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.176+0000 7fe7167fc640 1 --2- 172.21.15.150:0/908947671 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7281609e0 0x7fe72815b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.176+0000 7fe7167fc640 1 --2- 172.21.15.150:0/908947671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72808bdb0 0x7fe728153780 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.176+0000 7fe7167fc640 1 --2- 172.21.15.150:0/908947671 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72808b880 0x7fe728152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.176+0000 7fe7167fc640 1 -- 172.21.15.150:0/908947671 >> 172.21.15.150:0/908947671 conn(0x7fe72806ce50 msgr2=0x7fe728161a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.176+0000 7fe7167fc640 1 -- 172.21.15.150:0/908947671 shutdown_connections 2024-09-17T04:49:07.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.176+0000 7fe7167fc640 1 -- 172.21.15.150:0/908947671 wait complete. 2024-09-17T04:49:07.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.180+0000 7f1112a61640 1 -- 172.21.15.150:0/3022401607 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f10ec005590 con 0x7f110c0c2e80 2024-09-17T04:49:07.184 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.184+0000 7f2a6b7fe640 1 -- 172.21.15.150:0/4073369974 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v17)=0 v17) ==== 115+0+0 (secure 0 0 0) 0x7f2a7400ad70 con 0x7f2a7c0c40c0 2024-09-17T04:49:07.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.184+0000 7fcdfc52b640 1 -- 172.21.15.150:0/1864869572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7fcddc005590 con 0x7fcdf408b5a0 2024-09-17T04:49:07.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.192+0000 7f1108ff9640 1 -- 172.21.15.150:0/3022401607 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v18)=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7f110c1585e0 con 0x7f110c0c2e80 2024-09-17T04:49:07.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f1112a61640 1 -- 172.21.15.150:0/3022401607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f110c0c2e80 msgr2=0x7f110c149a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f1112a61640 1 --2- 172.21.15.150:0/3022401607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f110c0c2e80 0x7f110c149a80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f110400cbc0 tx=0x7f110400ccb0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f1112a61640 1 -- 172.21.15.150:0/3022401607 shutdown_connections 2024-09-17T04:49:07.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f1112a61640 1 --2- 172.21.15.150:0/3022401607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110c0c7700 0x7f110c1eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f1112a61640 1 --2- 172.21.15.150:0/3022401607 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f110c0c40c0 0x7f110c158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f1112a61640 1 --2- 172.21.15.150:0/3022401607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f110c0c2e80 0x7f110c149a80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f1112a61640 1 -- 172.21.15.150:0/3022401607 >> 172.21.15.150:0/3022401607 conn(0x7f110c0c7be0 msgr2=0x7f110c0ac040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f0f03026640 1 -- 172.21.15.150:0/3853123604 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f0edc005590 con 0x7f0efc074f70 2024-09-17T04:49:07.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f1112a61640 1 -- 172.21.15.150:0/3022401607 shutdown_connections 2024-09-17T04:49:07.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f1112a61640 1 -- 172.21.15.150:0/3022401607 wait complete. 2024-09-17T04:49:07.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f2a8128b640 1 -- 172.21.15.150:0/4073369974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c0c40c0 msgr2=0x7f2a7c165380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f2a8128b640 1 --2- 172.21.15.150:0/4073369974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c0c40c0 0x7f2a7c165380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2a7400cb00 tx=0x7f2a74002120 comp rx=0 tx=0).stop 2024-09-17T04:49:07.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f2a8128b640 1 -- 172.21.15.150:0/4073369974 shutdown_connections 2024-09-17T04:49:07.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f2a8128b640 1 --2- 172.21.15.150:0/4073369974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a7c0c40c0 0x7f2a7c165380 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f2a8128b640 1 --2- 172.21.15.150:0/4073369974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a7c0c7700 0x7f2a7c14bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f2a8128b640 1 --2- 172.21.15.150:0/4073369974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2a7c06f840 0x7f2a7c158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f2a8128b640 1 -- 172.21.15.150:0/4073369974 >> 172.21.15.150:0/4073369974 conn(0x7f2a7c0c7be0 msgr2=0x7f2a7c0c4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7f532dc21640 1 -- 172.21.15.150:0/1900322876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7f5308005590 con 0x7f5328070d60 2024-09-17T04:49:07.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.196+0000 7fcdeaffd640 1 -- 172.21.15.150:0/1864869572 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v19)=0 v19) ==== 125+0+0 (secure 0 0 0) 0x7fcdec005680 con 0x7fcdf408b5a0 2024-09-17T04:49:07.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.204+0000 7f2a8128b640 1 -- 172.21.15.150:0/4073369974 shutdown_connections 2024-09-17T04:49:07.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.204+0000 7f2a8128b640 1 -- 172.21.15.150:0/4073369974 wait complete. 2024-09-17T04:49:07.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.208+0000 7f0ef97fa640 1 -- 172.21.15.150:0/3853123604 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v20)=0 v20) ==== 119+0+0 (secure 0 0 0) 0x7f0ef0010830 con 0x7f0efc074f70 2024-09-17T04:49:07.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.212+0000 7fcdfc52b640 1 -- 172.21.15.150:0/1864869572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdf408b5a0 msgr2=0x7fcdf40c09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.212+0000 7fcdfc52b640 1 --2- 172.21.15.150:0/1864869572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdf408b5a0 0x7fcdf40c09a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00c8a0 tx=0x7fcdec00cd60 comp rx=0 tx=0).stop 2024-09-17T04:49:07.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.212+0000 7fa18645a640 1 -- 172.21.15.150:0/1501746750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=debug_mgr}] v 0) -- 0x7fa15c005860 con 0x7fa1800c7670 2024-09-17T04:49:07.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.212+0000 7fcdfc52b640 1 -- 172.21.15.150:0/1864869572 shutdown_connections 2024-09-17T04:49:07.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.212+0000 7fcdfc52b640 1 --2- 172.21.15.150:0/1864869572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdf40c5dc0 0x7fcdf4201f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.212+0000 7fcdfc52b640 1 --2- 172.21.15.150:0/1864869572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcdf408b5a0 0x7fcdf40c09a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.212+0000 7fcdfc52b640 1 --2- 172.21.15.150:0/1864869572 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcdf4075250 0x7fcdf40c0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.212+0000 7fcdfc52b640 1 -- 172.21.15.150:0/1864869572 >> 172.21.15.150:0/1864869572 conn(0x7fcdf40c7be0 msgr2=0x7fcdf4068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.212+0000 7fcdfc52b640 1 -- 172.21.15.150:0/1864869572 shutdown_connections 2024-09-17T04:49:07.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.212+0000 7fcdfc52b640 1 -- 172.21.15.150:0/1864869572 wait complete. 2024-09-17T04:49:07.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.216+0000 7fcbb329a640 1 -- 172.21.15.150:0/622736408 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7fcbac0707f0 con 0x7fcbac0c7700 2024-09-17T04:49:07.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.216+0000 7f5ab850b640 1 -- 172.21.15.150:0/612241646 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f5a98005590 con 0x7f5ab00c40c0 2024-09-17T04:49:07.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.216+0000 7f0f03026640 1 -- 172.21.15.150:0/3853123604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc074f70 msgr2=0x7f0efc153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.216+0000 7f0f03026640 1 --2- 172.21.15.150:0/3853123604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc074f70 0x7f0efc153270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000d6d0 tx=0x7f0ef000db90 comp rx=0 tx=0).stop 2024-09-17T04:49:07.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.216+0000 7f0f03026640 1 -- 172.21.15.150:0/3853123604 shutdown_connections 2024-09-17T04:49:07.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.216+0000 7f0f03026640 1 --2- 172.21.15.150:0/3853123604 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0efc156010 0x7f0efc0bd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.216+0000 7f0f03026640 1 --2- 172.21.15.150:0/3853123604 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0efc156840 0x7f0efc155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.216+0000 7f0f03026640 1 --2- 172.21.15.150:0/3853123604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0efc074f70 0x7f0efc153270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.216+0000 7f0f03026640 1 -- 172.21.15.150:0/3853123604 >> 172.21.15.150:0/3853123604 conn(0x7f0efc0c4e30 msgr2=0x7f0efc1546b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.216+0000 7f0f03026640 1 -- 172.21.15.150:0/3853123604 shutdown_connections 2024-09-17T04:49:07.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.216+0000 7f0f03026640 1 -- 172.21.15.150:0/3853123604 wait complete. 2024-09-17T04:49:07.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.220+0000 7f531d7fa640 1 -- 172.21.15.150:0/1900322876 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v21)=0 v21) ==== 118+0+0 (secure 0 0 0) 0x7f5324003d90 con 0x7f5328070d60 2024-09-17T04:49:07.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.224+0000 7fe5f8ff9640 1 -- 172.21.15.150:0/4260560417 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7fe604071150 con 0x7fe6040c34c0 2024-09-17T04:49:07.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.224+0000 7fa17cff9640 1 -- 172.21.15.150:0/1501746750 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mgr}]=0 v22)=0 v22) ==== 108+0+0 (secure 0 0 0) 0x7fa174003600 con 0x7fa1800c7670 2024-09-17T04:49:07.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.228+0000 7fa18645a640 1 -- 172.21.15.150:0/1501746750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1800c7670 msgr2=0x7fa180167af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.228+0000 7fa18645a640 1 --2- 172.21.15.150:0/1501746750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1800c7670 0x7fa180167af0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa17400bb20 tx=0x7fa1740071c0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.228+0000 7fa18645a640 1 -- 172.21.15.150:0/1501746750 shutdown_connections 2024-09-17T04:49:07.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.228+0000 7fa18645a640 1 --2- 172.21.15.150:0/1501746750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1800c7670 0x7fa180167af0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.228+0000 7fa18645a640 1 --2- 172.21.15.150:0/1501746750 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa18008b5a0 0x7fa18020d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.228+0000 7fa18645a640 1 --2- 172.21.15.150:0/1501746750 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa180075250 0x7fa180169cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.228+0000 7fa18645a640 1 -- 172.21.15.150:0/1501746750 >> 172.21.15.150:0/1501746750 conn(0x7fa1800c7be0 msgr2=0x7fa1800c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.228+0000 7fa18645a640 1 -- 172.21.15.150:0/1501746750 shutdown_connections 2024-09-17T04:49:07.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.228+0000 7fa18645a640 1 -- 172.21.15.150:0/1501746750 wait complete. 2024-09-17T04:49:07.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.232+0000 7fcba1ffb640 1 -- 172.21.15.150:0/622736408 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v23)=0 v23) ==== 111+0+0 (secure 0 0 0) 0x7fcba8016350 con 0x7fcbac0c7700 2024-09-17T04:49:07.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.236+0000 7fcbb329a640 1 -- 172.21.15.150:0/622736408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbac0c7700 msgr2=0x7fcbac158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.236+0000 7fcbb329a640 1 --2- 172.21.15.150:0/622736408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbac0c7700 0x7fcbac158a90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcbac071380 tx=0x7fcba80a9460 comp rx=0 tx=0).stop 2024-09-17T04:49:07.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.236+0000 7fcbb329a640 1 -- 172.21.15.150:0/622736408 shutdown_connections 2024-09-17T04:49:07.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.236+0000 7fcbb329a640 1 --2- 172.21.15.150:0/622736408 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbac0c7700 0x7fcbac158a90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.236+0000 7fcbb329a640 1 --2- 172.21.15.150:0/622736408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbac0c40c0 0x7fcbac151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.236+0000 7fcbb329a640 1 --2- 172.21.15.150:0/622736408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbac0c2e80 0x7fcbac1664d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.236+0000 7fcbb329a640 1 -- 172.21.15.150:0/622736408 >> 172.21.15.150:0/622736408 conn(0x7fcbac0c7be0 msgr2=0x7fcbac0c4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.236+0000 7fcbb329a640 1 -- 172.21.15.150:0/622736408 shutdown_connections 2024-09-17T04:49:07.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.236+0000 7fcbb329a640 1 -- 172.21.15.150:0/622736408 wait complete. 2024-09-17T04:49:07.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.240+0000 7f0f1d43f640 1 -- 172.21.15.150:0/1177860568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f0efc005590 con 0x7f0f180c3ac0 2024-09-17T04:49:07.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.248+0000 7f532dc21640 1 -- 172.21.15.150:0/1900322876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5328070d60 msgr2=0x7f53280a9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.248+0000 7f532dc21640 1 --2- 172.21.15.150:0/1900322876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5328070d60 0x7f53280a9ad0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f532400e770 tx=0x7f532400ec30 comp rx=0 tx=0).stop 2024-09-17T04:49:07.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.248+0000 7f532dc21640 1 -- 172.21.15.150:0/1900322876 shutdown_connections 2024-09-17T04:49:07.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.248+0000 7f532dc21640 1 --2- 172.21.15.150:0/1900322876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53280ad310 0x7f5328146af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.248+0000 7f532dc21640 1 --2- 172.21.15.150:0/1900322876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5328071740 0x7f5328165b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.248+0000 7f532dc21640 1 --2- 172.21.15.150:0/1900322876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5328070d60 0x7f53280a9ad0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.248+0000 7f532dc21640 1 -- 172.21.15.150:0/1900322876 >> 172.21.15.150:0/1900322876 conn(0x7f532806d250 msgr2=0x7f532814d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.248+0000 7f532dc21640 1 -- 172.21.15.150:0/1900322876 shutdown_connections 2024-09-17T04:49:07.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.248+0000 7f532dc21640 1 -- 172.21.15.150:0/1900322876 wait complete. 2024-09-17T04:49:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.252+0000 7f5aa6ffd640 1 -- 172.21.15.150:0/612241646 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v24)=0 v24) ==== 120+0+0 (secure 0 0 0) 0x7f5aa80279e0 con 0x7f5ab00c40c0 2024-09-17T04:49:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.252+0000 7f5ab850b640 1 -- 172.21.15.150:0/612241646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ab00c40c0 msgr2=0x7f5ab0156560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.252+0000 7f5ab850b640 1 --2- 172.21.15.150:0/612241646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ab00c40c0 0x7f5ab0156560 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5ab00c8b20 tx=0x7f5aa801a860 comp rx=0 tx=0).stop 2024-09-17T04:49:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.252+0000 7f5ab850b640 1 -- 172.21.15.150:0/612241646 shutdown_connections 2024-09-17T04:49:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.252+0000 7f5ab850b640 1 --2- 172.21.15.150:0/612241646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ab00c7700 0x7f5ab0156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.252+0000 7f5ab850b640 1 --2- 172.21.15.150:0/612241646 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ab00c40c0 0x7f5ab0156560 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.252+0000 7f5ab850b640 1 --2- 172.21.15.150:0/612241646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ab00c2e80 0x7f5ab00a7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.252+0000 7f5ab850b640 1 -- 172.21.15.150:0/612241646 >> 172.21.15.150:0/612241646 conn(0x7f5ab00c7be0 msgr2=0x7f5ab00c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.256+0000 7f5ab850b640 1 -- 172.21.15.150:0/612241646 shutdown_connections 2024-09-17T04:49:07.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.256+0000 7f5ab850b640 1 -- 172.21.15.150:0/612241646 wait complete. 2024-09-17T04:49:07.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.252+0000 7fe5faffd640 1 -- 172.21.15.150:0/4260560417 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v25)=0 v25) ==== 121+0+0 (secure 0 0 0) 0x7fe5fc0169b0 con 0x7fe6040c34c0 2024-09-17T04:49:07.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.264+0000 7f0f077fe640 1 -- 172.21.15.150:0/1177860568 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v26)=0 v26) ==== 121+0+0 (secure 0 0 0) 0x7f0f08003dd0 con 0x7f0f180c3ac0 2024-09-17T04:49:07.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7f0f1d43f640 1 -- 172.21.15.150:0/1177860568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f180c3ac0 msgr2=0x7f0f180a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7f0f1d43f640 1 --2- 172.21.15.150:0/1177860568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f180c3ac0 0x7f0f180a5000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0f0800c430 tx=0x7f0f0800c8f0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7fe5f8ff9640 1 -- 172.21.15.150:0/4260560417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6040c34c0 msgr2=0x7fe604200480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7fe5f8ff9640 1 --2- 172.21.15.150:0/4260560417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6040c34c0 0x7fe604200480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe604153bc0 tx=0x7fe5fc004a50 comp rx=0 tx=0).stop 2024-09-17T04:49:07.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7fe5f8ff9640 1 -- 172.21.15.150:0/4260560417 shutdown_connections 2024-09-17T04:49:07.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7fe5f8ff9640 1 --2- 172.21.15.150:0/4260560417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6042009c0 0x7fe6040c0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7fe5f8ff9640 1 --2- 172.21.15.150:0/4260560417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6040c34c0 0x7fe604200480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7fe5f8ff9640 1 --2- 172.21.15.150:0/4260560417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe604073150 0x7fe60414ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7fe5f8ff9640 1 -- 172.21.15.150:0/4260560417 >> 172.21.15.150:0/4260560417 conn(0x7fe60406ce50 msgr2=0x7fe60415c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7f0f1d43f640 1 -- 172.21.15.150:0/1177860568 shutdown_connections 2024-09-17T04:49:07.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7f0f1d43f640 1 --2- 172.21.15.150:0/1177860568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f181501c0 0x7f0f181f7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7f0f1d43f640 1 --2- 172.21.15.150:0/1177860568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f180c3ac0 0x7f0f180a5000 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7f0f1d43f640 1 --2- 172.21.15.150:0/1177860568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f18071310 0x7f0f1814d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7f0f1d43f640 1 -- 172.21.15.150:0/1177860568 >> 172.21.15.150:0/1177860568 conn(0x7f0f180c4a00 msgr2=0x7f0f18073660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7fe5f8ff9640 1 -- 172.21.15.150:0/4260560417 shutdown_connections 2024-09-17T04:49:07.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7fe5f8ff9640 1 -- 172.21.15.150:0/4260560417 wait complete. 2024-09-17T04:49:07.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7f0f1d43f640 1 -- 172.21.15.150:0/1177860568 shutdown_connections 2024-09-17T04:49:07.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.268+0000 7f0f1d43f640 1 -- 172.21.15.150:0/1177860568 wait complete. 2024-09-17T04:49:07.282 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.282 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.282 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.284+0000 7f56b6e92640 1 -- 172.21.15.150:0/56425817 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f5698005590 con 0x7f56b0156680 2024-09-17T04:49:07.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.284+0000 7f56a67fc640 1 -- 172.21.15.150:0/56425817 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v27)=0 v27) ==== 121+0+0 (secure 0 0 0) 0x7f56a00193b0 con 0x7f56b0156680 2024-09-17T04:49:07.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.288+0000 7f56b6e92640 1 -- 172.21.15.150:0/56425817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56b0156680 msgr2=0x7f56b00c5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.288+0000 7f56b6e92640 1 --2- 172.21.15.150:0/56425817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56b0156680 0x7f56b00c5820 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f56b00738c0 tx=0x7f56a000b870 comp rx=0 tx=0).stop 2024-09-17T04:49:07.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.288+0000 7f56b6e92640 1 -- 172.21.15.150:0/56425817 shutdown_connections 2024-09-17T04:49:07.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.288+0000 7f56b6e92640 1 --2- 172.21.15.150:0/56425817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56b00c4890 0x7f56b00c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.288+0000 7f56b6e92640 1 --2- 172.21.15.150:0/56425817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56b0156680 0x7f56b00c5820 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.288+0000 7f56b6e92640 1 --2- 172.21.15.150:0/56425817 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56b00c4480 0x7f56b00c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.288+0000 7f56b6e92640 1 -- 172.21.15.150:0/56425817 >> 172.21.15.150:0/56425817 conn(0x7f56b006ce50 msgr2=0x7f56b013e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.288+0000 7f56b6e92640 1 -- 172.21.15.150:0/56425817 shutdown_connections 2024-09-17T04:49:07.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.288+0000 7f56b6e92640 1 -- 172.21.15.150:0/56425817 wait complete. 2024-09-17T04:49:07.302 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.303 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.311 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.311 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.311 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'client_mount_timeout', '30'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.311 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'debug_client', '20'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.311 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'debug_mgr', '20'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.312 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'debug_ms', '1'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.312 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.312 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.312 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi150') 2024-09-17T04:49:07.312 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config dump 2024-09-17T04:49:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.488+0000 7f33fd393640 1 -- 172.21.15.150:0/3605429410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33e0004d50 msgr2=0x7f33e0005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.488+0000 7f33fd393640 1 --2- 172.21.15.150:0/3605429410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33e0004d50 0x7f33e0005160 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f33f80719d0 tx=0x7f33f001d300 comp rx=0 tx=0).stop 2024-09-17T04:49:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.488+0000 7f33f77fe640 1 -- 172.21.15.150:0/3605429410 reap_dead start 2024-09-17T04:49:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.488+0000 7f33fd393640 1 -- 172.21.15.150:0/3605429410 shutdown_connections 2024-09-17T04:49:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.488+0000 7f33fd393640 1 -- 172.21.15.150:0/3605429410 >> 172.21.15.150:0/3605429410 conn(0x7f33f806e360 msgr2=0x7f33f806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.488+0000 7f33fd393640 1 -- 172.21.15.150:0/3605429410 shutdown_connections 2024-09-17T04:49:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.488+0000 7f33fd393640 1 -- 172.21.15.150:0/3605429410 wait complete. 2024-09-17T04:49:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.488+0000 7f33fd393640 1 Processor -- start 2024-09-17T04:49:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.488+0000 7f33fd393640 1 -- start start 2024-09-17T04:49:07.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33fd393640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e0004d50 0x7f33f80bdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:07.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33fd393640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33f80ae650 0x7f33f80beef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:07.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33fd393640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f81ebd30 0x7f33f814fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:07.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33fd393640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f33f80c47e0 con 0x7f33f81ebd30 2024-09-17T04:49:07.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33fd393640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f33f80c4660 con 0x7f33e0004d50 2024-09-17T04:49:07.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33fd393640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f33f8070fa0 con 0x7f33f80ae650 2024-09-17T04:49:07.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33f77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f81ebd30 0x7f33f814fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33f77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f81ebd30 0x7f33f814fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54716/0 (socket says 172.21.15.150:54716) 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33f77fe640 1 -- 172.21.15.150:0/1037852887 learned_addr learned my addr 172.21.15.150:0/1037852887 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33f67fc640 1 --2- 172.21.15.150:0/1037852887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33f80ae650 0x7f33f80beef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33f6ffd640 1 --2- 172.21.15.150:0/1037852887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e0004d50 0x7f33f80bdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33f77fe640 1 -- 172.21.15.150:0/1037852887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33f80ae650 msgr2=0x7f33f80beef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33f77fe640 1 --2- 172.21.15.150:0/1037852887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33f80ae650 0x7f33f80beef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33f77fe640 1 -- 172.21.15.150:0/1037852887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e0004d50 msgr2=0x7f33f80bdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33f77fe640 1 --2- 172.21.15.150:0/1037852887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e0004d50 0x7f33f80bdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33f77fe640 1 -- 172.21.15.150:0/1037852887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33f80acbf0 con 0x7f33f81ebd30 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33f77fe640 1 --2- 172.21.15.150:0/1037852887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f81ebd30 0x7f33f814fd40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f33f0000f90 tx=0x7f33f0004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33df7fe640 1 -- 172.21.15.150:0/1037852887 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f00bd070 con 0x7f33f81ebd30 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33df7fe640 1 -- 172.21.15.150:0/1037852887 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f33f0001ca0 con 0x7f33f81ebd30 2024-09-17T04:49:07.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33df7fe640 1 -- 172.21.15.150:0/1037852887 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f00b2040 con 0x7f33f81ebd30 2024-09-17T04:49:07.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33fd393640 1 -- 172.21.15.150:0/1037852887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33f80bf430 con 0x7f33f81ebd30 2024-09-17T04:49:07.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33fd393640 1 -- 172.21.15.150:0/1037852887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33f81ec680 con 0x7f33f81ebd30 2024-09-17T04:49:07.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33df7fe640 1 -- 172.21.15.150:0/1037852887 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f33f0003be0 con 0x7f33f81ebd30 2024-09-17T04:49:07.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33df7fe640 1 -- 172.21.15.150:0/1037852887 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f33f00c14e0 con 0x7f33f81ebd30 2024-09-17T04:49:07.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.492+0000 7f33fd393640 1 -- 172.21.15.150:0/1037852887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33f8072db0 con 0x7f33f81ebd30 2024-09-17T04:49:07.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.496+0000 7f33df7fe640 1 -- 172.21.15.150:0/1037852887 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f33f00c6070 con 0x7f33f81ebd30 2024-09-17T04:49:07.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.552+0000 7f33fd393640 1 -- 172.21.15.150:0/1037852887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f33f80ada20 con 0x7f33f81ebd30 2024-09-17T04:49:07.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.552+0000 7f33df7fe640 1 -- 172.21.15.150:0/1037852887 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v27) ==== 59+0+1782 (secure 0 0 0) 0x7f33f80bf430 con 0x7f33f81ebd30 2024-09-17T04:49:07.551 INFO:teuthology.orchestra.run.smithi150.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T04:49:07.551 INFO:teuthology.orchestra.run.smithi150.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T04:49:07.551 INFO:teuthology.orchestra.run.smithi150.stdout:mgr advanced client_mount_timeout 30 2024-09-17T04:49:07.551 INFO:teuthology.orchestra.run.smithi150.stdout:mgr advanced debug_client 20/20 2024-09-17T04:49:07.551 INFO:teuthology.orchestra.run.smithi150.stdout:mgr advanced debug_mgr 20/20 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mgr advanced debug_ms 1/1 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mds advanced debug_mds 20/20 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mds advanced debug_ms 2/2 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mds advanced mds_bal_fragment_size_max 10000 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mds advanced mds_bal_merge_size 5 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mds advanced mds_bal_split_bits 3 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mds advanced mds_bal_split_size 100 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mds dev mds_debug_frag true 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mds dev mds_debug_scatterstat true 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T04:49:07.552 INFO:teuthology.orchestra.run.smithi150.stdout:mds dev mds_verify_scatter true 2024-09-17T04:49:07.553 INFO:teuthology.orchestra.run.smithi150.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T04:49:07.553 INFO:teuthology.orchestra.run.smithi150.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T04:49:07.553 INFO:teuthology.orchestra.run.smithi150.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T04:49:07.553 INFO:teuthology.orchestra.run.smithi150.stdout:client advanced client_mount_timeout 600 2024-09-17T04:49:07.553 INFO:teuthology.orchestra.run.smithi150.stdout:client advanced debug_client 20/20 2024-09-17T04:49:07.553 INFO:teuthology.orchestra.run.smithi150.stdout:client advanced debug_ms 1/1 2024-09-17T04:49:07.553 INFO:teuthology.orchestra.run.smithi150.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T04:49:07.553 INFO:teuthology.orchestra.run.smithi150.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T04:49:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.552+0000 7f33fd393640 1 -- 172.21.15.150:0/1037852887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f81ebd30 msgr2=0x7f33f814fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.552+0000 7f33fd393640 1 --2- 172.21.15.150:0/1037852887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f81ebd30 0x7f33f814fd40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f33f0000f90 tx=0x7f33f0004ef0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.552+0000 7f33fd393640 1 -- 172.21.15.150:0/1037852887 shutdown_connections 2024-09-17T04:49:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.552+0000 7f33fd393640 1 --2- 172.21.15.150:0/1037852887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f81ebd30 0x7f33f814fd40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.552+0000 7f33fd393640 1 --2- 172.21.15.150:0/1037852887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33f80ae650 0x7f33f80beef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.552+0000 7f33fd393640 1 --2- 172.21.15.150:0/1037852887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33e0004d50 0x7f33f80bdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.552+0000 7f33fd393640 1 -- 172.21.15.150:0/1037852887 >> 172.21.15.150:0/1037852887 conn(0x7f33f806e360 msgr2=0x7f33f80c68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.552+0000 7f33fd393640 1 -- 172.21.15.150:0/1037852887 shutdown_connections 2024-09-17T04:49:07.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.552+0000 7f33fd393640 1 -- 172.21.15.150:0/1037852887 wait complete. 2024-09-17T04:49:07.571 INFO:tasks.ceph:Setting crush tunables to default 2024-09-17T04:49:07.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd crush tunables default 2024-09-17T04:49:07.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.720+0000 7fa7058d3640 1 -- 172.21.15.150:0/609849688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7000ab9d0 msgr2=0x7fa70014da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.720+0000 7fa7058d3640 1 --2- 172.21.15.150:0/609849688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7000ab9d0 0x7fa70014da10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009840 tx=0x7fa6f001c700 comp rx=0 tx=0).stop 2024-09-17T04:49:07.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.720+0000 7fa7058d3640 1 -- 172.21.15.150:0/609849688 shutdown_connections 2024-09-17T04:49:07.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.720+0000 7fa7058d3640 1 --2- 172.21.15.150:0/609849688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7000ab9d0 0x7fa70014da10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.720+0000 7fa7058d3640 1 -- 172.21.15.150:0/609849688 >> 172.21.15.150:0/609849688 conn(0x7fa700070b50 msgr2=0x7fa700070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.720+0000 7fa7058d3640 1 -- 172.21.15.150:0/609849688 shutdown_connections 2024-09-17T04:49:07.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.720+0000 7fa7058d3640 1 -- 172.21.15.150:0/609849688 wait complete. 2024-09-17T04:49:07.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa7058d3640 1 Processor -- start 2024-09-17T04:49:07.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa7058d3640 1 -- start start 2024-09-17T04:49:07.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa7058d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa70006df60 0x7fa7000ac270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:07.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa7058d3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa70006e370 0x7fa7000c1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa7058d3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7000ab9d0 0x7fa700146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa7058d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa7001618c0 con 0x7fa70006df60 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa7058d3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa700069e00 con 0x7fa7000ab9d0 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa7058d3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa700161a40 con 0x7fa70006e370 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6feffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa70006df60 0x7fa7000ac270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6feffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa70006df60 0x7fa7000ac270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54736/0 (socket says 172.21.15.150:54736) 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6feffd640 1 -- 172.21.15.150:0/1556849462 learned_addr learned my addr 172.21.15.150:0/1556849462 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6f7fff640 1 --2- 172.21.15.150:0/1556849462 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa70006e370 0x7fa7000c1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6ff7fe640 1 --2- 172.21.15.150:0/1556849462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7000ab9d0 0x7fa700146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6feffd640 1 -- 172.21.15.150:0/1556849462 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa70006e370 msgr2=0x7fa7000c1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6feffd640 1 --2- 172.21.15.150:0/1556849462 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa70006e370 0x7fa7000c1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6feffd640 1 -- 172.21.15.150:0/1556849462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7000ab9d0 msgr2=0x7fa700146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6feffd640 1 --2- 172.21.15.150:0/1556849462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7000ab9d0 0x7fa700146c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6feffd640 1 -- 172.21.15.150:0/1556849462 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7001f0c40 con 0x7fa70006df60 2024-09-17T04:49:07.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6ff7fe640 1 --2- 172.21.15.150:0/1556849462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7000ab9d0 0x7fa700146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:07.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6feffd640 1 --2- 172.21.15.150:0/1556849462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa70006df60 0x7fa7000ac270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009790 tx=0x7fa6f00a86e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:07.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6fcff9640 1 -- 172.21.15.150:0/1556849462 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f00a8a70 con 0x7fa70006df60 2024-09-17T04:49:07.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6fcff9640 1 -- 172.21.15.150:0/1556849462 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa6f00a8c10 con 0x7fa70006df60 2024-09-17T04:49:07.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6fcff9640 1 -- 172.21.15.150:0/1556849462 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0005140 con 0x7fa70006df60 2024-09-17T04:49:07.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa7058d3640 1 -- 172.21.15.150:0/1556849462 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7001f0f20 con 0x7fa70006df60 2024-09-17T04:49:07.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa7058d3640 1 -- 172.21.15.150:0/1556849462 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7000adb30 con 0x7fa70006df60 2024-09-17T04:49:07.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6fcff9640 1 -- 172.21.15.150:0/1556849462 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa6f00052e0 con 0x7fa70006df60 2024-09-17T04:49:07.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa6fcff9640 1 -- 172.21.15.150:0/1556849462 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa6f00aa110 con 0x7fa70006df60 2024-09-17T04:49:07.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.724+0000 7fa7058d3640 1 -- 172.21.15.150:0/1556849462 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa70014da10 con 0x7fa70006df60 2024-09-17T04:49:07.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.728+0000 7fa6fcff9640 1 -- 172.21.15.150:0/1556849462 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fa6f00c1030 con 0x7fa70006df60 2024-09-17T04:49:07.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.788+0000 7fa7058d3640 1 -- 172.21.15.150:0/1556849462 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fa700075060 con 0x7fa70006df60 2024-09-17T04:49:07.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.792+0000 7fa6fcff9640 1 -- 172.21.15.150:0/1556849462 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fa6f00ac030 con 0x7fa70006df60 2024-09-17T04:49:07.791 INFO:teuthology.orchestra.run.smithi150.stderr:adjusted tunables profile to default 2024-09-17T04:49:07.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.792+0000 7fa7058d3640 1 -- 172.21.15.150:0/1556849462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa70006df60 msgr2=0x7fa7000ac270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:07.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.792+0000 7fa7058d3640 1 --2- 172.21.15.150:0/1556849462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa70006df60 0x7fa7000ac270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009790 tx=0x7fa6f00a86e0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.792+0000 7fa7058d3640 1 -- 172.21.15.150:0/1556849462 shutdown_connections 2024-09-17T04:49:07.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.792+0000 7fa7058d3640 1 --2- 172.21.15.150:0/1556849462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7000ab9d0 0x7fa700146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.792+0000 7fa7058d3640 1 --2- 172.21.15.150:0/1556849462 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa70006e370 0x7fa7000c1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.792+0000 7fa7058d3640 1 --2- 172.21.15.150:0/1556849462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa70006df60 0x7fa7000ac270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:07.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.792+0000 7fa7058d3640 1 -- 172.21.15.150:0/1556849462 >> 172.21.15.150:0/1556849462 conn(0x7fa700070b50 msgr2=0x7fa7000a9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:07.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.792+0000 7fa7058d3640 1 -- 172.21.15.150:0/1556849462 shutdown_connections 2024-09-17T04:49:07.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:07.792+0000 7fa7058d3640 1 -- 172.21.15.150:0/1556849462 wait complete. 2024-09-17T04:49:07.810 INFO:tasks.ceph:check_enable_crimson: False 2024-09-17T04:49:07.811 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-17T04:49:07.811 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:49:07.811 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-17T04:49:07.823 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:49:07.823 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-17T04:49:07.874 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:49:07.874 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-17T04:49:07.927 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:49:07.927 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-17T04:49:07.980 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:49:07.980 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-17T04:49:07.994 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:49:07.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-17T04:49:08.047 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:49:08.047 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-17T04:49:08.104 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:49:08.104 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-17T04:49:08.158 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new e04420c7-04e9-46b3-93e4-f85feeeeb97a 0 2024-09-17T04:49:08.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.354+0000 7ff8e3247640 1 -- 172.21.15.176:0/1541450976 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8dc0c5680 msgr2=0x7ff8dc0c7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:08.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.354+0000 7ff8e3247640 1 --2- 172.21.15.176:0/1541450976 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8dc0c5680 0x7ff8dc0c7d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009900 tx=0x7ff8cc01c8e0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.354+0000 7ff8e3247640 1 -- 172.21.15.176:0/1541450976 shutdown_connections 2024-09-17T04:49:08.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.354+0000 7ff8e3247640 1 --2- 172.21.15.176:0/1541450976 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8dc0c5680 0x7ff8dc0c7d60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.354+0000 7ff8e3247640 1 -- 172.21.15.176:0/1541450976 >> 172.21.15.176:0/1541450976 conn(0x7ff8dc0ca460 msgr2=0x7ff8dc0c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:08.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.354+0000 7ff8e3247640 1 -- 172.21.15.176:0/1541450976 shutdown_connections 2024-09-17T04:49:08.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.354+0000 7ff8e3247640 1 -- 172.21.15.176:0/1541450976 wait complete. 2024-09-17T04:49:08.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.354+0000 7ff8e3247640 1 Processor -- start 2024-09-17T04:49:08.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.354+0000 7ff8e3247640 1 -- start start 2024-09-17T04:49:08.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e3247640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8dc071670 0x7ff8dc13bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:08.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e3247640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8dc0c5680 0x7ff8dc145ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:08.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e3247640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8dc157c00 0x7ff8dc141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:08.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e3247640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc0c6740 con 0x7ff8dc071670 2024-09-17T04:49:08.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e3247640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc0c65c0 con 0x7ff8dc0c5680 2024-09-17T04:49:08.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e3247640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff8dc072180 con 0x7ff8dc157c00 2024-09-17T04:49:08.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8dbfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8dc0c5680 0x7ff8dc145ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e17e4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8dc157c00 0x7ff8dc141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8dbfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8dc0c5680 0x7ff8dc145ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:58548/0 (socket says 172.21.15.176:58548) 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8dbfff640 1 -- 172.21.15.176:0/3988130091 learned_addr learned my addr 172.21.15.176:0/3988130091 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e0fe3640 1 --2- 172.21.15.176:0/3988130091 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8dc071670 0x7ff8dc13bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e17e4640 1 -- 172.21.15.176:0/3988130091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8dc0c5680 msgr2=0x7ff8dc145ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e17e4640 1 --2- 172.21.15.176:0/3988130091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8dc0c5680 0x7ff8dc145ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e17e4640 1 -- 172.21.15.176:0/3988130091 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8dc071670 msgr2=0x7ff8dc13bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e17e4640 1 --2- 172.21.15.176:0/3988130091 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8dc071670 0x7ff8dc13bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e17e4640 1 -- 172.21.15.176:0/3988130091 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc1463e0 con 0x7ff8dc157c00 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8dbfff640 1 --2- 172.21.15.176:0/3988130091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8dc0c5680 0x7ff8dc145ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e0fe3640 1 --2- 172.21.15.176:0/3988130091 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8dc071670 0x7ff8dc13bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e17e4640 1 --2- 172.21.15.176:0/3988130091 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8dc157c00 0x7ff8dc141a20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0c76c0 tx=0x7ff8d000b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:08.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8d97fa640 1 -- 172.21.15.176:0/3988130091 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0015020 con 0x7ff8dc157c00 2024-09-17T04:49:08.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8d97fa640 1 -- 172.21.15.176:0/3988130091 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8d0011920 con 0x7ff8dc157c00 2024-09-17T04:49:08.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e3247640 1 -- 172.21.15.176:0/3988130091 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8dc13eb70 con 0x7ff8dc157c00 2024-09-17T04:49:08.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8d97fa640 1 -- 172.21.15.176:0/3988130091 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d0011db0 con 0x7ff8dc157c00 2024-09-17T04:49:08.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e3247640 1 -- 172.21.15.176:0/3988130091 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8dc143ae0 con 0x7ff8dc157c00 2024-09-17T04:49:08.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8d97fa640 1 -- 172.21.15.176:0/3988130091 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff8d0018020 con 0x7ff8dc157c00 2024-09-17T04:49:08.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8d97fa640 1 -- 172.21.15.176:0/3988130091 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7ff8d001a530 con 0x7ff8dc157c00 2024-09-17T04:49:08.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.358+0000 7ff8e3247640 1 -- 172.21.15.176:0/3988130091 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8dc154800 con 0x7ff8dc157c00 2024-09-17T04:49:08.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.362+0000 7ff8d97fa640 1 -- 172.21.15.176:0/3988130091 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7ff8dc154800 con 0x7ff8dc157c00 2024-09-17T04:49:08.434 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.430+0000 7ff8e3247640 1 -- 172.21.15.176:0/3988130091 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e04420c7-04e9-46b3-93e4-f85feeeeb97a", "id": 0} v 0) -- 0x7ff8dc152860 con 0x7ff8dc157c00 2024-09-17T04:49:08.443 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.442+0000 7ff8d97fa640 1 -- 172.21.15.176:0/3988130091 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e04420c7-04e9-46b3-93e4-f85feeeeb97a", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7ff8d001f030 con 0x7ff8dc157c00 2024-09-17T04:49:08.443 INFO:teuthology.orchestra.run.smithi176.stdout:0 2024-09-17T04:49:08.445 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.442+0000 7ff8e3247640 1 -- 172.21.15.176:0/3988130091 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8dc157c00 msgr2=0x7ff8dc141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:08.445 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.442+0000 7ff8e3247640 1 --2- 172.21.15.176:0/3988130091 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8dc157c00 0x7ff8dc141a20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0c76c0 tx=0x7ff8d000b870 comp rx=0 tx=0).stop 2024-09-17T04:49:08.445 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.442+0000 7ff8e3247640 1 -- 172.21.15.176:0/3988130091 shutdown_connections 2024-09-17T04:49:08.445 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.442+0000 7ff8e3247640 1 --2- 172.21.15.176:0/3988130091 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8dc0c5680 0x7ff8dc145ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.445 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.442+0000 7ff8e3247640 1 --2- 172.21.15.176:0/3988130091 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8dc157c00 0x7ff8dc141a20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.445 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.442+0000 7ff8e3247640 1 --2- 172.21.15.176:0/3988130091 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8dc071670 0x7ff8dc13bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.445 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.442+0000 7ff8e3247640 1 -- 172.21.15.176:0/3988130091 >> 172.21.15.176:0/3988130091 conn(0x7ff8dc0ca460 msgr2=0x7ff8dc149e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:08.445 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.442+0000 7ff8e3247640 1 -- 172.21.15.176:0/3988130091 shutdown_connections 2024-09-17T04:49:08.445 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.442+0000 7ff8e3247640 1 -- 172.21.15.176:0/3988130091 wait complete. 2024-09-17T04:49:08.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 4f065472-fc2f-4355-9e30-3ca9a6615ba6 1 2024-09-17T04:49:08.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.626+0000 7f16f0a2b640 1 -- 172.21.15.176:0/2329538748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16ec156c50 msgr2=0x7f16ec1659d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:08.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.626+0000 7f16f0a2b640 1 --2- 172.21.15.176:0/2329538748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16ec156c50 0x7f16ec1659d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f16dc009900 tx=0x7f16dc01c7f0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.626+0000 7f16f0a2b640 1 -- 172.21.15.176:0/2329538748 shutdown_connections 2024-09-17T04:49:08.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.626+0000 7f16f0a2b640 1 --2- 172.21.15.176:0/2329538748 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16ec156c50 0x7f16ec1659d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.626+0000 7f16f0a2b640 1 --2- 172.21.15.176:0/2329538748 >> v2:172.21.15.150:3300/0 conn(0x7f16ec074f70 0x7f16ec0734d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.626+0000 7f16f0a2b640 1 -- 172.21.15.176:0/2329538748 >> 172.21.15.176:0/2329538748 conn(0x7f16ec0c4e30 msgr2=0x7f16ec06ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:08.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.626+0000 7f16f0a2b640 1 -- 172.21.15.176:0/2329538748 shutdown_connections 2024-09-17T04:49:08.632 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.626+0000 7f16f0a2b640 1 -- 172.21.15.176:0/2329538748 wait complete. 2024-09-17T04:49:08.632 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.630+0000 7f16f0a2b640 1 Processor -- start 2024-09-17T04:49:08.635 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.630+0000 7f16f0a2b640 1 -- start start 2024-09-17T04:49:08.635 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16f0a2b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16ec074f70 0x7f16ec157c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:08.635 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16f0a2b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ec156840 0x7f16ec200450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:08.636 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16f0a2b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16ec156c50 0x7f16ec1f9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:08.636 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16f0a2b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec06ddf0 con 0x7f16ec156840 2024-09-17T04:49:08.636 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16f0a2b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec06dc70 con 0x7f16ec156c50 2024-09-17T04:49:08.636 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16f0a2b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f16ec071c50 con 0x7f16ec074f70 2024-09-17T04:49:08.637 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16ea59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16ec074f70 0x7f16ec157c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:08.637 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16ea59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16ec074f70 0x7f16ec157c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:56674/0 (socket says 172.21.15.176:56674) 2024-09-17T04:49:08.637 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16ea59c640 1 -- 172.21.15.176:0/3730487635 learned_addr learned my addr 172.21.15.176:0/3730487635 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-17T04:49:08.637 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16e9d9b640 1 --2- 172.21.15.176:0/3730487635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ec156840 0x7f16ec200450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:08.638 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16ead9d640 1 --2- 172.21.15.176:0/3730487635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16ec156c50 0x7f16ec1f9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:08.638 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16ead9d640 1 -- 172.21.15.176:0/3730487635 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16ec074f70 msgr2=0x7f16ec157c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:08.638 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16ead9d640 1 --2- 172.21.15.176:0/3730487635 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16ec074f70 0x7f16ec157c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.638 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16ead9d640 1 -- 172.21.15.176:0/3730487635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ec156840 msgr2=0x7f16ec200450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:08.638 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16ead9d640 1 --2- 172.21.15.176:0/3730487635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ec156840 0x7f16ec200450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.638 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16ead9d640 1 -- 172.21.15.176:0/3730487635 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16ec151060 con 0x7f16ec156c50 2024-09-17T04:49:08.638 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16e9d9b640 1 --2- 172.21.15.176:0/3730487635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ec156840 0x7f16ec200450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:08.638 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16ea59c640 1 --2- 172.21.15.176:0/3730487635 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16ec074f70 0x7f16ec157c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:08.638 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16ead9d640 1 --2- 172.21.15.176:0/3730487635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16ec156c50 0x7f16ec1f9f80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f16dc0ac000 tx=0x7f16dc0a6780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:08.639 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16daffd640 1 -- 172.21.15.176:0/3730487635 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc004400 con 0x7f16ec156c50 2024-09-17T04:49:08.639 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16f0a2b640 1 -- 172.21.15.176:0/3730487635 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16ec200d30 con 0x7f16ec156c50 2024-09-17T04:49:08.639 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16f0a2b640 1 -- 172.21.15.176:0/3730487635 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16ec158270 con 0x7f16ec156c50 2024-09-17T04:49:08.639 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16daffd640 1 -- 172.21.15.176:0/3730487635 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16dc0bd070 con 0x7f16ec156c50 2024-09-17T04:49:08.639 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.634+0000 7f16daffd640 1 -- 172.21.15.176:0/3730487635 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc005290 con 0x7f16ec156c50 2024-09-17T04:49:08.639 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.638+0000 7f16daffd640 1 -- 172.21.15.176:0/3730487635 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f16dc0a6950 con 0x7f16ec156c50 2024-09-17T04:49:08.639 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.638+0000 7f16daffd640 1 -- 172.21.15.176:0/3730487635 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f16dc0039b0 con 0x7f16ec156c50 2024-09-17T04:49:08.640 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.638+0000 7f16f0a2b640 1 -- 172.21.15.176:0/3730487635 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d00052a0 con 0x7f16ec156c50 2024-09-17T04:49:08.645 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.642+0000 7f16daffd640 1 -- 172.21.15.176:0/3730487635 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f16dc0c6020 con 0x7f16ec156c50 2024-09-17T04:49:08.724 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.722+0000 7f16f0a2b640 1 -- 172.21.15.176:0/3730487635 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4f065472-fc2f-4355-9e30-3ca9a6615ba6", "id": 1} v 0) -- 0x7f16d0005590 con 0x7f16ec156c50 2024-09-17T04:49:08.731 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.730+0000 7f16daffd640 1 -- 172.21.15.176:0/3730487635 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4f065472-fc2f-4355-9e30-3ca9a6615ba6", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f16dc0164d0 con 0x7f16ec156c50 2024-09-17T04:49:08.731 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-09-17T04:49:08.733 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.730+0000 7f16f0a2b640 1 -- 172.21.15.176:0/3730487635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16ec156c50 msgr2=0x7f16ec1f9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:08.733 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.730+0000 7f16f0a2b640 1 --2- 172.21.15.176:0/3730487635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16ec156c50 0x7f16ec1f9f80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f16dc0ac000 tx=0x7f16dc0a6780 comp rx=0 tx=0).stop 2024-09-17T04:49:08.733 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.730+0000 7f16f0a2b640 1 -- 172.21.15.176:0/3730487635 shutdown_connections 2024-09-17T04:49:08.733 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.730+0000 7f16f0a2b640 1 --2- 172.21.15.176:0/3730487635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16ec156c50 0x7f16ec1f9f80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.733 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.730+0000 7f16f0a2b640 1 --2- 172.21.15.176:0/3730487635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16ec156840 0x7f16ec200450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.733 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.730+0000 7f16f0a2b640 1 --2- 172.21.15.176:0/3730487635 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16ec074f70 0x7f16ec157c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.733 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.730+0000 7f16f0a2b640 1 -- 172.21.15.176:0/3730487635 >> 172.21.15.176:0/3730487635 conn(0x7f16ec0c4e30 msgr2=0x7f16ec1556e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:08.733 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.730+0000 7f16f0a2b640 1 -- 172.21.15.176:0/3730487635 shutdown_connections 2024-09-17T04:49:08.733 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.730+0000 7f16f0a2b640 1 -- 172.21.15.176:0/3730487635 wait complete. 2024-09-17T04:49:08.749 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 6fc6848e-0c10-4f16-85f3-48fc378d762e 2 2024-09-17T04:49:08.892 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 -- 172.21.15.176:0/186583417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5280004810 msgr2=0x7f5280004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:08.892 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 --2- 172.21.15.176:0/186583417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5280004810 0x7f5280004c20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f52840019c0 tx=0x7f528401c880 comp rx=0 tx=0).stop 2024-09-17T04:49:08.892 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 -- 172.21.15.176:0/186583417 shutdown_connections 2024-09-17T04:49:08.892 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 --2- 172.21.15.176:0/186583417 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5280004810 0x7f5280004c20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.892 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 -- 172.21.15.176:0/186583417 >> 172.21.15.176:0/186583417 conn(0x7f52940aab60 msgr2=0x7f52940aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:08.892 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 -- 172.21.15.176:0/186583417 shutdown_connections 2024-09-17T04:49:08.893 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 -- 172.21.15.176:0/186583417 wait complete. 2024-09-17T04:49:08.893 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 Processor -- start 2024-09-17T04:49:08.893 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 -- start start 2024-09-17T04:49:08.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5280004810 0x7f52940c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:08.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5294157eb0 0x7f52941f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:08.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52941f38e0 0x7f5294147060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:08.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f529406d5f0 con 0x7f52941f38e0 2024-09-17T04:49:08.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f529406d470 con 0x7f5280004810 2024-09-17T04:49:08.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f529b43c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f529406d770 con 0x7f5294157eb0 2024-09-17T04:49:08.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f52989d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5294157eb0 0x7f52941f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:08.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f52991d8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5280004810 0x7f52940c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:08.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f52989d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5294157eb0 0x7f52941f33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:56690/0 (socket says 172.21.15.176:56690) 2024-09-17T04:49:08.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f52991d8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5280004810 0x7f52940c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:58562/0 (socket says 172.21.15.176:58562) 2024-09-17T04:49:08.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.890+0000 7f52989d7640 1 -- 172.21.15.176:0/3342511689 learned_addr learned my addr 172.21.15.176:0/3342511689 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-17T04:49:08.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f52999d9640 1 --2- 172.21.15.176:0/3342511689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52941f38e0 0x7f5294147060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:08.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f52989d7640 1 -- 172.21.15.176:0/3342511689 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5280004810 msgr2=0x7f52940c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:08.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f52989d7640 1 --2- 172.21.15.176:0/3342511689 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5280004810 0x7f52940c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f52989d7640 1 -- 172.21.15.176:0/3342511689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52941f38e0 msgr2=0x7f5294147060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:08.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f52989d7640 1 --2- 172.21.15.176:0/3342511689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52941f38e0 0x7f5294147060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:08.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f52989d7640 1 -- 172.21.15.176:0/3342511689 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52941f40d0 con 0x7f5294157eb0 2024-09-17T04:49:08.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f52991d8640 1 --2- 172.21.15.176:0/3342511689 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5280004810 0x7f52940c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:08.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f52989d7640 1 --2- 172.21.15.176:0/3342511689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5294157eb0 0x7f52941f33a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f528401cd60 tx=0x7f5284000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:08.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f5289ffb640 1 -- 172.21.15.176:0/3342511689 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52840015e0 con 0x7f5294157eb0 2024-09-17T04:49:08.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f5289ffb640 1 -- 172.21.15.176:0/3342511689 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5284001120 con 0x7f5294157eb0 2024-09-17T04:49:08.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f529b43c640 1 -- 172.21.15.176:0/3342511689 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5294158af0 con 0x7f5294157eb0 2024-09-17T04:49:08.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f5289ffb640 1 -- 172.21.15.176:0/3342511689 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5284004850 con 0x7f5294157eb0 2024-09-17T04:49:08.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f529b43c640 1 -- 172.21.15.176:0/3342511689 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5294159b50 con 0x7f5294157eb0 2024-09-17T04:49:08.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f5289ffb640 1 -- 172.21.15.176:0/3342511689 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f52840a5530 con 0x7f5294157eb0 2024-09-17T04:49:08.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f5289ffb640 1 -- 172.21.15.176:0/3342511689 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f5284004d80 con 0x7f5294157eb0 2024-09-17T04:49:08.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f529b43c640 1 -- 172.21.15.176:0/3342511689 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5294159e50 con 0x7f5294157eb0 2024-09-17T04:49:08.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.894+0000 7f5289ffb640 1 -- 172.21.15.176:0/3342511689 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f52941544a0 con 0x7f5294157eb0 2024-09-17T04:49:08.900 INFO:tasks.ceph.mgr.x.smithi176.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-17T04:49:08.900 INFO:tasks.ceph.mgr.x.smithi176.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-17T04:49:08.900 INFO:tasks.ceph.mgr.x.smithi176.stderr: from numpy import show_config as show_numpy_config 2024-09-17T04:49:08.997 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:08.994+0000 7f529b43c640 1 -- 172.21.15.176:0/3342511689 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6fc6848e-0c10-4f16-85f3-48fc378d762e", "id": 2} v 0) -- 0x7f52940c4050 con 0x7f5294157eb0 2024-09-17T04:49:09.003 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.002+0000 7f5289ffb640 1 -- 172.21.15.176:0/3342511689 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6fc6848e-0c10-4f16-85f3-48fc378d762e", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f52840a5a00 con 0x7f5294157eb0 2024-09-17T04:49:09.004 INFO:teuthology.orchestra.run.smithi176.stdout:2 2024-09-17T04:49:09.005 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.002+0000 7f529b43c640 1 -- 172.21.15.176:0/3342511689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5294157eb0 msgr2=0x7f52941f33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.005 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.002+0000 7f529b43c640 1 --2- 172.21.15.176:0/3342511689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5294157eb0 0x7f52941f33a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f528401cd60 tx=0x7f5284000ee0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.006 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.002+0000 7f529b43c640 1 -- 172.21.15.176:0/3342511689 shutdown_connections 2024-09-17T04:49:09.006 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.002+0000 7f529b43c640 1 --2- 172.21.15.176:0/3342511689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52941f38e0 0x7f5294147060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.006 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.002+0000 7f529b43c640 1 --2- 172.21.15.176:0/3342511689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5294157eb0 0x7f52941f33a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.006 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.002+0000 7f529b43c640 1 --2- 172.21.15.176:0/3342511689 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5280004810 0x7f52940c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.006 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.002+0000 7f529b43c640 1 -- 172.21.15.176:0/3342511689 >> 172.21.15.176:0/3342511689 conn(0x7f52940aab60 msgr2=0x7f52941e1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:09.006 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.002+0000 7f529b43c640 1 -- 172.21.15.176:0/3342511689 shutdown_connections 2024-09-17T04:49:09.006 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.002+0000 7f529b43c640 1 -- 172.21.15.176:0/3342511689 wait complete. 2024-09-17T04:49:09.031 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 50d85cf9-61f7-4476-9f71-3b5fc940fdda 3 2024-09-17T04:49:09.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.186+0000 7f6802660640 1 -- 172.21.15.176:0/2869980333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67fc0beda0 msgr2=0x7f67fc0bf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.186+0000 7f6802660640 1 --2- 172.21.15.176:0/2869980333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67fc0beda0 0x7f67fc0bf720 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009c80 tx=0x7f67ec01d800 comp rx=0 tx=0).stop 2024-09-17T04:49:09.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.186+0000 7f6802660640 1 -- 172.21.15.176:0/2869980333 shutdown_connections 2024-09-17T04:49:09.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.186+0000 7f6802660640 1 --2- 172.21.15.176:0/2869980333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67fc0c4bd0 0x7f67fc0c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.186+0000 7f6802660640 1 --2- 172.21.15.176:0/2869980333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67fc0beda0 0x7f67fc0bf720 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.186+0000 7f6802660640 1 -- 172.21.15.176:0/2869980333 >> 172.21.15.176:0/2869980333 conn(0x7f67fc0c9ec0 msgr2=0x7f67fc0ca2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:09.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.186+0000 7f6802660640 1 -- 172.21.15.176:0/2869980333 shutdown_connections 2024-09-17T04:49:09.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.186+0000 7f6802660640 1 -- 172.21.15.176:0/2869980333 wait complete. 2024-09-17T04:49:09.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.186+0000 7f6802660640 1 Processor -- start 2024-09-17T04:49:09.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.186+0000 7f6802660640 1 -- start start 2024-09-17T04:49:09.193 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f6802660640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67fc0beda0 0x7f67fc151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:09.193 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f6802660640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67fc0c4bd0 0x7f67fc0aad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:09.193 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f6802660640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67fc0c7dc0 0x7f67fc145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:09.193 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f6802660640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc14eb60 con 0x7f67fc0c4bd0 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f6802660640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc14e9e0 con 0x7f67fc0beda0 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f6802660640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f67fc1618c0 con 0x7f67fc0c7dc0 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67fbfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67fc0beda0 0x7f67fc151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67fbfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67fc0beda0 0x7f67fc151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:58568/0 (socket says 172.21.15.176:58568) 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67fbfff640 1 -- 172.21.15.176:0/1295875072 learned_addr learned my addr 172.21.15.176:0/1295875072 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f6800bfd640 1 --2- 172.21.15.176:0/1295875072 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67fc0c7dc0 0x7f67fc145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67fb7fe640 1 --2- 172.21.15.176:0/1295875072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67fc0c4bd0 0x7f67fc0aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67fbfff640 1 -- 172.21.15.176:0/1295875072 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67fc0c7dc0 msgr2=0x7f67fc145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67fbfff640 1 --2- 172.21.15.176:0/1295875072 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67fc0c7dc0 0x7f67fc145710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67fbfff640 1 -- 172.21.15.176:0/1295875072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67fc0c4bd0 msgr2=0x7f67fc0aad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67fbfff640 1 --2- 172.21.15.176:0/1295875072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67fc0c4bd0 0x7f67fc0aad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67fbfff640 1 -- 172.21.15.176:0/1295875072 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67fc14ff10 con 0x7f67fc0beda0 2024-09-17T04:49:09.194 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67fbfff640 1 --2- 172.21.15.176:0/1295875072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67fc0beda0 0x7f67fc151220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f67ec01a9d0 tx=0x7f67ec0092c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:09.195 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67f8ff9640 1 -- 172.21.15.176:0/1295875072 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0b2040 con 0x7f67fc0beda0 2024-09-17T04:49:09.195 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67f8ff9640 1 -- 172.21.15.176:0/1295875072 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67ec0ad8e0 con 0x7f67fc0beda0 2024-09-17T04:49:09.195 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67f8ff9640 1 -- 172.21.15.176:0/1295875072 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0adcd0 con 0x7f67fc0beda0 2024-09-17T04:49:09.195 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f6802660640 1 -- 172.21.15.176:0/1295875072 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67fc1eeb30 con 0x7f67fc0beda0 2024-09-17T04:49:09.195 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f6802660640 1 -- 172.21.15.176:0/1295875072 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67fc15e120 con 0x7f67fc0beda0 2024-09-17T04:49:09.195 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67f8ff9640 1 -- 172.21.15.176:0/1295875072 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f67ec0bd060 con 0x7f67fc0beda0 2024-09-17T04:49:09.195 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f67f8ff9640 1 -- 172.21.15.176:0/1295875072 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f67ec0bfa70 con 0x7f67fc0beda0 2024-09-17T04:49:09.195 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.190+0000 7f6802660640 1 -- 172.21.15.176:0/1295875072 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67fc0c4fe0 con 0x7f67fc0beda0 2024-09-17T04:49:09.195 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.194+0000 7f67f8ff9640 1 -- 172.21.15.176:0/1295875072 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f67ec0c4020 con 0x7f67fc0beda0 2024-09-17T04:49:09.267 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.262+0000 7f6802660640 1 -- 172.21.15.176:0/1295875072 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "50d85cf9-61f7-4476-9f71-3b5fc940fdda", "id": 3} v 0) -- 0x7f67fc15ee80 con 0x7f67fc0beda0 2024-09-17T04:49:09.272 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.270+0000 7f67f8ff9640 1 -- 172.21.15.176:0/1295875072 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "50d85cf9-61f7-4476-9f71-3b5fc940fdda", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f67fc14ff10 con 0x7f67fc0beda0 2024-09-17T04:49:09.272 INFO:teuthology.orchestra.run.smithi176.stdout:3 2024-09-17T04:49:09.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.270+0000 7f6802660640 1 -- 172.21.15.176:0/1295875072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67fc0beda0 msgr2=0x7f67fc151220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.270+0000 7f6802660640 1 --2- 172.21.15.176:0/1295875072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67fc0beda0 0x7f67fc151220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f67ec01a9d0 tx=0x7f67ec0092c0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.270+0000 7f6802660640 1 -- 172.21.15.176:0/1295875072 shutdown_connections 2024-09-17T04:49:09.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.270+0000 7f6802660640 1 --2- 172.21.15.176:0/1295875072 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67fc0c7dc0 0x7f67fc145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.270+0000 7f6802660640 1 --2- 172.21.15.176:0/1295875072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67fc0c4bd0 0x7f67fc0aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.270+0000 7f6802660640 1 --2- 172.21.15.176:0/1295875072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67fc0beda0 0x7f67fc151220 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.270+0000 7f6802660640 1 -- 172.21.15.176:0/1295875072 >> 172.21.15.176:0/1295875072 conn(0x7f67fc0c9ec0 msgr2=0x7f67fc0c81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:09.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.270+0000 7f6802660640 1 -- 172.21.15.176:0/1295875072 shutdown_connections 2024-09-17T04:49:09.274 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.270+0000 7f6802660640 1 -- 172.21.15.176:0/1295875072 wait complete. 2024-09-17T04:49:09.291 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 55c7a772-ac2c-4f68-bba2-cf7cbc9c9569 4 2024-09-17T04:49:09.437 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.434+0000 7f45fa4c7640 1 -- 172.21.15.176:0/2832151951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f4157c00 msgr2=0x7f45f4158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.437 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.434+0000 7f45fa4c7640 1 --2- 172.21.15.176:0/2832151951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f4157c00 0x7f45f4158010 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f45f40a9a30 tx=0x7f45ec01cb80 comp rx=0 tx=0).stop 2024-09-17T04:49:09.437 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.434+0000 7f45fa4c7640 1 -- 172.21.15.176:0/2832151951 shutdown_connections 2024-09-17T04:49:09.437 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.434+0000 7f45fa4c7640 1 --2- 172.21.15.176:0/2832151951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f4157c00 0x7f45f4158010 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.437 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.434+0000 7f45fa4c7640 1 --2- 172.21.15.176:0/2832151951 >> v2:172.21.15.150:3300/0 conn(0x7f45f4071670 0x7f45f40c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.437 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.434+0000 7f45fa4c7640 1 -- 172.21.15.176:0/2832151951 >> 172.21.15.176:0/2832151951 conn(0x7f45f40ca460 msgr2=0x7f45f40c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:09.437 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.434+0000 7f45fa4c7640 1 -- 172.21.15.176:0/2832151951 shutdown_connections 2024-09-17T04:49:09.437 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.434+0000 7f45fa4c7640 1 -- 172.21.15.176:0/2832151951 wait complete. 2024-09-17T04:49:09.438 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.434+0000 7f45fa4c7640 1 Processor -- start 2024-09-17T04:49:09.438 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.434+0000 7f45fa4c7640 1 -- start start 2024-09-17T04:49:09.441 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45fa4c7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f4071670 0x7f45f4153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:09.441 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45fa4c7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45f414e490 0x7f45f406f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:09.441 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45fa4c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f4157c00 0x7f45f4145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:09.441 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45fa4c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f45f414ea20 con 0x7f45f4157c00 2024-09-17T04:49:09.441 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45fa4c7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f45f414e8a0 con 0x7f45f4071670 2024-09-17T04:49:09.441 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45fa4c7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f45f4072180 con 0x7f45f414e490 2024-09-17T04:49:09.441 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f3fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f4071670 0x7f45f4153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:09.441 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f3fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f4071670 0x7f45f4153e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:58586/0 (socket says 172.21.15.176:58586) 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f37fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45f414e490 0x7f45f406f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f3fff640 1 -- 172.21.15.176:0/1899650587 learned_addr learned my addr 172.21.15.176:0/1899650587 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f8a64640 1 --2- 172.21.15.176:0/1899650587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f4157c00 0x7f45f4145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f37fe640 1 -- 172.21.15.176:0/1899650587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f4071670 msgr2=0x7f45f4153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f37fe640 1 --2- 172.21.15.176:0/1899650587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f4071670 0x7f45f4153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f37fe640 1 -- 172.21.15.176:0/1899650587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f4157c00 msgr2=0x7f45f4145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f37fe640 1 --2- 172.21.15.176:0/1899650587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f4157c00 0x7f45f4145b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f37fe640 1 -- 172.21.15.176:0/1899650587 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f4152900 con 0x7f45f414e490 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f3fff640 1 --2- 172.21.15.176:0/1899650587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f4071670 0x7f45f4153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f37fe640 1 --2- 172.21.15.176:0/1899650587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45f414e490 0x7f45f406f2b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f45e400c3f0 tx=0x7f45e400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f0ff9640 1 -- 172.21.15.176:0/1899650587 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4017070 con 0x7f45f414e490 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f0ff9640 1 -- 172.21.15.176:0/1899650587 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45e4003600 con 0x7f45f414e490 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45fa4c7640 1 -- 172.21.15.176:0/1899650587 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45f4152620 con 0x7f45f414e490 2024-09-17T04:49:09.442 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f0ff9640 1 -- 172.21.15.176:0/1899650587 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4003a50 con 0x7f45f414e490 2024-09-17T04:49:09.443 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45fa4c7640 1 -- 172.21.15.176:0/1899650587 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f45f41536c0 con 0x7f45f414e490 2024-09-17T04:49:09.443 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f0ff9640 1 -- 172.21.15.176:0/1899650587 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f45e401a070 con 0x7f45f414e490 2024-09-17T04:49:09.443 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45f0ff9640 1 -- 172.21.15.176:0/1899650587 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f45e4012910 con 0x7f45f414e490 2024-09-17T04:49:09.443 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.438+0000 7f45fa4c7640 1 -- 172.21.15.176:0/1899650587 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45f41f3620 con 0x7f45f414e490 2024-09-17T04:49:09.444 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.442+0000 7f45f0ff9640 1 -- 172.21.15.176:0/1899650587 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f45e4021070 con 0x7f45f414e490 2024-09-17T04:49:09.512 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.510+0000 7f45fa4c7640 1 -- 172.21.15.176:0/1899650587 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "55c7a772-ac2c-4f68-bba2-cf7cbc9c9569", "id": 4} v 0) -- 0x7f45f40c5680 con 0x7f45f414e490 2024-09-17T04:49:09.518 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.514+0000 7f45f0ff9640 1 -- 172.21.15.176:0/1899650587 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "55c7a772-ac2c-4f68-bba2-cf7cbc9c9569", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f45e4003d20 con 0x7f45f414e490 2024-09-17T04:49:09.518 INFO:teuthology.orchestra.run.smithi176.stdout:4 2024-09-17T04:49:09.520 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.518+0000 7f45fa4c7640 1 -- 172.21.15.176:0/1899650587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45f414e490 msgr2=0x7f45f406f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.520 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.518+0000 7f45fa4c7640 1 --2- 172.21.15.176:0/1899650587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45f414e490 0x7f45f406f2b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f45e400c3f0 tx=0x7f45e400c8b0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.520 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.518+0000 7f45fa4c7640 1 -- 172.21.15.176:0/1899650587 shutdown_connections 2024-09-17T04:49:09.520 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.518+0000 7f45fa4c7640 1 --2- 172.21.15.176:0/1899650587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f4157c00 0x7f45f4145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.520 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.518+0000 7f45fa4c7640 1 --2- 172.21.15.176:0/1899650587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45f414e490 0x7f45f406f2b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.520 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.518+0000 7f45fa4c7640 1 --2- 172.21.15.176:0/1899650587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f4071670 0x7f45f4153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.520 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.518+0000 7f45fa4c7640 1 -- 172.21.15.176:0/1899650587 >> 172.21.15.176:0/1899650587 conn(0x7f45f40ca460 msgr2=0x7f45f414c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:09.520 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.518+0000 7f45fa4c7640 1 -- 172.21.15.176:0/1899650587 shutdown_connections 2024-09-17T04:49:09.520 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.518+0000 7f45fa4c7640 1 -- 172.21.15.176:0/1899650587 wait complete. 2024-09-17T04:49:09.537 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 62b9fc78-e4c5-4c20-8391-63c4b411d840 5 2024-09-17T04:49:09.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.674+0000 7fdfa7452640 1 -- 172.21.15.176:0/3213788519 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfa006e370 msgr2=0x7fdfa0153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.674+0000 7fdfa7452640 1 --2- 172.21.15.176:0/3213788519 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfa006e370 0x7fdfa0153df0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009bc0 tx=0x7fdf9801ca80 comp rx=0 tx=0).stop 2024-09-17T04:49:09.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.674+0000 7fdfa7452640 1 -- 172.21.15.176:0/3213788519 shutdown_connections 2024-09-17T04:49:09.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.674+0000 7fdfa7452640 1 --2- 172.21.15.176:0/3213788519 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfa006e370 0x7fdfa0153df0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.677 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.674+0000 7fdfa7452640 1 --2- 172.21.15.176:0/3213788519 >> v2:172.21.15.150:3300/0 conn(0x7fdfa00ab9d0 0x7fdfa014da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.674+0000 7fdfa7452640 1 -- 172.21.15.176:0/3213788519 >> 172.21.15.176:0/3213788519 conn(0x7fdfa0070b50 msgr2=0x7fdfa0070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:09.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.674+0000 7fdfa7452640 1 -- 172.21.15.176:0/3213788519 shutdown_connections 2024-09-17T04:49:09.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.674+0000 7fdfa7452640 1 -- 172.21.15.176:0/3213788519 wait complete. 2024-09-17T04:49:09.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.674+0000 7fdfa7452640 1 Processor -- start 2024-09-17T04:49:09.678 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa7452640 1 -- start start 2024-09-17T04:49:09.682 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa7452640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa006df60 0x7fdfa00be1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:09.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa7452640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa006e370 0x7fdfa0150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:09.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa7452640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfa00ab9d0 0x7fdfa00c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:09.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa7452640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa0069d70 con 0x7fdfa006e370 2024-09-17T04:49:09.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa7452640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa0069bf0 con 0x7fdfa006df60 2024-09-17T04:49:09.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa7452640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdfa01608b0 con 0x7fdfa00ab9d0 2024-09-17T04:49:09.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa51ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa006df60 0x7fdfa00be1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:09.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa51ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa006df60 0x7fdfa00be1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:58588/0 (socket says 172.21.15.176:58588) 2024-09-17T04:49:09.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa51ee640 1 -- 172.21.15.176:0/2359821893 learned_addr learned my addr 172.21.15.176:0/2359821893 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-17T04:49:09.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa59ef640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfa00ab9d0 0x7fdfa00c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:09.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa49ed640 1 --2- 172.21.15.176:0/2359821893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa006e370 0x7fdfa0150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:09.683 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa51ee640 1 -- 172.21.15.176:0/2359821893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfa00ab9d0 msgr2=0x7fdfa00c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa51ee640 1 --2- 172.21.15.176:0/2359821893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfa00ab9d0 0x7fdfa00c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa51ee640 1 -- 172.21.15.176:0/2359821893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa006e370 msgr2=0x7fdfa0150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa51ee640 1 --2- 172.21.15.176:0/2359821893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa006e370 0x7fdfa0150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa51ee640 1 -- 172.21.15.176:0/2359821893 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa014e910 con 0x7fdfa006df60 2024-09-17T04:49:09.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa49ed640 1 --2- 172.21.15.176:0/2359821893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa006e370 0x7fdfa0150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:09.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa51ee640 1 --2- 172.21.15.176:0/2359821893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa006df60 0x7fdfa00be1c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0098a0 tx=0x7fdf9c009d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:09.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdf95ffb640 1 -- 172.21.15.176:0/2359821893 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c0079b0 con 0x7fdfa006df60 2024-09-17T04:49:09.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdf95ffb640 1 -- 172.21.15.176:0/2359821893 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf9c007b50 con 0x7fdfa006df60 2024-09-17T04:49:09.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdf95ffb640 1 -- 172.21.15.176:0/2359821893 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c0163f0 con 0x7fdfa006df60 2024-09-17T04:49:09.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa7452640 1 -- 172.21.15.176:0/2359821893 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa0147700 con 0x7fdfa006df60 2024-09-17T04:49:09.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa7452640 1 -- 172.21.15.176:0/2359821893 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa014e360 con 0x7fdfa006df60 2024-09-17T04:49:09.684 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdf95ffb640 1 -- 172.21.15.176:0/2359821893 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdf9c016590 con 0x7fdfa006df60 2024-09-17T04:49:09.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdf95ffb640 1 -- 172.21.15.176:0/2359821893 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fdf9c01f400 con 0x7fdfa006df60 2024-09-17T04:49:09.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.678+0000 7fdfa7452640 1 -- 172.21.15.176:0/2359821893 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfa0147500 con 0x7fdfa006df60 2024-09-17T04:49:09.685 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.682+0000 7fdf95ffb640 1 -- 172.21.15.176:0/2359821893 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fdf9c01f900 con 0x7fdfa006df60 2024-09-17T04:49:09.768 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.766+0000 7fdfa7452640 1 -- 172.21.15.176:0/2359821893 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "62b9fc78-e4c5-4c20-8391-63c4b411d840", "id": 5} v 0) -- 0x7fdfa014da10 con 0x7fdfa006df60 2024-09-17T04:49:09.774 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.770+0000 7fdf95ffb640 1 -- 172.21.15.176:0/2359821893 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "62b9fc78-e4c5-4c20-8391-63c4b411d840", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fdf9c01fae0 con 0x7fdfa006df60 2024-09-17T04:49:09.774 INFO:teuthology.orchestra.run.smithi176.stdout:5 2024-09-17T04:49:09.774 INFO:tasks.ceph.mgr.y.smithi150.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-17T04:49:09.774 INFO:tasks.ceph.mgr.y.smithi150.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-17T04:49:09.775 INFO:tasks.ceph.mgr.y.smithi150.stderr: from numpy import show_config as show_numpy_config 2024-09-17T04:49:09.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.774+0000 7fdfa7452640 1 -- 172.21.15.176:0/2359821893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa006df60 msgr2=0x7fdfa00be1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.774+0000 7fdfa7452640 1 --2- 172.21.15.176:0/2359821893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa006df60 0x7fdfa00be1c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0098a0 tx=0x7fdf9c009d60 comp rx=0 tx=0).stop 2024-09-17T04:49:09.775 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.774+0000 7fdfa7452640 1 -- 172.21.15.176:0/2359821893 shutdown_connections 2024-09-17T04:49:09.776 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.774+0000 7fdfa7452640 1 --2- 172.21.15.176:0/2359821893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfa00ab9d0 0x7fdfa00c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.776 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.774+0000 7fdfa7452640 1 --2- 172.21.15.176:0/2359821893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa006e370 0x7fdfa0150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.776 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.774+0000 7fdfa7452640 1 --2- 172.21.15.176:0/2359821893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfa006df60 0x7fdfa00be1c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.776 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.774+0000 7fdfa7452640 1 -- 172.21.15.176:0/2359821893 >> 172.21.15.176:0/2359821893 conn(0x7fdfa0070b50 msgr2=0x7fdfa00a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:09.776 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.774+0000 7fdfa7452640 1 -- 172.21.15.176:0/2359821893 shutdown_connections 2024-09-17T04:49:09.776 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.774+0000 7fdfa7452640 1 -- 172.21.15.176:0/2359821893 wait complete. 2024-09-17T04:49:09.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 142e4b00-3eee-422a-8274-65396c592493 6 2024-09-17T04:49:09.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.942+0000 7fefed565640 1 -- 172.21.15.176:0/4278310523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe806e420 msgr2=0x7fefe8156b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.942+0000 7fefed565640 1 --2- 172.21.15.176:0/4278310523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe806e420 0x7fefe8156b40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fefe8069040 tx=0x7fefdc01c770 comp rx=0 tx=0).stop 2024-09-17T04:49:09.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.942+0000 7fefed565640 1 -- 172.21.15.176:0/4278310523 shutdown_connections 2024-09-17T04:49:09.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.942+0000 7fefed565640 1 --2- 172.21.15.176:0/4278310523 >> v2:172.21.15.150:3300/0 conn(0x7fefe8074d80 0x7fefe8157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.942+0000 7fefed565640 1 --2- 172.21.15.176:0/4278310523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe806e420 0x7fefe8156b40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.942+0000 7fefed565640 1 --2- 172.21.15.176:0/4278310523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefe8070580 0x7fefe806dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.942+0000 7fefed565640 1 -- 172.21.15.176:0/4278310523 >> 172.21.15.176:0/4278310523 conn(0x7fefe80c5970 msgr2=0x7fefe8073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:09.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.942+0000 7fefed565640 1 -- 172.21.15.176:0/4278310523 shutdown_connections 2024-09-17T04:49:09.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.942+0000 7fefed565640 1 -- 172.21.15.176:0/4278310523 wait complete. 2024-09-17T04:49:09.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.942+0000 7fefed565640 1 Processor -- start 2024-09-17T04:49:09.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.942+0000 7fefed565640 1 -- start start 2024-09-17T04:49:09.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefed565640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe806e420 0x7fefe81598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:09.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefed565640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefe8070580 0x7fefe80a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:09.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefed565640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe8074d80 0x7fefe814a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:09.949 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefed565640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fefe8153f50 con 0x7fefe806e420 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefed565640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fefe8153dd0 con 0x7fefe8074d80 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefed565640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fefe81540d0 con 0x7fefe8070580 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefe77fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe8074d80 0x7fefe814a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefe67fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefe8070580 0x7fefe80a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefe77fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe8074d80 0x7fefe814a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:58598/0 (socket says 172.21.15.176:58598) 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefe77fe640 1 -- 172.21.15.176:0/2489781430 learned_addr learned my addr 172.21.15.176:0/2489781430 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefe6ffd640 1 --2- 172.21.15.176:0/2489781430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe806e420 0x7fefe81598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefe67fc640 1 -- 172.21.15.176:0/2489781430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe8074d80 msgr2=0x7fefe814a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefe67fc640 1 --2- 172.21.15.176:0/2489781430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe8074d80 0x7fefe814a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefe67fc640 1 -- 172.21.15.176:0/2489781430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe806e420 msgr2=0x7fefe81598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefe67fc640 1 --2- 172.21.15.176:0/2489781430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe806e420 0x7fefe81598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefe67fc640 1 -- 172.21.15.176:0/2489781430 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe814ce60 con 0x7fefe8070580 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefe77fe640 1 --2- 172.21.15.176:0/2489781430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe8074d80 0x7fefe814a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:09.950 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefe67fc640 1 --2- 172.21.15.176:0/2489781430 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefe8070580 0x7fefe80a9a70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fefe80c4bc0 tx=0x7fefdc0a65e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:09.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefcf7fe640 1 -- 172.21.15.176:0/2489781430 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc0bc020 con 0x7fefe8070580 2024-09-17T04:49:09.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefcf7fe640 1 -- 172.21.15.176:0/2489781430 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fefdc004340 con 0x7fefe8070580 2024-09-17T04:49:09.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefed565640 1 -- 172.21.15.176:0/2489781430 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe8158c80 con 0x7fefe8070580 2024-09-17T04:49:09.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefcf7fe640 1 -- 172.21.15.176:0/2489781430 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc005280 con 0x7fefe8070580 2024-09-17T04:49:09.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefed565640 1 -- 172.21.15.176:0/2489781430 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe815c6b0 con 0x7fefe8070580 2024-09-17T04:49:09.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefcf7fe640 1 -- 172.21.15.176:0/2489781430 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fefdc004a60 con 0x7fefe8070580 2024-09-17T04:49:09.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefcf7fe640 1 -- 172.21.15.176:0/2489781430 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fefdc0b7480 con 0x7fefe8070580 2024-09-17T04:49:09.951 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.946+0000 7fefed565640 1 -- 172.21.15.176:0/2489781430 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe806dee0 con 0x7fefe8070580 2024-09-17T04:49:09.952 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:09.950+0000 7fefcf7fe640 1 -- 172.21.15.176:0/2489781430 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fefdc0c6020 con 0x7fefe8070580 2024-09-17T04:49:10.018 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.014+0000 7fefed565640 1 -- 172.21.15.176:0/2489781430 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "142e4b00-3eee-422a-8274-65396c592493", "id": 6} v 0) -- 0x7fefe80c2d00 con 0x7fefe8070580 2024-09-17T04:49:10.024 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.022+0000 7fefcf7fe640 1 -- 172.21.15.176:0/2489781430 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "142e4b00-3eee-422a-8274-65396c592493", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fefdc0163f0 con 0x7fefe8070580 2024-09-17T04:49:10.024 INFO:teuthology.orchestra.run.smithi176.stdout:6 2024-09-17T04:49:10.026 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.022+0000 7fefed565640 1 -- 172.21.15.176:0/2489781430 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefe8070580 msgr2=0x7fefe80a9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:10.026 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.022+0000 7fefed565640 1 --2- 172.21.15.176:0/2489781430 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefe8070580 0x7fefe80a9a70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fefe80c4bc0 tx=0x7fefdc0a65e0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.026 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.022+0000 7fefed565640 1 -- 172.21.15.176:0/2489781430 shutdown_connections 2024-09-17T04:49:10.026 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.022+0000 7fefed565640 1 --2- 172.21.15.176:0/2489781430 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe8074d80 0x7fefe814a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.026 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.022+0000 7fefed565640 1 --2- 172.21.15.176:0/2489781430 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefe8070580 0x7fefe80a9a70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.026 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.022+0000 7fefed565640 1 --2- 172.21.15.176:0/2489781430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe806e420 0x7fefe81598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.026 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.022+0000 7fefed565640 1 -- 172.21.15.176:0/2489781430 >> 172.21.15.176:0/2489781430 conn(0x7fefe80c5970 msgr2=0x7fefe815d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:10.026 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.022+0000 7fefed565640 1 -- 172.21.15.176:0/2489781430 shutdown_connections 2024-09-17T04:49:10.026 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.022+0000 7fefed565640 1 -- 172.21.15.176:0/2489781430 wait complete. 2024-09-17T04:49:10.043 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new da214a44-feda-404e-b901-920961e6f786 7 2024-09-17T04:49:10.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.182+0000 7f75f839d640 1 -- 172.21.15.176:0/2084684215 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f0074e60 msgr2=0x7f75f015df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:10.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.182+0000 7f75f839d640 1 --2- 172.21.15.176:0/2084684215 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f0074e60 0x7f75f015df70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009bc0 tx=0x7f75e801cba0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.182+0000 7f75f839d640 1 -- 172.21.15.176:0/2084684215 shutdown_connections 2024-09-17T04:49:10.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.182+0000 7f75f839d640 1 --2- 172.21.15.176:0/2084684215 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f0074e60 0x7f75f015df70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.182+0000 7f75f839d640 1 --2- 172.21.15.176:0/2084684215 >> v2:172.21.15.150:3300/0 conn(0x7f75f00c3900 0x7f75f0074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.182+0000 7f75f839d640 1 -- 172.21.15.176:0/2084684215 >> 172.21.15.176:0/2084684215 conn(0x7f75f006e360 msgr2=0x7f75f006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:10.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.182+0000 7f75f839d640 1 -- 172.21.15.176:0/2084684215 shutdown_connections 2024-09-17T04:49:10.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.182+0000 7f75f839d640 1 -- 172.21.15.176:0/2084684215 wait complete. 2024-09-17T04:49:10.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.182+0000 7f75f839d640 1 Processor -- start 2024-09-17T04:49:10.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.182+0000 7f75f839d640 1 -- start start 2024-09-17T04:49:10.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f839d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75f0074e60 0x7f75f014b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:10.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f839d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f00c3900 0x7f75f0156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f839d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f00c4250 0x7f75f01f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f839d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f75f0160e50 con 0x7f75f0074e60 2024-09-17T04:49:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f839d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f75f0160cd0 con 0x7f75f00c3900 2024-09-17T04:49:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f839d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f75f0160fd0 con 0x7f75f00c4250 2024-09-17T04:49:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f693a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f00c4250 0x7f75f01f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f5938640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f00c3900 0x7f75f0156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f693a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f00c4250 0x7f75f01f2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.176:56778/0 (socket says 172.21.15.176:56778) 2024-09-17T04:49:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f693a640 1 -- 172.21.15.176:0/1944982811 learned_addr learned my addr 172.21.15.176:0/1944982811 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-17T04:49:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f6139640 1 --2- 172.21.15.176:0/1944982811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75f0074e60 0x7f75f014b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f693a640 1 -- 172.21.15.176:0/1944982811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f00c3900 msgr2=0x7f75f0156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f693a640 1 --2- 172.21.15.176:0/1944982811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f00c3900 0x7f75f0156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f693a640 1 -- 172.21.15.176:0/1944982811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75f0074e60 msgr2=0x7f75f014b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:10.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f693a640 1 --2- 172.21.15.176:0/1944982811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75f0074e60 0x7f75f014b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f693a640 1 -- 172.21.15.176:0/1944982811 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f0154ae0 con 0x7f75f00c4250 2024-09-17T04:49:10.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f5938640 1 --2- 172.21.15.176:0/1944982811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f00c3900 0x7f75f0156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:10.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f6139640 1 --2- 172.21.15.176:0/1944982811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75f0074e60 0x7f75f014b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:49:10.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f693a640 1 --2- 172.21.15.176:0/1944982811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f00c4250 0x7f75f01f2e90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f75e801a7d0 tx=0x7f75e8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:10.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75e6ffd640 1 -- 172.21.15.176:0/1944982811 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e80bb070 con 0x7f75f00c4250 2024-09-17T04:49:10.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75e6ffd640 1 -- 172.21.15.176:0/1944982811 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75e8004d40 con 0x7f75f00c4250 2024-09-17T04:49:10.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75e6ffd640 1 -- 172.21.15.176:0/1944982811 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e80b2040 con 0x7f75f00c4250 2024-09-17T04:49:10.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f839d640 1 -- 172.21.15.176:0/1944982811 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f00c32b0 con 0x7f75f00c4250 2024-09-17T04:49:10.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f839d640 1 -- 172.21.15.176:0/1944982811 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f015e150 con 0x7f75f00c4250 2024-09-17T04:49:10.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75e6ffd640 1 -- 172.21.15.176:0/1944982811 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f75e80be070 con 0x7f75f00c4250 2024-09-17T04:49:10.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75e6ffd640 1 -- 172.21.15.176:0/1944982811 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f75e8005780 con 0x7f75f00c4250 2024-09-17T04:49:10.191 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.186+0000 7f75f839d640 1 -- 172.21.15.176:0/1944982811 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f0074920 con 0x7f75f00c4250 2024-09-17T04:49:10.192 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.190+0000 7f75e6ffd640 1 -- 172.21.15.176:0/1944982811 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f75e80d8860 con 0x7f75f00c4250 2024-09-17T04:49:10.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.274+0000 7f75f839d640 1 -- 172.21.15.176:0/1944982811 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd new", "uuid": "da214a44-feda-404e-b901-920961e6f786", "id": 7} v 0) -- 0x7f75f006d900 con 0x7f75f00c4250 2024-09-17T04:49:10.281 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.278+0000 7f75e6ffd640 1 -- 172.21.15.176:0/1944982811 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "da214a44-feda-404e-b901-920961e6f786", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f75e80c6030 con 0x7f75f00c4250 2024-09-17T04:49:10.281 INFO:teuthology.orchestra.run.smithi176.stdout:7 2024-09-17T04:49:10.282 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.278+0000 7f75f839d640 1 -- 172.21.15.176:0/1944982811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f00c4250 msgr2=0x7f75f01f2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:10.282 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.278+0000 7f75f839d640 1 --2- 172.21.15.176:0/1944982811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f00c4250 0x7f75f01f2e90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f75e801a7d0 tx=0x7f75e8009260 comp rx=0 tx=0).stop 2024-09-17T04:49:10.282 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.278+0000 7f75f839d640 1 -- 172.21.15.176:0/1944982811 shutdown_connections 2024-09-17T04:49:10.282 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.278+0000 7f75f839d640 1 --2- 172.21.15.176:0/1944982811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75f00c3900 0x7f75f0156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.282 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.278+0000 7f75f839d640 1 --2- 172.21.15.176:0/1944982811 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75f00c4250 0x7f75f01f2e90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.283 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.278+0000 7f75f839d640 1 --2- 172.21.15.176:0/1944982811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75f0074e60 0x7f75f014b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.283 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.278+0000 7f75f839d640 1 -- 172.21.15.176:0/1944982811 >> 172.21.15.176:0/1944982811 conn(0x7f75f006e360 msgr2=0x7f75f006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:10.283 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.278+0000 7f75f839d640 1 -- 172.21.15.176:0/1944982811 shutdown_connections 2024-09-17T04:49:10.283 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:49:10.278+0000 7f75f839d640 1 -- 172.21.15.176:0/1944982811 wait complete. 2024-09-17T04:49:10.300 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-17T04:49:10.300 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-17T04:49:10.303 INFO:tasks.ceph.osd.0:Started 2024-09-17T04:49:10.304 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-17T04:49:10.304 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-17T04:49:10.306 INFO:tasks.ceph.osd.1:Started 2024-09-17T04:49:10.306 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-17T04:49:10.306 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-17T04:49:10.308 INFO:tasks.ceph.osd.2:Started 2024-09-17T04:49:10.308 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-17T04:49:10.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-17T04:49:10.310 INFO:tasks.ceph.osd.3:Started 2024-09-17T04:49:10.310 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-17T04:49:10.310 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-17T04:49:10.312 INFO:tasks.ceph.osd.4:Started 2024-09-17T04:49:10.312 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-17T04:49:10.312 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-17T04:49:10.314 INFO:tasks.ceph.osd.5:Started 2024-09-17T04:49:10.314 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-17T04:49:10.314 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-17T04:49:10.317 INFO:tasks.ceph.osd.6:Started 2024-09-17T04:49:10.317 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-17T04:49:10.317 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-17T04:49:10.319 INFO:tasks.ceph.osd.7:Started 2024-09-17T04:49:10.320 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:10.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.487+0000 7fabc7d10640 1 -- 172.21.15.150:0/2824026598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc01568a0 msgr2=0x7fabc00c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:10.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.487+0000 7fabc7d10640 1 --2- 172.21.15.150:0/2824026598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc01568a0 0x7fabc00c5810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009900 tx=0x7fabbc01c8b0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.487+0000 7fabc7d10640 1 -- 172.21.15.150:0/2824026598 shutdown_connections 2024-09-17T04:49:10.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.487+0000 7fabc7d10640 1 --2- 172.21.15.150:0/2824026598 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc00c35b0 0x7fabc00c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.487+0000 7fabc7d10640 1 --2- 172.21.15.150:0/2824026598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc01568a0 0x7fabc00c5810 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.487+0000 7fabc7d10640 1 -- 172.21.15.150:0/2824026598 >> 172.21.15.150:0/2824026598 conn(0x7fabc0074b80 msgr2=0x7fabc006f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:10.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.487+0000 7fabc7d10640 1 -- 172.21.15.150:0/2824026598 shutdown_connections 2024-09-17T04:49:10.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.487+0000 7fabc7d10640 1 -- 172.21.15.150:0/2824026598 wait complete. 2024-09-17T04:49:10.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.487+0000 7fabc7d10640 1 Processor -- start 2024-09-17T04:49:10.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.487+0000 7fabc7d10640 1 -- start start 2024-09-17T04:49:10.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc7d10640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc00c35b0 0x7fabc014ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:10.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc7d10640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c8fc0 0x7fabc014f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:10.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc7d10640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc01568a0 0x7fabc01f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:10.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc7d10640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fabc00c6b70 con 0x7fabc00c8fc0 2024-09-17T04:49:10.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc7d10640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fabc00c69f0 con 0x7fabc00c35b0 2024-09-17T04:49:10.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc7d10640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fabc00c27b0 con 0x7fabc01568a0 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc52ab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c8fc0 0x7fabc014f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc5aac640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc00c35b0 0x7fabc014ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc5aac640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc00c35b0 0x7fabc014ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:56566/0 (socket says 172.21.15.150:56566) 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc52ab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c8fc0 0x7fabc014f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35130/0 (socket says 172.21.15.150:35130) 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc5aac640 1 -- 172.21.15.150:0/2581957418 learned_addr learned my addr 172.21.15.150:0/2581957418 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc62ad640 1 --2- 172.21.15.150:0/2581957418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc01568a0 0x7fabc01f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc52ab640 1 -- 172.21.15.150:0/2581957418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc01568a0 msgr2=0x7fabc01f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc52ab640 1 --2- 172.21.15.150:0/2581957418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc01568a0 0x7fabc01f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc52ab640 1 -- 172.21.15.150:0/2581957418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc00c35b0 msgr2=0x7fabc014ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc52ab640 1 --2- 172.21.15.150:0/2581957418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc00c35b0 0x7fabc014ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc52ab640 1 -- 172.21.15.150:0/2581957418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc014d350 con 0x7fabc00c8fc0 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc5aac640 1 --2- 172.21.15.150:0/2581957418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc00c35b0 0x7fabc014ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:10.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc52ab640 1 --2- 172.21.15.150:0/2581957418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c8fc0 0x7fabc014f840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fabb000d650 tx=0x7fabb000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:10.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabb67fc640 1 -- 172.21.15.150:0/2581957418 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0017070 con 0x7fabc00c8fc0 2024-09-17T04:49:10.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabb67fc640 1 -- 172.21.15.150:0/2581957418 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabb0004220 con 0x7fabc00c8fc0 2024-09-17T04:49:10.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc7d10640 1 -- 172.21.15.150:0/2581957418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc014f550 con 0x7fabc00c8fc0 2024-09-17T04:49:10.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabb67fc640 1 -- 172.21.15.150:0/2581957418 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0005020 con 0x7fabc00c8fc0 2024-09-17T04:49:10.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc7d10640 1 -- 172.21.15.150:0/2581957418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc014d820 con 0x7fabc00c8fc0 2024-09-17T04:49:10.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabb67fc640 1 -- 172.21.15.150:0/2581957418 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fabb00097c0 con 0x7fabc00c8fc0 2024-09-17T04:49:10.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabb67fc640 1 -- 172.21.15.150:0/2581957418 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fabb0004df0 con 0x7fabc00c8fc0 2024-09-17T04:49:10.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.491+0000 7fabc7d10640 1 -- 172.21.15.150:0/2581957418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc00c8a80 con 0x7fabc00c8fc0 2024-09-17T04:49:10.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.495+0000 7fabb67fc640 1 -- 172.21.15.150:0/2581957418 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fabb0020020 con 0x7fabc00c8fc0 2024-09-17T04:49:10.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.555+0000 7fabc7d10640 1 -- 172.21.15.150:0/2581957418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fabc014e360 con 0x7fabc00c8fc0 2024-09-17T04:49:10.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.559+0000 7fabb67fc640 1 -- 172.21.15.150:0/2581957418 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fabb0005610 con 0x7fabc00c8fc0 2024-09-17T04:49:10.558 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:10.558 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":10,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:10.281212+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:10.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.559+0000 7fabc7d10640 1 -- 172.21.15.150:0/2581957418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c8fc0 msgr2=0x7fabc014f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:10.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.559+0000 7fabc7d10640 1 --2- 172.21.15.150:0/2581957418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c8fc0 0x7fabc014f840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fabb000d650 tx=0x7fabb000db10 comp rx=0 tx=0).stop 2024-09-17T04:49:10.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.559+0000 7fabc7d10640 1 -- 172.21.15.150:0/2581957418 shutdown_connections 2024-09-17T04:49:10.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.559+0000 7fabc7d10640 1 --2- 172.21.15.150:0/2581957418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc01568a0 0x7fabc01f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.559+0000 7fabc7d10640 1 --2- 172.21.15.150:0/2581957418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c8fc0 0x7fabc014f840 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.559+0000 7fabc7d10640 1 --2- 172.21.15.150:0/2581957418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc00c35b0 0x7fabc014ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:10.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.559+0000 7fabc7d10640 1 -- 172.21.15.150:0/2581957418 >> 172.21.15.150:0/2581957418 conn(0x7fabc0074b80 msgr2=0x7fabc0068fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:10.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.559+0000 7fabc7d10640 1 -- 172.21.15.150:0/2581957418 shutdown_connections 2024-09-17T04:49:10.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:10.559+0000 7fabc7d10640 1 -- 172.21.15.150:0/2581957418 wait complete. 2024-09-17T04:49:10.579 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-17T04:49:10.579 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-17T04:49:11.385 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:11.532 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.531+0000 7f335ed51640 1 -- 172.21.15.150:0/322639065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f335806de60 msgr2=0x7f335806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:11.532 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.531+0000 7f335ed51640 1 --2- 172.21.15.150:0/322639065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f335806de60 0x7f335806e270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3358073b20 tx=0x7f334001c830 comp rx=0 tx=0).stop 2024-09-17T04:49:11.532 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.531+0000 7f335ed51640 1 -- 172.21.15.150:0/322639065 shutdown_connections 2024-09-17T04:49:11.532 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.531+0000 7f335ed51640 1 --2- 172.21.15.150:0/322639065 >> v2:172.21.15.176:3300/0 conn(0x7f3358153b60 0x7f3358153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:11.532 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.531+0000 7f335ed51640 1 --2- 172.21.15.150:0/322639065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f335806de60 0x7f335806e270 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:11.532 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.531+0000 7f335ed51640 1 --2- 172.21.15.150:0/322639065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33580ab2d0 0x7f335814d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:11.532 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.531+0000 7f335ed51640 1 -- 172.21.15.150:0/322639065 >> 172.21.15.150:0/322639065 conn(0x7f3358074190 msgr2=0x7f33580745a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:11.533 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.531+0000 7f335ed51640 1 -- 172.21.15.150:0/322639065 shutdown_connections 2024-09-17T04:49:11.533 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.531+0000 7f335ed51640 1 -- 172.21.15.150:0/322639065 wait complete. 2024-09-17T04:49:11.534 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335ed51640 1 Processor -- start 2024-09-17T04:49:11.534 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335ed51640 1 -- start start 2024-09-17T04:49:11.537 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335ed51640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335806de60 0x7f3358145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:11.538 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335ed51640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33580ab2d0 0x7f33580adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:11.538 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335ed51640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3358153b60 0x7f33581ef710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:11.538 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335ed51640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f33580c2f10 con 0x7f3358153b60 2024-09-17T04:49:11.538 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335ed51640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f33581544b0 con 0x7f335806de60 2024-09-17T04:49:11.538 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335ed51640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f33580aae50 con 0x7f33580ab2d0 2024-09-17T04:49:11.538 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335d2ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3358153b60 0x7f33581ef710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:11.538 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335d2ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3358153b60 0x7f33581ef710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35154/0 (socket says 172.21.15.150:35154) 2024-09-17T04:49:11.538 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335d2ee640 1 -- 172.21.15.150:0/227581831 learned_addr learned my addr 172.21.15.150:0/227581831 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:11.538 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f3357fff640 1 --2- 172.21.15.150:0/227581831 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33580ab2d0 0x7f33580adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:11.538 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335caed640 1 --2- 172.21.15.150:0/227581831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335806de60 0x7f3358145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:11.538 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335d2ee640 1 -- 172.21.15.150:0/227581831 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33580ab2d0 msgr2=0x7f33580adbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:11.538 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335d2ee640 1 --2- 172.21.15.150:0/227581831 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33580ab2d0 0x7f33580adbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:11.539 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335d2ee640 1 -- 172.21.15.150:0/227581831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335806de60 msgr2=0x7f3358145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:11.539 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335d2ee640 1 --2- 172.21.15.150:0/227581831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335806de60 0x7f3358145930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:11.539 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335d2ee640 1 -- 172.21.15.150:0/227581831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33580ac770 con 0x7f3358153b60 2024-09-17T04:49:11.539 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335d2ee640 1 --2- 172.21.15.150:0/227581831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3358153b60 0x7f33581ef710 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f335000cab0 tx=0x7f335000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:11.539 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f33557fa640 1 -- 172.21.15.150:0/227581831 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335000adc0 con 0x7f3358153b60 2024-09-17T04:49:11.539 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f33557fa640 1 -- 172.21.15.150:0/227581831 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f335000f560 con 0x7f3358153b60 2024-09-17T04:49:11.539 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f33557fa640 1 -- 172.21.15.150:0/227581831 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335000f9b0 con 0x7f3358153b60 2024-09-17T04:49:11.539 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335ed51640 1 -- 172.21.15.150:0/227581831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f335814dbd0 con 0x7f3358153b60 2024-09-17T04:49:11.539 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.535+0000 7f335ed51640 1 -- 172.21.15.150:0/227581831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33580ac250 con 0x7f3358153b60 2024-09-17T04:49:11.540 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.539+0000 7f335ed51640 1 -- 172.21.15.150:0/227581831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33581f04a0 con 0x7f3358153b60 2024-09-17T04:49:11.540 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.539+0000 7f33557fa640 1 -- 172.21.15.150:0/227581831 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 2) ==== 48655+0+0 (secure 0 0 0) 0x7f335000fb50 con 0x7f3358153b60 2024-09-17T04:49:11.540 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.539+0000 7f33557fa640 1 -- 172.21.15.150:0/227581831 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f335004fb90 con 0x7f3358153b60 2024-09-17T04:49:11.542 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.543+0000 7f33557fa640 1 -- 172.21.15.150:0/227581831 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f33581f04a0 con 0x7f3358153b60 2024-09-17T04:49:11.602 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.599+0000 7f335ed51640 1 -- 172.21.15.150:0/227581831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3358147770 con 0x7f3358153b60 2024-09-17T04:49:11.602 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.603+0000 7f33557fa640 1 -- 172.21.15.150:0/227581831 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f3358147770 con 0x7f3358153b60 2024-09-17T04:49:11.603 INFO:teuthology.misc.health.smithi150.stdout: 2024-09-17T04:49:11.603 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":10,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:10.281212+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:11.604 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.603+0000 7f335ed51640 1 -- 172.21.15.150:0/227581831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3358153b60 msgr2=0x7f33581ef710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:11.604 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.603+0000 7f335ed51640 1 --2- 172.21.15.150:0/227581831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3358153b60 0x7f33581ef710 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f335000cab0 tx=0x7f335000cf70 comp rx=0 tx=0).stop 2024-09-17T04:49:11.604 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.603+0000 7f335ed51640 1 -- 172.21.15.150:0/227581831 shutdown_connections 2024-09-17T04:49:11.605 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.603+0000 7f335ed51640 1 --2- 172.21.15.150:0/227581831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3358153b60 0x7f33581ef710 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:11.605 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.603+0000 7f335ed51640 1 --2- 172.21.15.150:0/227581831 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33580ab2d0 0x7f33580adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:11.605 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.603+0000 7f335ed51640 1 --2- 172.21.15.150:0/227581831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f335806de60 0x7f3358145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:11.605 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.603+0000 7f335ed51640 1 -- 172.21.15.150:0/227581831 >> 172.21.15.150:0/227581831 conn(0x7f3358074190 msgr2=0x7f33580c26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:11.605 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.603+0000 7f335ed51640 1 -- 172.21.15.150:0/227581831 shutdown_connections 2024-09-17T04:49:11.605 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:11.603+0000 7f335ed51640 1 -- 172.21.15.150:0/227581831 wait complete. 2024-09-17T04:49:11.620 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-17T04:49:11.681 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-09-17T04:49:11.678+0000 7f1fb88dc7c0 -1 Falling back to public interface 2024-09-17T04:49:11.808 INFO:tasks.ceph.osd.0.smithi150.stderr:2024-09-17T04:49:11.807+0000 7faa2074d7c0 -1 Falling back to public interface 2024-09-17T04:49:11.839 INFO:tasks.ceph.osd.1.smithi150.stderr:2024-09-17T04:49:11.839+0000 7f80d63567c0 -1 Falling back to public interface 2024-09-17T04:49:11.951 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-09-17T04:49:11.950+0000 7fc117918640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-17T04:49:11.954 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-09-17T04:49:11.950+0000 7fc117918640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T04:49:11.989 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-09-17T04:49:11.986+0000 7f455287e7c0 -1 Falling back to public interface 2024-09-17T04:49:12.016 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-09-17T04:49:12.014+0000 7fa82902e7c0 -1 Falling back to public interface 2024-09-17T04:49:12.060 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-09-17T04:49:12.059+0000 7f3bf5b247c0 -1 Falling back to public interface 2024-09-17T04:49:12.220 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-09-17T04:49:12.219+0000 7f57e980d7c0 -1 Falling back to public interface 2024-09-17T04:49:12.233 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-09-17T04:49:12.230+0000 7f0711f887c0 -1 Falling back to public interface 2024-09-17T04:49:15.772 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-09-17T04:49:15.770+0000 7f1fb88dc7c0 -1 osd.6 0 log_to_monitors true 2024-09-17T04:49:15.863 INFO:tasks.ceph.osd.0.smithi150.stderr:2024-09-17T04:49:15.863+0000 7faa2074d7c0 -1 osd.0 0 log_to_monitors true 2024-09-17T04:49:15.891 INFO:tasks.ceph.osd.1.smithi150.stderr:2024-09-17T04:49:15.891+0000 7f80d63567c0 -1 osd.1 0 log_to_monitors true 2024-09-17T04:49:15.972 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-09-17T04:49:15.970+0000 7f455287e7c0 -1 osd.5 0 log_to_monitors true 2024-09-17T04:49:16.320 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-09-17T04:49:16.319+0000 7f3bf5b247c0 -1 osd.2 0 log_to_monitors true 2024-09-17T04:49:16.454 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-09-17T04:49:16.450+0000 7f0711f887c0 -1 osd.4 0 log_to_monitors true 2024-09-17T04:49:16.531 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-09-17T04:49:16.531+0000 7f57e980d7c0 -1 osd.3 0 log_to_monitors true 2024-09-17T04:49:16.584 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-09-17T04:49:16.582+0000 7fa82902e7c0 -1 osd.7 0 log_to_monitors true 2024-09-17T04:49:17.945 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-09-17T04:49:17.942+0000 7f1fb5097640 -1 osd.6 0 waiting for initial osdmap 2024-09-17T04:49:18.039 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-09-17T04:49:18.038+0000 7f1fac67f640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T04:49:18.426 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:18.594 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.590+0000 7f85d3dbe640 1 -- 172.21.15.150:0/2813983027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85cc0c1170 msgr2=0x7f85cc0c1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:18.594 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.590+0000 7f85d3dbe640 1 --2- 172.21.15.150:0/2813983027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85cc0c1170 0x7f85cc0c1580 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0c3a70 tx=0x7f85c801c800 comp rx=0 tx=0).stop 2024-09-17T04:49:18.594 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.590+0000 7f85d3dbe640 1 -- 172.21.15.150:0/2813983027 shutdown_connections 2024-09-17T04:49:18.594 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.590+0000 7f85d3dbe640 1 --2- 172.21.15.150:0/2813983027 >> v2:172.21.15.176:3300/0 conn(0x7f85cc0c6920 0x7f85cc0c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:18.594 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.590+0000 7f85d3dbe640 1 --2- 172.21.15.150:0/2813983027 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85cc1568f0 0x7f85cc156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:18.594 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.590+0000 7f85d3dbe640 1 --2- 172.21.15.150:0/2813983027 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85cc0c1170 0x7f85cc0c1580 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:18.594 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.590+0000 7f85d3dbe640 1 -- 172.21.15.150:0/2813983027 >> 172.21.15.150:0/2813983027 conn(0x7f85cc074750 msgr2=0x7f85cc074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:18.594 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.590+0000 7f85d3dbe640 1 -- 172.21.15.150:0/2813983027 shutdown_connections 2024-09-17T04:49:18.595 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.590+0000 7f85d3dbe640 1 -- 172.21.15.150:0/2813983027 wait complete. 2024-09-17T04:49:18.595 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d3dbe640 1 Processor -- start 2024-09-17T04:49:18.595 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d3dbe640 1 -- start start 2024-09-17T04:49:18.598 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d3dbe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85cc0c1170 0x7f85cc14dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:18.598 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d3dbe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85cc0c6920 0x7f85cc1f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:18.598 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d3dbe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85cc1568f0 0x7f85cc14ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:18.598 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d3dbe640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc0c45d0 con 0x7f85cc0c6920 2024-09-17T04:49:18.598 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d3dbe640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc157240 con 0x7f85cc1568f0 2024-09-17T04:49:18.598 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d3dbe640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f85cc0c4750 con 0x7f85cc0c1170 2024-09-17T04:49:18.598 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d1359640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85cc0c6920 0x7f85cc1f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:18.598 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d1359640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85cc0c6920 0x7f85cc1f3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35184/0 (socket says 172.21.15.150:35184) 2024-09-17T04:49:18.598 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d1359640 1 -- 172.21.15.150:0/1482092137 learned_addr learned my addr 172.21.15.150:0/1482092137 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:18.598 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d1b5a640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85cc0c1170 0x7f85cc14dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:18.599 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d235b640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85cc1568f0 0x7f85cc14ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:18.599 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d1359640 1 -- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85cc0c1170 msgr2=0x7f85cc14dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:18.599 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d1359640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85cc0c1170 0x7f85cc14dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:18.599 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d1359640 1 -- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85cc1568f0 msgr2=0x7f85cc14ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:18.599 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d1359640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85cc1568f0 0x7f85cc14ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:18.599 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d1359640 1 -- 172.21.15.150:0/1482092137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85cc1f3bc0 con 0x7f85cc0c6920 2024-09-17T04:49:18.599 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d1359640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85cc0c6920 0x7f85cc1f3290 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f85bc00e790 tx=0x7f85bc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:18.599 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85c27fc640 1 -- 172.21.15.150:0/1482092137 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc00eea0 con 0x7f85cc0c6920 2024-09-17T04:49:18.599 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85c27fc640 1 -- 172.21.15.150:0/1482092137 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85bc00cc60 con 0x7f85cc0c6920 2024-09-17T04:49:18.599 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d3dbe640 1 -- 172.21.15.150:0/1482092137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85cc1f3ea0 con 0x7f85cc0c6920 2024-09-17T04:49:18.599 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85c27fc640 1 -- 172.21.15.150:0/1482092137 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc005280 con 0x7f85cc0c6920 2024-09-17T04:49:18.599 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.594+0000 7f85d3dbe640 1 -- 172.21.15.150:0/1482092137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85cc150740 con 0x7f85cc0c6920 2024-09-17T04:49:18.600 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.598+0000 7f85c27fc640 1 -- 172.21.15.150:0/1482092137 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) ==== 97178+0+0 (secure 0 0 0) 0x7f85bc005420 con 0x7f85cc0c6920 2024-09-17T04:49:18.600 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.598+0000 7f85d3dbe640 1 -- 172.21.15.150:0/1482092137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85cc0b1f10 con 0x7f85cc0c6920 2024-09-17T04:49:18.600 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.598+0000 7f85c27fc640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85b0073170 0x7f85b0075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:18.600 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.598+0000 7f85c27fc640 1 -- 172.21.15.150:0/1482092137 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2231+0+0 (secure 0 0 0) 0x7f85bc093b40 con 0x7f85cc0c6920 2024-09-17T04:49:18.601 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.598+0000 7f85d1b5a640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85b0073170 0x7f85b0075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:18.601 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.598+0000 7f85d1b5a640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85b0073170 0x7f85b0075680 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f85c80af000 tx=0x7f85c80a7040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:18.605 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.602+0000 7f85c27fc640 1 -- 172.21.15.150:0/1482092137 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f85bc05f520 con 0x7f85cc0c6920 2024-09-17T04:49:18.817 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.814+0000 7f85d3dbe640 1 -- 172.21.15.150:0/1482092137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85cc0c8560 con 0x7f85cc0c6920 2024-09-17T04:49:18.818 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.814+0000 7f85c27fc640 1 -- 172.21.15.150:0/1482092137 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+7058 (secure 0 0 0) 0x7f85bc063f50 con 0x7f85cc0c6920 2024-09-17T04:49:18.818 INFO:teuthology.misc.health.smithi150.stdout: 2024-09-17T04:49:18.818 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":13,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:17.843607+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:18.821 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.818+0000 7f85d3dbe640 1 -- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85b0073170 msgr2=0x7f85b0075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:18.821 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.818+0000 7f85d3dbe640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85b0073170 0x7f85b0075680 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f85c80af000 tx=0x7f85c80a7040 comp rx=0 tx=0).stop 2024-09-17T04:49:18.821 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.818+0000 7f85d3dbe640 1 -- 172.21.15.150:0/1482092137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85cc0c6920 msgr2=0x7f85cc1f3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:18.821 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.818+0000 7f85d3dbe640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85cc0c6920 0x7f85cc1f3290 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f85bc00e790 tx=0x7f85bc00ec50 comp rx=0 tx=0).stop 2024-09-17T04:49:18.821 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.818+0000 7f85d3dbe640 1 -- 172.21.15.150:0/1482092137 shutdown_connections 2024-09-17T04:49:18.821 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.818+0000 7f85d3dbe640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85b0073170 0x7f85b0075680 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:18.821 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.818+0000 7f85d3dbe640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85cc1568f0 0x7f85cc14ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:18.821 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.818+0000 7f85d3dbe640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85cc0c6920 0x7f85cc1f3290 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:18.821 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.818+0000 7f85d3dbe640 1 --2- 172.21.15.150:0/1482092137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85cc0c1170 0x7f85cc14dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:18.821 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.818+0000 7f85d3dbe640 1 -- 172.21.15.150:0/1482092137 >> 172.21.15.150:0/1482092137 conn(0x7f85cc074750 msgr2=0x7f85cc1605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:18.822 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.818+0000 7f85d3dbe640 1 -- 172.21.15.150:0/1482092137 shutdown_connections 2024-09-17T04:49:18.822 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:18.818+0000 7f85d3dbe640 1 -- 172.21.15.150:0/1482092137 wait complete. 2024-09-17T04:49:18.841 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-17T04:49:19.613 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-09-17T04:49:19.610+0000 7f454e824640 -1 osd.5 0 waiting for initial osdmap 2024-09-17T04:49:19.636 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-09-17T04:49:19.634+0000 7f4546621640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T04:49:19.646 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-09-17T04:49:19.642+0000 7f070e743640 -1 osd.4 0 waiting for initial osdmap 2024-09-17T04:49:19.657 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-09-17T04:49:19.654+0000 7fa824fd4640 -1 osd.7 0 waiting for initial osdmap 2024-09-17T04:49:19.667 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-09-17T04:49:19.666+0000 7f0705d2b640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T04:49:19.683 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-09-17T04:49:19.682+0000 7fa81cdd1640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T04:49:19.955 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-09-17T04:49:19.954+0000 7f57e5fc8640 -1 osd.3 0 waiting for initial osdmap 2024-09-17T04:49:19.984 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-09-17T04:49:19.982+0000 7f57dd5b0640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T04:49:19.993 INFO:tasks.ceph.osd.0.smithi150.stderr:2024-09-17T04:49:19.990+0000 7faa1c6f3640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T04:49:20.019 INFO:tasks.ceph.osd.0.smithi150.stderr:2024-09-17T04:49:20.018+0000 7faa144f0640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T04:49:20.020 INFO:tasks.ceph.osd.1.smithi150.stderr:2024-09-17T04:49:20.018+0000 7f80d22fc640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T04:49:20.020 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-09-17T04:49:20.018+0000 7f3bf1aca640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T04:49:20.060 INFO:tasks.ceph.osd.1.smithi150.stderr:2024-09-17T04:49:20.058+0000 7f80ca0f9640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T04:49:20.060 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-09-17T04:49:20.058+0000 7f3be98c7640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T04:49:25.648 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:25.807 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.806+0000 7fa013de2640 1 -- 172.21.15.150:0/537996607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff8004810 msgr2=0x7f9ff8004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:25.807 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.806+0000 7fa013de2640 1 --2- 172.21.15.150:0/537996607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff8004810 0x7f9ff8004c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc00c260 tx=0x7f9ffc01cf00 comp rx=0 tx=0).stop 2024-09-17T04:49:25.807 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.806+0000 7fa013de2640 1 -- 172.21.15.150:0/537996607 shutdown_connections 2024-09-17T04:49:25.807 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.806+0000 7fa013de2640 1 --2- 172.21.15.150:0/537996607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ff8004810 0x7f9ff8004c20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:25.807 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.806+0000 7fa013de2640 1 -- 172.21.15.150:0/537996607 >> 172.21.15.150:0/537996607 conn(0x7fa00c070a90 msgr2=0x7fa00c070ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:25.807 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.806+0000 7fa013de2640 1 -- 172.21.15.150:0/537996607 shutdown_connections 2024-09-17T04:49:25.808 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.806+0000 7fa013de2640 1 -- 172.21.15.150:0/537996607 wait complete. 2024-09-17T04:49:25.808 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa013de2640 1 Processor -- start 2024-09-17T04:49:25.808 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa013de2640 1 -- start start 2024-09-17T04:49:25.811 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa013de2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ff8004810 0x7fa00c1f01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:25.811 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa013de2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa00c14f3e0 0x7fa00c14f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:25.811 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa013de2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c1499b0 0x7fa00c149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:25.811 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa013de2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa00c0715b0 con 0x7fa00c14f3e0 2024-09-17T04:49:25.811 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa013de2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa00c0abcf0 con 0x7fa00c1499b0 2024-09-17T04:49:25.811 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa013de2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa00c069da0 con 0x7f9ff8004810 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa01137d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa00c14f3e0 0x7fa00c14f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa01137d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa00c14f3e0 0x7fa00c14f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58386/0 (socket says 172.21.15.150:58386) 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa01137d640 1 -- 172.21.15.150:0/34395071 learned_addr learned my addr 172.21.15.150:0/34395071 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa01237f640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c1499b0 0x7fa00c149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa011b7e640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ff8004810 0x7fa00c1f01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa01137d640 1 -- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ff8004810 msgr2=0x7fa00c1f01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa01137d640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ff8004810 0x7fa00c1f01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa01137d640 1 -- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c1499b0 msgr2=0x7fa00c149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa01137d640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c1499b0 0x7fa00c149dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa01137d640 1 -- 172.21.15.150:0/34395071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00c0c7730 con 0x7fa00c14f3e0 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa011b7e640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ff8004810 0x7fa00c1f01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa01137d640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa00c14f3e0 0x7fa00c14f7f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc098980 tx=0x7f9ffc001c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa0027fc640 1 -- 172.21.15.150:0/34395071 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc0014c0 con 0x7fa00c14f3e0 2024-09-17T04:49:25.812 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa0027fc640 1 -- 172.21.15.150:0/34395071 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ffc001660 con 0x7fa00c14f3e0 2024-09-17T04:49:25.813 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa013de2640 1 -- 172.21.15.150:0/34395071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa00c0c7a10 con 0x7fa00c14f3e0 2024-09-17T04:49:25.813 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa0027fc640 1 -- 172.21.15.150:0/34395071 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc0064e0 con 0x7fa00c14f3e0 2024-09-17T04:49:25.813 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.810+0000 7fa013de2640 1 -- 172.21.15.150:0/34395071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa00c15c810 con 0x7fa00c14f3e0 2024-09-17T04:49:25.813 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.814+0000 7fa013de2640 1 -- 172.21.15.150:0/34395071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa00c150fc0 con 0x7fa00c14f3e0 2024-09-17T04:49:25.814 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.814+0000 7fa0027fc640 1 -- 172.21.15.150:0/34395071 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9ffc003600 con 0x7fa00c14f3e0 2024-09-17T04:49:25.814 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.814+0000 7fa0027fc640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ff0073230 0x7f9ff0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:25.814 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.814+0000 7fa0027fc640 1 -- 172.21.15.150:0/34395071 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f9ffc0b4020 con 0x7fa00c14f3e0 2024-09-17T04:49:25.814 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.814+0000 7fa011b7e640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ff0073230 0x7f9ff0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:25.814 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.814+0000 7fa011b7e640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ff0073230 0x7f9ff0075740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa00c075030 tx=0x7fa008006a40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:25.818 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.818+0000 7fa0027fc640 1 -- 172.21.15.150:0/34395071 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9ffc016cf0 con 0x7fa00c14f3e0 2024-09-17T04:49:25.986 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.986+0000 7fa013de2640 1 -- 172.21.15.150:0/34395071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa00c14a510 con 0x7fa00c14f3e0 2024-09-17T04:49:25.987 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.986+0000 7fa0027fc640 1 -- 172.21.15.150:0/34395071 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12932 (secure 0 0 0) 0x7f9ffc0f8ea0 con 0x7fa00c14f3e0 2024-09-17T04:49:25.987 INFO:teuthology.misc.health.smithi150.stdout: 2024-09-17T04:49:25.987 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":18,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:23.669463+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:25.989 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.990+0000 7fa013de2640 1 -- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ff0073230 msgr2=0x7f9ff0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:25.989 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.990+0000 7fa013de2640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ff0073230 0x7f9ff0075740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa00c075030 tx=0x7fa008006a40 comp rx=0 tx=0).stop 2024-09-17T04:49:25.989 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.990+0000 7fa013de2640 1 -- 172.21.15.150:0/34395071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa00c14f3e0 msgr2=0x7fa00c14f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:25.989 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.990+0000 7fa013de2640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa00c14f3e0 0x7fa00c14f7f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc098980 tx=0x7f9ffc001c80 comp rx=0 tx=0).stop 2024-09-17T04:49:25.990 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.990+0000 7fa013de2640 1 -- 172.21.15.150:0/34395071 shutdown_connections 2024-09-17T04:49:25.990 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.990+0000 7fa013de2640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ff0073230 0x7f9ff0075740 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:25.990 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.990+0000 7fa013de2640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c1499b0 0x7fa00c149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:25.990 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.990+0000 7fa013de2640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa00c14f3e0 0x7fa00c14f7f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:25.990 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.990+0000 7fa013de2640 1 --2- 172.21.15.150:0/34395071 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ff8004810 0x7fa00c1f01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:25.990 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.990+0000 7fa013de2640 1 -- 172.21.15.150:0/34395071 >> 172.21.15.150:0/34395071 conn(0x7fa00c070a90 msgr2=0x7fa00c0aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:25.990 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.990+0000 7fa013de2640 1 -- 172.21.15.150:0/34395071 shutdown_connections 2024-09-17T04:49:25.990 INFO:teuthology.misc.health.smithi150.stderr:2024-09-17T04:49:25.990+0000 7fa013de2640 1 -- 172.21.15.150:0/34395071 wait complete. 2024-09-17T04:49:26.006 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-17T04:49:26.007 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-17T04:49:26.007 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:49:26.007 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:49:26.010 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:49:26.010 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:49:26.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:49:26.012 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:49:26.012 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T04:49:26.012 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T04:49:26.015 INFO:tasks.ceph.mds.b:Started 2024-09-17T04:49:26.015 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:49:26.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:49:26.018 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:49:26.018 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-17T04:49:26.018 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:49:26.120 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T04:49:26.120 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T04:49:26.130 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-09-17T04:49:26.135 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-09-17T04:49:26.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.166+0000 7fb788f6b640 1 -- 172.21.15.150:0/3415744507 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb78406fca0 msgr2=0x7fb784153790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:26.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.166+0000 7fb788f6b640 1 --2- 172.21.15.150:0/3415744507 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb78406fca0 0x7fb784153790 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb774009840 tx=0x7fb77401c790 comp rx=0 tx=0).stop 2024-09-17T04:49:26.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.166+0000 7fb788f6b640 1 -- 172.21.15.150:0/3415744507 shutdown_connections 2024-09-17T04:49:26.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.166+0000 7fb788f6b640 1 --2- 172.21.15.150:0/3415744507 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb78406fca0 0x7fb784153790 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:26.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.166+0000 7fb788f6b640 1 -- 172.21.15.150:0/3415744507 >> 172.21.15.150:0/3415744507 conn(0x7fb7840c5ae0 msgr2=0x7fb784073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:26.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.166+0000 7fb788f6b640 1 -- 172.21.15.150:0/3415744507 shutdown_connections 2024-09-17T04:49:26.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.166+0000 7fb788f6b640 1 -- 172.21.15.150:0/3415744507 wait complete. 2024-09-17T04:49:26.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb788f6b640 1 Processor -- start 2024-09-17T04:49:26.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb788f6b640 1 -- start start 2024-09-17T04:49:26.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb788f6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb78406fca0 0x7fb7841486d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:26.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb788f6b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb784074570 0x7fb7840ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:26.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb788f6b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb784074980 0x7fb7840acd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:26.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb788f6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb784160830 con 0x7fb78406fca0 2024-09-17T04:49:26.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb788f6b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb7841606b0 con 0x7fb784074980 2024-09-17T04:49:26.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb788f6b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb7841609b0 con 0x7fb784074570 2024-09-17T04:49:26.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb78259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb78406fca0 0x7fb7841486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:26.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb78259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb78406fca0 0x7fb7841486d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58410/0 (socket says 172.21.15.150:58410) 2024-09-17T04:49:26.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb78259c640 1 -- 172.21.15.150:0/3131917377 learned_addr learned my addr 172.21.15.150:0/3131917377 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:26.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb782d9d640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb784074980 0x7fb7840acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:26.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb781d9b640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb784074570 0x7fb7840ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:26.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb78259c640 1 -- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb784074570 msgr2=0x7fb7840ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:26.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb78259c640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb784074570 0x7fb7840ac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:26.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb78259c640 1 -- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb784074980 msgr2=0x7fb7840acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:26.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb78259c640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb784074980 0x7fb7840acd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:26.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb78259c640 1 -- 172.21.15.150:0/3131917377 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78414e7e0 con 0x7fb78406fca0 2024-09-17T04:49:26.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb78259c640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb78406fca0 0x7fb7841486d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb774009790 tx=0x7fb774005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:26.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb772ffd640 1 -- 172.21.15.150:0/3131917377 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7740b2040 con 0x7fb78406fca0 2024-09-17T04:49:26.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb772ffd640 1 -- 172.21.15.150:0/3131917377 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7740043e0 con 0x7fb78406fca0 2024-09-17T04:49:26.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb788f6b640 1 -- 172.21.15.150:0/3131917377 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7841423d0 con 0x7fb78406fca0 2024-09-17T04:49:26.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb772ffd640 1 -- 172.21.15.150:0/3131917377 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7740052d0 con 0x7fb78406fca0 2024-09-17T04:49:26.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb788f6b640 1 -- 172.21.15.150:0/3131917377 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7841482c0 con 0x7fb78406fca0 2024-09-17T04:49:26.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.170+0000 7fb788f6b640 1 -- 172.21.15.150:0/3131917377 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb784153790 con 0x7fb78406fca0 2024-09-17T04:49:26.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.174+0000 7fb772ffd640 1 -- 172.21.15.150:0/3131917377 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb774005470 con 0x7fb78406fca0 2024-09-17T04:49:26.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.174+0000 7fb772ffd640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb764073200 0x7fb764075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:26.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.174+0000 7fb772ffd640 1 -- 172.21.15.150:0/3131917377 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fb77412db70 con 0x7fb78406fca0 2024-09-17T04:49:26.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.174+0000 7fb781d9b640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb764073200 0x7fb764075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:26.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.174+0000 7fb781d9b640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb764073200 0x7fb764075710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb78415ded0 tx=0x7fb778009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:26.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.178+0000 7fb772ffd640 1 -- 172.21.15.150:0/3131917377 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb7740bd060 con 0x7fb78406fca0 2024-09-17T04:49:26.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 -- 172.21.15.150:0/3131917377 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7840c6590 con 0x7fb78406fca0 2024-09-17T04:49:26.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb772ffd640 1 -- 172.21.15.150:0/3131917377 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1437 (secure 0 0 0) 0x7fb774016410 con 0x7fb78406fca0 2024-09-17T04:49:26.368 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:26.369 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":2,"btime":"2024-09-17T04:49:26:156750+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4404,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-17T04:49:26.369 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 2 2024-09-17T04:49:26.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 -- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb764073200 msgr2=0x7fb764075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:26.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb764073200 0x7fb764075710 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb78415ded0 tx=0x7fb778009040 comp rx=0 tx=0).stop 2024-09-17T04:49:26.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 -- 172.21.15.150:0/3131917377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb78406fca0 msgr2=0x7fb7841486d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:26.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb78406fca0 0x7fb7841486d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb774009790 tx=0x7fb774005d90 comp rx=0 tx=0).stop 2024-09-17T04:49:26.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 -- 172.21.15.150:0/3131917377 shutdown_connections 2024-09-17T04:49:26.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb764073200 0x7fb764075710 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:26.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb784074980 0x7fb7840acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:26.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb784074570 0x7fb7840ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:26.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 --2- 172.21.15.150:0/3131917377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb78406fca0 0x7fb7841486d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:26.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 -- 172.21.15.150:0/3131917377 >> 172.21.15.150:0/3131917377 conn(0x7fb7840c5ae0 msgr2=0x7fb7840c7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:26.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 -- 172.21.15.150:0/3131917377 shutdown_connections 2024-09-17T04:49:26.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:26.370+0000 7fb788f6b640 1 -- 172.21.15.150:0/3131917377 wait complete. 2024-09-17T04:49:27.914 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:49:27.916 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:49:28.389 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:49:28.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.553+0000 7f90f8f7c640 1 -- 172.21.15.150:0/1638970368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90f414a180 msgr2=0x7f90f40c7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:28.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.553+0000 7f90f8f7c640 1 --2- 172.21.15.150:0/1638970368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90f414a180 0x7f90f40c7190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f90e4009900 tx=0x7f90e401c8a0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.553+0000 7f90f8f7c640 1 -- 172.21.15.150:0/1638970368 shutdown_connections 2024-09-17T04:49:28.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.553+0000 7f90f8f7c640 1 --2- 172.21.15.150:0/1638970368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90f414a180 0x7f90f40c7190 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.553+0000 7f90f8f7c640 1 --2- 172.21.15.150:0/1638970368 >> v2:172.21.15.176:3300/0 conn(0x7f90f40c7eb0 0x7f90f41540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.553+0000 7f90f8f7c640 1 -- 172.21.15.150:0/1638970368 >> 172.21.15.150:0/1638970368 conn(0x7f90f40aab60 msgr2=0x7f90f40aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:28.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.553+0000 7f90f8f7c640 1 -- 172.21.15.150:0/1638970368 shutdown_connections 2024-09-17T04:49:28.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.553+0000 7f90f8f7c640 1 -- 172.21.15.150:0/1638970368 wait complete. 2024-09-17T04:49:28.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f8f7c640 1 Processor -- start 2024-09-17T04:49:28.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f8f7c640 1 -- start start 2024-09-17T04:49:28.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f8f7c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90f40c7eb0 0x7f90f414c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:28.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f8f7c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f90f4149d70 0x7f90f4148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:28.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f8f7c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90f414a180 0x7f90f40c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:28.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f8f7c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f90f40c7c60 con 0x7f90f414a180 2024-09-17T04:49:28.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f8f7c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f90f40c7ae0 con 0x7f90f40c7eb0 2024-09-17T04:49:28.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f8f7c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f90f4156540 con 0x7f90f4149d70 2024-09-17T04:49:28.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90f414a180 0x7f90f40c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:28.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90f414a180 0x7f90f40c21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58440/0 (socket says 172.21.15.150:58440) 2024-09-17T04:49:28.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f2d9d640 1 -- 172.21.15.150:0/960183419 learned_addr learned my addr 172.21.15.150:0/960183419 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:28.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f259c640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90f40c7eb0 0x7f90f414c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:28.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f1d9b640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f90f4149d70 0x7f90f4148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:28.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f2d9d640 1 -- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f90f4149d70 msgr2=0x7f90f4148b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:28.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f2d9d640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f90f4149d70 0x7f90f4148b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f2d9d640 1 -- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90f40c7eb0 msgr2=0x7f90f414c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:28.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f2d9d640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90f40c7eb0 0x7f90f414c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f2d9d640 1 -- 172.21.15.150:0/960183419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f414ce60 con 0x7f90f414a180 2024-09-17T04:49:28.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f1d9b640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f90f4149d70 0x7f90f4148b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:28.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f2d9d640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90f414a180 0x7f90f40c21e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f90e401d140 tx=0x7f90e40a5520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:28.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90e2ffd640 1 -- 172.21.15.150:0/960183419 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e40b2040 con 0x7f90f414a180 2024-09-17T04:49:28.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90e2ffd640 1 -- 172.21.15.150:0/960183419 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90e40041b0 con 0x7f90f414a180 2024-09-17T04:49:28.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f8f7c640 1 -- 172.21.15.150:0/960183419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90f414ae60 con 0x7f90f414a180 2024-09-17T04:49:28.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90e2ffd640 1 -- 172.21.15.150:0/960183419 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4003600 con 0x7f90f414a180 2024-09-17T04:49:28.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f8f7c640 1 -- 172.21.15.150:0/960183419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90f41f09a0 con 0x7f90f414a180 2024-09-17T04:49:28.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.557+0000 7f90f8f7c640 1 -- 172.21.15.150:0/960183419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90f41f0d80 con 0x7f90f414a180 2024-09-17T04:49:28.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.561+0000 7f90e2ffd640 1 -- 172.21.15.150:0/960183419 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f90e40bd020 con 0x7f90f414a180 2024-09-17T04:49:28.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.561+0000 7f90e2ffd640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f90d4073230 0x7f90d4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:28.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.561+0000 7f90e2ffd640 1 -- 172.21.15.150:0/960183419 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f90e412dae0 con 0x7f90f414a180 2024-09-17T04:49:28.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.561+0000 7f90f259c640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f90d4073230 0x7f90d4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:28.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.561+0000 7f90f259c640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f90d4073230 0x7f90d4075740 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f90ec005fd0 tx=0x7f90ec00a3b0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:28.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.565+0000 7f90e2ffd640 1 -- 172.21.15.150:0/960183419 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f90f41f0d80 con 0x7f90f414a180 2024-09-17T04:49:28.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.769+0000 7f90f8f7c640 1 -- 172.21.15.150:0/960183419 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90f40c7190 con 0x7f90f414a180 2024-09-17T04:49:28.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.769+0000 7f90e2ffd640 1 -- 172.21.15.150:0/960183419 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3912 (secure 0 0 0) 0x7f90e40c4020 con 0x7f90f414a180 2024-09-17T04:49:28.771 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:28.771 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":3,"btime":"2024-09-17T04:49:27:158005+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4404,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-17T04:49:28.771 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 3 2024-09-17T04:49:28.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.773+0000 7f90f8f7c640 1 -- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f90d4073230 msgr2=0x7f90d4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:28.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.773+0000 7f90f8f7c640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f90d4073230 0x7f90d4075740 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f90ec005fd0 tx=0x7f90ec00a3b0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.773+0000 7f90f8f7c640 1 -- 172.21.15.150:0/960183419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90f414a180 msgr2=0x7f90f40c21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:28.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.773+0000 7f90f8f7c640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90f414a180 0x7f90f40c21e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f90e401d140 tx=0x7f90e40a5520 comp rx=0 tx=0).stop 2024-09-17T04:49:28.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.773+0000 7f90f8f7c640 1 -- 172.21.15.150:0/960183419 shutdown_connections 2024-09-17T04:49:28.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.773+0000 7f90f8f7c640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f90d4073230 0x7f90d4075740 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.773+0000 7f90f8f7c640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90f414a180 0x7f90f40c21e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.773+0000 7f90f8f7c640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f90f4149d70 0x7f90f4148b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.773+0000 7f90f8f7c640 1 --2- 172.21.15.150:0/960183419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90f40c7eb0 0x7f90f414c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.773+0000 7f90f8f7c640 1 -- 172.21.15.150:0/960183419 >> 172.21.15.150:0/960183419 conn(0x7f90f40aab60 msgr2=0x7f90f415e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:28.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.773+0000 7f90f8f7c640 1 -- 172.21.15.150:0/960183419 shutdown_connections 2024-09-17T04:49:28.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.773+0000 7f90f8f7c640 1 -- 172.21.15.150:0/960183419 wait complete. 2024-09-17T04:49:28.792 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T04:49:28.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.949+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3602788550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b8070e50 msgr2=0x7f96b8071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:28.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.949+0000 7f96bfb8d640 1 --2- 172.21.15.150:0/3602788550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b8070e50 0x7f96b8071260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f96b8072370 tx=0x7f96a801c690 comp rx=0 tx=0).stop 2024-09-17T04:49:28.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.949+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3602788550 shutdown_connections 2024-09-17T04:49:28.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.949+0000 7f96bfb8d640 1 --2- 172.21.15.150:0/3602788550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b8070e50 0x7f96b8071260 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.949+0000 7f96bfb8d640 1 --2- 172.21.15.150:0/3602788550 >> v2:172.21.15.176:3300/0 conn(0x7f96b8073440 0x7f96b80a7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.949+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3602788550 >> 172.21.15.150:0/3602788550 conn(0x7f96b80c4a00 msgr2=0x7f96b80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:28.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.949+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3602788550 shutdown_connections 2024-09-17T04:49:28.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.949+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3602788550 wait complete. 2024-09-17T04:49:28.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.949+0000 7f96bfb8d640 1 Processor -- start 2024-09-17T04:49:28.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.949+0000 7f96bfb8d640 1 -- start start 2024-09-17T04:49:28.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bfb8d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b8070e50 0x7f96b8154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:28.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bfb8d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96b80717a0 0x7f96b81efd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:28.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bfb8d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96b8073440 0x7f96b81570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:28.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bfb8d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f96b8074f70 con 0x7f96b8070e50 2024-09-17T04:49:28.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bfb8d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f96b80adef0 con 0x7f96b80717a0 2024-09-17T04:49:28.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bfb8d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f96b80750f0 con 0x7f96b8073440 2024-09-17T04:49:28.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bd929640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b8070e50 0x7f96b8154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:28.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bd929640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b8070e50 0x7f96b8154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58454/0 (socket says 172.21.15.150:58454) 2024-09-17T04:49:28.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bd929640 1 -- 172.21.15.150:0/3317544554 learned_addr learned my addr 172.21.15.150:0/3317544554 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:28.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bd128640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96b80717a0 0x7f96b81efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:28.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96be12a640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96b8073440 0x7f96b81570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:28.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bd929640 1 -- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96b8073440 msgr2=0x7f96b81570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:28.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bd929640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96b8073440 0x7f96b81570c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bd929640 1 -- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96b80717a0 msgr2=0x7f96b81efd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:28.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bd929640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96b80717a0 0x7f96b81efd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:28.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bd929640 1 -- 172.21.15.150:0/3317544554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b81ef840 con 0x7f96b8070e50 2024-09-17T04:49:28.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bd128640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96b80717a0 0x7f96b81efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:28.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bd929640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b8070e50 0x7f96b8154b60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f96b400d7a0 tx=0x7f96b400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:28.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96ae7fc640 1 -- 172.21.15.150:0/3317544554 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b4017070 con 0x7f96b8070e50 2024-09-17T04:49:28.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96ae7fc640 1 -- 172.21.15.150:0/3317544554 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96b4009d20 con 0x7f96b8070e50 2024-09-17T04:49:28.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3317544554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b8151530 con 0x7f96b8070e50 2024-09-17T04:49:28.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96ae7fc640 1 -- 172.21.15.150:0/3317544554 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b40106c0 con 0x7f96b8070e50 2024-09-17T04:49:28.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3317544554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b81f0760 con 0x7f96b8070e50 2024-09-17T04:49:28.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3317544554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b80ad8c0 con 0x7f96b8070e50 2024-09-17T04:49:28.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96ae7fc640 1 -- 172.21.15.150:0/3317544554 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f96b401a070 con 0x7f96b8070e50 2024-09-17T04:49:28.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.953+0000 7f96ae7fc640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f969c073230 0x7f969c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:28.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.957+0000 7f96ae7fc640 1 -- 172.21.15.150:0/3317544554 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f96b40943f0 con 0x7f96b8070e50 2024-09-17T04:49:28.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.957+0000 7f96bd128640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f969c073230 0x7f969c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:28.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.957+0000 7f96bd128640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f969c073230 0x7f969c075740 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f96a80af000 tx=0x7f96a801aed0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:28.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:28.961+0000 7f96ae7fc640 1 -- 172.21.15.150:0/3317544554 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f96b81ef840 con 0x7f96b8070e50 2024-09-17T04:49:29.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.133+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3317544554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f96b80c60f0 con 0x7f96b8070e50 2024-09-17T04:49:29.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.133+0000 7f96ae7fc640 1 -- 172.21.15.150:0/3317544554 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) ==== 84+0+65 (secure 0 0 0) 0x7f96b405f880 con 0x7f96b8070e50 2024-09-17T04:49:29.133 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:29.133 INFO:teuthology.orchestra.run.smithi150.stdout:[ 2024-09-17T04:49:29.133 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-17T04:49:29.133 INFO:teuthology.orchestra.run.smithi150.stdout: "poolnum": 1, 2024-09-17T04:49:29.133 INFO:teuthology.orchestra.run.smithi150.stdout: "poolname": ".mgr" 2024-09-17T04:49:29.134 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-17T04:49:29.134 INFO:teuthology.orchestra.run.smithi150.stdout:] 2024-09-17T04:49:29.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.137+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f969c073230 msgr2=0x7f969c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:29.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.137+0000 7f96bfb8d640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f969c073230 0x7f969c075740 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f96a80af000 tx=0x7f96a801aed0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.137+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3317544554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b8070e50 msgr2=0x7f96b8154b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:29.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.137+0000 7f96bfb8d640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b8070e50 0x7f96b8154b60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f96b400d7a0 tx=0x7f96b400dc60 comp rx=0 tx=0).stop 2024-09-17T04:49:29.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.137+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3317544554 shutdown_connections 2024-09-17T04:49:29.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.137+0000 7f96bfb8d640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f969c073230 0x7f969c075740 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.137+0000 7f96bfb8d640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96b80717a0 0x7f96b81efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.137+0000 7f96bfb8d640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96b8073440 0x7f96b81570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.137+0000 7f96bfb8d640 1 --2- 172.21.15.150:0/3317544554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b8070e50 0x7f96b8154b60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.137+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3317544554 >> 172.21.15.150:0/3317544554 conn(0x7f96b80c4a00 msgr2=0x7f96b80c2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:29.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.137+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3317544554 shutdown_connections 2024-09-17T04:49:29.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.137+0000 7f96bfb8d640 1 -- 172.21.15.150:0/3317544554 wait complete. 2024-09-17T04:49:29.154 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'a' 2024-09-17T04:49:29.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_metadata --pg_num_min 64 2024-09-17T04:49:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.305+0000 7f4f23cce640 1 -- 172.21.15.150:0/2343620650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f080053b0 msgr2=0x7f4f0800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.305+0000 7f4f23cce640 1 --2- 172.21.15.150:0/2343620650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f080053b0 0x7f4f0800ba60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4f140016f0 tx=0x7f4f1401c7b0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.305+0000 7f4f23cce640 1 -- 172.21.15.150:0/2343620650 shutdown_connections 2024-09-17T04:49:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.305+0000 7f4f23cce640 1 --2- 172.21.15.150:0/2343620650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f080053b0 0x7f4f0800ba60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.305+0000 7f4f23cce640 1 -- 172.21.15.150:0/2343620650 >> 172.21.15.150:0/2343620650 conn(0x7f4f1c0c5940 msgr2=0x7f4f1c0c5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.305+0000 7f4f23cce640 1 -- 172.21.15.150:0/2343620650 shutdown_connections 2024-09-17T04:49:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.305+0000 7f4f23cce640 1 -- 172.21.15.150:0/2343620650 wait complete. 2024-09-17T04:49:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.305+0000 7f4f23cce640 1 Processor -- start 2024-09-17T04:49:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.305+0000 7f4f23cce640 1 -- start start 2024-09-17T04:49:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f23cce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f080053b0 0x7f4f1c1428d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f23cce640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f1c142f70 0x7f4f1c06d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f23cce640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f1c0c2270 0x7f4f1c0c2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f23cce640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c149f40 con 0x7f4f1c0c2270 2024-09-17T04:49:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f23cce640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1c149dc0 con 0x7f4f1c142f70 2024-09-17T04:49:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f23cce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4f1c0c7e00 con 0x7f4f080053b0 2024-09-17T04:49:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f2226b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f1c0c2270 0x7f4f1c0c2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f2226b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f1c0c2270 0x7f4f1c0c2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42906/0 (socket says 172.21.15.150:42906) 2024-09-17T04:49:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f2226b640 1 -- 172.21.15.150:0/2269774140 learned_addr learned my addr 172.21.15.150:0/2269774140 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f21269640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f1c142f70 0x7f4f1c06d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f21a6a640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f080053b0 0x7f4f1c1428d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f2226b640 1 -- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f080053b0 msgr2=0x7f4f1c1428d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f2226b640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f080053b0 0x7f4f1c1428d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f2226b640 1 -- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f1c142f70 msgr2=0x7f4f1c06d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f2226b640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f1c142f70 0x7f4f1c06d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f2226b640 1 -- 172.21.15.150:0/2269774140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f1c0bea80 con 0x7f4f1c0c2270 2024-09-17T04:49:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f21a6a640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f080053b0 0x7f4f1c1428d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f2226b640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f1c0c2270 0x7f4f1c0c2680 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4f1401d050 tx=0x7f4f14004c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f127fc640 1 -- 172.21.15.150:0/2269774140 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f140ac070 con 0x7f4f1c0c2270 2024-09-17T04:49:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f127fc640 1 -- 172.21.15.150:0/2269774140 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f140b7070 con 0x7f4f1c0c2270 2024-09-17T04:49:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f23cce640 1 -- 172.21.15.150:0/2269774140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f1c0aee00 con 0x7f4f1c0c2270 2024-09-17T04:49:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f127fc640 1 -- 172.21.15.150:0/2269774140 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f14006930 con 0x7f4f1c0c2270 2024-09-17T04:49:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f23cce640 1 -- 172.21.15.150:0/2269774140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f1c1ebb60 con 0x7f4f1c0c2270 2024-09-17T04:49:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f23cce640 1 -- 172.21.15.150:0/2269774140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f1c1535e0 con 0x7f4f1c0c2270 2024-09-17T04:49:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f127fc640 1 -- 172.21.15.150:0/2269774140 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4f140bc460 con 0x7f4f1c0c2270 2024-09-17T04:49:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f127fc640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f00073230 0x7f4f00075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.309+0000 7f4f127fc640 1 -- 172.21.15.150:0/2269774140 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f4f1412da10 con 0x7f4f1c0c2270 2024-09-17T04:49:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.313+0000 7f4f21a6a640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f00073230 0x7f4f00075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.313+0000 7f4f21a6a640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f00073230 0x7f4f00075740 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f18003f40 tx=0x7f4f18009500 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.317+0000 7f4f127fc640 1 -- 172.21.15.150:0/2269774140 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4f14016510 con 0x7f4f1c0c2270 2024-09-17T04:49:29.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.473+0000 7f4f23cce640 1 -- 172.21.15.150:0/2269774140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64} v 0) -- 0x7f4f1c150e60 con 0x7f4f1c0c2270 2024-09-17T04:49:29.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.833+0000 7f4f127fc640 1 -- 172.21.15.150:0/2269774140 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64}]=0 pool 'a_metadata' created v19) ==== 128+0+0 (secure 0 0 0) 0x7f4f140f8ea0 con 0x7f4f1c0c2270 2024-09-17T04:49:29.835 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'a_metadata' created 2024-09-17T04:49:29.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.837+0000 7f4f23cce640 1 -- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f00073230 msgr2=0x7f4f00075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:29.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.837+0000 7f4f23cce640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f00073230 0x7f4f00075740 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f18003f40 tx=0x7f4f18009500 comp rx=0 tx=0).stop 2024-09-17T04:49:29.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.837+0000 7f4f23cce640 1 -- 172.21.15.150:0/2269774140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f1c0c2270 msgr2=0x7f4f1c0c2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:29.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.837+0000 7f4f23cce640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f1c0c2270 0x7f4f1c0c2680 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4f1401d050 tx=0x7f4f14004c50 comp rx=0 tx=0).stop 2024-09-17T04:49:29.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.837+0000 7f4f23cce640 1 -- 172.21.15.150:0/2269774140 shutdown_connections 2024-09-17T04:49:29.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.837+0000 7f4f23cce640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f00073230 0x7f4f00075740 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.837+0000 7f4f23cce640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f1c142f70 0x7f4f1c06d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.837+0000 7f4f23cce640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f1c0c2270 0x7f4f1c0c2680 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.837+0000 7f4f23cce640 1 --2- 172.21.15.150:0/2269774140 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f080053b0 0x7f4f1c1428d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:29.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.837+0000 7f4f23cce640 1 -- 172.21.15.150:0/2269774140 >> 172.21.15.150:0/2269774140 conn(0x7f4f1c0c5940 msgr2=0x7f4f1c074da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:29.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.837+0000 7f4f23cce640 1 -- 172.21.15.150:0/2269774140 shutdown_connections 2024-09-17T04:49:29.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:29.837+0000 7f4f23cce640 1 -- 172.21.15.150:0/2269774140 wait complete. 2024-09-17T04:49:29.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T04:49:30.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 -- 172.21.15.150:0/1284251099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f41505a0 msgr2=0x7f60f4152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:30.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 --2- 172.21.15.150:0/1284251099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f41505a0 0x7f60f4152cf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f60f406ffd0 tx=0x7f60ec01cb80 comp rx=0 tx=0).stop 2024-09-17T04:49:30.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 -- 172.21.15.150:0/1284251099 shutdown_connections 2024-09-17T04:49:30.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 --2- 172.21.15.150:0/1284251099 >> v2:172.21.15.176:3300/0 conn(0x7f60f40a7350 0x7f60f4150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:30.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 --2- 172.21.15.150:0/1284251099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f41505a0 0x7f60f4152cf0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:30.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 -- 172.21.15.150:0/1284251099 >> 172.21.15.150:0/1284251099 conn(0x7f60f40c4a00 msgr2=0x7f60f40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:30.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 -- 172.21.15.150:0/1284251099 shutdown_connections 2024-09-17T04:49:30.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 -- 172.21.15.150:0/1284251099 wait complete. 2024-09-17T04:49:30.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 Processor -- start 2024-09-17T04:49:30.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 -- start start 2024-09-17T04:49:30.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f40a7350 0x7f60f4147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:30.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60f40c42e0 0x7f60f4151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:30.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60f41505a0 0x7f60f40aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:30.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f60f4153640 con 0x7f60f40a7350 2024-09-17T04:49:30.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f60f4070b80 con 0x7f60f40c42e0 2024-09-17T04:49:30.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.037+0000 7f60f964f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f60f41537c0 con 0x7f60f41505a0 2024-09-17T04:49:30.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f40a7350 0x7f60f4147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:30.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f40a7350 0x7f60f4147a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42920/0 (socket says 172.21.15.150:42920) 2024-09-17T04:49:30.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f2ffd640 1 -- 172.21.15.150:0/1944736683 learned_addr learned my addr 172.21.15.150:0/1944736683 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f37fe640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60f41505a0 0x7f60f40aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f27fc640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60f40c42e0 0x7f60f4151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f2ffd640 1 -- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60f41505a0 msgr2=0x7f60f40aaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f2ffd640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60f41505a0 0x7f60f40aaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f2ffd640 1 -- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60f40c42e0 msgr2=0x7f60f4151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f2ffd640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60f40c42e0 0x7f60f4151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f2ffd640 1 -- 172.21.15.150:0/1944736683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f41556e0 con 0x7f60f40a7350 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f2ffd640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f40a7350 0x7f60f4147a90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f60e400c460 tx=0x7f60e400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60db7fe640 1 -- 172.21.15.150:0/1944736683 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e4016020 con 0x7f60f40a7350 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60db7fe640 1 -- 172.21.15.150:0/1944736683 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60e4012940 con 0x7f60f40a7350 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f964f640 1 -- 172.21.15.150:0/1944736683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f4152a50 con 0x7f60f40a7350 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60db7fe640 1 -- 172.21.15.150:0/1944736683 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e4012d90 con 0x7f60f40a7350 2024-09-17T04:49:30.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f964f640 1 -- 172.21.15.150:0/1944736683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f41ed2f0 con 0x7f60f40a7350 2024-09-17T04:49:30.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f964f640 1 -- 172.21.15.150:0/1944736683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60f4150060 con 0x7f60f40a7350 2024-09-17T04:49:30.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60db7fe640 1 -- 172.21.15.150:0/1944736683 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f60e4019020 con 0x7f60f40a7350 2024-09-17T04:49:30.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60db7fe640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60e0073200 0x7f60e0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:30.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60db7fe640 1 -- 172.21.15.150:0/1944736683 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5222+0+0 (secure 0 0 0) 0x7f60e4093cd0 con 0x7f60f40a7350 2024-09-17T04:49:30.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.041+0000 7f60f27fc640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60e0073200 0x7f60e0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:30.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.045+0000 7f60f27fc640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60e0073200 0x7f60e0075710 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f60e80058d0 tx=0x7f60e8005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:30.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.049+0000 7f60db7fe640 1 -- 172.21.15.150:0/1944736683 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f60e4020030 con 0x7f60f40a7350 2024-09-17T04:49:30.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.217+0000 7f60f964f640 1 -- 172.21.15.150:0/1944736683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f60f40c61f0 con 0x7f60f40a7350 2024-09-17T04:49:30.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.845+0000 7f60db7fe640 1 -- 172.21.15.150:0/1944736683 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'a_data' created v20) ==== 160+0+0 (secure 0 0 0) 0x7f60e405f020 con 0x7f60f40a7350 2024-09-17T04:49:30.846 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'a_data' created 2024-09-17T04:49:30.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.849+0000 7f60f964f640 1 -- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60e0073200 msgr2=0x7f60e0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:30.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.849+0000 7f60f964f640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60e0073200 0x7f60e0075710 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f60e80058d0 tx=0x7f60e8005860 comp rx=0 tx=0).stop 2024-09-17T04:49:30.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.849+0000 7f60f964f640 1 -- 172.21.15.150:0/1944736683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f40a7350 msgr2=0x7f60f4147a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:30.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.849+0000 7f60f964f640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f40a7350 0x7f60f4147a90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f60e400c460 tx=0x7f60e400c920 comp rx=0 tx=0).stop 2024-09-17T04:49:30.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.849+0000 7f60f964f640 1 -- 172.21.15.150:0/1944736683 shutdown_connections 2024-09-17T04:49:30.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.849+0000 7f60f964f640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60e0073200 0x7f60e0075710 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:30.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.849+0000 7f60f964f640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60f40c42e0 0x7f60f4151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:30.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.849+0000 7f60f964f640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60f41505a0 0x7f60f40aaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:30.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.849+0000 7f60f964f640 1 --2- 172.21.15.150:0/1944736683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60f40a7350 0x7f60f4147a90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:30.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.849+0000 7f60f964f640 1 -- 172.21.15.150:0/1944736683 >> 172.21.15.150:0/1944736683 conn(0x7f60f40c4a00 msgr2=0x7f60f414f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:30.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.849+0000 7f60f964f640 1 -- 172.21.15.150:0/1944736683 shutdown_connections 2024-09-17T04:49:30.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:30.849+0000 7f60f964f640 1 -- 172.21.15.150:0/1944736683 wait complete. 2024-09-17T04:49:30.886 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new a a_metadata a_data 2024-09-17T04:49:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 -- 172.21.15.150:0/3017577408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98f00c42e0 msgr2=0x7f98f00c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 --2- 172.21.15.150:0/3017577408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98f00c42e0 0x7f98f00c6690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009900 tx=0x7f98e001c8e0 comp rx=0 tx=0).stop 2024-09-17T04:49:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 -- 172.21.15.150:0/3017577408 shutdown_connections 2024-09-17T04:49:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 --2- 172.21.15.150:0/3017577408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98f00a7350 0x7f98f0150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:31.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 --2- 172.21.15.150:0/3017577408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98f00c42e0 0x7f98f00c6690 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:31.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 -- 172.21.15.150:0/3017577408 >> 172.21.15.150:0/3017577408 conn(0x7f98f00c4a00 msgr2=0x7f98f00c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:31.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 -- 172.21.15.150:0/3017577408 shutdown_connections 2024-09-17T04:49:31.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 -- 172.21.15.150:0/3017577408 wait complete. 2024-09-17T04:49:31.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 Processor -- start 2024-09-17T04:49:31.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 -- start start 2024-09-17T04:49:31.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98f00a7350 0x7f98f00aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:31.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98f00c42e0 0x7f98f00c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:31.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98f01505a0 0x7f98f0148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:31.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f98f00c6bd0 con 0x7f98f00c42e0 2024-09-17T04:49:31.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f98f0070b40 con 0x7f98f01505a0 2024-09-17T04:49:31.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98f626f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f98f00c6d50 con 0x7f98f00a7350 2024-09-17T04:49:31.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98e77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98f00c42e0 0x7f98f00c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:31.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98e77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98f00c42e0 0x7f98f00c1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42928/0 (socket says 172.21.15.150:42928) 2024-09-17T04:49:31.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98e77fe640 1 -- 172.21.15.150:0/1024552886 learned_addr learned my addr 172.21.15.150:0/1024552886 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:31.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.049+0000 7f98effff640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98f00a7350 0x7f98f00aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:31.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98f480c640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98f01505a0 0x7f98f0148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:31.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98e77fe640 1 -- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98f00a7350 msgr2=0x7f98f00aaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:31.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98e77fe640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98f00a7350 0x7f98f00aaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:31.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98e77fe640 1 -- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98f01505a0 msgr2=0x7f98f0148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:31.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98e77fe640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98f01505a0 0x7f98f0148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:31.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98e77fe640 1 -- 172.21.15.150:0/1024552886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f01f3e60 con 0x7f98f00c42e0 2024-09-17T04:49:31.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98effff640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98f00a7350 0x7f98f00aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:49:31.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98f480c640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98f01505a0 0x7f98f0148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:49:31.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98e77fe640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98f00c42e0 0x7f98f00c1330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f98d800c940 tx=0x7f98d800ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:31.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98ed7fa640 1 -- 172.21.15.150:0/1024552886 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d8004220 con 0x7f98f00c42e0 2024-09-17T04:49:31.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98ed7fa640 1 -- 172.21.15.150:0/1024552886 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98d80079b0 con 0x7f98f00c42e0 2024-09-17T04:49:31.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98ed7fa640 1 -- 172.21.15.150:0/1024552886 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d80044b0 con 0x7f98f00c42e0 2024-09-17T04:49:31.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98f626f640 1 -- 172.21.15.150:0/1024552886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98f01f39c0 con 0x7f98f00c42e0 2024-09-17T04:49:31.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98f626f640 1 -- 172.21.15.150:0/1024552886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98f00ac5f0 con 0x7f98f00c42e0 2024-09-17T04:49:31.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98f626f640 1 -- 172.21.15.150:0/1024552886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98f0150060 con 0x7f98f00c42e0 2024-09-17T04:49:31.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98ed7fa640 1 -- 172.21.15.150:0/1024552886 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f98d8005020 con 0x7f98f00c42e0 2024-09-17T04:49:31.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98ed7fa640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f98d0073230 0x7f98d0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:31.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98ed7fa640 1 -- 172.21.15.150:0/1024552886 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5599+0+0 (secure 0 0 0) 0x7f98d8094580 con 0x7f98f00c42e0 2024-09-17T04:49:31.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.053+0000 7f98effff640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f98d0073230 0x7f98d0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:31.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.057+0000 7f98effff640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f98d0073230 0x7f98d0075740 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f98e00af000 tx=0x7f98e001b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:31.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.057+0000 7f98ed7fa640 1 -- 172.21.15.150:0/1024552886 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f98d8014030 con 0x7f98f00c42e0 2024-09-17T04:49:31.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.249+0000 7f98f626f640 1 -- 172.21.15.150:0/1024552886 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"} v 0) -- 0x7f98f01f3b90 con 0x7f98f00c42e0 2024-09-17T04:49:31.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.881+0000 7f98ed7fa640 1 -- 172.21.15.150:0/1024552886 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 157+0+0 (secure 0 0 0) 0x7f98d805f810 con 0x7f98f00c42e0 2024-09-17T04:49:31.906 INFO:teuthology.orchestra.run.smithi150.stderr:new fs with metadata pool 2 and data pool 3 2024-09-17T04:49:31.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.909+0000 7f98e67fc640 1 -- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f98d0073230 msgr2=0x7f98d0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:31.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.909+0000 7f98e67fc640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f98d0073230 0x7f98d0075740 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f98e00af000 tx=0x7f98e001b120 comp rx=0 tx=0).stop 2024-09-17T04:49:31.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.909+0000 7f98e67fc640 1 -- 172.21.15.150:0/1024552886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98f00c42e0 msgr2=0x7f98f00c1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:31.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.909+0000 7f98e67fc640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98f00c42e0 0x7f98f00c1330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f98d800c940 tx=0x7f98d800ce00 comp rx=0 tx=0).stop 2024-09-17T04:49:31.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.909+0000 7f98e67fc640 1 -- 172.21.15.150:0/1024552886 shutdown_connections 2024-09-17T04:49:31.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.909+0000 7f98e67fc640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f98d0073230 0x7f98d0075740 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:31.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.909+0000 7f98e67fc640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98f01505a0 0x7f98f0148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:31.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.909+0000 7f98e67fc640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98f00c42e0 0x7f98f00c1330 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:31.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.909+0000 7f98e67fc640 1 --2- 172.21.15.150:0/1024552886 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98f00a7350 0x7f98f00aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:31.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.909+0000 7f98e67fc640 1 -- 172.21.15.150:0/1024552886 >> 172.21.15.150:0/1024552886 conn(0x7f98f00c4a00 msgr2=0x7f98f00c3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:31.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.909+0000 7f98e67fc640 1 -- 172.21.15.150:0/1024552886 shutdown_connections 2024-09-17T04:49:31.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:31.909+0000 7f98e67fc640 1 -- 172.21.15.150:0/1024552886 wait complete. 2024-09-17T04:49:31.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:32.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.081+0000 7ff23a15c640 1 -- 172.21.15.150:0/2013508547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff21c004810 msgr2=0x7ff21c004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.081+0000 7ff23a15c640 1 --2- 172.21.15.150:0/2013508547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff21c004810 0x7ff21c004c20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff228009b60 tx=0x7ff22801d270 comp rx=0 tx=0).stop 2024-09-17T04:49:32.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.081+0000 7ff23a15c640 1 -- 172.21.15.150:0/2013508547 shutdown_connections 2024-09-17T04:49:32.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.081+0000 7ff23a15c640 1 --2- 172.21.15.150:0/2013508547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff21c004810 0x7ff21c004c20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.081+0000 7ff23a15c640 1 -- 172.21.15.150:0/2013508547 >> 172.21.15.150:0/2013508547 conn(0x7ff2340c4a00 msgr2=0x7ff2340c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:32.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.081+0000 7ff23a15c640 1 -- 172.21.15.150:0/2013508547 shutdown_connections 2024-09-17T04:49:32.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.081+0000 7ff23a15c640 1 -- 172.21.15.150:0/2013508547 wait complete. 2024-09-17T04:49:32.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff23a15c640 1 Processor -- start 2024-09-17T04:49:32.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff23a15c640 1 -- start start 2024-09-17T04:49:32.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff23a15c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff21c004810 0x7ff2340c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:32.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff23a15c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2341548a0 0x7ff234154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:32.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff23a15c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff234155330 0x7ff23414b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:32.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff23a15c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff234070ac0 con 0x7ff234155330 2024-09-17T04:49:32.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff23a15c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff2341509b0 con 0x7ff21c004810 2024-09-17T04:49:32.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff23a15c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff234070da0 con 0x7ff2341548a0 2024-09-17T04:49:32.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff233fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff234155330 0x7ff23414b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:32.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff233fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff234155330 0x7ff23414b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42942/0 (socket says 172.21.15.150:42942) 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff233fff640 1 -- 172.21.15.150:0/3908234907 learned_addr learned my addr 172.21.15.150:0/3908234907 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff2337fe640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff21c004810 0x7ff2340c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff232ffd640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2341548a0 0x7ff234154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff233fff640 1 -- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2341548a0 msgr2=0x7ff234154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff233fff640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2341548a0 0x7ff234154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff233fff640 1 -- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff21c004810 msgr2=0x7ff2340c1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff233fff640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff21c004810 0x7ff2340c1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff233fff640 1 -- 172.21.15.150:0/3908234907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2340c3690 con 0x7ff234155330 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff2337fe640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff21c004810 0x7ff2340c1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff233fff640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff234155330 0x7ff23414b2d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff2340719d0 tx=0x7ff22c010db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff21bfff640 1 -- 172.21.15.150:0/3908234907 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c00ec40 con 0x7ff234155330 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff21bfff640 1 -- 172.21.15.150:0/3908234907 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff22c013480 con 0x7ff234155330 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff23a15c640 1 -- 172.21.15.150:0/3908234907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff234156f30 con 0x7ff234155330 2024-09-17T04:49:32.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff21bfff640 1 -- 172.21.15.150:0/3908234907 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c0138b0 con 0x7ff234155330 2024-09-17T04:49:32.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff23a15c640 1 -- 172.21.15.150:0/3908234907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff234153750 con 0x7ff234155330 2024-09-17T04:49:32.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.085+0000 7ff23a15c640 1 -- 172.21.15.150:0/3908234907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff234150320 con 0x7ff234155330 2024-09-17T04:49:32.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.089+0000 7ff21bfff640 1 -- 172.21.15.150:0/3908234907 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff22c013a50 con 0x7ff234155330 2024-09-17T04:49:32.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.089+0000 7ff21bfff640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff220073200 0x7ff220075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:32.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.089+0000 7ff21bfff640 1 -- 172.21.15.150:0/3908234907 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7ff22c097fb0 con 0x7ff234155330 2024-09-17T04:49:32.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.089+0000 7ff2337fe640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff220073200 0x7ff220075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:32.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.089+0000 7ff2337fe640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff220073200 0x7ff220075710 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff224006bc0 tx=0x7ff224006b50 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:32.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.093+0000 7ff21bfff640 1 -- 172.21.15.150:0/3908234907 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff2340c3690 con 0x7ff234155330 2024-09-17T04:49:32.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.269+0000 7ff23a15c640 1 -- 172.21.15.150:0/3908234907 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff2341540e0 con 0x7ff234155330 2024-09-17T04:49:32.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.269+0000 7ff21bfff640 1 -- 172.21.15.150:0/3908234907 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16698 (secure 0 0 0) 0x7ff22c062ff0 con 0x7ff234155330 2024-09-17T04:49:32.272 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:32.272 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":21,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:31.253715+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:32.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.273+0000 7ff23a15c640 1 -- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff220073200 msgr2=0x7ff220075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.273+0000 7ff23a15c640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff220073200 0x7ff220075710 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff224006bc0 tx=0x7ff224006b50 comp rx=0 tx=0).stop 2024-09-17T04:49:32.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.273+0000 7ff23a15c640 1 -- 172.21.15.150:0/3908234907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff234155330 msgr2=0x7ff23414b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.273+0000 7ff23a15c640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff234155330 0x7ff23414b2d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff2340719d0 tx=0x7ff22c010db0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.273+0000 7ff23a15c640 1 -- 172.21.15.150:0/3908234907 shutdown_connections 2024-09-17T04:49:32.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.273+0000 7ff23a15c640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff220073200 0x7ff220075710 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.273+0000 7ff23a15c640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff234155330 0x7ff23414b2d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.273+0000 7ff23a15c640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2341548a0 0x7ff234154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.273+0000 7ff23a15c640 1 --2- 172.21.15.150:0/3908234907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff21c004810 0x7ff2340c1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.273+0000 7ff23a15c640 1 -- 172.21.15.150:0/3908234907 >> 172.21.15.150:0/3908234907 conn(0x7ff2340c4a00 msgr2=0x7ff23414d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:32.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.273+0000 7ff23a15c640 1 -- 172.21.15.150:0/3908234907 shutdown_connections 2024-09-17T04:49:32.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.273+0000 7ff23a15c640 1 -- 172.21.15.150:0/3908234907 wait complete. 2024-09-17T04:49:32.294 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:32.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.445+0000 7f418ff01640 1 -- 172.21.15.150:0/358010562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174004810 msgr2=0x7f4174004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.445+0000 7f418ff01640 1 --2- 172.21.15.150:0/358010562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174004810 0x7f4174004c20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f41780016f0 tx=0x7f417801c750 comp rx=0 tx=0).stop 2024-09-17T04:49:32.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.445+0000 7f418ff01640 1 -- 172.21.15.150:0/358010562 shutdown_connections 2024-09-17T04:49:32.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.445+0000 7f418ff01640 1 --2- 172.21.15.150:0/358010562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4174004810 0x7f4174004c20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.445+0000 7f418ff01640 1 -- 172.21.15.150:0/358010562 >> 172.21.15.150:0/358010562 conn(0x7f41880c4a00 msgr2=0x7f41880c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:32.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.445+0000 7f418ff01640 1 -- 172.21.15.150:0/358010562 shutdown_connections 2024-09-17T04:49:32.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.445+0000 7f418ff01640 1 -- 172.21.15.150:0/358010562 wait complete. 2024-09-17T04:49:32.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.445+0000 7f418ff01640 1 Processor -- start 2024-09-17T04:49:32.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.445+0000 7f418ff01640 1 -- start start 2024-09-17T04:49:32.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418ff01640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4174004810 0x7f4188156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:32.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418ff01640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f418814b640 0x7f4188152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:32.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418ff01640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4188157060 0x7f41881f0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:32.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418ff01640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f41880749e0 con 0x7f4188157060 2024-09-17T04:49:32.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418ff01640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4188074860 con 0x7f4174004810 2024-09-17T04:49:32.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418ff01640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4188074b60 con 0x7f418814b640 2024-09-17T04:49:32.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418e49e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4188157060 0x7f41881f0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:32.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418e49e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4188157060 0x7f41881f0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42970/0 (socket says 172.21.15.150:42970) 2024-09-17T04:49:32.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418e49e640 1 -- 172.21.15.150:0/1010950292 learned_addr learned my addr 172.21.15.150:0/1010950292 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:32.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418d49c640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f418814b640 0x7f4188152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:32.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418dc9d640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4174004810 0x7f4188156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:32.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418e49e640 1 -- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f418814b640 msgr2=0x7f4188152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418e49e640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f418814b640 0x7f4188152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418e49e640 1 -- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4174004810 msgr2=0x7f4188156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418e49e640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4174004810 0x7f4188156b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418e49e640 1 -- 172.21.15.150:0/1010950292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41881ee240 con 0x7f4188157060 2024-09-17T04:49:32.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418d49c640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f418814b640 0x7f4188152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:32.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418dc9d640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4174004810 0x7f4188156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:32.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418e49e640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4188157060 0x7f41881f0e90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4188151170 tx=0x7f4180009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:32.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f417e7fc640 1 -- 172.21.15.150:0/1010950292 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418000e900 con 0x7f4188157060 2024-09-17T04:49:32.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f417e7fc640 1 -- 172.21.15.150:0/1010950292 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f418000ebb0 con 0x7f4188157060 2024-09-17T04:49:32.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418ff01640 1 -- 172.21.15.150:0/1010950292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4188157b30 con 0x7f4188157060 2024-09-17T04:49:32.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f417e7fc640 1 -- 172.21.15.150:0/1010950292 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180011470 con 0x7f4188157060 2024-09-17T04:49:32.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418ff01640 1 -- 172.21.15.150:0/1010950292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41881f18f0 con 0x7f4188157060 2024-09-17T04:49:32.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f418ff01640 1 -- 172.21.15.150:0/1010950292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f418814b0f0 con 0x7f4188157060 2024-09-17T04:49:32.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f417e7fc640 1 -- 172.21.15.150:0/1010950292 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4180011610 con 0x7f4188157060 2024-09-17T04:49:32.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.449+0000 7f417e7fc640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f416c073230 0x7f416c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:32.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.453+0000 7f417e7fc640 1 -- 172.21.15.150:0/1010950292 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7f4180094b30 con 0x7f4188157060 2024-09-17T04:49:32.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.453+0000 7f418dc9d640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f416c073230 0x7f416c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.453+0000 7f418dc9d640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f416c073230 0x7f416c075740 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4184001870 tx=0x7f4184001e80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:32.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.457+0000 7f417e7fc640 1 -- 172.21.15.150:0/1010950292 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f418005fc80 con 0x7f4188157060 2024-09-17T04:49:32.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.613+0000 7f418ff01640 1 -- 172.21.15.150:0/1010950292 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4188075070 con 0x7f4188157060 2024-09-17T04:49:32.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f417e7fc640 1 -- 172.21.15.150:0/1010950292 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16698 (secure 0 0 0) 0x7f418005f580 con 0x7f4188157060 2024-09-17T04:49:32.617 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:32.617 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":21,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:31.253715+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:32.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f418ff01640 1 -- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f416c073230 msgr2=0x7f416c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f418ff01640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f416c073230 0x7f416c075740 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4184001870 tx=0x7f4184001e80 comp rx=0 tx=0).stop 2024-09-17T04:49:32.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f418ff01640 1 -- 172.21.15.150:0/1010950292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4188157060 msgr2=0x7f41881f0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f418ff01640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4188157060 0x7f41881f0e90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4188151170 tx=0x7f4180009b00 comp rx=0 tx=0).stop 2024-09-17T04:49:32.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f418ff01640 1 -- 172.21.15.150:0/1010950292 shutdown_connections 2024-09-17T04:49:32.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f418ff01640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f416c073230 0x7f416c075740 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f418ff01640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4188157060 0x7f41881f0e90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f418ff01640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f418814b640 0x7f4188152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f418ff01640 1 --2- 172.21.15.150:0/1010950292 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4174004810 0x7f4188156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f418ff01640 1 -- 172.21.15.150:0/1010950292 >> 172.21.15.150:0/1010950292 conn(0x7f41880c4a00 msgr2=0x7f41880c2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:32.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f418ff01640 1 -- 172.21.15.150:0/1010950292 shutdown_connections 2024-09-17T04:49:32.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.617+0000 7f418ff01640 1 -- 172.21.15.150:0/1010950292 wait complete. 2024-09-17T04:49:32.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a standby_count_wanted 0 2024-09-17T04:49:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.797+0000 7efee0344640 1 -- 172.21.15.150:0/2815588835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efed8072ea0 msgr2=0x7efed80732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.797+0000 7efee0344640 1 --2- 172.21.15.150:0/2815588835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efed8072ea0 0x7efed80732b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efed8071a80 tx=0x7efec801c860 comp rx=0 tx=0).stop 2024-09-17T04:49:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.797+0000 7efee0344640 1 -- 172.21.15.150:0/2815588835 shutdown_connections 2024-09-17T04:49:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.797+0000 7efee0344640 1 --2- 172.21.15.150:0/2815588835 >> v2:172.21.15.176:3300/0 conn(0x7efed8157be0 0x7efed8157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.797+0000 7efee0344640 1 --2- 172.21.15.150:0/2815588835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efed8072ea0 0x7efed80732b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.797+0000 7efee0344640 1 --2- 172.21.15.150:0/2815588835 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efed80c97a0 0x7efed8151c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.797+0000 7efee0344640 1 -- 172.21.15.150:0/2815588835 >> 172.21.15.150:0/2815588835 conn(0x7efed80abb00 msgr2=0x7efed80abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.797+0000 7efee0344640 1 -- 172.21.15.150:0/2815588835 shutdown_connections 2024-09-17T04:49:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.797+0000 7efee0344640 1 -- 172.21.15.150:0/2815588835 wait complete. 2024-09-17T04:49:32.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.797+0000 7efee0344640 1 Processor -- start 2024-09-17T04:49:32.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.797+0000 7efee0344640 1 -- start start 2024-09-17T04:49:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efee0344640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efed8072ea0 0x7efed814bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efee0344640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efed80c97a0 0x7efed8147400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efee0344640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efed8157be0 0x7efed8157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efee0344640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efed80c4d60 con 0x7efed80c97a0 2024-09-17T04:49:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efee0344640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efed80c4be0 con 0x7efed8072ea0 2024-09-17T04:49:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efee0344640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efed80c84e0 con 0x7efed8157be0 2024-09-17T04:49:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efedd8df640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efed80c97a0 0x7efed8147400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efedd8df640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efed80c97a0 0x7efed8147400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42994/0 (socket says 172.21.15.150:42994) 2024-09-17T04:49:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efedd8df640 1 -- 172.21.15.150:0/607437288 learned_addr learned my addr 172.21.15.150:0/607437288 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efede0e0640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efed8072ea0 0x7efed814bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efede8e1640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efed8157be0 0x7efed8157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efedd8df640 1 -- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efed8157be0 msgr2=0x7efed8157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efedd8df640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efed8157be0 0x7efed8157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efedd8df640 1 -- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efed8072ea0 msgr2=0x7efed814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efedd8df640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efed8072ea0 0x7efed814bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efedd8df640 1 -- 172.21.15.150:0/607437288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed8150010 con 0x7efed80c97a0 2024-09-17T04:49:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efedd8df640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efed80c97a0 0x7efed8147400 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efec80b0040 tx=0x7efec80a5720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efeceffd640 1 -- 172.21.15.150:0/607437288 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec80b2040 con 0x7efed80c97a0 2024-09-17T04:49:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efeceffd640 1 -- 172.21.15.150:0/607437288 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efec8005030 con 0x7efed80c97a0 2024-09-17T04:49:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efeceffd640 1 -- 172.21.15.150:0/607437288 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec80044f0 con 0x7efed80c97a0 2024-09-17T04:49:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efee0344640 1 -- 172.21.15.150:0/607437288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed8155a80 con 0x7efed80c97a0 2024-09-17T04:49:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efee0344640 1 -- 172.21.15.150:0/607437288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed814c930 con 0x7efed80c97a0 2024-09-17T04:49:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efee0344640 1 -- 172.21.15.150:0/607437288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efed80c89b0 con 0x7efed80c97a0 2024-09-17T04:49:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efeceffd640 1 -- 172.21.15.150:0/607437288 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7efec8005460 con 0x7efed80c97a0 2024-09-17T04:49:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efeceffd640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efebc073230 0x7efebc075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.801+0000 7efeceffd640 1 -- 172.21.15.150:0/607437288 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5689+0+0 (secure 0 0 0) 0x7efec812e490 con 0x7efed80c97a0 2024-09-17T04:49:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.805+0000 7efede0e0640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efebc073230 0x7efebc075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:32.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.805+0000 7efede0e0640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efebc073230 0x7efebc075740 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efed40058d0 tx=0x7efed4005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:32.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:32.809+0000 7efeceffd640 1 -- 172.21.15.150:0/607437288 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7efec8016450 con 0x7efed80c97a0 2024-09-17T04:49:33.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.025+0000 7efee0344640 1 -- 172.21.15.150:0/607437288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7efed814b810 con 0x7efed80c97a0 2024-09-17T04:49:33.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.913+0000 7efeceffd640 1 -- 172.21.15.150:0/607437288 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 113+0+0 (secure 0 0 0) 0x7efec80f94c0 con 0x7efed80c97a0 2024-09-17T04:49:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.917+0000 7efee0344640 1 -- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efebc073230 msgr2=0x7efebc075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.917+0000 7efee0344640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efebc073230 0x7efebc075740 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efed40058d0 tx=0x7efed4005860 comp rx=0 tx=0).stop 2024-09-17T04:49:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.917+0000 7efee0344640 1 -- 172.21.15.150:0/607437288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efed80c97a0 msgr2=0x7efed8147400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.917+0000 7efee0344640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efed80c97a0 0x7efed8147400 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efec80b0040 tx=0x7efec80a5720 comp rx=0 tx=0).stop 2024-09-17T04:49:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.917+0000 7efee0344640 1 -- 172.21.15.150:0/607437288 shutdown_connections 2024-09-17T04:49:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.917+0000 7efee0344640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efebc073230 0x7efebc075740 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.917+0000 7efee0344640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efed8157be0 0x7efed8157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.917+0000 7efee0344640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efed80c97a0 0x7efed8147400 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.917+0000 7efee0344640 1 --2- 172.21.15.150:0/607437288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efed8072ea0 0x7efed814bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.917+0000 7efee0344640 1 -- 172.21.15.150:0/607437288 >> 172.21.15.150:0/607437288 conn(0x7efed80abb00 msgr2=0x7efed8152e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.917+0000 7efee0344640 1 -- 172.21.15.150:0/607437288 shutdown_connections 2024-09-17T04:49:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:33.917+0000 7efee0344640 1 -- 172.21.15.150:0/607437288 wait complete. 2024-09-17T04:49:33.941 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-09-17T04:49:33.941 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false 2024-09-17T04:49:34.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.101+0000 7f95f1d41640 1 -- 172.21.15.150:0/2268231206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95ec070e50 msgr2=0x7f95ec071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:34.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.101+0000 7f95f1d41640 1 --2- 172.21.15.150:0/2268231206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95ec070e50 0x7f95ec071260 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f95ec072370 tx=0x7f95e001c830 comp rx=0 tx=0).stop 2024-09-17T04:49:34.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.101+0000 7f95f1d41640 1 -- 172.21.15.150:0/2268231206 shutdown_connections 2024-09-17T04:49:34.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.101+0000 7f95f1d41640 1 --2- 172.21.15.150:0/2268231206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95ec070e50 0x7f95ec071260 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:34.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.101+0000 7f95f1d41640 1 --2- 172.21.15.150:0/2268231206 >> v2:172.21.15.176:3300/0 conn(0x7f95ec073440 0x7f95ec0a7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:34.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.101+0000 7f95f1d41640 1 -- 172.21.15.150:0/2268231206 >> 172.21.15.150:0/2268231206 conn(0x7f95ec0c4a00 msgr2=0x7f95ec0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:34.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.101+0000 7f95f1d41640 1 -- 172.21.15.150:0/2268231206 shutdown_connections 2024-09-17T04:49:34.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.101+0000 7f95f1d41640 1 -- 172.21.15.150:0/2268231206 wait complete. 2024-09-17T04:49:34.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95f1d41640 1 Processor -- start 2024-09-17T04:49:34.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95f1d41640 1 -- start start 2024-09-17T04:49:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95f1d41640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95ec070e50 0x7f95ec14c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95f1d41640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95ec0717a0 0x7f95ec1f3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95f1d41640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95ec073440 0x7f95ec1f0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95f1d41640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec074f70 con 0x7f95ec070e50 2024-09-17T04:49:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95f1d41640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec0adef0 con 0x7f95ec073440 2024-09-17T04:49:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95f1d41640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f95ec0750f0 con 0x7f95ec0717a0 2024-09-17T04:49:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95eb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95ec070e50 0x7f95ec14c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95eb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95ec070e50 0x7f95ec14c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43014/0 (socket says 172.21.15.150:43014) 2024-09-17T04:49:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95eb7fe640 1 -- 172.21.15.150:0/4151561630 learned_addr learned my addr 172.21.15.150:0/4151561630 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95ebfff640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95ec073440 0x7f95ec1f0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95eaffd640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95ec0717a0 0x7f95ec1f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95eb7fe640 1 -- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95ec0717a0 msgr2=0x7f95ec1f3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95eb7fe640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95ec0717a0 0x7f95ec1f3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95eb7fe640 1 -- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95ec073440 msgr2=0x7f95ec1f0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95eb7fe640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95ec073440 0x7f95ec1f0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95eb7fe640 1 -- 172.21.15.150:0/4151561630 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ec1f3f70 con 0x7f95ec070e50 2024-09-17T04:49:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95ebfff640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95ec073440 0x7f95ec1f0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:49:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95eb7fe640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95ec070e50 0x7f95ec14c800 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f95e400e9f0 tx=0x7f95e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95d3fff640 1 -- 172.21.15.150:0/4151561630 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e400ccc0 con 0x7f95ec070e50 2024-09-17T04:49:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95d3fff640 1 -- 172.21.15.150:0/4151561630 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f95e4003980 con 0x7f95ec070e50 2024-09-17T04:49:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95f1d41640 1 -- 172.21.15.150:0/4151561630 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95ec155440 con 0x7f95ec070e50 2024-09-17T04:49:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95d3fff640 1 -- 172.21.15.150:0/4151561630 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4003d90 con 0x7f95ec070e50 2024-09-17T04:49:34.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.105+0000 7f95f1d41640 1 -- 172.21.15.150:0/4151561630 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95ec154f90 con 0x7f95ec070e50 2024-09-17T04:49:34.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.109+0000 7f95f1d41640 1 -- 172.21.15.150:0/4151561630 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95ec0ad8c0 con 0x7f95ec070e50 2024-09-17T04:49:34.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.109+0000 7f95d3fff640 1 -- 172.21.15.150:0/4151561630 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f95e4004020 con 0x7f95ec070e50 2024-09-17T04:49:34.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.109+0000 7f95d3fff640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f95d8073200 0x7f95d8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:34.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.109+0000 7f95d3fff640 1 -- 172.21.15.150:0/4151561630 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f95e4094aa0 con 0x7f95ec070e50 2024-09-17T04:49:34.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.109+0000 7f95eaffd640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f95d8073200 0x7f95d8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:34.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.109+0000 7f95eaffd640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f95d8073200 0x7f95d8075710 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95e00af000 tx=0x7f95e001b070 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:34.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.113+0000 7f95d3fff640 1 -- 172.21.15.150:0/4151561630 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f95e4016030 con 0x7f95ec070e50 2024-09-17T04:49:34.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.325+0000 7f95f1d41640 1 -- 172.21.15.150:0/4151561630 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f95ec1f3130 con 0x7f95ec070e50 2024-09-17T04:49:34.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.925+0000 7f95d3fff640 1 -- 172.21.15.150:0/4151561630 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 117+0+0 (secure 0 0 0) 0x7f95e405fb50 con 0x7f95ec070e50 2024-09-17T04:49:34.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.929+0000 7f95f1d41640 1 -- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f95d8073200 msgr2=0x7f95d8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:34.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.929+0000 7f95f1d41640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f95d8073200 0x7f95d8075710 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95e00af000 tx=0x7f95e001b070 comp rx=0 tx=0).stop 2024-09-17T04:49:34.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.929+0000 7f95f1d41640 1 -- 172.21.15.150:0/4151561630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95ec070e50 msgr2=0x7f95ec14c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:34.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.929+0000 7f95f1d41640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95ec070e50 0x7f95ec14c800 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f95e400e9f0 tx=0x7f95e400eeb0 comp rx=0 tx=0).stop 2024-09-17T04:49:34.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.929+0000 7f95f1d41640 1 -- 172.21.15.150:0/4151561630 shutdown_connections 2024-09-17T04:49:34.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.929+0000 7f95f1d41640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f95d8073200 0x7f95d8075710 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:34.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.929+0000 7f95f1d41640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95ec073440 0x7f95ec1f0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:34.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.929+0000 7f95f1d41640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f95ec0717a0 0x7f95ec1f3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:34.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.929+0000 7f95f1d41640 1 --2- 172.21.15.150:0/4151561630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95ec070e50 0x7f95ec14c800 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:34.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.929+0000 7f95f1d41640 1 -- 172.21.15.150:0/4151561630 >> 172.21.15.150:0/4151561630 conn(0x7f95ec0c4a00 msgr2=0x7f95ec0c2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:34.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.929+0000 7f95f1d41640 1 -- 172.21.15.150:0/4151561630 shutdown_connections 2024-09-17T04:49:34.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:34.929+0000 7f95f1d41640 1 -- 172.21.15.150:0/4151561630 wait complete. 2024-09-17T04:49:34.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a session_timeout 300 2024-09-17T04:49:35.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.125+0000 7fa2ba291640 1 -- 172.21.15.150:0/4043195065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b40c7eb0 msgr2=0x7fa2b41540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:35.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.125+0000 7fa2ba291640 1 --2- 172.21.15.150:0/4043195065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b40c7eb0 0x7fa2b41540b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40755f0 tx=0x7fa2a401c8e0 comp rx=0 tx=0).stop 2024-09-17T04:49:35.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.125+0000 7fa2ba291640 1 -- 172.21.15.150:0/4043195065 shutdown_connections 2024-09-17T04:49:35.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.125+0000 7fa2ba291640 1 --2- 172.21.15.150:0/4043195065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2b4149d70 0x7fa2b414a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:35.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.125+0000 7fa2ba291640 1 --2- 172.21.15.150:0/4043195065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b40c7eb0 0x7fa2b41540b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:35.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.125+0000 7fa2ba291640 1 -- 172.21.15.150:0/4043195065 >> 172.21.15.150:0/4043195065 conn(0x7fa2b40aab60 msgr2=0x7fa2b40aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:35.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.125+0000 7fa2ba291640 1 -- 172.21.15.150:0/4043195065 shutdown_connections 2024-09-17T04:49:35.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.125+0000 7fa2ba291640 1 -- 172.21.15.150:0/4043195065 wait complete. 2024-09-17T04:49:35.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.125+0000 7fa2ba291640 1 Processor -- start 2024-09-17T04:49:35.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.125+0000 7fa2ba291640 1 -- start start 2024-09-17T04:49:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2ba291640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b40c7eb0 0x7fa2b40acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2ba291640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2b40c8330 0x7fa2b41f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2ba291640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2b4149d70 0x7fa2b41f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2ba291640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b41568d0 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2ba291640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b4156750 con 0x7fa2b4149d70 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2ba291640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b40c8b50 con 0x7fa2b40c8330 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b40c7eb0 0x7fa2b40acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b40c7eb0 0x7fa2b40acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43034/0 (socket says 172.21.15.150:43034) 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b3fff640 1 -- 172.21.15.150:0/3325794813 learned_addr learned my addr 172.21.15.150:0/3325794813 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2ab7fe640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2b40c8330 0x7fa2b41f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b3fff640 1 -- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2b40c8330 msgr2=0x7fa2b41f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b3fff640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2b40c8330 0x7fa2b41f37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b3fff640 1 -- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2b4149d70 msgr2=0x7fa2b41f3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b882e640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2b4149d70 0x7fa2b41f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b3fff640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2b4149d70 0x7fa2b41f3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b3fff640 1 -- 172.21.15.150:0/3325794813 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b40ac140 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b3fff640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b40c7eb0 0x7fa2b40acdd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa2a40b0040 tx=0x7fa2a40a55a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:35.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b17fa640 1 -- 172.21.15.150:0/3325794813 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a40b2040 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b17fa640 1 -- 172.21.15.150:0/3325794813 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2a40043b0 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2ba291640 1 -- 172.21.15.150:0/3325794813 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b4150280 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b17fa640 1 -- 172.21.15.150:0/3325794813 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4003860 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2ba291640 1 -- 172.21.15.150:0/3325794813 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b40c5cc0 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2ba291640 1 -- 172.21.15.150:0/3325794813 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2b41540b0 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b17fa640 1 -- 172.21.15.150:0/3325794813 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa2a4003a00 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b17fa640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa294073230 0x7fa294075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2b17fa640 1 -- 172.21.15.150:0/3325794813 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7fa2a412dfa0 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.129+0000 7fa2ab7fe640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa294073230 0x7fa294075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.133+0000 7fa2ab7fe640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa294073230 0x7fa294075740 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0058d0 tx=0x7fa29c005bd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:35.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.133+0000 7fa2b17fa640 1 -- 172.21.15.150:0/3325794813 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa2a40bd060 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.345+0000 7fa2ba291640 1 -- 172.21.15.150:0/3325794813 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"} v 0) -- 0x7fa2b415e7e0 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.937+0000 7fa2b17fa640 1 -- 172.21.15.150:0/3325794813 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"}]=0 v9) ==== 110+0+0 (secure 0 0 0) 0x7fa2a40164d0 con 0x7fa2b40c7eb0 2024-09-17T04:49:35.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.941+0000 7fa2ba291640 1 -- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa294073230 msgr2=0x7fa294075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:35.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.941+0000 7fa2ba291640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa294073230 0x7fa294075740 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa29c0058d0 tx=0x7fa29c005bd0 comp rx=0 tx=0).stop 2024-09-17T04:49:35.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.941+0000 7fa2ba291640 1 -- 172.21.15.150:0/3325794813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b40c7eb0 msgr2=0x7fa2b40acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:35.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.941+0000 7fa2ba291640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b40c7eb0 0x7fa2b40acdd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa2a40b0040 tx=0x7fa2a40a55a0 comp rx=0 tx=0).stop 2024-09-17T04:49:35.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.941+0000 7fa2ba291640 1 -- 172.21.15.150:0/3325794813 shutdown_connections 2024-09-17T04:49:35.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.941+0000 7fa2ba291640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa294073230 0x7fa294075740 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:35.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.941+0000 7fa2ba291640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2b4149d70 0x7fa2b41f3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:35.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.941+0000 7fa2ba291640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2b40c8330 0x7fa2b41f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:35.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.941+0000 7fa2ba291640 1 --2- 172.21.15.150:0/3325794813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b40c7eb0 0x7fa2b40acdd0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:35.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.941+0000 7fa2ba291640 1 -- 172.21.15.150:0/3325794813 >> 172.21.15.150:0/3325794813 conn(0x7fa2b40aab60 msgr2=0x7fa2b4068e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:35.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.941+0000 7fa2ba291640 1 -- 172.21.15.150:0/3325794813 shutdown_connections 2024-09-17T04:49:35.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:35.941+0000 7fa2ba291640 1 -- 172.21.15.150:0/3325794813 wait complete. 2024-09-17T04:49:35.968 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'a' 2024-09-17T04:49:35.969 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:49:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.129+0000 7f020e30e640 1 -- 172.21.15.150:0/2277711885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0208156850 msgr2=0x7f02080c3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.129+0000 7f020e30e640 1 --2- 172.21.15.150:0/2277711885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0208156850 0x7f02080c3110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009840 tx=0x7f01fc01c800 comp rx=0 tx=0).stop 2024-09-17T04:49:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.129+0000 7f020e30e640 1 -- 172.21.15.150:0/2277711885 shutdown_connections 2024-09-17T04:49:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.129+0000 7f020e30e640 1 --2- 172.21.15.150:0/2277711885 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02080c3650 0x7f020806ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.129+0000 7f020e30e640 1 --2- 172.21.15.150:0/2277711885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0208156850 0x7f02080c3110 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.129+0000 7f020e30e640 1 -- 172.21.15.150:0/2277711885 >> 172.21.15.150:0/2277711885 conn(0x7f02080c9de0 msgr2=0x7f02080ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.129+0000 7f020e30e640 1 -- 172.21.15.150:0/2277711885 shutdown_connections 2024-09-17T04:49:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.129+0000 7f020e30e640 1 -- 172.21.15.150:0/2277711885 wait complete. 2024-09-17T04:49:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020e30e640 1 Processor -- start 2024-09-17T04:49:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020e30e640 1 -- start start 2024-09-17T04:49:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020e30e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0208073b40 0x7f0208152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:36.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020e30e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02080c3650 0x7f02081534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:36.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020e30e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0208156850 0x7f02081f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:36.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020e30e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f02080c9c40 con 0x7f0208156850 2024-09-17T04:49:36.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020e30e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f02080c9ac0 con 0x7f0208073b40 2024-09-17T04:49:36.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020e30e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f02081639f0 con 0x7f02080c3650 2024-09-17T04:49:36.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020c8ab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0208156850 0x7f02081f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:36.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020c8ab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0208156850 0x7f02081f34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43046/0 (socket says 172.21.15.150:43046) 2024-09-17T04:49:36.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020c8ab640 1 -- 172.21.15.150:0/2623518131 learned_addr learned my addr 172.21.15.150:0/2623518131 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:36.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f0207fff640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0208073b40 0x7f0208152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:36.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f02077fe640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02080c3650 0x7f02081534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:36.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020c8ab640 1 -- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02080c3650 msgr2=0x7f02081534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020c8ab640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02080c3650 0x7f02081534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020c8ab640 1 -- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0208073b40 msgr2=0x7f0208152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020c8ab640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0208073b40 0x7f0208152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020c8ab640 1 -- 172.21.15.150:0/2623518131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020806a920 con 0x7f0208156850 2024-09-17T04:49:36.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f0207fff640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0208073b40 0x7f0208152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:49:36.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020c8ab640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0208156850 0x7f02081f34a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f020000caa0 tx=0x7f020000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:36.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f0204ff9640 1 -- 172.21.15.150:0/2623518131 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200016020 con 0x7f0208156850 2024-09-17T04:49:36.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f0204ff9640 1 -- 172.21.15.150:0/2623518131 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f020000a930 con 0x7f0208156850 2024-09-17T04:49:36.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020e30e640 1 -- 172.21.15.150:0/2623518131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0208159bb0 con 0x7f0208156850 2024-09-17T04:49:36.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f0204ff9640 1 -- 172.21.15.150:0/2623518131 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020000ad80 con 0x7f0208156850 2024-09-17T04:49:36.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020e30e640 1 -- 172.21.15.150:0/2623518131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f020815aaf0 con 0x7f0208156850 2024-09-17T04:49:36.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.133+0000 7f020e30e640 1 -- 172.21.15.150:0/2623518131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02080c3110 con 0x7f0208156850 2024-09-17T04:49:36.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.137+0000 7f0204ff9640 1 -- 172.21.15.150:0/2623518131 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0200019020 con 0x7f0208156850 2024-09-17T04:49:36.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.137+0000 7f0204ff9640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01f4073200 0x7f01f4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:36.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.137+0000 7f0204ff9640 1 -- 172.21.15.150:0/2623518131 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f02000949c0 con 0x7f0208156850 2024-09-17T04:49:36.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.137+0000 7f0207fff640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01f4073200 0x7f01f4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:36.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.137+0000 7f0207fff640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01f4073200 0x7f01f4075710 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f01fc0ae000 tx=0x7f01fc01b040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:36.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.141+0000 7f0204ff9640 1 -- 172.21.15.150:0/2623518131 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0200020020 con 0x7f0208156850 2024-09-17T04:49:36.162 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T04:49:36.161+0000 7f7345a67640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:49:36.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.345+0000 7f020e30e640 1 -- 172.21.15.150:0/2623518131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02081f3200 con 0x7f0208156850 2024-09-17T04:49:36.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.345+0000 7f0204ff9640 1 -- 172.21.15.150:0/2623518131 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5219 (secure 0 0 0) 0x7f020005fb10 con 0x7f0208156850 2024-09-17T04:49:36.348 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:36.348 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":9,"btime":"2024-09-17T04:49:35:930403+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1}]} 2024-09-17T04:49:36.348 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 9 2024-09-17T04:49:36.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.349+0000 7f020e30e640 1 -- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01f4073200 msgr2=0x7f01f4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.349+0000 7f020e30e640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01f4073200 0x7f01f4075710 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f01fc0ae000 tx=0x7f01fc01b040 comp rx=0 tx=0).stop 2024-09-17T04:49:36.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.349+0000 7f020e30e640 1 -- 172.21.15.150:0/2623518131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0208156850 msgr2=0x7f02081f34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.349+0000 7f020e30e640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0208156850 0x7f02081f34a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f020000caa0 tx=0x7f020000cb90 comp rx=0 tx=0).stop 2024-09-17T04:49:36.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.349+0000 7f020e30e640 1 -- 172.21.15.150:0/2623518131 shutdown_connections 2024-09-17T04:49:36.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.349+0000 7f020e30e640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01f4073200 0x7f01f4075710 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.349+0000 7f020e30e640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0208156850 0x7f02081f34a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.349+0000 7f020e30e640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02080c3650 0x7f02081534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.349+0000 7f020e30e640 1 --2- 172.21.15.150:0/2623518131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0208073b40 0x7f0208152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.349+0000 7f020e30e640 1 -- 172.21.15.150:0/2623518131 >> 172.21.15.150:0/2623518131 conn(0x7f02080c9de0 msgr2=0x7f02080aeac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:36.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.349+0000 7f020e30e640 1 -- 172.21.15.150:0/2623518131 shutdown_connections 2024-09-17T04:49:36.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.349+0000 7f020e30e640 1 -- 172.21.15.150:0/2623518131 wait complete. 2024-09-17T04:49:36.370 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:36.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.525+0000 7feff0307640 1 -- 172.21.15.150:0/2836303038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe8152cf0 msgr2=0x7fefe8153100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.525+0000 7feff0307640 1 --2- 172.21.15.150:0/2836303038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe8152cf0 0x7fefe8153100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fefe80c7010 tx=0x7fefe001cb50 comp rx=0 tx=0).stop 2024-09-17T04:49:36.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.525+0000 7feff0307640 1 -- 172.21.15.150:0/2836303038 shutdown_connections 2024-09-17T04:49:36.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.525+0000 7feff0307640 1 --2- 172.21.15.150:0/2836303038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe8152cf0 0x7fefe8153100 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.525+0000 7feff0307640 1 --2- 172.21.15.150:0/2836303038 >> v2:172.21.15.176:3300/0 conn(0x7fefe80c42e0 0x7fefe80c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.525+0000 7feff0307640 1 -- 172.21.15.150:0/2836303038 >> 172.21.15.150:0/2836303038 conn(0x7fefe80c4a00 msgr2=0x7fefe80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:36.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.525+0000 7feff0307640 1 -- 172.21.15.150:0/2836303038 shutdown_connections 2024-09-17T04:49:36.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.525+0000 7feff0307640 1 -- 172.21.15.150:0/2836303038 wait complete. 2024-09-17T04:49:36.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.525+0000 7feff0307640 1 Processor -- start 2024-09-17T04:49:36.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.525+0000 7feff0307640 1 -- start start 2024-09-17T04:49:36.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7feff0307640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe80a7350 0x7fefe80ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:36.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7feff0307640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe80c42e0 0x7fefe80ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:36.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7feff0307640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefe8152cf0 0x7fefe814a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:36.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7feff0307640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fefe80c6d50 con 0x7fefe80a7350 2024-09-17T04:49:36.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7feff0307640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fefe80c6bd0 con 0x7fefe80c42e0 2024-09-17T04:49:36.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7feff0307640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fefe8153640 con 0x7fefe8152cf0 2024-09-17T04:49:36.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefee0a3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe80a7350 0x7fefe80ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:36.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefee0a3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe80a7350 0x7fefe80ad7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43064/0 (socket says 172.21.15.150:43064) 2024-09-17T04:49:36.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefee0a3640 1 -- 172.21.15.150:0/3299955956 learned_addr learned my addr 172.21.15.150:0/3299955956 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:36.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefee0a3640 1 -- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefe8152cf0 msgr2=0x7fefe814a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefee0a3640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefe8152cf0 0x7fefe814a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefee0a3640 1 -- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe80c42e0 msgr2=0x7fefe80ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefed8a2640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe80c42e0 0x7fefe80ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:36.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefee0a3640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe80c42e0 0x7fefe80ac2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefee0a3640 1 -- 172.21.15.150:0/3299955956 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe8147cc0 con 0x7fefe80a7350 2024-09-17T04:49:36.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefee0a3640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe80a7350 0x7fefe80ad7e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fefd800e7f0 tx=0x7fefd800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:36.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefdeffd640 1 -- 172.21.15.150:0/3299955956 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd800ccc0 con 0x7fefe80a7350 2024-09-17T04:49:36.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefdeffd640 1 -- 172.21.15.150:0/3299955956 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fefd800ce80 con 0x7fefe80a7350 2024-09-17T04:49:36.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7feff0307640 1 -- 172.21.15.150:0/3299955956 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe80acb60 con 0x7fefe80a7350 2024-09-17T04:49:36.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefdeffd640 1 -- 172.21.15.150:0/3299955956 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8005280 con 0x7fefe80a7350 2024-09-17T04:49:36.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7feff0307640 1 -- 172.21.15.150:0/3299955956 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe814a630 con 0x7fefe80a7350 2024-09-17T04:49:36.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7feff0307640 1 -- 172.21.15.150:0/3299955956 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe81508b0 con 0x7fefe80a7350 2024-09-17T04:49:36.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefdeffd640 1 -- 172.21.15.150:0/3299955956 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fefd8005420 con 0x7fefe80a7350 2024-09-17T04:49:36.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.529+0000 7fefdeffd640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fefcc073200 0x7fefcc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:36.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.533+0000 7fefdeffd640 1 -- 172.21.15.150:0/3299955956 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7fefd8094bb0 con 0x7fefe80a7350 2024-09-17T04:49:36.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.533+0000 7fefed8a2640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fefcc073200 0x7fefcc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:36.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.533+0000 7fefed8a2640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fefcc073200 0x7fefcc075710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fefe4001230 tx=0x7fefe4009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:36.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.537+0000 7fefdeffd640 1 -- 172.21.15.150:0/3299955956 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fefd805fc70 con 0x7fefe80a7350 2024-09-17T04:49:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.705+0000 7feff0307640 1 -- 172.21.15.150:0/3299955956 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fefe8153230 con 0x7fefe80a7350 2024-09-17T04:49:36.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.705+0000 7fefdeffd640 1 -- 172.21.15.150:0/3299955956 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16685 (secure 0 0 0) 0x7fefd805f570 con 0x7fefe80a7350 2024-09-17T04:49:36.706 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:36.706 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":22,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:32.881578+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.705+0000 7feff0307640 1 -- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fefcc073200 msgr2=0x7fefcc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.705+0000 7feff0307640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fefcc073200 0x7fefcc075710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fefe4001230 tx=0x7fefe4009040 comp rx=0 tx=0).stop 2024-09-17T04:49:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.705+0000 7feff0307640 1 -- 172.21.15.150:0/3299955956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe80a7350 msgr2=0x7fefe80ad7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.705+0000 7feff0307640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe80a7350 0x7fefe80ad7e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fefd800e7f0 tx=0x7fefd800ecb0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.709+0000 7feff0307640 1 -- 172.21.15.150:0/3299955956 shutdown_connections 2024-09-17T04:49:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.709+0000 7feff0307640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fefcc073200 0x7fefcc075710 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.709+0000 7feff0307640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefe80c42e0 0x7fefe80ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.709+0000 7feff0307640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fefe8152cf0 0x7fefe814a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.709+0000 7feff0307640 1 --2- 172.21.15.150:0/3299955956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fefe80a7350 0x7fefe80ad7e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.709+0000 7feff0307640 1 -- 172.21.15.150:0/3299955956 >> 172.21.15.150:0/3299955956 conn(0x7fefe80c4a00 msgr2=0x7fefe814d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:36.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.709+0000 7feff0307640 1 -- 172.21.15.150:0/3299955956 shutdown_connections 2024-09-17T04:49:36.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.709+0000 7feff0307640 1 -- 172.21.15.150:0/3299955956 wait complete. 2024-09-17T04:49:36.726 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:49:36.726 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:49:36.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.877+0000 7f8a22c67640 1 -- 172.21.15.150:0/4159711440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a1c074450 msgr2=0x7f8a1c074860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.877+0000 7f8a22c67640 1 --2- 172.21.15.150:0/4159711440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a1c074450 0x7f8a1c074860 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c072370 tx=0x7f8a0c01c830 comp rx=0 tx=0).stop 2024-09-17T04:49:36.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.877+0000 7f8a22c67640 1 -- 172.21.15.150:0/4159711440 shutdown_connections 2024-09-17T04:49:36.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.877+0000 7f8a22c67640 1 --2- 172.21.15.150:0/4159711440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a1c074450 0x7f8a1c074860 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.877+0000 7f8a22c67640 1 --2- 172.21.15.150:0/4159711440 >> v2:172.21.15.176:3300/0 conn(0x7f8a1c073440 0x7f8a1c0a7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.877+0000 7f8a22c67640 1 -- 172.21.15.150:0/4159711440 >> 172.21.15.150:0/4159711440 conn(0x7f8a1c0c4a00 msgr2=0x7f8a1c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:36.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.877+0000 7f8a22c67640 1 -- 172.21.15.150:0/4159711440 shutdown_connections 2024-09-17T04:49:36.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.877+0000 7f8a22c67640 1 -- 172.21.15.150:0/4159711440 wait complete. 2024-09-17T04:49:36.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a22c67640 1 Processor -- start 2024-09-17T04:49:36.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a22c67640 1 -- start start 2024-09-17T04:49:36.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a22c67640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1c070e50 0x7f8a1c1ed6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:36.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a22c67640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a1c073440 0x7f8a1c156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:36.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a22c67640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a1c074450 0x7f8a1c143d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:36.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a22c67640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1c0ade60 con 0x7f8a1c074450 2024-09-17T04:49:36.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a22c67640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1c0adce0 con 0x7f8a1c073440 2024-09-17T04:49:36.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a22c67640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8a1c0ad610 con 0x7f8a1c070e50 2024-09-17T04:49:36.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a21204640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a1c074450 0x7f8a1c143d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:36.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a21204640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a1c074450 0x7f8a1c143d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43076/0 (socket says 172.21.15.150:43076) 2024-09-17T04:49:36.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a21204640 1 -- 172.21.15.150:0/1176621512 learned_addr learned my addr 172.21.15.150:0/1176621512 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:36.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a20a03640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1c070e50 0x7f8a1c1ed6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:36.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a1bfff640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a1c073440 0x7f8a1c156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:36.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a21204640 1 -- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1c070e50 msgr2=0x7f8a1c1ed6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a21204640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1c070e50 0x7f8a1c1ed6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a21204640 1 -- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a1c073440 msgr2=0x7f8a1c156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:36.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a21204640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a1c073440 0x7f8a1c156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:36.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a21204640 1 -- 172.21.15.150:0/1176621512 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a1c147eb0 con 0x7f8a1c074450 2024-09-17T04:49:36.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a21204640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a1c074450 0x7f8a1c143d10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c0b0040 tx=0x7f8a0c0a6780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:36.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a197fa640 1 -- 172.21.15.150:0/1176621512 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c0bb020 con 0x7f8a1c074450 2024-09-17T04:49:36.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a197fa640 1 -- 172.21.15.150:0/1176621512 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a0c0044d0 con 0x7f8a1c074450 2024-09-17T04:49:36.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a22c67640 1 -- 172.21.15.150:0/1176621512 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a1c1f0d60 con 0x7f8a1c074450 2024-09-17T04:49:36.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a197fa640 1 -- 172.21.15.150:0/1176621512 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c003860 con 0x7f8a1c074450 2024-09-17T04:49:36.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a22c67640 1 -- 172.21.15.150:0/1176621512 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a1c1f1000 con 0x7f8a1c074450 2024-09-17T04:49:36.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a22c67640 1 -- 172.21.15.150:0/1176621512 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a1c1475b0 con 0x7f8a1c074450 2024-09-17T04:49:36.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.881+0000 7f8a197fa640 1 -- 172.21.15.150:0/1176621512 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8a0c0be020 con 0x7f8a1c074450 2024-09-17T04:49:36.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.885+0000 7f8a197fa640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89fc073200 0x7f89fc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:36.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.885+0000 7f8a197fa640 1 -- 172.21.15.150:0/1176621512 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f8a0c12ee90 con 0x7f8a1c074450 2024-09-17T04:49:36.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.885+0000 7f8a20a03640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89fc073200 0x7f89fc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:36.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.885+0000 7f8a20a03640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89fc073200 0x7f89fc075710 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8a140058d0 tx=0x7f8a14005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:36.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:36.889+0000 7f8a197fa640 1 -- 172.21.15.150:0/1176621512 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8a0c0c5030 con 0x7f8a1c074450 2024-09-17T04:49:37.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.057+0000 7f8a22c67640 1 -- 172.21.15.150:0/1176621512 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8a1c0ad9e0 con 0x7f89fc073200 2024-09-17T04:49:37.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.069+0000 7f8a197fa640 1 -- 172.21.15.150:0/1176621512 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251941 (secure 0 0 0) 0x7f8a1c0ad9e0 con 0x7f89fc073200 2024-09-17T04:49:37.075 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:37.076 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T04:49:37.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.077+0000 7f8a22c67640 1 -- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89fc073200 msgr2=0x7f89fc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:37.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.077+0000 7f8a22c67640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89fc073200 0x7f89fc075710 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8a140058d0 tx=0x7f8a14005860 comp rx=0 tx=0).stop 2024-09-17T04:49:37.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.077+0000 7f8a22c67640 1 -- 172.21.15.150:0/1176621512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a1c074450 msgr2=0x7f8a1c143d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:37.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.077+0000 7f8a22c67640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a1c074450 0x7f8a1c143d10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c0b0040 tx=0x7f8a0c0a6780 comp rx=0 tx=0).stop 2024-09-17T04:49:37.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.077+0000 7f8a22c67640 1 -- 172.21.15.150:0/1176621512 shutdown_connections 2024-09-17T04:49:37.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.077+0000 7f8a22c67640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89fc073200 0x7f89fc075710 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.077+0000 7f8a22c67640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a1c073440 0x7f8a1c156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.077+0000 7f8a22c67640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a1c074450 0x7f8a1c143d10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.077+0000 7f8a22c67640 1 --2- 172.21.15.150:0/1176621512 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a1c070e50 0x7f8a1c1ed6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.077+0000 7f8a22c67640 1 -- 172.21.15.150:0/1176621512 >> 172.21.15.150:0/1176621512 conn(0x7f8a1c0c4a00 msgr2=0x7f8a1c15e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:37.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.077+0000 7f8a22c67640 1 -- 172.21.15.150:0/1176621512 shutdown_connections 2024-09-17T04:49:37.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.077+0000 7f8a22c67640 1 -- 172.21.15.150:0/1176621512 wait complete. 2024-09-17T04:49:37.096 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-17T04:49:36.942217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223376,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751344,"statfs":{"total":773094113280,"available":772865376256,"internally_reserved":0,"allocated":3022848,"data_stored":1795188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":56,"apply_latency_ms":56,"commit_latency_ns":56000000,"apply_latency_ns":56000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.093037"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406997+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T04:49:33.406997+0000","last_peered":"2024-09-17T04:49:33.406997+0000","last_clean":"2024-09-17T04:49:33.406997+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T04:49:33.406997+0000","last_undegraded":"2024-09-17T04:49:33.406997+0000","last_fullsized":"2024-09-17T04:49:33.406997+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403685+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T04:49:33.403685+0000","last_peered":"2024-09-17T04:49:33.403685+0000","last_clean":"2024-09-17T04:49:33.403685+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T04:49:33.403685+0000","last_undegraded":"2024-09-17T04:49:33.403685+0000","last_fullsized":"2024-09-17T04:49:33.403685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403283+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T04:49:33.403283+0000","last_peered":"2024-09-17T04:49:33.403283+0000","last_clean":"2024-09-17T04:49:33.403283+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T04:49:33.403283+0000","last_undegraded":"2024-09-17T04:49:33.403283+0000","last_fullsized":"2024-09-17T04:49:33.403283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.405271+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T04:49:33.405271+0000","last_peered":"2024-09-17T04:49:33.405271+0000","last_clean":"2024-09-17T04:49:33.405271+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T04:49:33.405271+0000","last_undegraded":"2024-09-17T04:49:33.405271+0000","last_fullsized":"2024-09-17T04:49:33.405271+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407231+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T04:49:33.407231+0000","last_peered":"2024-09-17T04:49:33.407231+0000","last_clean":"2024-09-17T04:49:33.407231+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T04:49:33.407231+0000","last_undegraded":"2024-09-17T04:49:33.407231+0000","last_fullsized":"2024-09-17T04:49:33.407231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404936+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T04:49:33.404936+0000","last_peered":"2024-09-17T04:49:33.404936+0000","last_clean":"2024-09-17T04:49:33.404936+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T04:49:33.404936+0000","last_undegraded":"2024-09-17T04:49:33.404936+0000","last_fullsized":"2024-09-17T04:49:33.404936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407864+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T04:49:33.407864+0000","last_peered":"2024-09-17T04:49:33.407864+0000","last_clean":"2024-09-17T04:49:33.407864+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T04:49:33.407864+0000","last_undegraded":"2024-09-17T04:49:33.407864+0000","last_fullsized":"2024-09-17T04:49:33.407864+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403454+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T04:49:33.403454+0000","last_peered":"2024-09-17T04:49:33.403454+0000","last_clean":"2024-09-17T04:49:33.403454+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T04:49:33.403454+0000","last_undegraded":"2024-09-17T04:49:33.403454+0000","last_fullsized":"2024-09-17T04:49:33.403454+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890786+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T04:49:32.890786+0000","last_peered":"2024-09-17T04:49:32.890786+0000","last_clean":"2024-09-17T04:49:32.890786+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T04:49:32.890786+0000","last_undegraded":"2024-09-17T04:49:32.890786+0000","last_fullsized":"2024-09-17T04:49:32.890786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407254+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T04:49:33.407254+0000","last_peered":"2024-09-17T04:49:33.407254+0000","last_clean":"2024-09-17T04:49:33.407254+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T04:49:33.407254+0000","last_undegraded":"2024-09-17T04:49:33.407254+0000","last_fullsized":"2024-09-17T04:49:33.407254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.405405+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T04:49:33.405405+0000","last_peered":"2024-09-17T04:49:33.405405+0000","last_clean":"2024-09-17T04:49:33.405405+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T04:49:33.405405+0000","last_undegraded":"2024-09-17T04:49:33.405405+0000","last_fullsized":"2024-09-17T04:49:33.405405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.408154+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T04:49:33.408154+0000","last_peered":"2024-09-17T04:49:33.408154+0000","last_clean":"2024-09-17T04:49:33.408154+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T04:49:33.408154+0000","last_undegraded":"2024-09-17T04:49:33.408154+0000","last_fullsized":"2024-09-17T04:49:33.408154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892510+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T04:49:32.892510+0000","last_peered":"2024-09-17T04:49:32.892510+0000","last_clean":"2024-09-17T04:49:32.892510+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T04:49:32.892510+0000","last_undegraded":"2024-09-17T04:49:32.892510+0000","last_fullsized":"2024-09-17T04:49:32.892510+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892102+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T04:49:32.892102+0000","last_peered":"2024-09-17T04:49:32.892102+0000","last_clean":"2024-09-17T04:49:32.892102+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T04:49:32.892102+0000","last_undegraded":"2024-09-17T04:49:32.892102+0000","last_fullsized":"2024-09-17T04:49:32.892102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.402966+0000","last_change":"2024-09-17T04:49:31.911036+0000","last_active":"2024-09-17T04:49:33.402966+0000","last_peered":"2024-09-17T04:49:33.402966+0000","last_clean":"2024-09-17T04:49:33.402966+0000","last_became_active":"2024-09-17T04:49:31.910285+0000","last_became_peered":"2024-09-17T04:49:31.910285+0000","last_unstale":"2024-09-17T04:49:33.402966+0000","last_undegraded":"2024-09-17T04:49:33.402966+0000","last_fullsized":"2024-09-17T04:49:33.402966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406489+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T04:49:33.406489+0000","last_peered":"2024-09-17T04:49:33.406489+0000","last_clean":"2024-09-17T04:49:33.406489+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T04:49:33.406489+0000","last_undegraded":"2024-09-17T04:49:33.406489+0000","last_fullsized":"2024-09-17T04:49:33.406489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410347+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T04:49:33.410347+0000","last_peered":"2024-09-17T04:49:33.410347+0000","last_clean":"2024-09-17T04:49:33.410347+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T04:49:33.410347+0000","last_undegraded":"2024-09-17T04:49:33.410347+0000","last_fullsized":"2024-09-17T04:49:33.410347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.995271+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T04:49:32.995271+0000","last_peered":"2024-09-17T04:49:32.995271+0000","last_clean":"2024-09-17T04:49:32.995271+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T04:49:32.995271+0000","last_undegraded":"2024-09-17T04:49:32.995271+0000","last_fullsized":"2024-09-17T04:49:32.995271+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410470+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T04:49:33.410470+0000","last_peered":"2024-09-17T04:49:33.410470+0000","last_clean":"2024-09-17T04:49:33.410470+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T04:49:33.410470+0000","last_undegraded":"2024-09-17T04:49:33.410470+0000","last_fullsized":"2024-09-17T04:49:33.410470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.408650+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T04:49:33.408650+0000","last_peered":"2024-09-17T04:49:33.408650+0000","last_clean":"2024-09-17T04:49:33.408650+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T04:49:33.408650+0000","last_undegraded":"2024-09-17T04:49:33.408650+0000","last_fullsized":"2024-09-17T04:49:33.408650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889265+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T04:49:32.889265+0000","last_peered":"2024-09-17T04:49:32.889265+0000","last_clean":"2024-09-17T04:49:32.889265+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T04:49:32.889265+0000","last_undegraded":"2024-09-17T04:49:32.889265+0000","last_fullsized":"2024-09-17T04:49:32.889265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889697+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T04:49:32.889697+0000","last_peered":"2024-09-17T04:49:32.889697+0000","last_clean":"2024-09-17T04:49:32.889697+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T04:49:32.889697+0000","last_undegraded":"2024-09-17T04:49:32.889697+0000","last_fullsized":"2024-09-17T04:49:32.889697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403082+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T04:49:33.403082+0000","last_peered":"2024-09-17T04:49:33.403082+0000","last_clean":"2024-09-17T04:49:33.403082+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T04:49:33.403082+0000","last_undegraded":"2024-09-17T04:49:33.403082+0000","last_fullsized":"2024-09-17T04:49:33.403082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404523+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T04:49:33.404523+0000","last_peered":"2024-09-17T04:49:33.404523+0000","last_clean":"2024-09-17T04:49:33.404523+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T04:49:33.404523+0000","last_undegraded":"2024-09-17T04:49:33.404523+0000","last_fullsized":"2024-09-17T04:49:33.404523+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403996+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T04:49:33.403996+0000","last_peered":"2024-09-17T04:49:33.403996+0000","last_clean":"2024-09-17T04:49:33.403996+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T04:49:33.403996+0000","last_undegraded":"2024-09-17T04:49:33.403996+0000","last_fullsized":"2024-09-17T04:49:33.403996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406592+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T04:49:33.406592+0000","last_peered":"2024-09-17T04:49:33.406592+0000","last_clean":"2024-09-17T04:49:33.406592+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T04:49:33.406592+0000","last_undegraded":"2024-09-17T04:49:33.406592+0000","last_fullsized":"2024-09-17T04:49:33.406592+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892882+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T04:49:32.892882+0000","last_peered":"2024-09-17T04:49:32.892882+0000","last_clean":"2024-09-17T04:49:32.892882+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T04:49:32.892882+0000","last_undegraded":"2024-09-17T04:49:32.892882+0000","last_fullsized":"2024-09-17T04:49:32.892882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892925+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T04:49:32.892925+0000","last_peered":"2024-09-17T04:49:32.892925+0000","last_clean":"2024-09-17T04:49:32.892925+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T04:49:32.892925+0000","last_undegraded":"2024-09-17T04:49:32.892925+0000","last_fullsized":"2024-09-17T04:49:32.892925+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403575+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T04:49:33.403575+0000","last_peered":"2024-09-17T04:49:33.403575+0000","last_clean":"2024-09-17T04:49:33.403575+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T04:49:33.403575+0000","last_undegraded":"2024-09-17T04:49:33.403575+0000","last_fullsized":"2024-09-17T04:49:33.403575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409144+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T04:49:33.409144+0000","last_peered":"2024-09-17T04:49:33.409144+0000","last_clean":"2024-09-17T04:49:33.409144+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T04:49:33.409144+0000","last_undegraded":"2024-09-17T04:49:33.409144+0000","last_fullsized":"2024-09-17T04:49:33.409144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.994822+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T04:49:32.994822+0000","last_peered":"2024-09-17T04:49:32.994822+0000","last_clean":"2024-09-17T04:49:32.994822+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T04:49:32.994822+0000","last_undegraded":"2024-09-17T04:49:32.994822+0000","last_fullsized":"2024-09-17T04:49:32.994822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404526+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T04:49:33.404526+0000","last_peered":"2024-09-17T04:49:33.404526+0000","last_clean":"2024-09-17T04:49:33.404526+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T04:49:33.404526+0000","last_undegraded":"2024-09-17T04:49:33.404526+0000","last_fullsized":"2024-09-17T04:49:33.404526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890207+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T04:49:32.890207+0000","last_peered":"2024-09-17T04:49:32.890207+0000","last_clean":"2024-09-17T04:49:32.890207+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T04:49:32.890207+0000","last_undegraded":"2024-09-17T04:49:32.890207+0000","last_fullsized":"2024-09-17T04:49:32.890207+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403470+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T04:49:33.403470+0000","last_peered":"2024-09-17T04:49:33.403470+0000","last_clean":"2024-09-17T04:49:33.403470+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T04:49:33.403470+0000","last_undegraded":"2024-09-17T04:49:33.403470+0000","last_fullsized":"2024-09-17T04:49:33.403470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890205+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T04:49:32.890205+0000","last_peered":"2024-09-17T04:49:32.890205+0000","last_clean":"2024-09-17T04:49:32.890205+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T04:49:32.890205+0000","last_undegraded":"2024-09-17T04:49:32.890205+0000","last_fullsized":"2024-09-17T04:49:32.890205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409641+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T04:49:33.409641+0000","last_peered":"2024-09-17T04:49:33.409641+0000","last_clean":"2024-09-17T04:49:33.409641+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T04:49:33.409641+0000","last_undegraded":"2024-09-17T04:49:33.409641+0000","last_fullsized":"2024-09-17T04:49:33.409641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.891281+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T04:49:32.891281+0000","last_peered":"2024-09-17T04:49:32.891281+0000","last_clean":"2024-09-17T04:49:32.891281+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T04:49:32.891281+0000","last_undegraded":"2024-09-17T04:49:32.891281+0000","last_fullsized":"2024-09-17T04:49:32.891281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404563+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T04:49:33.404563+0000","last_peered":"2024-09-17T04:49:33.404563+0000","last_clean":"2024-09-17T04:49:33.404563+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T04:49:33.404563+0000","last_undegraded":"2024-09-17T04:49:33.404563+0000","last_fullsized":"2024-09-17T04:49:33.404563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406532+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T04:49:33.406532+0000","last_peered":"2024-09-17T04:49:33.406532+0000","last_clean":"2024-09-17T04:49:33.406532+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T04:49:33.406532+0000","last_undegraded":"2024-09-17T04:49:33.406532+0000","last_fullsized":"2024-09-17T04:49:33.406532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889317+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T04:49:32.889317+0000","last_peered":"2024-09-17T04:49:32.889317+0000","last_clean":"2024-09-17T04:49:32.889317+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T04:49:32.889317+0000","last_undegraded":"2024-09-17T04:49:32.889317+0000","last_fullsized":"2024-09-17T04:49:32.889317+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890106+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T04:49:32.890106+0000","last_peered":"2024-09-17T04:49:32.890106+0000","last_clean":"2024-09-17T04:49:32.890106+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T04:49:32.890106+0000","last_undegraded":"2024-09-17T04:49:32.890106+0000","last_fullsized":"2024-09-17T04:49:32.890106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407222+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T04:49:33.407222+0000","last_peered":"2024-09-17T04:49:33.407222+0000","last_clean":"2024-09-17T04:49:33.407222+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T04:49:33.407222+0000","last_undegraded":"2024-09-17T04:49:33.407222+0000","last_fullsized":"2024-09-17T04:49:33.407222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407474+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T04:49:33.407474+0000","last_peered":"2024-09-17T04:49:33.407474+0000","last_clean":"2024-09-17T04:49:33.407474+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T04:49:33.407474+0000","last_undegraded":"2024-09-17T04:49:33.407474+0000","last_fullsized":"2024-09-17T04:49:33.407474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404483+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T04:49:33.404483+0000","last_peered":"2024-09-17T04:49:33.404483+0000","last_clean":"2024-09-17T04:49:33.404483+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T04:49:33.404483+0000","last_undegraded":"2024-09-17T04:49:33.404483+0000","last_fullsized":"2024-09-17T04:49:33.404483+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.994691+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T04:49:32.994691+0000","last_peered":"2024-09-17T04:49:32.994691+0000","last_clean":"2024-09-17T04:49:32.994691+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T04:49:32.994691+0000","last_undegraded":"2024-09-17T04:49:32.994691+0000","last_fullsized":"2024-09-17T04:49:32.994691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409414+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T04:49:33.409414+0000","last_peered":"2024-09-17T04:49:33.409414+0000","last_clean":"2024-09-17T04:49:33.409414+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T04:49:33.409414+0000","last_undegraded":"2024-09-17T04:49:33.409414+0000","last_fullsized":"2024-09-17T04:49:33.409414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.405756+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T04:49:33.405756+0000","last_peered":"2024-09-17T04:49:33.405756+0000","last_clean":"2024-09-17T04:49:33.405756+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T04:49:33.405756+0000","last_undegraded":"2024-09-17T04:49:33.405756+0000","last_fullsized":"2024-09-17T04:49:33.405756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.405787+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T04:49:33.405787+0000","last_peered":"2024-09-17T04:49:33.405787+0000","last_clean":"2024-09-17T04:49:33.405787+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T04:49:33.405787+0000","last_undegraded":"2024-09-17T04:49:33.405787+0000","last_fullsized":"2024-09-17T04:49:33.405787+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.995219+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T04:49:32.995219+0000","last_peered":"2024-09-17T04:49:32.995219+0000","last_clean":"2024-09-17T04:49:32.995219+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T04:49:32.995219+0000","last_undegraded":"2024-09-17T04:49:32.995219+0000","last_fullsized":"2024-09-17T04:49:32.995219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892585+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T04:49:32.892585+0000","last_peered":"2024-09-17T04:49:32.892585+0000","last_clean":"2024-09-17T04:49:32.892585+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T04:49:32.892585+0000","last_undegraded":"2024-09-17T04:49:32.892585+0000","last_fullsized":"2024-09-17T04:49:32.892585+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410773+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T04:49:33.410773+0000","last_peered":"2024-09-17T04:49:33.410773+0000","last_clean":"2024-09-17T04:49:33.410773+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T04:49:33.410773+0000","last_undegraded":"2024-09-17T04:49:33.410773+0000","last_fullsized":"2024-09-17T04:49:33.410773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409979+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T04:49:33.409979+0000","last_peered":"2024-09-17T04:49:33.409979+0000","last_clean":"2024-09-17T04:49:33.409979+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T04:49:33.409979+0000","last_undegraded":"2024-09-17T04:49:33.409979+0000","last_fullsized":"2024-09-17T04:49:33.409979+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406471+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T04:49:33.406471+0000","last_peered":"2024-09-17T04:49:33.406471+0000","last_clean":"2024-09-17T04:49:33.406471+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T04:49:33.406471+0000","last_undegraded":"2024-09-17T04:49:33.406471+0000","last_fullsized":"2024-09-17T04:49:33.406471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889195+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T04:49:32.889195+0000","last_peered":"2024-09-17T04:49:32.889195+0000","last_clean":"2024-09-17T04:49:32.889195+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T04:49:32.889195+0000","last_undegraded":"2024-09-17T04:49:32.889195+0000","last_fullsized":"2024-09-17T04:49:32.889195+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889539+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T04:49:32.889539+0000","last_peered":"2024-09-17T04:49:32.889539+0000","last_clean":"2024-09-17T04:49:32.889539+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T04:49:32.889539+0000","last_undegraded":"2024-09-17T04:49:32.889539+0000","last_fullsized":"2024-09-17T04:49:32.889539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407977+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T04:49:33.407977+0000","last_peered":"2024-09-17T04:49:33.407977+0000","last_clean":"2024-09-17T04:49:33.407977+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T04:49:33.407977+0000","last_undegraded":"2024-09-17T04:49:33.407977+0000","last_fullsized":"2024-09-17T04:49:33.407977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407515+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T04:49:33.407515+0000","last_peered":"2024-09-17T04:49:33.407515+0000","last_clean":"2024-09-17T04:49:33.407515+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T04:49:33.407515+0000","last_undegraded":"2024-09-17T04:49:33.407515+0000","last_fullsized":"2024-09-17T04:49:33.407515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407798+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T04:49:33.407798+0000","last_peered":"2024-09-17T04:49:33.407798+0000","last_clean":"2024-09-17T04:49:33.407798+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T04:49:33.407798+0000","last_undegraded":"2024-09-17T04:49:33.407798+0000","last_fullsized":"2024-09-17T04:49:33.407798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890372+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T04:49:32.890372+0000","last_peered":"2024-09-17T04:49:32.890372+0000","last_clean":"2024-09-17T04:49:32.890372+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T04:49:32.890372+0000","last_undegraded":"2024-09-17T04:49:32.890372+0000","last_fullsized":"2024-09-17T04:49:32.890372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407145+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T04:49:33.407145+0000","last_peered":"2024-09-17T04:49:33.407145+0000","last_clean":"2024-09-17T04:49:33.407145+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T04:49:33.407145+0000","last_undegraded":"2024-09-17T04:49:33.407145+0000","last_fullsized":"2024-09-17T04:49:33.407145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403310+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T04:49:33.403310+0000","last_peered":"2024-09-17T04:49:33.403310+0000","last_clean":"2024-09-17T04:49:33.403310+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T04:49:33.403310+0000","last_undegraded":"2024-09-17T04:49:33.403310+0000","last_fullsized":"2024-09-17T04:49:33.403310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.408579+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T04:49:33.408579+0000","last_peered":"2024-09-17T04:49:33.408579+0000","last_clean":"2024-09-17T04:49:33.408579+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T04:49:33.408579+0000","last_undegraded":"2024-09-17T04:49:33.408579+0000","last_fullsized":"2024-09-17T04:49:33.408579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892968+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T04:49:32.892968+0000","last_peered":"2024-09-17T04:49:32.892968+0000","last_clean":"2024-09-17T04:49:32.892968+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T04:49:32.892968+0000","last_undegraded":"2024-09-17T04:49:32.892968+0000","last_fullsized":"2024-09-17T04:49:32.892968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410052+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T04:49:33.410052+0000","last_peered":"2024-09-17T04:49:33.410052+0000","last_clean":"2024-09-17T04:49:33.410052+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T04:49:33.410052+0000","last_undegraded":"2024-09-17T04:49:33.410052+0000","last_fullsized":"2024-09-17T04:49:33.410052+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":111,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889214+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T04:49:32.889214+0000","last_peered":"2024-09-17T04:49:32.889214+0000","last_clean":"2024-09-17T04:49:32.889214+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T04:49:32.889214+0000","last_undegraded":"2024-09-17T04:49:32.889214+0000","last_fullsized":"2024-09-17T04:49:32.889214+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409562+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T04:49:33.409562+0000","last_peered":"2024-09-17T04:49:33.409562+0000","last_clean":"2024-09-17T04:49:33.409562+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T04:49:33.409562+0000","last_undegraded":"2024-09-17T04:49:33.409562+0000","last_fullsized":"2024-09-17T04:49:33.409562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889412+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T04:49:32.889412+0000","last_peered":"2024-09-17T04:49:32.889412+0000","last_clean":"2024-09-17T04:49:32.889412+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T04:49:32.889412+0000","last_undegraded":"2024-09-17T04:49:32.889412+0000","last_fullsized":"2024-09-17T04:49:32.889412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406888+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T04:49:33.406888+0000","last_peered":"2024-09-17T04:49:33.406888+0000","last_clean":"2024-09-17T04:49:33.406888+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T04:49:33.406888+0000","last_undegraded":"2024-09-17T04:49:33.406888+0000","last_fullsized":"2024-09-17T04:49:33.406888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.891390+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T04:49:32.891390+0000","last_peered":"2024-09-17T04:49:32.891390+0000","last_clean":"2024-09-17T04:49:32.891390+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T04:49:32.891390+0000","last_undegraded":"2024-09-17T04:49:32.891390+0000","last_fullsized":"2024-09-17T04:49:32.891390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.994261+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T04:49:32.994261+0000","last_peered":"2024-09-17T04:49:32.994261+0000","last_clean":"2024-09-17T04:49:32.994261+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T04:49:32.994261+0000","last_undegraded":"2024-09-17T04:49:32.994261+0000","last_fullsized":"2024-09-17T04:49:32.994261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403329+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T04:49:33.403329+0000","last_peered":"2024-09-17T04:49:33.403329+0000","last_clean":"2024-09-17T04:49:33.403329+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T04:49:33.403329+0000","last_undegraded":"2024-09-17T04:49:33.403329+0000","last_fullsized":"2024-09-17T04:49:33.403329+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410800+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T04:49:33.410800+0000","last_peered":"2024-09-17T04:49:33.410800+0000","last_clean":"2024-09-17T04:49:33.410800+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T04:49:33.410800+0000","last_undegraded":"2024-09-17T04:49:33.410800+0000","last_fullsized":"2024-09-17T04:49:33.410800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407387+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T04:49:33.407387+0000","last_peered":"2024-09-17T04:49:33.407387+0000","last_clean":"2024-09-17T04:49:33.407387+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T04:49:33.407387+0000","last_undegraded":"2024-09-17T04:49:33.407387+0000","last_fullsized":"2024-09-17T04:49:33.407387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.402938+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T04:49:33.402938+0000","last_peered":"2024-09-17T04:49:33.402938+0000","last_clean":"2024-09-17T04:49:33.402938+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T04:49:33.402938+0000","last_undegraded":"2024-09-17T04:49:33.402938+0000","last_fullsized":"2024-09-17T04:49:33.402938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.405916+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T04:49:33.405916+0000","last_peered":"2024-09-17T04:49:33.405916+0000","last_clean":"2024-09-17T04:49:33.405916+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T04:49:33.405916+0000","last_undegraded":"2024-09-17T04:49:33.405916+0000","last_fullsized":"2024-09-17T04:49:33.405916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410517+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T04:49:33.410517+0000","last_peered":"2024-09-17T04:49:33.410517+0000","last_clean":"2024-09-17T04:49:33.410517+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T04:49:33.410517+0000","last_undegraded":"2024-09-17T04:49:33.410517+0000","last_fullsized":"2024-09-17T04:49:33.410517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892861+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T04:49:32.892861+0000","last_peered":"2024-09-17T04:49:32.892861+0000","last_clean":"2024-09-17T04:49:32.892861+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T04:49:32.892861+0000","last_undegraded":"2024-09-17T04:49:32.892861+0000","last_fullsized":"2024-09-17T04:49:32.892861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890648+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T04:49:32.890648+0000","last_peered":"2024-09-17T04:49:32.890648+0000","last_clean":"2024-09-17T04:49:32.890648+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T04:49:32.890648+0000","last_undegraded":"2024-09-17T04:49:32.890648+0000","last_fullsized":"2024-09-17T04:49:32.890648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.891083+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T04:49:32.891083+0000","last_peered":"2024-09-17T04:49:32.891083+0000","last_clean":"2024-09-17T04:49:32.891083+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T04:49:32.891083+0000","last_undegraded":"2024-09-17T04:49:32.891083+0000","last_fullsized":"2024-09-17T04:49:32.891083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410223+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T04:49:33.410223+0000","last_peered":"2024-09-17T04:49:33.410223+0000","last_clean":"2024-09-17T04:49:33.410223+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T04:49:33.410223+0000","last_undegraded":"2024-09-17T04:49:33.410223+0000","last_fullsized":"2024-09-17T04:49:33.410223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890056+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T04:49:32.890056+0000","last_peered":"2024-09-17T04:49:32.890056+0000","last_clean":"2024-09-17T04:49:32.890056+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T04:49:32.890056+0000","last_undegraded":"2024-09-17T04:49:32.890056+0000","last_fullsized":"2024-09-17T04:49:32.890056+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.405369+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T04:49:33.405369+0000","last_peered":"2024-09-17T04:49:33.405369+0000","last_clean":"2024-09-17T04:49:33.405369+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T04:49:33.405369+0000","last_undegraded":"2024-09-17T04:49:33.405369+0000","last_fullsized":"2024-09-17T04:49:33.405369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.408507+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T04:49:33.408507+0000","last_peered":"2024-09-17T04:49:33.408507+0000","last_clean":"2024-09-17T04:49:33.408507+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T04:49:33.408507+0000","last_undegraded":"2024-09-17T04:49:33.408507+0000","last_fullsized":"2024-09-17T04:49:33.408507+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889173+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T04:49:32.889173+0000","last_peered":"2024-09-17T04:49:32.889173+0000","last_clean":"2024-09-17T04:49:32.889173+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T04:49:32.889173+0000","last_undegraded":"2024-09-17T04:49:32.889173+0000","last_fullsized":"2024-09-17T04:49:32.889173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889026+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T04:49:32.889026+0000","last_peered":"2024-09-17T04:49:32.889026+0000","last_clean":"2024-09-17T04:49:32.889026+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T04:49:32.889026+0000","last_undegraded":"2024-09-17T04:49:32.889026+0000","last_fullsized":"2024-09-17T04:49:32.889026+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892293+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T04:49:32.892293+0000","last_peered":"2024-09-17T04:49:32.892293+0000","last_clean":"2024-09-17T04:49:32.892293+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T04:49:32.892293+0000","last_undegraded":"2024-09-17T04:49:32.892293+0000","last_fullsized":"2024-09-17T04:49:32.892293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.995392+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T04:49:32.995392+0000","last_peered":"2024-09-17T04:49:32.995392+0000","last_clean":"2024-09-17T04:49:32.995392+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T04:49:32.995392+0000","last_undegraded":"2024-09-17T04:49:32.995392+0000","last_fullsized":"2024-09-17T04:49:32.995392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409194+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T04:49:33.409194+0000","last_peered":"2024-09-17T04:49:33.409194+0000","last_clean":"2024-09-17T04:49:33.409194+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T04:49:33.409194+0000","last_undegraded":"2024-09-17T04:49:33.409194+0000","last_fullsized":"2024-09-17T04:49:33.409194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409215+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T04:49:33.409215+0000","last_peered":"2024-09-17T04:49:33.409215+0000","last_clean":"2024-09-17T04:49:33.409215+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T04:49:33.409215+0000","last_undegraded":"2024-09-17T04:49:33.409215+0000","last_fullsized":"2024-09-17T04:49:33.409215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409539+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T04:49:33.409539+0000","last_peered":"2024-09-17T04:49:33.409539+0000","last_clean":"2024-09-17T04:49:33.409539+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T04:49:33.409539+0000","last_undegraded":"2024-09-17T04:49:33.409539+0000","last_fullsized":"2024-09-17T04:49:33.409539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.408917+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T04:49:33.408917+0000","last_peered":"2024-09-17T04:49:33.408917+0000","last_clean":"2024-09-17T04:49:33.408917+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T04:49:33.408917+0000","last_undegraded":"2024-09-17T04:49:33.408917+0000","last_fullsized":"2024-09-17T04:49:33.408917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890034+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T04:49:32.890034+0000","last_peered":"2024-09-17T04:49:32.890034+0000","last_clean":"2024-09-17T04:49:32.890034+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T04:49:32.890034+0000","last_undegraded":"2024-09-17T04:49:32.890034+0000","last_fullsized":"2024-09-17T04:49:32.890034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406048+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T04:49:33.406048+0000","last_peered":"2024-09-17T04:49:33.406048+0000","last_clean":"2024-09-17T04:49:33.406048+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T04:49:33.406048+0000","last_undegraded":"2024-09-17T04:49:33.406048+0000","last_fullsized":"2024-09-17T04:49:33.406048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404083+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T04:49:33.404083+0000","last_peered":"2024-09-17T04:49:33.404083+0000","last_clean":"2024-09-17T04:49:33.404083+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T04:49:33.404083+0000","last_undegraded":"2024-09-17T04:49:33.404083+0000","last_fullsized":"2024-09-17T04:49:33.404083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404168+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T04:49:33.404168+0000","last_peered":"2024-09-17T04:49:33.404168+0000","last_clean":"2024-09-17T04:49:33.404168+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T04:49:33.404168+0000","last_undegraded":"2024-09-17T04:49:33.404168+0000","last_fullsized":"2024-09-17T04:49:33.404168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404076+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T04:49:33.404076+0000","last_peered":"2024-09-17T04:49:33.404076+0000","last_clean":"2024-09-17T04:49:33.404076+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T04:49:33.404076+0000","last_undegraded":"2024-09-17T04:49:33.404076+0000","last_fullsized":"2024-09-17T04:49:33.404076+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.891572+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T04:49:32.891572+0000","last_peered":"2024-09-17T04:49:32.891572+0000","last_clean":"2024-09-17T04:49:32.891572+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T04:49:32.891572+0000","last_undegraded":"2024-09-17T04:49:32.891572+0000","last_fullsized":"2024-09-17T04:49:32.891572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":76328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":823296,"data_stored":666664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":241664,"data_stored":78488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":77816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":819200,"data_stored":666642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":229376,"data_stored":76260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":76204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:49:37.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:49:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.285+0000 7f42626ef640 1 -- 172.21.15.150:0/3808892097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c0c3db0 msgr2=0x7f425c0ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.285+0000 7f42626ef640 1 --2- 172.21.15.150:0/3808892097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c0c3db0 0x7f425c0ad2a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f425c06f7a0 tx=0x7f424c01c8a0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.285+0000 7f42626ef640 1 -- 172.21.15.150:0/3808892097 shutdown_connections 2024-09-17T04:49:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.285+0000 7f42626ef640 1 --2- 172.21.15.150:0/3808892097 >> v2:172.21.15.176:3300/0 conn(0x7f425c0c41c0 0x7f425c0ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.285+0000 7f42626ef640 1 --2- 172.21.15.150:0/3808892097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c0c3db0 0x7f425c0ad2a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.285+0000 7f42626ef640 1 -- 172.21.15.150:0/3808892097 >> 172.21.15.150:0/3808892097 conn(0x7f425c0c4a00 msgr2=0x7f425c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:37.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.285+0000 7f42626ef640 1 -- 172.21.15.150:0/3808892097 shutdown_connections 2024-09-17T04:49:37.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.285+0000 7f42626ef640 1 -- 172.21.15.150:0/3808892097 wait complete. 2024-09-17T04:49:37.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f42626ef640 1 Processor -- start 2024-09-17T04:49:37.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f42626ef640 1 -- start start 2024-09-17T04:49:37.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f42626ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c06fab0 0x7f425c1479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f42626ef640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f425c0c3db0 0x7f425c1f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f42626ef640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f425c0c41c0 0x7f425c150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f42626ef640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f425c070750 con 0x7f425c06fab0 2024-09-17T04:49:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f42626ef640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f425c0705d0 con 0x7f425c0c41c0 2024-09-17T04:49:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f42626ef640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f425c0a75f0 con 0x7f425c0c3db0 2024-09-17T04:49:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f425bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c06fab0 0x7f425c1479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f425bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c06fab0 0x7f425c1479e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43090/0 (socket says 172.21.15.150:43090) 2024-09-17T04:49:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f425bfff640 1 -- 172.21.15.150:0/1821371536 learned_addr learned my addr 172.21.15.150:0/1821371536 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f425bfff640 1 -- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f425c0c3db0 msgr2=0x7f425c1f0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f425bfff640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f425c0c3db0 0x7f425c1f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f425bfff640 1 -- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f425c0c41c0 msgr2=0x7f425c150540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f4260c8c640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f425c0c41c0 0x7f425c150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f425bfff640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f425c0c41c0 0x7f425c150540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f425bfff640 1 -- 172.21.15.150:0/1821371536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f425c06e6d0 con 0x7f425c06fab0 2024-09-17T04:49:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f4260c8c640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f425c0c41c0 0x7f425c150540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f425bfff640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c06fab0 0x7f425c1479e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f424c0b0040 tx=0x7f424c0a5720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f4258ff9640 1 -- 172.21.15.150:0/1821371536 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c0b2040 con 0x7f425c06fab0 2024-09-17T04:49:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f4258ff9640 1 -- 172.21.15.150:0/1821371536 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f424c005030 con 0x7f425c06fab0 2024-09-17T04:49:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f42626ef640 1 -- 172.21.15.150:0/1821371536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f425c1f1c40 con 0x7f425c06fab0 2024-09-17T04:49:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f4258ff9640 1 -- 172.21.15.150:0/1821371536 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c0044f0 con 0x7f425c06fab0 2024-09-17T04:49:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.289+0000 7f42626ef640 1 -- 172.21.15.150:0/1821371536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f425c146f00 con 0x7f425c06fab0 2024-09-17T04:49:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.293+0000 7f42626ef640 1 -- 172.21.15.150:0/1821371536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f425c06fec0 con 0x7f425c06fab0 2024-09-17T04:49:37.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.293+0000 7f4258ff9640 1 -- 172.21.15.150:0/1821371536 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f424c0bd060 con 0x7f425c06fab0 2024-09-17T04:49:37.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.293+0000 7f4258ff9640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4248073230 0x7f4248075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:37.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.293+0000 7f4258ff9640 1 -- 172.21.15.150:0/1821371536 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f424c12d8a0 con 0x7f425c06fab0 2024-09-17T04:49:37.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.293+0000 7f425b7fe640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4248073230 0x7f4248075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:37.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.297+0000 7f425b7fe640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4248073230 0x7f4248075740 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f425c1550c0 tx=0x7f425000b370 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:37.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.297+0000 7f4258ff9640 1 -- 172.21.15.150:0/1821371536 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f424c016490 con 0x7f425c06fab0 2024-09-17T04:49:37.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.453+0000 7f42626ef640 1 -- 172.21.15.150:0/1821371536 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f425c15ee30 con 0x7f4248073230 2024-09-17T04:49:37.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.465+0000 7f4258ff9640 1 -- 172.21.15.150:0/1821371536 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251941 (secure 0 0 0) 0x7f425c15ee30 con 0x7f4248073230 2024-09-17T04:49:37.469 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:37.471 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T04:49:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.469+0000 7f42626ef640 1 -- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4248073230 msgr2=0x7f4248075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.469+0000 7f42626ef640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4248073230 0x7f4248075740 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f425c1550c0 tx=0x7f425000b370 comp rx=0 tx=0).stop 2024-09-17T04:49:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.469+0000 7f42626ef640 1 -- 172.21.15.150:0/1821371536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c06fab0 msgr2=0x7f425c1479e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.469+0000 7f42626ef640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c06fab0 0x7f425c1479e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f424c0b0040 tx=0x7f424c0a5720 comp rx=0 tx=0).stop 2024-09-17T04:49:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.469+0000 7f42626ef640 1 -- 172.21.15.150:0/1821371536 shutdown_connections 2024-09-17T04:49:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.469+0000 7f42626ef640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4248073230 0x7f4248075740 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.469+0000 7f42626ef640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f425c0c41c0 0x7f425c150540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.469+0000 7f42626ef640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f425c0c3db0 0x7f425c1f0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.469+0000 7f42626ef640 1 --2- 172.21.15.150:0/1821371536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f425c06fab0 0x7f425c1479e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.469+0000 7f42626ef640 1 -- 172.21.15.150:0/1821371536 >> 172.21.15.150:0/1821371536 conn(0x7f425c0c4a00 msgr2=0x7f425c149de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:37.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.469+0000 7f42626ef640 1 -- 172.21.15.150:0/1821371536 shutdown_connections 2024-09-17T04:49:37.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.469+0000 7f42626ef640 1 -- 172.21.15.150:0/1821371536 wait complete. 2024-09-17T04:49:37.488 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-17T04:49:36.942217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223376,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751344,"statfs":{"total":773094113280,"available":772865376256,"internally_reserved":0,"allocated":3022848,"data_stored":1795188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":56,"apply_latency_ms":56,"commit_latency_ns":56000000,"apply_latency_ns":56000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.093037"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406997+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T04:49:33.406997+0000","last_peered":"2024-09-17T04:49:33.406997+0000","last_clean":"2024-09-17T04:49:33.406997+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T04:49:33.406997+0000","last_undegraded":"2024-09-17T04:49:33.406997+0000","last_fullsized":"2024-09-17T04:49:33.406997+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403685+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T04:49:33.403685+0000","last_peered":"2024-09-17T04:49:33.403685+0000","last_clean":"2024-09-17T04:49:33.403685+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T04:49:33.403685+0000","last_undegraded":"2024-09-17T04:49:33.403685+0000","last_fullsized":"2024-09-17T04:49:33.403685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403283+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T04:49:33.403283+0000","last_peered":"2024-09-17T04:49:33.403283+0000","last_clean":"2024-09-17T04:49:33.403283+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T04:49:33.403283+0000","last_undegraded":"2024-09-17T04:49:33.403283+0000","last_fullsized":"2024-09-17T04:49:33.403283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.405271+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T04:49:33.405271+0000","last_peered":"2024-09-17T04:49:33.405271+0000","last_clean":"2024-09-17T04:49:33.405271+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T04:49:33.405271+0000","last_undegraded":"2024-09-17T04:49:33.405271+0000","last_fullsized":"2024-09-17T04:49:33.405271+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407231+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T04:49:33.407231+0000","last_peered":"2024-09-17T04:49:33.407231+0000","last_clean":"2024-09-17T04:49:33.407231+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T04:49:33.407231+0000","last_undegraded":"2024-09-17T04:49:33.407231+0000","last_fullsized":"2024-09-17T04:49:33.407231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404936+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T04:49:33.404936+0000","last_peered":"2024-09-17T04:49:33.404936+0000","last_clean":"2024-09-17T04:49:33.404936+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T04:49:33.404936+0000","last_undegraded":"2024-09-17T04:49:33.404936+0000","last_fullsized":"2024-09-17T04:49:33.404936+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407864+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T04:49:33.407864+0000","last_peered":"2024-09-17T04:49:33.407864+0000","last_clean":"2024-09-17T04:49:33.407864+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T04:49:33.407864+0000","last_undegraded":"2024-09-17T04:49:33.407864+0000","last_fullsized":"2024-09-17T04:49:33.407864+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403454+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T04:49:33.403454+0000","last_peered":"2024-09-17T04:49:33.403454+0000","last_clean":"2024-09-17T04:49:33.403454+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T04:49:33.403454+0000","last_undegraded":"2024-09-17T04:49:33.403454+0000","last_fullsized":"2024-09-17T04:49:33.403454+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890786+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T04:49:32.890786+0000","last_peered":"2024-09-17T04:49:32.890786+0000","last_clean":"2024-09-17T04:49:32.890786+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T04:49:32.890786+0000","last_undegraded":"2024-09-17T04:49:32.890786+0000","last_fullsized":"2024-09-17T04:49:32.890786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407254+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T04:49:33.407254+0000","last_peered":"2024-09-17T04:49:33.407254+0000","last_clean":"2024-09-17T04:49:33.407254+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T04:49:33.407254+0000","last_undegraded":"2024-09-17T04:49:33.407254+0000","last_fullsized":"2024-09-17T04:49:33.407254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.405405+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T04:49:33.405405+0000","last_peered":"2024-09-17T04:49:33.405405+0000","last_clean":"2024-09-17T04:49:33.405405+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T04:49:33.405405+0000","last_undegraded":"2024-09-17T04:49:33.405405+0000","last_fullsized":"2024-09-17T04:49:33.405405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.408154+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T04:49:33.408154+0000","last_peered":"2024-09-17T04:49:33.408154+0000","last_clean":"2024-09-17T04:49:33.408154+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T04:49:33.408154+0000","last_undegraded":"2024-09-17T04:49:33.408154+0000","last_fullsized":"2024-09-17T04:49:33.408154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892510+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T04:49:32.892510+0000","last_peered":"2024-09-17T04:49:32.892510+0000","last_clean":"2024-09-17T04:49:32.892510+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T04:49:32.892510+0000","last_undegraded":"2024-09-17T04:49:32.892510+0000","last_fullsized":"2024-09-17T04:49:32.892510+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892102+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T04:49:32.892102+0000","last_peered":"2024-09-17T04:49:32.892102+0000","last_clean":"2024-09-17T04:49:32.892102+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T04:49:32.892102+0000","last_undegraded":"2024-09-17T04:49:32.892102+0000","last_fullsized":"2024-09-17T04:49:32.892102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.402966+0000","last_change":"2024-09-17T04:49:31.911036+0000","last_active":"2024-09-17T04:49:33.402966+0000","last_peered":"2024-09-17T04:49:33.402966+0000","last_clean":"2024-09-17T04:49:33.402966+0000","last_became_active":"2024-09-17T04:49:31.910285+0000","last_became_peered":"2024-09-17T04:49:31.910285+0000","last_unstale":"2024-09-17T04:49:33.402966+0000","last_undegraded":"2024-09-17T04:49:33.402966+0000","last_fullsized":"2024-09-17T04:49:33.402966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406489+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T04:49:33.406489+0000","last_peered":"2024-09-17T04:49:33.406489+0000","last_clean":"2024-09-17T04:49:33.406489+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T04:49:33.406489+0000","last_undegraded":"2024-09-17T04:49:33.406489+0000","last_fullsized":"2024-09-17T04:49:33.406489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410347+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T04:49:33.410347+0000","last_peered":"2024-09-17T04:49:33.410347+0000","last_clean":"2024-09-17T04:49:33.410347+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T04:49:33.410347+0000","last_undegraded":"2024-09-17T04:49:33.410347+0000","last_fullsized":"2024-09-17T04:49:33.410347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.995271+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T04:49:32.995271+0000","last_peered":"2024-09-17T04:49:32.995271+0000","last_clean":"2024-09-17T04:49:32.995271+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T04:49:32.995271+0000","last_undegraded":"2024-09-17T04:49:32.995271+0000","last_fullsized":"2024-09-17T04:49:32.995271+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410470+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T04:49:33.410470+0000","last_peered":"2024-09-17T04:49:33.410470+0000","last_clean":"2024-09-17T04:49:33.410470+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T04:49:33.410470+0000","last_undegraded":"2024-09-17T04:49:33.410470+0000","last_fullsized":"2024-09-17T04:49:33.410470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.408650+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T04:49:33.408650+0000","last_peered":"2024-09-17T04:49:33.408650+0000","last_clean":"2024-09-17T04:49:33.408650+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T04:49:33.408650+0000","last_undegraded":"2024-09-17T04:49:33.408650+0000","last_fullsized":"2024-09-17T04:49:33.408650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889265+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T04:49:32.889265+0000","last_peered":"2024-09-17T04:49:32.889265+0000","last_clean":"2024-09-17T04:49:32.889265+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T04:49:32.889265+0000","last_undegraded":"2024-09-17T04:49:32.889265+0000","last_fullsized":"2024-09-17T04:49:32.889265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889697+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T04:49:32.889697+0000","last_peered":"2024-09-17T04:49:32.889697+0000","last_clean":"2024-09-17T04:49:32.889697+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T04:49:32.889697+0000","last_undegraded":"2024-09-17T04:49:32.889697+0000","last_fullsized":"2024-09-17T04:49:32.889697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403082+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T04:49:33.403082+0000","last_peered":"2024-09-17T04:49:33.403082+0000","last_clean":"2024-09-17T04:49:33.403082+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T04:49:33.403082+0000","last_undegraded":"2024-09-17T04:49:33.403082+0000","last_fullsized":"2024-09-17T04:49:33.403082+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404523+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T04:49:33.404523+0000","last_peered":"2024-09-17T04:49:33.404523+0000","last_clean":"2024-09-17T04:49:33.404523+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T04:49:33.404523+0000","last_undegraded":"2024-09-17T04:49:33.404523+0000","last_fullsized":"2024-09-17T04:49:33.404523+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403996+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T04:49:33.403996+0000","last_peered":"2024-09-17T04:49:33.403996+0000","last_clean":"2024-09-17T04:49:33.403996+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T04:49:33.403996+0000","last_undegraded":"2024-09-17T04:49:33.403996+0000","last_fullsized":"2024-09-17T04:49:33.403996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406592+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T04:49:33.406592+0000","last_peered":"2024-09-17T04:49:33.406592+0000","last_clean":"2024-09-17T04:49:33.406592+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T04:49:33.406592+0000","last_undegraded":"2024-09-17T04:49:33.406592+0000","last_fullsized":"2024-09-17T04:49:33.406592+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892882+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T04:49:32.892882+0000","last_peered":"2024-09-17T04:49:32.892882+0000","last_clean":"2024-09-17T04:49:32.892882+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T04:49:32.892882+0000","last_undegraded":"2024-09-17T04:49:32.892882+0000","last_fullsized":"2024-09-17T04:49:32.892882+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892925+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T04:49:32.892925+0000","last_peered":"2024-09-17T04:49:32.892925+0000","last_clean":"2024-09-17T04:49:32.892925+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T04:49:32.892925+0000","last_undegraded":"2024-09-17T04:49:32.892925+0000","last_fullsized":"2024-09-17T04:49:32.892925+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403575+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T04:49:33.403575+0000","last_peered":"2024-09-17T04:49:33.403575+0000","last_clean":"2024-09-17T04:49:33.403575+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T04:49:33.403575+0000","last_undegraded":"2024-09-17T04:49:33.403575+0000","last_fullsized":"2024-09-17T04:49:33.403575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409144+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T04:49:33.409144+0000","last_peered":"2024-09-17T04:49:33.409144+0000","last_clean":"2024-09-17T04:49:33.409144+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T04:49:33.409144+0000","last_undegraded":"2024-09-17T04:49:33.409144+0000","last_fullsized":"2024-09-17T04:49:33.409144+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.994822+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T04:49:32.994822+0000","last_peered":"2024-09-17T04:49:32.994822+0000","last_clean":"2024-09-17T04:49:32.994822+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T04:49:32.994822+0000","last_undegraded":"2024-09-17T04:49:32.994822+0000","last_fullsized":"2024-09-17T04:49:32.994822+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404526+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T04:49:33.404526+0000","last_peered":"2024-09-17T04:49:33.404526+0000","last_clean":"2024-09-17T04:49:33.404526+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T04:49:33.404526+0000","last_undegraded":"2024-09-17T04:49:33.404526+0000","last_fullsized":"2024-09-17T04:49:33.404526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890207+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T04:49:32.890207+0000","last_peered":"2024-09-17T04:49:32.890207+0000","last_clean":"2024-09-17T04:49:32.890207+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T04:49:32.890207+0000","last_undegraded":"2024-09-17T04:49:32.890207+0000","last_fullsized":"2024-09-17T04:49:32.890207+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403470+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T04:49:33.403470+0000","last_peered":"2024-09-17T04:49:33.403470+0000","last_clean":"2024-09-17T04:49:33.403470+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T04:49:33.403470+0000","last_undegraded":"2024-09-17T04:49:33.403470+0000","last_fullsized":"2024-09-17T04:49:33.403470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890205+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T04:49:32.890205+0000","last_peered":"2024-09-17T04:49:32.890205+0000","last_clean":"2024-09-17T04:49:32.890205+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T04:49:32.890205+0000","last_undegraded":"2024-09-17T04:49:32.890205+0000","last_fullsized":"2024-09-17T04:49:32.890205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409641+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T04:49:33.409641+0000","last_peered":"2024-09-17T04:49:33.409641+0000","last_clean":"2024-09-17T04:49:33.409641+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T04:49:33.409641+0000","last_undegraded":"2024-09-17T04:49:33.409641+0000","last_fullsized":"2024-09-17T04:49:33.409641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.891281+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T04:49:32.891281+0000","last_peered":"2024-09-17T04:49:32.891281+0000","last_clean":"2024-09-17T04:49:32.891281+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T04:49:32.891281+0000","last_undegraded":"2024-09-17T04:49:32.891281+0000","last_fullsized":"2024-09-17T04:49:32.891281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404563+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T04:49:33.404563+0000","last_peered":"2024-09-17T04:49:33.404563+0000","last_clean":"2024-09-17T04:49:33.404563+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T04:49:33.404563+0000","last_undegraded":"2024-09-17T04:49:33.404563+0000","last_fullsized":"2024-09-17T04:49:33.404563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406532+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T04:49:33.406532+0000","last_peered":"2024-09-17T04:49:33.406532+0000","last_clean":"2024-09-17T04:49:33.406532+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T04:49:33.406532+0000","last_undegraded":"2024-09-17T04:49:33.406532+0000","last_fullsized":"2024-09-17T04:49:33.406532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889317+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T04:49:32.889317+0000","last_peered":"2024-09-17T04:49:32.889317+0000","last_clean":"2024-09-17T04:49:32.889317+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T04:49:32.889317+0000","last_undegraded":"2024-09-17T04:49:32.889317+0000","last_fullsized":"2024-09-17T04:49:32.889317+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890106+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T04:49:32.890106+0000","last_peered":"2024-09-17T04:49:32.890106+0000","last_clean":"2024-09-17T04:49:32.890106+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T04:49:32.890106+0000","last_undegraded":"2024-09-17T04:49:32.890106+0000","last_fullsized":"2024-09-17T04:49:32.890106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407222+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T04:49:33.407222+0000","last_peered":"2024-09-17T04:49:33.407222+0000","last_clean":"2024-09-17T04:49:33.407222+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T04:49:33.407222+0000","last_undegraded":"2024-09-17T04:49:33.407222+0000","last_fullsized":"2024-09-17T04:49:33.407222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407474+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T04:49:33.407474+0000","last_peered":"2024-09-17T04:49:33.407474+0000","last_clean":"2024-09-17T04:49:33.407474+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T04:49:33.407474+0000","last_undegraded":"2024-09-17T04:49:33.407474+0000","last_fullsized":"2024-09-17T04:49:33.407474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404483+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T04:49:33.404483+0000","last_peered":"2024-09-17T04:49:33.404483+0000","last_clean":"2024-09-17T04:49:33.404483+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T04:49:33.404483+0000","last_undegraded":"2024-09-17T04:49:33.404483+0000","last_fullsized":"2024-09-17T04:49:33.404483+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.994691+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T04:49:32.994691+0000","last_peered":"2024-09-17T04:49:32.994691+0000","last_clean":"2024-09-17T04:49:32.994691+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T04:49:32.994691+0000","last_undegraded":"2024-09-17T04:49:32.994691+0000","last_fullsized":"2024-09-17T04:49:32.994691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409414+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T04:49:33.409414+0000","last_peered":"2024-09-17T04:49:33.409414+0000","last_clean":"2024-09-17T04:49:33.409414+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T04:49:33.409414+0000","last_undegraded":"2024-09-17T04:49:33.409414+0000","last_fullsized":"2024-09-17T04:49:33.409414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.405756+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T04:49:33.405756+0000","last_peered":"2024-09-17T04:49:33.405756+0000","last_clean":"2024-09-17T04:49:33.405756+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T04:49:33.405756+0000","last_undegraded":"2024-09-17T04:49:33.405756+0000","last_fullsized":"2024-09-17T04:49:33.405756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.405787+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T04:49:33.405787+0000","last_peered":"2024-09-17T04:49:33.405787+0000","last_clean":"2024-09-17T04:49:33.405787+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T04:49:33.405787+0000","last_undegraded":"2024-09-17T04:49:33.405787+0000","last_fullsized":"2024-09-17T04:49:33.405787+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.995219+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T04:49:32.995219+0000","last_peered":"2024-09-17T04:49:32.995219+0000","last_clean":"2024-09-17T04:49:32.995219+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T04:49:32.995219+0000","last_undegraded":"2024-09-17T04:49:32.995219+0000","last_fullsized":"2024-09-17T04:49:32.995219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892585+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T04:49:32.892585+0000","last_peered":"2024-09-17T04:49:32.892585+0000","last_clean":"2024-09-17T04:49:32.892585+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T04:49:32.892585+0000","last_undegraded":"2024-09-17T04:49:32.892585+0000","last_fullsized":"2024-09-17T04:49:32.892585+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410773+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T04:49:33.410773+0000","last_peered":"2024-09-17T04:49:33.410773+0000","last_clean":"2024-09-17T04:49:33.410773+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T04:49:33.410773+0000","last_undegraded":"2024-09-17T04:49:33.410773+0000","last_fullsized":"2024-09-17T04:49:33.410773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409979+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T04:49:33.409979+0000","last_peered":"2024-09-17T04:49:33.409979+0000","last_clean":"2024-09-17T04:49:33.409979+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T04:49:33.409979+0000","last_undegraded":"2024-09-17T04:49:33.409979+0000","last_fullsized":"2024-09-17T04:49:33.409979+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406471+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T04:49:33.406471+0000","last_peered":"2024-09-17T04:49:33.406471+0000","last_clean":"2024-09-17T04:49:33.406471+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T04:49:33.406471+0000","last_undegraded":"2024-09-17T04:49:33.406471+0000","last_fullsized":"2024-09-17T04:49:33.406471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889195+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T04:49:32.889195+0000","last_peered":"2024-09-17T04:49:32.889195+0000","last_clean":"2024-09-17T04:49:32.889195+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T04:49:32.889195+0000","last_undegraded":"2024-09-17T04:49:32.889195+0000","last_fullsized":"2024-09-17T04:49:32.889195+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889539+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T04:49:32.889539+0000","last_peered":"2024-09-17T04:49:32.889539+0000","last_clean":"2024-09-17T04:49:32.889539+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T04:49:32.889539+0000","last_undegraded":"2024-09-17T04:49:32.889539+0000","last_fullsized":"2024-09-17T04:49:32.889539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407977+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T04:49:33.407977+0000","last_peered":"2024-09-17T04:49:33.407977+0000","last_clean":"2024-09-17T04:49:33.407977+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T04:49:33.407977+0000","last_undegraded":"2024-09-17T04:49:33.407977+0000","last_fullsized":"2024-09-17T04:49:33.407977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407515+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T04:49:33.407515+0000","last_peered":"2024-09-17T04:49:33.407515+0000","last_clean":"2024-09-17T04:49:33.407515+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T04:49:33.407515+0000","last_undegraded":"2024-09-17T04:49:33.407515+0000","last_fullsized":"2024-09-17T04:49:33.407515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407798+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T04:49:33.407798+0000","last_peered":"2024-09-17T04:49:33.407798+0000","last_clean":"2024-09-17T04:49:33.407798+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T04:49:33.407798+0000","last_undegraded":"2024-09-17T04:49:33.407798+0000","last_fullsized":"2024-09-17T04:49:33.407798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890372+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T04:49:32.890372+0000","last_peered":"2024-09-17T04:49:32.890372+0000","last_clean":"2024-09-17T04:49:32.890372+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T04:49:32.890372+0000","last_undegraded":"2024-09-17T04:49:32.890372+0000","last_fullsized":"2024-09-17T04:49:32.890372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407145+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T04:49:33.407145+0000","last_peered":"2024-09-17T04:49:33.407145+0000","last_clean":"2024-09-17T04:49:33.407145+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T04:49:33.407145+0000","last_undegraded":"2024-09-17T04:49:33.407145+0000","last_fullsized":"2024-09-17T04:49:33.407145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403310+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T04:49:33.403310+0000","last_peered":"2024-09-17T04:49:33.403310+0000","last_clean":"2024-09-17T04:49:33.403310+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T04:49:33.403310+0000","last_undegraded":"2024-09-17T04:49:33.403310+0000","last_fullsized":"2024-09-17T04:49:33.403310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.408579+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T04:49:33.408579+0000","last_peered":"2024-09-17T04:49:33.408579+0000","last_clean":"2024-09-17T04:49:33.408579+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T04:49:33.408579+0000","last_undegraded":"2024-09-17T04:49:33.408579+0000","last_fullsized":"2024-09-17T04:49:33.408579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892968+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T04:49:32.892968+0000","last_peered":"2024-09-17T04:49:32.892968+0000","last_clean":"2024-09-17T04:49:32.892968+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T04:49:32.892968+0000","last_undegraded":"2024-09-17T04:49:32.892968+0000","last_fullsized":"2024-09-17T04:49:32.892968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410052+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T04:49:33.410052+0000","last_peered":"2024-09-17T04:49:33.410052+0000","last_clean":"2024-09-17T04:49:33.410052+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T04:49:33.410052+0000","last_undegraded":"2024-09-17T04:49:33.410052+0000","last_fullsized":"2024-09-17T04:49:33.410052+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":111,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889214+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T04:49:32.889214+0000","last_peered":"2024-09-17T04:49:32.889214+0000","last_clean":"2024-09-17T04:49:32.889214+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T04:49:32.889214+0000","last_undegraded":"2024-09-17T04:49:32.889214+0000","last_fullsized":"2024-09-17T04:49:32.889214+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409562+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T04:49:33.409562+0000","last_peered":"2024-09-17T04:49:33.409562+0000","last_clean":"2024-09-17T04:49:33.409562+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T04:49:33.409562+0000","last_undegraded":"2024-09-17T04:49:33.409562+0000","last_fullsized":"2024-09-17T04:49:33.409562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889412+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T04:49:32.889412+0000","last_peered":"2024-09-17T04:49:32.889412+0000","last_clean":"2024-09-17T04:49:32.889412+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T04:49:32.889412+0000","last_undegraded":"2024-09-17T04:49:32.889412+0000","last_fullsized":"2024-09-17T04:49:32.889412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406888+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T04:49:33.406888+0000","last_peered":"2024-09-17T04:49:33.406888+0000","last_clean":"2024-09-17T04:49:33.406888+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T04:49:33.406888+0000","last_undegraded":"2024-09-17T04:49:33.406888+0000","last_fullsized":"2024-09-17T04:49:33.406888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.891390+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T04:49:32.891390+0000","last_peered":"2024-09-17T04:49:32.891390+0000","last_clean":"2024-09-17T04:49:32.891390+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T04:49:32.891390+0000","last_undegraded":"2024-09-17T04:49:32.891390+0000","last_fullsized":"2024-09-17T04:49:32.891390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.994261+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T04:49:32.994261+0000","last_peered":"2024-09-17T04:49:32.994261+0000","last_clean":"2024-09-17T04:49:32.994261+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T04:49:32.994261+0000","last_undegraded":"2024-09-17T04:49:32.994261+0000","last_fullsized":"2024-09-17T04:49:32.994261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.403329+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T04:49:33.403329+0000","last_peered":"2024-09-17T04:49:33.403329+0000","last_clean":"2024-09-17T04:49:33.403329+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T04:49:33.403329+0000","last_undegraded":"2024-09-17T04:49:33.403329+0000","last_fullsized":"2024-09-17T04:49:33.403329+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410800+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T04:49:33.410800+0000","last_peered":"2024-09-17T04:49:33.410800+0000","last_clean":"2024-09-17T04:49:33.410800+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T04:49:33.410800+0000","last_undegraded":"2024-09-17T04:49:33.410800+0000","last_fullsized":"2024-09-17T04:49:33.410800+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.407387+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T04:49:33.407387+0000","last_peered":"2024-09-17T04:49:33.407387+0000","last_clean":"2024-09-17T04:49:33.407387+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T04:49:33.407387+0000","last_undegraded":"2024-09-17T04:49:33.407387+0000","last_fullsized":"2024-09-17T04:49:33.407387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.402938+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T04:49:33.402938+0000","last_peered":"2024-09-17T04:49:33.402938+0000","last_clean":"2024-09-17T04:49:33.402938+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T04:49:33.402938+0000","last_undegraded":"2024-09-17T04:49:33.402938+0000","last_fullsized":"2024-09-17T04:49:33.402938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.405916+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T04:49:33.405916+0000","last_peered":"2024-09-17T04:49:33.405916+0000","last_clean":"2024-09-17T04:49:33.405916+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T04:49:33.405916+0000","last_undegraded":"2024-09-17T04:49:33.405916+0000","last_fullsized":"2024-09-17T04:49:33.405916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410517+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T04:49:33.410517+0000","last_peered":"2024-09-17T04:49:33.410517+0000","last_clean":"2024-09-17T04:49:33.410517+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T04:49:33.410517+0000","last_undegraded":"2024-09-17T04:49:33.410517+0000","last_fullsized":"2024-09-17T04:49:33.410517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892861+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T04:49:32.892861+0000","last_peered":"2024-09-17T04:49:32.892861+0000","last_clean":"2024-09-17T04:49:32.892861+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T04:49:32.892861+0000","last_undegraded":"2024-09-17T04:49:32.892861+0000","last_fullsized":"2024-09-17T04:49:32.892861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890648+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T04:49:32.890648+0000","last_peered":"2024-09-17T04:49:32.890648+0000","last_clean":"2024-09-17T04:49:32.890648+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T04:49:32.890648+0000","last_undegraded":"2024-09-17T04:49:32.890648+0000","last_fullsized":"2024-09-17T04:49:32.890648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.891083+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T04:49:32.891083+0000","last_peered":"2024-09-17T04:49:32.891083+0000","last_clean":"2024-09-17T04:49:32.891083+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T04:49:32.891083+0000","last_undegraded":"2024-09-17T04:49:32.891083+0000","last_fullsized":"2024-09-17T04:49:32.891083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.410223+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T04:49:33.410223+0000","last_peered":"2024-09-17T04:49:33.410223+0000","last_clean":"2024-09-17T04:49:33.410223+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T04:49:33.410223+0000","last_undegraded":"2024-09-17T04:49:33.410223+0000","last_fullsized":"2024-09-17T04:49:33.410223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890056+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T04:49:32.890056+0000","last_peered":"2024-09-17T04:49:32.890056+0000","last_clean":"2024-09-17T04:49:32.890056+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T04:49:32.890056+0000","last_undegraded":"2024-09-17T04:49:32.890056+0000","last_fullsized":"2024-09-17T04:49:32.890056+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.405369+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T04:49:33.405369+0000","last_peered":"2024-09-17T04:49:33.405369+0000","last_clean":"2024-09-17T04:49:33.405369+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T04:49:33.405369+0000","last_undegraded":"2024-09-17T04:49:33.405369+0000","last_fullsized":"2024-09-17T04:49:33.405369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.408507+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T04:49:33.408507+0000","last_peered":"2024-09-17T04:49:33.408507+0000","last_clean":"2024-09-17T04:49:33.408507+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T04:49:33.408507+0000","last_undegraded":"2024-09-17T04:49:33.408507+0000","last_fullsized":"2024-09-17T04:49:33.408507+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889173+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T04:49:32.889173+0000","last_peered":"2024-09-17T04:49:32.889173+0000","last_clean":"2024-09-17T04:49:32.889173+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T04:49:32.889173+0000","last_undegraded":"2024-09-17T04:49:32.889173+0000","last_fullsized":"2024-09-17T04:49:32.889173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.889026+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T04:49:32.889026+0000","last_peered":"2024-09-17T04:49:32.889026+0000","last_clean":"2024-09-17T04:49:32.889026+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T04:49:32.889026+0000","last_undegraded":"2024-09-17T04:49:32.889026+0000","last_fullsized":"2024-09-17T04:49:32.889026+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.892293+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T04:49:32.892293+0000","last_peered":"2024-09-17T04:49:32.892293+0000","last_clean":"2024-09-17T04:49:32.892293+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T04:49:32.892293+0000","last_undegraded":"2024-09-17T04:49:32.892293+0000","last_fullsized":"2024-09-17T04:49:32.892293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.995392+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T04:49:32.995392+0000","last_peered":"2024-09-17T04:49:32.995392+0000","last_clean":"2024-09-17T04:49:32.995392+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T04:49:32.995392+0000","last_undegraded":"2024-09-17T04:49:32.995392+0000","last_fullsized":"2024-09-17T04:49:32.995392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409194+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T04:49:33.409194+0000","last_peered":"2024-09-17T04:49:33.409194+0000","last_clean":"2024-09-17T04:49:33.409194+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T04:49:33.409194+0000","last_undegraded":"2024-09-17T04:49:33.409194+0000","last_fullsized":"2024-09-17T04:49:33.409194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409215+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T04:49:33.409215+0000","last_peered":"2024-09-17T04:49:33.409215+0000","last_clean":"2024-09-17T04:49:33.409215+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T04:49:33.409215+0000","last_undegraded":"2024-09-17T04:49:33.409215+0000","last_fullsized":"2024-09-17T04:49:33.409215+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.409539+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T04:49:33.409539+0000","last_peered":"2024-09-17T04:49:33.409539+0000","last_clean":"2024-09-17T04:49:33.409539+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T04:49:33.409539+0000","last_undegraded":"2024-09-17T04:49:33.409539+0000","last_fullsized":"2024-09-17T04:49:33.409539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.408917+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T04:49:33.408917+0000","last_peered":"2024-09-17T04:49:33.408917+0000","last_clean":"2024-09-17T04:49:33.408917+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T04:49:33.408917+0000","last_undegraded":"2024-09-17T04:49:33.408917+0000","last_fullsized":"2024-09-17T04:49:33.408917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.890034+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T04:49:32.890034+0000","last_peered":"2024-09-17T04:49:32.890034+0000","last_clean":"2024-09-17T04:49:32.890034+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T04:49:32.890034+0000","last_undegraded":"2024-09-17T04:49:32.890034+0000","last_fullsized":"2024-09-17T04:49:32.890034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.406048+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T04:49:33.406048+0000","last_peered":"2024-09-17T04:49:33.406048+0000","last_clean":"2024-09-17T04:49:33.406048+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T04:49:33.406048+0000","last_undegraded":"2024-09-17T04:49:33.406048+0000","last_fullsized":"2024-09-17T04:49:33.406048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404083+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T04:49:33.404083+0000","last_peered":"2024-09-17T04:49:33.404083+0000","last_clean":"2024-09-17T04:49:33.404083+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T04:49:33.404083+0000","last_undegraded":"2024-09-17T04:49:33.404083+0000","last_fullsized":"2024-09-17T04:49:33.404083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404168+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T04:49:33.404168+0000","last_peered":"2024-09-17T04:49:33.404168+0000","last_clean":"2024-09-17T04:49:33.404168+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T04:49:33.404168+0000","last_undegraded":"2024-09-17T04:49:33.404168+0000","last_fullsized":"2024-09-17T04:49:33.404168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:33.404076+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T04:49:33.404076+0000","last_peered":"2024-09-17T04:49:33.404076+0000","last_clean":"2024-09-17T04:49:33.404076+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T04:49:33.404076+0000","last_undegraded":"2024-09-17T04:49:33.404076+0000","last_fullsized":"2024-09-17T04:49:33.404076+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T04:49:32.891572+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T04:49:32.891572+0000","last_peered":"2024-09-17T04:49:32.891572+0000","last_clean":"2024-09-17T04:49:32.891572+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T04:49:32.891572+0000","last_undegraded":"2024-09-17T04:49:32.891572+0000","last_fullsized":"2024-09-17T04:49:32.891572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":76328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":823296,"data_stored":666664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":241664,"data_stored":78488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":77816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":819200,"data_stored":666642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":229376,"data_stored":76260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":76204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:49:37.492 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:49:37.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T04:49:37.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.681+0000 7f1588e2d640 1 -- 172.21.15.150:0/4205235801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1570003a40 msgr2=0x7f1570003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:37.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.681+0000 7f1588e2d640 1 --2- 172.21.15.150:0/4205235801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1570003a40 0x7f1570003eb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f15841513e0 tx=0x7f157c01cda0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.681+0000 7f1582d9d640 1 -- 172.21.15.150:0/4205235801 reap_dead start 2024-09-17T04:49:37.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.681+0000 7f1588e2d640 1 -- 172.21.15.150:0/4205235801 shutdown_connections 2024-09-17T04:49:37.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.681+0000 7f1588e2d640 1 -- 172.21.15.150:0/4205235801 >> 172.21.15.150:0/4205235801 conn(0x7f15840c4a00 msgr2=0x7f15840c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:37.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.681+0000 7f1588e2d640 1 -- 172.21.15.150:0/4205235801 shutdown_connections 2024-09-17T04:49:37.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.681+0000 7f1588e2d640 1 -- 172.21.15.150:0/4205235801 wait complete. 2024-09-17T04:49:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.681+0000 7f1588e2d640 1 Processor -- start 2024-09-17T04:49:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.681+0000 7f1588e2d640 1 -- start start 2024-09-17T04:49:37.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1588e2d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1570003a40 0x7f15841dfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:37.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1588e2d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f158414f070 0x7f158414fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:37.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1588e2d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15840ac6b0 0x7f158414c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:37.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1588e2d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f158406f7d0 con 0x7f158414f070 2024-09-17T04:49:37.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1588e2d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f15840adef0 con 0x7f15840ac6b0 2024-09-17T04:49:37.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1588e2d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1584074f70 con 0x7f1570003a40 2024-09-17T04:49:37.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1581d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f158414f070 0x7f158414fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:37.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1581d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f158414f070 0x7f158414fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43106/0 (socket says 172.21.15.150:43106) 2024-09-17T04:49:37.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1581d9b640 1 -- 172.21.15.150:0/640534980 learned_addr learned my addr 172.21.15.150:0/640534980 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:37.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f158259c640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1570003a40 0x7f15841dfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:37.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1581d9b640 1 -- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1570003a40 msgr2=0x7f15841dfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:37.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1582d9d640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15840ac6b0 0x7f158414c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1581d9b640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1570003a40 0x7f15841dfdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1581d9b640 1 -- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15840ac6b0 msgr2=0x7f158414c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1581d9b640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15840ac6b0 0x7f158414c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1581d9b640 1 -- 172.21.15.150:0/640534980 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158414d150 con 0x7f158414f070 2024-09-17T04:49:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1582d9d640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15840ac6b0 0x7f158414c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1581d9b640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f158414f070 0x7f158414fcb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f156c00d5b0 tx=0x7f156c00da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f157affd640 1 -- 172.21.15.150:0/640534980 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c016020 con 0x7f158414f070 2024-09-17T04:49:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f157affd640 1 -- 172.21.15.150:0/640534980 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f156c012630 con 0x7f158414f070 2024-09-17T04:49:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1588e2d640 1 -- 172.21.15.150:0/640534980 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f158414cee0 con 0x7f158414f070 2024-09-17T04:49:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f157affd640 1 -- 172.21.15.150:0/640534980 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c012a80 con 0x7f158414f070 2024-09-17T04:49:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1588e2d640 1 -- 172.21.15.150:0/640534980 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15840a9050 con 0x7f158414f070 2024-09-17T04:49:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f1588e2d640 1 -- 172.21.15.150:0/640534980 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15840ad7d0 con 0x7f158414f070 2024-09-17T04:49:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f157affd640 1 -- 172.21.15.150:0/640534980 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f156c019020 con 0x7f158414f070 2024-09-17T04:49:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f157affd640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1564073200 0x7f1564075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:37.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f157affd640 1 -- 172.21.15.150:0/640534980 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f156c094020 con 0x7f158414f070 2024-09-17T04:49:37.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.685+0000 7f158259c640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1564073200 0x7f1564075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:37.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.689+0000 7f158259c640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1564073200 0x7f1564075710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f157c0b0000 tx=0x7f157c0a7040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:37.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.689+0000 7f157affd640 1 -- 172.21.15.150:0/640534980 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f156c020020 con 0x7f158414f070 2024-09-17T04:49:37.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 -- 172.21.15.150:0/640534980 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f15840c60f0 con 0x7f158414f070 2024-09-17T04:49:37.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f157affd640 1 -- 172.21.15.150:0/640534980 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v22) ==== 84+0+197 (secure 0 0 0) 0x7f156c05f170 con 0x7f158414f070 2024-09-17T04:49:37.849 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:37.849 INFO:teuthology.orchestra.run.smithi150.stdout:[ 2024-09-17T04:49:37.849 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-17T04:49:37.849 INFO:teuthology.orchestra.run.smithi150.stdout: "poolnum": 1, 2024-09-17T04:49:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: "poolname": ".mgr" 2024-09-17T04:49:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-17T04:49:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-17T04:49:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: "poolnum": 2, 2024-09-17T04:49:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: "poolname": "a_metadata" 2024-09-17T04:49:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-17T04:49:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-09-17T04:49:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: "poolnum": 3, 2024-09-17T04:49:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: "poolname": "a_data" 2024-09-17T04:49:37.850 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-17T04:49:37.850 INFO:teuthology.orchestra.run.smithi150.stdout:] 2024-09-17T04:49:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 -- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1564073200 msgr2=0x7f1564075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1564073200 0x7f1564075710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f157c0b0000 tx=0x7f157c0a7040 comp rx=0 tx=0).stop 2024-09-17T04:49:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 -- 172.21.15.150:0/640534980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f158414f070 msgr2=0x7f158414fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f158414f070 0x7f158414fcb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f156c00d5b0 tx=0x7f156c00da70 comp rx=0 tx=0).stop 2024-09-17T04:49:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 -- 172.21.15.150:0/640534980 shutdown_connections 2024-09-17T04:49:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1564073200 0x7f1564075710 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15840ac6b0 0x7f158414c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f158414f070 0x7f158414fcb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 --2- 172.21.15.150:0/640534980 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1570003a40 0x7f15841dfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 -- 172.21.15.150:0/640534980 >> 172.21.15.150:0/640534980 conn(0x7f15840c4a00 msgr2=0x7f1584152120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 -- 172.21.15.150:0/640534980 shutdown_connections 2024-09-17T04:49:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:37.849+0000 7f1588e2d640 1 -- 172.21.15.150:0/640534980 wait complete. 2024-09-17T04:49:37.869 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'b' 2024-09-17T04:49:37.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_metadata --pg_num_min 64 2024-09-17T04:49:38.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 -- 172.21.15.150:0/1551526957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d94003a40 msgr2=0x7f7d94003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:38.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 --2- 172.21.15.150:0/1551526957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d94003a40 0x7f7d94003eb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7da4009b30 tx=0x7f7da401d270 comp rx=0 tx=0).stop 2024-09-17T04:49:38.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7dadca7640 1 -- 172.21.15.150:0/1551526957 reap_dead start 2024-09-17T04:49:38.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 -- 172.21.15.150:0/1551526957 shutdown_connections 2024-09-17T04:49:38.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 -- 172.21.15.150:0/1551526957 >> 172.21.15.150:0/1551526957 conn(0x7f7da80c7fc0 msgr2=0x7f7da80c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:38.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 -- 172.21.15.150:0/1551526957 shutdown_connections 2024-09-17T04:49:38.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 -- 172.21.15.150:0/1551526957 wait complete. 2024-09-17T04:49:38.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 Processor -- start 2024-09-17T04:49:38.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 -- start start 2024-09-17T04:49:38.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d94003a40 0x7f7da815c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:38.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da815f6a0 0x7f7da814e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:38.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da81f1880 0x7f7da81f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:38.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7da80c2bd0 con 0x7f7d94003a40 2024-09-17T04:49:38.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7da80a7830 con 0x7f7da81f1880 2024-09-17T04:49:38.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7da80c7900 con 0x7f7da815f6a0 2024-09-17T04:49:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7dad4a6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d94003a40 0x7f7da815c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7dad4a6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d94003a40 0x7f7da815c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43122/0 (socket says 172.21.15.150:43122) 2024-09-17T04:49:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7dad4a6640 1 -- 172.21.15.150:0/3564588587 learned_addr learned my addr 172.21.15.150:0/3564588587 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7dacca5640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da815f6a0 0x7f7da814e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7dadca7640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da81f1880 0x7f7da81f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7dad4a6640 1 -- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da815f6a0 msgr2=0x7f7da814e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7dad4a6640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da815f6a0 0x7f7da814e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7dad4a6640 1 -- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da81f1880 msgr2=0x7f7da81f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7dad4a6640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da81f1880 0x7f7da81f1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7dad4a6640 1 -- 172.21.15.150:0/3564588587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da81542c0 con 0x7f7d94003a40 2024-09-17T04:49:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7dad4a6640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d94003a40 0x7f7da815c070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7da401cce0 tx=0x7f7da40ac9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7d99ffb640 1 -- 172.21.15.150:0/3564588587 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da40ba030 con 0x7f7d94003a40 2024-09-17T04:49:38.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7d99ffb640 1 -- 172.21.15.150:0/3564588587 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7da40acbd0 con 0x7f7d94003a40 2024-09-17T04:49:38.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7d99ffb640 1 -- 172.21.15.150:0/3564588587 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da40b1040 con 0x7f7d94003a40 2024-09-17T04:49:38.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.029+0000 7f7daf70a640 1 -- 172.21.15.150:0/3564588587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da8160030 con 0x7f7d94003a40 2024-09-17T04:49:38.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.033+0000 7f7daf70a640 1 -- 172.21.15.150:0/3564588587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da81f24f0 con 0x7f7d94003a40 2024-09-17T04:49:38.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.033+0000 7f7d8b7fe640 1 -- 172.21.15.150:0/3564588587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d840052a0 con 0x7f7d94003a40 2024-09-17T04:49:38.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.033+0000 7f7d99ffb640 1 -- 172.21.15.150:0/3564588587 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7da40bd070 con 0x7f7d94003a40 2024-09-17T04:49:38.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.033+0000 7f7d99ffb640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d8c073200 0x7f7d8c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:38.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.033+0000 7f7d99ffb640 1 -- 172.21.15.150:0/3564588587 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f7da412f0e0 con 0x7f7d94003a40 2024-09-17T04:49:38.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.033+0000 7f7dacca5640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d8c073200 0x7f7d8c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:38.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.033+0000 7f7dacca5640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d8c073200 0x7f7d8c075710 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7da8156410 tx=0x7f7d9c006a50 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:38.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.037+0000 7f7d99ffb640 1 -- 172.21.15.150:0/3564588587 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7da40c5020 con 0x7f7d94003a40 2024-09-17T04:49:38.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.209+0000 7f7d8b7fe640 1 -- 172.21.15.150:0/3564588587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64} v 0) -- 0x7f7d84005860 con 0x7f7d94003a40 2024-09-17T04:49:38.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.957+0000 7f7d99ffb640 1 -- 172.21.15.150:0/3564588587 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64}]=0 pool 'b_metadata' created v23) ==== 128+0+0 (secure 0 0 0) 0x7f7da4016f80 con 0x7f7d94003a40 2024-09-17T04:49:38.958 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'b_metadata' created 2024-09-17T04:49:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.961+0000 7f7d8b7fe640 1 -- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d8c073200 msgr2=0x7f7d8c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.961+0000 7f7d8b7fe640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d8c073200 0x7f7d8c075710 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7da8156410 tx=0x7f7d9c006a50 comp rx=0 tx=0).stop 2024-09-17T04:49:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.961+0000 7f7d8b7fe640 1 -- 172.21.15.150:0/3564588587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d94003a40 msgr2=0x7f7da815c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.961+0000 7f7d8b7fe640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d94003a40 0x7f7da815c070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7da401cce0 tx=0x7f7da40ac9e0 comp rx=0 tx=0).stop 2024-09-17T04:49:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.961+0000 7f7d8b7fe640 1 -- 172.21.15.150:0/3564588587 shutdown_connections 2024-09-17T04:49:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.961+0000 7f7d8b7fe640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d8c073200 0x7f7d8c075710 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.961+0000 7f7d8b7fe640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da81f1880 0x7f7da81f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:38.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.961+0000 7f7d8b7fe640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da815f6a0 0x7f7da814e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:38.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.961+0000 7f7d8b7fe640 1 --2- 172.21.15.150:0/3564588587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d94003a40 0x7f7da815c070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:38.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.961+0000 7f7d8b7fe640 1 -- 172.21.15.150:0/3564588587 >> 172.21.15.150:0/3564588587 conn(0x7f7da80c7fc0 msgr2=0x7f7da81572f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:38.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.961+0000 7f7d8b7fe640 1 -- 172.21.15.150:0/3564588587 shutdown_connections 2024-09-17T04:49:38.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:38.961+0000 7f7d8b7fe640 1 -- 172.21.15.150:0/3564588587 wait complete. 2024-09-17T04:49:38.990 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T04:49:39.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.153+0000 7f3c7e548640 1 -- 172.21.15.150:0/2880132877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c780a85b0 msgr2=0x7f3c780c7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:39.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.153+0000 7f3c7e548640 1 --2- 172.21.15.150:0/2880132877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c780a85b0 0x7f3c780c7270 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c009840 tx=0x7f3c6c01c790 comp rx=0 tx=0).stop 2024-09-17T04:49:39.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.153+0000 7f3c7e548640 1 -- 172.21.15.150:0/2880132877 shutdown_connections 2024-09-17T04:49:39.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.153+0000 7f3c7e548640 1 --2- 172.21.15.150:0/2880132877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c780a85b0 0x7f3c780c7270 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:39.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.153+0000 7f3c7e548640 1 -- 172.21.15.150:0/2880132877 >> 172.21.15.150:0/2880132877 conn(0x7f3c780aab60 msgr2=0x7f3c780aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:39.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.153+0000 7f3c7e548640 1 -- 172.21.15.150:0/2880132877 shutdown_connections 2024-09-17T04:49:39.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.153+0000 7f3c7e548640 1 -- 172.21.15.150:0/2880132877 wait complete. 2024-09-17T04:49:39.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.153+0000 7f3c7e548640 1 Processor -- start 2024-09-17T04:49:39.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.153+0000 7f3c7e548640 1 -- start start 2024-09-17T04:49:39.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c7e548640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c780a85b0 0x7f3c78070de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:39.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c7e548640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c780c6cb0 0x7f3c78148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:39.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c7e548640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c780c77b0 0x7f3c7814d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:39.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c7e548640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78162ab0 con 0x7f3c780a85b0 2024-09-17T04:49:39.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c7e548640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3c780a8120 con 0x7f3c780c6cb0 2024-09-17T04:49:39.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c7e548640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3c78162c30 con 0x7f3c780c77b0 2024-09-17T04:49:39.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c77fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c780a85b0 0x7f3c78070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:39.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c77fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c780a85b0 0x7f3c78070de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43146/0 (socket says 172.21.15.150:43146) 2024-09-17T04:49:39.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c77fff640 1 -- 172.21.15.150:0/4153318812 learned_addr learned my addr 172.21.15.150:0/4153318812 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:39.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c777fe640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c780c6cb0 0x7f3c78148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:39.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c7cae5640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c780c77b0 0x7f3c7814d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:39.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c77fff640 1 -- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c780c77b0 msgr2=0x7f3c7814d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:39.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c77fff640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c780c77b0 0x7f3c7814d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:39.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c77fff640 1 -- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c780c6cb0 msgr2=0x7f3c78148db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:39.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c77fff640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c780c6cb0 0x7f3c78148db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:39.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c77fff640 1 -- 172.21.15.150:0/4153318812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c7814a470 con 0x7f3c780a85b0 2024-09-17T04:49:39.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c777fe640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c780c6cb0 0x7f3c78148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:39.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c77fff640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c780a85b0 0x7f3c78070de0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800c430 tx=0x7f3c6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:39.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c74ff9640 1 -- 172.21.15.150:0/4153318812 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c68011790 con 0x7f3c780a85b0 2024-09-17T04:49:39.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c74ff9640 1 -- 172.21.15.150:0/4153318812 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c68017070 con 0x7f3c780a85b0 2024-09-17T04:49:39.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c7e548640 1 -- 172.21.15.150:0/4153318812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c781492f0 con 0x7f3c780a85b0 2024-09-17T04:49:39.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c74ff9640 1 -- 172.21.15.150:0/4153318812 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c68011cc0 con 0x7f3c780a85b0 2024-09-17T04:49:39.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c7e548640 1 -- 172.21.15.150:0/4153318812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c781f1ac0 con 0x7f3c780a85b0 2024-09-17T04:49:39.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c7e548640 1 -- 172.21.15.150:0/4153318812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c78071370 con 0x7f3c780a85b0 2024-09-17T04:49:39.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.157+0000 7f3c74ff9640 1 -- 172.21.15.150:0/4153318812 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3c6801c460 con 0x7f3c780a85b0 2024-09-17T04:49:39.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.161+0000 7f3c74ff9640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c64073200 0x7f3c64075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:39.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.161+0000 7f3c74ff9640 1 -- 172.21.15.150:0/4153318812 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6054+0+0 (secure 0 0 0) 0x7f3c680941c0 con 0x7f3c780a85b0 2024-09-17T04:49:39.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.161+0000 7f3c777fe640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c64073200 0x7f3c64075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:39.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.161+0000 7f3c777fe640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c64073200 0x7f3c64075710 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0ae000 tx=0x7f3c6c01afd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:39.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.165+0000 7f3c74ff9640 1 -- 172.21.15.150:0/4153318812 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3c6805f080 con 0x7f3c780a85b0 2024-09-17T04:49:39.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.333+0000 7f3c7e548640 1 -- 172.21.15.150:0/4153318812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f3c780c7270 con 0x7f3c780a85b0 2024-09-17T04:49:39.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.961+0000 7f3c74ff9640 1 -- 172.21.15.150:0/4153318812 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'b_data' created v24) ==== 160+0+0 (secure 0 0 0) 0x7f3c6805e980 con 0x7f3c780a85b0 2024-09-17T04:49:39.962 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'b_data' created 2024-09-17T04:49:39.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.965+0000 7f3c7e548640 1 -- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c64073200 msgr2=0x7f3c64075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:39.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.965+0000 7f3c7e548640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c64073200 0x7f3c64075710 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0ae000 tx=0x7f3c6c01afd0 comp rx=0 tx=0).stop 2024-09-17T04:49:39.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.965+0000 7f3c7e548640 1 -- 172.21.15.150:0/4153318812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c780a85b0 msgr2=0x7f3c78070de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:39.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.965+0000 7f3c7e548640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c780a85b0 0x7f3c78070de0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800c430 tx=0x7f3c6800c8f0 comp rx=0 tx=0).stop 2024-09-17T04:49:39.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.965+0000 7f3c7e548640 1 -- 172.21.15.150:0/4153318812 shutdown_connections 2024-09-17T04:49:39.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.965+0000 7f3c7e548640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c64073200 0x7f3c64075710 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:39.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.965+0000 7f3c7e548640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c780c6cb0 0x7f3c78148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:39.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.965+0000 7f3c7e548640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c780c77b0 0x7f3c7814d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:39.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.965+0000 7f3c7e548640 1 --2- 172.21.15.150:0/4153318812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c780a85b0 0x7f3c78070de0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:39.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.965+0000 7f3c7e548640 1 -- 172.21.15.150:0/4153318812 >> 172.21.15.150:0/4153318812 conn(0x7f3c780aab60 msgr2=0x7f3c7814e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:39.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.965+0000 7f3c7e548640 1 -- 172.21.15.150:0/4153318812 shutdown_connections 2024-09-17T04:49:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:39.965+0000 7f3c7e548640 1 -- 172.21.15.150:0/4153318812 wait complete. 2024-09-17T04:49:40.005 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new b b_metadata b_data 2024-09-17T04:49:40.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.157+0000 7fc59c60e640 1 -- 172.21.15.150:0/2284830920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc594151d10 msgr2=0x7fc5940adae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:40.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.157+0000 7fc59c60e640 1 --2- 172.21.15.150:0/2284830920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc594151d10 0x7fc5940adae0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009bc0 tx=0x7fc58c01ca80 comp rx=0 tx=0).stop 2024-09-17T04:49:40.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.157+0000 7fc59c60e640 1 -- 172.21.15.150:0/2284830920 shutdown_connections 2024-09-17T04:49:40.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.157+0000 7fc59c60e640 1 --2- 172.21.15.150:0/2284830920 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5940699b0 0x7fc5940719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:40.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.157+0000 7fc59c60e640 1 --2- 172.21.15.150:0/2284830920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc594151d10 0x7fc5940adae0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:40.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.157+0000 7fc59c60e640 1 -- 172.21.15.150:0/2284830920 >> 172.21.15.150:0/2284830920 conn(0x7fc594070b50 msgr2=0x7fc594070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:40.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.157+0000 7fc59c60e640 1 -- 172.21.15.150:0/2284830920 shutdown_connections 2024-09-17T04:49:40.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.157+0000 7fc59c60e640 1 -- 172.21.15.150:0/2284830920 wait complete. 2024-09-17T04:49:40.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59c60e640 1 Processor -- start 2024-09-17T04:49:40.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59c60e640 1 -- start start 2024-09-17T04:49:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59c60e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5940699b0 0x7fc5941f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59c60e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59406e590 0x7fc59406d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59c60e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc594151d10 0x7fc594140c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59c60e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc594161990 con 0x7fc5940699b0 2024-09-17T04:49:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59c60e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc5940a9fb0 con 0x7fc594151d10 2024-09-17T04:49:40.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59c60e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc594161b10 con 0x7fc59406e590 2024-09-17T04:49:40.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59a3aa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5940699b0 0x7fc5941f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:40.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59a3aa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5940699b0 0x7fc5941f0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53114/0 (socket says 172.21.15.150:53114) 2024-09-17T04:49:40.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59a3aa640 1 -- 172.21.15.150:0/2054204577 learned_addr learned my addr 172.21.15.150:0/2054204577 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:40.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc599ba9640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59406e590 0x7fc59406d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:40.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59abab640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc594151d10 0x7fc594140c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:40.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59a3aa640 1 -- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59406e590 msgr2=0x7fc59406d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:40.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59a3aa640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59406e590 0x7fc59406d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:40.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59a3aa640 1 -- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc594151d10 msgr2=0x7fc594140c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:40.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59a3aa640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc594151d10 0x7fc594140c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:40.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59a3aa640 1 -- 172.21.15.150:0/2054204577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5941456c0 con 0x7fc5940699b0 2024-09-17T04:49:40.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59abab640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc594151d10 0x7fc594140c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:40.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc599ba9640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59406e590 0x7fc59406d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:40.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59a3aa640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5940699b0 0x7fc5941f0a70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc59000c430 tx=0x7fc59000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:40.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc58affd640 1 -- 172.21.15.150:0/2054204577 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc590011790 con 0x7fc5940699b0 2024-09-17T04:49:40.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59c60e640 1 -- 172.21.15.150:0/2054204577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc594145950 con 0x7fc5940699b0 2024-09-17T04:49:40.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc58affd640 1 -- 172.21.15.150:0/2054204577 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc590017070 con 0x7fc5940699b0 2024-09-17T04:49:40.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc58affd640 1 -- 172.21.15.150:0/2054204577 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc590011ce0 con 0x7fc5940699b0 2024-09-17T04:49:40.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.161+0000 7fc59c60e640 1 -- 172.21.15.150:0/2054204577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5940c7720 con 0x7fc5940699b0 2024-09-17T04:49:40.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.165+0000 7fc59c60e640 1 -- 172.21.15.150:0/2054204577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5940719e0 con 0x7fc5940699b0 2024-09-17T04:49:40.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.165+0000 7fc58affd640 1 -- 172.21.15.150:0/2054204577 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc59001c460 con 0x7fc5940699b0 2024-09-17T04:49:40.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.165+0000 7fc58affd640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc578073200 0x7fc578075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:40.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.165+0000 7fc58affd640 1 -- 172.21.15.150:0/2054204577 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6431+0+0 (secure 0 0 0) 0x7fc590094ce0 con 0x7fc5940699b0 2024-09-17T04:49:40.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.165+0000 7fc599ba9640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc578073200 0x7fc578075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:40.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.165+0000 7fc599ba9640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc578073200 0x7fc578075710 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc584005fd0 tx=0x7fc584005de0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:40.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.169+0000 7fc58affd640 1 -- 172.21.15.150:0/2054204577 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc59005fb80 con 0x7fc5940699b0 2024-09-17T04:49:40.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:40.373+0000 7fc59c60e640 1 -- 172.21.15.150:0/2054204577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"} v 0) -- 0x7fc5941541c0 con 0x7fc5940699b0 2024-09-17T04:49:41.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.025+0000 7fc58affd640 1 -- 172.21.15.150:0/2054204577 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"}]=0 new fs with metadata pool 4 and data pool 5 v10) ==== 157+0+0 (secure 0 0 0) 0x7fc59005f480 con 0x7fc5940699b0 2024-09-17T04:49:41.042 INFO:teuthology.orchestra.run.smithi150.stderr:new fs with metadata pool 4 and data pool 5 2024-09-17T04:49:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.053+0000 7fc59c60e640 1 -- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc578073200 msgr2=0x7fc578075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.053+0000 7fc59c60e640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc578073200 0x7fc578075710 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc584005fd0 tx=0x7fc584005de0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.053+0000 7fc59c60e640 1 -- 172.21.15.150:0/2054204577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5940699b0 msgr2=0x7fc5941f0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.053+0000 7fc59c60e640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5940699b0 0x7fc5941f0a70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc59000c430 tx=0x7fc59000c8f0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.053+0000 7fc59c60e640 1 -- 172.21.15.150:0/2054204577 shutdown_connections 2024-09-17T04:49:41.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.053+0000 7fc59c60e640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc578073200 0x7fc578075710 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.053+0000 7fc59c60e640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc594151d10 0x7fc594140c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.053+0000 7fc59c60e640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc59406e590 0x7fc59406d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.053+0000 7fc59c60e640 1 --2- 172.21.15.150:0/2054204577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5940699b0 0x7fc5941f0a70 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc59000c430 tx=0x7fc59000c8f0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.053+0000 7fc59c60e640 1 -- 172.21.15.150:0/2054204577 >> 172.21.15.150:0/2054204577 conn(0x7fc594070b50 msgr2=0x7fc59414e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:41.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.053+0000 7fc59c60e640 1 -- 172.21.15.150:0/2054204577 shutdown_connections 2024-09-17T04:49:41.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.053+0000 7fc59c60e640 1 -- 172.21.15.150:0/2054204577 wait complete. 2024-09-17T04:49:41.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:41.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/1438531502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c6c004d50 msgr2=0x7f8c6c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 --2- 172.21.15.150:0/1438531502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c6c004d50 0x7f8c6c005160 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00c640 tx=0x7f8c7c01fa60 comp rx=0 tx=0).stop 2024-09-17T04:49:41.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c89c73640 1 -- 172.21.15.150:0/1438531502 reap_dead start 2024-09-17T04:49:41.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/1438531502 shutdown_connections 2024-09-17T04:49:41.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/1438531502 >> 172.21.15.150:0/1438531502 conn(0x7f8c840c4a00 msgr2=0x7f8c840c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:41.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/1438531502 shutdown_connections 2024-09-17T04:49:41.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/1438531502 wait complete. 2024-09-17T04:49:41.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 Processor -- start 2024-09-17T04:49:41.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 -- start start 2024-09-17T04:49:41.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c841499c0 0x7f8c84148600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:41.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c84147490 0x7f8c841478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:41.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c8414a640 0x7f8c8414aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:41.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8c840adce0 con 0x7f8c8414a640 2024-09-17T04:49:41.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8406f4a0 con 0x7f8c841499c0 2024-09-17T04:49:41.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c8b6d6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8c840ade60 con 0x7f8c84147490 2024-09-17T04:49:41.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.233+0000 7f8c89c73640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c8414a640 0x7f8c8414aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:41.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c89c73640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c8414a640 0x7f8c8414aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53126/0 (socket says 172.21.15.150:53126) 2024-09-17T04:49:41.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c89c73640 1 -- 172.21.15.150:0/3222002577 learned_addr learned my addr 172.21.15.150:0/3222002577 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:41.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c88c71640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c84147490 0x7f8c841478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:41.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c89472640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c841499c0 0x7f8c84148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:41.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c89c73640 1 -- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c84147490 msgr2=0x7f8c841478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c89c73640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c84147490 0x7f8c841478a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c89c73640 1 -- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c841499c0 msgr2=0x7f8c84148600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c89c73640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c841499c0 0x7f8c84148600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c89c73640 1 -- 172.21.15.150:0/3222002577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c84147de0 con 0x7f8c8414a640 2024-09-17T04:49:41.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c89c73640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c8414a640 0x7f8c8414aa50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c09c350 tx=0x7f8c7c01fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:41.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c81ffb640 1 -- 172.21.15.150:0/3222002577 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c004e60 con 0x7f8c8414a640 2024-09-17T04:49:41.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c81ffb640 1 -- 172.21.15.150:0/3222002577 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c7c001840 con 0x7f8c8414a640 2024-09-17T04:49:41.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/3222002577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c84150830 con 0x7f8c8414a640 2024-09-17T04:49:41.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c81ffb640 1 -- 172.21.15.150:0/3222002577 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c002040 con 0x7f8c8414a640 2024-09-17T04:49:41.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/3222002577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c84149dd0 con 0x7f8c8414a640 2024-09-17T04:49:41.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/3222002577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c841eb150 con 0x7f8c8414a640 2024-09-17T04:49:41.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c81ffb640 1 -- 172.21.15.150:0/3222002577 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8c7c0bb030 con 0x7f8c8414a640 2024-09-17T04:49:41.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c81ffb640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c70073200 0x7f8c70075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:41.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.237+0000 7f8c81ffb640 1 -- 172.21.15.150:0/3222002577 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f8c7c1311c0 con 0x7f8c8414a640 2024-09-17T04:49:41.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.241+0000 7f8c89472640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c70073200 0x7f8c70075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:41.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.241+0000 7f8c89472640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c70073200 0x7f8c70075710 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8c841513e0 tx=0x7f8c740091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:41.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.241+0000 7f8c81ffb640 1 -- 172.21.15.150:0/3222002577 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8c7c019810 con 0x7f8c8414a640 2024-09-17T04:49:41.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.413+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/3222002577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c8414a290 con 0x7f8c8414a640 2024-09-17T04:49:41.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c81ffb640 1 -- 172.21.15.150:0/3222002577 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+20444 (secure 0 0 0) 0x7f8c8414a290 con 0x7f8c8414a640 2024-09-17T04:49:41.417 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:41.417 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":25,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:40.378387+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:41.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c70073200 msgr2=0x7f8c70075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c8b6d6640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c70073200 0x7f8c70075710 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8c841513e0 tx=0x7f8c740091e0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/3222002577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c8414a640 msgr2=0x7f8c8414aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c8b6d6640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c8414a640 0x7f8c8414aa50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c09c350 tx=0x7f8c7c01fcb0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/3222002577 shutdown_connections 2024-09-17T04:49:41.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c8b6d6640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c70073200 0x7f8c70075710 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c8b6d6640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c8414a640 0x7f8c8414aa50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c8b6d6640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c84147490 0x7f8c841478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c8b6d6640 1 --2- 172.21.15.150:0/3222002577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c841499c0 0x7f8c84148600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/3222002577 >> 172.21.15.150:0/3222002577 conn(0x7f8c840c4a00 msgr2=0x7f8c840c64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:41.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/3222002577 shutdown_connections 2024-09-17T04:49:41.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.417+0000 7f8c8b6d6640 1 -- 172.21.15.150:0/3222002577 wait complete. 2024-09-17T04:49:41.440 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:41.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.593+0000 7f778aa54640 1 -- 172.21.15.150:0/737209873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7784072ea0 msgr2=0x7f77840732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.593+0000 7f778aa54640 1 --2- 172.21.15.150:0/737209873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7784072ea0 0x7f77840732b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f776c009c80 tx=0x7f776c01cda0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.593+0000 7f778aa54640 1 -- 172.21.15.150:0/737209873 shutdown_connections 2024-09-17T04:49:41.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.593+0000 7f778aa54640 1 --2- 172.21.15.150:0/737209873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7784072ea0 0x7f77840732b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.593+0000 7f778aa54640 1 --2- 172.21.15.150:0/737209873 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77840c97a0 0x7f7784151c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.593+0000 7f778aa54640 1 -- 172.21.15.150:0/737209873 >> 172.21.15.150:0/737209873 conn(0x7f77840abb00 msgr2=0x7f77840abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:41.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.593+0000 7f778aa54640 1 -- 172.21.15.150:0/737209873 shutdown_connections 2024-09-17T04:49:41.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.593+0000 7f778aa54640 1 -- 172.21.15.150:0/737209873 wait complete. 2024-09-17T04:49:41.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f778aa54640 1 Processor -- start 2024-09-17T04:49:41.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f778aa54640 1 -- start start 2024-09-17T04:49:41.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f778aa54640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7784072ea0 0x7f7784147d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:41.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f778aa54640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77840c97a0 0x7f77841468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:41.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f778aa54640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7784157be0 0x7f7784146e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:41.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f778aa54640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7784161790 con 0x7f77840c97a0 2024-09-17T04:49:41.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f778aa54640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f77840c4df0 con 0x7f7784072ea0 2024-09-17T04:49:41.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f778aa54640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7784161910 con 0x7f7784157be0 2024-09-17T04:49:41.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f77837fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77840c97a0 0x7f77841468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:41.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f77837fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77840c97a0 0x7f77841468f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53136/0 (socket says 172.21.15.150:53136) 2024-09-17T04:49:41.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f77837fe640 1 -- 172.21.15.150:0/2270902340 learned_addr learned my addr 172.21.15.150:0/2270902340 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:41.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f7788ff1640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7784157be0 0x7f7784146e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:41.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f7783fff640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7784072ea0 0x7f7784147d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:41.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f77837fe640 1 -- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7784157be0 msgr2=0x7f7784146e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f77837fe640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7784157be0 0x7f7784146e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f77837fe640 1 -- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7784072ea0 msgr2=0x7f7784147d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f77837fe640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7784072ea0 0x7f7784147d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f77837fe640 1 -- 172.21.15.150:0/2270902340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77841edd20 con 0x7f77840c97a0 2024-09-17T04:49:41.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f7788ff1640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7784157be0 0x7f7784146e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:41.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f77837fe640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77840c97a0 0x7f77841468f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f776c009bd0 tx=0x7f776c0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:41.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f7780ff9640 1 -- 172.21.15.150:0/2270902340 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c0bb070 con 0x7f77840c97a0 2024-09-17T04:49:41.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f7780ff9640 1 -- 172.21.15.150:0/2270902340 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f776c004e00 con 0x7f77840c97a0 2024-09-17T04:49:41.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f778aa54640 1 -- 172.21.15.150:0/2270902340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77840acb40 con 0x7f77840c97a0 2024-09-17T04:49:41.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f7780ff9640 1 -- 172.21.15.150:0/2270902340 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c0b2040 con 0x7f77840c97a0 2024-09-17T04:49:41.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f778aa54640 1 -- 172.21.15.150:0/2270902340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77840ad100 con 0x7f77840c97a0 2024-09-17T04:49:41.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.597+0000 7f778aa54640 1 -- 172.21.15.150:0/2270902340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77840732b0 con 0x7f77840c97a0 2024-09-17T04:49:41.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.601+0000 7f7780ff9640 1 -- 172.21.15.150:0/2270902340 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f776c0ad850 con 0x7f77840c97a0 2024-09-17T04:49:41.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.601+0000 7f7780ff9640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7764073230 0x7f7764075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:41.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.601+0000 7f7780ff9640 1 -- 172.21.15.150:0/2270902340 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f776c12eab0 con 0x7f77840c97a0 2024-09-17T04:49:41.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.601+0000 7f7783fff640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7764073230 0x7f7764075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:41.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.601+0000 7f7783fff640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7764073230 0x7f7764075740 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f777c0058d0 tx=0x7f777c005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:41.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.605+0000 7f7780ff9640 1 -- 172.21.15.150:0/2270902340 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f776c0c4020 con 0x7f77840c97a0 2024-09-17T04:49:41.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.773+0000 7f778aa54640 1 -- 172.21.15.150:0/2270902340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77840c3e50 con 0x7f77840c97a0 2024-09-17T04:49:41.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.777+0000 7f7780ff9640 1 -- 172.21.15.150:0/2270902340 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+20444 (secure 0 0 0) 0x7f776c016990 con 0x7f77840c97a0 2024-09-17T04:49:41.779 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:41.779 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":25,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:40.378387+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:41.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.781+0000 7f778aa54640 1 -- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7764073230 msgr2=0x7f7764075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.781+0000 7f778aa54640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7764073230 0x7f7764075740 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f777c0058d0 tx=0x7f777c005860 comp rx=0 tx=0).stop 2024-09-17T04:49:41.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.781+0000 7f778aa54640 1 -- 172.21.15.150:0/2270902340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77840c97a0 msgr2=0x7f77841468f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.781+0000 7f778aa54640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77840c97a0 0x7f77841468f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f776c009bd0 tx=0x7f776c0093f0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.781+0000 7f778aa54640 1 -- 172.21.15.150:0/2270902340 shutdown_connections 2024-09-17T04:49:41.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.781+0000 7f778aa54640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7764073230 0x7f7764075740 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.781+0000 7f778aa54640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7784157be0 0x7f7784146e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.781+0000 7f778aa54640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77840c97a0 0x7f77841468f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.781+0000 7f778aa54640 1 --2- 172.21.15.150:0/2270902340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7784072ea0 0x7f7784147d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.781+0000 7f778aa54640 1 -- 172.21.15.150:0/2270902340 >> 172.21.15.150:0/2270902340 conn(0x7f77840abb00 msgr2=0x7f77840aa480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:41.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.781+0000 7f778aa54640 1 -- 172.21.15.150:0/2270902340 shutdown_connections 2024-09-17T04:49:41.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.781+0000 7f778aa54640 1 -- 172.21.15.150:0/2270902340 wait complete. 2024-09-17T04:49:41.800 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b standby_count_wanted 0 2024-09-17T04:49:41.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.945+0000 7f1d83a52640 1 -- 172.21.15.150:0/678367362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d68003fe0 msgr2=0x7f1d68004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.945+0000 7f1d83a52640 1 --2- 172.21.15.150:0/678367362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d68003fe0 0x7f1d68004810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c009b30 tx=0x7f1d6c01d270 comp rx=0 tx=0).stop 2024-09-17T04:49:41.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.945+0000 7f1d81fef640 1 -- 172.21.15.150:0/678367362 reap_dead start 2024-09-17T04:49:41.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.945+0000 7f1d83a52640 1 -- 172.21.15.150:0/678367362 shutdown_connections 2024-09-17T04:49:41.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.945+0000 7f1d83a52640 1 -- 172.21.15.150:0/678367362 >> 172.21.15.150:0/678367362 conn(0x7f1d7c0c5cb0 msgr2=0x7f1d7c0c60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:41.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.945+0000 7f1d83a52640 1 -- 172.21.15.150:0/678367362 shutdown_connections 2024-09-17T04:49:41.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.945+0000 7f1d83a52640 1 -- 172.21.15.150:0/678367362 wait complete. 2024-09-17T04:49:41.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d83a52640 1 Processor -- start 2024-09-17T04:49:41.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d83a52640 1 -- start start 2024-09-17T04:49:41.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d83a52640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d68003fe0 0x7f1d7c14c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:41.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d83a52640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d7c0ac320 0x7f1d7c1ed720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:41.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d83a52640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d7c1edc60 0x7f1d7c0adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:41.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d83a52640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c06ebd0 con 0x7f1d7c1edc60 2024-09-17T04:49:41.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d83a52640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7c06ea50 con 0x7f1d7c0ac320 2024-09-17T04:49:41.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d83a52640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1d7c072e00 con 0x7f1d68003fe0 2024-09-17T04:49:41.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d81fef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d7c1edc60 0x7f1d7c0adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:41.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d81fef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d7c1edc60 0x7f1d7c0adef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53152/0 (socket says 172.21.15.150:53152) 2024-09-17T04:49:41.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d81fef640 1 -- 172.21.15.150:0/1281989350 learned_addr learned my addr 172.21.15.150:0/1281989350 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:41.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d817ee640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d68003fe0 0x7f1d7c14c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:41.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d80fed640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d7c0ac320 0x7f1d7c1ed720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:41.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d81fef640 1 -- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d68003fe0 msgr2=0x7f1d7c14c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d81fef640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d68003fe0 0x7f1d7c14c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d81fef640 1 -- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d7c0ac320 msgr2=0x7f1d7c1ed720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:41.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d81fef640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d7c0ac320 0x7f1d7c1ed720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:41.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d81fef640 1 -- 172.21.15.150:0/1281989350 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7c0c6160 con 0x7f1d7c1edc60 2024-09-17T04:49:41.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d80fed640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d7c0ac320 0x7f1d7c1ed720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:41.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d81fef640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d7c1edc60 0x7f1d7c0adef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0c4bc0 tx=0x7f1d7400ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:41.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d71ffb640 1 -- 172.21.15.150:0/1281989350 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d74004b30 con 0x7f1d7c1edc60 2024-09-17T04:49:41.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d71ffb640 1 -- 172.21.15.150:0/1281989350 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d74004dc0 con 0x7f1d7c1edc60 2024-09-17T04:49:41.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d83a52640 1 -- 172.21.15.150:0/1281989350 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7c154970 con 0x7f1d7c1edc60 2024-09-17T04:49:41.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d71ffb640 1 -- 172.21.15.150:0/1281989350 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7400cd30 con 0x7f1d7c1edc60 2024-09-17T04:49:41.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d83a52640 1 -- 172.21.15.150:0/1281989350 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d7c157010 con 0x7f1d7c1edc60 2024-09-17T04:49:41.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.949+0000 7f1d83a52640 1 -- 172.21.15.150:0/1281989350 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d7c154b40 con 0x7f1d7c1edc60 2024-09-17T04:49:41.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.953+0000 7f1d71ffb640 1 -- 172.21.15.150:0/1281989350 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1d7401b030 con 0x7f1d7c1edc60 2024-09-17T04:49:41.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.953+0000 7f1d71ffb640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d60073200 0x7f1d60075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:41.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.953+0000 7f1d71ffb640 1 -- 172.21.15.150:0/1281989350 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6521+0+0 (secure 0 0 0) 0x7f1d740947f0 con 0x7f1d7c1edc60 2024-09-17T04:49:41.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.953+0000 7f1d817ee640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d60073200 0x7f1d60075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:41.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.953+0000 7f1d817ee640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d60073200 0x7f1d60075710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d78001870 tx=0x7f1d780015e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:41.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:41.957+0000 7f1d71ffb640 1 -- 172.21.15.150:0/1281989350 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1d74022060 con 0x7f1d7c1edc60 2024-09-17T04:49:42.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:42.153+0000 7f1d83a52640 1 -- 172.21.15.150:0/1281989350 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f1d7c1f11a0 con 0x7f1d7c1edc60 2024-09-17T04:49:42.175 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T04:49:42.173+0000 7f3714f81640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:49:43.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.044+0000 7f1d71ffb640 1 -- 172.21.15.150:0/1281989350 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"}]=0 v13) ==== 113+0+0 (secure 0 0 0) 0x7f1d7405f650 con 0x7f1d7c1edc60 2024-09-17T04:49:43.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.048+0000 7f1d83a52640 1 -- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d60073200 msgr2=0x7f1d60075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:43.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.048+0000 7f1d83a52640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d60073200 0x7f1d60075710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d78001870 tx=0x7f1d780015e0 comp rx=0 tx=0).stop 2024-09-17T04:49:43.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.048+0000 7f1d83a52640 1 -- 172.21.15.150:0/1281989350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d7c1edc60 msgr2=0x7f1d7c0adef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:43.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.048+0000 7f1d83a52640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d7c1edc60 0x7f1d7c0adef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0c4bc0 tx=0x7f1d7400ee10 comp rx=0 tx=0).stop 2024-09-17T04:49:43.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.048+0000 7f1d83a52640 1 -- 172.21.15.150:0/1281989350 shutdown_connections 2024-09-17T04:49:43.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.048+0000 7f1d83a52640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d60073200 0x7f1d60075710 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:43.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.048+0000 7f1d83a52640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d7c0ac320 0x7f1d7c1ed720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:43.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.048+0000 7f1d83a52640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d7c1edc60 0x7f1d7c0adef0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:43.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.048+0000 7f1d83a52640 1 --2- 172.21.15.150:0/1281989350 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d68003fe0 0x7f1d7c14c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:43.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.048+0000 7f1d83a52640 1 -- 172.21.15.150:0/1281989350 >> 172.21.15.150:0/1281989350 conn(0x7f1d7c0c5cb0 msgr2=0x7f1d7c0c7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:43.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.048+0000 7f1d83a52640 1 -- 172.21.15.150:0/1281989350 shutdown_connections 2024-09-17T04:49:43.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.048+0000 7f1d83a52640 1 -- 172.21.15.150:0/1281989350 wait complete. 2024-09-17T04:49:43.075 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-09-17T04:49:43.076 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b allow_standby_replay false 2024-09-17T04:49:43.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.236+0000 7f1bf7759640 1 -- 172.21.15.150:0/334846773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bf01536b0 msgr2=0x7f1bf0153ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:43.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.236+0000 7f1bf7759640 1 --2- 172.21.15.150:0/334846773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bf01536b0 0x7f1bf0153ac0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0073f60 tx=0x7f1bec01ccf0 comp rx=0 tx=0).stop 2024-09-17T04:49:43.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.236+0000 7f1bf7759640 1 -- 172.21.15.150:0/334846773 shutdown_connections 2024-09-17T04:49:43.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.236+0000 7f1bf7759640 1 --2- 172.21.15.150:0/334846773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bf01536b0 0x7f1bf0153ac0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:43.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.236+0000 7f1bf7759640 1 --2- 172.21.15.150:0/334846773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bf00c5840 0x7f1bf00c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:43.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.236+0000 7f1bf7759640 1 --2- 172.21.15.150:0/334846773 >> v2:172.21.15.176:3300/0 conn(0x7f1bf0150d80 0x7f1bf00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:43.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.236+0000 7f1bf7759640 1 -- 172.21.15.150:0/334846773 >> 172.21.15.150:0/334846773 conn(0x7f1bf00c8db0 msgr2=0x7f1bf00c91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:43.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.236+0000 7f1bf7759640 1 -- 172.21.15.150:0/334846773 shutdown_connections 2024-09-17T04:49:43.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.236+0000 7f1bf7759640 1 -- 172.21.15.150:0/334846773 wait complete. 2024-09-17T04:49:43.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf7759640 1 Processor -- start 2024-09-17T04:49:43.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf7759640 1 -- start start 2024-09-17T04:49:43.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf7759640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bf00c5840 0x7f1bf00be010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:43.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf7759640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bf0150d80 0x7f1bf014f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:43.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf7759640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bf01536b0 0x7f1bf014d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:43.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf7759640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf0154180 con 0x7f1bf00c5840 2024-09-17T04:49:43.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf7759640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf0154000 con 0x7f1bf01536b0 2024-09-17T04:49:43.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf7759640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf00c86a0 con 0x7f1bf0150d80 2024-09-17T04:49:43.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf54f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bf00c5840 0x7f1bf00be010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:43.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf54f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bf00c5840 0x7f1bf00be010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53174/0 (socket says 172.21.15.150:53174) 2024-09-17T04:49:43.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf54f5640 1 -- 172.21.15.150:0/2620268597 learned_addr learned my addr 172.21.15.150:0/2620268597 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:43.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf4cf4640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bf0150d80 0x7f1bf014f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:43.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf5cf6640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bf01536b0 0x7f1bf014d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:43.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf54f5640 1 -- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bf0150d80 msgr2=0x7f1bf014f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:43.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf54f5640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bf0150d80 0x7f1bf014f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:43.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf54f5640 1 -- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bf01536b0 msgr2=0x7f1bf014d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:43.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf54f5640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bf01536b0 0x7f1bf014d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:43.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf54f5640 1 -- 172.21.15.150:0/2620268597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf00aa6c0 con 0x7f1bf00c5840 2024-09-17T04:49:43.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf4cf4640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bf0150d80 0x7f1bf014f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:43.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf5cf6640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bf01536b0 0x7f1bf014d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:43.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf54f5640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bf00c5840 0x7f1bf00be010 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1be80058d0 tx=0x7f1be8009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:43.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1be5ffb640 1 -- 172.21.15.150:0/2620268597 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be801d070 con 0x7f1bf00c5840 2024-09-17T04:49:43.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf7759640 1 -- 172.21.15.150:0/2620268597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf00bf300 con 0x7f1bf00c5840 2024-09-17T04:49:43.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1be5ffb640 1 -- 172.21.15.150:0/2620268597 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1be8007770 con 0x7f1bf00c5840 2024-09-17T04:49:43.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1be5ffb640 1 -- 172.21.15.150:0/2620268597 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8007ba0 con 0x7f1bf00c5840 2024-09-17T04:49:43.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.240+0000 7f1bf7759640 1 -- 172.21.15.150:0/2620268597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf01f0660 con 0x7f1bf00c5840 2024-09-17T04:49:43.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.244+0000 7f1bf7759640 1 -- 172.21.15.150:0/2620268597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf00702c0 con 0x7f1bf00c5840 2024-09-17T04:49:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.244+0000 7f1be5ffb640 1 -- 172.21.15.150:0/2620268597 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1be8020070 con 0x7f1bf00c5840 2024-09-17T04:49:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.244+0000 7f1be5ffb640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1bd4073200 0x7f1bd4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.244+0000 7f1be5ffb640 1 -- 172.21.15.150:0/2620268597 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f1be8095d30 con 0x7f1bf00c5840 2024-09-17T04:49:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.244+0000 7f1bf4cf4640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1bd4073200 0x7f1bd4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.244+0000 7f1bf4cf4640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1bd4073200 0x7f1bd4075710 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1be0005fd0 tx=0x7f1be0005d60 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:43.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.248+0000 7f1be5ffb640 1 -- 172.21.15.150:0/2620268597 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1be80603e0 con 0x7f1bf00c5840 2024-09-17T04:49:43.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:43.440+0000 7f1bf7759640 1 -- 172.21.15.150:0/2620268597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f1bf00c8480 con 0x7f1bf00c5840 2024-09-17T04:49:44.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.048+0000 7f1be5ffb640 1 -- 172.21.15.150:0/2620268597 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"}]=0 v14) ==== 117+0+0 (secure 0 0 0) 0x7f1be8064e10 con 0x7f1bf00c5840 2024-09-17T04:49:44.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.052+0000 7f1bf7759640 1 -- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1bd4073200 msgr2=0x7f1bd4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:44.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.052+0000 7f1bf7759640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1bd4073200 0x7f1bd4075710 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1be0005fd0 tx=0x7f1be0005d60 comp rx=0 tx=0).stop 2024-09-17T04:49:44.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.052+0000 7f1bf7759640 1 -- 172.21.15.150:0/2620268597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bf00c5840 msgr2=0x7f1bf00be010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:44.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.052+0000 7f1bf7759640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bf00c5840 0x7f1bf00be010 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1be80058d0 tx=0x7f1be8009d30 comp rx=0 tx=0).stop 2024-09-17T04:49:44.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.052+0000 7f1bf7759640 1 -- 172.21.15.150:0/2620268597 shutdown_connections 2024-09-17T04:49:44.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.052+0000 7f1bf7759640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1bd4073200 0x7f1bd4075710 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:44.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.052+0000 7f1bf7759640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1bf01536b0 0x7f1bf014d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:44.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.052+0000 7f1bf7759640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1bf0150d80 0x7f1bf014f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:44.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.052+0000 7f1bf7759640 1 --2- 172.21.15.150:0/2620268597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bf00c5840 0x7f1bf00be010 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:44.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.052+0000 7f1bf7759640 1 -- 172.21.15.150:0/2620268597 >> 172.21.15.150:0/2620268597 conn(0x7f1bf00c8db0 msgr2=0x7f1bf00c36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:44.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.052+0000 7f1bf7759640 1 -- 172.21.15.150:0/2620268597 shutdown_connections 2024-09-17T04:49:44.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.052+0000 7f1bf7759640 1 -- 172.21.15.150:0/2620268597 wait complete. 2024-09-17T04:49:44.083 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b session_timeout 300 2024-09-17T04:49:44.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.244+0000 7f7b758e5640 1 -- 172.21.15.150:0/2859566451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b5c004d50 msgr2=0x7f7b5c005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:44.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.244+0000 7f7b758e5640 1 --2- 172.21.15.150:0/2859566451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b5c004d50 0x7f7b5c005200 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009e90 tx=0x7f7b6801d5f0 comp rx=0 tx=0).stop 2024-09-17T04:49:44.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.244+0000 7f7b6f7fe640 1 -- 172.21.15.150:0/2859566451 reap_dead start 2024-09-17T04:49:44.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.244+0000 7f7b758e5640 1 -- 172.21.15.150:0/2859566451 shutdown_connections 2024-09-17T04:49:44.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.244+0000 7f7b758e5640 1 -- 172.21.15.150:0/2859566451 >> 172.21.15.150:0/2859566451 conn(0x7f7b700c4a00 msgr2=0x7f7b700c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:44.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.244+0000 7f7b758e5640 1 -- 172.21.15.150:0/2859566451 shutdown_connections 2024-09-17T04:49:44.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.244+0000 7f7b758e5640 1 -- 172.21.15.150:0/2859566451 wait complete. 2024-09-17T04:49:44.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b758e5640 1 Processor -- start 2024-09-17T04:49:44.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b758e5640 1 -- start start 2024-09-17T04:49:44.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b758e5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b5c004d50 0x7f7b701e9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:44.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b758e5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b700be610 0x7f7b70156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:44.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b758e5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b70150490 0x7f7b701508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:44.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b758e5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7b700addb0 con 0x7f7b70150490 2024-09-17T04:49:44.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b758e5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7b700704f0 con 0x7f7b5c004d50 2024-09-17T04:49:44.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b758e5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7b700c44b0 con 0x7f7b700be610 2024-09-17T04:49:44.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b70150490 0x7f7b701508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:44.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b70150490 0x7f7b701508a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53198/0 (socket says 172.21.15.150:53198) 2024-09-17T04:49:44.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6f7fe640 1 -- 172.21.15.150:0/74629463 learned_addr learned my addr 172.21.15.150:0/74629463 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:44.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b67fff640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b700be610 0x7f7b70156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:44.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6effd640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b5c004d50 0x7f7b701e9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:44.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6f7fe640 1 -- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b700be610 msgr2=0x7f7b70156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:44.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6f7fe640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b700be610 0x7f7b70156dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6f7fe640 1 -- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b5c004d50 msgr2=0x7f7b701e9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6f7fe640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b5c004d50 0x7f7b701e9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6f7fe640 1 -- 172.21.15.150:0/74629463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b700bcdf0 con 0x7f7b70150490 2024-09-17T04:49:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b67fff640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b700be610 0x7f7b70156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6effd640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b5c004d50 0x7f7b701e9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6f7fe640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b70150490 0x7f7b701508a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7b6801d060 tx=0x7f7b68004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6cff9640 1 -- 172.21.15.150:0/74629463 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b680ad030 con 0x7f7b70150490 2024-09-17T04:49:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6cff9640 1 -- 172.21.15.150:0/74629463 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b680a7a90 con 0x7f7b70150490 2024-09-17T04:49:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b758e5640 1 -- 172.21.15.150:0/74629463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b700bd9f0 con 0x7f7b70150490 2024-09-17T04:49:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b6cff9640 1 -- 172.21.15.150:0/74629463 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b680a7e80 con 0x7f7b70150490 2024-09-17T04:49:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b758e5640 1 -- 172.21.15.150:0/74629463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b70157740 con 0x7f7b70150490 2024-09-17T04:49:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.248+0000 7f7b758e5640 1 -- 172.21.15.150:0/74629463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b700c3fe0 con 0x7f7b70150490 2024-09-17T04:49:44.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.252+0000 7f7b6cff9640 1 -- 172.21.15.150:0/74629463 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7b680b0070 con 0x7f7b70150490 2024-09-17T04:49:44.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.252+0000 7f7b6cff9640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7b50073200 0x7f7b50075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:44.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.252+0000 7f7b6cff9640 1 -- 172.21.15.150:0/74629463 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f7b6812fa30 con 0x7f7b70150490 2024-09-17T04:49:44.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.252+0000 7f7b6effd640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7b50073200 0x7f7b50075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:44.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.252+0000 7f7b6effd640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7b50073200 0x7f7b50075710 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7b60001170 tx=0x7f7b60009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:44.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.256+0000 7f7b6cff9640 1 -- 172.21.15.150:0/74629463 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7b680b7070 con 0x7f7b70150490 2024-09-17T04:49:44.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:44.464+0000 7f7b758e5640 1 -- 172.21.15.150:0/74629463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"} v 0) -- 0x7f7b700a7730 con 0x7f7b70150490 2024-09-17T04:49:45.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.068+0000 7f7b6cff9640 1 -- 172.21.15.150:0/74629463 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"}]=0 v15) ==== 110+0+0 (secure 0 0 0) 0x7f7b68017300 con 0x7f7b70150490 2024-09-17T04:49:45.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.072+0000 7f7b758e5640 1 -- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7b50073200 msgr2=0x7f7b50075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.072+0000 7f7b758e5640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7b50073200 0x7f7b50075710 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7b60001170 tx=0x7f7b60009040 comp rx=0 tx=0).stop 2024-09-17T04:49:45.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.072+0000 7f7b758e5640 1 -- 172.21.15.150:0/74629463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b70150490 msgr2=0x7f7b701508a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.072+0000 7f7b758e5640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b70150490 0x7f7b701508a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7b6801d060 tx=0x7f7b68004f50 comp rx=0 tx=0).stop 2024-09-17T04:49:45.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.072+0000 7f7b758e5640 1 -- 172.21.15.150:0/74629463 shutdown_connections 2024-09-17T04:49:45.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.072+0000 7f7b758e5640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7b50073200 0x7f7b50075710 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.072+0000 7f7b758e5640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b70150490 0x7f7b701508a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.072+0000 7f7b758e5640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b700be610 0x7f7b70156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.072+0000 7f7b758e5640 1 --2- 172.21.15.150:0/74629463 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b5c004d50 0x7f7b701e9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.072+0000 7f7b758e5640 1 -- 172.21.15.150:0/74629463 >> 172.21.15.150:0/74629463 conn(0x7f7b700c4a00 msgr2=0x7f7b700ab850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:45.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.072+0000 7f7b758e5640 1 -- 172.21.15.150:0/74629463 shutdown_connections 2024-09-17T04:49:45.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.072+0000 7f7b758e5640 1 -- 172.21.15.150:0/74629463 wait complete. 2024-09-17T04:49:45.099 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'b' 2024-09-17T04:49:45.099 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:49:45.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 -- 172.21.15.150:0/1070051139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d0003a60 msgr2=0x7f03d0003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 --2- 172.21.15.150:0/1070051139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d0003a60 0x7f03d0003ed0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0016f0 tx=0x7f03dc01c720 comp rx=0 tx=0).stop 2024-09-17T04:49:45.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 -- 172.21.15.150:0/1070051139 shutdown_connections 2024-09-17T04:49:45.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 --2- 172.21.15.150:0/1070051139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d0003a60 0x7f03d0003ed0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 -- 172.21.15.150:0/1070051139 >> 172.21.15.150:0/1070051139 conn(0x7f03e40abb00 msgr2=0x7f03e40abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:45.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 -- 172.21.15.150:0/1070051139 shutdown_connections 2024-09-17T04:49:45.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 -- 172.21.15.150:0/1070051139 wait complete. 2024-09-17T04:49:45.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 Processor -- start 2024-09-17T04:49:45.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 -- start start 2024-09-17T04:49:45.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03d0003a60 0x7f03e40aded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:45.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03e41eec20 0x7f03e40bee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:45.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e41e2dd0 0x7f03e4138f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:45.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f03e4158120 con 0x7f03e41eec20 2024-09-17T04:49:45.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f03e4157fa0 con 0x7f03e41e2dd0 2024-09-17T04:49:45.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e9c70640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f03e41582a0 con 0x7f03d0003a60 2024-09-17T04:49:45.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03e41eec20 0x7f03e40bee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:45.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03e41eec20 0x7f03e40bee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53222/0 (socket says 172.21.15.150:53222) 2024-09-17T04:49:45.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e2ffd640 1 -- 172.21.15.150:0/2443936561 learned_addr learned my addr 172.21.15.150:0/2443936561 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:45.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e37fe640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03d0003a60 0x7f03e40aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.252+0000 7f03e3fff640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e41e2dd0 0x7f03e4138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03e2ffd640 1 -- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03d0003a60 msgr2=0x7f03e40aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03e2ffd640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03d0003a60 0x7f03e40aded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03e2ffd640 1 -- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e41e2dd0 msgr2=0x7f03e4138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03e2ffd640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e41e2dd0 0x7f03e4138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03e2ffd640 1 -- 172.21.15.150:0/2443936561 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e41e3a70 con 0x7f03e41eec20 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03e3fff640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e41e2dd0 0x7f03e4138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03e2ffd640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03e41eec20 0x7f03e40bee90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f03cc010bc0 tx=0x7f03cc00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03dbfff640 1 -- 172.21.15.150:0/2443936561 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc012650 con 0x7f03e41eec20 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03dbfff640 1 -- 172.21.15.150:0/2443936561 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03cc0128e0 con 0x7f03e41eec20 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03e9c70640 1 -- 172.21.15.150:0/2443936561 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e40aeb90 con 0x7f03e41eec20 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03dbfff640 1 -- 172.21.15.150:0/2443936561 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc012d10 con 0x7f03e41eec20 2024-09-17T04:49:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03e9c70640 1 -- 172.21.15.150:0/2443936561 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e40ae410 con 0x7f03e41eec20 2024-09-17T04:49:45.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03e9c70640 1 -- 172.21.15.150:0/2443936561 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03e4158800 con 0x7f03e41eec20 2024-09-17T04:49:45.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03dbfff640 1 -- 172.21.15.150:0/2443936561 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f03cc01a020 con 0x7f03e41eec20 2024-09-17T04:49:45.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03dbfff640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03c4073200 0x7f03c4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:45.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03dbfff640 1 -- 172.21.15.150:0/2443936561 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f03cc0959e0 con 0x7f03e41eec20 2024-09-17T04:49:45.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.256+0000 7f03e37fe640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03c4073200 0x7f03c4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:45.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.260+0000 7f03e37fe640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03c4073200 0x7f03c4075710 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0016c0 tx=0x7f03dc019510 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:45.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.260+0000 7f03dbfff640 1 -- 172.21.15.150:0/2443936561 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f03cc02a020 con 0x7f03e41eec20 2024-09-17T04:49:45.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.456+0000 7f03e9c70640 1 -- 172.21.15.150:0/2443936561 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03e4157910 con 0x7f03e41eec20 2024-09-17T04:49:45.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03dbfff640 1 -- 172.21.15.150:0/2443936561 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f03e41e3a70 con 0x7f03e41eec20 2024-09-17T04:49:45.461 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:45.461 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:49:45.461 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:49:45.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03e9c70640 1 -- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03c4073200 msgr2=0x7f03c4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03e9c70640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03c4073200 0x7f03c4075710 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f03dc0016c0 tx=0x7f03dc019510 comp rx=0 tx=0).stop 2024-09-17T04:49:45.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03e9c70640 1 -- 172.21.15.150:0/2443936561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03e41eec20 msgr2=0x7f03e40bee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03e9c70640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03e41eec20 0x7f03e40bee90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f03cc010bc0 tx=0x7f03cc00e6c0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03e9c70640 1 -- 172.21.15.150:0/2443936561 shutdown_connections 2024-09-17T04:49:45.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03e9c70640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03c4073200 0x7f03c4075710 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03e9c70640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03e41e2dd0 0x7f03e4138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03e9c70640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03e41eec20 0x7f03e40bee90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03e9c70640 1 --2- 172.21.15.150:0/2443936561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03d0003a60 0x7f03e40aded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03e9c70640 1 -- 172.21.15.150:0/2443936561 >> 172.21.15.150:0/2443936561 conn(0x7f03e40abb00 msgr2=0x7f03e413c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:45.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03e9c70640 1 -- 172.21.15.150:0/2443936561 shutdown_connections 2024-09-17T04:49:45.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.460+0000 7f03e9c70640 1 -- 172.21.15.150:0/2443936561 wait complete. 2024-09-17T04:49:45.482 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:45.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.632+0000 7fd2d176b640 1 -- 172.21.15.150:0/61311307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc0c42e0 msgr2=0x7fd2cc0c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.632+0000 7fd2d176b640 1 --2- 172.21.15.150:0/61311307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc0c42e0 0x7fd2cc0c6690 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000aa10 tx=0x7fd2c001dc30 comp rx=0 tx=0).stop 2024-09-17T04:49:45.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.632+0000 7fd2d176b640 1 -- 172.21.15.150:0/61311307 shutdown_connections 2024-09-17T04:49:45.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.632+0000 7fd2d176b640 1 --2- 172.21.15.150:0/61311307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc0c42e0 0x7fd2cc0c6690 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.632+0000 7fd2d176b640 1 -- 172.21.15.150:0/61311307 >> 172.21.15.150:0/61311307 conn(0x7fd2cc0c4a00 msgr2=0x7fd2cc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:45.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.632+0000 7fd2d176b640 1 -- 172.21.15.150:0/61311307 shutdown_connections 2024-09-17T04:49:45.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.632+0000 7fd2d176b640 1 -- 172.21.15.150:0/61311307 wait complete. 2024-09-17T04:49:45.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.632+0000 7fd2d176b640 1 Processor -- start 2024-09-17T04:49:45.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.632+0000 7fd2d176b640 1 -- start start 2024-09-17T04:49:45.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2d176b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc0a7350 0x7fd2cc0a8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:45.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2d176b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2cc0c42e0 0x7fd2cc149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:45.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2d176b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2cc152cf0 0x7fd2cc147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:45.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2d176b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc0c6d50 con 0x7fd2cc0a7350 2024-09-17T04:49:45.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2d176b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc0c6bd0 con 0x7fd2cc152cf0 2024-09-17T04:49:45.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2d176b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd2cc150770 con 0x7fd2cc0c42e0 2024-09-17T04:49:45.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2caffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc0a7350 0x7fd2cc0a8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:45.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2caffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc0a7350 0x7fd2cc0a8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53234/0 (socket says 172.21.15.150:53234) 2024-09-17T04:49:45.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2caffd640 1 -- 172.21.15.150:0/218843595 learned_addr learned my addr 172.21.15.150:0/218843595 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:45.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2ca7fc640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2cc0c42e0 0x7fd2cc149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:45.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2cb7fe640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2cc152cf0 0x7fd2cc147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:45.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2caffd640 1 -- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2cc0c42e0 msgr2=0x7fd2cc149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2caffd640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2cc0c42e0 0x7fd2cc149990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2caffd640 1 -- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2cc152cf0 msgr2=0x7fd2cc147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2caffd640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2cc152cf0 0x7fd2cc147190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2caffd640 1 -- 172.21.15.150:0/218843595 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2cc0c1260 con 0x7fd2cc0a7350 2024-09-17T04:49:45.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2cb7fe640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2cc152cf0 0x7fd2cc147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:45.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2caffd640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc0a7350 0x7fd2cc0a8ff0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000a960 tx=0x7fd2c00a6960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:45.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2b37fe640 1 -- 172.21.15.150:0/218843595 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c00b3070 con 0x7fd2cc0a7350 2024-09-17T04:49:45.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2b37fe640 1 -- 172.21.15.150:0/218843595 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2c0003980 con 0x7fd2cc0a7350 2024-09-17T04:49:45.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2d176b640 1 -- 172.21.15.150:0/218843595 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2cc0a85b0 con 0x7fd2cc0a7350 2024-09-17T04:49:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2b37fe640 1 -- 172.21.15.150:0/218843595 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0003d90 con 0x7fd2cc0a7350 2024-09-17T04:49:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2d176b640 1 -- 172.21.15.150:0/218843595 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2cc148a00 con 0x7fd2cc0a7350 2024-09-17T04:49:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2d176b640 1 -- 172.21.15.150:0/218843595 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2cc0c6690 con 0x7fd2cc0a7350 2024-09-17T04:49:45.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2b37fe640 1 -- 172.21.15.150:0/218843595 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd2c00bd060 con 0x7fd2cc0a7350 2024-09-17T04:49:45.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2b37fe640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd2b8073200 0x7fd2b8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:45.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2b37fe640 1 -- 172.21.15.150:0/218843595 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fd2c012e310 con 0x7fd2cc0a7350 2024-09-17T04:49:45.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.636+0000 7fd2ca7fc640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd2b8073200 0x7fd2b8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:45.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.640+0000 7fd2ca7fc640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd2b8073200 0x7fd2b8075710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0a8ea0 tx=0x7fd2bc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:45.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.644+0000 7fd2b37fe640 1 -- 172.21.15.150:0/218843595 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd2c0017820 con 0x7fd2cc0a7350 2024-09-17T04:49:45.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.812+0000 7fd2d176b640 1 -- 172.21.15.150:0/218843595 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2cc0a7130 con 0x7fd2cc0a7350 2024-09-17T04:49:45.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2b37fe640 1 -- 172.21.15.150:0/218843595 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20431 (secure 0 0 0) 0x7fd2c00f9120 con 0x7fd2cc0a7350 2024-09-17T04:49:45.818 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:45.818 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:42.029799+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:45.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2d176b640 1 -- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd2b8073200 msgr2=0x7fd2b8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2d176b640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd2b8073200 0x7fd2b8075710 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0a8ea0 tx=0x7fd2bc009040 comp rx=0 tx=0).stop 2024-09-17T04:49:45.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2d176b640 1 -- 172.21.15.150:0/218843595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc0a7350 msgr2=0x7fd2cc0a8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2d176b640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc0a7350 0x7fd2cc0a8ff0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000a960 tx=0x7fd2c00a6960 comp rx=0 tx=0).stop 2024-09-17T04:49:45.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2d176b640 1 -- 172.21.15.150:0/218843595 shutdown_connections 2024-09-17T04:49:45.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2d176b640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd2b8073200 0x7fd2b8075710 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2d176b640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2cc152cf0 0x7fd2cc147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2d176b640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2cc0c42e0 0x7fd2cc149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2d176b640 1 --2- 172.21.15.150:0/218843595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2cc0a7350 0x7fd2cc0a8ff0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2d176b640 1 -- 172.21.15.150:0/218843595 >> 172.21.15.150:0/218843595 conn(0x7fd2cc0c4a00 msgr2=0x7fd2cc15de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:45.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2d176b640 1 -- 172.21.15.150:0/218843595 shutdown_connections 2024-09-17T04:49:45.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.816+0000 7fd2d176b640 1 -- 172.21.15.150:0/218843595 wait complete. 2024-09-17T04:49:45.840 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:49:45.841 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:49:45.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.992+0000 7f3317322640 1 -- 172.21.15.150:0/439432985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32fc003a40 msgr2=0x7f32fc003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.992+0000 7f3317322640 1 --2- 172.21.15.150:0/439432985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32fc003a40 0x7f32fc003eb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3308009ee0 tx=0x7f330801d610 comp rx=0 tx=0).stop 2024-09-17T04:49:45.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.992+0000 7f33158bf640 1 -- 172.21.15.150:0/439432985 reap_dead start 2024-09-17T04:49:45.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.992+0000 7f3317322640 1 -- 172.21.15.150:0/439432985 shutdown_connections 2024-09-17T04:49:45.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.992+0000 7f3317322640 1 -- 172.21.15.150:0/439432985 >> 172.21.15.150:0/439432985 conn(0x7f331006e360 msgr2=0x7f331006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:45.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.992+0000 7f3317322640 1 -- 172.21.15.150:0/439432985 shutdown_connections 2024-09-17T04:49:45.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.992+0000 7f3317322640 1 -- 172.21.15.150:0/439432985 wait complete. 2024-09-17T04:49:45.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.992+0000 7f3317322640 1 Processor -- start 2024-09-17T04:49:45.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.992+0000 7f3317322640 1 -- start start 2024-09-17T04:49:45.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3317322640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32fc003a40 0x7f331015da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:45.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3317322640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f331015df40 0x7f331015e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:45.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3317322640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f331015ee30 0x7f33101f0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:45.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3317322640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3310071120 con 0x7f32fc003a40 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3317322640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3310070fa0 con 0x7f331015ee30 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3317322640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3310072db0 con 0x7f331015df40 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f33150be640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32fc003a40 0x7f331015da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f33150be640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32fc003a40 0x7f331015da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53248/0 (socket says 172.21.15.150:53248) 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f33150be640 1 -- 172.21.15.150:0/2987538486 learned_addr learned my addr 172.21.15.150:0/2987538486 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f33148bd640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f331015df40 0x7f331015e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f33158bf640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f331015ee30 0x7f33101f0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f33150be640 1 -- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f331015df40 msgr2=0x7f331015e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f33150be640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f331015df40 0x7f331015e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f33150be640 1 -- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f331015ee30 msgr2=0x7f33101f0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f33150be640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f331015ee30 0x7f33101f0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f33150be640 1 -- 172.21.15.150:0/2987538486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33100bfa90 con 0x7f32fc003a40 2024-09-17T04:49:45.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f33158bf640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f331015ee30 0x7f33101f0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:46.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f33150be640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32fc003a40 0x7f331015da00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f33080ae040 tx=0x7f3308009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:46.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3305ffb640 1 -- 172.21.15.150:0/2987538486 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3308003b80 con 0x7f32fc003a40 2024-09-17T04:49:46.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3305ffb640 1 -- 172.21.15.150:0/2987538486 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3308003e10 con 0x7f32fc003a40 2024-09-17T04:49:46.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3317322640 1 -- 172.21.15.150:0/2987538486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33101ee480 con 0x7f32fc003a40 2024-09-17T04:49:46.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3305ffb640 1 -- 172.21.15.150:0/2987538486 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3308004290 con 0x7f32fc003a40 2024-09-17T04:49:46.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3317322640 1 -- 172.21.15.150:0/2987538486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33100c73b0 con 0x7f32fc003a40 2024-09-17T04:49:46.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3317322640 1 -- 172.21.15.150:0/2987538486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3310074920 con 0x7f32fc003a40 2024-09-17T04:49:46.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:45.996+0000 7f3305ffb640 1 -- 172.21.15.150:0/2987538486 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3308004430 con 0x7f32fc003a40 2024-09-17T04:49:46.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.000+0000 7f3305ffb640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f32f4073200 0x7f32f4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:46.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.000+0000 7f3305ffb640 1 -- 172.21.15.150:0/2987538486 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f3308130400 con 0x7f32fc003a40 2024-09-17T04:49:46.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.000+0000 7f33148bd640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f32f4073200 0x7f32f4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:46.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.000+0000 7f33148bd640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f32f4073200 0x7f32f4075710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f33100c6f30 tx=0x7f3300001560 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:46.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.004+0000 7f3305ffb640 1 -- 172.21.15.150:0/2987538486 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f33101ee480 con 0x7f32fc003a40 2024-09-17T04:49:46.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.164+0000 7f3317322640 1 -- 172.21.15.150:0/2987538486 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f33100be650 con 0x7f32f4073200 2024-09-17T04:49:46.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.208+0000 7f3305ffb640 1 -- 172.21.15.150:0/2987538486 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+485723 (secure 0 0 0) 0x7f33100be650 con 0x7f32f4073200 2024-09-17T04:49:46.214 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:46.215 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T04:49:46.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.212+0000 7f3317322640 1 -- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f32f4073200 msgr2=0x7f32f4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:46.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.212+0000 7f3317322640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f32f4073200 0x7f32f4075710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f33100c6f30 tx=0x7f3300001560 comp rx=0 tx=0).stop 2024-09-17T04:49:46.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.212+0000 7f3317322640 1 -- 172.21.15.150:0/2987538486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32fc003a40 msgr2=0x7f331015da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:46.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.212+0000 7f3317322640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32fc003a40 0x7f331015da00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f33080ae040 tx=0x7f3308009b30 comp rx=0 tx=0).stop 2024-09-17T04:49:46.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.212+0000 7f3317322640 1 -- 172.21.15.150:0/2987538486 shutdown_connections 2024-09-17T04:49:46.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.212+0000 7f3317322640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f32f4073200 0x7f32f4075710 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.212+0000 7f3317322640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f331015ee30 0x7f33101f0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.212+0000 7f3317322640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f331015df40 0x7f331015e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.212+0000 7f3317322640 1 --2- 172.21.15.150:0/2987538486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32fc003a40 0x7f331015da00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.212+0000 7f3317322640 1 -- 172.21.15.150:0/2987538486 >> 172.21.15.150:0/2987538486 conn(0x7f331006e360 msgr2=0x7f331006d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:46.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.212+0000 7f3317322640 1 -- 172.21.15.150:0/2987538486 shutdown_connections 2024-09-17T04:49:46.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.212+0000 7f3317322640 1 -- 172.21.15.150:0/2987538486 wait complete. 2024-09-17T04:49:46.232 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-09-17T04:49:44.946025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594390,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":155,"num_write_kb":1645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":237,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223572,"kb_used_data":3180,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751148,"statfs":{"total":773094113280,"available":772865175552,"internally_reserved":0,"allocated":3256320,"data_stored":1921332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.986753"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.985065+0000","last_change":"2024-09-17T04:49:39.985065+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.985065+0000","last_undegraded":"2024-09-17T04:49:39.985065+0000","last_fullsized":"2024-09-17T04:49:39.985065+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.988582+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T04:49:39.988582+0000","last_peered":"2024-09-17T04:49:39.988582+0000","last_clean":"2024-09-17T04:49:39.988582+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T04:49:39.988582+0000","last_undegraded":"2024-09-17T04:49:39.988582+0000","last_fullsized":"2024-09-17T04:49:39.988582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018012+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T04:49:40.018012+0000","last_peered":"2024-09-17T04:49:40.018012+0000","last_clean":"2024-09-17T04:49:40.018012+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T04:49:40.018012+0000","last_undegraded":"2024-09-17T04:49:40.018012+0000","last_fullsized":"2024-09-17T04:49:40.018012+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.015771+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T04:49:40.015771+0000","last_peered":"2024-09-17T04:49:40.015771+0000","last_clean":"2024-09-17T04:49:40.015771+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T04:49:40.015771+0000","last_undegraded":"2024-09-17T04:49:40.015771+0000","last_fullsized":"2024-09-17T04:49:40.015771+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:05.898867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.986079+0000","last_change":"2024-09-17T04:49:39.986079+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.986079+0000","last_undegraded":"2024-09-17T04:49:39.986079+0000","last_fullsized":"2024-09-17T04:49:39.986079+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018560+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T04:49:40.018560+0000","last_peered":"2024-09-17T04:49:40.018560+0000","last_clean":"2024-09-17T04:49:40.018560+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T04:49:40.018560+0000","last_undegraded":"2024-09-17T04:49:40.018560+0000","last_fullsized":"2024-09-17T04:49:40.018560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.020356+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T04:49:40.020356+0000","last_peered":"2024-09-17T04:49:40.020356+0000","last_clean":"2024-09-17T04:49:40.020356+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T04:49:40.020356+0000","last_undegraded":"2024-09-17T04:49:40.020356+0000","last_fullsized":"2024-09-17T04:49:40.020356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.008050+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T04:49:40.008050+0000","last_peered":"2024-09-17T04:49:40.008050+0000","last_clean":"2024-09-17T04:49:40.008050+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T04:49:40.008050+0000","last_undegraded":"2024-09-17T04:49:40.008050+0000","last_fullsized":"2024-09-17T04:49:40.008050+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:04.923610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978452+0000","last_change":"2024-09-17T04:49:39.978452+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978452+0000","last_undegraded":"2024-09-17T04:49:39.978452+0000","last_fullsized":"2024-09-17T04:49:39.978452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991377+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T04:49:39.991377+0000","last_peered":"2024-09-17T04:49:39.991377+0000","last_clean":"2024-09-17T04:49:39.991377+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T04:49:39.991377+0000","last_undegraded":"2024-09-17T04:49:39.991377+0000","last_fullsized":"2024-09-17T04:49:39.991377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.983748+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T04:49:39.983748+0000","last_peered":"2024-09-17T04:49:39.983748+0000","last_clean":"2024-09-17T04:49:39.983748+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T04:49:39.983748+0000","last_undegraded":"2024-09-17T04:49:39.983748+0000","last_fullsized":"2024-09-17T04:49:39.983748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.019964+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T04:49:40.019964+0000","last_peered":"2024-09-17T04:49:40.019964+0000","last_clean":"2024-09-17T04:49:40.019964+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T04:49:40.019964+0000","last_undegraded":"2024-09-17T04:49:40.019964+0000","last_fullsized":"2024-09-17T04:49:40.019964+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:09.540046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.982521+0000","last_change":"2024-09-17T04:49:39.982521+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.982521+0000","last_undegraded":"2024-09-17T04:49:39.982521+0000","last_fullsized":"2024-09-17T04:49:39.982521+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991897+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T04:49:39.991897+0000","last_peered":"2024-09-17T04:49:39.991897+0000","last_clean":"2024-09-17T04:49:39.991897+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T04:49:39.991897+0000","last_undegraded":"2024-09-17T04:49:39.991897+0000","last_fullsized":"2024-09-17T04:49:39.991897+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.972255+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T04:49:39.972255+0000","last_peered":"2024-09-17T04:49:39.972255+0000","last_clean":"2024-09-17T04:49:39.972255+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T04:49:39.972255+0000","last_undegraded":"2024-09-17T04:49:39.972255+0000","last_fullsized":"2024-09-17T04:49:39.972255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.008116+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T04:49:40.008116+0000","last_peered":"2024-09-17T04:49:40.008116+0000","last_clean":"2024-09-17T04:49:40.008116+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T04:49:40.008116+0000","last_undegraded":"2024-09-17T04:49:40.008116+0000","last_fullsized":"2024-09-17T04:49:40.008116+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:13.890297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.983148+0000","last_change":"2024-09-17T04:49:39.983148+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.983148+0000","last_undegraded":"2024-09-17T04:49:39.983148+0000","last_fullsized":"2024-09-17T04:49:39.983148+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.970984+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T04:49:39.970984+0000","last_peered":"2024-09-17T04:49:39.970984+0000","last_clean":"2024-09-17T04:49:39.970984+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T04:49:39.970984+0000","last_undegraded":"2024-09-17T04:49:39.970984+0000","last_fullsized":"2024-09-17T04:49:39.970984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971797+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T04:49:39.971797+0000","last_peered":"2024-09-17T04:49:39.971797+0000","last_clean":"2024-09-17T04:49:39.971797+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T04:49:39.971797+0000","last_undegraded":"2024-09-17T04:49:39.971797+0000","last_fullsized":"2024-09-17T04:49:39.971797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.996335+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T04:49:39.996335+0000","last_peered":"2024-09-17T04:49:39.996335+0000","last_clean":"2024-09-17T04:49:39.996335+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T04:49:39.996335+0000","last_undegraded":"2024-09-17T04:49:39.996335+0000","last_fullsized":"2024-09-17T04:49:39.996335+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:23:23.098901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.982950+0000","last_change":"2024-09-17T04:49:39.982950+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.982950+0000","last_undegraded":"2024-09-17T04:49:39.982950+0000","last_fullsized":"2024-09-17T04:49:39.982950+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987119+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T04:49:39.987119+0000","last_peered":"2024-09-17T04:49:39.987119+0000","last_clean":"2024-09-17T04:49:39.987119+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T04:49:39.987119+0000","last_undegraded":"2024-09-17T04:49:39.987119+0000","last_fullsized":"2024-09-17T04:49:39.987119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052248+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T04:49:42.052248+0000","last_peered":"2024-09-17T04:49:42.052248+0000","last_clean":"2024-09-17T04:49:42.052248+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T04:49:42.052248+0000","last_undegraded":"2024-09-17T04:49:42.052248+0000","last_fullsized":"2024-09-17T04:49:42.052248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.011662+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T04:49:40.011662+0000","last_peered":"2024-09-17T04:49:40.011662+0000","last_clean":"2024-09-17T04:49:40.011662+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T04:49:40.011662+0000","last_undegraded":"2024-09-17T04:49:40.011662+0000","last_fullsized":"2024-09-17T04:49:40.011662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:11:52.754063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.976214+0000","last_change":"2024-09-17T04:49:39.976214+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.976214+0000","last_undegraded":"2024-09-17T04:49:39.976214+0000","last_fullsized":"2024-09-17T04:49:39.976214+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.984466+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T04:49:39.984466+0000","last_peered":"2024-09-17T04:49:39.984466+0000","last_clean":"2024-09-17T04:49:39.984466+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T04:49:39.984466+0000","last_undegraded":"2024-09-17T04:49:39.984466+0000","last_fullsized":"2024-09-17T04:49:39.984466+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987576+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T04:49:39.987576+0000","last_peered":"2024-09-17T04:49:39.987576+0000","last_clean":"2024-09-17T04:49:39.987576+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T04:49:39.987576+0000","last_undegraded":"2024-09-17T04:49:39.987576+0000","last_fullsized":"2024-09-17T04:49:39.987576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.010631+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T04:49:40.010631+0000","last_peered":"2024-09-17T04:49:40.010631+0000","last_clean":"2024-09-17T04:49:40.010631+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T04:49:40.010631+0000","last_undegraded":"2024-09-17T04:49:40.010631+0000","last_fullsized":"2024-09-17T04:49:40.010631+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:54.939034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978320+0000","last_change":"2024-09-17T04:49:39.978320+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978320+0000","last_undegraded":"2024-09-17T04:49:39.978320+0000","last_fullsized":"2024-09-17T04:49:39.978320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.970864+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T04:49:39.970864+0000","last_peered":"2024-09-17T04:49:39.970864+0000","last_clean":"2024-09-17T04:49:39.970864+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T04:49:39.970864+0000","last_undegraded":"2024-09-17T04:49:39.970864+0000","last_fullsized":"2024-09-17T04:49:39.970864+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987169+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T04:49:39.987169+0000","last_peered":"2024-09-17T04:49:39.987169+0000","last_clean":"2024-09-17T04:49:39.987169+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T04:49:39.987169+0000","last_undegraded":"2024-09-17T04:49:39.987169+0000","last_fullsized":"2024-09-17T04:49:39.987169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.000386+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T04:49:40.000386+0000","last_peered":"2024-09-17T04:49:40.000386+0000","last_clean":"2024-09-17T04:49:40.000386+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T04:49:40.000386+0000","last_undegraded":"2024-09-17T04:49:40.000386+0000","last_fullsized":"2024-09-17T04:49:40.000386+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:48.926905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978035+0000","last_change":"2024-09-17T04:49:39.978035+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978035+0000","last_undegraded":"2024-09-17T04:49:39.978035+0000","last_fullsized":"2024-09-17T04:49:39.978035+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.986541+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T04:49:39.986541+0000","last_peered":"2024-09-17T04:49:39.986541+0000","last_clean":"2024-09-17T04:49:39.986541+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T04:49:39.986541+0000","last_undegraded":"2024-09-17T04:49:39.986541+0000","last_fullsized":"2024-09-17T04:49:39.986541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.972535+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T04:49:39.972535+0000","last_peered":"2024-09-17T04:49:39.972535+0000","last_clean":"2024-09-17T04:49:39.972535+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T04:49:39.972535+0000","last_undegraded":"2024-09-17T04:49:39.972535+0000","last_fullsized":"2024-09-17T04:49:39.972535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.998590+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T04:49:39.998590+0000","last_peered":"2024-09-17T04:49:39.998590+0000","last_clean":"2024-09-17T04:49:39.998590+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T04:49:39.998590+0000","last_undegraded":"2024-09-17T04:49:39.998590+0000","last_fullsized":"2024-09-17T04:49:39.998590+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:14.734836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.976199+0000","last_change":"2024-09-17T04:49:39.976199+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.976199+0000","last_undegraded":"2024-09-17T04:49:39.976199+0000","last_fullsized":"2024-09-17T04:49:39.976199+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.988283+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T04:49:39.988283+0000","last_peered":"2024-09-17T04:49:39.988283+0000","last_clean":"2024-09-17T04:49:39.988283+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T04:49:39.988283+0000","last_undegraded":"2024-09-17T04:49:39.988283+0000","last_fullsized":"2024-09-17T04:49:39.988283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.986846+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T04:49:39.986846+0000","last_peered":"2024-09-17T04:49:39.986846+0000","last_clean":"2024-09-17T04:49:39.986846+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T04:49:39.986846+0000","last_undegraded":"2024-09-17T04:49:39.986846+0000","last_fullsized":"2024-09-17T04:49:39.986846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.013654+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T04:49:40.013654+0000","last_peered":"2024-09-17T04:49:40.013654+0000","last_clean":"2024-09-17T04:49:40.013654+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T04:49:40.013654+0000","last_undegraded":"2024-09-17T04:49:40.013654+0000","last_fullsized":"2024-09-17T04:49:40.013654+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.980946+0000","last_change":"2024-09-17T04:49:39.980946+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.980946+0000","last_undegraded":"2024-09-17T04:49:39.980946+0000","last_fullsized":"2024-09-17T04:49:39.980946+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055299+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T04:49:42.055299+0000","last_peered":"2024-09-17T04:49:42.055299+0000","last_clean":"2024-09-17T04:49:42.055299+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T04:49:42.055299+0000","last_undegraded":"2024-09-17T04:49:42.055299+0000","last_fullsized":"2024-09-17T04:49:42.055299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.972605+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T04:49:39.972605+0000","last_peered":"2024-09-17T04:49:39.972605+0000","last_clean":"2024-09-17T04:49:39.972605+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T04:49:39.972605+0000","last_undegraded":"2024-09-17T04:49:39.972605+0000","last_fullsized":"2024-09-17T04:49:39.972605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053169+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T04:49:42.053169+0000","last_peered":"2024-09-17T04:49:42.053169+0000","last_clean":"2024-09-17T04:49:42.053169+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T04:49:42.053169+0000","last_undegraded":"2024-09-17T04:49:42.053169+0000","last_fullsized":"2024-09-17T04:49:42.053169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.976674+0000","last_change":"2024-09-17T04:49:39.976674+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.976674+0000","last_undegraded":"2024-09-17T04:49:39.976674+0000","last_fullsized":"2024-09-17T04:49:39.976674+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985149+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T04:49:39.985149+0000","last_peered":"2024-09-17T04:49:39.985149+0000","last_clean":"2024-09-17T04:49:39.985149+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T04:49:39.985149+0000","last_undegraded":"2024-09-17T04:49:39.985149+0000","last_fullsized":"2024-09-17T04:49:39.985149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018285+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T04:49:40.018285+0000","last_peered":"2024-09-17T04:49:40.018285+0000","last_clean":"2024-09-17T04:49:40.018285+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T04:49:40.018285+0000","last_undegraded":"2024-09-17T04:49:40.018285+0000","last_fullsized":"2024-09-17T04:49:40.018285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050191+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T04:49:42.050191+0000","last_peered":"2024-09-17T04:49:42.050191+0000","last_clean":"2024-09-17T04:49:42.050191+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T04:49:42.050191+0000","last_undegraded":"2024-09-17T04:49:42.050191+0000","last_fullsized":"2024-09-17T04:49:42.050191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.980131+0000","last_change":"2024-09-17T04:49:39.980131+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.980131+0000","last_undegraded":"2024-09-17T04:49:39.980131+0000","last_fullsized":"2024-09-17T04:49:39.980131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991475+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T04:49:39.991475+0000","last_peered":"2024-09-17T04:49:39.991475+0000","last_clean":"2024-09-17T04:49:39.991475+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T04:49:39.991475+0000","last_undegraded":"2024-09-17T04:49:39.991475+0000","last_fullsized":"2024-09-17T04:49:39.991475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971397+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T04:49:39.971397+0000","last_peered":"2024-09-17T04:49:39.971397+0000","last_clean":"2024-09-17T04:49:39.971397+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T04:49:39.971397+0000","last_undegraded":"2024-09-17T04:49:39.971397+0000","last_fullsized":"2024-09-17T04:49:39.971397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054900+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T04:49:42.054900+0000","last_peered":"2024-09-17T04:49:42.054900+0000","last_clean":"2024-09-17T04:49:42.054900+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T04:49:42.054900+0000","last_undegraded":"2024-09-17T04:49:42.054900+0000","last_fullsized":"2024-09-17T04:49:42.054900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.980753+0000","last_change":"2024-09-17T04:49:39.980753+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.980753+0000","last_undegraded":"2024-09-17T04:49:39.980753+0000","last_fullsized":"2024-09-17T04:49:39.980753+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.010825+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T04:49:40.010825+0000","last_peered":"2024-09-17T04:49:40.010825+0000","last_clean":"2024-09-17T04:49:40.010825+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T04:49:40.010825+0000","last_undegraded":"2024-09-17T04:49:40.010825+0000","last_fullsized":"2024-09-17T04:49:40.010825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.989072+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T04:49:39.989072+0000","last_peered":"2024-09-17T04:49:39.989072+0000","last_clean":"2024-09-17T04:49:39.989072+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T04:49:39.989072+0000","last_undegraded":"2024-09-17T04:49:39.989072+0000","last_fullsized":"2024-09-17T04:49:39.989072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054921+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T04:49:42.054921+0000","last_peered":"2024-09-17T04:49:42.054921+0000","last_clean":"2024-09-17T04:49:42.054921+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T04:49:42.054921+0000","last_undegraded":"2024-09-17T04:49:42.054921+0000","last_fullsized":"2024-09-17T04:49:42.054921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.976361+0000","last_change":"2024-09-17T04:49:39.976361+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.976361+0000","last_undegraded":"2024-09-17T04:49:39.976361+0000","last_fullsized":"2024-09-17T04:49:39.976361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050299+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T04:49:42.050299+0000","last_peered":"2024-09-17T04:49:42.050299+0000","last_clean":"2024-09-17T04:49:42.050299+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T04:49:42.050299+0000","last_undegraded":"2024-09-17T04:49:42.050299+0000","last_fullsized":"2024-09-17T04:49:42.050299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971260+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T04:49:39.971260+0000","last_peered":"2024-09-17T04:49:39.971260+0000","last_clean":"2024-09-17T04:49:39.971260+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T04:49:39.971260+0000","last_undegraded":"2024-09-17T04:49:39.971260+0000","last_fullsized":"2024-09-17T04:49:39.971260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.022656+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T04:49:40.022656+0000","last_peered":"2024-09-17T04:49:40.022656+0000","last_clean":"2024-09-17T04:49:40.022656+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T04:49:40.022656+0000","last_undegraded":"2024-09-17T04:49:40.022656+0000","last_fullsized":"2024-09-17T04:49:40.022656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:40:52.141955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.987046+0000","last_change":"2024-09-17T04:49:39.987046+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.987046+0000","last_undegraded":"2024-09-17T04:49:39.987046+0000","last_fullsized":"2024-09-17T04:49:39.987046+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.977611+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T04:49:39.977611+0000","last_peered":"2024-09-17T04:49:39.977611+0000","last_clean":"2024-09-17T04:49:39.977611+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T04:49:39.977611+0000","last_undegraded":"2024-09-17T04:49:39.977611+0000","last_fullsized":"2024-09-17T04:49:39.977611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971993+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T04:49:39.971993+0000","last_peered":"2024-09-17T04:49:39.971993+0000","last_clean":"2024-09-17T04:49:39.971993+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T04:49:39.971993+0000","last_undegraded":"2024-09-17T04:49:39.971993+0000","last_fullsized":"2024-09-17T04:49:39.971993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":115,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985677+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T04:49:39.985677+0000","last_peered":"2024-09-17T04:49:39.985677+0000","last_clean":"2024-09-17T04:49:39.985677+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T04:49:39.985677+0000","last_undegraded":"2024-09-17T04:49:39.985677+0000","last_fullsized":"2024-09-17T04:49:39.985677+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.000144+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T04:49:40.000144+0000","last_peered":"2024-09-17T04:49:40.000144+0000","last_clean":"2024-09-17T04:49:40.000144+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T04:49:40.000144+0000","last_undegraded":"2024-09-17T04:49:40.000144+0000","last_fullsized":"2024-09-17T04:49:40.000144+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:09.621194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:40.007026+0000","last_change":"2024-09-17T04:49:40.007026+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:40.007026+0000","last_undegraded":"2024-09-17T04:49:40.007026+0000","last_fullsized":"2024-09-17T04:49:40.007026+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.972380+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T04:49:39.972380+0000","last_peered":"2024-09-17T04:49:39.972380+0000","last_clean":"2024-09-17T04:49:39.972380+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T04:49:39.972380+0000","last_undegraded":"2024-09-17T04:49:39.972380+0000","last_fullsized":"2024-09-17T04:49:39.972380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055292+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T04:49:42.055292+0000","last_peered":"2024-09-17T04:49:42.055292+0000","last_clean":"2024-09-17T04:49:42.055292+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T04:49:42.055292+0000","last_undegraded":"2024-09-17T04:49:42.055292+0000","last_fullsized":"2024-09-17T04:49:42.055292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.999080+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T04:49:39.999080+0000","last_peered":"2024-09-17T04:49:39.999080+0000","last_clean":"2024-09-17T04:49:39.999080+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T04:49:39.999080+0000","last_undegraded":"2024-09-17T04:49:39.999080+0000","last_fullsized":"2024-09-17T04:49:39.999080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:14:11.002522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050257+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T04:49:42.050257+0000","last_peered":"2024-09-17T04:49:42.050257+0000","last_clean":"2024-09-17T04:49:42.050257+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T04:49:42.050257+0000","last_undegraded":"2024-09-17T04:49:42.050257+0000","last_fullsized":"2024-09-17T04:49:42.050257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991821+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T04:49:39.991821+0000","last_peered":"2024-09-17T04:49:39.991821+0000","last_clean":"2024-09-17T04:49:39.991821+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T04:49:39.991821+0000","last_undegraded":"2024-09-17T04:49:39.991821+0000","last_fullsized":"2024-09-17T04:49:39.991821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018583+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T04:49:40.018583+0000","last_peered":"2024-09-17T04:49:40.018583+0000","last_clean":"2024-09-17T04:49:40.018583+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T04:49:40.018583+0000","last_undegraded":"2024-09-17T04:49:40.018583+0000","last_fullsized":"2024-09-17T04:49:40.018583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.023455+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T04:49:40.023455+0000","last_peered":"2024-09-17T04:49:40.023455+0000","last_clean":"2024-09-17T04:49:40.023455+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T04:49:40.023455+0000","last_undegraded":"2024-09-17T04:49:40.023455+0000","last_fullsized":"2024-09-17T04:49:40.023455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:59.283113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.983355+0000","last_change":"2024-09-17T04:49:39.983355+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.983355+0000","last_undegraded":"2024-09-17T04:49:39.983355+0000","last_fullsized":"2024-09-17T04:49:39.983355+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991055+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T04:49:39.991055+0000","last_peered":"2024-09-17T04:49:39.991055+0000","last_clean":"2024-09-17T04:49:39.991055+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T04:49:39.991055+0000","last_undegraded":"2024-09-17T04:49:39.991055+0000","last_fullsized":"2024-09-17T04:49:39.991055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.986635+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T04:49:39.986635+0000","last_peered":"2024-09-17T04:49:39.986635+0000","last_clean":"2024-09-17T04:49:39.986635+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T04:49:39.986635+0000","last_undegraded":"2024-09-17T04:49:39.986635+0000","last_fullsized":"2024-09-17T04:49:39.986635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.995394+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T04:49:39.995394+0000","last_peered":"2024-09-17T04:49:39.995394+0000","last_clean":"2024-09-17T04:49:39.995394+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T04:49:39.995394+0000","last_undegraded":"2024-09-17T04:49:39.995394+0000","last_fullsized":"2024-09-17T04:49:39.995394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:59.682768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048488+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T04:49:42.048488+0000","last_peered":"2024-09-17T04:49:42.048488+0000","last_clean":"2024-09-17T04:49:42.048488+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T04:49:42.048488+0000","last_undegraded":"2024-09-17T04:49:42.048488+0000","last_fullsized":"2024-09-17T04:49:42.048488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.986487+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T04:49:39.986487+0000","last_peered":"2024-09-17T04:49:39.986487+0000","last_clean":"2024-09-17T04:49:39.986487+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T04:49:39.986487+0000","last_undegraded":"2024-09-17T04:49:39.986487+0000","last_fullsized":"2024-09-17T04:49:39.986487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991208+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T04:49:39.991208+0000","last_peered":"2024-09-17T04:49:39.991208+0000","last_clean":"2024-09-17T04:49:39.991208+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T04:49:39.991208+0000","last_undegraded":"2024-09-17T04:49:39.991208+0000","last_fullsized":"2024-09-17T04:49:39.991208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.008066+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T04:49:40.008066+0000","last_peered":"2024-09-17T04:49:40.008066+0000","last_clean":"2024-09-17T04:49:40.008066+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T04:49:40.008066+0000","last_undegraded":"2024-09-17T04:49:40.008066+0000","last_fullsized":"2024-09-17T04:49:40.008066+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:05.898867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.984785+0000","last_change":"2024-09-17T04:49:39.984785+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.984785+0000","last_undegraded":"2024-09-17T04:49:39.984785+0000","last_fullsized":"2024-09-17T04:49:39.984785+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971014+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T04:49:39.971014+0000","last_peered":"2024-09-17T04:49:39.971014+0000","last_clean":"2024-09-17T04:49:39.971014+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T04:49:39.971014+0000","last_undegraded":"2024-09-17T04:49:39.971014+0000","last_fullsized":"2024-09-17T04:49:39.971014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.984798+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T04:49:39.984798+0000","last_peered":"2024-09-17T04:49:39.984798+0000","last_clean":"2024-09-17T04:49:39.984798+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T04:49:39.984798+0000","last_undegraded":"2024-09-17T04:49:39.984798+0000","last_fullsized":"2024-09-17T04:49:39.984798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.024158+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T04:49:40.024158+0000","last_peered":"2024-09-17T04:49:40.024158+0000","last_clean":"2024-09-17T04:49:40.024158+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T04:49:40.024158+0000","last_undegraded":"2024-09-17T04:49:40.024158+0000","last_fullsized":"2024-09-17T04:49:40.024158+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:47.755668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978813+0000","last_change":"2024-09-17T04:49:39.978813+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978813+0000","last_undegraded":"2024-09-17T04:49:39.978813+0000","last_fullsized":"2024-09-17T04:49:39.978813+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985495+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T04:49:39.985495+0000","last_peered":"2024-09-17T04:49:39.985495+0000","last_clean":"2024-09-17T04:49:39.985495+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T04:49:39.985495+0000","last_undegraded":"2024-09-17T04:49:39.985495+0000","last_fullsized":"2024-09-17T04:49:39.985495+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.990794+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T04:49:39.990794+0000","last_peered":"2024-09-17T04:49:39.990794+0000","last_clean":"2024-09-17T04:49:39.990794+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T04:49:39.990794+0000","last_undegraded":"2024-09-17T04:49:39.990794+0000","last_fullsized":"2024-09-17T04:49:39.990794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.023219+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T04:49:40.023219+0000","last_peered":"2024-09-17T04:49:40.023219+0000","last_clean":"2024-09-17T04:49:40.023219+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T04:49:40.023219+0000","last_undegraded":"2024-09-17T04:49:40.023219+0000","last_fullsized":"2024-09-17T04:49:40.023219+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:02:51.630455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.977289+0000","last_change":"2024-09-17T04:49:39.977289+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.977289+0000","last_undegraded":"2024-09-17T04:49:39.977289+0000","last_fullsized":"2024-09-17T04:49:39.977289+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971619+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T04:49:39.971619+0000","last_peered":"2024-09-17T04:49:39.971619+0000","last_clean":"2024-09-17T04:49:39.971619+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T04:49:39.971619+0000","last_undegraded":"2024-09-17T04:49:39.971619+0000","last_fullsized":"2024-09-17T04:49:39.971619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971315+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T04:49:39.971315+0000","last_peered":"2024-09-17T04:49:39.971315+0000","last_clean":"2024-09-17T04:49:39.971315+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T04:49:39.971315+0000","last_undegraded":"2024-09-17T04:49:39.971315+0000","last_fullsized":"2024-09-17T04:49:39.971315+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054288+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T04:49:42.054288+0000","last_peered":"2024-09-17T04:49:42.054288+0000","last_clean":"2024-09-17T04:49:42.054288+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T04:49:42.054288+0000","last_undegraded":"2024-09-17T04:49:42.054288+0000","last_fullsized":"2024-09-17T04:49:42.054288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.979905+0000","last_change":"2024-09-17T04:49:39.979905+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.979905+0000","last_undegraded":"2024-09-17T04:49:39.979905+0000","last_fullsized":"2024-09-17T04:49:39.979905+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051977+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T04:49:42.051977+0000","last_peered":"2024-09-17T04:49:42.051977+0000","last_clean":"2024-09-17T04:49:42.051977+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T04:49:42.051977+0000","last_undegraded":"2024-09-17T04:49:42.051977+0000","last_fullsized":"2024-09-17T04:49:42.051977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987237+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T04:49:39.987237+0000","last_peered":"2024-09-17T04:49:39.987237+0000","last_clean":"2024-09-17T04:49:39.987237+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T04:49:39.987237+0000","last_undegraded":"2024-09-17T04:49:39.987237+0000","last_fullsized":"2024-09-17T04:49:39.987237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.997661+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T04:49:39.997661+0000","last_peered":"2024-09-17T04:49:39.997661+0000","last_clean":"2024-09-17T04:49:39.997661+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T04:49:39.997661+0000","last_undegraded":"2024-09-17T04:49:39.997661+0000","last_fullsized":"2024-09-17T04:49:39.997661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:32.751253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.979302+0000","last_change":"2024-09-17T04:49:39.979302+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.979302+0000","last_undegraded":"2024-09-17T04:49:39.979302+0000","last_fullsized":"2024-09-17T04:49:39.979302+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991259+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T04:49:39.991259+0000","last_peered":"2024-09-17T04:49:39.991259+0000","last_clean":"2024-09-17T04:49:39.991259+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T04:49:39.991259+0000","last_undegraded":"2024-09-17T04:49:39.991259+0000","last_fullsized":"2024-09-17T04:49:39.991259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985784+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T04:49:39.985784+0000","last_peered":"2024-09-17T04:49:39.985784+0000","last_clean":"2024-09-17T04:49:39.985784+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T04:49:39.985784+0000","last_undegraded":"2024-09-17T04:49:39.985784+0000","last_fullsized":"2024-09-17T04:49:39.985784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.001601+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T04:49:40.001601+0000","last_peered":"2024-09-17T04:49:40.001601+0000","last_clean":"2024-09-17T04:49:40.001601+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T04:49:40.001601+0000","last_undegraded":"2024-09-17T04:49:40.001601+0000","last_fullsized":"2024-09-17T04:49:40.001601+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051373+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T04:49:42.051373+0000","last_peered":"2024-09-17T04:49:42.051373+0000","last_clean":"2024-09-17T04:49:42.051373+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T04:49:42.051373+0000","last_undegraded":"2024-09-17T04:49:42.051373+0000","last_fullsized":"2024-09-17T04:49:42.051373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971437+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T04:49:39.971437+0000","last_peered":"2024-09-17T04:49:39.971437+0000","last_clean":"2024-09-17T04:49:39.971437+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T04:49:39.971437+0000","last_undegraded":"2024-09-17T04:49:39.971437+0000","last_fullsized":"2024-09-17T04:49:39.971437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.988938+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T04:49:39.988938+0000","last_peered":"2024-09-17T04:49:39.988938+0000","last_clean":"2024-09-17T04:49:39.988938+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T04:49:39.988938+0000","last_undegraded":"2024-09-17T04:49:39.988938+0000","last_fullsized":"2024-09-17T04:49:39.988938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.998706+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T04:49:39.998706+0000","last_peered":"2024-09-17T04:49:39.998706+0000","last_clean":"2024-09-17T04:49:39.998706+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T04:49:39.998706+0000","last_undegraded":"2024-09-17T04:49:39.998706+0000","last_fullsized":"2024-09-17T04:49:39.998706+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:41:14.644133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.977218+0000","last_change":"2024-09-17T04:49:39.977218+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.977218+0000","last_undegraded":"2024-09-17T04:49:39.977218+0000","last_fullsized":"2024-09-17T04:49:39.977218+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.020525+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T04:49:40.020525+0000","last_peered":"2024-09-17T04:49:40.020525+0000","last_clean":"2024-09-17T04:49:40.020525+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T04:49:40.020525+0000","last_undegraded":"2024-09-17T04:49:40.020525+0000","last_fullsized":"2024-09-17T04:49:40.020525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.988040+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T04:49:39.988040+0000","last_peered":"2024-09-17T04:49:39.988040+0000","last_clean":"2024-09-17T04:49:39.988040+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T04:49:39.988040+0000","last_undegraded":"2024-09-17T04:49:39.988040+0000","last_fullsized":"2024-09-17T04:49:39.988040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.004496+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T04:49:40.004496+0000","last_peered":"2024-09-17T04:49:40.004496+0000","last_clean":"2024-09-17T04:49:40.004496+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T04:49:40.004496+0000","last_undegraded":"2024-09-17T04:49:40.004496+0000","last_fullsized":"2024-09-17T04:49:40.004496+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:32.066883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.982220+0000","last_change":"2024-09-17T04:49:39.982220+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.982220+0000","last_undegraded":"2024-09-17T04:49:39.982220+0000","last_fullsized":"2024-09-17T04:49:39.982220+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.986328+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T04:49:39.986328+0000","last_peered":"2024-09-17T04:49:39.986328+0000","last_clean":"2024-09-17T04:49:39.986328+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T04:49:39.986328+0000","last_undegraded":"2024-09-17T04:49:39.986328+0000","last_fullsized":"2024-09-17T04:49:39.986328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.984922+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T04:49:39.984922+0000","last_peered":"2024-09-17T04:49:39.984922+0000","last_clean":"2024-09-17T04:49:39.984922+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T04:49:39.984922+0000","last_undegraded":"2024-09-17T04:49:39.984922+0000","last_fullsized":"2024-09-17T04:49:39.984922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.013699+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T04:49:40.013699+0000","last_peered":"2024-09-17T04:49:40.013699+0000","last_clean":"2024-09-17T04:49:40.013699+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T04:49:40.013699+0000","last_undegraded":"2024-09-17T04:49:40.013699+0000","last_fullsized":"2024-09-17T04:49:40.013699+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978988+0000","last_change":"2024-09-17T04:49:39.978988+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978988+0000","last_undegraded":"2024-09-17T04:49:39.978988+0000","last_fullsized":"2024-09-17T04:49:39.978988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.013760+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T04:49:40.013760+0000","last_peered":"2024-09-17T04:49:40.013760+0000","last_clean":"2024-09-17T04:49:40.013760+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T04:49:40.013760+0000","last_undegraded":"2024-09-17T04:49:40.013760+0000","last_fullsized":"2024-09-17T04:49:40.013760+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987238+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T04:49:39.987238+0000","last_peered":"2024-09-17T04:49:39.987238+0000","last_clean":"2024-09-17T04:49:39.987238+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T04:49:39.987238+0000","last_undegraded":"2024-09-17T04:49:39.987238+0000","last_fullsized":"2024-09-17T04:49:39.987238+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971132+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T04:49:39.971132+0000","last_peered":"2024-09-17T04:49:39.971132+0000","last_clean":"2024-09-17T04:49:39.971132+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T04:49:39.971132+0000","last_undegraded":"2024-09-17T04:49:39.971132+0000","last_fullsized":"2024-09-17T04:49:39.971132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055365+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T04:49:42.055365+0000","last_peered":"2024-09-17T04:49:42.055365+0000","last_clean":"2024-09-17T04:49:42.055365+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T04:49:42.055365+0000","last_undegraded":"2024-09-17T04:49:42.055365+0000","last_fullsized":"2024-09-17T04:49:42.055365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.005637+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T04:49:40.005637+0000","last_peered":"2024-09-17T04:49:40.005637+0000","last_clean":"2024-09-17T04:49:40.005637+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T04:49:40.005637+0000","last_undegraded":"2024-09-17T04:49:40.005637+0000","last_fullsized":"2024-09-17T04:49:40.005637+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:38.789358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.011236+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T04:49:40.011236+0000","last_peered":"2024-09-17T04:49:40.011236+0000","last_clean":"2024-09-17T04:49:40.011236+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T04:49:40.011236+0000","last_undegraded":"2024-09-17T04:49:40.011236+0000","last_fullsized":"2024-09-17T04:49:40.011236+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050522+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T04:49:42.050522+0000","last_peered":"2024-09-17T04:49:42.050522+0000","last_clean":"2024-09-17T04:49:42.050522+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T04:49:42.050522+0000","last_undegraded":"2024-09-17T04:49:42.050522+0000","last_fullsized":"2024-09-17T04:49:42.050522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.976107+0000","last_change":"2024-09-17T04:49:39.976107+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.976107+0000","last_undegraded":"2024-09-17T04:49:39.976107+0000","last_fullsized":"2024-09-17T04:49:39.976107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971090+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T04:49:39.971090+0000","last_peered":"2024-09-17T04:49:39.971090+0000","last_clean":"2024-09-17T04:49:39.971090+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T04:49:39.971090+0000","last_undegraded":"2024-09-17T04:49:39.971090+0000","last_fullsized":"2024-09-17T04:49:39.971090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.984521+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T04:49:39.984521+0000","last_peered":"2024-09-17T04:49:39.984521+0000","last_clean":"2024-09-17T04:49:39.984521+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T04:49:39.984521+0000","last_undegraded":"2024-09-17T04:49:39.984521+0000","last_fullsized":"2024-09-17T04:49:39.984521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.015171+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T04:49:40.015171+0000","last_peered":"2024-09-17T04:49:40.015171+0000","last_clean":"2024-09-17T04:49:40.015171+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T04:49:40.015171+0000","last_undegraded":"2024-09-17T04:49:40.015171+0000","last_fullsized":"2024-09-17T04:49:40.015171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978682+0000","last_change":"2024-09-17T04:49:39.978682+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978682+0000","last_undegraded":"2024-09-17T04:49:39.978682+0000","last_fullsized":"2024-09-17T04:49:39.978682+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018754+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T04:49:40.018754+0000","last_peered":"2024-09-17T04:49:40.018754+0000","last_clean":"2024-09-17T04:49:40.018754+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T04:49:40.018754+0000","last_undegraded":"2024-09-17T04:49:40.018754+0000","last_fullsized":"2024-09-17T04:49:40.018754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.979076+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T04:49:39.979076+0000","last_peered":"2024-09-17T04:49:39.979076+0000","last_clean":"2024-09-17T04:49:39.979076+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T04:49:39.979076+0000","last_undegraded":"2024-09-17T04:49:39.979076+0000","last_fullsized":"2024-09-17T04:49:39.979076+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.020077+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T04:49:40.020077+0000","last_peered":"2024-09-17T04:49:40.020077+0000","last_clean":"2024-09-17T04:49:40.020077+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T04:49:40.020077+0000","last_undegraded":"2024-09-17T04:49:40.020077+0000","last_fullsized":"2024-09-17T04:49:40.020077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:14.699029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.985837+0000","last_change":"2024-09-17T04:49:39.985837+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.985837+0000","last_undegraded":"2024-09-17T04:49:39.985837+0000","last_fullsized":"2024-09-17T04:49:39.985837+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985295+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T04:49:39.985295+0000","last_peered":"2024-09-17T04:49:39.985295+0000","last_clean":"2024-09-17T04:49:39.985295+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T04:49:39.985295+0000","last_undegraded":"2024-09-17T04:49:39.985295+0000","last_fullsized":"2024-09-17T04:49:39.985295+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.975188+0000","last_change":"2024-09-17T04:49:39.975188+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.975188+0000","last_undegraded":"2024-09-17T04:49:39.975188+0000","last_fullsized":"2024-09-17T04:49:39.975188+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987738+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T04:49:39.987738+0000","last_peered":"2024-09-17T04:49:39.987738+0000","last_clean":"2024-09-17T04:49:39.987738+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T04:49:39.987738+0000","last_undegraded":"2024-09-17T04:49:39.987738+0000","last_fullsized":"2024-09-17T04:49:39.987738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.981350+0000","last_change":"2024-09-17T04:49:39.981350+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.981350+0000","last_undegraded":"2024-09-17T04:49:39.981350+0000","last_fullsized":"2024-09-17T04:49:39.981350+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.970914+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T04:49:39.970914+0000","last_peered":"2024-09-17T04:49:39.970914+0000","last_clean":"2024-09-17T04:49:39.970914+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T04:49:39.970914+0000","last_undegraded":"2024-09-17T04:49:39.970914+0000","last_fullsized":"2024-09-17T04:49:39.970914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.996610+0000","last_change":"2024-09-17T04:49:39.996610+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.996610+0000","last_undegraded":"2024-09-17T04:49:39.996610+0000","last_fullsized":"2024-09-17T04:49:39.996610+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018286+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T04:49:40.018286+0000","last_peered":"2024-09-17T04:49:40.018286+0000","last_clean":"2024-09-17T04:49:40.018286+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T04:49:40.018286+0000","last_undegraded":"2024-09-17T04:49:40.018286+0000","last_fullsized":"2024-09-17T04:49:40.018286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.987074+0000","last_change":"2024-09-17T04:49:39.987074+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.987074+0000","last_undegraded":"2024-09-17T04:49:39.987074+0000","last_fullsized":"2024-09-17T04:49:39.987074+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054935+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T04:49:42.054935+0000","last_peered":"2024-09-17T04:49:42.054935+0000","last_clean":"2024-09-17T04:49:42.054935+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T04:49:42.054935+0000","last_undegraded":"2024-09-17T04:49:42.054935+0000","last_fullsized":"2024-09-17T04:49:42.054935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.987846+0000","last_change":"2024-09-17T04:49:39.987846+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.987846+0000","last_undegraded":"2024-09-17T04:49:39.987846+0000","last_fullsized":"2024-09-17T04:49:39.987846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054811+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T04:49:42.054811+0000","last_peered":"2024-09-17T04:49:42.054811+0000","last_clean":"2024-09-17T04:49:42.054811+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T04:49:42.054811+0000","last_undegraded":"2024-09-17T04:49:42.054811+0000","last_fullsized":"2024-09-17T04:49:42.054811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.977503+0000","last_change":"2024-09-17T04:49:39.977503+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.977503+0000","last_undegraded":"2024-09-17T04:49:39.977503+0000","last_fullsized":"2024-09-17T04:49:39.977503+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.986168+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T04:49:39.986168+0000","last_peered":"2024-09-17T04:49:39.986168+0000","last_clean":"2024-09-17T04:49:39.986168+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T04:49:39.986168+0000","last_undegraded":"2024-09-17T04:49:39.986168+0000","last_fullsized":"2024-09-17T04:49:39.986168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.980932+0000","last_change":"2024-09-17T04:49:39.980932+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.980932+0000","last_undegraded":"2024-09-17T04:49:39.980932+0000","last_fullsized":"2024-09-17T04:49:39.980932+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018697+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T04:49:40.018697+0000","last_peered":"2024-09-17T04:49:40.018697+0000","last_clean":"2024-09-17T04:49:40.018697+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T04:49:40.018697+0000","last_undegraded":"2024-09-17T04:49:40.018697+0000","last_fullsized":"2024-09-17T04:49:40.018697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.982103+0000","last_change":"2024-09-17T04:49:39.982103+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.982103+0000","last_undegraded":"2024-09-17T04:49:39.982103+0000","last_fullsized":"2024-09-17T04:49:39.982103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.970983+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T04:49:39.970983+0000","last_peered":"2024-09-17T04:49:39.970983+0000","last_clean":"2024-09-17T04:49:39.970983+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T04:49:39.970983+0000","last_undegraded":"2024-09-17T04:49:39.970983+0000","last_fullsized":"2024-09-17T04:49:39.970983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.977416+0000","last_change":"2024-09-17T04:49:39.977416+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.977416+0000","last_undegraded":"2024-09-17T04:49:39.977416+0000","last_fullsized":"2024-09-17T04:49:39.977416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.016212+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T04:49:40.016212+0000","last_peered":"2024-09-17T04:49:40.016212+0000","last_clean":"2024-09-17T04:49:40.016212+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T04:49:40.016212+0000","last_undegraded":"2024-09-17T04:49:40.016212+0000","last_fullsized":"2024-09-17T04:49:40.016212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054699+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T04:49:42.054699+0000","last_peered":"2024-09-17T04:49:42.054699+0000","last_clean":"2024-09-17T04:49:42.054699+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T04:49:42.054699+0000","last_undegraded":"2024-09-17T04:49:42.054699+0000","last_fullsized":"2024-09-17T04:49:42.054699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.978927+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T04:49:39.978927+0000","last_peered":"2024-09-17T04:49:39.978927+0000","last_clean":"2024-09-17T04:49:39.978927+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T04:49:39.978927+0000","last_undegraded":"2024-09-17T04:49:39.978927+0000","last_fullsized":"2024-09-17T04:49:39.978927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.984280+0000","last_change":"2024-09-17T04:49:39.984280+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.984280+0000","last_undegraded":"2024-09-17T04:49:39.984280+0000","last_fullsized":"2024-09-17T04:49:39.984280+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.984182+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T04:49:39.984182+0000","last_peered":"2024-09-17T04:49:39.984182+0000","last_clean":"2024-09-17T04:49:39.984182+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T04:49:39.984182+0000","last_undegraded":"2024-09-17T04:49:39.984182+0000","last_fullsized":"2024-09-17T04:49:39.984182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.982084+0000","last_change":"2024-09-17T04:49:39.982084+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.982084+0000","last_undegraded":"2024-09-17T04:49:39.982084+0000","last_fullsized":"2024-09-17T04:49:39.982084+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991784+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T04:49:39.991784+0000","last_peered":"2024-09-17T04:49:39.991784+0000","last_clean":"2024-09-17T04:49:39.991784+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T04:49:39.991784+0000","last_undegraded":"2024-09-17T04:49:39.991784+0000","last_fullsized":"2024-09-17T04:49:39.991784+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.987090+0000","last_change":"2024-09-17T04:49:39.987090+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.987090+0000","last_undegraded":"2024-09-17T04:49:39.987090+0000","last_fullsized":"2024-09-17T04:49:39.987090+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.972492+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T04:49:39.972492+0000","last_peered":"2024-09-17T04:49:39.972492+0000","last_clean":"2024-09-17T04:49:39.972492+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T04:49:39.972492+0000","last_undegraded":"2024-09-17T04:49:39.972492+0000","last_fullsized":"2024-09-17T04:49:39.972492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.984293+0000","last_change":"2024-09-17T04:49:39.984293+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.984293+0000","last_undegraded":"2024-09-17T04:49:39.984293+0000","last_fullsized":"2024-09-17T04:49:39.984293+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.989187+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T04:49:39.989187+0000","last_peered":"2024-09-17T04:49:39.989187+0000","last_clean":"2024-09-17T04:49:39.989187+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T04:49:39.989187+0000","last_undegraded":"2024-09-17T04:49:39.989187+0000","last_fullsized":"2024-09-17T04:49:39.989187+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050316+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T04:49:42.050316+0000","last_peered":"2024-09-17T04:49:42.050316+0000","last_clean":"2024-09-17T04:49:42.050316+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T04:49:42.050316+0000","last_undegraded":"2024-09-17T04:49:42.050316+0000","last_fullsized":"2024-09-17T04:49:42.050316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.972470+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T04:49:39.972470+0000","last_peered":"2024-09-17T04:49:39.972470+0000","last_clean":"2024-09-17T04:49:39.972470+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T04:49:39.972470+0000","last_undegraded":"2024-09-17T04:49:39.972470+0000","last_fullsized":"2024-09-17T04:49:39.972470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.976385+0000","last_change":"2024-09-17T04:49:39.976385+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.976385+0000","last_undegraded":"2024-09-17T04:49:39.976385+0000","last_fullsized":"2024-09-17T04:49:39.976385+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.970750+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T04:49:39.970750+0000","last_peered":"2024-09-17T04:49:39.970750+0000","last_clean":"2024-09-17T04:49:39.970750+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T04:49:39.970750+0000","last_undegraded":"2024-09-17T04:49:39.970750+0000","last_fullsized":"2024-09-17T04:49:39.970750+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.979153+0000","last_change":"2024-09-17T04:49:39.979153+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.979153+0000","last_undegraded":"2024-09-17T04:49:39.979153+0000","last_fullsized":"2024-09-17T04:49:39.979153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.011778+0000","last_change":"2024-09-17T04:49:31.911036+0000","last_active":"2024-09-17T04:49:40.011778+0000","last_peered":"2024-09-17T04:49:40.011778+0000","last_clean":"2024-09-17T04:49:40.011778+0000","last_became_active":"2024-09-17T04:49:31.910285+0000","last_became_peered":"2024-09-17T04:49:31.910285+0000","last_unstale":"2024-09-17T04:49:40.011778+0000","last_undegraded":"2024-09-17T04:49:40.011778+0000","last_fullsized":"2024-09-17T04:49:40.011778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.979656+0000","last_change":"2024-09-17T04:49:39.979656+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.979656+0000","last_undegraded":"2024-09-17T04:49:39.979656+0000","last_fullsized":"2024-09-17T04:49:39.979656+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051528+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T04:49:42.051528+0000","last_peered":"2024-09-17T04:49:42.051528+0000","last_clean":"2024-09-17T04:49:42.051528+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T04:49:42.051528+0000","last_undegraded":"2024-09-17T04:49:42.051528+0000","last_fullsized":"2024-09-17T04:49:42.051528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.984690+0000","last_change":"2024-09-17T04:49:39.984690+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.984690+0000","last_undegraded":"2024-09-17T04:49:39.984690+0000","last_fullsized":"2024-09-17T04:49:39.984690+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053942+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T04:49:42.053942+0000","last_peered":"2024-09-17T04:49:42.053942+0000","last_clean":"2024-09-17T04:49:42.053942+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T04:49:42.053942+0000","last_undegraded":"2024-09-17T04:49:42.053942+0000","last_fullsized":"2024-09-17T04:49:42.053942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.975437+0000","last_change":"2024-09-17T04:49:39.975437+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.975437+0000","last_undegraded":"2024-09-17T04:49:39.975437+0000","last_fullsized":"2024-09-17T04:49:39.975437+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991687+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T04:49:39.991687+0000","last_peered":"2024-09-17T04:49:39.991687+0000","last_clean":"2024-09-17T04:49:39.991687+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T04:49:39.991687+0000","last_undegraded":"2024-09-17T04:49:39.991687+0000","last_fullsized":"2024-09-17T04:49:39.991687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053498+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T04:49:42.053498+0000","last_peered":"2024-09-17T04:49:42.053498+0000","last_clean":"2024-09-17T04:49:42.053498+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T04:49:42.053498+0000","last_undegraded":"2024-09-17T04:49:42.053498+0000","last_fullsized":"2024-09-17T04:49:42.053498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991111+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T04:49:39.991111+0000","last_peered":"2024-09-17T04:49:39.991111+0000","last_clean":"2024-09-17T04:49:39.991111+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T04:49:39.991111+0000","last_undegraded":"2024-09-17T04:49:39.991111+0000","last_fullsized":"2024-09-17T04:49:39.991111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.980619+0000","last_change":"2024-09-17T04:49:39.980619+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.980619+0000","last_undegraded":"2024-09-17T04:49:39.980619+0000","last_fullsized":"2024-09-17T04:49:39.980619+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987814+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T04:49:39.987814+0000","last_peered":"2024-09-17T04:49:39.987814+0000","last_clean":"2024-09-17T04:49:39.987814+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T04:49:39.987814+0000","last_undegraded":"2024-09-17T04:49:39.987814+0000","last_fullsized":"2024-09-17T04:49:39.987814+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.981542+0000","last_change":"2024-09-17T04:49:39.981542+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.981542+0000","last_undegraded":"2024-09-17T04:49:39.981542+0000","last_fullsized":"2024-09-17T04:49:39.981542+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.984808+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T04:49:39.984808+0000","last_peered":"2024-09-17T04:49:39.984808+0000","last_clean":"2024-09-17T04:49:39.984808+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T04:49:39.984808+0000","last_undegraded":"2024-09-17T04:49:39.984808+0000","last_fullsized":"2024-09-17T04:49:39.984808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:40.006583+0000","last_change":"2024-09-17T04:49:40.006583+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:40.006583+0000","last_undegraded":"2024-09-17T04:49:40.006583+0000","last_fullsized":"2024-09-17T04:49:40.006583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.010550+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T04:49:40.010550+0000","last_peered":"2024-09-17T04:49:40.010550+0000","last_clean":"2024-09-17T04:49:40.010550+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T04:49:40.010550+0000","last_undegraded":"2024-09-17T04:49:40.010550+0000","last_fullsized":"2024-09-17T04:49:40.010550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.979767+0000","last_change":"2024-09-17T04:49:39.979767+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.979767+0000","last_undegraded":"2024-09-17T04:49:39.979767+0000","last_fullsized":"2024-09-17T04:49:39.979767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.988053+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T04:49:39.988053+0000","last_peered":"2024-09-17T04:49:39.988053+0000","last_clean":"2024-09-17T04:49:39.988053+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T04:49:39.988053+0000","last_undegraded":"2024-09-17T04:49:39.988053+0000","last_fullsized":"2024-09-17T04:49:39.988053+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.984817+0000","last_change":"2024-09-17T04:49:39.984817+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.984817+0000","last_undegraded":"2024-09-17T04:49:39.984817+0000","last_fullsized":"2024-09-17T04:49:39.984817+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985677+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T04:49:39.985677+0000","last_peered":"2024-09-17T04:49:39.985677+0000","last_clean":"2024-09-17T04:49:39.985677+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T04:49:39.985677+0000","last_undegraded":"2024-09-17T04:49:39.985677+0000","last_fullsized":"2024-09-17T04:49:39.985677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.985016+0000","last_change":"2024-09-17T04:49:39.985016+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.985016+0000","last_undegraded":"2024-09-17T04:49:39.985016+0000","last_fullsized":"2024-09-17T04:49:39.985016+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991312+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T04:49:39.991312+0000","last_peered":"2024-09-17T04:49:39.991312+0000","last_clean":"2024-09-17T04:49:39.991312+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T04:49:39.991312+0000","last_undegraded":"2024-09-17T04:49:39.991312+0000","last_fullsized":"2024-09-17T04:49:39.991312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.981661+0000","last_change":"2024-09-17T04:49:39.981661+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.981661+0000","last_undegraded":"2024-09-17T04:49:39.981661+0000","last_fullsized":"2024-09-17T04:49:39.981661+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985688+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T04:49:39.985688+0000","last_peered":"2024-09-17T04:49:39.985688+0000","last_clean":"2024-09-17T04:49:39.985688+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T04:49:39.985688+0000","last_undegraded":"2024-09-17T04:49:39.985688+0000","last_fullsized":"2024-09-17T04:49:39.985688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:40.008239+0000","last_change":"2024-09-17T04:49:40.008239+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:40.008239+0000","last_undegraded":"2024-09-17T04:49:40.008239+0000","last_fullsized":"2024-09-17T04:49:40.008239+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018466+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T04:49:40.018466+0000","last_peered":"2024-09-17T04:49:40.018466+0000","last_clean":"2024-09-17T04:49:40.018466+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T04:49:40.018466+0000","last_undegraded":"2024-09-17T04:49:40.018466+0000","last_fullsized":"2024-09-17T04:49:40.018466+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978152+0000","last_change":"2024-09-17T04:49:39.978152+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978152+0000","last_undegraded":"2024-09-17T04:49:39.978152+0000","last_fullsized":"2024-09-17T04:49:39.978152+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.017691+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T04:49:40.017691+0000","last_peered":"2024-09-17T04:49:40.017691+0000","last_clean":"2024-09-17T04:49:40.017691+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T04:49:40.017691+0000","last_undegraded":"2024-09-17T04:49:40.017691+0000","last_fullsized":"2024-09-17T04:49:40.017691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.981911+0000","last_change":"2024-09-17T04:49:39.981911+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.981911+0000","last_undegraded":"2024-09-17T04:49:39.981911+0000","last_fullsized":"2024-09-17T04:49:39.981911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.990901+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T04:49:39.990901+0000","last_peered":"2024-09-17T04:49:39.990901+0000","last_clean":"2024-09-17T04:49:39.990901+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T04:49:39.990901+0000","last_undegraded":"2024-09-17T04:49:39.990901+0000","last_fullsized":"2024-09-17T04:49:39.990901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":90059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":847872,"data_stored":680395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":92219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":90517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":258048,"data_stored":91547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":843776,"data_stored":680373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":89991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:49:46.240 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:49:46.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 -- 172.21.15.150:0/2081961677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85b80c4b20 msgr2=0x7f85b8153bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:46.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 --2- 172.21.15.150:0/2081961677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85b80c4b20 0x7f85b8153bc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f85b0009bc0 tx=0x7f85b001cba0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 -- 172.21.15.150:0/2081961677 shutdown_connections 2024-09-17T04:49:46.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 --2- 172.21.15.150:0/2081961677 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85b80c6340 0x7f85b8157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 --2- 172.21.15.150:0/2081961677 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85b80c4b20 0x7f85b8153bc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 -- 172.21.15.150:0/2081961677 >> 172.21.15.150:0/2081961677 conn(0x7f85b8074750 msgr2=0x7f85b8074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:46.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 -- 172.21.15.150:0/2081961677 shutdown_connections 2024-09-17T04:49:46.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 -- 172.21.15.150:0/2081961677 wait complete. 2024-09-17T04:49:46.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 Processor -- start 2024-09-17T04:49:46.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 -- start start 2024-09-17T04:49:46.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85b80c4b20 0x7f85b80cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:46.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85b80c5df0 0x7f85b814c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:46.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85b80c6340 0x7f85b8149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:46.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f85b8163d10 con 0x7f85b80c4b20 2024-09-17T04:49:46.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f85b8163b90 con 0x7f85b80c5df0 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.416+0000 7f85be31b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f85b8163e90 con 0x7f85b80c6340 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85b80c4b20 0x7f85b80cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85b80c4b20 0x7f85b80cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53264/0 (socket says 172.21.15.150:53264) 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b7fff640 1 -- 172.21.15.150:0/3131158818 learned_addr learned my addr 172.21.15.150:0/3131158818 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b7fff640 1 -- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85b80c6340 msgr2=0x7f85b8149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85bc8b8640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85b80c6340 0x7f85b8149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b7fff640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85b80c6340 0x7f85b8149e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b7fff640 1 -- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85b80c5df0 msgr2=0x7f85b814c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b77fe640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85b80c5df0 0x7f85b814c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b7fff640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85b80c5df0 0x7f85b814c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b7fff640 1 -- 172.21.15.150:0/3131158818 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b8152e40 con 0x7f85b80c4b20 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b7fff640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85b80c4b20 0x7f85b80cc5f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f85a800c430 tx=0x7f85a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:46.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b4ff9640 1 -- 172.21.15.150:0/3131158818 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a8011790 con 0x7f85b80c4b20 2024-09-17T04:49:46.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b4ff9640 1 -- 172.21.15.150:0/3131158818 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85a8017070 con 0x7f85b80c4b20 2024-09-17T04:49:46.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85be31b640 1 -- 172.21.15.150:0/3131158818 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85b81f0ee0 con 0x7f85b80c4b20 2024-09-17T04:49:46.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b4ff9640 1 -- 172.21.15.150:0/3131158818 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a8011cc0 con 0x7f85b80c4b20 2024-09-17T04:49:46.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85be31b640 1 -- 172.21.15.150:0/3131158818 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85b814b360 con 0x7f85b80c4b20 2024-09-17T04:49:46.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85be31b640 1 -- 172.21.15.150:0/3131158818 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85b814ae70 con 0x7f85b80c4b20 2024-09-17T04:49:46.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b4ff9640 1 -- 172.21.15.150:0/3131158818 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f85a801c460 con 0x7f85b80c4b20 2024-09-17T04:49:46.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b4ff9640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85a4073200 0x7f85a4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:46.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b4ff9640 1 -- 172.21.15.150:0/3131158818 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f85a80942a0 con 0x7f85b80c4b20 2024-09-17T04:49:46.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.420+0000 7f85b77fe640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85a4073200 0x7f85a4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:46.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.424+0000 7f85b77fe640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85a4073200 0x7f85a4075710 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f85b8153330 tx=0x7f85ac005bf0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:46.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.424+0000 7f85b4ff9640 1 -- 172.21.15.150:0/3131158818 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f85b80b20f0 con 0x7f85b80c4b20 2024-09-17T04:49:46.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.588+0000 7f85be31b640 1 -- 172.21.15.150:0/3131158818 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f85b8153bc0 con 0x7f85a4073200 2024-09-17T04:49:46.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.612+0000 7f85b4ff9640 1 -- 172.21.15.150:0/3131158818 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+485723 (secure 0 0 0) 0x7f85b8153bc0 con 0x7f85a4073200 2024-09-17T04:49:46.615 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:46.620 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T04:49:46.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.616+0000 7f85be31b640 1 -- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85a4073200 msgr2=0x7f85a4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:46.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.616+0000 7f85be31b640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85a4073200 0x7f85a4075710 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f85b8153330 tx=0x7f85ac005bf0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.616+0000 7f85be31b640 1 -- 172.21.15.150:0/3131158818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85b80c4b20 msgr2=0x7f85b80cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:46.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.616+0000 7f85be31b640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85b80c4b20 0x7f85b80cc5f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f85a800c430 tx=0x7f85a800c8f0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.616+0000 7f85be31b640 1 -- 172.21.15.150:0/3131158818 shutdown_connections 2024-09-17T04:49:46.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.616+0000 7f85be31b640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85a4073200 0x7f85a4075710 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.616+0000 7f85be31b640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85b80c5df0 0x7f85b814c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.616+0000 7f85be31b640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85b80c6340 0x7f85b8149e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.616+0000 7f85be31b640 1 --2- 172.21.15.150:0/3131158818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85b80c4b20 0x7f85b80cc5f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.616+0000 7f85be31b640 1 -- 172.21.15.150:0/3131158818 >> 172.21.15.150:0/3131158818 conn(0x7f85b8074750 msgr2=0x7f85b80c7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:46.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.616+0000 7f85be31b640 1 -- 172.21.15.150:0/3131158818 shutdown_connections 2024-09-17T04:49:46.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.616+0000 7f85be31b640 1 -- 172.21.15.150:0/3131158818 wait complete. 2024-09-17T04:49:46.640 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-09-17T04:49:44.946025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594390,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":155,"num_write_kb":1645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":237,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223572,"kb_used_data":3180,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751148,"statfs":{"total":773094113280,"available":772865175552,"internally_reserved":0,"allocated":3256320,"data_stored":1921332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.986753"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.985065+0000","last_change":"2024-09-17T04:49:39.985065+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.985065+0000","last_undegraded":"2024-09-17T04:49:39.985065+0000","last_fullsized":"2024-09-17T04:49:39.985065+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.988582+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T04:49:39.988582+0000","last_peered":"2024-09-17T04:49:39.988582+0000","last_clean":"2024-09-17T04:49:39.988582+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T04:49:39.988582+0000","last_undegraded":"2024-09-17T04:49:39.988582+0000","last_fullsized":"2024-09-17T04:49:39.988582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018012+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T04:49:40.018012+0000","last_peered":"2024-09-17T04:49:40.018012+0000","last_clean":"2024-09-17T04:49:40.018012+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T04:49:40.018012+0000","last_undegraded":"2024-09-17T04:49:40.018012+0000","last_fullsized":"2024-09-17T04:49:40.018012+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.015771+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T04:49:40.015771+0000","last_peered":"2024-09-17T04:49:40.015771+0000","last_clean":"2024-09-17T04:49:40.015771+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T04:49:40.015771+0000","last_undegraded":"2024-09-17T04:49:40.015771+0000","last_fullsized":"2024-09-17T04:49:40.015771+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:05.898867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.986079+0000","last_change":"2024-09-17T04:49:39.986079+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.986079+0000","last_undegraded":"2024-09-17T04:49:39.986079+0000","last_fullsized":"2024-09-17T04:49:39.986079+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018560+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T04:49:40.018560+0000","last_peered":"2024-09-17T04:49:40.018560+0000","last_clean":"2024-09-17T04:49:40.018560+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T04:49:40.018560+0000","last_undegraded":"2024-09-17T04:49:40.018560+0000","last_fullsized":"2024-09-17T04:49:40.018560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.020356+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T04:49:40.020356+0000","last_peered":"2024-09-17T04:49:40.020356+0000","last_clean":"2024-09-17T04:49:40.020356+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T04:49:40.020356+0000","last_undegraded":"2024-09-17T04:49:40.020356+0000","last_fullsized":"2024-09-17T04:49:40.020356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.008050+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T04:49:40.008050+0000","last_peered":"2024-09-17T04:49:40.008050+0000","last_clean":"2024-09-17T04:49:40.008050+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T04:49:40.008050+0000","last_undegraded":"2024-09-17T04:49:40.008050+0000","last_fullsized":"2024-09-17T04:49:40.008050+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:04.923610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978452+0000","last_change":"2024-09-17T04:49:39.978452+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978452+0000","last_undegraded":"2024-09-17T04:49:39.978452+0000","last_fullsized":"2024-09-17T04:49:39.978452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991377+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T04:49:39.991377+0000","last_peered":"2024-09-17T04:49:39.991377+0000","last_clean":"2024-09-17T04:49:39.991377+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T04:49:39.991377+0000","last_undegraded":"2024-09-17T04:49:39.991377+0000","last_fullsized":"2024-09-17T04:49:39.991377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.983748+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T04:49:39.983748+0000","last_peered":"2024-09-17T04:49:39.983748+0000","last_clean":"2024-09-17T04:49:39.983748+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T04:49:39.983748+0000","last_undegraded":"2024-09-17T04:49:39.983748+0000","last_fullsized":"2024-09-17T04:49:39.983748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.019964+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T04:49:40.019964+0000","last_peered":"2024-09-17T04:49:40.019964+0000","last_clean":"2024-09-17T04:49:40.019964+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T04:49:40.019964+0000","last_undegraded":"2024-09-17T04:49:40.019964+0000","last_fullsized":"2024-09-17T04:49:40.019964+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:09.540046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.982521+0000","last_change":"2024-09-17T04:49:39.982521+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.982521+0000","last_undegraded":"2024-09-17T04:49:39.982521+0000","last_fullsized":"2024-09-17T04:49:39.982521+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991897+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T04:49:39.991897+0000","last_peered":"2024-09-17T04:49:39.991897+0000","last_clean":"2024-09-17T04:49:39.991897+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T04:49:39.991897+0000","last_undegraded":"2024-09-17T04:49:39.991897+0000","last_fullsized":"2024-09-17T04:49:39.991897+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.972255+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T04:49:39.972255+0000","last_peered":"2024-09-17T04:49:39.972255+0000","last_clean":"2024-09-17T04:49:39.972255+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T04:49:39.972255+0000","last_undegraded":"2024-09-17T04:49:39.972255+0000","last_fullsized":"2024-09-17T04:49:39.972255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.008116+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T04:49:40.008116+0000","last_peered":"2024-09-17T04:49:40.008116+0000","last_clean":"2024-09-17T04:49:40.008116+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T04:49:40.008116+0000","last_undegraded":"2024-09-17T04:49:40.008116+0000","last_fullsized":"2024-09-17T04:49:40.008116+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:13.890297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.983148+0000","last_change":"2024-09-17T04:49:39.983148+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.983148+0000","last_undegraded":"2024-09-17T04:49:39.983148+0000","last_fullsized":"2024-09-17T04:49:39.983148+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.970984+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T04:49:39.970984+0000","last_peered":"2024-09-17T04:49:39.970984+0000","last_clean":"2024-09-17T04:49:39.970984+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T04:49:39.970984+0000","last_undegraded":"2024-09-17T04:49:39.970984+0000","last_fullsized":"2024-09-17T04:49:39.970984+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971797+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T04:49:39.971797+0000","last_peered":"2024-09-17T04:49:39.971797+0000","last_clean":"2024-09-17T04:49:39.971797+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T04:49:39.971797+0000","last_undegraded":"2024-09-17T04:49:39.971797+0000","last_fullsized":"2024-09-17T04:49:39.971797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.996335+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T04:49:39.996335+0000","last_peered":"2024-09-17T04:49:39.996335+0000","last_clean":"2024-09-17T04:49:39.996335+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T04:49:39.996335+0000","last_undegraded":"2024-09-17T04:49:39.996335+0000","last_fullsized":"2024-09-17T04:49:39.996335+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:23:23.098901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.982950+0000","last_change":"2024-09-17T04:49:39.982950+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.982950+0000","last_undegraded":"2024-09-17T04:49:39.982950+0000","last_fullsized":"2024-09-17T04:49:39.982950+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987119+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T04:49:39.987119+0000","last_peered":"2024-09-17T04:49:39.987119+0000","last_clean":"2024-09-17T04:49:39.987119+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T04:49:39.987119+0000","last_undegraded":"2024-09-17T04:49:39.987119+0000","last_fullsized":"2024-09-17T04:49:39.987119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052248+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T04:49:42.052248+0000","last_peered":"2024-09-17T04:49:42.052248+0000","last_clean":"2024-09-17T04:49:42.052248+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T04:49:42.052248+0000","last_undegraded":"2024-09-17T04:49:42.052248+0000","last_fullsized":"2024-09-17T04:49:42.052248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.011662+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T04:49:40.011662+0000","last_peered":"2024-09-17T04:49:40.011662+0000","last_clean":"2024-09-17T04:49:40.011662+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T04:49:40.011662+0000","last_undegraded":"2024-09-17T04:49:40.011662+0000","last_fullsized":"2024-09-17T04:49:40.011662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:11:52.754063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.976214+0000","last_change":"2024-09-17T04:49:39.976214+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.976214+0000","last_undegraded":"2024-09-17T04:49:39.976214+0000","last_fullsized":"2024-09-17T04:49:39.976214+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.984466+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T04:49:39.984466+0000","last_peered":"2024-09-17T04:49:39.984466+0000","last_clean":"2024-09-17T04:49:39.984466+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T04:49:39.984466+0000","last_undegraded":"2024-09-17T04:49:39.984466+0000","last_fullsized":"2024-09-17T04:49:39.984466+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987576+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T04:49:39.987576+0000","last_peered":"2024-09-17T04:49:39.987576+0000","last_clean":"2024-09-17T04:49:39.987576+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T04:49:39.987576+0000","last_undegraded":"2024-09-17T04:49:39.987576+0000","last_fullsized":"2024-09-17T04:49:39.987576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.010631+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T04:49:40.010631+0000","last_peered":"2024-09-17T04:49:40.010631+0000","last_clean":"2024-09-17T04:49:40.010631+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T04:49:40.010631+0000","last_undegraded":"2024-09-17T04:49:40.010631+0000","last_fullsized":"2024-09-17T04:49:40.010631+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:54.939034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978320+0000","last_change":"2024-09-17T04:49:39.978320+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978320+0000","last_undegraded":"2024-09-17T04:49:39.978320+0000","last_fullsized":"2024-09-17T04:49:39.978320+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.970864+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T04:49:39.970864+0000","last_peered":"2024-09-17T04:49:39.970864+0000","last_clean":"2024-09-17T04:49:39.970864+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T04:49:39.970864+0000","last_undegraded":"2024-09-17T04:49:39.970864+0000","last_fullsized":"2024-09-17T04:49:39.970864+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987169+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T04:49:39.987169+0000","last_peered":"2024-09-17T04:49:39.987169+0000","last_clean":"2024-09-17T04:49:39.987169+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T04:49:39.987169+0000","last_undegraded":"2024-09-17T04:49:39.987169+0000","last_fullsized":"2024-09-17T04:49:39.987169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.000386+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T04:49:40.000386+0000","last_peered":"2024-09-17T04:49:40.000386+0000","last_clean":"2024-09-17T04:49:40.000386+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T04:49:40.000386+0000","last_undegraded":"2024-09-17T04:49:40.000386+0000","last_fullsized":"2024-09-17T04:49:40.000386+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:48.926905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978035+0000","last_change":"2024-09-17T04:49:39.978035+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978035+0000","last_undegraded":"2024-09-17T04:49:39.978035+0000","last_fullsized":"2024-09-17T04:49:39.978035+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.986541+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T04:49:39.986541+0000","last_peered":"2024-09-17T04:49:39.986541+0000","last_clean":"2024-09-17T04:49:39.986541+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T04:49:39.986541+0000","last_undegraded":"2024-09-17T04:49:39.986541+0000","last_fullsized":"2024-09-17T04:49:39.986541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.972535+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T04:49:39.972535+0000","last_peered":"2024-09-17T04:49:39.972535+0000","last_clean":"2024-09-17T04:49:39.972535+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T04:49:39.972535+0000","last_undegraded":"2024-09-17T04:49:39.972535+0000","last_fullsized":"2024-09-17T04:49:39.972535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.998590+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T04:49:39.998590+0000","last_peered":"2024-09-17T04:49:39.998590+0000","last_clean":"2024-09-17T04:49:39.998590+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T04:49:39.998590+0000","last_undegraded":"2024-09-17T04:49:39.998590+0000","last_fullsized":"2024-09-17T04:49:39.998590+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:14.734836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.976199+0000","last_change":"2024-09-17T04:49:39.976199+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.976199+0000","last_undegraded":"2024-09-17T04:49:39.976199+0000","last_fullsized":"2024-09-17T04:49:39.976199+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.988283+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T04:49:39.988283+0000","last_peered":"2024-09-17T04:49:39.988283+0000","last_clean":"2024-09-17T04:49:39.988283+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T04:49:39.988283+0000","last_undegraded":"2024-09-17T04:49:39.988283+0000","last_fullsized":"2024-09-17T04:49:39.988283+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.986846+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T04:49:39.986846+0000","last_peered":"2024-09-17T04:49:39.986846+0000","last_clean":"2024-09-17T04:49:39.986846+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T04:49:39.986846+0000","last_undegraded":"2024-09-17T04:49:39.986846+0000","last_fullsized":"2024-09-17T04:49:39.986846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.013654+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T04:49:40.013654+0000","last_peered":"2024-09-17T04:49:40.013654+0000","last_clean":"2024-09-17T04:49:40.013654+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T04:49:40.013654+0000","last_undegraded":"2024-09-17T04:49:40.013654+0000","last_fullsized":"2024-09-17T04:49:40.013654+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.980946+0000","last_change":"2024-09-17T04:49:39.980946+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.980946+0000","last_undegraded":"2024-09-17T04:49:39.980946+0000","last_fullsized":"2024-09-17T04:49:39.980946+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055299+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T04:49:42.055299+0000","last_peered":"2024-09-17T04:49:42.055299+0000","last_clean":"2024-09-17T04:49:42.055299+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T04:49:42.055299+0000","last_undegraded":"2024-09-17T04:49:42.055299+0000","last_fullsized":"2024-09-17T04:49:42.055299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.972605+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T04:49:39.972605+0000","last_peered":"2024-09-17T04:49:39.972605+0000","last_clean":"2024-09-17T04:49:39.972605+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T04:49:39.972605+0000","last_undegraded":"2024-09-17T04:49:39.972605+0000","last_fullsized":"2024-09-17T04:49:39.972605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053169+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T04:49:42.053169+0000","last_peered":"2024-09-17T04:49:42.053169+0000","last_clean":"2024-09-17T04:49:42.053169+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T04:49:42.053169+0000","last_undegraded":"2024-09-17T04:49:42.053169+0000","last_fullsized":"2024-09-17T04:49:42.053169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.976674+0000","last_change":"2024-09-17T04:49:39.976674+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.976674+0000","last_undegraded":"2024-09-17T04:49:39.976674+0000","last_fullsized":"2024-09-17T04:49:39.976674+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985149+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T04:49:39.985149+0000","last_peered":"2024-09-17T04:49:39.985149+0000","last_clean":"2024-09-17T04:49:39.985149+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T04:49:39.985149+0000","last_undegraded":"2024-09-17T04:49:39.985149+0000","last_fullsized":"2024-09-17T04:49:39.985149+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018285+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T04:49:40.018285+0000","last_peered":"2024-09-17T04:49:40.018285+0000","last_clean":"2024-09-17T04:49:40.018285+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T04:49:40.018285+0000","last_undegraded":"2024-09-17T04:49:40.018285+0000","last_fullsized":"2024-09-17T04:49:40.018285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050191+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T04:49:42.050191+0000","last_peered":"2024-09-17T04:49:42.050191+0000","last_clean":"2024-09-17T04:49:42.050191+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T04:49:42.050191+0000","last_undegraded":"2024-09-17T04:49:42.050191+0000","last_fullsized":"2024-09-17T04:49:42.050191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.980131+0000","last_change":"2024-09-17T04:49:39.980131+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.980131+0000","last_undegraded":"2024-09-17T04:49:39.980131+0000","last_fullsized":"2024-09-17T04:49:39.980131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991475+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T04:49:39.991475+0000","last_peered":"2024-09-17T04:49:39.991475+0000","last_clean":"2024-09-17T04:49:39.991475+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T04:49:39.991475+0000","last_undegraded":"2024-09-17T04:49:39.991475+0000","last_fullsized":"2024-09-17T04:49:39.991475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971397+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T04:49:39.971397+0000","last_peered":"2024-09-17T04:49:39.971397+0000","last_clean":"2024-09-17T04:49:39.971397+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T04:49:39.971397+0000","last_undegraded":"2024-09-17T04:49:39.971397+0000","last_fullsized":"2024-09-17T04:49:39.971397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054900+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T04:49:42.054900+0000","last_peered":"2024-09-17T04:49:42.054900+0000","last_clean":"2024-09-17T04:49:42.054900+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T04:49:42.054900+0000","last_undegraded":"2024-09-17T04:49:42.054900+0000","last_fullsized":"2024-09-17T04:49:42.054900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.980753+0000","last_change":"2024-09-17T04:49:39.980753+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.980753+0000","last_undegraded":"2024-09-17T04:49:39.980753+0000","last_fullsized":"2024-09-17T04:49:39.980753+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.010825+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T04:49:40.010825+0000","last_peered":"2024-09-17T04:49:40.010825+0000","last_clean":"2024-09-17T04:49:40.010825+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T04:49:40.010825+0000","last_undegraded":"2024-09-17T04:49:40.010825+0000","last_fullsized":"2024-09-17T04:49:40.010825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.989072+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T04:49:39.989072+0000","last_peered":"2024-09-17T04:49:39.989072+0000","last_clean":"2024-09-17T04:49:39.989072+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T04:49:39.989072+0000","last_undegraded":"2024-09-17T04:49:39.989072+0000","last_fullsized":"2024-09-17T04:49:39.989072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054921+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T04:49:42.054921+0000","last_peered":"2024-09-17T04:49:42.054921+0000","last_clean":"2024-09-17T04:49:42.054921+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T04:49:42.054921+0000","last_undegraded":"2024-09-17T04:49:42.054921+0000","last_fullsized":"2024-09-17T04:49:42.054921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.976361+0000","last_change":"2024-09-17T04:49:39.976361+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.976361+0000","last_undegraded":"2024-09-17T04:49:39.976361+0000","last_fullsized":"2024-09-17T04:49:39.976361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050299+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T04:49:42.050299+0000","last_peered":"2024-09-17T04:49:42.050299+0000","last_clean":"2024-09-17T04:49:42.050299+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T04:49:42.050299+0000","last_undegraded":"2024-09-17T04:49:42.050299+0000","last_fullsized":"2024-09-17T04:49:42.050299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971260+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T04:49:39.971260+0000","last_peered":"2024-09-17T04:49:39.971260+0000","last_clean":"2024-09-17T04:49:39.971260+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T04:49:39.971260+0000","last_undegraded":"2024-09-17T04:49:39.971260+0000","last_fullsized":"2024-09-17T04:49:39.971260+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.022656+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T04:49:40.022656+0000","last_peered":"2024-09-17T04:49:40.022656+0000","last_clean":"2024-09-17T04:49:40.022656+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T04:49:40.022656+0000","last_undegraded":"2024-09-17T04:49:40.022656+0000","last_fullsized":"2024-09-17T04:49:40.022656+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:40:52.141955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.987046+0000","last_change":"2024-09-17T04:49:39.987046+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.987046+0000","last_undegraded":"2024-09-17T04:49:39.987046+0000","last_fullsized":"2024-09-17T04:49:39.987046+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.977611+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T04:49:39.977611+0000","last_peered":"2024-09-17T04:49:39.977611+0000","last_clean":"2024-09-17T04:49:39.977611+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T04:49:39.977611+0000","last_undegraded":"2024-09-17T04:49:39.977611+0000","last_fullsized":"2024-09-17T04:49:39.977611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971993+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T04:49:39.971993+0000","last_peered":"2024-09-17T04:49:39.971993+0000","last_clean":"2024-09-17T04:49:39.971993+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T04:49:39.971993+0000","last_undegraded":"2024-09-17T04:49:39.971993+0000","last_fullsized":"2024-09-17T04:49:39.971993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":115,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985677+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T04:49:39.985677+0000","last_peered":"2024-09-17T04:49:39.985677+0000","last_clean":"2024-09-17T04:49:39.985677+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T04:49:39.985677+0000","last_undegraded":"2024-09-17T04:49:39.985677+0000","last_fullsized":"2024-09-17T04:49:39.985677+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.000144+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T04:49:40.000144+0000","last_peered":"2024-09-17T04:49:40.000144+0000","last_clean":"2024-09-17T04:49:40.000144+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T04:49:40.000144+0000","last_undegraded":"2024-09-17T04:49:40.000144+0000","last_fullsized":"2024-09-17T04:49:40.000144+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:09.621194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:40.007026+0000","last_change":"2024-09-17T04:49:40.007026+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:40.007026+0000","last_undegraded":"2024-09-17T04:49:40.007026+0000","last_fullsized":"2024-09-17T04:49:40.007026+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.972380+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T04:49:39.972380+0000","last_peered":"2024-09-17T04:49:39.972380+0000","last_clean":"2024-09-17T04:49:39.972380+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T04:49:39.972380+0000","last_undegraded":"2024-09-17T04:49:39.972380+0000","last_fullsized":"2024-09-17T04:49:39.972380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055292+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T04:49:42.055292+0000","last_peered":"2024-09-17T04:49:42.055292+0000","last_clean":"2024-09-17T04:49:42.055292+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T04:49:42.055292+0000","last_undegraded":"2024-09-17T04:49:42.055292+0000","last_fullsized":"2024-09-17T04:49:42.055292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.999080+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T04:49:39.999080+0000","last_peered":"2024-09-17T04:49:39.999080+0000","last_clean":"2024-09-17T04:49:39.999080+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T04:49:39.999080+0000","last_undegraded":"2024-09-17T04:49:39.999080+0000","last_fullsized":"2024-09-17T04:49:39.999080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:14:11.002522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050257+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T04:49:42.050257+0000","last_peered":"2024-09-17T04:49:42.050257+0000","last_clean":"2024-09-17T04:49:42.050257+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T04:49:42.050257+0000","last_undegraded":"2024-09-17T04:49:42.050257+0000","last_fullsized":"2024-09-17T04:49:42.050257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991821+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T04:49:39.991821+0000","last_peered":"2024-09-17T04:49:39.991821+0000","last_clean":"2024-09-17T04:49:39.991821+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T04:49:39.991821+0000","last_undegraded":"2024-09-17T04:49:39.991821+0000","last_fullsized":"2024-09-17T04:49:39.991821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018583+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T04:49:40.018583+0000","last_peered":"2024-09-17T04:49:40.018583+0000","last_clean":"2024-09-17T04:49:40.018583+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T04:49:40.018583+0000","last_undegraded":"2024-09-17T04:49:40.018583+0000","last_fullsized":"2024-09-17T04:49:40.018583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.023455+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T04:49:40.023455+0000","last_peered":"2024-09-17T04:49:40.023455+0000","last_clean":"2024-09-17T04:49:40.023455+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T04:49:40.023455+0000","last_undegraded":"2024-09-17T04:49:40.023455+0000","last_fullsized":"2024-09-17T04:49:40.023455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:59.283113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.983355+0000","last_change":"2024-09-17T04:49:39.983355+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.983355+0000","last_undegraded":"2024-09-17T04:49:39.983355+0000","last_fullsized":"2024-09-17T04:49:39.983355+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991055+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T04:49:39.991055+0000","last_peered":"2024-09-17T04:49:39.991055+0000","last_clean":"2024-09-17T04:49:39.991055+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T04:49:39.991055+0000","last_undegraded":"2024-09-17T04:49:39.991055+0000","last_fullsized":"2024-09-17T04:49:39.991055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.986635+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T04:49:39.986635+0000","last_peered":"2024-09-17T04:49:39.986635+0000","last_clean":"2024-09-17T04:49:39.986635+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T04:49:39.986635+0000","last_undegraded":"2024-09-17T04:49:39.986635+0000","last_fullsized":"2024-09-17T04:49:39.986635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.995394+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T04:49:39.995394+0000","last_peered":"2024-09-17T04:49:39.995394+0000","last_clean":"2024-09-17T04:49:39.995394+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T04:49:39.995394+0000","last_undegraded":"2024-09-17T04:49:39.995394+0000","last_fullsized":"2024-09-17T04:49:39.995394+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:59.682768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048488+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T04:49:42.048488+0000","last_peered":"2024-09-17T04:49:42.048488+0000","last_clean":"2024-09-17T04:49:42.048488+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T04:49:42.048488+0000","last_undegraded":"2024-09-17T04:49:42.048488+0000","last_fullsized":"2024-09-17T04:49:42.048488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.986487+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T04:49:39.986487+0000","last_peered":"2024-09-17T04:49:39.986487+0000","last_clean":"2024-09-17T04:49:39.986487+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T04:49:39.986487+0000","last_undegraded":"2024-09-17T04:49:39.986487+0000","last_fullsized":"2024-09-17T04:49:39.986487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991208+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T04:49:39.991208+0000","last_peered":"2024-09-17T04:49:39.991208+0000","last_clean":"2024-09-17T04:49:39.991208+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T04:49:39.991208+0000","last_undegraded":"2024-09-17T04:49:39.991208+0000","last_fullsized":"2024-09-17T04:49:39.991208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.008066+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T04:49:40.008066+0000","last_peered":"2024-09-17T04:49:40.008066+0000","last_clean":"2024-09-17T04:49:40.008066+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T04:49:40.008066+0000","last_undegraded":"2024-09-17T04:49:40.008066+0000","last_fullsized":"2024-09-17T04:49:40.008066+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:05.898867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.984785+0000","last_change":"2024-09-17T04:49:39.984785+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.984785+0000","last_undegraded":"2024-09-17T04:49:39.984785+0000","last_fullsized":"2024-09-17T04:49:39.984785+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971014+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T04:49:39.971014+0000","last_peered":"2024-09-17T04:49:39.971014+0000","last_clean":"2024-09-17T04:49:39.971014+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T04:49:39.971014+0000","last_undegraded":"2024-09-17T04:49:39.971014+0000","last_fullsized":"2024-09-17T04:49:39.971014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.984798+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T04:49:39.984798+0000","last_peered":"2024-09-17T04:49:39.984798+0000","last_clean":"2024-09-17T04:49:39.984798+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T04:49:39.984798+0000","last_undegraded":"2024-09-17T04:49:39.984798+0000","last_fullsized":"2024-09-17T04:49:39.984798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.024158+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T04:49:40.024158+0000","last_peered":"2024-09-17T04:49:40.024158+0000","last_clean":"2024-09-17T04:49:40.024158+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T04:49:40.024158+0000","last_undegraded":"2024-09-17T04:49:40.024158+0000","last_fullsized":"2024-09-17T04:49:40.024158+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:47.755668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978813+0000","last_change":"2024-09-17T04:49:39.978813+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978813+0000","last_undegraded":"2024-09-17T04:49:39.978813+0000","last_fullsized":"2024-09-17T04:49:39.978813+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985495+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T04:49:39.985495+0000","last_peered":"2024-09-17T04:49:39.985495+0000","last_clean":"2024-09-17T04:49:39.985495+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T04:49:39.985495+0000","last_undegraded":"2024-09-17T04:49:39.985495+0000","last_fullsized":"2024-09-17T04:49:39.985495+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.990794+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T04:49:39.990794+0000","last_peered":"2024-09-17T04:49:39.990794+0000","last_clean":"2024-09-17T04:49:39.990794+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T04:49:39.990794+0000","last_undegraded":"2024-09-17T04:49:39.990794+0000","last_fullsized":"2024-09-17T04:49:39.990794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.023219+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T04:49:40.023219+0000","last_peered":"2024-09-17T04:49:40.023219+0000","last_clean":"2024-09-17T04:49:40.023219+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T04:49:40.023219+0000","last_undegraded":"2024-09-17T04:49:40.023219+0000","last_fullsized":"2024-09-17T04:49:40.023219+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:02:51.630455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.977289+0000","last_change":"2024-09-17T04:49:39.977289+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.977289+0000","last_undegraded":"2024-09-17T04:49:39.977289+0000","last_fullsized":"2024-09-17T04:49:39.977289+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971619+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T04:49:39.971619+0000","last_peered":"2024-09-17T04:49:39.971619+0000","last_clean":"2024-09-17T04:49:39.971619+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T04:49:39.971619+0000","last_undegraded":"2024-09-17T04:49:39.971619+0000","last_fullsized":"2024-09-17T04:49:39.971619+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971315+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T04:49:39.971315+0000","last_peered":"2024-09-17T04:49:39.971315+0000","last_clean":"2024-09-17T04:49:39.971315+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T04:49:39.971315+0000","last_undegraded":"2024-09-17T04:49:39.971315+0000","last_fullsized":"2024-09-17T04:49:39.971315+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054288+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T04:49:42.054288+0000","last_peered":"2024-09-17T04:49:42.054288+0000","last_clean":"2024-09-17T04:49:42.054288+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T04:49:42.054288+0000","last_undegraded":"2024-09-17T04:49:42.054288+0000","last_fullsized":"2024-09-17T04:49:42.054288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.979905+0000","last_change":"2024-09-17T04:49:39.979905+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.979905+0000","last_undegraded":"2024-09-17T04:49:39.979905+0000","last_fullsized":"2024-09-17T04:49:39.979905+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051977+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T04:49:42.051977+0000","last_peered":"2024-09-17T04:49:42.051977+0000","last_clean":"2024-09-17T04:49:42.051977+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T04:49:42.051977+0000","last_undegraded":"2024-09-17T04:49:42.051977+0000","last_fullsized":"2024-09-17T04:49:42.051977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987237+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T04:49:39.987237+0000","last_peered":"2024-09-17T04:49:39.987237+0000","last_clean":"2024-09-17T04:49:39.987237+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T04:49:39.987237+0000","last_undegraded":"2024-09-17T04:49:39.987237+0000","last_fullsized":"2024-09-17T04:49:39.987237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.997661+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T04:49:39.997661+0000","last_peered":"2024-09-17T04:49:39.997661+0000","last_clean":"2024-09-17T04:49:39.997661+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T04:49:39.997661+0000","last_undegraded":"2024-09-17T04:49:39.997661+0000","last_fullsized":"2024-09-17T04:49:39.997661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:32.751253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.979302+0000","last_change":"2024-09-17T04:49:39.979302+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.979302+0000","last_undegraded":"2024-09-17T04:49:39.979302+0000","last_fullsized":"2024-09-17T04:49:39.979302+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991259+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T04:49:39.991259+0000","last_peered":"2024-09-17T04:49:39.991259+0000","last_clean":"2024-09-17T04:49:39.991259+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T04:49:39.991259+0000","last_undegraded":"2024-09-17T04:49:39.991259+0000","last_fullsized":"2024-09-17T04:49:39.991259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985784+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T04:49:39.985784+0000","last_peered":"2024-09-17T04:49:39.985784+0000","last_clean":"2024-09-17T04:49:39.985784+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T04:49:39.985784+0000","last_undegraded":"2024-09-17T04:49:39.985784+0000","last_fullsized":"2024-09-17T04:49:39.985784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.001601+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T04:49:40.001601+0000","last_peered":"2024-09-17T04:49:40.001601+0000","last_clean":"2024-09-17T04:49:40.001601+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T04:49:40.001601+0000","last_undegraded":"2024-09-17T04:49:40.001601+0000","last_fullsized":"2024-09-17T04:49:40.001601+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051373+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T04:49:42.051373+0000","last_peered":"2024-09-17T04:49:42.051373+0000","last_clean":"2024-09-17T04:49:42.051373+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T04:49:42.051373+0000","last_undegraded":"2024-09-17T04:49:42.051373+0000","last_fullsized":"2024-09-17T04:49:42.051373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971437+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T04:49:39.971437+0000","last_peered":"2024-09-17T04:49:39.971437+0000","last_clean":"2024-09-17T04:49:39.971437+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T04:49:39.971437+0000","last_undegraded":"2024-09-17T04:49:39.971437+0000","last_fullsized":"2024-09-17T04:49:39.971437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.988938+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T04:49:39.988938+0000","last_peered":"2024-09-17T04:49:39.988938+0000","last_clean":"2024-09-17T04:49:39.988938+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T04:49:39.988938+0000","last_undegraded":"2024-09-17T04:49:39.988938+0000","last_fullsized":"2024-09-17T04:49:39.988938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.998706+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T04:49:39.998706+0000","last_peered":"2024-09-17T04:49:39.998706+0000","last_clean":"2024-09-17T04:49:39.998706+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T04:49:39.998706+0000","last_undegraded":"2024-09-17T04:49:39.998706+0000","last_fullsized":"2024-09-17T04:49:39.998706+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:41:14.644133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.977218+0000","last_change":"2024-09-17T04:49:39.977218+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.977218+0000","last_undegraded":"2024-09-17T04:49:39.977218+0000","last_fullsized":"2024-09-17T04:49:39.977218+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.020525+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T04:49:40.020525+0000","last_peered":"2024-09-17T04:49:40.020525+0000","last_clean":"2024-09-17T04:49:40.020525+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T04:49:40.020525+0000","last_undegraded":"2024-09-17T04:49:40.020525+0000","last_fullsized":"2024-09-17T04:49:40.020525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.988040+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T04:49:39.988040+0000","last_peered":"2024-09-17T04:49:39.988040+0000","last_clean":"2024-09-17T04:49:39.988040+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T04:49:39.988040+0000","last_undegraded":"2024-09-17T04:49:39.988040+0000","last_fullsized":"2024-09-17T04:49:39.988040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.004496+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T04:49:40.004496+0000","last_peered":"2024-09-17T04:49:40.004496+0000","last_clean":"2024-09-17T04:49:40.004496+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T04:49:40.004496+0000","last_undegraded":"2024-09-17T04:49:40.004496+0000","last_fullsized":"2024-09-17T04:49:40.004496+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:32.066883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.982220+0000","last_change":"2024-09-17T04:49:39.982220+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.982220+0000","last_undegraded":"2024-09-17T04:49:39.982220+0000","last_fullsized":"2024-09-17T04:49:39.982220+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.986328+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T04:49:39.986328+0000","last_peered":"2024-09-17T04:49:39.986328+0000","last_clean":"2024-09-17T04:49:39.986328+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T04:49:39.986328+0000","last_undegraded":"2024-09-17T04:49:39.986328+0000","last_fullsized":"2024-09-17T04:49:39.986328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.984922+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T04:49:39.984922+0000","last_peered":"2024-09-17T04:49:39.984922+0000","last_clean":"2024-09-17T04:49:39.984922+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T04:49:39.984922+0000","last_undegraded":"2024-09-17T04:49:39.984922+0000","last_fullsized":"2024-09-17T04:49:39.984922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.013699+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T04:49:40.013699+0000","last_peered":"2024-09-17T04:49:40.013699+0000","last_clean":"2024-09-17T04:49:40.013699+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T04:49:40.013699+0000","last_undegraded":"2024-09-17T04:49:40.013699+0000","last_fullsized":"2024-09-17T04:49:40.013699+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978988+0000","last_change":"2024-09-17T04:49:39.978988+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978988+0000","last_undegraded":"2024-09-17T04:49:39.978988+0000","last_fullsized":"2024-09-17T04:49:39.978988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.013760+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T04:49:40.013760+0000","last_peered":"2024-09-17T04:49:40.013760+0000","last_clean":"2024-09-17T04:49:40.013760+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T04:49:40.013760+0000","last_undegraded":"2024-09-17T04:49:40.013760+0000","last_fullsized":"2024-09-17T04:49:40.013760+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987238+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T04:49:39.987238+0000","last_peered":"2024-09-17T04:49:39.987238+0000","last_clean":"2024-09-17T04:49:39.987238+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T04:49:39.987238+0000","last_undegraded":"2024-09-17T04:49:39.987238+0000","last_fullsized":"2024-09-17T04:49:39.987238+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971132+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T04:49:39.971132+0000","last_peered":"2024-09-17T04:49:39.971132+0000","last_clean":"2024-09-17T04:49:39.971132+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T04:49:39.971132+0000","last_undegraded":"2024-09-17T04:49:39.971132+0000","last_fullsized":"2024-09-17T04:49:39.971132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055365+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T04:49:42.055365+0000","last_peered":"2024-09-17T04:49:42.055365+0000","last_clean":"2024-09-17T04:49:42.055365+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T04:49:42.055365+0000","last_undegraded":"2024-09-17T04:49:42.055365+0000","last_fullsized":"2024-09-17T04:49:42.055365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.005637+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T04:49:40.005637+0000","last_peered":"2024-09-17T04:49:40.005637+0000","last_clean":"2024-09-17T04:49:40.005637+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T04:49:40.005637+0000","last_undegraded":"2024-09-17T04:49:40.005637+0000","last_fullsized":"2024-09-17T04:49:40.005637+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:38.789358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.011236+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T04:49:40.011236+0000","last_peered":"2024-09-17T04:49:40.011236+0000","last_clean":"2024-09-17T04:49:40.011236+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T04:49:40.011236+0000","last_undegraded":"2024-09-17T04:49:40.011236+0000","last_fullsized":"2024-09-17T04:49:40.011236+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050522+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T04:49:42.050522+0000","last_peered":"2024-09-17T04:49:42.050522+0000","last_clean":"2024-09-17T04:49:42.050522+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T04:49:42.050522+0000","last_undegraded":"2024-09-17T04:49:42.050522+0000","last_fullsized":"2024-09-17T04:49:42.050522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.976107+0000","last_change":"2024-09-17T04:49:39.976107+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.976107+0000","last_undegraded":"2024-09-17T04:49:39.976107+0000","last_fullsized":"2024-09-17T04:49:39.976107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.971090+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T04:49:39.971090+0000","last_peered":"2024-09-17T04:49:39.971090+0000","last_clean":"2024-09-17T04:49:39.971090+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T04:49:39.971090+0000","last_undegraded":"2024-09-17T04:49:39.971090+0000","last_fullsized":"2024-09-17T04:49:39.971090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.984521+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T04:49:39.984521+0000","last_peered":"2024-09-17T04:49:39.984521+0000","last_clean":"2024-09-17T04:49:39.984521+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T04:49:39.984521+0000","last_undegraded":"2024-09-17T04:49:39.984521+0000","last_fullsized":"2024-09-17T04:49:39.984521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.015171+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T04:49:40.015171+0000","last_peered":"2024-09-17T04:49:40.015171+0000","last_clean":"2024-09-17T04:49:40.015171+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T04:49:40.015171+0000","last_undegraded":"2024-09-17T04:49:40.015171+0000","last_fullsized":"2024-09-17T04:49:40.015171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978682+0000","last_change":"2024-09-17T04:49:39.978682+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978682+0000","last_undegraded":"2024-09-17T04:49:39.978682+0000","last_fullsized":"2024-09-17T04:49:39.978682+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018754+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T04:49:40.018754+0000","last_peered":"2024-09-17T04:49:40.018754+0000","last_clean":"2024-09-17T04:49:40.018754+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T04:49:40.018754+0000","last_undegraded":"2024-09-17T04:49:40.018754+0000","last_fullsized":"2024-09-17T04:49:40.018754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.979076+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T04:49:39.979076+0000","last_peered":"2024-09-17T04:49:39.979076+0000","last_clean":"2024-09-17T04:49:39.979076+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T04:49:39.979076+0000","last_undegraded":"2024-09-17T04:49:39.979076+0000","last_fullsized":"2024-09-17T04:49:39.979076+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.020077+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T04:49:40.020077+0000","last_peered":"2024-09-17T04:49:40.020077+0000","last_clean":"2024-09-17T04:49:40.020077+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T04:49:40.020077+0000","last_undegraded":"2024-09-17T04:49:40.020077+0000","last_fullsized":"2024-09-17T04:49:40.020077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:14.699029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.985837+0000","last_change":"2024-09-17T04:49:39.985837+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.985837+0000","last_undegraded":"2024-09-17T04:49:39.985837+0000","last_fullsized":"2024-09-17T04:49:39.985837+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985295+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T04:49:39.985295+0000","last_peered":"2024-09-17T04:49:39.985295+0000","last_clean":"2024-09-17T04:49:39.985295+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T04:49:39.985295+0000","last_undegraded":"2024-09-17T04:49:39.985295+0000","last_fullsized":"2024-09-17T04:49:39.985295+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.975188+0000","last_change":"2024-09-17T04:49:39.975188+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.975188+0000","last_undegraded":"2024-09-17T04:49:39.975188+0000","last_fullsized":"2024-09-17T04:49:39.975188+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987738+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T04:49:39.987738+0000","last_peered":"2024-09-17T04:49:39.987738+0000","last_clean":"2024-09-17T04:49:39.987738+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T04:49:39.987738+0000","last_undegraded":"2024-09-17T04:49:39.987738+0000","last_fullsized":"2024-09-17T04:49:39.987738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.981350+0000","last_change":"2024-09-17T04:49:39.981350+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.981350+0000","last_undegraded":"2024-09-17T04:49:39.981350+0000","last_fullsized":"2024-09-17T04:49:39.981350+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.970914+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T04:49:39.970914+0000","last_peered":"2024-09-17T04:49:39.970914+0000","last_clean":"2024-09-17T04:49:39.970914+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T04:49:39.970914+0000","last_undegraded":"2024-09-17T04:49:39.970914+0000","last_fullsized":"2024-09-17T04:49:39.970914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.996610+0000","last_change":"2024-09-17T04:49:39.996610+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.996610+0000","last_undegraded":"2024-09-17T04:49:39.996610+0000","last_fullsized":"2024-09-17T04:49:39.996610+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018286+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T04:49:40.018286+0000","last_peered":"2024-09-17T04:49:40.018286+0000","last_clean":"2024-09-17T04:49:40.018286+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T04:49:40.018286+0000","last_undegraded":"2024-09-17T04:49:40.018286+0000","last_fullsized":"2024-09-17T04:49:40.018286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.987074+0000","last_change":"2024-09-17T04:49:39.987074+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.987074+0000","last_undegraded":"2024-09-17T04:49:39.987074+0000","last_fullsized":"2024-09-17T04:49:39.987074+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054935+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T04:49:42.054935+0000","last_peered":"2024-09-17T04:49:42.054935+0000","last_clean":"2024-09-17T04:49:42.054935+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T04:49:42.054935+0000","last_undegraded":"2024-09-17T04:49:42.054935+0000","last_fullsized":"2024-09-17T04:49:42.054935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.987846+0000","last_change":"2024-09-17T04:49:39.987846+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.987846+0000","last_undegraded":"2024-09-17T04:49:39.987846+0000","last_fullsized":"2024-09-17T04:49:39.987846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054811+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T04:49:42.054811+0000","last_peered":"2024-09-17T04:49:42.054811+0000","last_clean":"2024-09-17T04:49:42.054811+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T04:49:42.054811+0000","last_undegraded":"2024-09-17T04:49:42.054811+0000","last_fullsized":"2024-09-17T04:49:42.054811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.977503+0000","last_change":"2024-09-17T04:49:39.977503+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.977503+0000","last_undegraded":"2024-09-17T04:49:39.977503+0000","last_fullsized":"2024-09-17T04:49:39.977503+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.986168+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T04:49:39.986168+0000","last_peered":"2024-09-17T04:49:39.986168+0000","last_clean":"2024-09-17T04:49:39.986168+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T04:49:39.986168+0000","last_undegraded":"2024-09-17T04:49:39.986168+0000","last_fullsized":"2024-09-17T04:49:39.986168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.980932+0000","last_change":"2024-09-17T04:49:39.980932+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.980932+0000","last_undegraded":"2024-09-17T04:49:39.980932+0000","last_fullsized":"2024-09-17T04:49:39.980932+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018697+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T04:49:40.018697+0000","last_peered":"2024-09-17T04:49:40.018697+0000","last_clean":"2024-09-17T04:49:40.018697+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T04:49:40.018697+0000","last_undegraded":"2024-09-17T04:49:40.018697+0000","last_fullsized":"2024-09-17T04:49:40.018697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.982103+0000","last_change":"2024-09-17T04:49:39.982103+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.982103+0000","last_undegraded":"2024-09-17T04:49:39.982103+0000","last_fullsized":"2024-09-17T04:49:39.982103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.970983+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T04:49:39.970983+0000","last_peered":"2024-09-17T04:49:39.970983+0000","last_clean":"2024-09-17T04:49:39.970983+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T04:49:39.970983+0000","last_undegraded":"2024-09-17T04:49:39.970983+0000","last_fullsized":"2024-09-17T04:49:39.970983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.977416+0000","last_change":"2024-09-17T04:49:39.977416+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.977416+0000","last_undegraded":"2024-09-17T04:49:39.977416+0000","last_fullsized":"2024-09-17T04:49:39.977416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.016212+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T04:49:40.016212+0000","last_peered":"2024-09-17T04:49:40.016212+0000","last_clean":"2024-09-17T04:49:40.016212+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T04:49:40.016212+0000","last_undegraded":"2024-09-17T04:49:40.016212+0000","last_fullsized":"2024-09-17T04:49:40.016212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054699+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T04:49:42.054699+0000","last_peered":"2024-09-17T04:49:42.054699+0000","last_clean":"2024-09-17T04:49:42.054699+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T04:49:42.054699+0000","last_undegraded":"2024-09-17T04:49:42.054699+0000","last_fullsized":"2024-09-17T04:49:42.054699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.978927+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T04:49:39.978927+0000","last_peered":"2024-09-17T04:49:39.978927+0000","last_clean":"2024-09-17T04:49:39.978927+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T04:49:39.978927+0000","last_undegraded":"2024-09-17T04:49:39.978927+0000","last_fullsized":"2024-09-17T04:49:39.978927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.984280+0000","last_change":"2024-09-17T04:49:39.984280+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.984280+0000","last_undegraded":"2024-09-17T04:49:39.984280+0000","last_fullsized":"2024-09-17T04:49:39.984280+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.984182+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T04:49:39.984182+0000","last_peered":"2024-09-17T04:49:39.984182+0000","last_clean":"2024-09-17T04:49:39.984182+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T04:49:39.984182+0000","last_undegraded":"2024-09-17T04:49:39.984182+0000","last_fullsized":"2024-09-17T04:49:39.984182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.982084+0000","last_change":"2024-09-17T04:49:39.982084+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.982084+0000","last_undegraded":"2024-09-17T04:49:39.982084+0000","last_fullsized":"2024-09-17T04:49:39.982084+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991784+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T04:49:39.991784+0000","last_peered":"2024-09-17T04:49:39.991784+0000","last_clean":"2024-09-17T04:49:39.991784+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T04:49:39.991784+0000","last_undegraded":"2024-09-17T04:49:39.991784+0000","last_fullsized":"2024-09-17T04:49:39.991784+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.987090+0000","last_change":"2024-09-17T04:49:39.987090+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.987090+0000","last_undegraded":"2024-09-17T04:49:39.987090+0000","last_fullsized":"2024-09-17T04:49:39.987090+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.972492+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T04:49:39.972492+0000","last_peered":"2024-09-17T04:49:39.972492+0000","last_clean":"2024-09-17T04:49:39.972492+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T04:49:39.972492+0000","last_undegraded":"2024-09-17T04:49:39.972492+0000","last_fullsized":"2024-09-17T04:49:39.972492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.984293+0000","last_change":"2024-09-17T04:49:39.984293+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.984293+0000","last_undegraded":"2024-09-17T04:49:39.984293+0000","last_fullsized":"2024-09-17T04:49:39.984293+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.989187+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T04:49:39.989187+0000","last_peered":"2024-09-17T04:49:39.989187+0000","last_clean":"2024-09-17T04:49:39.989187+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T04:49:39.989187+0000","last_undegraded":"2024-09-17T04:49:39.989187+0000","last_fullsized":"2024-09-17T04:49:39.989187+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050316+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T04:49:42.050316+0000","last_peered":"2024-09-17T04:49:42.050316+0000","last_clean":"2024-09-17T04:49:42.050316+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T04:49:42.050316+0000","last_undegraded":"2024-09-17T04:49:42.050316+0000","last_fullsized":"2024-09-17T04:49:42.050316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.972470+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T04:49:39.972470+0000","last_peered":"2024-09-17T04:49:39.972470+0000","last_clean":"2024-09-17T04:49:39.972470+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T04:49:39.972470+0000","last_undegraded":"2024-09-17T04:49:39.972470+0000","last_fullsized":"2024-09-17T04:49:39.972470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.976385+0000","last_change":"2024-09-17T04:49:39.976385+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.976385+0000","last_undegraded":"2024-09-17T04:49:39.976385+0000","last_fullsized":"2024-09-17T04:49:39.976385+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.970750+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T04:49:39.970750+0000","last_peered":"2024-09-17T04:49:39.970750+0000","last_clean":"2024-09-17T04:49:39.970750+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T04:49:39.970750+0000","last_undegraded":"2024-09-17T04:49:39.970750+0000","last_fullsized":"2024-09-17T04:49:39.970750+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.979153+0000","last_change":"2024-09-17T04:49:39.979153+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.979153+0000","last_undegraded":"2024-09-17T04:49:39.979153+0000","last_fullsized":"2024-09-17T04:49:39.979153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.011778+0000","last_change":"2024-09-17T04:49:31.911036+0000","last_active":"2024-09-17T04:49:40.011778+0000","last_peered":"2024-09-17T04:49:40.011778+0000","last_clean":"2024-09-17T04:49:40.011778+0000","last_became_active":"2024-09-17T04:49:31.910285+0000","last_became_peered":"2024-09-17T04:49:31.910285+0000","last_unstale":"2024-09-17T04:49:40.011778+0000","last_undegraded":"2024-09-17T04:49:40.011778+0000","last_fullsized":"2024-09-17T04:49:40.011778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.979656+0000","last_change":"2024-09-17T04:49:39.979656+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.979656+0000","last_undegraded":"2024-09-17T04:49:39.979656+0000","last_fullsized":"2024-09-17T04:49:39.979656+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051528+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T04:49:42.051528+0000","last_peered":"2024-09-17T04:49:42.051528+0000","last_clean":"2024-09-17T04:49:42.051528+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T04:49:42.051528+0000","last_undegraded":"2024-09-17T04:49:42.051528+0000","last_fullsized":"2024-09-17T04:49:42.051528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.984690+0000","last_change":"2024-09-17T04:49:39.984690+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.984690+0000","last_undegraded":"2024-09-17T04:49:39.984690+0000","last_fullsized":"2024-09-17T04:49:39.984690+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053942+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T04:49:42.053942+0000","last_peered":"2024-09-17T04:49:42.053942+0000","last_clean":"2024-09-17T04:49:42.053942+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T04:49:42.053942+0000","last_undegraded":"2024-09-17T04:49:42.053942+0000","last_fullsized":"2024-09-17T04:49:42.053942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.975437+0000","last_change":"2024-09-17T04:49:39.975437+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.975437+0000","last_undegraded":"2024-09-17T04:49:39.975437+0000","last_fullsized":"2024-09-17T04:49:39.975437+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991687+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T04:49:39.991687+0000","last_peered":"2024-09-17T04:49:39.991687+0000","last_clean":"2024-09-17T04:49:39.991687+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T04:49:39.991687+0000","last_undegraded":"2024-09-17T04:49:39.991687+0000","last_fullsized":"2024-09-17T04:49:39.991687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053498+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T04:49:42.053498+0000","last_peered":"2024-09-17T04:49:42.053498+0000","last_clean":"2024-09-17T04:49:42.053498+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T04:49:42.053498+0000","last_undegraded":"2024-09-17T04:49:42.053498+0000","last_fullsized":"2024-09-17T04:49:42.053498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991111+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T04:49:39.991111+0000","last_peered":"2024-09-17T04:49:39.991111+0000","last_clean":"2024-09-17T04:49:39.991111+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T04:49:39.991111+0000","last_undegraded":"2024-09-17T04:49:39.991111+0000","last_fullsized":"2024-09-17T04:49:39.991111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.980619+0000","last_change":"2024-09-17T04:49:39.980619+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.980619+0000","last_undegraded":"2024-09-17T04:49:39.980619+0000","last_fullsized":"2024-09-17T04:49:39.980619+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.987814+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T04:49:39.987814+0000","last_peered":"2024-09-17T04:49:39.987814+0000","last_clean":"2024-09-17T04:49:39.987814+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T04:49:39.987814+0000","last_undegraded":"2024-09-17T04:49:39.987814+0000","last_fullsized":"2024-09-17T04:49:39.987814+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.981542+0000","last_change":"2024-09-17T04:49:39.981542+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.981542+0000","last_undegraded":"2024-09-17T04:49:39.981542+0000","last_fullsized":"2024-09-17T04:49:39.981542+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.984808+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T04:49:39.984808+0000","last_peered":"2024-09-17T04:49:39.984808+0000","last_clean":"2024-09-17T04:49:39.984808+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T04:49:39.984808+0000","last_undegraded":"2024-09-17T04:49:39.984808+0000","last_fullsized":"2024-09-17T04:49:39.984808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:40.006583+0000","last_change":"2024-09-17T04:49:40.006583+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:40.006583+0000","last_undegraded":"2024-09-17T04:49:40.006583+0000","last_fullsized":"2024-09-17T04:49:40.006583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.010550+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T04:49:40.010550+0000","last_peered":"2024-09-17T04:49:40.010550+0000","last_clean":"2024-09-17T04:49:40.010550+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T04:49:40.010550+0000","last_undegraded":"2024-09-17T04:49:40.010550+0000","last_fullsized":"2024-09-17T04:49:40.010550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.979767+0000","last_change":"2024-09-17T04:49:39.979767+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.979767+0000","last_undegraded":"2024-09-17T04:49:39.979767+0000","last_fullsized":"2024-09-17T04:49:39.979767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.988053+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T04:49:39.988053+0000","last_peered":"2024-09-17T04:49:39.988053+0000","last_clean":"2024-09-17T04:49:39.988053+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T04:49:39.988053+0000","last_undegraded":"2024-09-17T04:49:39.988053+0000","last_fullsized":"2024-09-17T04:49:39.988053+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.984817+0000","last_change":"2024-09-17T04:49:39.984817+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.984817+0000","last_undegraded":"2024-09-17T04:49:39.984817+0000","last_fullsized":"2024-09-17T04:49:39.984817+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985677+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T04:49:39.985677+0000","last_peered":"2024-09-17T04:49:39.985677+0000","last_clean":"2024-09-17T04:49:39.985677+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T04:49:39.985677+0000","last_undegraded":"2024-09-17T04:49:39.985677+0000","last_fullsized":"2024-09-17T04:49:39.985677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.985016+0000","last_change":"2024-09-17T04:49:39.985016+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.985016+0000","last_undegraded":"2024-09-17T04:49:39.985016+0000","last_fullsized":"2024-09-17T04:49:39.985016+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.991312+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T04:49:39.991312+0000","last_peered":"2024-09-17T04:49:39.991312+0000","last_clean":"2024-09-17T04:49:39.991312+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T04:49:39.991312+0000","last_undegraded":"2024-09-17T04:49:39.991312+0000","last_fullsized":"2024-09-17T04:49:39.991312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.981661+0000","last_change":"2024-09-17T04:49:39.981661+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.981661+0000","last_undegraded":"2024-09-17T04:49:39.981661+0000","last_fullsized":"2024-09-17T04:49:39.981661+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.985688+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T04:49:39.985688+0000","last_peered":"2024-09-17T04:49:39.985688+0000","last_clean":"2024-09-17T04:49:39.985688+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T04:49:39.985688+0000","last_undegraded":"2024-09-17T04:49:39.985688+0000","last_fullsized":"2024-09-17T04:49:39.985688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:40.008239+0000","last_change":"2024-09-17T04:49:40.008239+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:40.008239+0000","last_undegraded":"2024-09-17T04:49:40.008239+0000","last_fullsized":"2024-09-17T04:49:40.008239+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.018466+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T04:49:40.018466+0000","last_peered":"2024-09-17T04:49:40.018466+0000","last_clean":"2024-09-17T04:49:40.018466+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T04:49:40.018466+0000","last_undegraded":"2024-09-17T04:49:40.018466+0000","last_fullsized":"2024-09-17T04:49:40.018466+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.978152+0000","last_change":"2024-09-17T04:49:39.978152+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.978152+0000","last_undegraded":"2024-09-17T04:49:39.978152+0000","last_fullsized":"2024-09-17T04:49:39.978152+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:40.017691+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T04:49:40.017691+0000","last_peered":"2024-09-17T04:49:40.017691+0000","last_clean":"2024-09-17T04:49:40.017691+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T04:49:40.017691+0000","last_undegraded":"2024-09-17T04:49:40.017691+0000","last_fullsized":"2024-09-17T04:49:40.017691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":24,"state":"creating+peering","last_fresh":"2024-09-17T04:49:39.981911+0000","last_change":"2024-09-17T04:49:39.981911+0000","last_active":"2024-09-17T04:49:39.959607+0000","last_peered":"2024-09-17T04:49:39.959607+0000","last_clean":"2024-09-17T04:49:39.959607+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:49:39.981911+0000","last_undegraded":"2024-09-17T04:49:39.981911+0000","last_fullsized":"2024-09-17T04:49:39.981911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-17T04:49:39.990901+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T04:49:39.990901+0000","last_peered":"2024-09-17T04:49:39.990901+0000","last_clean":"2024-09-17T04:49:39.990901+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T04:49:39.990901+0000","last_undegraded":"2024-09-17T04:49:39.990901+0000","last_fullsized":"2024-09-17T04:49:39.990901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":90059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":847872,"data_stored":680395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":92219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":90517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":258048,"data_stored":91547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":843776,"data_stored":680373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509448,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":89991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:49:46.648 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:49:46.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.836+0000 7fbe6bb16640 1 -- 172.21.15.150:0/1822146259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe50003a90 msgr2=0x7fbe50003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:46.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.836+0000 7fbe6bb16640 1 --2- 172.21.15.150:0/1822146259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe50003a90 0x7fbe50003f00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbe60006fd0 tx=0x7fbe6001cc20 comp rx=0 tx=0).stop 2024-09-17T04:49:46.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.836+0000 7fbe6bb16640 1 -- 172.21.15.150:0/1822146259 shutdown_connections 2024-09-17T04:49:46.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.836+0000 7fbe6bb16640 1 --2- 172.21.15.150:0/1822146259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe50003a90 0x7fbe50003f00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.836+0000 7fbe6bb16640 1 -- 172.21.15.150:0/1822146259 >> 172.21.15.150:0/1822146259 conn(0x7fbe640aab60 msgr2=0x7fbe640aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:46.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.836+0000 7fbe6bb16640 1 -- 172.21.15.150:0/1822146259 shutdown_connections 2024-09-17T04:49:46.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.836+0000 7fbe6bb16640 1 -- 172.21.15.150:0/1822146259 wait complete. 2024-09-17T04:49:46.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe6bb16640 1 Processor -- start 2024-09-17T04:49:46.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe6bb16640 1 -- start start 2024-09-17T04:49:46.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe6bb16640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe50003a90 0x7fbe6414e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:46.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe6bb16640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe6406e070 0x7fbe6414bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:46.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe6bb16640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe6406e480 0x7fbe6414c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:46.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe6bb16640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbe64149ef0 con 0x7fbe6406e070 2024-09-17T04:49:46.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe6bb16640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbe64149d70 con 0x7fbe50003a90 2024-09-17T04:49:46.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe6bb16640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbe6414a070 con 0x7fbe6406e480 2024-09-17T04:49:46.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe690b1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe6406e070 0x7fbe6414bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:46.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe690b1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe6406e070 0x7fbe6414bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53268/0 (socket says 172.21.15.150:53268) 2024-09-17T04:49:46.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe690b1640 1 -- 172.21.15.150:0/2877934875 learned_addr learned my addr 172.21.15.150:0/2877934875 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe698b2640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe50003a90 0x7fbe6414e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe6a0b3640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe6406e480 0x7fbe6414c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe690b1640 1 -- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe6406e480 msgr2=0x7fbe6414c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe690b1640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe6406e480 0x7fbe6414c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe690b1640 1 -- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe50003a90 msgr2=0x7fbe6414e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe690b1640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe50003a90 0x7fbe6414e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe690b1640 1 -- 172.21.15.150:0/2877934875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6414b800 con 0x7fbe6406e070 2024-09-17T04:49:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe690b1640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe6406e070 0x7fbe6414bdb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbe5400c530 tx=0x7fbe5400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe5a7fc640 1 -- 172.21.15.150:0/2877934875 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe540052d0 con 0x7fbe6406e070 2024-09-17T04:49:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe5a7fc640 1 -- 172.21.15.150:0/2877934875 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe54005490 con 0x7fbe6406e070 2024-09-17T04:49:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe6bb16640 1 -- 172.21.15.150:0/2877934875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe6414f2b0 con 0x7fbe6406e070 2024-09-17T04:49:46.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe5a7fc640 1 -- 172.21.15.150:0/2877934875 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe54012870 con 0x7fbe6406e070 2024-09-17T04:49:46.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe6bb16640 1 -- 172.21.15.150:0/2877934875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe6414cea0 con 0x7fbe6406e070 2024-09-17T04:49:46.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.840+0000 7fbe6bb16640 1 -- 172.21.15.150:0/2877934875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe6414a980 con 0x7fbe6406e070 2024-09-17T04:49:46.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.844+0000 7fbe5a7fc640 1 -- 172.21.15.150:0/2877934875 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbe54003600 con 0x7fbe6406e070 2024-09-17T04:49:46.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.844+0000 7fbe5a7fc640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe480731a0 0x7fbe480756b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:46.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.844+0000 7fbe5a7fc640 1 -- 172.21.15.150:0/2877934875 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fbe54094bd0 con 0x7fbe6406e070 2024-09-17T04:49:46.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.844+0000 7fbe698b2640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe480731a0 0x7fbe480756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:46.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.844+0000 7fbe698b2640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe480731a0 0x7fbe480756b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbe600ac000 tx=0x7fbe60019d80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:46.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:46.848+0000 7fbe5a7fc640 1 -- 172.21.15.150:0/2877934875 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbe6414a980 con 0x7fbe6406e070 2024-09-17T04:49:47.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.008+0000 7fbe6bb16640 1 -- 172.21.15.150:0/2877934875 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbe64157bd0 con 0x7fbe480731a0 2024-09-17T04:49:47.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.032+0000 7fbe5a7fc640 1 -- 172.21.15.150:0/2877934875 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490420 (secure 0 0 0) 0x7fbe64157bd0 con 0x7fbe480731a0 2024-09-17T04:49:47.037 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:47.041 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T04:49:47.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.036+0000 7fbe6bb16640 1 -- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe480731a0 msgr2=0x7fbe480756b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:47.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.036+0000 7fbe6bb16640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe480731a0 0x7fbe480756b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbe600ac000 tx=0x7fbe60019d80 comp rx=0 tx=0).stop 2024-09-17T04:49:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.036+0000 7fbe6bb16640 1 -- 172.21.15.150:0/2877934875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe6406e070 msgr2=0x7fbe6414bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.036+0000 7fbe6bb16640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe6406e070 0x7fbe6414bdb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbe5400c530 tx=0x7fbe5400c9f0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.036+0000 7fbe6bb16640 1 -- 172.21.15.150:0/2877934875 shutdown_connections 2024-09-17T04:49:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.036+0000 7fbe6bb16640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe480731a0 0x7fbe480756b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.036+0000 7fbe6bb16640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe6406e480 0x7fbe6414c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.036+0000 7fbe6bb16640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe6406e070 0x7fbe6414bdb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.036+0000 7fbe6bb16640 1 --2- 172.21.15.150:0/2877934875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe50003a90 0x7fbe6414e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.036+0000 7fbe6bb16640 1 -- 172.21.15.150:0/2877934875 >> 172.21.15.150:0/2877934875 conn(0x7fbe640aab60 msgr2=0x7fbe64068e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.036+0000 7fbe6bb16640 1 -- 172.21.15.150:0/2877934875 shutdown_connections 2024-09-17T04:49:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.036+0000 7fbe6bb16640 1 -- 172.21.15.150:0/2877934875 wait complete. 2024-09-17T04:49:47.058 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-17T04:49:46.946858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":191,"num_write_kb":1656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":373,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223812,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750908,"statfs":{"total":773094113280,"available":772864929792,"internally_reserved":0,"allocated":3481600,"data_stored":2027328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":59,"apply_latency_ms":59,"commit_latency_ns":59000000,"apply_latency_ns":59000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.979341"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671077+0000","last_change":"2024-09-17T04:49:41.025271+0000","last_active":"2024-09-17T04:49:42.671077+0000","last_peered":"2024-09-17T04:49:42.671077+0000","last_clean":"2024-09-17T04:49:42.671077+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T04:49:42.671077+0000","last_undegraded":"2024-09-17T04:49:42.671077+0000","last_fullsized":"2024-09-17T04:49:42.671077+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:39.715602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054640+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T04:49:42.054640+0000","last_peered":"2024-09-17T04:49:42.054640+0000","last_clean":"2024-09-17T04:49:42.054640+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T04:49:42.054640+0000","last_undegraded":"2024-09-17T04:49:42.054640+0000","last_fullsized":"2024-09-17T04:49:42.054640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671706+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T04:49:42.671706+0000","last_peered":"2024-09-17T04:49:42.671706+0000","last_clean":"2024-09-17T04:49:42.671706+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T04:49:42.671706+0000","last_undegraded":"2024-09-17T04:49:42.671706+0000","last_fullsized":"2024-09-17T04:49:42.671706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668736+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T04:49:42.668736+0000","last_peered":"2024-09-17T04:49:42.668736+0000","last_clean":"2024-09-17T04:49:42.668736+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T04:49:42.668736+0000","last_undegraded":"2024-09-17T04:49:42.668736+0000","last_fullsized":"2024-09-17T04:49:42.668736+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:38.417840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047972+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T04:49:42.047972+0000","last_peered":"2024-09-17T04:49:42.047972+0000","last_clean":"2024-09-17T04:49:42.047972+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T04:49:42.047972+0000","last_undegraded":"2024-09-17T04:49:42.047972+0000","last_fullsized":"2024-09-17T04:49:42.047972+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672300+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T04:49:42.672300+0000","last_peered":"2024-09-17T04:49:42.672300+0000","last_clean":"2024-09-17T04:49:42.672300+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T04:49:42.672300+0000","last_undegraded":"2024-09-17T04:49:42.672300+0000","last_fullsized":"2024-09-17T04:49:42.672300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672014+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T04:49:42.672014+0000","last_peered":"2024-09-17T04:49:42.672014+0000","last_clean":"2024-09-17T04:49:42.672014+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T04:49:42.672014+0000","last_undegraded":"2024-09-17T04:49:42.672014+0000","last_fullsized":"2024-09-17T04:49:42.672014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535134+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T04:49:42.535134+0000","last_peered":"2024-09-17T04:49:42.535134+0000","last_clean":"2024-09-17T04:49:42.535134+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T04:49:42.535134+0000","last_undegraded":"2024-09-17T04:49:42.535134+0000","last_fullsized":"2024-09-17T04:49:42.535134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672206+0000","last_change":"2024-09-17T04:49:41.067317+0000","last_active":"2024-09-17T04:49:42.672206+0000","last_peered":"2024-09-17T04:49:42.672206+0000","last_clean":"2024-09-17T04:49:42.672206+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T04:49:42.672206+0000","last_undegraded":"2024-09-17T04:49:42.672206+0000","last_fullsized":"2024-09-17T04:49:42.672206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:37.637477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670707+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T04:49:42.670707+0000","last_peered":"2024-09-17T04:49:42.670707+0000","last_clean":"2024-09-17T04:49:42.670707+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T04:49:42.670707+0000","last_undegraded":"2024-09-17T04:49:42.670707+0000","last_fullsized":"2024-09-17T04:49:42.670707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674274+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T04:49:42.674274+0000","last_peered":"2024-09-17T04:49:42.674274+0000","last_clean":"2024-09-17T04:49:42.674274+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T04:49:42.674274+0000","last_undegraded":"2024-09-17T04:49:42.674274+0000","last_fullsized":"2024-09-17T04:49:42.674274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535118+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T04:49:42.535118+0000","last_peered":"2024-09-17T04:49:42.535118+0000","last_clean":"2024-09-17T04:49:42.535118+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T04:49:42.535118+0000","last_undegraded":"2024-09-17T04:49:42.535118+0000","last_fullsized":"2024-09-17T04:49:42.535118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535259+0000","last_change":"2024-09-17T04:49:41.043531+0000","last_active":"2024-09-17T04:49:42.535259+0000","last_peered":"2024-09-17T04:49:42.535259+0000","last_clean":"2024-09-17T04:49:42.535259+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T04:49:42.535259+0000","last_undegraded":"2024-09-17T04:49:42.535259+0000","last_fullsized":"2024-09-17T04:49:42.535259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672162+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T04:49:42.672162+0000","last_peered":"2024-09-17T04:49:42.672162+0000","last_clean":"2024-09-17T04:49:42.672162+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T04:49:42.672162+0000","last_undegraded":"2024-09-17T04:49:42.672162+0000","last_fullsized":"2024-09-17T04:49:42.672162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042945+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T04:49:42.042945+0000","last_peered":"2024-09-17T04:49:42.042945+0000","last_clean":"2024-09-17T04:49:42.042945+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T04:49:42.042945+0000","last_undegraded":"2024-09-17T04:49:42.042945+0000","last_fullsized":"2024-09-17T04:49:42.042945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535003+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T04:49:42.535003+0000","last_peered":"2024-09-17T04:49:42.535003+0000","last_clean":"2024-09-17T04:49:42.535003+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T04:49:42.535003+0000","last_undegraded":"2024-09-17T04:49:42.535003+0000","last_fullsized":"2024-09-17T04:49:42.535003+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:16.854365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669588+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T04:49:42.669588+0000","last_peered":"2024-09-17T04:49:42.669588+0000","last_clean":"2024-09-17T04:49:42.669588+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T04:49:42.669588+0000","last_undegraded":"2024-09-17T04:49:42.669588+0000","last_fullsized":"2024-09-17T04:49:42.669588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042526+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T04:49:42.042526+0000","last_peered":"2024-09-17T04:49:42.042526+0000","last_clean":"2024-09-17T04:49:42.042526+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T04:49:42.042526+0000","last_undegraded":"2024-09-17T04:49:42.042526+0000","last_fullsized":"2024-09-17T04:49:42.042526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041407+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T04:49:42.041407+0000","last_peered":"2024-09-17T04:49:42.041407+0000","last_clean":"2024-09-17T04:49:42.041407+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T04:49:42.041407+0000","last_undegraded":"2024-09-17T04:49:42.041407+0000","last_fullsized":"2024-09-17T04:49:42.041407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042319+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T04:49:42.042319+0000","last_peered":"2024-09-17T04:49:42.042319+0000","last_clean":"2024-09-17T04:49:42.042319+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T04:49:42.042319+0000","last_undegraded":"2024-09-17T04:49:42.042319+0000","last_fullsized":"2024-09-17T04:49:42.042319+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:28.365399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669339+0000","last_change":"2024-09-17T04:49:41.038415+0000","last_active":"2024-09-17T04:49:42.669339+0000","last_peered":"2024-09-17T04:49:42.669339+0000","last_clean":"2024-09-17T04:49:42.669339+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T04:49:42.669339+0000","last_undegraded":"2024-09-17T04:49:42.669339+0000","last_fullsized":"2024-09-17T04:49:42.669339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:17.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668616+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T04:49:42.668616+0000","last_peered":"2024-09-17T04:49:42.668616+0000","last_clean":"2024-09-17T04:49:42.668616+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T04:49:42.668616+0000","last_undegraded":"2024-09-17T04:49:42.668616+0000","last_fullsized":"2024-09-17T04:49:42.668616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052248+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T04:49:42.052248+0000","last_peered":"2024-09-17T04:49:42.052248+0000","last_clean":"2024-09-17T04:49:42.052248+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T04:49:42.052248+0000","last_undegraded":"2024-09-17T04:49:42.052248+0000","last_fullsized":"2024-09-17T04:49:42.052248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671586+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T04:49:42.671586+0000","last_peered":"2024-09-17T04:49:42.671586+0000","last_clean":"2024-09-17T04:49:42.671586+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T04:49:42.671586+0000","last_undegraded":"2024-09-17T04:49:42.671586+0000","last_fullsized":"2024-09-17T04:49:42.671586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:17.424265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042028+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T04:49:42.042028+0000","last_peered":"2024-09-17T04:49:42.042028+0000","last_clean":"2024-09-17T04:49:42.042028+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T04:49:42.042028+0000","last_undegraded":"2024-09-17T04:49:42.042028+0000","last_fullsized":"2024-09-17T04:49:42.042028+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:06.516432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048455+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T04:49:42.048455+0000","last_peered":"2024-09-17T04:49:42.048455+0000","last_clean":"2024-09-17T04:49:42.048455+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T04:49:42.048455+0000","last_undegraded":"2024-09-17T04:49:42.048455+0000","last_fullsized":"2024-09-17T04:49:42.048455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048500+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T04:49:42.048500+0000","last_peered":"2024-09-17T04:49:42.048500+0000","last_clean":"2024-09-17T04:49:42.048500+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T04:49:42.048500+0000","last_undegraded":"2024-09-17T04:49:42.048500+0000","last_fullsized":"2024-09-17T04:49:42.048500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668827+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T04:49:42.668827+0000","last_peered":"2024-09-17T04:49:42.668827+0000","last_clean":"2024-09-17T04:49:42.668827+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T04:49:42.668827+0000","last_undegraded":"2024-09-17T04:49:42.668827+0000","last_fullsized":"2024-09-17T04:49:42.668827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:00.010365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675406+0000","last_change":"2024-09-17T04:49:41.036658+0000","last_active":"2024-09-17T04:49:42.675406+0000","last_peered":"2024-09-17T04:49:42.675406+0000","last_clean":"2024-09-17T04:49:42.675406+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T04:49:42.675406+0000","last_undegraded":"2024-09-17T04:49:42.675406+0000","last_fullsized":"2024-09-17T04:49:42.675406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042824+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T04:49:42.042824+0000","last_peered":"2024-09-17T04:49:42.042824+0000","last_clean":"2024-09-17T04:49:42.042824+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T04:49:42.042824+0000","last_undegraded":"2024-09-17T04:49:42.042824+0000","last_fullsized":"2024-09-17T04:49:42.042824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668829+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T04:49:42.668829+0000","last_peered":"2024-09-17T04:49:42.668829+0000","last_clean":"2024-09-17T04:49:42.668829+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T04:49:42.668829+0000","last_undegraded":"2024-09-17T04:49:42.668829+0000","last_fullsized":"2024-09-17T04:49:42.668829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536259+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T04:49:42.536259+0000","last_peered":"2024-09-17T04:49:42.536259+0000","last_clean":"2024-09-17T04:49:42.536259+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T04:49:42.536259+0000","last_undegraded":"2024-09-17T04:49:42.536259+0000","last_fullsized":"2024-09-17T04:49:42.536259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048942+0000","last_change":"2024-09-17T04:49:41.050170+0000","last_active":"2024-09-17T04:49:42.048942+0000","last_peered":"2024-09-17T04:49:42.048942+0000","last_clean":"2024-09-17T04:49:42.048942+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T04:49:42.048942+0000","last_undegraded":"2024-09-17T04:49:42.048942+0000","last_fullsized":"2024-09-17T04:49:42.048942+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051929+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T04:49:42.051929+0000","last_peered":"2024-09-17T04:49:42.051929+0000","last_clean":"2024-09-17T04:49:42.051929+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T04:49:42.051929+0000","last_undegraded":"2024-09-17T04:49:42.051929+0000","last_fullsized":"2024-09-17T04:49:42.051929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042185+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T04:49:42.042185+0000","last_peered":"2024-09-17T04:49:42.042185+0000","last_clean":"2024-09-17T04:49:42.042185+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T04:49:42.042185+0000","last_undegraded":"2024-09-17T04:49:42.042185+0000","last_fullsized":"2024-09-17T04:49:42.042185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535553+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T04:49:42.535553+0000","last_peered":"2024-09-17T04:49:42.535553+0000","last_clean":"2024-09-17T04:49:42.535553+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T04:49:42.535553+0000","last_undegraded":"2024-09-17T04:49:42.535553+0000","last_fullsized":"2024-09-17T04:49:42.535553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041211+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T04:49:42.041211+0000","last_peered":"2024-09-17T04:49:42.041211+0000","last_clean":"2024-09-17T04:49:42.041211+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T04:49:42.041211+0000","last_undegraded":"2024-09-17T04:49:42.041211+0000","last_fullsized":"2024-09-17T04:49:42.041211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054336+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T04:49:42.054336+0000","last_peered":"2024-09-17T04:49:42.054336+0000","last_clean":"2024-09-17T04:49:42.054336+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T04:49:42.054336+0000","last_undegraded":"2024-09-17T04:49:42.054336+0000","last_fullsized":"2024-09-17T04:49:42.054336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052826+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T04:49:42.052826+0000","last_peered":"2024-09-17T04:49:42.052826+0000","last_clean":"2024-09-17T04:49:42.052826+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T04:49:42.052826+0000","last_undegraded":"2024-09-17T04:49:42.052826+0000","last_fullsized":"2024-09-17T04:49:42.052826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048145+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T04:49:42.048145+0000","last_peered":"2024-09-17T04:49:42.048145+0000","last_clean":"2024-09-17T04:49:42.048145+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T04:49:42.048145+0000","last_undegraded":"2024-09-17T04:49:42.048145+0000","last_fullsized":"2024-09-17T04:49:42.048145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669693+0000","last_change":"2024-09-17T04:49:41.042314+0000","last_active":"2024-09-17T04:49:42.669693+0000","last_peered":"2024-09-17T04:49:42.669693+0000","last_clean":"2024-09-17T04:49:42.669693+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T04:49:42.669693+0000","last_undegraded":"2024-09-17T04:49:42.669693+0000","last_fullsized":"2024-09-17T04:49:42.669693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055299+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T04:49:42.055299+0000","last_peered":"2024-09-17T04:49:42.055299+0000","last_clean":"2024-09-17T04:49:42.055299+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T04:49:42.055299+0000","last_undegraded":"2024-09-17T04:49:42.055299+0000","last_fullsized":"2024-09-17T04:49:42.055299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043663+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T04:49:42.043663+0000","last_peered":"2024-09-17T04:49:42.043663+0000","last_clean":"2024-09-17T04:49:42.043663+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T04:49:42.043663+0000","last_undegraded":"2024-09-17T04:49:42.043663+0000","last_fullsized":"2024-09-17T04:49:42.043663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053169+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T04:49:42.053169+0000","last_peered":"2024-09-17T04:49:42.053169+0000","last_clean":"2024-09-17T04:49:42.053169+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T04:49:42.053169+0000","last_undegraded":"2024-09-17T04:49:42.053169+0000","last_fullsized":"2024-09-17T04:49:42.053169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052959+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T04:49:42.052959+0000","last_peered":"2024-09-17T04:49:42.052959+0000","last_clean":"2024-09-17T04:49:42.052959+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T04:49:42.052959+0000","last_undegraded":"2024-09-17T04:49:42.052959+0000","last_fullsized":"2024-09-17T04:49:42.052959+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535986+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T04:49:42.535986+0000","last_peered":"2024-09-17T04:49:42.535986+0000","last_clean":"2024-09-17T04:49:42.535986+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T04:49:42.535986+0000","last_undegraded":"2024-09-17T04:49:42.535986+0000","last_fullsized":"2024-09-17T04:49:42.535986+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670900+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T04:49:42.670900+0000","last_peered":"2024-09-17T04:49:42.670900+0000","last_clean":"2024-09-17T04:49:42.670900+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T04:49:42.670900+0000","last_undegraded":"2024-09-17T04:49:42.670900+0000","last_fullsized":"2024-09-17T04:49:42.670900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050191+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T04:49:42.050191+0000","last_peered":"2024-09-17T04:49:42.050191+0000","last_clean":"2024-09-17T04:49:42.050191+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T04:49:42.050191+0000","last_undegraded":"2024-09-17T04:49:42.050191+0000","last_fullsized":"2024-09-17T04:49:42.050191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670215+0000","last_change":"2024-09-17T04:49:41.069177+0000","last_active":"2024-09-17T04:49:42.670215+0000","last_peered":"2024-09-17T04:49:42.670215+0000","last_clean":"2024-09-17T04:49:42.670215+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T04:49:42.670215+0000","last_undegraded":"2024-09-17T04:49:42.670215+0000","last_fullsized":"2024-09-17T04:49:42.670215+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:16.855206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670381+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T04:49:42.670381+0000","last_peered":"2024-09-17T04:49:42.670381+0000","last_clean":"2024-09-17T04:49:42.670381+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T04:49:42.670381+0000","last_undegraded":"2024-09-17T04:49:42.670381+0000","last_fullsized":"2024-09-17T04:49:42.670381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042706+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T04:49:42.042706+0000","last_peered":"2024-09-17T04:49:42.042706+0000","last_clean":"2024-09-17T04:49:42.042706+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T04:49:42.042706+0000","last_undegraded":"2024-09-17T04:49:42.042706+0000","last_fullsized":"2024-09-17T04:49:42.042706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054900+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T04:49:42.054900+0000","last_peered":"2024-09-17T04:49:42.054900+0000","last_clean":"2024-09-17T04:49:42.054900+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T04:49:42.054900+0000","last_undegraded":"2024-09-17T04:49:42.054900+0000","last_fullsized":"2024-09-17T04:49:42.054900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669308+0000","last_change":"2024-09-17T04:49:41.068537+0000","last_active":"2024-09-17T04:49:42.669308+0000","last_peered":"2024-09-17T04:49:42.669308+0000","last_clean":"2024-09-17T04:49:42.669308+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T04:49:42.669308+0000","last_undegraded":"2024-09-17T04:49:42.669308+0000","last_fullsized":"2024-09-17T04:49:42.669308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667397+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T04:49:42.667397+0000","last_peered":"2024-09-17T04:49:42.667397+0000","last_clean":"2024-09-17T04:49:42.667397+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T04:49:42.667397+0000","last_undegraded":"2024-09-17T04:49:42.667397+0000","last_fullsized":"2024-09-17T04:49:42.667397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668940+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T04:49:42.668940+0000","last_peered":"2024-09-17T04:49:42.668940+0000","last_clean":"2024-09-17T04:49:42.668940+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T04:49:42.668940+0000","last_undegraded":"2024-09-17T04:49:42.668940+0000","last_fullsized":"2024-09-17T04:49:42.668940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054921+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T04:49:42.054921+0000","last_peered":"2024-09-17T04:49:42.054921+0000","last_clean":"2024-09-17T04:49:42.054921+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T04:49:42.054921+0000","last_undegraded":"2024-09-17T04:49:42.054921+0000","last_fullsized":"2024-09-17T04:49:42.054921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041762+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T04:49:42.041762+0000","last_peered":"2024-09-17T04:49:42.041762+0000","last_clean":"2024-09-17T04:49:42.041762+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T04:49:42.041762+0000","last_undegraded":"2024-09-17T04:49:42.041762+0000","last_fullsized":"2024-09-17T04:49:42.041762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050299+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T04:49:42.050299+0000","last_peered":"2024-09-17T04:49:42.050299+0000","last_clean":"2024-09-17T04:49:42.050299+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T04:49:42.050299+0000","last_undegraded":"2024-09-17T04:49:42.050299+0000","last_fullsized":"2024-09-17T04:49:42.050299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041771+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T04:49:42.041771+0000","last_peered":"2024-09-17T04:49:42.041771+0000","last_clean":"2024-09-17T04:49:42.041771+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T04:49:42.041771+0000","last_undegraded":"2024-09-17T04:49:42.041771+0000","last_fullsized":"2024-09-17T04:49:42.041771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672234+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T04:49:42.672234+0000","last_peered":"2024-09-17T04:49:42.672234+0000","last_clean":"2024-09-17T04:49:42.672234+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T04:49:42.672234+0000","last_undegraded":"2024-09-17T04:49:42.672234+0000","last_fullsized":"2024-09-17T04:49:42.672234+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671183+0000","last_change":"2024-09-17T04:49:41.052564+0000","last_active":"2024-09-17T04:49:42.671183+0000","last_peered":"2024-09-17T04:49:42.671183+0000","last_clean":"2024-09-17T04:49:42.671183+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T04:49:42.671183+0000","last_undegraded":"2024-09-17T04:49:42.671183+0000","last_fullsized":"2024-09-17T04:49:42.671183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049438+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T04:49:42.049438+0000","last_peered":"2024-09-17T04:49:42.049438+0000","last_clean":"2024-09-17T04:49:42.049438+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T04:49:42.049438+0000","last_undegraded":"2024-09-17T04:49:42.049438+0000","last_fullsized":"2024-09-17T04:49:42.049438+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041458+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T04:49:42.041458+0000","last_peered":"2024-09-17T04:49:42.041458+0000","last_clean":"2024-09-17T04:49:42.041458+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T04:49:42.041458+0000","last_undegraded":"2024-09-17T04:49:42.041458+0000","last_fullsized":"2024-09-17T04:49:42.041458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671735+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T04:49:42.671735+0000","last_peered":"2024-09-17T04:49:42.671735+0000","last_clean":"2024-09-17T04:49:42.671735+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T04:49:42.671735+0000","last_undegraded":"2024-09-17T04:49:42.671735+0000","last_fullsized":"2024-09-17T04:49:42.671735+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043078+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T04:49:42.043078+0000","last_peered":"2024-09-17T04:49:42.043078+0000","last_clean":"2024-09-17T04:49:42.043078+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T04:49:42.043078+0000","last_undegraded":"2024-09-17T04:49:42.043078+0000","last_fullsized":"2024-09-17T04:49:42.043078+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:21.202398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669578+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T04:49:42.669578+0000","last_peered":"2024-09-17T04:49:42.669578+0000","last_clean":"2024-09-17T04:49:42.669578+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T04:49:42.669578+0000","last_undegraded":"2024-09-17T04:49:42.669578+0000","last_fullsized":"2024-09-17T04:49:42.669578+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:48.128609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043280+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T04:49:42.043280+0000","last_peered":"2024-09-17T04:49:42.043280+0000","last_clean":"2024-09-17T04:49:42.043280+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T04:49:42.043280+0000","last_undegraded":"2024-09-17T04:49:42.043280+0000","last_fullsized":"2024-09-17T04:49:42.043280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055292+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T04:49:42.055292+0000","last_peered":"2024-09-17T04:49:42.055292+0000","last_clean":"2024-09-17T04:49:42.055292+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T04:49:42.055292+0000","last_undegraded":"2024-09-17T04:49:42.055292+0000","last_fullsized":"2024-09-17T04:49:42.055292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536164+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T04:49:42.536164+0000","last_peered":"2024-09-17T04:49:42.536164+0000","last_clean":"2024-09-17T04:49:42.536164+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T04:49:42.536164+0000","last_undegraded":"2024-09-17T04:49:42.536164+0000","last_fullsized":"2024-09-17T04:49:42.536164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050257+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T04:49:42.050257+0000","last_peered":"2024-09-17T04:49:42.050257+0000","last_clean":"2024-09-17T04:49:42.050257+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T04:49:42.050257+0000","last_undegraded":"2024-09-17T04:49:42.050257+0000","last_fullsized":"2024-09-17T04:49:42.050257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671927+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T04:49:42.671927+0000","last_peered":"2024-09-17T04:49:42.671927+0000","last_clean":"2024-09-17T04:49:42.671927+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T04:49:42.671927+0000","last_undegraded":"2024-09-17T04:49:42.671927+0000","last_fullsized":"2024-09-17T04:49:42.671927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671411+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T04:49:42.671411+0000","last_peered":"2024-09-17T04:49:42.671411+0000","last_clean":"2024-09-17T04:49:42.671411+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T04:49:42.671411+0000","last_undegraded":"2024-09-17T04:49:42.671411+0000","last_fullsized":"2024-09-17T04:49:42.671411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670087+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T04:49:42.670087+0000","last_peered":"2024-09-17T04:49:42.670087+0000","last_clean":"2024-09-17T04:49:42.670087+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T04:49:42.670087+0000","last_undegraded":"2024-09-17T04:49:42.670087+0000","last_fullsized":"2024-09-17T04:49:42.670087+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:18.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672287+0000","last_change":"2024-09-17T04:49:41.031036+0000","last_active":"2024-09-17T04:49:42.672287+0000","last_peered":"2024-09-17T04:49:42.672287+0000","last_clean":"2024-09-17T04:49:42.672287+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T04:49:42.672287+0000","last_undegraded":"2024-09-17T04:49:42.672287+0000","last_fullsized":"2024-09-17T04:49:42.672287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:48.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672111+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T04:49:42.672111+0000","last_peered":"2024-09-17T04:49:42.672111+0000","last_clean":"2024-09-17T04:49:42.672111+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T04:49:42.672111+0000","last_undegraded":"2024-09-17T04:49:42.672111+0000","last_fullsized":"2024-09-17T04:49:42.672111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052688+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T04:49:42.052688+0000","last_peered":"2024-09-17T04:49:42.052688+0000","last_clean":"2024-09-17T04:49:42.052688+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T04:49:42.052688+0000","last_undegraded":"2024-09-17T04:49:42.052688+0000","last_fullsized":"2024-09-17T04:49:42.052688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041095+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T04:49:42.041095+0000","last_peered":"2024-09-17T04:49:42.041095+0000","last_clean":"2024-09-17T04:49:42.041095+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T04:49:42.041095+0000","last_undegraded":"2024-09-17T04:49:42.041095+0000","last_fullsized":"2024-09-17T04:49:42.041095+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048488+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T04:49:42.048488+0000","last_peered":"2024-09-17T04:49:42.048488+0000","last_clean":"2024-09-17T04:49:42.048488+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T04:49:42.048488+0000","last_undegraded":"2024-09-17T04:49:42.048488+0000","last_fullsized":"2024-09-17T04:49:42.048488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536967+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T04:49:42.536967+0000","last_peered":"2024-09-17T04:49:42.536967+0000","last_clean":"2024-09-17T04:49:42.536967+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T04:49:42.536967+0000","last_undegraded":"2024-09-17T04:49:42.536967+0000","last_fullsized":"2024-09-17T04:49:42.536967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672404+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T04:49:42.672404+0000","last_peered":"2024-09-17T04:49:42.672404+0000","last_clean":"2024-09-17T04:49:42.672404+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T04:49:42.672404+0000","last_undegraded":"2024-09-17T04:49:42.672404+0000","last_fullsized":"2024-09-17T04:49:42.672404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672966+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T04:49:42.672966+0000","last_peered":"2024-09-17T04:49:42.672966+0000","last_clean":"2024-09-17T04:49:42.672966+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T04:49:42.672966+0000","last_undegraded":"2024-09-17T04:49:42.672966+0000","last_fullsized":"2024-09-17T04:49:42.672966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:09.712272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052007+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T04:49:42.052007+0000","last_peered":"2024-09-17T04:49:42.052007+0000","last_clean":"2024-09-17T04:49:42.052007+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T04:49:42.052007+0000","last_undegraded":"2024-09-17T04:49:42.052007+0000","last_fullsized":"2024-09-17T04:49:42.052007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:43.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042787+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T04:49:42.042787+0000","last_peered":"2024-09-17T04:49:42.042787+0000","last_clean":"2024-09-17T04:49:42.042787+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T04:49:42.042787+0000","last_undegraded":"2024-09-17T04:49:42.042787+0000","last_fullsized":"2024-09-17T04:49:42.042787+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049500+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T04:49:42.049500+0000","last_peered":"2024-09-17T04:49:42.049500+0000","last_clean":"2024-09-17T04:49:42.049500+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T04:49:42.049500+0000","last_undegraded":"2024-09-17T04:49:42.049500+0000","last_fullsized":"2024-09-17T04:49:42.049500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671253+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T04:49:42.671253+0000","last_peered":"2024-09-17T04:49:42.671253+0000","last_clean":"2024-09-17T04:49:42.671253+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T04:49:42.671253+0000","last_undegraded":"2024-09-17T04:49:42.671253+0000","last_fullsized":"2024-09-17T04:49:42.671253+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:52.364568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041504+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T04:49:42.041504+0000","last_peered":"2024-09-17T04:49:42.041504+0000","last_clean":"2024-09-17T04:49:42.041504+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T04:49:42.041504+0000","last_undegraded":"2024-09-17T04:49:42.041504+0000","last_fullsized":"2024-09-17T04:49:42.041504+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:03.537392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671439+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T04:49:42.671439+0000","last_peered":"2024-09-17T04:49:42.671439+0000","last_clean":"2024-09-17T04:49:42.671439+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T04:49:42.671439+0000","last_undegraded":"2024-09-17T04:49:42.671439+0000","last_fullsized":"2024-09-17T04:49:42.671439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670943+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T04:49:42.670943+0000","last_peered":"2024-09-17T04:49:42.670943+0000","last_clean":"2024-09-17T04:49:42.670943+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T04:49:42.670943+0000","last_undegraded":"2024-09-17T04:49:42.670943+0000","last_fullsized":"2024-09-17T04:49:42.670943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671787+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T04:49:42.671787+0000","last_peered":"2024-09-17T04:49:42.671787+0000","last_clean":"2024-09-17T04:49:42.671787+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T04:49:42.671787+0000","last_undegraded":"2024-09-17T04:49:42.671787+0000","last_fullsized":"2024-09-17T04:49:42.671787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:52.120543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041402+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T04:49:42.041402+0000","last_peered":"2024-09-17T04:49:42.041402+0000","last_clean":"2024-09-17T04:49:42.041402+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T04:49:42.041402+0000","last_undegraded":"2024-09-17T04:49:42.041402+0000","last_fullsized":"2024-09-17T04:49:42.041402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:56.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041285+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T04:49:42.041285+0000","last_peered":"2024-09-17T04:49:42.041285+0000","last_clean":"2024-09-17T04:49:42.041285+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T04:49:42.041285+0000","last_undegraded":"2024-09-17T04:49:42.041285+0000","last_fullsized":"2024-09-17T04:49:42.041285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042566+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T04:49:42.042566+0000","last_peered":"2024-09-17T04:49:42.042566+0000","last_clean":"2024-09-17T04:49:42.042566+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T04:49:42.042566+0000","last_undegraded":"2024-09-17T04:49:42.042566+0000","last_fullsized":"2024-09-17T04:49:42.042566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054288+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T04:49:42.054288+0000","last_peered":"2024-09-17T04:49:42.054288+0000","last_clean":"2024-09-17T04:49:42.054288+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T04:49:42.054288+0000","last_undegraded":"2024-09-17T04:49:42.054288+0000","last_fullsized":"2024-09-17T04:49:42.054288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051153+0000","last_change":"2024-09-17T04:49:41.058280+0000","last_active":"2024-09-17T04:49:42.051153+0000","last_peered":"2024-09-17T04:49:42.051153+0000","last_clean":"2024-09-17T04:49:42.051153+0000","last_became_active":"2024-09-17T04:49:41.052009+0000","last_became_peered":"2024-09-17T04:49:41.052009+0000","last_unstale":"2024-09-17T04:49:42.051153+0000","last_undegraded":"2024-09-17T04:49:42.051153+0000","last_fullsized":"2024-09-17T04:49:42.051153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:14.100818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051977+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T04:49:42.051977+0000","last_peered":"2024-09-17T04:49:42.051977+0000","last_clean":"2024-09-17T04:49:42.051977+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T04:49:42.051977+0000","last_undegraded":"2024-09-17T04:49:42.051977+0000","last_fullsized":"2024-09-17T04:49:42.051977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669355+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T04:49:42.669355+0000","last_peered":"2024-09-17T04:49:42.669355+0000","last_clean":"2024-09-17T04:49:42.669355+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T04:49:42.669355+0000","last_undegraded":"2024-09-17T04:49:42.669355+0000","last_fullsized":"2024-09-17T04:49:42.669355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042782+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T04:49:42.042782+0000","last_peered":"2024-09-17T04:49:42.042782+0000","last_clean":"2024-09-17T04:49:42.042782+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T04:49:42.042782+0000","last_undegraded":"2024-09-17T04:49:42.042782+0000","last_fullsized":"2024-09-17T04:49:42.042782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:07.945355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052280+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T04:49:42.052280+0000","last_peered":"2024-09-17T04:49:42.052280+0000","last_clean":"2024-09-17T04:49:42.052280+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T04:49:42.052280+0000","last_undegraded":"2024-09-17T04:49:42.052280+0000","last_fullsized":"2024-09-17T04:49:42.052280+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670412+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T04:49:42.670412+0000","last_peered":"2024-09-17T04:49:42.670412+0000","last_clean":"2024-09-17T04:49:42.670412+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T04:49:42.670412+0000","last_undegraded":"2024-09-17T04:49:42.670412+0000","last_fullsized":"2024-09-17T04:49:42.670412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537054+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T04:49:42.537054+0000","last_peered":"2024-09-17T04:49:42.537054+0000","last_clean":"2024-09-17T04:49:42.537054+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T04:49:42.537054+0000","last_undegraded":"2024-09-17T04:49:42.537054+0000","last_fullsized":"2024-09-17T04:49:42.537054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048913+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T04:49:42.048913+0000","last_peered":"2024-09-17T04:49:42.048913+0000","last_clean":"2024-09-17T04:49:42.048913+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T04:49:42.048913+0000","last_undegraded":"2024-09-17T04:49:42.048913+0000","last_fullsized":"2024-09-17T04:49:42.048913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:16.447835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051373+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T04:49:42.051373+0000","last_peered":"2024-09-17T04:49:42.051373+0000","last_clean":"2024-09-17T04:49:42.051373+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T04:49:42.051373+0000","last_undegraded":"2024-09-17T04:49:42.051373+0000","last_fullsized":"2024-09-17T04:49:42.051373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042063+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T04:49:42.042063+0000","last_peered":"2024-09-17T04:49:42.042063+0000","last_clean":"2024-09-17T04:49:42.042063+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T04:49:42.042063+0000","last_undegraded":"2024-09-17T04:49:42.042063+0000","last_fullsized":"2024-09-17T04:49:42.042063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670596+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T04:49:42.670596+0000","last_peered":"2024-09-17T04:49:42.670596+0000","last_clean":"2024-09-17T04:49:42.670596+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T04:49:42.670596+0000","last_undegraded":"2024-09-17T04:49:42.670596+0000","last_fullsized":"2024-09-17T04:49:42.670596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535510+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T04:49:42.535510+0000","last_peered":"2024-09-17T04:49:42.535510+0000","last_clean":"2024-09-17T04:49:42.535510+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T04:49:42.535510+0000","last_undegraded":"2024-09-17T04:49:42.535510+0000","last_fullsized":"2024-09-17T04:49:42.535510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041322+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T04:49:42.041322+0000","last_peered":"2024-09-17T04:49:42.041322+0000","last_clean":"2024-09-17T04:49:42.041322+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T04:49:42.041322+0000","last_undegraded":"2024-09-17T04:49:42.041322+0000","last_fullsized":"2024-09-17T04:49:42.041322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672182+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T04:49:42.672182+0000","last_peered":"2024-09-17T04:49:42.672182+0000","last_clean":"2024-09-17T04:49:42.672182+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T04:49:42.672182+0000","last_undegraded":"2024-09-17T04:49:42.672182+0000","last_fullsized":"2024-09-17T04:49:42.672182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671261+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T04:49:42.671261+0000","last_peered":"2024-09-17T04:49:42.671261+0000","last_clean":"2024-09-17T04:49:42.671261+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T04:49:42.671261+0000","last_undegraded":"2024-09-17T04:49:42.671261+0000","last_fullsized":"2024-09-17T04:49:42.671261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667575+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T04:49:42.667575+0000","last_peered":"2024-09-17T04:49:42.667575+0000","last_clean":"2024-09-17T04:49:42.667575+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T04:49:42.667575+0000","last_undegraded":"2024-09-17T04:49:42.667575+0000","last_fullsized":"2024-09-17T04:49:42.667575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:40.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534654+0000","last_change":"2024-09-17T04:49:41.040278+0000","last_active":"2024-09-17T04:49:42.534654+0000","last_peered":"2024-09-17T04:49:42.534654+0000","last_clean":"2024-09-17T04:49:42.534654+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T04:49:42.534654+0000","last_undegraded":"2024-09-17T04:49:42.534654+0000","last_fullsized":"2024-09-17T04:49:42.534654+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.895223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536732+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T04:49:42.536732+0000","last_peered":"2024-09-17T04:49:42.536732+0000","last_clean":"2024-09-17T04:49:42.536732+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T04:49:42.536732+0000","last_undegraded":"2024-09-17T04:49:42.536732+0000","last_fullsized":"2024-09-17T04:49:42.536732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675753+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T04:49:42.675753+0000","last_peered":"2024-09-17T04:49:42.675753+0000","last_clean":"2024-09-17T04:49:42.675753+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T04:49:42.675753+0000","last_undegraded":"2024-09-17T04:49:42.675753+0000","last_fullsized":"2024-09-17T04:49:42.675753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047646+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T04:49:42.047646+0000","last_peered":"2024-09-17T04:49:42.047646+0000","last_clean":"2024-09-17T04:49:42.047646+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T04:49:42.047646+0000","last_undegraded":"2024-09-17T04:49:42.047646+0000","last_fullsized":"2024-09-17T04:49:42.047646+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:15.845612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537298+0000","last_change":"2024-09-17T04:49:41.050551+0000","last_active":"2024-09-17T04:49:42.537298+0000","last_peered":"2024-09-17T04:49:42.537298+0000","last_clean":"2024-09-17T04:49:42.537298+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T04:49:42.537298+0000","last_undegraded":"2024-09-17T04:49:42.537298+0000","last_fullsized":"2024-09-17T04:49:42.537298+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:11.088230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051592+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T04:49:42.051592+0000","last_peered":"2024-09-17T04:49:42.051592+0000","last_clean":"2024-09-17T04:49:42.051592+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T04:49:42.051592+0000","last_undegraded":"2024-09-17T04:49:42.051592+0000","last_fullsized":"2024-09-17T04:49:42.051592+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:37.778756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048401+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T04:49:42.048401+0000","last_peered":"2024-09-17T04:49:42.048401+0000","last_clean":"2024-09-17T04:49:42.048401+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T04:49:42.048401+0000","last_undegraded":"2024-09-17T04:49:42.048401+0000","last_fullsized":"2024-09-17T04:49:42.048401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041797+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T04:49:42.041797+0000","last_peered":"2024-09-17T04:49:42.041797+0000","last_clean":"2024-09-17T04:49:42.041797+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T04:49:42.041797+0000","last_undegraded":"2024-09-17T04:49:42.041797+0000","last_fullsized":"2024-09-17T04:49:42.041797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055365+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T04:49:42.055365+0000","last_peered":"2024-09-17T04:49:42.055365+0000","last_clean":"2024-09-17T04:49:42.055365+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T04:49:42.055365+0000","last_undegraded":"2024-09-17T04:49:42.055365+0000","last_fullsized":"2024-09-17T04:49:42.055365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041767+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T04:49:42.041767+0000","last_peered":"2024-09-17T04:49:42.041767+0000","last_clean":"2024-09-17T04:49:42.041767+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T04:49:42.041767+0000","last_undegraded":"2024-09-17T04:49:42.041767+0000","last_fullsized":"2024-09-17T04:49:42.041767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667708+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T04:49:42.667708+0000","last_peered":"2024-09-17T04:49:42.667708+0000","last_clean":"2024-09-17T04:49:42.667708+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T04:49:42.667708+0000","last_undegraded":"2024-09-17T04:49:42.667708+0000","last_fullsized":"2024-09-17T04:49:42.667708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050522+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T04:49:42.050522+0000","last_peered":"2024-09-17T04:49:42.050522+0000","last_clean":"2024-09-17T04:49:42.050522+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T04:49:42.050522+0000","last_undegraded":"2024-09-17T04:49:42.050522+0000","last_fullsized":"2024-09-17T04:49:42.050522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049954+0000","last_change":"2024-09-17T04:49:41.057647+0000","last_active":"2024-09-17T04:49:42.049954+0000","last_peered":"2024-09-17T04:49:42.049954+0000","last_clean":"2024-09-17T04:49:42.049954+0000","last_became_active":"2024-09-17T04:49:41.056213+0000","last_became_peered":"2024-09-17T04:49:41.056213+0000","last_unstale":"2024-09-17T04:49:42.049954+0000","last_undegraded":"2024-09-17T04:49:42.049954+0000","last_fullsized":"2024-09-17T04:49:42.049954+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042498+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T04:49:42.042498+0000","last_peered":"2024-09-17T04:49:42.042498+0000","last_clean":"2024-09-17T04:49:42.042498+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T04:49:42.042498+0000","last_undegraded":"2024-09-17T04:49:42.042498+0000","last_fullsized":"2024-09-17T04:49:42.042498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053714+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T04:49:42.053714+0000","last_peered":"2024-09-17T04:49:42.053714+0000","last_clean":"2024-09-17T04:49:42.053714+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T04:49:42.053714+0000","last_undegraded":"2024-09-17T04:49:42.053714+0000","last_fullsized":"2024-09-17T04:49:42.053714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051282+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T04:49:42.051282+0000","last_peered":"2024-09-17T04:49:42.051282+0000","last_clean":"2024-09-17T04:49:42.051282+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T04:49:42.051282+0000","last_undegraded":"2024-09-17T04:49:42.051282+0000","last_fullsized":"2024-09-17T04:49:42.051282+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:27.230604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536526+0000","last_change":"2024-09-17T04:49:41.055263+0000","last_active":"2024-09-17T04:49:42.536526+0000","last_peered":"2024-09-17T04:49:42.536526+0000","last_clean":"2024-09-17T04:49:42.536526+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T04:49:42.536526+0000","last_undegraded":"2024-09-17T04:49:42.536526+0000","last_fullsized":"2024-09-17T04:49:42.536526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670438+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T04:49:42.670438+0000","last_peered":"2024-09-17T04:49:42.670438+0000","last_clean":"2024-09-17T04:49:42.670438+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T04:49:42.670438+0000","last_undegraded":"2024-09-17T04:49:42.670438+0000","last_fullsized":"2024-09-17T04:49:42.670438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673048+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T04:49:42.673048+0000","last_peered":"2024-09-17T04:49:42.673048+0000","last_clean":"2024-09-17T04:49:42.673048+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T04:49:42.673048+0000","last_undegraded":"2024-09-17T04:49:42.673048+0000","last_fullsized":"2024-09-17T04:49:42.673048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670447+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T04:49:42.670447+0000","last_peered":"2024-09-17T04:49:42.670447+0000","last_clean":"2024-09-17T04:49:42.670447+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T04:49:42.670447+0000","last_undegraded":"2024-09-17T04:49:42.670447+0000","last_fullsized":"2024-09-17T04:49:42.670447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:24.586834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051128+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.051128+0000","last_peered":"2024-09-17T04:49:42.051128+0000","last_clean":"2024-09-17T04:49:42.051128+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T04:49:42.051128+0000","last_undegraded":"2024-09-17T04:49:42.051128+0000","last_fullsized":"2024-09-17T04:49:42.051128+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536163+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T04:49:42.536163+0000","last_peered":"2024-09-17T04:49:42.536163+0000","last_clean":"2024-09-17T04:49:42.536163+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T04:49:42.536163+0000","last_undegraded":"2024-09-17T04:49:42.536163+0000","last_fullsized":"2024-09-17T04:49:42.536163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052121+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T04:49:42.052121+0000","last_peered":"2024-09-17T04:49:42.052121+0000","last_clean":"2024-09-17T04:49:42.052121+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T04:49:42.052121+0000","last_undegraded":"2024-09-17T04:49:42.052121+0000","last_fullsized":"2024-09-17T04:49:42.052121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668467+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T04:49:42.668467+0000","last_peered":"2024-09-17T04:49:42.668467+0000","last_clean":"2024-09-17T04:49:42.668467+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T04:49:42.668467+0000","last_undegraded":"2024-09-17T04:49:42.668467+0000","last_fullsized":"2024-09-17T04:49:42.668467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536096+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T04:49:42.536096+0000","last_peered":"2024-09-17T04:49:42.536096+0000","last_clean":"2024-09-17T04:49:42.536096+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T04:49:42.536096+0000","last_undegraded":"2024-09-17T04:49:42.536096+0000","last_fullsized":"2024-09-17T04:49:42.536096+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042322+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T04:49:42.042322+0000","last_peered":"2024-09-17T04:49:42.042322+0000","last_clean":"2024-09-17T04:49:42.042322+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T04:49:42.042322+0000","last_undegraded":"2024-09-17T04:49:42.042322+0000","last_fullsized":"2024-09-17T04:49:42.042322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671200+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T04:49:42.671200+0000","last_peered":"2024-09-17T04:49:42.671200+0000","last_clean":"2024-09-17T04:49:42.671200+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T04:49:42.671200+0000","last_undegraded":"2024-09-17T04:49:42.671200+0000","last_fullsized":"2024-09-17T04:49:42.671200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670309+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T04:49:42.670309+0000","last_peered":"2024-09-17T04:49:42.670309+0000","last_clean":"2024-09-17T04:49:42.670309+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T04:49:42.670309+0000","last_undegraded":"2024-09-17T04:49:42.670309+0000","last_fullsized":"2024-09-17T04:49:42.670309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668969+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T04:49:42.668969+0000","last_peered":"2024-09-17T04:49:42.668969+0000","last_clean":"2024-09-17T04:49:42.668969+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T04:49:42.668969+0000","last_undegraded":"2024-09-17T04:49:42.668969+0000","last_fullsized":"2024-09-17T04:49:42.668969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054935+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T04:49:42.054935+0000","last_peered":"2024-09-17T04:49:42.054935+0000","last_clean":"2024-09-17T04:49:42.054935+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T04:49:42.054935+0000","last_undegraded":"2024-09-17T04:49:42.054935+0000","last_fullsized":"2024-09-17T04:49:42.054935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669059+0000","last_change":"2024-09-17T04:49:41.073709+0000","last_active":"2024-09-17T04:49:42.669059+0000","last_peered":"2024-09-17T04:49:42.669059+0000","last_clean":"2024-09-17T04:49:42.669059+0000","last_became_active":"2024-09-17T04:49:41.073330+0000","last_became_peered":"2024-09-17T04:49:41.073330+0000","last_unstale":"2024-09-17T04:49:42.669059+0000","last_undegraded":"2024-09-17T04:49:42.669059+0000","last_fullsized":"2024-09-17T04:49:42.669059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054811+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T04:49:42.054811+0000","last_peered":"2024-09-17T04:49:42.054811+0000","last_clean":"2024-09-17T04:49:42.054811+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T04:49:42.054811+0000","last_undegraded":"2024-09-17T04:49:42.054811+0000","last_fullsized":"2024-09-17T04:49:42.054811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041256+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T04:49:42.041256+0000","last_peered":"2024-09-17T04:49:42.041256+0000","last_clean":"2024-09-17T04:49:42.041256+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T04:49:42.041256+0000","last_undegraded":"2024-09-17T04:49:42.041256+0000","last_fullsized":"2024-09-17T04:49:42.041256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536669+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T04:49:42.536669+0000","last_peered":"2024-09-17T04:49:42.536669+0000","last_clean":"2024-09-17T04:49:42.536669+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T04:49:42.536669+0000","last_undegraded":"2024-09-17T04:49:42.536669+0000","last_fullsized":"2024-09-17T04:49:42.536669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534968+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T04:49:42.534968+0000","last_peered":"2024-09-17T04:49:42.534968+0000","last_clean":"2024-09-17T04:49:42.534968+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T04:49:42.534968+0000","last_undegraded":"2024-09-17T04:49:42.534968+0000","last_fullsized":"2024-09-17T04:49:42.534968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672019+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T04:49:42.672019+0000","last_peered":"2024-09-17T04:49:42.672019+0000","last_clean":"2024-09-17T04:49:42.672019+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T04:49:42.672019+0000","last_undegraded":"2024-09-17T04:49:42.672019+0000","last_fullsized":"2024-09-17T04:49:42.672019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668565+0000","last_change":"2024-09-17T04:49:41.045502+0000","last_active":"2024-09-17T04:49:42.668565+0000","last_peered":"2024-09-17T04:49:42.668565+0000","last_clean":"2024-09-17T04:49:42.668565+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T04:49:42.668565+0000","last_undegraded":"2024-09-17T04:49:42.668565+0000","last_fullsized":"2024-09-17T04:49:42.668565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:19.746356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041666+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T04:49:42.041666+0000","last_peered":"2024-09-17T04:49:42.041666+0000","last_clean":"2024-09-17T04:49:42.041666+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T04:49:42.041666+0000","last_undegraded":"2024-09-17T04:49:42.041666+0000","last_fullsized":"2024-09-17T04:49:42.041666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042310+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T04:49:42.042310+0000","last_peered":"2024-09-17T04:49:42.042310+0000","last_clean":"2024-09-17T04:49:42.042310+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T04:49:42.042310+0000","last_undegraded":"2024-09-17T04:49:42.042310+0000","last_fullsized":"2024-09-17T04:49:42.042310+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668045+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T04:49:42.668045+0000","last_peered":"2024-09-17T04:49:42.668045+0000","last_clean":"2024-09-17T04:49:42.668045+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T04:49:42.668045+0000","last_undegraded":"2024-09-17T04:49:42.668045+0000","last_fullsized":"2024-09-17T04:49:42.668045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054699+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T04:49:42.054699+0000","last_peered":"2024-09-17T04:49:42.054699+0000","last_clean":"2024-09-17T04:49:42.054699+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T04:49:42.054699+0000","last_undegraded":"2024-09-17T04:49:42.054699+0000","last_fullsized":"2024-09-17T04:49:42.054699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673038+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T04:49:42.673038+0000","last_peered":"2024-09-17T04:49:42.673038+0000","last_clean":"2024-09-17T04:49:42.673038+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T04:49:42.673038+0000","last_undegraded":"2024-09-17T04:49:42.673038+0000","last_fullsized":"2024-09-17T04:49:42.673038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669104+0000","last_change":"2024-09-17T04:49:41.038265+0000","last_active":"2024-09-17T04:49:42.669104+0000","last_peered":"2024-09-17T04:49:42.669104+0000","last_clean":"2024-09-17T04:49:42.669104+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T04:49:42.669104+0000","last_undegraded":"2024-09-17T04:49:42.669104+0000","last_fullsized":"2024-09-17T04:49:42.669104+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:41.638403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053416+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T04:49:42.053416+0000","last_peered":"2024-09-17T04:49:42.053416+0000","last_clean":"2024-09-17T04:49:42.053416+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T04:49:42.053416+0000","last_undegraded":"2024-09-17T04:49:42.053416+0000","last_fullsized":"2024-09-17T04:49:42.053416+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535001+0000","last_change":"2024-09-17T04:49:41.053443+0000","last_active":"2024-09-17T04:49:42.535001+0000","last_peered":"2024-09-17T04:49:42.535001+0000","last_clean":"2024-09-17T04:49:42.535001+0000","last_became_active":"2024-09-17T04:49:41.053090+0000","last_became_peered":"2024-09-17T04:49:41.053090+0000","last_unstale":"2024-09-17T04:49:42.535001+0000","last_undegraded":"2024-09-17T04:49:42.535001+0000","last_fullsized":"2024-09-17T04:49:42.535001+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671599+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T04:49:42.671599+0000","last_peered":"2024-09-17T04:49:42.671599+0000","last_clean":"2024-09-17T04:49:42.671599+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T04:49:42.671599+0000","last_undegraded":"2024-09-17T04:49:42.671599+0000","last_fullsized":"2024-09-17T04:49:42.671599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669669+0000","last_change":"2024-09-17T04:49:41.038072+0000","last_active":"2024-09-17T04:49:42.669669+0000","last_peered":"2024-09-17T04:49:42.669669+0000","last_clean":"2024-09-17T04:49:42.669669+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T04:49:42.669669+0000","last_undegraded":"2024-09-17T04:49:42.669669+0000","last_fullsized":"2024-09-17T04:49:42.669669+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043492+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T04:49:42.043492+0000","last_peered":"2024-09-17T04:49:42.043492+0000","last_clean":"2024-09-17T04:49:42.043492+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T04:49:42.043492+0000","last_undegraded":"2024-09-17T04:49:42.043492+0000","last_fullsized":"2024-09-17T04:49:42.043492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534462+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T04:49:42.534462+0000","last_peered":"2024-09-17T04:49:42.534462+0000","last_clean":"2024-09-17T04:49:42.534462+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T04:49:42.534462+0000","last_undegraded":"2024-09-17T04:49:42.534462+0000","last_fullsized":"2024-09-17T04:49:42.534462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669602+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T04:49:42.669602+0000","last_peered":"2024-09-17T04:49:42.669602+0000","last_clean":"2024-09-17T04:49:42.669602+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T04:49:42.669602+0000","last_undegraded":"2024-09-17T04:49:42.669602+0000","last_fullsized":"2024-09-17T04:49:42.669602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050316+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T04:49:42.050316+0000","last_peered":"2024-09-17T04:49:42.050316+0000","last_clean":"2024-09-17T04:49:42.050316+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T04:49:42.050316+0000","last_undegraded":"2024-09-17T04:49:42.050316+0000","last_fullsized":"2024-09-17T04:49:42.050316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042123+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T04:49:42.042123+0000","last_peered":"2024-09-17T04:49:42.042123+0000","last_clean":"2024-09-17T04:49:42.042123+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T04:49:42.042123+0000","last_undegraded":"2024-09-17T04:49:42.042123+0000","last_fullsized":"2024-09-17T04:49:42.042123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052378+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T04:49:42.052378+0000","last_peered":"2024-09-17T04:49:42.052378+0000","last_clean":"2024-09-17T04:49:42.052378+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T04:49:42.052378+0000","last_undegraded":"2024-09-17T04:49:42.052378+0000","last_fullsized":"2024-09-17T04:49:42.052378+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041588+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T04:49:42.041588+0000","last_peered":"2024-09-17T04:49:42.041588+0000","last_clean":"2024-09-17T04:49:42.041588+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T04:49:42.041588+0000","last_undegraded":"2024-09-17T04:49:42.041588+0000","last_fullsized":"2024-09-17T04:49:42.041588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674897+0000","last_change":"2024-09-17T04:49:41.067207+0000","last_active":"2024-09-17T04:49:42.674897+0000","last_peered":"2024-09-17T04:49:42.674897+0000","last_clean":"2024-09-17T04:49:42.674897+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T04:49:42.674897+0000","last_undegraded":"2024-09-17T04:49:42.674897+0000","last_fullsized":"2024-09-17T04:49:42.674897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:08.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667484+0000","last_change":"2024-09-17T04:49:31.911036+0000","last_active":"2024-09-17T04:49:42.667484+0000","last_peered":"2024-09-17T04:49:42.667484+0000","last_clean":"2024-09-17T04:49:42.667484+0000","last_became_active":"2024-09-17T04:49:31.910285+0000","last_became_peered":"2024-09-17T04:49:31.910285+0000","last_unstale":"2024-09-17T04:49:42.667484+0000","last_undegraded":"2024-09-17T04:49:42.667484+0000","last_fullsized":"2024-09-17T04:49:42.667484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673474+0000","last_change":"2024-09-17T04:49:41.046170+0000","last_active":"2024-09-17T04:49:42.673474+0000","last_peered":"2024-09-17T04:49:42.673474+0000","last_clean":"2024-09-17T04:49:42.673474+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T04:49:42.673474+0000","last_undegraded":"2024-09-17T04:49:42.673474+0000","last_fullsized":"2024-09-17T04:49:42.673474+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:01.683826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051528+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T04:49:42.051528+0000","last_peered":"2024-09-17T04:49:42.051528+0000","last_clean":"2024-09-17T04:49:42.051528+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T04:49:42.051528+0000","last_undegraded":"2024-09-17T04:49:42.051528+0000","last_fullsized":"2024-09-17T04:49:42.051528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041843+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T04:49:42.041843+0000","last_peered":"2024-09-17T04:49:42.041843+0000","last_clean":"2024-09-17T04:49:42.041843+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T04:49:42.041843+0000","last_undegraded":"2024-09-17T04:49:42.041843+0000","last_fullsized":"2024-09-17T04:49:42.041843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:25.574682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053942+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T04:49:42.053942+0000","last_peered":"2024-09-17T04:49:42.053942+0000","last_clean":"2024-09-17T04:49:42.053942+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T04:49:42.053942+0000","last_undegraded":"2024-09-17T04:49:42.053942+0000","last_fullsized":"2024-09-17T04:49:42.053942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048414+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T04:49:42.048414+0000","last_peered":"2024-09-17T04:49:42.048414+0000","last_clean":"2024-09-17T04:49:42.048414+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T04:49:42.048414+0000","last_undegraded":"2024-09-17T04:49:42.048414+0000","last_fullsized":"2024-09-17T04:49:42.048414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671218+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T04:49:42.671218+0000","last_peered":"2024-09-17T04:49:42.671218+0000","last_clean":"2024-09-17T04:49:42.671218+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T04:49:42.671218+0000","last_undegraded":"2024-09-17T04:49:42.671218+0000","last_fullsized":"2024-09-17T04:49:42.671218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053498+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T04:49:42.053498+0000","last_peered":"2024-09-17T04:49:42.053498+0000","last_clean":"2024-09-17T04:49:42.053498+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T04:49:42.053498+0000","last_undegraded":"2024-09-17T04:49:42.053498+0000","last_fullsized":"2024-09-17T04:49:42.053498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670119+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T04:49:42.670119+0000","last_peered":"2024-09-17T04:49:42.670119+0000","last_clean":"2024-09-17T04:49:42.670119+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T04:49:42.670119+0000","last_undegraded":"2024-09-17T04:49:42.670119+0000","last_fullsized":"2024-09-17T04:49:42.670119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668585+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T04:49:42.668585+0000","last_peered":"2024-09-17T04:49:42.668585+0000","last_clean":"2024-09-17T04:49:42.668585+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T04:49:42.668585+0000","last_undegraded":"2024-09-17T04:49:42.668585+0000","last_fullsized":"2024-09-17T04:49:42.668585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670988+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T04:49:42.670988+0000","last_peered":"2024-09-17T04:49:42.670988+0000","last_clean":"2024-09-17T04:49:42.670988+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T04:49:42.670988+0000","last_undegraded":"2024-09-17T04:49:42.670988+0000","last_fullsized":"2024-09-17T04:49:42.670988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671416+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.671416+0000","last_peered":"2024-09-17T04:49:42.671416+0000","last_clean":"2024-09-17T04:49:42.671416+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T04:49:42.671416+0000","last_undegraded":"2024-09-17T04:49:42.671416+0000","last_fullsized":"2024-09-17T04:49:42.671416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675766+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T04:49:42.675766+0000","last_peered":"2024-09-17T04:49:42.675766+0000","last_clean":"2024-09-17T04:49:42.675766+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T04:49:42.675766+0000","last_undegraded":"2024-09-17T04:49:42.675766+0000","last_fullsized":"2024-09-17T04:49:42.675766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671330+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T04:49:42.671330+0000","last_peered":"2024-09-17T04:49:42.671330+0000","last_clean":"2024-09-17T04:49:42.671330+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T04:49:42.671330+0000","last_undegraded":"2024-09-17T04:49:42.671330+0000","last_fullsized":"2024-09-17T04:49:42.671330+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667310+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T04:49:42.667310+0000","last_peered":"2024-09-17T04:49:42.667310+0000","last_clean":"2024-09-17T04:49:42.667310+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T04:49:42.667310+0000","last_undegraded":"2024-09-17T04:49:42.667310+0000","last_fullsized":"2024-09-17T04:49:42.667310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537133+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T04:49:42.537133+0000","last_peered":"2024-09-17T04:49:42.537133+0000","last_clean":"2024-09-17T04:49:42.537133+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T04:49:42.537133+0000","last_undegraded":"2024-09-17T04:49:42.537133+0000","last_fullsized":"2024-09-17T04:49:42.537133+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669347+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T04:49:42.669347+0000","last_peered":"2024-09-17T04:49:42.669347+0000","last_clean":"2024-09-17T04:49:42.669347+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T04:49:42.669347+0000","last_undegraded":"2024-09-17T04:49:42.669347+0000","last_fullsized":"2024-09-17T04:49:42.669347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050809+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T04:49:42.050809+0000","last_peered":"2024-09-17T04:49:42.050809+0000","last_clean":"2024-09-17T04:49:42.050809+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T04:49:42.050809+0000","last_undegraded":"2024-09-17T04:49:42.050809+0000","last_fullsized":"2024-09-17T04:49:42.050809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535865+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T04:49:42.535865+0000","last_peered":"2024-09-17T04:49:42.535865+0000","last_clean":"2024-09-17T04:49:42.535865+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T04:49:42.535865+0000","last_undegraded":"2024-09-17T04:49:42.535865+0000","last_fullsized":"2024-09-17T04:49:42.535865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673274+0000","last_change":"2024-09-17T04:49:41.041201+0000","last_active":"2024-09-17T04:49:42.673274+0000","last_peered":"2024-09-17T04:49:42.673274+0000","last_clean":"2024-09-17T04:49:42.673274+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T04:49:42.673274+0000","last_undegraded":"2024-09-17T04:49:42.673274+0000","last_fullsized":"2024-09-17T04:49:42.673274+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:02.588786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670272+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T04:49:42.670272+0000","last_peered":"2024-09-17T04:49:42.670272+0000","last_clean":"2024-09-17T04:49:42.670272+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T04:49:42.670272+0000","last_undegraded":"2024-09-17T04:49:42.670272+0000","last_fullsized":"2024-09-17T04:49:42.670272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667767+0000","last_change":"2024-09-17T04:49:41.061092+0000","last_active":"2024-09-17T04:49:42.667767+0000","last_peered":"2024-09-17T04:49:42.667767+0000","last_clean":"2024-09-17T04:49:42.667767+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T04:49:42.667767+0000","last_undegraded":"2024-09-17T04:49:42.667767+0000","last_fullsized":"2024-09-17T04:49:42.667767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:52.714614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536820+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T04:49:42.536820+0000","last_peered":"2024-09-17T04:49:42.536820+0000","last_clean":"2024-09-17T04:49:42.536820+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T04:49:42.536820+0000","last_undegraded":"2024-09-17T04:49:42.536820+0000","last_fullsized":"2024-09-17T04:49:42.536820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669397+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T04:49:42.669397+0000","last_peered":"2024-09-17T04:49:42.669397+0000","last_clean":"2024-09-17T04:49:42.669397+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T04:49:42.669397+0000","last_undegraded":"2024-09-17T04:49:42.669397+0000","last_fullsized":"2024-09-17T04:49:42.669397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671170+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T04:49:42.671170+0000","last_peered":"2024-09-17T04:49:42.671170+0000","last_clean":"2024-09-17T04:49:42.671170+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T04:49:42.671170+0000","last_undegraded":"2024-09-17T04:49:42.671170+0000","last_fullsized":"2024-09-17T04:49:42.671170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536018+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T04:49:42.536018+0000","last_peered":"2024-09-17T04:49:42.536018+0000","last_clean":"2024-09-17T04:49:42.536018+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T04:49:42.536018+0000","last_undegraded":"2024-09-17T04:49:42.536018+0000","last_fullsized":"2024-09-17T04:49:42.536018+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671602+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T04:49:42.671602+0000","last_peered":"2024-09-17T04:49:42.671602+0000","last_clean":"2024-09-17T04:49:42.671602+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T04:49:42.671602+0000","last_undegraded":"2024-09-17T04:49:42.671602+0000","last_fullsized":"2024-09-17T04:49:42.671602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534962+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T04:49:42.534962+0000","last_peered":"2024-09-17T04:49:42.534962+0000","last_clean":"2024-09-17T04:49:42.534962+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T04:49:42.534962+0000","last_undegraded":"2024-09-17T04:49:42.534962+0000","last_fullsized":"2024-09-17T04:49:42.534962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671825+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T04:49:42.671825+0000","last_peered":"2024-09-17T04:49:42.671825+0000","last_clean":"2024-09-17T04:49:42.671825+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T04:49:42.671825+0000","last_undegraded":"2024-09-17T04:49:42.671825+0000","last_fullsized":"2024-09-17T04:49:42.671825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509448,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":307200,"data_stored":108537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":286720,"data_stored":106265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509449,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509449,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:49:47.065 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T04:49:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 -- 172.21.15.150:0/3379894193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a04004d50 msgr2=0x7f4a04005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 --2- 172.21.15.150:0/3379894193 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a04004d50 0x7f4a04005200 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c075620 tx=0x7f4a1401cad0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a1ad9d640 1 -- 172.21.15.150:0/3379894193 reap_dead start 2024-09-17T04:49:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 -- 172.21.15.150:0/3379894193 shutdown_connections 2024-09-17T04:49:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 -- 172.21.15.150:0/3379894193 >> 172.21.15.150:0/3379894193 conn(0x7f4a1c06d450 msgr2=0x7f4a1c0a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 -- 172.21.15.150:0/3379894193 shutdown_connections 2024-09-17T04:49:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 -- 172.21.15.150:0/3379894193 wait complete. 2024-09-17T04:49:47.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 Processor -- start 2024-09-17T04:49:47.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 -- start start 2024-09-17T04:49:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a04004d50 0x7f4a1c14f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a1c0a79e0 0x7f4a1c0c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a1c14f970 0x7f4a1c14fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c0733d0 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1c0c44b0 con 0x7f4a04004d50 2024-09-17T04:49:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a20998640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4a1c0c2730 con 0x7f4a1c14f970 2024-09-17T04:49:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a19d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a1c0a79e0 0x7f4a1c0c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a19d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a1c0a79e0 0x7f4a1c0c1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53284/0 (socket says 172.21.15.150:53284) 2024-09-17T04:49:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a19d9b640 1 -- 172.21.15.150:0/2380297310 learned_addr learned my addr 172.21.15.150:0/2380297310 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a1ad9d640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a1c14f970 0x7f4a1c14fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a1a59c640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a04004d50 0x7f4a1c14f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a19d9b640 1 -- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a1c14f970 msgr2=0x7f4a1c14fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a19d9b640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a1c14f970 0x7f4a1c14fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a19d9b640 1 -- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a04004d50 msgr2=0x7f4a1c14f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a19d9b640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a04004d50 0x7f4a1c14f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a19d9b640 1 -- 172.21.15.150:0/2380297310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1c14f540 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.256+0000 7f4a1ad9d640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a1c14f970 0x7f4a1c14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.260+0000 7f4a19d9b640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a1c0a79e0 0x7f4a1c0c1330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4a10010a40 tx=0x7f4a10010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.260+0000 7f4a0affd640 1 -- 172.21.15.150:0/2380297310 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1000ecc0 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.260+0000 7f4a0affd640 1 -- 172.21.15.150:0/2380297310 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a10012450 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.260+0000 7f4a20998640 1 -- 172.21.15.150:0/2380297310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a1c150ac0 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.260+0000 7f4a0affd640 1 -- 172.21.15.150:0/2380297310 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a100128a0 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.260+0000 7f4a20998640 1 -- 172.21.15.150:0/2380297310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a1c14dff0 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.260+0000 7f4a20998640 1 -- 172.21.15.150:0/2380297310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a1c0c7150 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.260+0000 7f4a0affd640 1 -- 172.21.15.150:0/2380297310 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4a10012a40 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.260+0000 7f4a0affd640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f49fc073200 0x7f49fc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:47.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.260+0000 7f4a0affd640 1 -- 172.21.15.150:0/2380297310 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f4a10095380 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.260+0000 7f4a1a59c640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f49fc073200 0x7f49fc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:47.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.264+0000 7f4a1a59c640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f49fc073200 0x7f49fc075710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c001060 tx=0x7f4a0c0015d0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:47.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.264+0000 7f4a0affd640 1 -- 172.21.15.150:0/2380297310 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4a1c14f540 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.480+0000 7f4a20998640 1 -- 172.21.15.150:0/2380297310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f4a1c0c7460 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a0affd640 1 -- 172.21.15.150:0/2380297310 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1157 (secure 0 0 0) 0x7f4a10060190 con 0x7f4a1c0a79e0 2024-09-17T04:49:47.484 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:47.484 INFO:teuthology.orchestra.run.smithi150.stdout:{"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":41,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":26,"num_osds":8,"num_up_osds":8,"osd_up_since":1726548560,"num_in_osds":8,"osd_in_since":1726548550,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":193}],"num_pgs":193,"num_pools":5,"num_objects":46,"data_bytes":595188,"bytes_used":229183488,"bytes_avail":772864929792,"bytes_total":773094113280,"write_bytes_sec":2054,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","id":1,"up":1,"in":1,"max":1,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":4401},{"filesystem_id":1,"rank":0,"name":"c","status":"up:active","gid":4404}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-17T04:49:06.067546+0000","services":{}},"progress_events":{}} 2024-09-17T04:49:47.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a20998640 1 -- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f49fc073200 msgr2=0x7f49fc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:47.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a20998640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f49fc073200 0x7f49fc075710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c001060 tx=0x7f4a0c0015d0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a20998640 1 -- 172.21.15.150:0/2380297310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a1c0a79e0 msgr2=0x7f4a1c0c1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:47.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a20998640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a1c0a79e0 0x7f4a1c0c1330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4a10010a40 tx=0x7f4a10010f00 comp rx=0 tx=0).stop 2024-09-17T04:49:47.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a20998640 1 -- 172.21.15.150:0/2380297310 shutdown_connections 2024-09-17T04:49:47.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a20998640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f49fc073200 0x7f49fc075710 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a20998640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a1c14f970 0x7f4a1c14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a20998640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a1c0a79e0 0x7f4a1c0c1330 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a20998640 1 --2- 172.21.15.150:0/2380297310 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a04004d50 0x7f4a1c14f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a20998640 1 -- 172.21.15.150:0/2380297310 >> 172.21.15.150:0/2380297310 conn(0x7f4a1c06d450 msgr2=0x7f4a1c074b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:47.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a20998640 1 -- 172.21.15.150:0/2380297310 shutdown_connections 2024-09-17T04:49:47.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.484+0000 7f4a20998640 1 -- 172.21.15.150:0/2380297310 wait complete. 2024-09-17T04:49:47.505 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T04:49:47.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:49:47.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.644+0000 7f51ec668640 1 -- 172.21.15.150:0/4207062393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51e40c3b40 msgr2=0x7f51e4157c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:47.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.644+0000 7f51ec668640 1 --2- 172.21.15.150:0/4207062393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51e40c3b40 0x7f51e4157c30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009840 tx=0x7f51d801c7c0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.644+0000 7f51ec668640 1 -- 172.21.15.150:0/4207062393 shutdown_connections 2024-09-17T04:49:47.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.644+0000 7f51ec668640 1 --2- 172.21.15.150:0/4207062393 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51e4158170 0x7f51e4158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.644+0000 7f51ec668640 1 --2- 172.21.15.150:0/4207062393 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51e40c3b40 0x7f51e4157c30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.644+0000 7f51ec668640 1 -- 172.21.15.150:0/4207062393 >> 172.21.15.150:0/4207062393 conn(0x7f51e40c4a00 msgr2=0x7f51e40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:47.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.644+0000 7f51ec668640 1 -- 172.21.15.150:0/4207062393 shutdown_connections 2024-09-17T04:49:47.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.644+0000 7f51ec668640 1 -- 172.21.15.150:0/4207062393 wait complete. 2024-09-17T04:49:47.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.644+0000 7f51ec668640 1 Processor -- start 2024-09-17T04:49:47.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.644+0000 7f51ec668640 1 -- start start 2024-09-17T04:49:47.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51ec668640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51e40c3b40 0x7f51e414d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:47.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51ec668640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51e4151d00 0x7f51e41f3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:47.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51ec668640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51e4158170 0x7f51e414a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51ec668640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f51e40c3f50 con 0x7f51e4158170 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51ec668640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f51e40ade60 con 0x7f51e4151d00 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51ec668640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f51e40c40d0 con 0x7f51e40c3b40 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51eac05640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51e4158170 0x7f51e414a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51eac05640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51e4158170 0x7f51e414a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53298/0 (socket says 172.21.15.150:53298) 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51eac05640 1 -- 172.21.15.150:0/3453520550 learned_addr learned my addr 172.21.15.150:0/3453520550 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51e9c03640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51e4151d00 0x7f51e41f3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51ea404640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51e40c3b40 0x7f51e414d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51eac05640 1 -- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51e40c3b40 msgr2=0x7f51e414d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51eac05640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51e40c3b40 0x7f51e414d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51eac05640 1 -- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51e4151d00 msgr2=0x7f51e41f3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51eac05640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51e4151d00 0x7f51e41f3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51eac05640 1 -- 172.21.15.150:0/3453520550 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e414b1e0 con 0x7f51e4158170 2024-09-17T04:49:47.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51eac05640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51e4158170 0x7f51e414a1a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51d400e970 tx=0x7f51d400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:47.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51e2ffd640 1 -- 172.21.15.150:0/3453520550 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4004d30 con 0x7f51e4158170 2024-09-17T04:49:47.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51e2ffd640 1 -- 172.21.15.150:0/3453520550 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51d400cb20 con 0x7f51e4158170 2024-09-17T04:49:47.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51ec668640 1 -- 172.21.15.150:0/3453520550 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e414af40 con 0x7f51e4158170 2024-09-17T04:49:47.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51e2ffd640 1 -- 172.21.15.150:0/3453520550 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4005020 con 0x7f51e4158170 2024-09-17T04:49:47.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51ec668640 1 -- 172.21.15.150:0/3453520550 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e4075440 con 0x7f51e4158170 2024-09-17T04:49:47.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51ec668640 1 -- 172.21.15.150:0/3453520550 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51e414b6b0 con 0x7f51e4158170 2024-09-17T04:49:47.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51e2ffd640 1 -- 172.21.15.150:0/3453520550 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f51d40051c0 con 0x7f51e4158170 2024-09-17T04:49:47.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51e2ffd640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f51d0073230 0x7f51d0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:47.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51e2ffd640 1 -- 172.21.15.150:0/3453520550 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f51d4094ec0 con 0x7f51e4158170 2024-09-17T04:49:47.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.648+0000 7f51ea404640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f51d0073230 0x7f51d0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:47.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.652+0000 7f51ea404640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f51d0073230 0x7f51d0075740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f51dc000f70 tx=0x7f51dc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:47.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.652+0000 7f51e2ffd640 1 -- 172.21.15.150:0/3453520550 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f51d4016030 con 0x7f51e4158170 2024-09-17T04:49:47.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.820+0000 7f51ec668640 1 -- 172.21.15.150:0/3453520550 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f51e4157c30 con 0x7f51d0073230 2024-09-17T04:49:47.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.844+0000 7f51e2ffd640 1 -- 172.21.15.150:0/3453520550 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490420 (secure 0 0 0) 0x7f51e4157c30 con 0x7f51d0073230 2024-09-17T04:49:47.850 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:47.853 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T04:49:47.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.848+0000 7f51ec668640 1 -- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f51d0073230 msgr2=0x7f51d0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:47.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.848+0000 7f51ec668640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f51d0073230 0x7f51d0075740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f51dc000f70 tx=0x7f51dc009040 comp rx=0 tx=0).stop 2024-09-17T04:49:47.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.848+0000 7f51ec668640 1 -- 172.21.15.150:0/3453520550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51e4158170 msgr2=0x7f51e414a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:47.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.848+0000 7f51ec668640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51e4158170 0x7f51e414a1a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51d400e970 tx=0x7f51d400ee30 comp rx=0 tx=0).stop 2024-09-17T04:49:47.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.848+0000 7f51ec668640 1 -- 172.21.15.150:0/3453520550 shutdown_connections 2024-09-17T04:49:47.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.848+0000 7f51ec668640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f51d0073230 0x7f51d0075740 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.848+0000 7f51ec668640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51e4151d00 0x7f51e41f3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.848+0000 7f51ec668640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51e4158170 0x7f51e414a1a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.848+0000 7f51ec668640 1 --2- 172.21.15.150:0/3453520550 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51e40c3b40 0x7f51e414d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:47.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.848+0000 7f51ec668640 1 -- 172.21.15.150:0/3453520550 >> 172.21.15.150:0/3453520550 conn(0x7f51e40c4a00 msgr2=0x7f51e40c5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:47.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.848+0000 7f51ec668640 1 -- 172.21.15.150:0/3453520550 shutdown_connections 2024-09-17T04:49:47.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:47.848+0000 7f51ec668640 1 -- 172.21.15.150:0/3453520550 wait complete. 2024-09-17T04:49:47.872 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-17T04:49:46.946858+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":191,"num_write_kb":1656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":373,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223812,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750908,"statfs":{"total":773094113280,"available":772864929792,"internally_reserved":0,"allocated":3481600,"data_stored":2027328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":59,"apply_latency_ms":59,"commit_latency_ns":59000000,"apply_latency_ns":59000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.979341"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671077+0000","last_change":"2024-09-17T04:49:41.025271+0000","last_active":"2024-09-17T04:49:42.671077+0000","last_peered":"2024-09-17T04:49:42.671077+0000","last_clean":"2024-09-17T04:49:42.671077+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T04:49:42.671077+0000","last_undegraded":"2024-09-17T04:49:42.671077+0000","last_fullsized":"2024-09-17T04:49:42.671077+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:39.715602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054640+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T04:49:42.054640+0000","last_peered":"2024-09-17T04:49:42.054640+0000","last_clean":"2024-09-17T04:49:42.054640+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T04:49:42.054640+0000","last_undegraded":"2024-09-17T04:49:42.054640+0000","last_fullsized":"2024-09-17T04:49:42.054640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671706+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T04:49:42.671706+0000","last_peered":"2024-09-17T04:49:42.671706+0000","last_clean":"2024-09-17T04:49:42.671706+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T04:49:42.671706+0000","last_undegraded":"2024-09-17T04:49:42.671706+0000","last_fullsized":"2024-09-17T04:49:42.671706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668736+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T04:49:42.668736+0000","last_peered":"2024-09-17T04:49:42.668736+0000","last_clean":"2024-09-17T04:49:42.668736+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T04:49:42.668736+0000","last_undegraded":"2024-09-17T04:49:42.668736+0000","last_fullsized":"2024-09-17T04:49:42.668736+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:38.417840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047972+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T04:49:42.047972+0000","last_peered":"2024-09-17T04:49:42.047972+0000","last_clean":"2024-09-17T04:49:42.047972+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T04:49:42.047972+0000","last_undegraded":"2024-09-17T04:49:42.047972+0000","last_fullsized":"2024-09-17T04:49:42.047972+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672300+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T04:49:42.672300+0000","last_peered":"2024-09-17T04:49:42.672300+0000","last_clean":"2024-09-17T04:49:42.672300+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T04:49:42.672300+0000","last_undegraded":"2024-09-17T04:49:42.672300+0000","last_fullsized":"2024-09-17T04:49:42.672300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672014+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T04:49:42.672014+0000","last_peered":"2024-09-17T04:49:42.672014+0000","last_clean":"2024-09-17T04:49:42.672014+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T04:49:42.672014+0000","last_undegraded":"2024-09-17T04:49:42.672014+0000","last_fullsized":"2024-09-17T04:49:42.672014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535134+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T04:49:42.535134+0000","last_peered":"2024-09-17T04:49:42.535134+0000","last_clean":"2024-09-17T04:49:42.535134+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T04:49:42.535134+0000","last_undegraded":"2024-09-17T04:49:42.535134+0000","last_fullsized":"2024-09-17T04:49:42.535134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672206+0000","last_change":"2024-09-17T04:49:41.067317+0000","last_active":"2024-09-17T04:49:42.672206+0000","last_peered":"2024-09-17T04:49:42.672206+0000","last_clean":"2024-09-17T04:49:42.672206+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T04:49:42.672206+0000","last_undegraded":"2024-09-17T04:49:42.672206+0000","last_fullsized":"2024-09-17T04:49:42.672206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:37.637477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670707+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T04:49:42.670707+0000","last_peered":"2024-09-17T04:49:42.670707+0000","last_clean":"2024-09-17T04:49:42.670707+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T04:49:42.670707+0000","last_undegraded":"2024-09-17T04:49:42.670707+0000","last_fullsized":"2024-09-17T04:49:42.670707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674274+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T04:49:42.674274+0000","last_peered":"2024-09-17T04:49:42.674274+0000","last_clean":"2024-09-17T04:49:42.674274+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T04:49:42.674274+0000","last_undegraded":"2024-09-17T04:49:42.674274+0000","last_fullsized":"2024-09-17T04:49:42.674274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535118+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T04:49:42.535118+0000","last_peered":"2024-09-17T04:49:42.535118+0000","last_clean":"2024-09-17T04:49:42.535118+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T04:49:42.535118+0000","last_undegraded":"2024-09-17T04:49:42.535118+0000","last_fullsized":"2024-09-17T04:49:42.535118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535259+0000","last_change":"2024-09-17T04:49:41.043531+0000","last_active":"2024-09-17T04:49:42.535259+0000","last_peered":"2024-09-17T04:49:42.535259+0000","last_clean":"2024-09-17T04:49:42.535259+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T04:49:42.535259+0000","last_undegraded":"2024-09-17T04:49:42.535259+0000","last_fullsized":"2024-09-17T04:49:42.535259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672162+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T04:49:42.672162+0000","last_peered":"2024-09-17T04:49:42.672162+0000","last_clean":"2024-09-17T04:49:42.672162+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T04:49:42.672162+0000","last_undegraded":"2024-09-17T04:49:42.672162+0000","last_fullsized":"2024-09-17T04:49:42.672162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042945+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T04:49:42.042945+0000","last_peered":"2024-09-17T04:49:42.042945+0000","last_clean":"2024-09-17T04:49:42.042945+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T04:49:42.042945+0000","last_undegraded":"2024-09-17T04:49:42.042945+0000","last_fullsized":"2024-09-17T04:49:42.042945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535003+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T04:49:42.535003+0000","last_peered":"2024-09-17T04:49:42.535003+0000","last_clean":"2024-09-17T04:49:42.535003+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T04:49:42.535003+0000","last_undegraded":"2024-09-17T04:49:42.535003+0000","last_fullsized":"2024-09-17T04:49:42.535003+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:16.854365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669588+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T04:49:42.669588+0000","last_peered":"2024-09-17T04:49:42.669588+0000","last_clean":"2024-09-17T04:49:42.669588+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T04:49:42.669588+0000","last_undegraded":"2024-09-17T04:49:42.669588+0000","last_fullsized":"2024-09-17T04:49:42.669588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042526+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T04:49:42.042526+0000","last_peered":"2024-09-17T04:49:42.042526+0000","last_clean":"2024-09-17T04:49:42.042526+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T04:49:42.042526+0000","last_undegraded":"2024-09-17T04:49:42.042526+0000","last_fullsized":"2024-09-17T04:49:42.042526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041407+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T04:49:42.041407+0000","last_peered":"2024-09-17T04:49:42.041407+0000","last_clean":"2024-09-17T04:49:42.041407+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T04:49:42.041407+0000","last_undegraded":"2024-09-17T04:49:42.041407+0000","last_fullsized":"2024-09-17T04:49:42.041407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042319+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T04:49:42.042319+0000","last_peered":"2024-09-17T04:49:42.042319+0000","last_clean":"2024-09-17T04:49:42.042319+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T04:49:42.042319+0000","last_undegraded":"2024-09-17T04:49:42.042319+0000","last_fullsized":"2024-09-17T04:49:42.042319+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:28.365399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669339+0000","last_change":"2024-09-17T04:49:41.038415+0000","last_active":"2024-09-17T04:49:42.669339+0000","last_peered":"2024-09-17T04:49:42.669339+0000","last_clean":"2024-09-17T04:49:42.669339+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T04:49:42.669339+0000","last_undegraded":"2024-09-17T04:49:42.669339+0000","last_fullsized":"2024-09-17T04:49:42.669339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:17.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668616+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T04:49:42.668616+0000","last_peered":"2024-09-17T04:49:42.668616+0000","last_clean":"2024-09-17T04:49:42.668616+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T04:49:42.668616+0000","last_undegraded":"2024-09-17T04:49:42.668616+0000","last_fullsized":"2024-09-17T04:49:42.668616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052248+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T04:49:42.052248+0000","last_peered":"2024-09-17T04:49:42.052248+0000","last_clean":"2024-09-17T04:49:42.052248+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T04:49:42.052248+0000","last_undegraded":"2024-09-17T04:49:42.052248+0000","last_fullsized":"2024-09-17T04:49:42.052248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671586+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T04:49:42.671586+0000","last_peered":"2024-09-17T04:49:42.671586+0000","last_clean":"2024-09-17T04:49:42.671586+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T04:49:42.671586+0000","last_undegraded":"2024-09-17T04:49:42.671586+0000","last_fullsized":"2024-09-17T04:49:42.671586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:17.424265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042028+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T04:49:42.042028+0000","last_peered":"2024-09-17T04:49:42.042028+0000","last_clean":"2024-09-17T04:49:42.042028+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T04:49:42.042028+0000","last_undegraded":"2024-09-17T04:49:42.042028+0000","last_fullsized":"2024-09-17T04:49:42.042028+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:06.516432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048455+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T04:49:42.048455+0000","last_peered":"2024-09-17T04:49:42.048455+0000","last_clean":"2024-09-17T04:49:42.048455+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T04:49:42.048455+0000","last_undegraded":"2024-09-17T04:49:42.048455+0000","last_fullsized":"2024-09-17T04:49:42.048455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048500+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T04:49:42.048500+0000","last_peered":"2024-09-17T04:49:42.048500+0000","last_clean":"2024-09-17T04:49:42.048500+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T04:49:42.048500+0000","last_undegraded":"2024-09-17T04:49:42.048500+0000","last_fullsized":"2024-09-17T04:49:42.048500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668827+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T04:49:42.668827+0000","last_peered":"2024-09-17T04:49:42.668827+0000","last_clean":"2024-09-17T04:49:42.668827+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T04:49:42.668827+0000","last_undegraded":"2024-09-17T04:49:42.668827+0000","last_fullsized":"2024-09-17T04:49:42.668827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:00.010365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675406+0000","last_change":"2024-09-17T04:49:41.036658+0000","last_active":"2024-09-17T04:49:42.675406+0000","last_peered":"2024-09-17T04:49:42.675406+0000","last_clean":"2024-09-17T04:49:42.675406+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T04:49:42.675406+0000","last_undegraded":"2024-09-17T04:49:42.675406+0000","last_fullsized":"2024-09-17T04:49:42.675406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042824+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T04:49:42.042824+0000","last_peered":"2024-09-17T04:49:42.042824+0000","last_clean":"2024-09-17T04:49:42.042824+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T04:49:42.042824+0000","last_undegraded":"2024-09-17T04:49:42.042824+0000","last_fullsized":"2024-09-17T04:49:42.042824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668829+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T04:49:42.668829+0000","last_peered":"2024-09-17T04:49:42.668829+0000","last_clean":"2024-09-17T04:49:42.668829+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T04:49:42.668829+0000","last_undegraded":"2024-09-17T04:49:42.668829+0000","last_fullsized":"2024-09-17T04:49:42.668829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536259+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T04:49:42.536259+0000","last_peered":"2024-09-17T04:49:42.536259+0000","last_clean":"2024-09-17T04:49:42.536259+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T04:49:42.536259+0000","last_undegraded":"2024-09-17T04:49:42.536259+0000","last_fullsized":"2024-09-17T04:49:42.536259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048942+0000","last_change":"2024-09-17T04:49:41.050170+0000","last_active":"2024-09-17T04:49:42.048942+0000","last_peered":"2024-09-17T04:49:42.048942+0000","last_clean":"2024-09-17T04:49:42.048942+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T04:49:42.048942+0000","last_undegraded":"2024-09-17T04:49:42.048942+0000","last_fullsized":"2024-09-17T04:49:42.048942+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051929+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T04:49:42.051929+0000","last_peered":"2024-09-17T04:49:42.051929+0000","last_clean":"2024-09-17T04:49:42.051929+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T04:49:42.051929+0000","last_undegraded":"2024-09-17T04:49:42.051929+0000","last_fullsized":"2024-09-17T04:49:42.051929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042185+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T04:49:42.042185+0000","last_peered":"2024-09-17T04:49:42.042185+0000","last_clean":"2024-09-17T04:49:42.042185+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T04:49:42.042185+0000","last_undegraded":"2024-09-17T04:49:42.042185+0000","last_fullsized":"2024-09-17T04:49:42.042185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535553+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T04:49:42.535553+0000","last_peered":"2024-09-17T04:49:42.535553+0000","last_clean":"2024-09-17T04:49:42.535553+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T04:49:42.535553+0000","last_undegraded":"2024-09-17T04:49:42.535553+0000","last_fullsized":"2024-09-17T04:49:42.535553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041211+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T04:49:42.041211+0000","last_peered":"2024-09-17T04:49:42.041211+0000","last_clean":"2024-09-17T04:49:42.041211+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T04:49:42.041211+0000","last_undegraded":"2024-09-17T04:49:42.041211+0000","last_fullsized":"2024-09-17T04:49:42.041211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054336+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T04:49:42.054336+0000","last_peered":"2024-09-17T04:49:42.054336+0000","last_clean":"2024-09-17T04:49:42.054336+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T04:49:42.054336+0000","last_undegraded":"2024-09-17T04:49:42.054336+0000","last_fullsized":"2024-09-17T04:49:42.054336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052826+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T04:49:42.052826+0000","last_peered":"2024-09-17T04:49:42.052826+0000","last_clean":"2024-09-17T04:49:42.052826+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T04:49:42.052826+0000","last_undegraded":"2024-09-17T04:49:42.052826+0000","last_fullsized":"2024-09-17T04:49:42.052826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048145+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T04:49:42.048145+0000","last_peered":"2024-09-17T04:49:42.048145+0000","last_clean":"2024-09-17T04:49:42.048145+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T04:49:42.048145+0000","last_undegraded":"2024-09-17T04:49:42.048145+0000","last_fullsized":"2024-09-17T04:49:42.048145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669693+0000","last_change":"2024-09-17T04:49:41.042314+0000","last_active":"2024-09-17T04:49:42.669693+0000","last_peered":"2024-09-17T04:49:42.669693+0000","last_clean":"2024-09-17T04:49:42.669693+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T04:49:42.669693+0000","last_undegraded":"2024-09-17T04:49:42.669693+0000","last_fullsized":"2024-09-17T04:49:42.669693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055299+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T04:49:42.055299+0000","last_peered":"2024-09-17T04:49:42.055299+0000","last_clean":"2024-09-17T04:49:42.055299+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T04:49:42.055299+0000","last_undegraded":"2024-09-17T04:49:42.055299+0000","last_fullsized":"2024-09-17T04:49:42.055299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043663+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T04:49:42.043663+0000","last_peered":"2024-09-17T04:49:42.043663+0000","last_clean":"2024-09-17T04:49:42.043663+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T04:49:42.043663+0000","last_undegraded":"2024-09-17T04:49:42.043663+0000","last_fullsized":"2024-09-17T04:49:42.043663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053169+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T04:49:42.053169+0000","last_peered":"2024-09-17T04:49:42.053169+0000","last_clean":"2024-09-17T04:49:42.053169+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T04:49:42.053169+0000","last_undegraded":"2024-09-17T04:49:42.053169+0000","last_fullsized":"2024-09-17T04:49:42.053169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052959+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T04:49:42.052959+0000","last_peered":"2024-09-17T04:49:42.052959+0000","last_clean":"2024-09-17T04:49:42.052959+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T04:49:42.052959+0000","last_undegraded":"2024-09-17T04:49:42.052959+0000","last_fullsized":"2024-09-17T04:49:42.052959+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535986+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T04:49:42.535986+0000","last_peered":"2024-09-17T04:49:42.535986+0000","last_clean":"2024-09-17T04:49:42.535986+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T04:49:42.535986+0000","last_undegraded":"2024-09-17T04:49:42.535986+0000","last_fullsized":"2024-09-17T04:49:42.535986+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670900+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T04:49:42.670900+0000","last_peered":"2024-09-17T04:49:42.670900+0000","last_clean":"2024-09-17T04:49:42.670900+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T04:49:42.670900+0000","last_undegraded":"2024-09-17T04:49:42.670900+0000","last_fullsized":"2024-09-17T04:49:42.670900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050191+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T04:49:42.050191+0000","last_peered":"2024-09-17T04:49:42.050191+0000","last_clean":"2024-09-17T04:49:42.050191+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T04:49:42.050191+0000","last_undegraded":"2024-09-17T04:49:42.050191+0000","last_fullsized":"2024-09-17T04:49:42.050191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670215+0000","last_change":"2024-09-17T04:49:41.069177+0000","last_active":"2024-09-17T04:49:42.670215+0000","last_peered":"2024-09-17T04:49:42.670215+0000","last_clean":"2024-09-17T04:49:42.670215+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T04:49:42.670215+0000","last_undegraded":"2024-09-17T04:49:42.670215+0000","last_fullsized":"2024-09-17T04:49:42.670215+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:16.855206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670381+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T04:49:42.670381+0000","last_peered":"2024-09-17T04:49:42.670381+0000","last_clean":"2024-09-17T04:49:42.670381+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T04:49:42.670381+0000","last_undegraded":"2024-09-17T04:49:42.670381+0000","last_fullsized":"2024-09-17T04:49:42.670381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042706+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T04:49:42.042706+0000","last_peered":"2024-09-17T04:49:42.042706+0000","last_clean":"2024-09-17T04:49:42.042706+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T04:49:42.042706+0000","last_undegraded":"2024-09-17T04:49:42.042706+0000","last_fullsized":"2024-09-17T04:49:42.042706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054900+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T04:49:42.054900+0000","last_peered":"2024-09-17T04:49:42.054900+0000","last_clean":"2024-09-17T04:49:42.054900+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T04:49:42.054900+0000","last_undegraded":"2024-09-17T04:49:42.054900+0000","last_fullsized":"2024-09-17T04:49:42.054900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669308+0000","last_change":"2024-09-17T04:49:41.068537+0000","last_active":"2024-09-17T04:49:42.669308+0000","last_peered":"2024-09-17T04:49:42.669308+0000","last_clean":"2024-09-17T04:49:42.669308+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T04:49:42.669308+0000","last_undegraded":"2024-09-17T04:49:42.669308+0000","last_fullsized":"2024-09-17T04:49:42.669308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667397+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T04:49:42.667397+0000","last_peered":"2024-09-17T04:49:42.667397+0000","last_clean":"2024-09-17T04:49:42.667397+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T04:49:42.667397+0000","last_undegraded":"2024-09-17T04:49:42.667397+0000","last_fullsized":"2024-09-17T04:49:42.667397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668940+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T04:49:42.668940+0000","last_peered":"2024-09-17T04:49:42.668940+0000","last_clean":"2024-09-17T04:49:42.668940+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T04:49:42.668940+0000","last_undegraded":"2024-09-17T04:49:42.668940+0000","last_fullsized":"2024-09-17T04:49:42.668940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054921+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T04:49:42.054921+0000","last_peered":"2024-09-17T04:49:42.054921+0000","last_clean":"2024-09-17T04:49:42.054921+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T04:49:42.054921+0000","last_undegraded":"2024-09-17T04:49:42.054921+0000","last_fullsized":"2024-09-17T04:49:42.054921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041762+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T04:49:42.041762+0000","last_peered":"2024-09-17T04:49:42.041762+0000","last_clean":"2024-09-17T04:49:42.041762+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T04:49:42.041762+0000","last_undegraded":"2024-09-17T04:49:42.041762+0000","last_fullsized":"2024-09-17T04:49:42.041762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050299+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T04:49:42.050299+0000","last_peered":"2024-09-17T04:49:42.050299+0000","last_clean":"2024-09-17T04:49:42.050299+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T04:49:42.050299+0000","last_undegraded":"2024-09-17T04:49:42.050299+0000","last_fullsized":"2024-09-17T04:49:42.050299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041771+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T04:49:42.041771+0000","last_peered":"2024-09-17T04:49:42.041771+0000","last_clean":"2024-09-17T04:49:42.041771+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T04:49:42.041771+0000","last_undegraded":"2024-09-17T04:49:42.041771+0000","last_fullsized":"2024-09-17T04:49:42.041771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672234+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T04:49:42.672234+0000","last_peered":"2024-09-17T04:49:42.672234+0000","last_clean":"2024-09-17T04:49:42.672234+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T04:49:42.672234+0000","last_undegraded":"2024-09-17T04:49:42.672234+0000","last_fullsized":"2024-09-17T04:49:42.672234+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671183+0000","last_change":"2024-09-17T04:49:41.052564+0000","last_active":"2024-09-17T04:49:42.671183+0000","last_peered":"2024-09-17T04:49:42.671183+0000","last_clean":"2024-09-17T04:49:42.671183+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T04:49:42.671183+0000","last_undegraded":"2024-09-17T04:49:42.671183+0000","last_fullsized":"2024-09-17T04:49:42.671183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049438+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T04:49:42.049438+0000","last_peered":"2024-09-17T04:49:42.049438+0000","last_clean":"2024-09-17T04:49:42.049438+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T04:49:42.049438+0000","last_undegraded":"2024-09-17T04:49:42.049438+0000","last_fullsized":"2024-09-17T04:49:42.049438+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041458+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T04:49:42.041458+0000","last_peered":"2024-09-17T04:49:42.041458+0000","last_clean":"2024-09-17T04:49:42.041458+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T04:49:42.041458+0000","last_undegraded":"2024-09-17T04:49:42.041458+0000","last_fullsized":"2024-09-17T04:49:42.041458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671735+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T04:49:42.671735+0000","last_peered":"2024-09-17T04:49:42.671735+0000","last_clean":"2024-09-17T04:49:42.671735+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T04:49:42.671735+0000","last_undegraded":"2024-09-17T04:49:42.671735+0000","last_fullsized":"2024-09-17T04:49:42.671735+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043078+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T04:49:42.043078+0000","last_peered":"2024-09-17T04:49:42.043078+0000","last_clean":"2024-09-17T04:49:42.043078+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T04:49:42.043078+0000","last_undegraded":"2024-09-17T04:49:42.043078+0000","last_fullsized":"2024-09-17T04:49:42.043078+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:21.202398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669578+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T04:49:42.669578+0000","last_peered":"2024-09-17T04:49:42.669578+0000","last_clean":"2024-09-17T04:49:42.669578+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T04:49:42.669578+0000","last_undegraded":"2024-09-17T04:49:42.669578+0000","last_fullsized":"2024-09-17T04:49:42.669578+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:48.128609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043280+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T04:49:42.043280+0000","last_peered":"2024-09-17T04:49:42.043280+0000","last_clean":"2024-09-17T04:49:42.043280+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T04:49:42.043280+0000","last_undegraded":"2024-09-17T04:49:42.043280+0000","last_fullsized":"2024-09-17T04:49:42.043280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055292+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T04:49:42.055292+0000","last_peered":"2024-09-17T04:49:42.055292+0000","last_clean":"2024-09-17T04:49:42.055292+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T04:49:42.055292+0000","last_undegraded":"2024-09-17T04:49:42.055292+0000","last_fullsized":"2024-09-17T04:49:42.055292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536164+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T04:49:42.536164+0000","last_peered":"2024-09-17T04:49:42.536164+0000","last_clean":"2024-09-17T04:49:42.536164+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T04:49:42.536164+0000","last_undegraded":"2024-09-17T04:49:42.536164+0000","last_fullsized":"2024-09-17T04:49:42.536164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050257+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T04:49:42.050257+0000","last_peered":"2024-09-17T04:49:42.050257+0000","last_clean":"2024-09-17T04:49:42.050257+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T04:49:42.050257+0000","last_undegraded":"2024-09-17T04:49:42.050257+0000","last_fullsized":"2024-09-17T04:49:42.050257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671927+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T04:49:42.671927+0000","last_peered":"2024-09-17T04:49:42.671927+0000","last_clean":"2024-09-17T04:49:42.671927+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T04:49:42.671927+0000","last_undegraded":"2024-09-17T04:49:42.671927+0000","last_fullsized":"2024-09-17T04:49:42.671927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671411+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T04:49:42.671411+0000","last_peered":"2024-09-17T04:49:42.671411+0000","last_clean":"2024-09-17T04:49:42.671411+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T04:49:42.671411+0000","last_undegraded":"2024-09-17T04:49:42.671411+0000","last_fullsized":"2024-09-17T04:49:42.671411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670087+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T04:49:42.670087+0000","last_peered":"2024-09-17T04:49:42.670087+0000","last_clean":"2024-09-17T04:49:42.670087+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T04:49:42.670087+0000","last_undegraded":"2024-09-17T04:49:42.670087+0000","last_fullsized":"2024-09-17T04:49:42.670087+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:18.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672287+0000","last_change":"2024-09-17T04:49:41.031036+0000","last_active":"2024-09-17T04:49:42.672287+0000","last_peered":"2024-09-17T04:49:42.672287+0000","last_clean":"2024-09-17T04:49:42.672287+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T04:49:42.672287+0000","last_undegraded":"2024-09-17T04:49:42.672287+0000","last_fullsized":"2024-09-17T04:49:42.672287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:48.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672111+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T04:49:42.672111+0000","last_peered":"2024-09-17T04:49:42.672111+0000","last_clean":"2024-09-17T04:49:42.672111+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T04:49:42.672111+0000","last_undegraded":"2024-09-17T04:49:42.672111+0000","last_fullsized":"2024-09-17T04:49:42.672111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052688+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T04:49:42.052688+0000","last_peered":"2024-09-17T04:49:42.052688+0000","last_clean":"2024-09-17T04:49:42.052688+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T04:49:42.052688+0000","last_undegraded":"2024-09-17T04:49:42.052688+0000","last_fullsized":"2024-09-17T04:49:42.052688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041095+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T04:49:42.041095+0000","last_peered":"2024-09-17T04:49:42.041095+0000","last_clean":"2024-09-17T04:49:42.041095+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T04:49:42.041095+0000","last_undegraded":"2024-09-17T04:49:42.041095+0000","last_fullsized":"2024-09-17T04:49:42.041095+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048488+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T04:49:42.048488+0000","last_peered":"2024-09-17T04:49:42.048488+0000","last_clean":"2024-09-17T04:49:42.048488+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T04:49:42.048488+0000","last_undegraded":"2024-09-17T04:49:42.048488+0000","last_fullsized":"2024-09-17T04:49:42.048488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536967+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T04:49:42.536967+0000","last_peered":"2024-09-17T04:49:42.536967+0000","last_clean":"2024-09-17T04:49:42.536967+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T04:49:42.536967+0000","last_undegraded":"2024-09-17T04:49:42.536967+0000","last_fullsized":"2024-09-17T04:49:42.536967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672404+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T04:49:42.672404+0000","last_peered":"2024-09-17T04:49:42.672404+0000","last_clean":"2024-09-17T04:49:42.672404+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T04:49:42.672404+0000","last_undegraded":"2024-09-17T04:49:42.672404+0000","last_fullsized":"2024-09-17T04:49:42.672404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672966+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T04:49:42.672966+0000","last_peered":"2024-09-17T04:49:42.672966+0000","last_clean":"2024-09-17T04:49:42.672966+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T04:49:42.672966+0000","last_undegraded":"2024-09-17T04:49:42.672966+0000","last_fullsized":"2024-09-17T04:49:42.672966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:09.712272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052007+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T04:49:42.052007+0000","last_peered":"2024-09-17T04:49:42.052007+0000","last_clean":"2024-09-17T04:49:42.052007+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T04:49:42.052007+0000","last_undegraded":"2024-09-17T04:49:42.052007+0000","last_fullsized":"2024-09-17T04:49:42.052007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:43.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042787+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T04:49:42.042787+0000","last_peered":"2024-09-17T04:49:42.042787+0000","last_clean":"2024-09-17T04:49:42.042787+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T04:49:42.042787+0000","last_undegraded":"2024-09-17T04:49:42.042787+0000","last_fullsized":"2024-09-17T04:49:42.042787+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049500+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T04:49:42.049500+0000","last_peered":"2024-09-17T04:49:42.049500+0000","last_clean":"2024-09-17T04:49:42.049500+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T04:49:42.049500+0000","last_undegraded":"2024-09-17T04:49:42.049500+0000","last_fullsized":"2024-09-17T04:49:42.049500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671253+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T04:49:42.671253+0000","last_peered":"2024-09-17T04:49:42.671253+0000","last_clean":"2024-09-17T04:49:42.671253+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T04:49:42.671253+0000","last_undegraded":"2024-09-17T04:49:42.671253+0000","last_fullsized":"2024-09-17T04:49:42.671253+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:52.364568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041504+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T04:49:42.041504+0000","last_peered":"2024-09-17T04:49:42.041504+0000","last_clean":"2024-09-17T04:49:42.041504+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T04:49:42.041504+0000","last_undegraded":"2024-09-17T04:49:42.041504+0000","last_fullsized":"2024-09-17T04:49:42.041504+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:03.537392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671439+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T04:49:42.671439+0000","last_peered":"2024-09-17T04:49:42.671439+0000","last_clean":"2024-09-17T04:49:42.671439+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T04:49:42.671439+0000","last_undegraded":"2024-09-17T04:49:42.671439+0000","last_fullsized":"2024-09-17T04:49:42.671439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670943+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T04:49:42.670943+0000","last_peered":"2024-09-17T04:49:42.670943+0000","last_clean":"2024-09-17T04:49:42.670943+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T04:49:42.670943+0000","last_undegraded":"2024-09-17T04:49:42.670943+0000","last_fullsized":"2024-09-17T04:49:42.670943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671787+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T04:49:42.671787+0000","last_peered":"2024-09-17T04:49:42.671787+0000","last_clean":"2024-09-17T04:49:42.671787+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T04:49:42.671787+0000","last_undegraded":"2024-09-17T04:49:42.671787+0000","last_fullsized":"2024-09-17T04:49:42.671787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:52.120543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041402+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T04:49:42.041402+0000","last_peered":"2024-09-17T04:49:42.041402+0000","last_clean":"2024-09-17T04:49:42.041402+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T04:49:42.041402+0000","last_undegraded":"2024-09-17T04:49:42.041402+0000","last_fullsized":"2024-09-17T04:49:42.041402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:56.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041285+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T04:49:42.041285+0000","last_peered":"2024-09-17T04:49:42.041285+0000","last_clean":"2024-09-17T04:49:42.041285+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T04:49:42.041285+0000","last_undegraded":"2024-09-17T04:49:42.041285+0000","last_fullsized":"2024-09-17T04:49:42.041285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042566+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T04:49:42.042566+0000","last_peered":"2024-09-17T04:49:42.042566+0000","last_clean":"2024-09-17T04:49:42.042566+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T04:49:42.042566+0000","last_undegraded":"2024-09-17T04:49:42.042566+0000","last_fullsized":"2024-09-17T04:49:42.042566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054288+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T04:49:42.054288+0000","last_peered":"2024-09-17T04:49:42.054288+0000","last_clean":"2024-09-17T04:49:42.054288+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T04:49:42.054288+0000","last_undegraded":"2024-09-17T04:49:42.054288+0000","last_fullsized":"2024-09-17T04:49:42.054288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051153+0000","last_change":"2024-09-17T04:49:41.058280+0000","last_active":"2024-09-17T04:49:42.051153+0000","last_peered":"2024-09-17T04:49:42.051153+0000","last_clean":"2024-09-17T04:49:42.051153+0000","last_became_active":"2024-09-17T04:49:41.052009+0000","last_became_peered":"2024-09-17T04:49:41.052009+0000","last_unstale":"2024-09-17T04:49:42.051153+0000","last_undegraded":"2024-09-17T04:49:42.051153+0000","last_fullsized":"2024-09-17T04:49:42.051153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:14.100818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051977+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T04:49:42.051977+0000","last_peered":"2024-09-17T04:49:42.051977+0000","last_clean":"2024-09-17T04:49:42.051977+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T04:49:42.051977+0000","last_undegraded":"2024-09-17T04:49:42.051977+0000","last_fullsized":"2024-09-17T04:49:42.051977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669355+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T04:49:42.669355+0000","last_peered":"2024-09-17T04:49:42.669355+0000","last_clean":"2024-09-17T04:49:42.669355+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T04:49:42.669355+0000","last_undegraded":"2024-09-17T04:49:42.669355+0000","last_fullsized":"2024-09-17T04:49:42.669355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042782+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T04:49:42.042782+0000","last_peered":"2024-09-17T04:49:42.042782+0000","last_clean":"2024-09-17T04:49:42.042782+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T04:49:42.042782+0000","last_undegraded":"2024-09-17T04:49:42.042782+0000","last_fullsized":"2024-09-17T04:49:42.042782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:07.945355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052280+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T04:49:42.052280+0000","last_peered":"2024-09-17T04:49:42.052280+0000","last_clean":"2024-09-17T04:49:42.052280+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T04:49:42.052280+0000","last_undegraded":"2024-09-17T04:49:42.052280+0000","last_fullsized":"2024-09-17T04:49:42.052280+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670412+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T04:49:42.670412+0000","last_peered":"2024-09-17T04:49:42.670412+0000","last_clean":"2024-09-17T04:49:42.670412+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T04:49:42.670412+0000","last_undegraded":"2024-09-17T04:49:42.670412+0000","last_fullsized":"2024-09-17T04:49:42.670412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537054+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T04:49:42.537054+0000","last_peered":"2024-09-17T04:49:42.537054+0000","last_clean":"2024-09-17T04:49:42.537054+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T04:49:42.537054+0000","last_undegraded":"2024-09-17T04:49:42.537054+0000","last_fullsized":"2024-09-17T04:49:42.537054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048913+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T04:49:42.048913+0000","last_peered":"2024-09-17T04:49:42.048913+0000","last_clean":"2024-09-17T04:49:42.048913+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T04:49:42.048913+0000","last_undegraded":"2024-09-17T04:49:42.048913+0000","last_fullsized":"2024-09-17T04:49:42.048913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:16.447835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051373+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T04:49:42.051373+0000","last_peered":"2024-09-17T04:49:42.051373+0000","last_clean":"2024-09-17T04:49:42.051373+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T04:49:42.051373+0000","last_undegraded":"2024-09-17T04:49:42.051373+0000","last_fullsized":"2024-09-17T04:49:42.051373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042063+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T04:49:42.042063+0000","last_peered":"2024-09-17T04:49:42.042063+0000","last_clean":"2024-09-17T04:49:42.042063+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T04:49:42.042063+0000","last_undegraded":"2024-09-17T04:49:42.042063+0000","last_fullsized":"2024-09-17T04:49:42.042063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670596+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T04:49:42.670596+0000","last_peered":"2024-09-17T04:49:42.670596+0000","last_clean":"2024-09-17T04:49:42.670596+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T04:49:42.670596+0000","last_undegraded":"2024-09-17T04:49:42.670596+0000","last_fullsized":"2024-09-17T04:49:42.670596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535510+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T04:49:42.535510+0000","last_peered":"2024-09-17T04:49:42.535510+0000","last_clean":"2024-09-17T04:49:42.535510+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T04:49:42.535510+0000","last_undegraded":"2024-09-17T04:49:42.535510+0000","last_fullsized":"2024-09-17T04:49:42.535510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041322+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T04:49:42.041322+0000","last_peered":"2024-09-17T04:49:42.041322+0000","last_clean":"2024-09-17T04:49:42.041322+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T04:49:42.041322+0000","last_undegraded":"2024-09-17T04:49:42.041322+0000","last_fullsized":"2024-09-17T04:49:42.041322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672182+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T04:49:42.672182+0000","last_peered":"2024-09-17T04:49:42.672182+0000","last_clean":"2024-09-17T04:49:42.672182+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T04:49:42.672182+0000","last_undegraded":"2024-09-17T04:49:42.672182+0000","last_fullsized":"2024-09-17T04:49:42.672182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671261+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T04:49:42.671261+0000","last_peered":"2024-09-17T04:49:42.671261+0000","last_clean":"2024-09-17T04:49:42.671261+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T04:49:42.671261+0000","last_undegraded":"2024-09-17T04:49:42.671261+0000","last_fullsized":"2024-09-17T04:49:42.671261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667575+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T04:49:42.667575+0000","last_peered":"2024-09-17T04:49:42.667575+0000","last_clean":"2024-09-17T04:49:42.667575+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T04:49:42.667575+0000","last_undegraded":"2024-09-17T04:49:42.667575+0000","last_fullsized":"2024-09-17T04:49:42.667575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:40.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534654+0000","last_change":"2024-09-17T04:49:41.040278+0000","last_active":"2024-09-17T04:49:42.534654+0000","last_peered":"2024-09-17T04:49:42.534654+0000","last_clean":"2024-09-17T04:49:42.534654+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T04:49:42.534654+0000","last_undegraded":"2024-09-17T04:49:42.534654+0000","last_fullsized":"2024-09-17T04:49:42.534654+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.895223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536732+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T04:49:42.536732+0000","last_peered":"2024-09-17T04:49:42.536732+0000","last_clean":"2024-09-17T04:49:42.536732+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T04:49:42.536732+0000","last_undegraded":"2024-09-17T04:49:42.536732+0000","last_fullsized":"2024-09-17T04:49:42.536732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675753+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T04:49:42.675753+0000","last_peered":"2024-09-17T04:49:42.675753+0000","last_clean":"2024-09-17T04:49:42.675753+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T04:49:42.675753+0000","last_undegraded":"2024-09-17T04:49:42.675753+0000","last_fullsized":"2024-09-17T04:49:42.675753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047646+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T04:49:42.047646+0000","last_peered":"2024-09-17T04:49:42.047646+0000","last_clean":"2024-09-17T04:49:42.047646+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T04:49:42.047646+0000","last_undegraded":"2024-09-17T04:49:42.047646+0000","last_fullsized":"2024-09-17T04:49:42.047646+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:15.845612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537298+0000","last_change":"2024-09-17T04:49:41.050551+0000","last_active":"2024-09-17T04:49:42.537298+0000","last_peered":"2024-09-17T04:49:42.537298+0000","last_clean":"2024-09-17T04:49:42.537298+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T04:49:42.537298+0000","last_undegraded":"2024-09-17T04:49:42.537298+0000","last_fullsized":"2024-09-17T04:49:42.537298+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:11.088230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051592+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T04:49:42.051592+0000","last_peered":"2024-09-17T04:49:42.051592+0000","last_clean":"2024-09-17T04:49:42.051592+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T04:49:42.051592+0000","last_undegraded":"2024-09-17T04:49:42.051592+0000","last_fullsized":"2024-09-17T04:49:42.051592+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:37.778756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048401+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T04:49:42.048401+0000","last_peered":"2024-09-17T04:49:42.048401+0000","last_clean":"2024-09-17T04:49:42.048401+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T04:49:42.048401+0000","last_undegraded":"2024-09-17T04:49:42.048401+0000","last_fullsized":"2024-09-17T04:49:42.048401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041797+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T04:49:42.041797+0000","last_peered":"2024-09-17T04:49:42.041797+0000","last_clean":"2024-09-17T04:49:42.041797+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T04:49:42.041797+0000","last_undegraded":"2024-09-17T04:49:42.041797+0000","last_fullsized":"2024-09-17T04:49:42.041797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055365+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T04:49:42.055365+0000","last_peered":"2024-09-17T04:49:42.055365+0000","last_clean":"2024-09-17T04:49:42.055365+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T04:49:42.055365+0000","last_undegraded":"2024-09-17T04:49:42.055365+0000","last_fullsized":"2024-09-17T04:49:42.055365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041767+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T04:49:42.041767+0000","last_peered":"2024-09-17T04:49:42.041767+0000","last_clean":"2024-09-17T04:49:42.041767+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T04:49:42.041767+0000","last_undegraded":"2024-09-17T04:49:42.041767+0000","last_fullsized":"2024-09-17T04:49:42.041767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667708+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T04:49:42.667708+0000","last_peered":"2024-09-17T04:49:42.667708+0000","last_clean":"2024-09-17T04:49:42.667708+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T04:49:42.667708+0000","last_undegraded":"2024-09-17T04:49:42.667708+0000","last_fullsized":"2024-09-17T04:49:42.667708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050522+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T04:49:42.050522+0000","last_peered":"2024-09-17T04:49:42.050522+0000","last_clean":"2024-09-17T04:49:42.050522+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T04:49:42.050522+0000","last_undegraded":"2024-09-17T04:49:42.050522+0000","last_fullsized":"2024-09-17T04:49:42.050522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049954+0000","last_change":"2024-09-17T04:49:41.057647+0000","last_active":"2024-09-17T04:49:42.049954+0000","last_peered":"2024-09-17T04:49:42.049954+0000","last_clean":"2024-09-17T04:49:42.049954+0000","last_became_active":"2024-09-17T04:49:41.056213+0000","last_became_peered":"2024-09-17T04:49:41.056213+0000","last_unstale":"2024-09-17T04:49:42.049954+0000","last_undegraded":"2024-09-17T04:49:42.049954+0000","last_fullsized":"2024-09-17T04:49:42.049954+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042498+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T04:49:42.042498+0000","last_peered":"2024-09-17T04:49:42.042498+0000","last_clean":"2024-09-17T04:49:42.042498+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T04:49:42.042498+0000","last_undegraded":"2024-09-17T04:49:42.042498+0000","last_fullsized":"2024-09-17T04:49:42.042498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053714+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T04:49:42.053714+0000","last_peered":"2024-09-17T04:49:42.053714+0000","last_clean":"2024-09-17T04:49:42.053714+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T04:49:42.053714+0000","last_undegraded":"2024-09-17T04:49:42.053714+0000","last_fullsized":"2024-09-17T04:49:42.053714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051282+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T04:49:42.051282+0000","last_peered":"2024-09-17T04:49:42.051282+0000","last_clean":"2024-09-17T04:49:42.051282+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T04:49:42.051282+0000","last_undegraded":"2024-09-17T04:49:42.051282+0000","last_fullsized":"2024-09-17T04:49:42.051282+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:27.230604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536526+0000","last_change":"2024-09-17T04:49:41.055263+0000","last_active":"2024-09-17T04:49:42.536526+0000","last_peered":"2024-09-17T04:49:42.536526+0000","last_clean":"2024-09-17T04:49:42.536526+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T04:49:42.536526+0000","last_undegraded":"2024-09-17T04:49:42.536526+0000","last_fullsized":"2024-09-17T04:49:42.536526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670438+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T04:49:42.670438+0000","last_peered":"2024-09-17T04:49:42.670438+0000","last_clean":"2024-09-17T04:49:42.670438+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T04:49:42.670438+0000","last_undegraded":"2024-09-17T04:49:42.670438+0000","last_fullsized":"2024-09-17T04:49:42.670438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673048+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T04:49:42.673048+0000","last_peered":"2024-09-17T04:49:42.673048+0000","last_clean":"2024-09-17T04:49:42.673048+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T04:49:42.673048+0000","last_undegraded":"2024-09-17T04:49:42.673048+0000","last_fullsized":"2024-09-17T04:49:42.673048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670447+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T04:49:42.670447+0000","last_peered":"2024-09-17T04:49:42.670447+0000","last_clean":"2024-09-17T04:49:42.670447+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T04:49:42.670447+0000","last_undegraded":"2024-09-17T04:49:42.670447+0000","last_fullsized":"2024-09-17T04:49:42.670447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:24.586834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051128+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.051128+0000","last_peered":"2024-09-17T04:49:42.051128+0000","last_clean":"2024-09-17T04:49:42.051128+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T04:49:42.051128+0000","last_undegraded":"2024-09-17T04:49:42.051128+0000","last_fullsized":"2024-09-17T04:49:42.051128+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536163+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T04:49:42.536163+0000","last_peered":"2024-09-17T04:49:42.536163+0000","last_clean":"2024-09-17T04:49:42.536163+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T04:49:42.536163+0000","last_undegraded":"2024-09-17T04:49:42.536163+0000","last_fullsized":"2024-09-17T04:49:42.536163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052121+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T04:49:42.052121+0000","last_peered":"2024-09-17T04:49:42.052121+0000","last_clean":"2024-09-17T04:49:42.052121+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T04:49:42.052121+0000","last_undegraded":"2024-09-17T04:49:42.052121+0000","last_fullsized":"2024-09-17T04:49:42.052121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668467+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T04:49:42.668467+0000","last_peered":"2024-09-17T04:49:42.668467+0000","last_clean":"2024-09-17T04:49:42.668467+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T04:49:42.668467+0000","last_undegraded":"2024-09-17T04:49:42.668467+0000","last_fullsized":"2024-09-17T04:49:42.668467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536096+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T04:49:42.536096+0000","last_peered":"2024-09-17T04:49:42.536096+0000","last_clean":"2024-09-17T04:49:42.536096+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T04:49:42.536096+0000","last_undegraded":"2024-09-17T04:49:42.536096+0000","last_fullsized":"2024-09-17T04:49:42.536096+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042322+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T04:49:42.042322+0000","last_peered":"2024-09-17T04:49:42.042322+0000","last_clean":"2024-09-17T04:49:42.042322+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T04:49:42.042322+0000","last_undegraded":"2024-09-17T04:49:42.042322+0000","last_fullsized":"2024-09-17T04:49:42.042322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671200+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T04:49:42.671200+0000","last_peered":"2024-09-17T04:49:42.671200+0000","last_clean":"2024-09-17T04:49:42.671200+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T04:49:42.671200+0000","last_undegraded":"2024-09-17T04:49:42.671200+0000","last_fullsized":"2024-09-17T04:49:42.671200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670309+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T04:49:42.670309+0000","last_peered":"2024-09-17T04:49:42.670309+0000","last_clean":"2024-09-17T04:49:42.670309+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T04:49:42.670309+0000","last_undegraded":"2024-09-17T04:49:42.670309+0000","last_fullsized":"2024-09-17T04:49:42.670309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668969+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T04:49:42.668969+0000","last_peered":"2024-09-17T04:49:42.668969+0000","last_clean":"2024-09-17T04:49:42.668969+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T04:49:42.668969+0000","last_undegraded":"2024-09-17T04:49:42.668969+0000","last_fullsized":"2024-09-17T04:49:42.668969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054935+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T04:49:42.054935+0000","last_peered":"2024-09-17T04:49:42.054935+0000","last_clean":"2024-09-17T04:49:42.054935+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T04:49:42.054935+0000","last_undegraded":"2024-09-17T04:49:42.054935+0000","last_fullsized":"2024-09-17T04:49:42.054935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669059+0000","last_change":"2024-09-17T04:49:41.073709+0000","last_active":"2024-09-17T04:49:42.669059+0000","last_peered":"2024-09-17T04:49:42.669059+0000","last_clean":"2024-09-17T04:49:42.669059+0000","last_became_active":"2024-09-17T04:49:41.073330+0000","last_became_peered":"2024-09-17T04:49:41.073330+0000","last_unstale":"2024-09-17T04:49:42.669059+0000","last_undegraded":"2024-09-17T04:49:42.669059+0000","last_fullsized":"2024-09-17T04:49:42.669059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054811+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T04:49:42.054811+0000","last_peered":"2024-09-17T04:49:42.054811+0000","last_clean":"2024-09-17T04:49:42.054811+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T04:49:42.054811+0000","last_undegraded":"2024-09-17T04:49:42.054811+0000","last_fullsized":"2024-09-17T04:49:42.054811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041256+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T04:49:42.041256+0000","last_peered":"2024-09-17T04:49:42.041256+0000","last_clean":"2024-09-17T04:49:42.041256+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T04:49:42.041256+0000","last_undegraded":"2024-09-17T04:49:42.041256+0000","last_fullsized":"2024-09-17T04:49:42.041256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536669+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T04:49:42.536669+0000","last_peered":"2024-09-17T04:49:42.536669+0000","last_clean":"2024-09-17T04:49:42.536669+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T04:49:42.536669+0000","last_undegraded":"2024-09-17T04:49:42.536669+0000","last_fullsized":"2024-09-17T04:49:42.536669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534968+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T04:49:42.534968+0000","last_peered":"2024-09-17T04:49:42.534968+0000","last_clean":"2024-09-17T04:49:42.534968+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T04:49:42.534968+0000","last_undegraded":"2024-09-17T04:49:42.534968+0000","last_fullsized":"2024-09-17T04:49:42.534968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672019+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T04:49:42.672019+0000","last_peered":"2024-09-17T04:49:42.672019+0000","last_clean":"2024-09-17T04:49:42.672019+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T04:49:42.672019+0000","last_undegraded":"2024-09-17T04:49:42.672019+0000","last_fullsized":"2024-09-17T04:49:42.672019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668565+0000","last_change":"2024-09-17T04:49:41.045502+0000","last_active":"2024-09-17T04:49:42.668565+0000","last_peered":"2024-09-17T04:49:42.668565+0000","last_clean":"2024-09-17T04:49:42.668565+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T04:49:42.668565+0000","last_undegraded":"2024-09-17T04:49:42.668565+0000","last_fullsized":"2024-09-17T04:49:42.668565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:19.746356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041666+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T04:49:42.041666+0000","last_peered":"2024-09-17T04:49:42.041666+0000","last_clean":"2024-09-17T04:49:42.041666+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T04:49:42.041666+0000","last_undegraded":"2024-09-17T04:49:42.041666+0000","last_fullsized":"2024-09-17T04:49:42.041666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042310+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T04:49:42.042310+0000","last_peered":"2024-09-17T04:49:42.042310+0000","last_clean":"2024-09-17T04:49:42.042310+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T04:49:42.042310+0000","last_undegraded":"2024-09-17T04:49:42.042310+0000","last_fullsized":"2024-09-17T04:49:42.042310+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668045+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T04:49:42.668045+0000","last_peered":"2024-09-17T04:49:42.668045+0000","last_clean":"2024-09-17T04:49:42.668045+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T04:49:42.668045+0000","last_undegraded":"2024-09-17T04:49:42.668045+0000","last_fullsized":"2024-09-17T04:49:42.668045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054699+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T04:49:42.054699+0000","last_peered":"2024-09-17T04:49:42.054699+0000","last_clean":"2024-09-17T04:49:42.054699+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T04:49:42.054699+0000","last_undegraded":"2024-09-17T04:49:42.054699+0000","last_fullsized":"2024-09-17T04:49:42.054699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673038+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T04:49:42.673038+0000","last_peered":"2024-09-17T04:49:42.673038+0000","last_clean":"2024-09-17T04:49:42.673038+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T04:49:42.673038+0000","last_undegraded":"2024-09-17T04:49:42.673038+0000","last_fullsized":"2024-09-17T04:49:42.673038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669104+0000","last_change":"2024-09-17T04:49:41.038265+0000","last_active":"2024-09-17T04:49:42.669104+0000","last_peered":"2024-09-17T04:49:42.669104+0000","last_clean":"2024-09-17T04:49:42.669104+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T04:49:42.669104+0000","last_undegraded":"2024-09-17T04:49:42.669104+0000","last_fullsized":"2024-09-17T04:49:42.669104+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:41.638403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053416+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T04:49:42.053416+0000","last_peered":"2024-09-17T04:49:42.053416+0000","last_clean":"2024-09-17T04:49:42.053416+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T04:49:42.053416+0000","last_undegraded":"2024-09-17T04:49:42.053416+0000","last_fullsized":"2024-09-17T04:49:42.053416+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535001+0000","last_change":"2024-09-17T04:49:41.053443+0000","last_active":"2024-09-17T04:49:42.535001+0000","last_peered":"2024-09-17T04:49:42.535001+0000","last_clean":"2024-09-17T04:49:42.535001+0000","last_became_active":"2024-09-17T04:49:41.053090+0000","last_became_peered":"2024-09-17T04:49:41.053090+0000","last_unstale":"2024-09-17T04:49:42.535001+0000","last_undegraded":"2024-09-17T04:49:42.535001+0000","last_fullsized":"2024-09-17T04:49:42.535001+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671599+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T04:49:42.671599+0000","last_peered":"2024-09-17T04:49:42.671599+0000","last_clean":"2024-09-17T04:49:42.671599+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T04:49:42.671599+0000","last_undegraded":"2024-09-17T04:49:42.671599+0000","last_fullsized":"2024-09-17T04:49:42.671599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669669+0000","last_change":"2024-09-17T04:49:41.038072+0000","last_active":"2024-09-17T04:49:42.669669+0000","last_peered":"2024-09-17T04:49:42.669669+0000","last_clean":"2024-09-17T04:49:42.669669+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T04:49:42.669669+0000","last_undegraded":"2024-09-17T04:49:42.669669+0000","last_fullsized":"2024-09-17T04:49:42.669669+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043492+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T04:49:42.043492+0000","last_peered":"2024-09-17T04:49:42.043492+0000","last_clean":"2024-09-17T04:49:42.043492+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T04:49:42.043492+0000","last_undegraded":"2024-09-17T04:49:42.043492+0000","last_fullsized":"2024-09-17T04:49:42.043492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534462+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T04:49:42.534462+0000","last_peered":"2024-09-17T04:49:42.534462+0000","last_clean":"2024-09-17T04:49:42.534462+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T04:49:42.534462+0000","last_undegraded":"2024-09-17T04:49:42.534462+0000","last_fullsized":"2024-09-17T04:49:42.534462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669602+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T04:49:42.669602+0000","last_peered":"2024-09-17T04:49:42.669602+0000","last_clean":"2024-09-17T04:49:42.669602+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T04:49:42.669602+0000","last_undegraded":"2024-09-17T04:49:42.669602+0000","last_fullsized":"2024-09-17T04:49:42.669602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050316+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T04:49:42.050316+0000","last_peered":"2024-09-17T04:49:42.050316+0000","last_clean":"2024-09-17T04:49:42.050316+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T04:49:42.050316+0000","last_undegraded":"2024-09-17T04:49:42.050316+0000","last_fullsized":"2024-09-17T04:49:42.050316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042123+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T04:49:42.042123+0000","last_peered":"2024-09-17T04:49:42.042123+0000","last_clean":"2024-09-17T04:49:42.042123+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T04:49:42.042123+0000","last_undegraded":"2024-09-17T04:49:42.042123+0000","last_fullsized":"2024-09-17T04:49:42.042123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052378+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T04:49:42.052378+0000","last_peered":"2024-09-17T04:49:42.052378+0000","last_clean":"2024-09-17T04:49:42.052378+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T04:49:42.052378+0000","last_undegraded":"2024-09-17T04:49:42.052378+0000","last_fullsized":"2024-09-17T04:49:42.052378+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041588+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T04:49:42.041588+0000","last_peered":"2024-09-17T04:49:42.041588+0000","last_clean":"2024-09-17T04:49:42.041588+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T04:49:42.041588+0000","last_undegraded":"2024-09-17T04:49:42.041588+0000","last_fullsized":"2024-09-17T04:49:42.041588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674897+0000","last_change":"2024-09-17T04:49:41.067207+0000","last_active":"2024-09-17T04:49:42.674897+0000","last_peered":"2024-09-17T04:49:42.674897+0000","last_clean":"2024-09-17T04:49:42.674897+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T04:49:42.674897+0000","last_undegraded":"2024-09-17T04:49:42.674897+0000","last_fullsized":"2024-09-17T04:49:42.674897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:08.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667484+0000","last_change":"2024-09-17T04:49:31.911036+0000","last_active":"2024-09-17T04:49:42.667484+0000","last_peered":"2024-09-17T04:49:42.667484+0000","last_clean":"2024-09-17T04:49:42.667484+0000","last_became_active":"2024-09-17T04:49:31.910285+0000","last_became_peered":"2024-09-17T04:49:31.910285+0000","last_unstale":"2024-09-17T04:49:42.667484+0000","last_undegraded":"2024-09-17T04:49:42.667484+0000","last_fullsized":"2024-09-17T04:49:42.667484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673474+0000","last_change":"2024-09-17T04:49:41.046170+0000","last_active":"2024-09-17T04:49:42.673474+0000","last_peered":"2024-09-17T04:49:42.673474+0000","last_clean":"2024-09-17T04:49:42.673474+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T04:49:42.673474+0000","last_undegraded":"2024-09-17T04:49:42.673474+0000","last_fullsized":"2024-09-17T04:49:42.673474+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:01.683826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051528+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T04:49:42.051528+0000","last_peered":"2024-09-17T04:49:42.051528+0000","last_clean":"2024-09-17T04:49:42.051528+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T04:49:42.051528+0000","last_undegraded":"2024-09-17T04:49:42.051528+0000","last_fullsized":"2024-09-17T04:49:42.051528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041843+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T04:49:42.041843+0000","last_peered":"2024-09-17T04:49:42.041843+0000","last_clean":"2024-09-17T04:49:42.041843+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T04:49:42.041843+0000","last_undegraded":"2024-09-17T04:49:42.041843+0000","last_fullsized":"2024-09-17T04:49:42.041843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:25.574682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053942+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T04:49:42.053942+0000","last_peered":"2024-09-17T04:49:42.053942+0000","last_clean":"2024-09-17T04:49:42.053942+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T04:49:42.053942+0000","last_undegraded":"2024-09-17T04:49:42.053942+0000","last_fullsized":"2024-09-17T04:49:42.053942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048414+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T04:49:42.048414+0000","last_peered":"2024-09-17T04:49:42.048414+0000","last_clean":"2024-09-17T04:49:42.048414+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T04:49:42.048414+0000","last_undegraded":"2024-09-17T04:49:42.048414+0000","last_fullsized":"2024-09-17T04:49:42.048414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671218+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T04:49:42.671218+0000","last_peered":"2024-09-17T04:49:42.671218+0000","last_clean":"2024-09-17T04:49:42.671218+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T04:49:42.671218+0000","last_undegraded":"2024-09-17T04:49:42.671218+0000","last_fullsized":"2024-09-17T04:49:42.671218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053498+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T04:49:42.053498+0000","last_peered":"2024-09-17T04:49:42.053498+0000","last_clean":"2024-09-17T04:49:42.053498+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T04:49:42.053498+0000","last_undegraded":"2024-09-17T04:49:42.053498+0000","last_fullsized":"2024-09-17T04:49:42.053498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670119+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T04:49:42.670119+0000","last_peered":"2024-09-17T04:49:42.670119+0000","last_clean":"2024-09-17T04:49:42.670119+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T04:49:42.670119+0000","last_undegraded":"2024-09-17T04:49:42.670119+0000","last_fullsized":"2024-09-17T04:49:42.670119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668585+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T04:49:42.668585+0000","last_peered":"2024-09-17T04:49:42.668585+0000","last_clean":"2024-09-17T04:49:42.668585+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T04:49:42.668585+0000","last_undegraded":"2024-09-17T04:49:42.668585+0000","last_fullsized":"2024-09-17T04:49:42.668585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670988+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T04:49:42.670988+0000","last_peered":"2024-09-17T04:49:42.670988+0000","last_clean":"2024-09-17T04:49:42.670988+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T04:49:42.670988+0000","last_undegraded":"2024-09-17T04:49:42.670988+0000","last_fullsized":"2024-09-17T04:49:42.670988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671416+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.671416+0000","last_peered":"2024-09-17T04:49:42.671416+0000","last_clean":"2024-09-17T04:49:42.671416+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T04:49:42.671416+0000","last_undegraded":"2024-09-17T04:49:42.671416+0000","last_fullsized":"2024-09-17T04:49:42.671416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675766+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T04:49:42.675766+0000","last_peered":"2024-09-17T04:49:42.675766+0000","last_clean":"2024-09-17T04:49:42.675766+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T04:49:42.675766+0000","last_undegraded":"2024-09-17T04:49:42.675766+0000","last_fullsized":"2024-09-17T04:49:42.675766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671330+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T04:49:42.671330+0000","last_peered":"2024-09-17T04:49:42.671330+0000","last_clean":"2024-09-17T04:49:42.671330+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T04:49:42.671330+0000","last_undegraded":"2024-09-17T04:49:42.671330+0000","last_fullsized":"2024-09-17T04:49:42.671330+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667310+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T04:49:42.667310+0000","last_peered":"2024-09-17T04:49:42.667310+0000","last_clean":"2024-09-17T04:49:42.667310+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T04:49:42.667310+0000","last_undegraded":"2024-09-17T04:49:42.667310+0000","last_fullsized":"2024-09-17T04:49:42.667310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537133+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T04:49:42.537133+0000","last_peered":"2024-09-17T04:49:42.537133+0000","last_clean":"2024-09-17T04:49:42.537133+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T04:49:42.537133+0000","last_undegraded":"2024-09-17T04:49:42.537133+0000","last_fullsized":"2024-09-17T04:49:42.537133+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669347+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T04:49:42.669347+0000","last_peered":"2024-09-17T04:49:42.669347+0000","last_clean":"2024-09-17T04:49:42.669347+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T04:49:42.669347+0000","last_undegraded":"2024-09-17T04:49:42.669347+0000","last_fullsized":"2024-09-17T04:49:42.669347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050809+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T04:49:42.050809+0000","last_peered":"2024-09-17T04:49:42.050809+0000","last_clean":"2024-09-17T04:49:42.050809+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T04:49:42.050809+0000","last_undegraded":"2024-09-17T04:49:42.050809+0000","last_fullsized":"2024-09-17T04:49:42.050809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535865+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T04:49:42.535865+0000","last_peered":"2024-09-17T04:49:42.535865+0000","last_clean":"2024-09-17T04:49:42.535865+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T04:49:42.535865+0000","last_undegraded":"2024-09-17T04:49:42.535865+0000","last_fullsized":"2024-09-17T04:49:42.535865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673274+0000","last_change":"2024-09-17T04:49:41.041201+0000","last_active":"2024-09-17T04:49:42.673274+0000","last_peered":"2024-09-17T04:49:42.673274+0000","last_clean":"2024-09-17T04:49:42.673274+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T04:49:42.673274+0000","last_undegraded":"2024-09-17T04:49:42.673274+0000","last_fullsized":"2024-09-17T04:49:42.673274+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:02.588786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670272+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T04:49:42.670272+0000","last_peered":"2024-09-17T04:49:42.670272+0000","last_clean":"2024-09-17T04:49:42.670272+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T04:49:42.670272+0000","last_undegraded":"2024-09-17T04:49:42.670272+0000","last_fullsized":"2024-09-17T04:49:42.670272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667767+0000","last_change":"2024-09-17T04:49:41.061092+0000","last_active":"2024-09-17T04:49:42.667767+0000","last_peered":"2024-09-17T04:49:42.667767+0000","last_clean":"2024-09-17T04:49:42.667767+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T04:49:42.667767+0000","last_undegraded":"2024-09-17T04:49:42.667767+0000","last_fullsized":"2024-09-17T04:49:42.667767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:52.714614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536820+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T04:49:42.536820+0000","last_peered":"2024-09-17T04:49:42.536820+0000","last_clean":"2024-09-17T04:49:42.536820+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T04:49:42.536820+0000","last_undegraded":"2024-09-17T04:49:42.536820+0000","last_fullsized":"2024-09-17T04:49:42.536820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669397+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T04:49:42.669397+0000","last_peered":"2024-09-17T04:49:42.669397+0000","last_clean":"2024-09-17T04:49:42.669397+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T04:49:42.669397+0000","last_undegraded":"2024-09-17T04:49:42.669397+0000","last_fullsized":"2024-09-17T04:49:42.669397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671170+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T04:49:42.671170+0000","last_peered":"2024-09-17T04:49:42.671170+0000","last_clean":"2024-09-17T04:49:42.671170+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T04:49:42.671170+0000","last_undegraded":"2024-09-17T04:49:42.671170+0000","last_fullsized":"2024-09-17T04:49:42.671170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536018+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T04:49:42.536018+0000","last_peered":"2024-09-17T04:49:42.536018+0000","last_clean":"2024-09-17T04:49:42.536018+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T04:49:42.536018+0000","last_undegraded":"2024-09-17T04:49:42.536018+0000","last_fullsized":"2024-09-17T04:49:42.536018+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671602+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T04:49:42.671602+0000","last_peered":"2024-09-17T04:49:42.671602+0000","last_clean":"2024-09-17T04:49:42.671602+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T04:49:42.671602+0000","last_undegraded":"2024-09-17T04:49:42.671602+0000","last_fullsized":"2024-09-17T04:49:42.671602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534962+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T04:49:42.534962+0000","last_peered":"2024-09-17T04:49:42.534962+0000","last_clean":"2024-09-17T04:49:42.534962+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T04:49:42.534962+0000","last_undegraded":"2024-09-17T04:49:42.534962+0000","last_fullsized":"2024-09-17T04:49:42.534962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671825+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T04:49:42.671825+0000","last_peered":"2024-09-17T04:49:42.671825+0000","last_clean":"2024-09-17T04:49:42.671825+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T04:49:42.671825+0000","last_undegraded":"2024-09-17T04:49:42.671825+0000","last_fullsized":"2024-09-17T04:49:42.671825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509448,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":307200,"data_stored":108537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":286720,"data_stored":106265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509449,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509449,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:49:50.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:49:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.040+0000 7f902a97a640 1 -- 172.21.15.150:0/2752142905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90240c3b40 msgr2=0x7f9024157c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.040+0000 7f902a97a640 1 --2- 172.21.15.150:0/2752142905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90240c3b40 0x7f9024157c30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9014009840 tx=0x7f901401c640 comp rx=0 tx=0).stop 2024-09-17T04:49:51.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.040+0000 7f902a97a640 1 -- 172.21.15.150:0/2752142905 shutdown_connections 2024-09-17T04:49:51.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.040+0000 7f902a97a640 1 --2- 172.21.15.150:0/2752142905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90240c3b40 0x7f9024157c30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.040+0000 7f902a97a640 1 -- 172.21.15.150:0/2752142905 >> 172.21.15.150:0/2752142905 conn(0x7f90240c4a00 msgr2=0x7f90240c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:51.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.040+0000 7f902a97a640 1 -- 172.21.15.150:0/2752142905 shutdown_connections 2024-09-17T04:49:51.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.040+0000 7f902a97a640 1 -- 172.21.15.150:0/2752142905 wait complete. 2024-09-17T04:49:51.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.040+0000 7f902a97a640 1 Processor -- start 2024-09-17T04:49:51.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.040+0000 7f902a97a640 1 -- start start 2024-09-17T04:49:51.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f902a97a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f90240c3b40 0x7f9024074eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:51.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f902a97a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9024151d00 0x7f90241502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:51.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f902a97a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9024158170 0x7f9024154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:51.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f902a97a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f90240c4630 con 0x7f9024151d00 2024-09-17T04:49:51.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f902a97a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f90240c44b0 con 0x7f9024158170 2024-09-17T04:49:51.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f902a97a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f90240c3f50 con 0x7f90240c3b40 2024-09-17T04:49:51.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f90237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9024151d00 0x7f90241502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:51.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f90237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9024151d00 0x7f90241502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33000/0 (socket says 172.21.15.150:33000) 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f90237fe640 1 -- 172.21.15.150:0/413355724 learned_addr learned my addr 172.21.15.150:0/413355724 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f9023fff640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f90240c3b40 0x7f9024074eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f9028f17640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9024158170 0x7f9024154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f90237fe640 1 -- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f90240c3b40 msgr2=0x7f9024074eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f90237fe640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f90240c3b40 0x7f9024074eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f90237fe640 1 -- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9024158170 msgr2=0x7f9024154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f90237fe640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9024158170 0x7f9024154750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f90237fe640 1 -- 172.21.15.150:0/413355724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90240753f0 con 0x7f9024151d00 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f90237fe640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9024151d00 0x7f90241502a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f901401cb20 tx=0x7f90140a9430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f9020ff9640 1 -- 172.21.15.150:0/413355724 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90140ad070 con 0x7f9024151d00 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f9020ff9640 1 -- 172.21.15.150:0/413355724 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90140053b0 con 0x7f9024151d00 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f902a97a640 1 -- 172.21.15.150:0/413355724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902414d050 con 0x7f9024151d00 2024-09-17T04:49:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f9020ff9640 1 -- 172.21.15.150:0/413355724 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90140046d0 con 0x7f9024151d00 2024-09-17T04:49:51.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f902a97a640 1 -- 172.21.15.150:0/413355724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90241f3df0 con 0x7f9024151d00 2024-09-17T04:49:51.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.044+0000 7f902a97a640 1 -- 172.21.15.150:0/413355724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9024157c30 con 0x7f9024151d00 2024-09-17T04:49:51.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.048+0000 7f9020ff9640 1 -- 172.21.15.150:0/413355724 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f90140b0070 con 0x7f9024151d00 2024-09-17T04:49:51.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.048+0000 7f9020ff9640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9010073200 0x7f9010075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:51.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.048+0000 7f9020ff9640 1 -- 172.21.15.150:0/413355724 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f901412f000 con 0x7f9024151d00 2024-09-17T04:49:51.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.048+0000 7f9023fff640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9010073200 0x7f9010075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:51.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.048+0000 7f9023fff640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9010073200 0x7f9010075710 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9018000f70 tx=0x7f9018009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:51.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.052+0000 7f9020ff9640 1 -- 172.21.15.150:0/413355724 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9024070a60 con 0x7f9024151d00 2024-09-17T04:49:51.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.216+0000 7f902a97a640 1 -- 172.21.15.150:0/413355724 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f90240a7730 con 0x7f9010073200 2024-09-17T04:49:51.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.240+0000 7f9020ff9640 1 -- 172.21.15.150:0/413355724 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490296 (secure 0 0 0) 0x7f90240a7730 con 0x7f9010073200 2024-09-17T04:49:51.248 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:51.251 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T04:49:51.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.248+0000 7f902a97a640 1 -- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9010073200 msgr2=0x7f9010075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.248+0000 7f902a97a640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9010073200 0x7f9010075710 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9018000f70 tx=0x7f9018009040 comp rx=0 tx=0).stop 2024-09-17T04:49:51.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.248+0000 7f902a97a640 1 -- 172.21.15.150:0/413355724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9024151d00 msgr2=0x7f90241502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.248+0000 7f902a97a640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9024151d00 0x7f90241502a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f901401cb20 tx=0x7f90140a9430 comp rx=0 tx=0).stop 2024-09-17T04:49:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.248+0000 7f902a97a640 1 -- 172.21.15.150:0/413355724 shutdown_connections 2024-09-17T04:49:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.248+0000 7f902a97a640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9010073200 0x7f9010075710 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.248+0000 7f902a97a640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9024158170 0x7f9024154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.248+0000 7f902a97a640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9024151d00 0x7f90241502a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.248+0000 7f902a97a640 1 --2- 172.21.15.150:0/413355724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f90240c3b40 0x7f9024074eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.248+0000 7f902a97a640 1 -- 172.21.15.150:0/413355724 >> 172.21.15.150:0/413355724 conn(0x7f90240c4a00 msgr2=0x7f9024151320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.248+0000 7f902a97a640 1 -- 172.21.15.150:0/413355724 shutdown_connections 2024-09-17T04:49:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.248+0000 7f902a97a640 1 -- 172.21.15.150:0/413355724 wait complete. 2024-09-17T04:49:51.270 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-17T04:49:50.949097+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":191,"num_write_kb":1656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223812,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750908,"statfs":{"total":773094113280,"available":772864929792,"internally_reserved":0,"allocated":3481600,"data_stored":2027328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.950663"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671077+0000","last_change":"2024-09-17T04:49:41.025271+0000","last_active":"2024-09-17T04:49:42.671077+0000","last_peered":"2024-09-17T04:49:42.671077+0000","last_clean":"2024-09-17T04:49:42.671077+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T04:49:42.671077+0000","last_undegraded":"2024-09-17T04:49:42.671077+0000","last_fullsized":"2024-09-17T04:49:42.671077+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:39.715602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054640+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T04:49:42.054640+0000","last_peered":"2024-09-17T04:49:42.054640+0000","last_clean":"2024-09-17T04:49:42.054640+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T04:49:42.054640+0000","last_undegraded":"2024-09-17T04:49:42.054640+0000","last_fullsized":"2024-09-17T04:49:42.054640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671706+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T04:49:42.671706+0000","last_peered":"2024-09-17T04:49:42.671706+0000","last_clean":"2024-09-17T04:49:42.671706+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T04:49:42.671706+0000","last_undegraded":"2024-09-17T04:49:42.671706+0000","last_fullsized":"2024-09-17T04:49:42.671706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668736+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T04:49:42.668736+0000","last_peered":"2024-09-17T04:49:42.668736+0000","last_clean":"2024-09-17T04:49:42.668736+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T04:49:42.668736+0000","last_undegraded":"2024-09-17T04:49:42.668736+0000","last_fullsized":"2024-09-17T04:49:42.668736+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:38.417840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047972+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T04:49:42.047972+0000","last_peered":"2024-09-17T04:49:42.047972+0000","last_clean":"2024-09-17T04:49:42.047972+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T04:49:42.047972+0000","last_undegraded":"2024-09-17T04:49:42.047972+0000","last_fullsized":"2024-09-17T04:49:42.047972+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672300+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T04:49:42.672300+0000","last_peered":"2024-09-17T04:49:42.672300+0000","last_clean":"2024-09-17T04:49:42.672300+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T04:49:42.672300+0000","last_undegraded":"2024-09-17T04:49:42.672300+0000","last_fullsized":"2024-09-17T04:49:42.672300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672014+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T04:49:42.672014+0000","last_peered":"2024-09-17T04:49:42.672014+0000","last_clean":"2024-09-17T04:49:42.672014+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T04:49:42.672014+0000","last_undegraded":"2024-09-17T04:49:42.672014+0000","last_fullsized":"2024-09-17T04:49:42.672014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535134+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T04:49:42.535134+0000","last_peered":"2024-09-17T04:49:42.535134+0000","last_clean":"2024-09-17T04:49:42.535134+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T04:49:42.535134+0000","last_undegraded":"2024-09-17T04:49:42.535134+0000","last_fullsized":"2024-09-17T04:49:42.535134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672206+0000","last_change":"2024-09-17T04:49:41.067317+0000","last_active":"2024-09-17T04:49:42.672206+0000","last_peered":"2024-09-17T04:49:42.672206+0000","last_clean":"2024-09-17T04:49:42.672206+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T04:49:42.672206+0000","last_undegraded":"2024-09-17T04:49:42.672206+0000","last_fullsized":"2024-09-17T04:49:42.672206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:37.637477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670707+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T04:49:42.670707+0000","last_peered":"2024-09-17T04:49:42.670707+0000","last_clean":"2024-09-17T04:49:42.670707+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T04:49:42.670707+0000","last_undegraded":"2024-09-17T04:49:42.670707+0000","last_fullsized":"2024-09-17T04:49:42.670707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674274+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T04:49:42.674274+0000","last_peered":"2024-09-17T04:49:42.674274+0000","last_clean":"2024-09-17T04:49:42.674274+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T04:49:42.674274+0000","last_undegraded":"2024-09-17T04:49:42.674274+0000","last_fullsized":"2024-09-17T04:49:42.674274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535118+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T04:49:42.535118+0000","last_peered":"2024-09-17T04:49:42.535118+0000","last_clean":"2024-09-17T04:49:42.535118+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T04:49:42.535118+0000","last_undegraded":"2024-09-17T04:49:42.535118+0000","last_fullsized":"2024-09-17T04:49:42.535118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535259+0000","last_change":"2024-09-17T04:49:41.043531+0000","last_active":"2024-09-17T04:49:42.535259+0000","last_peered":"2024-09-17T04:49:42.535259+0000","last_clean":"2024-09-17T04:49:42.535259+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T04:49:42.535259+0000","last_undegraded":"2024-09-17T04:49:42.535259+0000","last_fullsized":"2024-09-17T04:49:42.535259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672162+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T04:49:42.672162+0000","last_peered":"2024-09-17T04:49:42.672162+0000","last_clean":"2024-09-17T04:49:42.672162+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T04:49:42.672162+0000","last_undegraded":"2024-09-17T04:49:42.672162+0000","last_fullsized":"2024-09-17T04:49:42.672162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042945+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T04:49:42.042945+0000","last_peered":"2024-09-17T04:49:42.042945+0000","last_clean":"2024-09-17T04:49:42.042945+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T04:49:42.042945+0000","last_undegraded":"2024-09-17T04:49:42.042945+0000","last_fullsized":"2024-09-17T04:49:42.042945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535003+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T04:49:42.535003+0000","last_peered":"2024-09-17T04:49:42.535003+0000","last_clean":"2024-09-17T04:49:42.535003+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T04:49:42.535003+0000","last_undegraded":"2024-09-17T04:49:42.535003+0000","last_fullsized":"2024-09-17T04:49:42.535003+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:16.854365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669588+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T04:49:42.669588+0000","last_peered":"2024-09-17T04:49:42.669588+0000","last_clean":"2024-09-17T04:49:42.669588+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T04:49:42.669588+0000","last_undegraded":"2024-09-17T04:49:42.669588+0000","last_fullsized":"2024-09-17T04:49:42.669588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042526+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T04:49:42.042526+0000","last_peered":"2024-09-17T04:49:42.042526+0000","last_clean":"2024-09-17T04:49:42.042526+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T04:49:42.042526+0000","last_undegraded":"2024-09-17T04:49:42.042526+0000","last_fullsized":"2024-09-17T04:49:42.042526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041407+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T04:49:42.041407+0000","last_peered":"2024-09-17T04:49:42.041407+0000","last_clean":"2024-09-17T04:49:42.041407+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T04:49:42.041407+0000","last_undegraded":"2024-09-17T04:49:42.041407+0000","last_fullsized":"2024-09-17T04:49:42.041407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042319+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T04:49:42.042319+0000","last_peered":"2024-09-17T04:49:42.042319+0000","last_clean":"2024-09-17T04:49:42.042319+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T04:49:42.042319+0000","last_undegraded":"2024-09-17T04:49:42.042319+0000","last_fullsized":"2024-09-17T04:49:42.042319+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:28.365399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669339+0000","last_change":"2024-09-17T04:49:41.038415+0000","last_active":"2024-09-17T04:49:42.669339+0000","last_peered":"2024-09-17T04:49:42.669339+0000","last_clean":"2024-09-17T04:49:42.669339+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T04:49:42.669339+0000","last_undegraded":"2024-09-17T04:49:42.669339+0000","last_fullsized":"2024-09-17T04:49:42.669339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:17.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668616+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T04:49:42.668616+0000","last_peered":"2024-09-17T04:49:42.668616+0000","last_clean":"2024-09-17T04:49:42.668616+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T04:49:42.668616+0000","last_undegraded":"2024-09-17T04:49:42.668616+0000","last_fullsized":"2024-09-17T04:49:42.668616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052248+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T04:49:42.052248+0000","last_peered":"2024-09-17T04:49:42.052248+0000","last_clean":"2024-09-17T04:49:42.052248+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T04:49:42.052248+0000","last_undegraded":"2024-09-17T04:49:42.052248+0000","last_fullsized":"2024-09-17T04:49:42.052248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671586+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T04:49:42.671586+0000","last_peered":"2024-09-17T04:49:42.671586+0000","last_clean":"2024-09-17T04:49:42.671586+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T04:49:42.671586+0000","last_undegraded":"2024-09-17T04:49:42.671586+0000","last_fullsized":"2024-09-17T04:49:42.671586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:17.424265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042028+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T04:49:42.042028+0000","last_peered":"2024-09-17T04:49:42.042028+0000","last_clean":"2024-09-17T04:49:42.042028+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T04:49:42.042028+0000","last_undegraded":"2024-09-17T04:49:42.042028+0000","last_fullsized":"2024-09-17T04:49:42.042028+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:06.516432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048455+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T04:49:42.048455+0000","last_peered":"2024-09-17T04:49:42.048455+0000","last_clean":"2024-09-17T04:49:42.048455+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T04:49:42.048455+0000","last_undegraded":"2024-09-17T04:49:42.048455+0000","last_fullsized":"2024-09-17T04:49:42.048455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048500+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T04:49:42.048500+0000","last_peered":"2024-09-17T04:49:42.048500+0000","last_clean":"2024-09-17T04:49:42.048500+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T04:49:42.048500+0000","last_undegraded":"2024-09-17T04:49:42.048500+0000","last_fullsized":"2024-09-17T04:49:42.048500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668827+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T04:49:42.668827+0000","last_peered":"2024-09-17T04:49:42.668827+0000","last_clean":"2024-09-17T04:49:42.668827+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T04:49:42.668827+0000","last_undegraded":"2024-09-17T04:49:42.668827+0000","last_fullsized":"2024-09-17T04:49:42.668827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:00.010365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675406+0000","last_change":"2024-09-17T04:49:41.036658+0000","last_active":"2024-09-17T04:49:42.675406+0000","last_peered":"2024-09-17T04:49:42.675406+0000","last_clean":"2024-09-17T04:49:42.675406+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T04:49:42.675406+0000","last_undegraded":"2024-09-17T04:49:42.675406+0000","last_fullsized":"2024-09-17T04:49:42.675406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042824+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T04:49:42.042824+0000","last_peered":"2024-09-17T04:49:42.042824+0000","last_clean":"2024-09-17T04:49:42.042824+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T04:49:42.042824+0000","last_undegraded":"2024-09-17T04:49:42.042824+0000","last_fullsized":"2024-09-17T04:49:42.042824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668829+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T04:49:42.668829+0000","last_peered":"2024-09-17T04:49:42.668829+0000","last_clean":"2024-09-17T04:49:42.668829+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T04:49:42.668829+0000","last_undegraded":"2024-09-17T04:49:42.668829+0000","last_fullsized":"2024-09-17T04:49:42.668829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536259+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T04:49:42.536259+0000","last_peered":"2024-09-17T04:49:42.536259+0000","last_clean":"2024-09-17T04:49:42.536259+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T04:49:42.536259+0000","last_undegraded":"2024-09-17T04:49:42.536259+0000","last_fullsized":"2024-09-17T04:49:42.536259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048942+0000","last_change":"2024-09-17T04:49:41.050170+0000","last_active":"2024-09-17T04:49:42.048942+0000","last_peered":"2024-09-17T04:49:42.048942+0000","last_clean":"2024-09-17T04:49:42.048942+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T04:49:42.048942+0000","last_undegraded":"2024-09-17T04:49:42.048942+0000","last_fullsized":"2024-09-17T04:49:42.048942+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051929+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T04:49:42.051929+0000","last_peered":"2024-09-17T04:49:42.051929+0000","last_clean":"2024-09-17T04:49:42.051929+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T04:49:42.051929+0000","last_undegraded":"2024-09-17T04:49:42.051929+0000","last_fullsized":"2024-09-17T04:49:42.051929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042185+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T04:49:42.042185+0000","last_peered":"2024-09-17T04:49:42.042185+0000","last_clean":"2024-09-17T04:49:42.042185+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T04:49:42.042185+0000","last_undegraded":"2024-09-17T04:49:42.042185+0000","last_fullsized":"2024-09-17T04:49:42.042185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535553+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T04:49:42.535553+0000","last_peered":"2024-09-17T04:49:42.535553+0000","last_clean":"2024-09-17T04:49:42.535553+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T04:49:42.535553+0000","last_undegraded":"2024-09-17T04:49:42.535553+0000","last_fullsized":"2024-09-17T04:49:42.535553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041211+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T04:49:42.041211+0000","last_peered":"2024-09-17T04:49:42.041211+0000","last_clean":"2024-09-17T04:49:42.041211+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T04:49:42.041211+0000","last_undegraded":"2024-09-17T04:49:42.041211+0000","last_fullsized":"2024-09-17T04:49:42.041211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054336+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T04:49:42.054336+0000","last_peered":"2024-09-17T04:49:42.054336+0000","last_clean":"2024-09-17T04:49:42.054336+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T04:49:42.054336+0000","last_undegraded":"2024-09-17T04:49:42.054336+0000","last_fullsized":"2024-09-17T04:49:42.054336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052826+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T04:49:42.052826+0000","last_peered":"2024-09-17T04:49:42.052826+0000","last_clean":"2024-09-17T04:49:42.052826+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T04:49:42.052826+0000","last_undegraded":"2024-09-17T04:49:42.052826+0000","last_fullsized":"2024-09-17T04:49:42.052826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048145+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T04:49:42.048145+0000","last_peered":"2024-09-17T04:49:42.048145+0000","last_clean":"2024-09-17T04:49:42.048145+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T04:49:42.048145+0000","last_undegraded":"2024-09-17T04:49:42.048145+0000","last_fullsized":"2024-09-17T04:49:42.048145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669693+0000","last_change":"2024-09-17T04:49:41.042314+0000","last_active":"2024-09-17T04:49:42.669693+0000","last_peered":"2024-09-17T04:49:42.669693+0000","last_clean":"2024-09-17T04:49:42.669693+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T04:49:42.669693+0000","last_undegraded":"2024-09-17T04:49:42.669693+0000","last_fullsized":"2024-09-17T04:49:42.669693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055299+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T04:49:42.055299+0000","last_peered":"2024-09-17T04:49:42.055299+0000","last_clean":"2024-09-17T04:49:42.055299+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T04:49:42.055299+0000","last_undegraded":"2024-09-17T04:49:42.055299+0000","last_fullsized":"2024-09-17T04:49:42.055299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043663+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T04:49:42.043663+0000","last_peered":"2024-09-17T04:49:42.043663+0000","last_clean":"2024-09-17T04:49:42.043663+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T04:49:42.043663+0000","last_undegraded":"2024-09-17T04:49:42.043663+0000","last_fullsized":"2024-09-17T04:49:42.043663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053169+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T04:49:42.053169+0000","last_peered":"2024-09-17T04:49:42.053169+0000","last_clean":"2024-09-17T04:49:42.053169+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T04:49:42.053169+0000","last_undegraded":"2024-09-17T04:49:42.053169+0000","last_fullsized":"2024-09-17T04:49:42.053169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052959+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T04:49:42.052959+0000","last_peered":"2024-09-17T04:49:42.052959+0000","last_clean":"2024-09-17T04:49:42.052959+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T04:49:42.052959+0000","last_undegraded":"2024-09-17T04:49:42.052959+0000","last_fullsized":"2024-09-17T04:49:42.052959+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535986+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T04:49:42.535986+0000","last_peered":"2024-09-17T04:49:42.535986+0000","last_clean":"2024-09-17T04:49:42.535986+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T04:49:42.535986+0000","last_undegraded":"2024-09-17T04:49:42.535986+0000","last_fullsized":"2024-09-17T04:49:42.535986+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670900+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T04:49:42.670900+0000","last_peered":"2024-09-17T04:49:42.670900+0000","last_clean":"2024-09-17T04:49:42.670900+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T04:49:42.670900+0000","last_undegraded":"2024-09-17T04:49:42.670900+0000","last_fullsized":"2024-09-17T04:49:42.670900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050191+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T04:49:42.050191+0000","last_peered":"2024-09-17T04:49:42.050191+0000","last_clean":"2024-09-17T04:49:42.050191+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T04:49:42.050191+0000","last_undegraded":"2024-09-17T04:49:42.050191+0000","last_fullsized":"2024-09-17T04:49:42.050191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670215+0000","last_change":"2024-09-17T04:49:41.069177+0000","last_active":"2024-09-17T04:49:42.670215+0000","last_peered":"2024-09-17T04:49:42.670215+0000","last_clean":"2024-09-17T04:49:42.670215+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T04:49:42.670215+0000","last_undegraded":"2024-09-17T04:49:42.670215+0000","last_fullsized":"2024-09-17T04:49:42.670215+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:16.855206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670381+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T04:49:42.670381+0000","last_peered":"2024-09-17T04:49:42.670381+0000","last_clean":"2024-09-17T04:49:42.670381+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T04:49:42.670381+0000","last_undegraded":"2024-09-17T04:49:42.670381+0000","last_fullsized":"2024-09-17T04:49:42.670381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042706+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T04:49:42.042706+0000","last_peered":"2024-09-17T04:49:42.042706+0000","last_clean":"2024-09-17T04:49:42.042706+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T04:49:42.042706+0000","last_undegraded":"2024-09-17T04:49:42.042706+0000","last_fullsized":"2024-09-17T04:49:42.042706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054900+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T04:49:42.054900+0000","last_peered":"2024-09-17T04:49:42.054900+0000","last_clean":"2024-09-17T04:49:42.054900+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T04:49:42.054900+0000","last_undegraded":"2024-09-17T04:49:42.054900+0000","last_fullsized":"2024-09-17T04:49:42.054900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669308+0000","last_change":"2024-09-17T04:49:41.068537+0000","last_active":"2024-09-17T04:49:42.669308+0000","last_peered":"2024-09-17T04:49:42.669308+0000","last_clean":"2024-09-17T04:49:42.669308+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T04:49:42.669308+0000","last_undegraded":"2024-09-17T04:49:42.669308+0000","last_fullsized":"2024-09-17T04:49:42.669308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667397+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T04:49:42.667397+0000","last_peered":"2024-09-17T04:49:42.667397+0000","last_clean":"2024-09-17T04:49:42.667397+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T04:49:42.667397+0000","last_undegraded":"2024-09-17T04:49:42.667397+0000","last_fullsized":"2024-09-17T04:49:42.667397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668940+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T04:49:42.668940+0000","last_peered":"2024-09-17T04:49:42.668940+0000","last_clean":"2024-09-17T04:49:42.668940+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T04:49:42.668940+0000","last_undegraded":"2024-09-17T04:49:42.668940+0000","last_fullsized":"2024-09-17T04:49:42.668940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054921+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T04:49:42.054921+0000","last_peered":"2024-09-17T04:49:42.054921+0000","last_clean":"2024-09-17T04:49:42.054921+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T04:49:42.054921+0000","last_undegraded":"2024-09-17T04:49:42.054921+0000","last_fullsized":"2024-09-17T04:49:42.054921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041762+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T04:49:42.041762+0000","last_peered":"2024-09-17T04:49:42.041762+0000","last_clean":"2024-09-17T04:49:42.041762+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T04:49:42.041762+0000","last_undegraded":"2024-09-17T04:49:42.041762+0000","last_fullsized":"2024-09-17T04:49:42.041762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050299+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T04:49:42.050299+0000","last_peered":"2024-09-17T04:49:42.050299+0000","last_clean":"2024-09-17T04:49:42.050299+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T04:49:42.050299+0000","last_undegraded":"2024-09-17T04:49:42.050299+0000","last_fullsized":"2024-09-17T04:49:42.050299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041771+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T04:49:42.041771+0000","last_peered":"2024-09-17T04:49:42.041771+0000","last_clean":"2024-09-17T04:49:42.041771+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T04:49:42.041771+0000","last_undegraded":"2024-09-17T04:49:42.041771+0000","last_fullsized":"2024-09-17T04:49:42.041771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672234+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T04:49:42.672234+0000","last_peered":"2024-09-17T04:49:42.672234+0000","last_clean":"2024-09-17T04:49:42.672234+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T04:49:42.672234+0000","last_undegraded":"2024-09-17T04:49:42.672234+0000","last_fullsized":"2024-09-17T04:49:42.672234+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671183+0000","last_change":"2024-09-17T04:49:41.052564+0000","last_active":"2024-09-17T04:49:42.671183+0000","last_peered":"2024-09-17T04:49:42.671183+0000","last_clean":"2024-09-17T04:49:42.671183+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T04:49:42.671183+0000","last_undegraded":"2024-09-17T04:49:42.671183+0000","last_fullsized":"2024-09-17T04:49:42.671183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049438+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T04:49:42.049438+0000","last_peered":"2024-09-17T04:49:42.049438+0000","last_clean":"2024-09-17T04:49:42.049438+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T04:49:42.049438+0000","last_undegraded":"2024-09-17T04:49:42.049438+0000","last_fullsized":"2024-09-17T04:49:42.049438+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041458+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T04:49:42.041458+0000","last_peered":"2024-09-17T04:49:42.041458+0000","last_clean":"2024-09-17T04:49:42.041458+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T04:49:42.041458+0000","last_undegraded":"2024-09-17T04:49:42.041458+0000","last_fullsized":"2024-09-17T04:49:42.041458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671735+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T04:49:42.671735+0000","last_peered":"2024-09-17T04:49:42.671735+0000","last_clean":"2024-09-17T04:49:42.671735+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T04:49:42.671735+0000","last_undegraded":"2024-09-17T04:49:42.671735+0000","last_fullsized":"2024-09-17T04:49:42.671735+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043078+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T04:49:42.043078+0000","last_peered":"2024-09-17T04:49:42.043078+0000","last_clean":"2024-09-17T04:49:42.043078+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T04:49:42.043078+0000","last_undegraded":"2024-09-17T04:49:42.043078+0000","last_fullsized":"2024-09-17T04:49:42.043078+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:21.202398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669578+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T04:49:42.669578+0000","last_peered":"2024-09-17T04:49:42.669578+0000","last_clean":"2024-09-17T04:49:42.669578+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T04:49:42.669578+0000","last_undegraded":"2024-09-17T04:49:42.669578+0000","last_fullsized":"2024-09-17T04:49:42.669578+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:48.128609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043280+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T04:49:42.043280+0000","last_peered":"2024-09-17T04:49:42.043280+0000","last_clean":"2024-09-17T04:49:42.043280+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T04:49:42.043280+0000","last_undegraded":"2024-09-17T04:49:42.043280+0000","last_fullsized":"2024-09-17T04:49:42.043280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055292+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T04:49:42.055292+0000","last_peered":"2024-09-17T04:49:42.055292+0000","last_clean":"2024-09-17T04:49:42.055292+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T04:49:42.055292+0000","last_undegraded":"2024-09-17T04:49:42.055292+0000","last_fullsized":"2024-09-17T04:49:42.055292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536164+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T04:49:42.536164+0000","last_peered":"2024-09-17T04:49:42.536164+0000","last_clean":"2024-09-17T04:49:42.536164+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T04:49:42.536164+0000","last_undegraded":"2024-09-17T04:49:42.536164+0000","last_fullsized":"2024-09-17T04:49:42.536164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050257+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T04:49:42.050257+0000","last_peered":"2024-09-17T04:49:42.050257+0000","last_clean":"2024-09-17T04:49:42.050257+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T04:49:42.050257+0000","last_undegraded":"2024-09-17T04:49:42.050257+0000","last_fullsized":"2024-09-17T04:49:42.050257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671927+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T04:49:42.671927+0000","last_peered":"2024-09-17T04:49:42.671927+0000","last_clean":"2024-09-17T04:49:42.671927+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T04:49:42.671927+0000","last_undegraded":"2024-09-17T04:49:42.671927+0000","last_fullsized":"2024-09-17T04:49:42.671927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671411+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T04:49:42.671411+0000","last_peered":"2024-09-17T04:49:42.671411+0000","last_clean":"2024-09-17T04:49:42.671411+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T04:49:42.671411+0000","last_undegraded":"2024-09-17T04:49:42.671411+0000","last_fullsized":"2024-09-17T04:49:42.671411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670087+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T04:49:42.670087+0000","last_peered":"2024-09-17T04:49:42.670087+0000","last_clean":"2024-09-17T04:49:42.670087+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T04:49:42.670087+0000","last_undegraded":"2024-09-17T04:49:42.670087+0000","last_fullsized":"2024-09-17T04:49:42.670087+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:18.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672287+0000","last_change":"2024-09-17T04:49:41.031036+0000","last_active":"2024-09-17T04:49:42.672287+0000","last_peered":"2024-09-17T04:49:42.672287+0000","last_clean":"2024-09-17T04:49:42.672287+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T04:49:42.672287+0000","last_undegraded":"2024-09-17T04:49:42.672287+0000","last_fullsized":"2024-09-17T04:49:42.672287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:48.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672111+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T04:49:42.672111+0000","last_peered":"2024-09-17T04:49:42.672111+0000","last_clean":"2024-09-17T04:49:42.672111+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T04:49:42.672111+0000","last_undegraded":"2024-09-17T04:49:42.672111+0000","last_fullsized":"2024-09-17T04:49:42.672111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052688+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T04:49:42.052688+0000","last_peered":"2024-09-17T04:49:42.052688+0000","last_clean":"2024-09-17T04:49:42.052688+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T04:49:42.052688+0000","last_undegraded":"2024-09-17T04:49:42.052688+0000","last_fullsized":"2024-09-17T04:49:42.052688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041095+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T04:49:42.041095+0000","last_peered":"2024-09-17T04:49:42.041095+0000","last_clean":"2024-09-17T04:49:42.041095+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T04:49:42.041095+0000","last_undegraded":"2024-09-17T04:49:42.041095+0000","last_fullsized":"2024-09-17T04:49:42.041095+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048488+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T04:49:42.048488+0000","last_peered":"2024-09-17T04:49:42.048488+0000","last_clean":"2024-09-17T04:49:42.048488+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T04:49:42.048488+0000","last_undegraded":"2024-09-17T04:49:42.048488+0000","last_fullsized":"2024-09-17T04:49:42.048488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536967+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T04:49:42.536967+0000","last_peered":"2024-09-17T04:49:42.536967+0000","last_clean":"2024-09-17T04:49:42.536967+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T04:49:42.536967+0000","last_undegraded":"2024-09-17T04:49:42.536967+0000","last_fullsized":"2024-09-17T04:49:42.536967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672404+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T04:49:42.672404+0000","last_peered":"2024-09-17T04:49:42.672404+0000","last_clean":"2024-09-17T04:49:42.672404+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T04:49:42.672404+0000","last_undegraded":"2024-09-17T04:49:42.672404+0000","last_fullsized":"2024-09-17T04:49:42.672404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672966+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T04:49:42.672966+0000","last_peered":"2024-09-17T04:49:42.672966+0000","last_clean":"2024-09-17T04:49:42.672966+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T04:49:42.672966+0000","last_undegraded":"2024-09-17T04:49:42.672966+0000","last_fullsized":"2024-09-17T04:49:42.672966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:09.712272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052007+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T04:49:42.052007+0000","last_peered":"2024-09-17T04:49:42.052007+0000","last_clean":"2024-09-17T04:49:42.052007+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T04:49:42.052007+0000","last_undegraded":"2024-09-17T04:49:42.052007+0000","last_fullsized":"2024-09-17T04:49:42.052007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:43.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042787+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T04:49:42.042787+0000","last_peered":"2024-09-17T04:49:42.042787+0000","last_clean":"2024-09-17T04:49:42.042787+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T04:49:42.042787+0000","last_undegraded":"2024-09-17T04:49:42.042787+0000","last_fullsized":"2024-09-17T04:49:42.042787+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049500+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T04:49:42.049500+0000","last_peered":"2024-09-17T04:49:42.049500+0000","last_clean":"2024-09-17T04:49:42.049500+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T04:49:42.049500+0000","last_undegraded":"2024-09-17T04:49:42.049500+0000","last_fullsized":"2024-09-17T04:49:42.049500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671253+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T04:49:42.671253+0000","last_peered":"2024-09-17T04:49:42.671253+0000","last_clean":"2024-09-17T04:49:42.671253+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T04:49:42.671253+0000","last_undegraded":"2024-09-17T04:49:42.671253+0000","last_fullsized":"2024-09-17T04:49:42.671253+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:52.364568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041504+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T04:49:42.041504+0000","last_peered":"2024-09-17T04:49:42.041504+0000","last_clean":"2024-09-17T04:49:42.041504+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T04:49:42.041504+0000","last_undegraded":"2024-09-17T04:49:42.041504+0000","last_fullsized":"2024-09-17T04:49:42.041504+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:03.537392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671439+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T04:49:42.671439+0000","last_peered":"2024-09-17T04:49:42.671439+0000","last_clean":"2024-09-17T04:49:42.671439+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T04:49:42.671439+0000","last_undegraded":"2024-09-17T04:49:42.671439+0000","last_fullsized":"2024-09-17T04:49:42.671439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670943+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T04:49:42.670943+0000","last_peered":"2024-09-17T04:49:42.670943+0000","last_clean":"2024-09-17T04:49:42.670943+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T04:49:42.670943+0000","last_undegraded":"2024-09-17T04:49:42.670943+0000","last_fullsized":"2024-09-17T04:49:42.670943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671787+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T04:49:42.671787+0000","last_peered":"2024-09-17T04:49:42.671787+0000","last_clean":"2024-09-17T04:49:42.671787+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T04:49:42.671787+0000","last_undegraded":"2024-09-17T04:49:42.671787+0000","last_fullsized":"2024-09-17T04:49:42.671787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:52.120543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041402+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T04:49:42.041402+0000","last_peered":"2024-09-17T04:49:42.041402+0000","last_clean":"2024-09-17T04:49:42.041402+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T04:49:42.041402+0000","last_undegraded":"2024-09-17T04:49:42.041402+0000","last_fullsized":"2024-09-17T04:49:42.041402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:56.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041285+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T04:49:42.041285+0000","last_peered":"2024-09-17T04:49:42.041285+0000","last_clean":"2024-09-17T04:49:42.041285+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T04:49:42.041285+0000","last_undegraded":"2024-09-17T04:49:42.041285+0000","last_fullsized":"2024-09-17T04:49:42.041285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042566+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T04:49:42.042566+0000","last_peered":"2024-09-17T04:49:42.042566+0000","last_clean":"2024-09-17T04:49:42.042566+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T04:49:42.042566+0000","last_undegraded":"2024-09-17T04:49:42.042566+0000","last_fullsized":"2024-09-17T04:49:42.042566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054288+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T04:49:42.054288+0000","last_peered":"2024-09-17T04:49:42.054288+0000","last_clean":"2024-09-17T04:49:42.054288+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T04:49:42.054288+0000","last_undegraded":"2024-09-17T04:49:42.054288+0000","last_fullsized":"2024-09-17T04:49:42.054288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051153+0000","last_change":"2024-09-17T04:49:41.058280+0000","last_active":"2024-09-17T04:49:42.051153+0000","last_peered":"2024-09-17T04:49:42.051153+0000","last_clean":"2024-09-17T04:49:42.051153+0000","last_became_active":"2024-09-17T04:49:41.052009+0000","last_became_peered":"2024-09-17T04:49:41.052009+0000","last_unstale":"2024-09-17T04:49:42.051153+0000","last_undegraded":"2024-09-17T04:49:42.051153+0000","last_fullsized":"2024-09-17T04:49:42.051153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:14.100818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051977+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T04:49:42.051977+0000","last_peered":"2024-09-17T04:49:42.051977+0000","last_clean":"2024-09-17T04:49:42.051977+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T04:49:42.051977+0000","last_undegraded":"2024-09-17T04:49:42.051977+0000","last_fullsized":"2024-09-17T04:49:42.051977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669355+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T04:49:42.669355+0000","last_peered":"2024-09-17T04:49:42.669355+0000","last_clean":"2024-09-17T04:49:42.669355+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T04:49:42.669355+0000","last_undegraded":"2024-09-17T04:49:42.669355+0000","last_fullsized":"2024-09-17T04:49:42.669355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042782+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T04:49:42.042782+0000","last_peered":"2024-09-17T04:49:42.042782+0000","last_clean":"2024-09-17T04:49:42.042782+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T04:49:42.042782+0000","last_undegraded":"2024-09-17T04:49:42.042782+0000","last_fullsized":"2024-09-17T04:49:42.042782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:07.945355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052280+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T04:49:42.052280+0000","last_peered":"2024-09-17T04:49:42.052280+0000","last_clean":"2024-09-17T04:49:42.052280+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T04:49:42.052280+0000","last_undegraded":"2024-09-17T04:49:42.052280+0000","last_fullsized":"2024-09-17T04:49:42.052280+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670412+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T04:49:42.670412+0000","last_peered":"2024-09-17T04:49:42.670412+0000","last_clean":"2024-09-17T04:49:42.670412+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T04:49:42.670412+0000","last_undegraded":"2024-09-17T04:49:42.670412+0000","last_fullsized":"2024-09-17T04:49:42.670412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537054+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T04:49:42.537054+0000","last_peered":"2024-09-17T04:49:42.537054+0000","last_clean":"2024-09-17T04:49:42.537054+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T04:49:42.537054+0000","last_undegraded":"2024-09-17T04:49:42.537054+0000","last_fullsized":"2024-09-17T04:49:42.537054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048913+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T04:49:42.048913+0000","last_peered":"2024-09-17T04:49:42.048913+0000","last_clean":"2024-09-17T04:49:42.048913+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T04:49:42.048913+0000","last_undegraded":"2024-09-17T04:49:42.048913+0000","last_fullsized":"2024-09-17T04:49:42.048913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:16.447835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051373+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T04:49:42.051373+0000","last_peered":"2024-09-17T04:49:42.051373+0000","last_clean":"2024-09-17T04:49:42.051373+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T04:49:42.051373+0000","last_undegraded":"2024-09-17T04:49:42.051373+0000","last_fullsized":"2024-09-17T04:49:42.051373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042063+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T04:49:42.042063+0000","last_peered":"2024-09-17T04:49:42.042063+0000","last_clean":"2024-09-17T04:49:42.042063+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T04:49:42.042063+0000","last_undegraded":"2024-09-17T04:49:42.042063+0000","last_fullsized":"2024-09-17T04:49:42.042063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670596+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T04:49:42.670596+0000","last_peered":"2024-09-17T04:49:42.670596+0000","last_clean":"2024-09-17T04:49:42.670596+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T04:49:42.670596+0000","last_undegraded":"2024-09-17T04:49:42.670596+0000","last_fullsized":"2024-09-17T04:49:42.670596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535510+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T04:49:42.535510+0000","last_peered":"2024-09-17T04:49:42.535510+0000","last_clean":"2024-09-17T04:49:42.535510+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T04:49:42.535510+0000","last_undegraded":"2024-09-17T04:49:42.535510+0000","last_fullsized":"2024-09-17T04:49:42.535510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041322+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T04:49:42.041322+0000","last_peered":"2024-09-17T04:49:42.041322+0000","last_clean":"2024-09-17T04:49:42.041322+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T04:49:42.041322+0000","last_undegraded":"2024-09-17T04:49:42.041322+0000","last_fullsized":"2024-09-17T04:49:42.041322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672182+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T04:49:42.672182+0000","last_peered":"2024-09-17T04:49:42.672182+0000","last_clean":"2024-09-17T04:49:42.672182+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T04:49:42.672182+0000","last_undegraded":"2024-09-17T04:49:42.672182+0000","last_fullsized":"2024-09-17T04:49:42.672182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671261+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T04:49:42.671261+0000","last_peered":"2024-09-17T04:49:42.671261+0000","last_clean":"2024-09-17T04:49:42.671261+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T04:49:42.671261+0000","last_undegraded":"2024-09-17T04:49:42.671261+0000","last_fullsized":"2024-09-17T04:49:42.671261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667575+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T04:49:42.667575+0000","last_peered":"2024-09-17T04:49:42.667575+0000","last_clean":"2024-09-17T04:49:42.667575+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T04:49:42.667575+0000","last_undegraded":"2024-09-17T04:49:42.667575+0000","last_fullsized":"2024-09-17T04:49:42.667575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:40.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534654+0000","last_change":"2024-09-17T04:49:41.040278+0000","last_active":"2024-09-17T04:49:42.534654+0000","last_peered":"2024-09-17T04:49:42.534654+0000","last_clean":"2024-09-17T04:49:42.534654+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T04:49:42.534654+0000","last_undegraded":"2024-09-17T04:49:42.534654+0000","last_fullsized":"2024-09-17T04:49:42.534654+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.895223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536732+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T04:49:42.536732+0000","last_peered":"2024-09-17T04:49:42.536732+0000","last_clean":"2024-09-17T04:49:42.536732+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T04:49:42.536732+0000","last_undegraded":"2024-09-17T04:49:42.536732+0000","last_fullsized":"2024-09-17T04:49:42.536732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675753+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T04:49:42.675753+0000","last_peered":"2024-09-17T04:49:42.675753+0000","last_clean":"2024-09-17T04:49:42.675753+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T04:49:42.675753+0000","last_undegraded":"2024-09-17T04:49:42.675753+0000","last_fullsized":"2024-09-17T04:49:42.675753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047646+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T04:49:42.047646+0000","last_peered":"2024-09-17T04:49:42.047646+0000","last_clean":"2024-09-17T04:49:42.047646+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T04:49:42.047646+0000","last_undegraded":"2024-09-17T04:49:42.047646+0000","last_fullsized":"2024-09-17T04:49:42.047646+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:15.845612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537298+0000","last_change":"2024-09-17T04:49:41.050551+0000","last_active":"2024-09-17T04:49:42.537298+0000","last_peered":"2024-09-17T04:49:42.537298+0000","last_clean":"2024-09-17T04:49:42.537298+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T04:49:42.537298+0000","last_undegraded":"2024-09-17T04:49:42.537298+0000","last_fullsized":"2024-09-17T04:49:42.537298+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:11.088230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051592+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T04:49:42.051592+0000","last_peered":"2024-09-17T04:49:42.051592+0000","last_clean":"2024-09-17T04:49:42.051592+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T04:49:42.051592+0000","last_undegraded":"2024-09-17T04:49:42.051592+0000","last_fullsized":"2024-09-17T04:49:42.051592+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:37.778756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048401+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T04:49:42.048401+0000","last_peered":"2024-09-17T04:49:42.048401+0000","last_clean":"2024-09-17T04:49:42.048401+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T04:49:42.048401+0000","last_undegraded":"2024-09-17T04:49:42.048401+0000","last_fullsized":"2024-09-17T04:49:42.048401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041797+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T04:49:42.041797+0000","last_peered":"2024-09-17T04:49:42.041797+0000","last_clean":"2024-09-17T04:49:42.041797+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T04:49:42.041797+0000","last_undegraded":"2024-09-17T04:49:42.041797+0000","last_fullsized":"2024-09-17T04:49:42.041797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055365+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T04:49:42.055365+0000","last_peered":"2024-09-17T04:49:42.055365+0000","last_clean":"2024-09-17T04:49:42.055365+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T04:49:42.055365+0000","last_undegraded":"2024-09-17T04:49:42.055365+0000","last_fullsized":"2024-09-17T04:49:42.055365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041767+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T04:49:42.041767+0000","last_peered":"2024-09-17T04:49:42.041767+0000","last_clean":"2024-09-17T04:49:42.041767+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T04:49:42.041767+0000","last_undegraded":"2024-09-17T04:49:42.041767+0000","last_fullsized":"2024-09-17T04:49:42.041767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667708+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T04:49:42.667708+0000","last_peered":"2024-09-17T04:49:42.667708+0000","last_clean":"2024-09-17T04:49:42.667708+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T04:49:42.667708+0000","last_undegraded":"2024-09-17T04:49:42.667708+0000","last_fullsized":"2024-09-17T04:49:42.667708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050522+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T04:49:42.050522+0000","last_peered":"2024-09-17T04:49:42.050522+0000","last_clean":"2024-09-17T04:49:42.050522+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T04:49:42.050522+0000","last_undegraded":"2024-09-17T04:49:42.050522+0000","last_fullsized":"2024-09-17T04:49:42.050522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049954+0000","last_change":"2024-09-17T04:49:41.057647+0000","last_active":"2024-09-17T04:49:42.049954+0000","last_peered":"2024-09-17T04:49:42.049954+0000","last_clean":"2024-09-17T04:49:42.049954+0000","last_became_active":"2024-09-17T04:49:41.056213+0000","last_became_peered":"2024-09-17T04:49:41.056213+0000","last_unstale":"2024-09-17T04:49:42.049954+0000","last_undegraded":"2024-09-17T04:49:42.049954+0000","last_fullsized":"2024-09-17T04:49:42.049954+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042498+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T04:49:42.042498+0000","last_peered":"2024-09-17T04:49:42.042498+0000","last_clean":"2024-09-17T04:49:42.042498+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T04:49:42.042498+0000","last_undegraded":"2024-09-17T04:49:42.042498+0000","last_fullsized":"2024-09-17T04:49:42.042498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053714+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T04:49:42.053714+0000","last_peered":"2024-09-17T04:49:42.053714+0000","last_clean":"2024-09-17T04:49:42.053714+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T04:49:42.053714+0000","last_undegraded":"2024-09-17T04:49:42.053714+0000","last_fullsized":"2024-09-17T04:49:42.053714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051282+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T04:49:42.051282+0000","last_peered":"2024-09-17T04:49:42.051282+0000","last_clean":"2024-09-17T04:49:42.051282+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T04:49:42.051282+0000","last_undegraded":"2024-09-17T04:49:42.051282+0000","last_fullsized":"2024-09-17T04:49:42.051282+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:27.230604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536526+0000","last_change":"2024-09-17T04:49:41.055263+0000","last_active":"2024-09-17T04:49:42.536526+0000","last_peered":"2024-09-17T04:49:42.536526+0000","last_clean":"2024-09-17T04:49:42.536526+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T04:49:42.536526+0000","last_undegraded":"2024-09-17T04:49:42.536526+0000","last_fullsized":"2024-09-17T04:49:42.536526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670438+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T04:49:42.670438+0000","last_peered":"2024-09-17T04:49:42.670438+0000","last_clean":"2024-09-17T04:49:42.670438+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T04:49:42.670438+0000","last_undegraded":"2024-09-17T04:49:42.670438+0000","last_fullsized":"2024-09-17T04:49:42.670438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673048+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T04:49:42.673048+0000","last_peered":"2024-09-17T04:49:42.673048+0000","last_clean":"2024-09-17T04:49:42.673048+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T04:49:42.673048+0000","last_undegraded":"2024-09-17T04:49:42.673048+0000","last_fullsized":"2024-09-17T04:49:42.673048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670447+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T04:49:42.670447+0000","last_peered":"2024-09-17T04:49:42.670447+0000","last_clean":"2024-09-17T04:49:42.670447+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T04:49:42.670447+0000","last_undegraded":"2024-09-17T04:49:42.670447+0000","last_fullsized":"2024-09-17T04:49:42.670447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:24.586834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051128+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.051128+0000","last_peered":"2024-09-17T04:49:42.051128+0000","last_clean":"2024-09-17T04:49:42.051128+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T04:49:42.051128+0000","last_undegraded":"2024-09-17T04:49:42.051128+0000","last_fullsized":"2024-09-17T04:49:42.051128+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536163+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T04:49:42.536163+0000","last_peered":"2024-09-17T04:49:42.536163+0000","last_clean":"2024-09-17T04:49:42.536163+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T04:49:42.536163+0000","last_undegraded":"2024-09-17T04:49:42.536163+0000","last_fullsized":"2024-09-17T04:49:42.536163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052121+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T04:49:42.052121+0000","last_peered":"2024-09-17T04:49:42.052121+0000","last_clean":"2024-09-17T04:49:42.052121+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T04:49:42.052121+0000","last_undegraded":"2024-09-17T04:49:42.052121+0000","last_fullsized":"2024-09-17T04:49:42.052121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668467+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T04:49:42.668467+0000","last_peered":"2024-09-17T04:49:42.668467+0000","last_clean":"2024-09-17T04:49:42.668467+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T04:49:42.668467+0000","last_undegraded":"2024-09-17T04:49:42.668467+0000","last_fullsized":"2024-09-17T04:49:42.668467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536096+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T04:49:42.536096+0000","last_peered":"2024-09-17T04:49:42.536096+0000","last_clean":"2024-09-17T04:49:42.536096+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T04:49:42.536096+0000","last_undegraded":"2024-09-17T04:49:42.536096+0000","last_fullsized":"2024-09-17T04:49:42.536096+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042322+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T04:49:42.042322+0000","last_peered":"2024-09-17T04:49:42.042322+0000","last_clean":"2024-09-17T04:49:42.042322+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T04:49:42.042322+0000","last_undegraded":"2024-09-17T04:49:42.042322+0000","last_fullsized":"2024-09-17T04:49:42.042322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671200+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T04:49:42.671200+0000","last_peered":"2024-09-17T04:49:42.671200+0000","last_clean":"2024-09-17T04:49:42.671200+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T04:49:42.671200+0000","last_undegraded":"2024-09-17T04:49:42.671200+0000","last_fullsized":"2024-09-17T04:49:42.671200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670309+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T04:49:42.670309+0000","last_peered":"2024-09-17T04:49:42.670309+0000","last_clean":"2024-09-17T04:49:42.670309+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T04:49:42.670309+0000","last_undegraded":"2024-09-17T04:49:42.670309+0000","last_fullsized":"2024-09-17T04:49:42.670309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668969+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T04:49:42.668969+0000","last_peered":"2024-09-17T04:49:42.668969+0000","last_clean":"2024-09-17T04:49:42.668969+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T04:49:42.668969+0000","last_undegraded":"2024-09-17T04:49:42.668969+0000","last_fullsized":"2024-09-17T04:49:42.668969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054935+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T04:49:42.054935+0000","last_peered":"2024-09-17T04:49:42.054935+0000","last_clean":"2024-09-17T04:49:42.054935+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T04:49:42.054935+0000","last_undegraded":"2024-09-17T04:49:42.054935+0000","last_fullsized":"2024-09-17T04:49:42.054935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669059+0000","last_change":"2024-09-17T04:49:41.073709+0000","last_active":"2024-09-17T04:49:42.669059+0000","last_peered":"2024-09-17T04:49:42.669059+0000","last_clean":"2024-09-17T04:49:42.669059+0000","last_became_active":"2024-09-17T04:49:41.073330+0000","last_became_peered":"2024-09-17T04:49:41.073330+0000","last_unstale":"2024-09-17T04:49:42.669059+0000","last_undegraded":"2024-09-17T04:49:42.669059+0000","last_fullsized":"2024-09-17T04:49:42.669059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054811+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T04:49:42.054811+0000","last_peered":"2024-09-17T04:49:42.054811+0000","last_clean":"2024-09-17T04:49:42.054811+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T04:49:42.054811+0000","last_undegraded":"2024-09-17T04:49:42.054811+0000","last_fullsized":"2024-09-17T04:49:42.054811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041256+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T04:49:42.041256+0000","last_peered":"2024-09-17T04:49:42.041256+0000","last_clean":"2024-09-17T04:49:42.041256+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T04:49:42.041256+0000","last_undegraded":"2024-09-17T04:49:42.041256+0000","last_fullsized":"2024-09-17T04:49:42.041256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536669+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T04:49:42.536669+0000","last_peered":"2024-09-17T04:49:42.536669+0000","last_clean":"2024-09-17T04:49:42.536669+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T04:49:42.536669+0000","last_undegraded":"2024-09-17T04:49:42.536669+0000","last_fullsized":"2024-09-17T04:49:42.536669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534968+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T04:49:42.534968+0000","last_peered":"2024-09-17T04:49:42.534968+0000","last_clean":"2024-09-17T04:49:42.534968+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T04:49:42.534968+0000","last_undegraded":"2024-09-17T04:49:42.534968+0000","last_fullsized":"2024-09-17T04:49:42.534968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672019+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T04:49:42.672019+0000","last_peered":"2024-09-17T04:49:42.672019+0000","last_clean":"2024-09-17T04:49:42.672019+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T04:49:42.672019+0000","last_undegraded":"2024-09-17T04:49:42.672019+0000","last_fullsized":"2024-09-17T04:49:42.672019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668565+0000","last_change":"2024-09-17T04:49:41.045502+0000","last_active":"2024-09-17T04:49:42.668565+0000","last_peered":"2024-09-17T04:49:42.668565+0000","last_clean":"2024-09-17T04:49:42.668565+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T04:49:42.668565+0000","last_undegraded":"2024-09-17T04:49:42.668565+0000","last_fullsized":"2024-09-17T04:49:42.668565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:19.746356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041666+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T04:49:42.041666+0000","last_peered":"2024-09-17T04:49:42.041666+0000","last_clean":"2024-09-17T04:49:42.041666+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T04:49:42.041666+0000","last_undegraded":"2024-09-17T04:49:42.041666+0000","last_fullsized":"2024-09-17T04:49:42.041666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042310+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T04:49:42.042310+0000","last_peered":"2024-09-17T04:49:42.042310+0000","last_clean":"2024-09-17T04:49:42.042310+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T04:49:42.042310+0000","last_undegraded":"2024-09-17T04:49:42.042310+0000","last_fullsized":"2024-09-17T04:49:42.042310+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668045+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T04:49:42.668045+0000","last_peered":"2024-09-17T04:49:42.668045+0000","last_clean":"2024-09-17T04:49:42.668045+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T04:49:42.668045+0000","last_undegraded":"2024-09-17T04:49:42.668045+0000","last_fullsized":"2024-09-17T04:49:42.668045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054699+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T04:49:42.054699+0000","last_peered":"2024-09-17T04:49:42.054699+0000","last_clean":"2024-09-17T04:49:42.054699+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T04:49:42.054699+0000","last_undegraded":"2024-09-17T04:49:42.054699+0000","last_fullsized":"2024-09-17T04:49:42.054699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673038+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T04:49:42.673038+0000","last_peered":"2024-09-17T04:49:42.673038+0000","last_clean":"2024-09-17T04:49:42.673038+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T04:49:42.673038+0000","last_undegraded":"2024-09-17T04:49:42.673038+0000","last_fullsized":"2024-09-17T04:49:42.673038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669104+0000","last_change":"2024-09-17T04:49:41.038265+0000","last_active":"2024-09-17T04:49:42.669104+0000","last_peered":"2024-09-17T04:49:42.669104+0000","last_clean":"2024-09-17T04:49:42.669104+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T04:49:42.669104+0000","last_undegraded":"2024-09-17T04:49:42.669104+0000","last_fullsized":"2024-09-17T04:49:42.669104+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:41.638403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053416+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T04:49:42.053416+0000","last_peered":"2024-09-17T04:49:42.053416+0000","last_clean":"2024-09-17T04:49:42.053416+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T04:49:42.053416+0000","last_undegraded":"2024-09-17T04:49:42.053416+0000","last_fullsized":"2024-09-17T04:49:42.053416+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535001+0000","last_change":"2024-09-17T04:49:41.053443+0000","last_active":"2024-09-17T04:49:42.535001+0000","last_peered":"2024-09-17T04:49:42.535001+0000","last_clean":"2024-09-17T04:49:42.535001+0000","last_became_active":"2024-09-17T04:49:41.053090+0000","last_became_peered":"2024-09-17T04:49:41.053090+0000","last_unstale":"2024-09-17T04:49:42.535001+0000","last_undegraded":"2024-09-17T04:49:42.535001+0000","last_fullsized":"2024-09-17T04:49:42.535001+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671599+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T04:49:42.671599+0000","last_peered":"2024-09-17T04:49:42.671599+0000","last_clean":"2024-09-17T04:49:42.671599+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T04:49:42.671599+0000","last_undegraded":"2024-09-17T04:49:42.671599+0000","last_fullsized":"2024-09-17T04:49:42.671599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669669+0000","last_change":"2024-09-17T04:49:41.038072+0000","last_active":"2024-09-17T04:49:42.669669+0000","last_peered":"2024-09-17T04:49:42.669669+0000","last_clean":"2024-09-17T04:49:42.669669+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T04:49:42.669669+0000","last_undegraded":"2024-09-17T04:49:42.669669+0000","last_fullsized":"2024-09-17T04:49:42.669669+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043492+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T04:49:42.043492+0000","last_peered":"2024-09-17T04:49:42.043492+0000","last_clean":"2024-09-17T04:49:42.043492+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T04:49:42.043492+0000","last_undegraded":"2024-09-17T04:49:42.043492+0000","last_fullsized":"2024-09-17T04:49:42.043492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534462+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T04:49:42.534462+0000","last_peered":"2024-09-17T04:49:42.534462+0000","last_clean":"2024-09-17T04:49:42.534462+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T04:49:42.534462+0000","last_undegraded":"2024-09-17T04:49:42.534462+0000","last_fullsized":"2024-09-17T04:49:42.534462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669602+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T04:49:42.669602+0000","last_peered":"2024-09-17T04:49:42.669602+0000","last_clean":"2024-09-17T04:49:42.669602+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T04:49:42.669602+0000","last_undegraded":"2024-09-17T04:49:42.669602+0000","last_fullsized":"2024-09-17T04:49:42.669602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050316+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T04:49:42.050316+0000","last_peered":"2024-09-17T04:49:42.050316+0000","last_clean":"2024-09-17T04:49:42.050316+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T04:49:42.050316+0000","last_undegraded":"2024-09-17T04:49:42.050316+0000","last_fullsized":"2024-09-17T04:49:42.050316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042123+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T04:49:42.042123+0000","last_peered":"2024-09-17T04:49:42.042123+0000","last_clean":"2024-09-17T04:49:42.042123+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T04:49:42.042123+0000","last_undegraded":"2024-09-17T04:49:42.042123+0000","last_fullsized":"2024-09-17T04:49:42.042123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052378+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T04:49:42.052378+0000","last_peered":"2024-09-17T04:49:42.052378+0000","last_clean":"2024-09-17T04:49:42.052378+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T04:49:42.052378+0000","last_undegraded":"2024-09-17T04:49:42.052378+0000","last_fullsized":"2024-09-17T04:49:42.052378+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041588+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T04:49:42.041588+0000","last_peered":"2024-09-17T04:49:42.041588+0000","last_clean":"2024-09-17T04:49:42.041588+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T04:49:42.041588+0000","last_undegraded":"2024-09-17T04:49:42.041588+0000","last_fullsized":"2024-09-17T04:49:42.041588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674897+0000","last_change":"2024-09-17T04:49:41.067207+0000","last_active":"2024-09-17T04:49:42.674897+0000","last_peered":"2024-09-17T04:49:42.674897+0000","last_clean":"2024-09-17T04:49:42.674897+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T04:49:42.674897+0000","last_undegraded":"2024-09-17T04:49:42.674897+0000","last_fullsized":"2024-09-17T04:49:42.674897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:08.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667484+0000","last_change":"2024-09-17T04:49:31.911036+0000","last_active":"2024-09-17T04:49:42.667484+0000","last_peered":"2024-09-17T04:49:42.667484+0000","last_clean":"2024-09-17T04:49:42.667484+0000","last_became_active":"2024-09-17T04:49:31.910285+0000","last_became_peered":"2024-09-17T04:49:31.910285+0000","last_unstale":"2024-09-17T04:49:42.667484+0000","last_undegraded":"2024-09-17T04:49:42.667484+0000","last_fullsized":"2024-09-17T04:49:42.667484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673474+0000","last_change":"2024-09-17T04:49:41.046170+0000","last_active":"2024-09-17T04:49:42.673474+0000","last_peered":"2024-09-17T04:49:42.673474+0000","last_clean":"2024-09-17T04:49:42.673474+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T04:49:42.673474+0000","last_undegraded":"2024-09-17T04:49:42.673474+0000","last_fullsized":"2024-09-17T04:49:42.673474+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:01.683826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051528+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T04:49:42.051528+0000","last_peered":"2024-09-17T04:49:42.051528+0000","last_clean":"2024-09-17T04:49:42.051528+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T04:49:42.051528+0000","last_undegraded":"2024-09-17T04:49:42.051528+0000","last_fullsized":"2024-09-17T04:49:42.051528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041843+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T04:49:42.041843+0000","last_peered":"2024-09-17T04:49:42.041843+0000","last_clean":"2024-09-17T04:49:42.041843+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T04:49:42.041843+0000","last_undegraded":"2024-09-17T04:49:42.041843+0000","last_fullsized":"2024-09-17T04:49:42.041843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:25.574682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053942+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T04:49:42.053942+0000","last_peered":"2024-09-17T04:49:42.053942+0000","last_clean":"2024-09-17T04:49:42.053942+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T04:49:42.053942+0000","last_undegraded":"2024-09-17T04:49:42.053942+0000","last_fullsized":"2024-09-17T04:49:42.053942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048414+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T04:49:42.048414+0000","last_peered":"2024-09-17T04:49:42.048414+0000","last_clean":"2024-09-17T04:49:42.048414+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T04:49:42.048414+0000","last_undegraded":"2024-09-17T04:49:42.048414+0000","last_fullsized":"2024-09-17T04:49:42.048414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671218+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T04:49:42.671218+0000","last_peered":"2024-09-17T04:49:42.671218+0000","last_clean":"2024-09-17T04:49:42.671218+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T04:49:42.671218+0000","last_undegraded":"2024-09-17T04:49:42.671218+0000","last_fullsized":"2024-09-17T04:49:42.671218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053498+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T04:49:42.053498+0000","last_peered":"2024-09-17T04:49:42.053498+0000","last_clean":"2024-09-17T04:49:42.053498+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T04:49:42.053498+0000","last_undegraded":"2024-09-17T04:49:42.053498+0000","last_fullsized":"2024-09-17T04:49:42.053498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670119+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T04:49:42.670119+0000","last_peered":"2024-09-17T04:49:42.670119+0000","last_clean":"2024-09-17T04:49:42.670119+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T04:49:42.670119+0000","last_undegraded":"2024-09-17T04:49:42.670119+0000","last_fullsized":"2024-09-17T04:49:42.670119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668585+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T04:49:42.668585+0000","last_peered":"2024-09-17T04:49:42.668585+0000","last_clean":"2024-09-17T04:49:42.668585+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T04:49:42.668585+0000","last_undegraded":"2024-09-17T04:49:42.668585+0000","last_fullsized":"2024-09-17T04:49:42.668585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670988+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T04:49:42.670988+0000","last_peered":"2024-09-17T04:49:42.670988+0000","last_clean":"2024-09-17T04:49:42.670988+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T04:49:42.670988+0000","last_undegraded":"2024-09-17T04:49:42.670988+0000","last_fullsized":"2024-09-17T04:49:42.670988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671416+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.671416+0000","last_peered":"2024-09-17T04:49:42.671416+0000","last_clean":"2024-09-17T04:49:42.671416+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T04:49:42.671416+0000","last_undegraded":"2024-09-17T04:49:42.671416+0000","last_fullsized":"2024-09-17T04:49:42.671416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675766+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T04:49:42.675766+0000","last_peered":"2024-09-17T04:49:42.675766+0000","last_clean":"2024-09-17T04:49:42.675766+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T04:49:42.675766+0000","last_undegraded":"2024-09-17T04:49:42.675766+0000","last_fullsized":"2024-09-17T04:49:42.675766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671330+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T04:49:42.671330+0000","last_peered":"2024-09-17T04:49:42.671330+0000","last_clean":"2024-09-17T04:49:42.671330+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T04:49:42.671330+0000","last_undegraded":"2024-09-17T04:49:42.671330+0000","last_fullsized":"2024-09-17T04:49:42.671330+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667310+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T04:49:42.667310+0000","last_peered":"2024-09-17T04:49:42.667310+0000","last_clean":"2024-09-17T04:49:42.667310+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T04:49:42.667310+0000","last_undegraded":"2024-09-17T04:49:42.667310+0000","last_fullsized":"2024-09-17T04:49:42.667310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537133+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T04:49:42.537133+0000","last_peered":"2024-09-17T04:49:42.537133+0000","last_clean":"2024-09-17T04:49:42.537133+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T04:49:42.537133+0000","last_undegraded":"2024-09-17T04:49:42.537133+0000","last_fullsized":"2024-09-17T04:49:42.537133+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669347+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T04:49:42.669347+0000","last_peered":"2024-09-17T04:49:42.669347+0000","last_clean":"2024-09-17T04:49:42.669347+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T04:49:42.669347+0000","last_undegraded":"2024-09-17T04:49:42.669347+0000","last_fullsized":"2024-09-17T04:49:42.669347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050809+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T04:49:42.050809+0000","last_peered":"2024-09-17T04:49:42.050809+0000","last_clean":"2024-09-17T04:49:42.050809+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T04:49:42.050809+0000","last_undegraded":"2024-09-17T04:49:42.050809+0000","last_fullsized":"2024-09-17T04:49:42.050809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535865+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T04:49:42.535865+0000","last_peered":"2024-09-17T04:49:42.535865+0000","last_clean":"2024-09-17T04:49:42.535865+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T04:49:42.535865+0000","last_undegraded":"2024-09-17T04:49:42.535865+0000","last_fullsized":"2024-09-17T04:49:42.535865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673274+0000","last_change":"2024-09-17T04:49:41.041201+0000","last_active":"2024-09-17T04:49:42.673274+0000","last_peered":"2024-09-17T04:49:42.673274+0000","last_clean":"2024-09-17T04:49:42.673274+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T04:49:42.673274+0000","last_undegraded":"2024-09-17T04:49:42.673274+0000","last_fullsized":"2024-09-17T04:49:42.673274+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:02.588786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670272+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T04:49:42.670272+0000","last_peered":"2024-09-17T04:49:42.670272+0000","last_clean":"2024-09-17T04:49:42.670272+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T04:49:42.670272+0000","last_undegraded":"2024-09-17T04:49:42.670272+0000","last_fullsized":"2024-09-17T04:49:42.670272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667767+0000","last_change":"2024-09-17T04:49:41.061092+0000","last_active":"2024-09-17T04:49:42.667767+0000","last_peered":"2024-09-17T04:49:42.667767+0000","last_clean":"2024-09-17T04:49:42.667767+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T04:49:42.667767+0000","last_undegraded":"2024-09-17T04:49:42.667767+0000","last_fullsized":"2024-09-17T04:49:42.667767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:52.714614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536820+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T04:49:42.536820+0000","last_peered":"2024-09-17T04:49:42.536820+0000","last_clean":"2024-09-17T04:49:42.536820+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T04:49:42.536820+0000","last_undegraded":"2024-09-17T04:49:42.536820+0000","last_fullsized":"2024-09-17T04:49:42.536820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669397+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T04:49:42.669397+0000","last_peered":"2024-09-17T04:49:42.669397+0000","last_clean":"2024-09-17T04:49:42.669397+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T04:49:42.669397+0000","last_undegraded":"2024-09-17T04:49:42.669397+0000","last_fullsized":"2024-09-17T04:49:42.669397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671170+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T04:49:42.671170+0000","last_peered":"2024-09-17T04:49:42.671170+0000","last_clean":"2024-09-17T04:49:42.671170+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T04:49:42.671170+0000","last_undegraded":"2024-09-17T04:49:42.671170+0000","last_fullsized":"2024-09-17T04:49:42.671170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536018+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T04:49:42.536018+0000","last_peered":"2024-09-17T04:49:42.536018+0000","last_clean":"2024-09-17T04:49:42.536018+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T04:49:42.536018+0000","last_undegraded":"2024-09-17T04:49:42.536018+0000","last_fullsized":"2024-09-17T04:49:42.536018+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671602+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T04:49:42.671602+0000","last_peered":"2024-09-17T04:49:42.671602+0000","last_clean":"2024-09-17T04:49:42.671602+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T04:49:42.671602+0000","last_undegraded":"2024-09-17T04:49:42.671602+0000","last_fullsized":"2024-09-17T04:49:42.671602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534962+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T04:49:42.534962+0000","last_peered":"2024-09-17T04:49:42.534962+0000","last_clean":"2024-09-17T04:49:42.534962+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T04:49:42.534962+0000","last_undegraded":"2024-09-17T04:49:42.534962+0000","last_fullsized":"2024-09-17T04:49:42.534962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671825+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T04:49:42.671825+0000","last_peered":"2024-09-17T04:49:42.671825+0000","last_clean":"2024-09-17T04:49:42.671825+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T04:49:42.671825+0000","last_undegraded":"2024-09-17T04:49:42.671825+0000","last_fullsized":"2024-09-17T04:49:42.671825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509449,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509449,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":307200,"data_stored":108537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509449,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509449,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":286720,"data_stored":106265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509450,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509450,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542153,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:49:51.277 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:49:51.278 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-17T04:49:51.279 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-17T04:49:51.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.468+0000 7f61213f4640 1 -- 172.21.15.150:0/3905356935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6104003a60 msgr2=0x7f6104003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.468+0000 7f61213f4640 1 --2- 172.21.15.150:0/3905356935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6104003a60 0x7f6104003ed0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f610c009eb0 tx=0x7f610c01d5b0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.468+0000 7f61213f4640 1 -- 172.21.15.150:0/3905356935 shutdown_connections 2024-09-17T04:49:51.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.468+0000 7f61213f4640 1 --2- 172.21.15.150:0/3905356935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6104003a60 0x7f6104003ed0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.468+0000 7f61213f4640 1 -- 172.21.15.150:0/3905356935 >> 172.21.15.150:0/3905356935 conn(0x7f611c06f760 msgr2=0x7f611c074580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:51.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.468+0000 7f61213f4640 1 -- 172.21.15.150:0/3905356935 shutdown_connections 2024-09-17T04:49:51.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.468+0000 7f61213f4640 1 -- 172.21.15.150:0/3905356935 wait complete. 2024-09-17T04:49:51.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61213f4640 1 Processor -- start 2024-09-17T04:49:51.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61213f4640 1 -- start start 2024-09-17T04:49:51.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61213f4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6104003a60 0x7f611c1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:51.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61213f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f611c153c30 0x7f611c0cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:51.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61213f4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f611c0cc590 0x7f611c14db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61213f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f611c0c2200 con 0x7f611c153c30 2024-09-17T04:49:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61213f4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f611c0c2080 con 0x7f611c0cc590 2024-09-17T04:49:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61213f4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f611c0c2380 con 0x7f6104003a60 2024-09-17T04:49:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f611a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f611c153c30 0x7f611c0cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f611a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f611c153c30 0x7f611c0cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33020/0 (socket says 172.21.15.150:33020) 2024-09-17T04:49:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f611a7fc640 1 -- 172.21.15.150:0/3519802715 learned_addr learned my addr 172.21.15.150:0/3519802715 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f611b7fe640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f611c0cc590 0x7f611c14db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f611affd640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6104003a60 0x7f611c1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f611a7fc640 1 -- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6104003a60 msgr2=0x7f611c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f611a7fc640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6104003a60 0x7f611c1536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f611a7fc640 1 -- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f611c0cc590 msgr2=0x7f611c14db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f611a7fc640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f611c0cc590 0x7f611c14db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f611a7fc640 1 -- 172.21.15.150:0/3519802715 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f611c0ab9d0 con 0x7f611c153c30 2024-09-17T04:49:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f611affd640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6104003a60 0x7f611c1536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f611a7fc640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f611c153c30 0x7f611c0cc050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6110004ee0 tx=0x7f6110001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61037fe640 1 -- 172.21.15.150:0/3519802715 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6110015020 con 0x7f611c153c30 2024-09-17T04:49:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61037fe640 1 -- 172.21.15.150:0/3519802715 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6110011960 con 0x7f611c153c30 2024-09-17T04:49:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61037fe640 1 -- 172.21.15.150:0/3519802715 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6110011d90 con 0x7f611c153c30 2024-09-17T04:49:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61213f4640 1 -- 172.21.15.150:0/3519802715 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f611c0abc40 con 0x7f611c153c30 2024-09-17T04:49:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.472+0000 7f61213f4640 1 -- 172.21.15.150:0/3519802715 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f611c0cc9a0 con 0x7f611c153c30 2024-09-17T04:49:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.476+0000 7f61213f4640 1 -- 172.21.15.150:0/3519802715 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f611c1f0b70 con 0x7f611c153c30 2024-09-17T04:49:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.476+0000 7f61037fe640 1 -- 172.21.15.150:0/3519802715 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6110018020 con 0x7f611c153c30 2024-09-17T04:49:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.476+0000 7f61037fe640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6108073200 0x7f6108075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.476+0000 7f61037fe640 1 -- 172.21.15.150:0/3519802715 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f6110095500 con 0x7f611c153c30 2024-09-17T04:49:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.476+0000 7f611affd640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6108073200 0x7f6108075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.476+0000 7f611affd640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6108073200 0x7f6108075710 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f610c0b0000 tx=0x7f610c01a350 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:51.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.480+0000 7f61037fe640 1 -- 172.21.15.150:0/3519802715 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f611001f020 con 0x7f611c153c30 2024-09-17T04:49:51.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.648+0000 7f61213f4640 1 -- 172.21.15.150:0/3519802715 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f611c0c27f0 con 0x7f611c153c30 2024-09-17T04:49:51.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.648+0000 7f61037fe640 1 -- 172.21.15.150:0/3519802715 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v35) ==== 86+0+160 (secure 0 0 0) 0x7f6110060360 con 0x7f611c153c30 2024-09-17T04:49:51.651 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:51.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.652+0000 7f61213f4640 1 -- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6108073200 msgr2=0x7f6108075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.652+0000 7f61213f4640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6108073200 0x7f6108075710 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f610c0b0000 tx=0x7f610c01a350 comp rx=0 tx=0).stop 2024-09-17T04:49:51.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.652+0000 7f61213f4640 1 -- 172.21.15.150:0/3519802715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f611c153c30 msgr2=0x7f611c0cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.652+0000 7f61213f4640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f611c153c30 0x7f611c0cc050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6110004ee0 tx=0x7f6110001640 comp rx=0 tx=0).stop 2024-09-17T04:49:51.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.652+0000 7f61213f4640 1 -- 172.21.15.150:0/3519802715 shutdown_connections 2024-09-17T04:49:51.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.652+0000 7f61213f4640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6108073200 0x7f6108075710 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.652+0000 7f61213f4640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f611c0cc590 0x7f611c14db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.652+0000 7f61213f4640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f611c153c30 0x7f611c0cc050 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.652+0000 7f61213f4640 1 --2- 172.21.15.150:0/3519802715 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6104003a60 0x7f611c1536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.652+0000 7f61213f4640 1 -- 172.21.15.150:0/3519802715 >> 172.21.15.150:0/3519802715 conn(0x7f611c06f760 msgr2=0x7f611c156e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:51.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.652+0000 7f61213f4640 1 -- 172.21.15.150:0/3519802715 shutdown_connections 2024-09-17T04:49:51.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.652+0000 7f61213f4640 1 -- 172.21.15.150:0/3519802715 wait complete. 2024-09-17T04:49:51.672 INFO:teuthology.orchestra.run.smithi150.stdout:[{"version":35,"timestamp":"2024-09-17T04:49:20.028951+0000","name":"","changes":[{"name":"osd.2/osd_mclock_max_capacity_iops_ssd","new_value":"4741.891249"}]}] 2024-09-17T04:49:51.672 INFO:tasks.ceph_manager:config epoch is 35 2024-09-17T04:49:51.672 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T04:49:51.673 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-17T04:49:51.673 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-17T04:49:51.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.828+0000 7f3df811b640 1 -- 172.21.15.150:0/909157046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ddc0053b0 msgr2=0x7f3ddc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.828+0000 7f3df811b640 1 --2- 172.21.15.150:0/909157046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ddc0053b0 0x7f3ddc00ba60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3df00c45d0 tx=0x7f3de801caf0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.828+0000 7f3df811b640 1 -- 172.21.15.150:0/909157046 shutdown_connections 2024-09-17T04:49:51.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.828+0000 7f3df811b640 1 --2- 172.21.15.150:0/909157046 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ddc0053b0 0x7f3ddc00ba60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.828+0000 7f3df811b640 1 -- 172.21.15.150:0/909157046 >> 172.21.15.150:0/909157046 conn(0x7f3df00aab60 msgr2=0x7f3df00aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:51.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.828+0000 7f3df811b640 1 -- 172.21.15.150:0/909157046 shutdown_connections 2024-09-17T04:49:51.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.828+0000 7f3df811b640 1 -- 172.21.15.150:0/909157046 wait complete. 2024-09-17T04:49:51.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df811b640 1 Processor -- start 2024-09-17T04:49:51.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df811b640 1 -- start start 2024-09-17T04:49:51.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df811b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ddc0053b0 0x7f3df00c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:51.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df811b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3df0148ee0 0x7f3df01ecdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:51.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df811b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01ed330 0x7f3df0143da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:51.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df811b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3df00a8680 con 0x7f3df01ed330 2024-09-17T04:49:51.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df811b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3df00a8500 con 0x7f3ddc0053b0 2024-09-17T04:49:51.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df811b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3df00a8800 con 0x7f3df0148ee0 2024-09-17T04:49:51.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df66b8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01ed330 0x7f3df0143da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:51.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df66b8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01ed330 0x7f3df0143da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33036/0 (socket says 172.21.15.150:33036) 2024-09-17T04:49:51.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df66b8640 1 -- 172.21.15.150:0/4012072290 learned_addr learned my addr 172.21.15.150:0/4012072290 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:51.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df5eb7640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ddc0053b0 0x7f3df00c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:51.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df56b6640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3df0148ee0 0x7f3df01ecdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:51.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df66b8640 1 -- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3df0148ee0 msgr2=0x7f3df01ecdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df66b8640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3df0148ee0 0x7f3df01ecdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df66b8640 1 -- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ddc0053b0 msgr2=0x7f3df00c23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:51.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df66b8640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ddc0053b0 0x7f3df00c23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:51.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df66b8640 1 -- 172.21.15.150:0/4012072290 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df0144810 con 0x7f3df01ed330 2024-09-17T04:49:51.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df56b6640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3df0148ee0 0x7f3df01ecdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:51.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df66b8640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01ed330 0x7f3df0143da0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3de801c5b0 tx=0x7f3de8004fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:51.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3de67fc640 1 -- 172.21.15.150:0/4012072290 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de80019f0 con 0x7f3df01ed330 2024-09-17T04:49:51.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3de67fc640 1 -- 172.21.15.150:0/4012072290 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3de80b5070 con 0x7f3df01ed330 2024-09-17T04:49:51.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3de67fc640 1 -- 172.21.15.150:0/4012072290 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8006520 con 0x7f3df01ed330 2024-09-17T04:49:51.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df811b640 1 -- 172.21.15.150:0/4012072290 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df014d990 con 0x7f3df01ed330 2024-09-17T04:49:51.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df811b640 1 -- 172.21.15.150:0/4012072290 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df014a1b0 con 0x7f3df01ed330 2024-09-17T04:49:51.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.832+0000 7f3df811b640 1 -- 172.21.15.150:0/4012072290 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3df006d810 con 0x7f3df01ed330 2024-09-17T04:49:51.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.836+0000 7f3de67fc640 1 -- 172.21.15.150:0/4012072290 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3de80066c0 con 0x7f3df01ed330 2024-09-17T04:49:51.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.836+0000 7f3de67fc640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3dd4073200 0x7f3dd4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:51.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.836+0000 7f3de67fc640 1 -- 172.21.15.150:0/4012072290 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f3de812f5a0 con 0x7f3df01ed330 2024-09-17T04:49:51.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.836+0000 7f3df5eb7640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3dd4073200 0x7f3dd4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:51.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.836+0000 7f3df5eb7640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3dd4073200 0x7f3dd4075710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3dec001250 tx=0x7f3dec001cd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:51.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:51.840+0000 7f3de67fc640 1 -- 172.21.15.150:0/4012072290 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3de8016850 con 0x7f3df01ed330 2024-09-17T04:49:52.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.060+0000 7f3df811b640 1 -- 172.21.15.150:0/4012072290 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f3df006d590 con 0x7f3df01ed330 2024-09-17T04:49:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.064+0000 7f3de67fc640 1 -- 172.21.15.150:0/4012072290 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+186461 (secure 0 0 0) 0x7f3de80fa3a0 con 0x7f3df01ed330 2024-09-17T04:49:52.070 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:52.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.068+0000 7f3df811b640 1 -- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3dd4073200 msgr2=0x7f3dd4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:52.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.068+0000 7f3df811b640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3dd4073200 0x7f3dd4075710 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3dec001250 tx=0x7f3dec001cd0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.068+0000 7f3df811b640 1 -- 172.21.15.150:0/4012072290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01ed330 msgr2=0x7f3df0143da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:52.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.068+0000 7f3df811b640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01ed330 0x7f3df0143da0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3de801c5b0 tx=0x7f3de8004fd0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.068+0000 7f3df811b640 1 -- 172.21.15.150:0/4012072290 shutdown_connections 2024-09-17T04:49:52.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.068+0000 7f3df811b640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3dd4073200 0x7f3dd4075710 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.068+0000 7f3df811b640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01ed330 0x7f3df0143da0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.068+0000 7f3df811b640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3df0148ee0 0x7f3df01ecdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.068+0000 7f3df811b640 1 --2- 172.21.15.150:0/4012072290 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3ddc0053b0 0x7f3df00c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.068+0000 7f3df811b640 1 -- 172.21.15.150:0/4012072290 >> 172.21.15.150:0/4012072290 conn(0x7f3df00aab60 msgr2=0x7f3df014e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:52.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.068+0000 7f3df811b640 1 -- 172.21.15.150:0/4012072290 shutdown_connections 2024-09-17T04:49:52.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.068+0000 7f3df811b640 1 -- 172.21.15.150:0/4012072290 wait complete. 2024-09-17T04:49:52.091 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":5,"flags":0,"active_gid":4117,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":882000330},{"type":"v1","addr":"172.21.15.176:6801","nonce":882000330}]},"active_addr":"172.21.15.176:6801/882000330","active_change":"2024-09-17T04:49:10.926712+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4140,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":3881284714}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":3494545386}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":1925415293}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":3770977356}]}]} 2024-09-17T04:49:52.095 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-17T04:49:52.095 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-17T04:49:52.095 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:52.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.244+0000 7f01fe640640 1 -- 172.21.15.150:0/2016281142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f8074450 msgr2=0x7f01f8074860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:52.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.244+0000 7f01fe640640 1 --2- 172.21.15.150:0/2016281142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f8074450 0x7f01f8074860 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f01f8072370 tx=0x7f01f001ccf0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.244+0000 7f01fe640640 1 -- 172.21.15.150:0/2016281142 shutdown_connections 2024-09-17T04:49:52.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.244+0000 7f01fe640640 1 --2- 172.21.15.150:0/2016281142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f8074450 0x7f01f8074860 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.244+0000 7f01fe640640 1 --2- 172.21.15.150:0/2016281142 >> v2:172.21.15.176:3300/0 conn(0x7f01f8073440 0x7f01f80a7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.244+0000 7f01fe640640 1 -- 172.21.15.150:0/2016281142 >> 172.21.15.150:0/2016281142 conn(0x7f01f80c4a00 msgr2=0x7f01f80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:52.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.244+0000 7f01fe640640 1 -- 172.21.15.150:0/2016281142 shutdown_connections 2024-09-17T04:49:52.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.244+0000 7f01fe640640 1 -- 172.21.15.150:0/2016281142 wait complete. 2024-09-17T04:49:52.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.244+0000 7f01fe640640 1 Processor -- start 2024-09-17T04:49:52.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.244+0000 7f01fe640640 1 -- start start 2024-09-17T04:49:52.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fe640640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01f8070e50 0x7f01f8147a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:52.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fe640640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01f8073440 0x7f01f81449f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:52.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fe640640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f8074450 0x7f01f81f0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:52.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fe640640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f01f80ade60 con 0x7f01f8074450 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fe640640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f01f80adce0 con 0x7f01f8070e50 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fe640640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f01f80ad610 con 0x7f01f8073440 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fcbdd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f8074450 0x7f01f81f0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fcbdd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f8074450 0x7f01f81f0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33056/0 (socket says 172.21.15.150:33056) 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fcbdd640 1 -- 172.21.15.150:0/1561851502 learned_addr learned my addr 172.21.15.150:0/1561851502 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01f77fe640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01f8073440 0x7f01f81449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01f7fff640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01f8070e50 0x7f01f8147a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fcbdd640 1 -- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01f8073440 msgr2=0x7f01f81449f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fcbdd640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01f8073440 0x7f01f81449f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fcbdd640 1 -- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01f8070e50 msgr2=0x7f01f8147a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fcbdd640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01f8070e50 0x7f01f8147a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fcbdd640 1 -- 172.21.15.150:0/1561851502 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f8148350 con 0x7f01f8074450 2024-09-17T04:49:52.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01f77fe640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01f8073440 0x7f01f81449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:52.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01f7fff640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01f8070e50 0x7f01f8147a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:52.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fcbdd640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f8074450 0x7f01f81f0250 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f01f0009bd0 tx=0x7f01f00a5980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:52.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01f4ff9640 1 -- 172.21.15.150:0/1561851502 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f00bb070 con 0x7f01f8074450 2024-09-17T04:49:52.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01f4ff9640 1 -- 172.21.15.150:0/1561851502 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01f0004e00 con 0x7f01f8074450 2024-09-17T04:49:52.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fe640640 1 -- 172.21.15.150:0/1561851502 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f8148630 con 0x7f01f8074450 2024-09-17T04:49:52.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01f4ff9640 1 -- 172.21.15.150:0/1561851502 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f00b2040 con 0x7f01f8074450 2024-09-17T04:49:52.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fe640640 1 -- 172.21.15.150:0/1561851502 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f8154990 con 0x7f01f8074450 2024-09-17T04:49:52.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01fe640640 1 -- 172.21.15.150:0/1561851502 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01f80ad9f0 con 0x7f01f8074450 2024-09-17T04:49:52.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01f4ff9640 1 -- 172.21.15.150:0/1561851502 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f01f00be070 con 0x7f01f8074450 2024-09-17T04:49:52.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01f4ff9640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01e4073230 0x7f01e4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:52.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01f4ff9640 1 -- 172.21.15.150:0/1561851502 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f01f012e660 con 0x7f01f8074450 2024-09-17T04:49:52.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.248+0000 7f01f7fff640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01e4073230 0x7f01e4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:52.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.252+0000 7f01f7fff640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01e4073230 0x7f01e4075740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f01f81513e0 tx=0x7f01ec008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:52.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.252+0000 7f01f4ff9640 1 -- 172.21.15.150:0/1561851502 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f01f00a5e20 con 0x7f01f8074450 2024-09-17T04:49:52.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.416+0000 7f01fe640640 1 -- 172.21.15.150:0/1561851502 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01f8146010 con 0x7f01f8074450 2024-09-17T04:49:52.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.416+0000 7f01f4ff9640 1 -- 172.21.15.150:0/1561851502 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20431 (secure 0 0 0) 0x7f01f0016970 con 0x7f01f8074450 2024-09-17T04:49:52.420 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:52.420 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:42.029799+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:52.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.420+0000 7f01fe640640 1 -- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01e4073230 msgr2=0x7f01e4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:52.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.420+0000 7f01fe640640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01e4073230 0x7f01e4075740 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f01f81513e0 tx=0x7f01ec008040 comp rx=0 tx=0).stop 2024-09-17T04:49:52.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.420+0000 7f01fe640640 1 -- 172.21.15.150:0/1561851502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f8074450 msgr2=0x7f01f81f0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:52.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.420+0000 7f01fe640640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f8074450 0x7f01f81f0250 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f01f0009bd0 tx=0x7f01f00a5980 comp rx=0 tx=0).stop 2024-09-17T04:49:52.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.420+0000 7f01fe640640 1 -- 172.21.15.150:0/1561851502 shutdown_connections 2024-09-17T04:49:52.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.420+0000 7f01fe640640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01e4073230 0x7f01e4075740 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.420+0000 7f01fe640640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01f8074450 0x7f01f81f0250 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.420+0000 7f01fe640640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01f8073440 0x7f01f81449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.420+0000 7f01fe640640 1 --2- 172.21.15.150:0/1561851502 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01f8070e50 0x7f01f8147a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.420+0000 7f01fe640640 1 -- 172.21.15.150:0/1561851502 >> 172.21.15.150:0/1561851502 conn(0x7f01f80c4a00 msgr2=0x7f01f815e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:52.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.420+0000 7f01fe640640 1 -- 172.21.15.150:0/1561851502 shutdown_connections 2024-09-17T04:49:52.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.420+0000 7f01fe640640 1 -- 172.21.15.150:0/1561851502 wait complete. 2024-09-17T04:49:52.441 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-17T04:49:52.441 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:52.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.576+0000 7fc753f64640 1 -- 172.21.15.150:0/1156294293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc74c0ad7f0 msgr2=0x7fc74c0c6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:52.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.576+0000 7fc753f64640 1 --2- 172.21.15.150:0/1156294293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc74c0ad7f0 0x7fc74c0c6230 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc748009c70 tx=0x7fc74801cc00 comp rx=0 tx=0).stop 2024-09-17T04:49:52.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.576+0000 7fc753f64640 1 -- 172.21.15.150:0/1156294293 shutdown_connections 2024-09-17T04:49:52.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.576+0000 7fc753f64640 1 --2- 172.21.15.150:0/1156294293 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc74c151e10 0x7fc74c0c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.576+0000 7fc753f64640 1 --2- 172.21.15.150:0/1156294293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc74c0ad7f0 0x7fc74c0c6230 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.576+0000 7fc753f64640 1 -- 172.21.15.150:0/1156294293 >> 172.21.15.150:0/1156294293 conn(0x7fc74c0c9de0 msgr2=0x7fc74c0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:52.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.576+0000 7fc753f64640 1 -- 172.21.15.150:0/1156294293 shutdown_connections 2024-09-17T04:49:52.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.576+0000 7fc753f64640 1 -- 172.21.15.150:0/1156294293 wait complete. 2024-09-17T04:49:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc753f64640 1 Processor -- start 2024-09-17T04:49:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc753f64640 1 -- start start 2024-09-17T04:49:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc753f64640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc74c0ad7f0 0x7fc74c1f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc753f64640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c151e10 0x7fc74c1441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc753f64640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc74c157ca0 0x7fc74c072b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:52.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc753f64640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c0c59f0 con 0x7fc74c151e10 2024-09-17T04:49:52.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc753f64640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc74c0c5870 con 0x7fc74c157ca0 2024-09-17T04:49:52.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc753f64640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc74c0c7ce0 con 0x7fc74c0ad7f0 2024-09-17T04:49:52.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc7514ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c151e10 0x7fc74c1441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:52.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc7514ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c151e10 0x7fc74c1441d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33062/0 (socket says 172.21.15.150:33062) 2024-09-17T04:49:52.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc7514ff640 1 -- 172.21.15.150:0/3179270787 learned_addr learned my addr 172.21.15.150:0/3179270787 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:52.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc751d00640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc74c0ad7f0 0x7fc74c1f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:52.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc752501640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc74c157ca0 0x7fc74c072b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc7514ff640 1 -- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc74c0ad7f0 msgr2=0x7fc74c1f0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc7514ff640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc74c0ad7f0 0x7fc74c1f0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc7514ff640 1 -- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc74c157ca0 msgr2=0x7fc74c072b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc7514ff640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc74c157ca0 0x7fc74c072b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc7514ff640 1 -- 172.21.15.150:0/3179270787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74c0c9850 con 0x7fc74c151e10 2024-09-17T04:49:52.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc7514ff640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c151e10 0x7fc74c1441d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00e7a0 tx=0x7fc73c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:52.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc7427fc640 1 -- 172.21.15.150:0/3179270787 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c00ccc0 con 0x7fc74c151e10 2024-09-17T04:49:52.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc7427fc640 1 -- 172.21.15.150:0/3179270787 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc73c00ce60 con 0x7fc74c151e10 2024-09-17T04:49:52.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc753f64640 1 -- 172.21.15.150:0/3179270787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc74c1504a0 con 0x7fc74c151e10 2024-09-17T04:49:52.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc7427fc640 1 -- 172.21.15.150:0/3179270787 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc73c005260 con 0x7fc74c151e10 2024-09-17T04:49:52.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.580+0000 7fc753f64640 1 -- 172.21.15.150:0/3179270787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc74c150a20 con 0x7fc74c151e10 2024-09-17T04:49:52.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.584+0000 7fc753f64640 1 -- 172.21.15.150:0/3179270787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc74c0c6230 con 0x7fc74c151e10 2024-09-17T04:49:52.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.584+0000 7fc7427fc640 1 -- 172.21.15.150:0/3179270787 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc73c005400 con 0x7fc74c151e10 2024-09-17T04:49:52.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.584+0000 7fc7427fc640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc730073200 0x7fc730075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:52.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.584+0000 7fc7427fc640 1 -- 172.21.15.150:0/3179270787 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fc73c0951d0 con 0x7fc74c151e10 2024-09-17T04:49:52.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.584+0000 7fc751d00640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc730073200 0x7fc730075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:52.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.584+0000 7fc751d00640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc730073200 0x7fc730075710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc7480af000 tx=0x7fc74801b440 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:52.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.588+0000 7fc7427fc640 1 -- 172.21.15.150:0/3179270787 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc73c0600c0 con 0x7fc74c151e10 2024-09-17T04:49:52.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.744+0000 7fc753f64640 1 -- 172.21.15.150:0/3179270787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc74c0c54c0 con 0x7fc74c151e10 2024-09-17T04:49:52.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc7427fc640 1 -- 172.21.15.150:0/3179270787 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20431 (secure 0 0 0) 0x7fc73c05f9c0 con 0x7fc74c151e10 2024-09-17T04:49:52.750 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:52.750 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:42.029799+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:52.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc753f64640 1 -- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc730073200 msgr2=0x7fc730075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:52.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc753f64640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc730073200 0x7fc730075710 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc7480af000 tx=0x7fc74801b440 comp rx=0 tx=0).stop 2024-09-17T04:49:52.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc753f64640 1 -- 172.21.15.150:0/3179270787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c151e10 msgr2=0x7fc74c1441d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:52.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc753f64640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c151e10 0x7fc74c1441d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00e7a0 tx=0x7fc73c00ec60 comp rx=0 tx=0).stop 2024-09-17T04:49:52.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc753f64640 1 -- 172.21.15.150:0/3179270787 shutdown_connections 2024-09-17T04:49:52.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc753f64640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc730073200 0x7fc730075710 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc753f64640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc74c157ca0 0x7fc74c072b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc753f64640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc74c151e10 0x7fc74c1441d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc753f64640 1 --2- 172.21.15.150:0/3179270787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc74c0ad7f0 0x7fc74c1f0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:52.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc753f64640 1 -- 172.21.15.150:0/3179270787 >> 172.21.15.150:0/3179270787 conn(0x7fc74c0c9de0 msgr2=0x7fc74c0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:52.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc753f64640 1 -- 172.21.15.150:0/3179270787 shutdown_connections 2024-09-17T04:49:52.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:52.748+0000 7fc753f64640 1 -- 172.21.15.150:0/3179270787 wait complete. 2024-09-17T04:49:52.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-17T04:49:52.773 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-17T04:49:52.773 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-17T04:49:52.774 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-17T04:49:52.774 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-17T04:49:52.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-17T04:49:52.776 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-17T04:49:52.776 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-17T04:49:53.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7fbabd405640 1 -- 172.21.15.150:0/2810508120 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab40a6ce0 con 0x7fbab80aa320 2024-09-17T04:49:53.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7fbac066b640 1 -- 172.21.15.150:0/2810508120 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbab80aa320 msgr2=0x7fbab80a9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7fbac066b640 1 --2- 172.21.15.150:0/2810508120 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbab80aa320 0x7fbab80a9820 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbab400aa10 tx=0x7fbab401d9c0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7fd98affd640 1 -- 172.21.15.150:0/1502793369 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984001a80 con 0x7fd98c0c3fd0 2024-09-17T04:49:53.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7f40cabce640 1 -- 172.21.15.150:0/1391270291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c40a82a0 msgr2=0x7f40c41508f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7f40cabce640 1 --2- 172.21.15.150:0/1391270291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c40a82a0 0x7f40c41508f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f40c40731c0 tx=0x7f40b401e650 comp rx=0 tx=0).stop 2024-09-17T04:49:53.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7f40cabce640 1 -- 172.21.15.150:0/1391270291 shutdown_connections 2024-09-17T04:49:53.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7f40cabce640 1 --2- 172.21.15.150:0/1391270291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c40a82a0 0x7f40c41508f0 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f40c40731c0 tx=0x7f40b401e650 comp rx=0 tx=0).stop 2024-09-17T04:49:53.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7f40cabce640 1 --2- 172.21.15.150:0/1391270291 >> v2:172.21.15.150:3300/0 conn(0x7f40c40a9be0 0x7f40c40a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7f40cabce640 1 -- 172.21.15.150:0/1391270291 >> 172.21.15.150:0/1391270291 conn(0x7f40c40c7cd0 msgr2=0x7f40c40c80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7f40cabce640 1 -- 172.21.15.150:0/1391270291 shutdown_connections 2024-09-17T04:49:53.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7fbac066b640 1 -- 172.21.15.150:0/2810508120 shutdown_connections 2024-09-17T04:49:53.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7fbac066b640 1 --2- 172.21.15.150:0/2810508120 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbab80aa320 0x7fbab80a9820 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7fbac066b640 1 --2- 172.21.15.150:0/2810508120 >> v2:172.21.15.150:3300/0 conn(0x7fbab8070aa0 0x7fbab8070eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7fbac066b640 1 -- 172.21.15.150:0/2810508120 >> 172.21.15.150:0/2810508120 conn(0x7fbab80c7cd0 msgr2=0x7fbab80c80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7fd992637640 1 -- 172.21.15.150:0/1502793369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c0c3fd0 msgr2=0x7fd98c0c3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7fbac066b640 1 -- 172.21.15.150:0/2810508120 shutdown_connections 2024-09-17T04:49:53.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7fd992637640 1 --2- 172.21.15.150:0/1502793369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c0c3fd0 0x7fd98c0c3160 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd97c009840 tx=0x7fd97c01c6b0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fbac066b640 1 -- 172.21.15.150:0/2810508120 wait complete. 2024-09-17T04:49:53.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fd992637640 1 -- 172.21.15.150:0/1502793369 shutdown_connections 2024-09-17T04:49:53.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fd992637640 1 --2- 172.21.15.150:0/1502793369 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c0c3fd0 0x7fd98c0c3160 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fd992637640 1 -- 172.21.15.150:0/1502793369 >> 172.21.15.150:0/1502793369 conn(0x7fd98c0c7cd0 msgr2=0x7fd98c0c80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.004+0000 7f40cabce640 1 -- 172.21.15.150:0/1391270291 wait complete. 2024-09-17T04:49:53.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fd992637640 1 -- 172.21.15.150:0/1502793369 shutdown_connections 2024-09-17T04:49:53.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f115cb88640 1 -- 172.21.15.150:0/577604111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11580a82a0 msgr2=0x7f11581508f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f115cb88640 1 --2- 172.21.15.150:0/577604111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11580a82a0 0x7f11581508f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f11580731c0 tx=0x7f115001ca60 comp rx=0 tx=0).stop 2024-09-17T04:49:53.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f115cb88640 1 -- 172.21.15.150:0/577604111 shutdown_connections 2024-09-17T04:49:53.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f115cb88640 1 --2- 172.21.15.150:0/577604111 >> v2:172.21.15.150:3300/0 conn(0x7f11580a9be0 0x7f11580a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f115cb88640 1 --2- 172.21.15.150:0/577604111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11580a82a0 0x7f11581508f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f115cb88640 1 -- 172.21.15.150:0/577604111 >> 172.21.15.150:0/577604111 conn(0x7f11580c7cd0 msgr2=0x7f11580c80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f0d21ffb640 1 -- 172.21.15.150:0/1250908906 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d18003600 con 0x7f0d0c0042d0 2024-09-17T04:49:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f5b6b524640 1 -- 172.21.15.150:0/145670221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b640c3000 msgr2=0x7f5b640c3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f5b6b524640 1 --2- 172.21.15.150:0/145670221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b640c3000 0x7f5b640c3410 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5b540074e0 tx=0x7f5b5401d450 comp rx=0 tx=0).stop 2024-09-17T04:49:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f5b6b524640 1 -- 172.21.15.150:0/145670221 shutdown_connections 2024-09-17T04:49:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f5b6b524640 1 --2- 172.21.15.150:0/145670221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b640c3000 0x7f5b640c3410 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f5b6b524640 1 --2- 172.21.15.150:0/145670221 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b641534f0 0x7f5b640abcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f5b6b524640 1 -- 172.21.15.150:0/145670221 >> 172.21.15.150:0/145670221 conn(0x7f5b6406ce50 msgr2=0x7f5b6406d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f8967ac0640 1 -- 172.21.15.150:0/1982689442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f895802e8e0 msgr2=0x7f89580d4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f8967ac0640 1 --2- 172.21.15.150:0/1982689442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f895802e8e0 0x7f89580d4330 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8950009840 tx=0x7f895001c820 comp rx=0 tx=0).stop 2024-09-17T04:49:53.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f0d293f7640 1 -- 172.21.15.150:0/1250908906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d0c0042d0 msgr2=0x7f0d0c00ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f0d293f7640 1 --2- 172.21.15.150:0/1250908906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d0c0042d0 0x7f0d0c00ba70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0d18001890 tx=0x7f0d1801c900 comp rx=0 tx=0).stop 2024-09-17T04:49:53.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f5b6b524640 1 -- 172.21.15.150:0/145670221 shutdown_connections 2024-09-17T04:49:53.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f8967ac0640 1 -- 172.21.15.150:0/1982689442 shutdown_connections 2024-09-17T04:49:53.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f8967ac0640 1 --2- 172.21.15.150:0/1982689442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f895802e8e0 0x7f89580d4330 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f8967ac0640 1 --2- 172.21.15.150:0/1982689442 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89580478a0 0x7f89580d79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f8967ac0640 1 -- 172.21.15.150:0/1982689442 >> 172.21.15.150:0/1982689442 conn(0x7f8958049590 msgr2=0x7f895802e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f0d293f7640 1 -- 172.21.15.150:0/1250908906 shutdown_connections 2024-09-17T04:49:53.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f0d293f7640 1 --2- 172.21.15.150:0/1250908906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d0c0042d0 0x7f0d0c00ba70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f0d293f7640 1 -- 172.21.15.150:0/1250908906 >> 172.21.15.150:0/1250908906 conn(0x7f0d24070ba0 msgr2=0x7f0d24070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fd992637640 1 -- 172.21.15.150:0/1502793369 wait complete. 2024-09-17T04:49:53.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f8967ac0640 1 -- 172.21.15.150:0/1982689442 shutdown_connections 2024-09-17T04:49:53.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f0d293f7640 1 -- 172.21.15.150:0/1250908906 shutdown_connections 2024-09-17T04:49:53.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f115cb88640 1 -- 172.21.15.150:0/577604111 shutdown_connections 2024-09-17T04:49:53.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fae6f320640 1 -- 172.21.15.150:0/3156498523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae680a82a0 msgr2=0x7fae681508f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fae6f320640 1 --2- 172.21.15.150:0/3156498523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae680a82a0 0x7fae681508f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fae60009c80 tx=0x7fae6001cc80 comp rx=0 tx=0).stop 2024-09-17T04:49:53.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f0d293f7640 1 -- 172.21.15.150:0/1250908906 wait complete. 2024-09-17T04:49:53.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f8967ac0640 1 -- 172.21.15.150:0/1982689442 wait complete. 2024-09-17T04:49:53.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f115cb88640 1 -- 172.21.15.150:0/577604111 wait complete. 2024-09-17T04:49:53.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f40cabce640 1 Processor -- start 2024-09-17T04:49:53.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f5b6b524640 1 -- 172.21.15.150:0/145670221 wait complete. 2024-09-17T04:49:53.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fae6f320640 1 -- 172.21.15.150:0/3156498523 shutdown_connections 2024-09-17T04:49:53.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fae6f320640 1 --2- 172.21.15.150:0/3156498523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae680a82a0 0x7fae681508f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fae6f320640 1 -- 172.21.15.150:0/3156498523 >> 172.21.15.150:0/3156498523 conn(0x7fae680c7cd0 msgr2=0x7fae680c80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fd992637640 1 Processor -- start 2024-09-17T04:49:53.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fbac066b640 1 Processor -- start 2024-09-17T04:49:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fae6f320640 1 -- 172.21.15.150:0/3156498523 shutdown_connections 2024-09-17T04:49:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fae6f320640 1 -- 172.21.15.150:0/3156498523 wait complete. 2024-09-17T04:49:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fbac066b640 1 -- start start 2024-09-17T04:49:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7fd992637640 1 -- start start 2024-09-17T04:49:53.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b6b524640 1 Processor -- start 2024-09-17T04:49:53.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f8967ac0640 1 Processor -- start 2024-09-17T04:49:53.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b6b524640 1 -- start start 2024-09-17T04:49:53.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbac066b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbab8070aa0 0x7fbab80aab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbac066b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbab80aa320 0x7fbab8152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbac066b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbab8156200 0x7fbab8149ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbac066b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbab80c40e0 con 0x7fbab8070aa0 2024-09-17T04:49:53.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbac066b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbab80a9380 con 0x7fbab8156200 2024-09-17T04:49:53.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbac066b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbab80c4260 con 0x7fbab80aa320 2024-09-17T04:49:53.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fae6f320640 1 Processor -- start 2024-09-17T04:49:53.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fae6f320640 1 -- start start 2024-09-17T04:49:53.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f115cb88640 1 Processor -- start 2024-09-17T04:49:53.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd992637640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c0703e0 0x7fd98c151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd992637640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd98c0707f0 0x7fd98c14fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd992637640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd98c0c3fd0 0x7fd98c14eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd992637640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c0c21b0 con 0x7fd98c0703e0 2024-09-17T04:49:53.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd992637640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c0c2030 con 0x7fd98c0707f0 2024-09-17T04:49:53.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd992637640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd98c160890 con 0x7fd98c0c3fd0 2024-09-17T04:49:53.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.008+0000 7f40cabce640 1 -- start start 2024-09-17T04:49:53.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40cabce640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40c4075040 0x7f40c41f7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40cabce640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c40a9be0 0x7f40c40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40cabce640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40c40a5cd0 0x7f40c40c03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40cabce640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f40c40c3ea0 con 0x7f40c40a5cd0 2024-09-17T04:49:53.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40cabce640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f40c40c3d20 con 0x7f40c40a9be0 2024-09-17T04:49:53.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40cabce640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f40c40aa530 con 0x7f40c4075040 2024-09-17T04:49:53.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f8967ac0640 1 -- start start 2024-09-17T04:49:53.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d293f7640 1 Processor -- start 2024-09-17T04:49:53.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd98b7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd98c0707f0 0x7fd98c14fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd98b7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd98c0707f0 0x7fd98c14fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:44768/0 (socket says 172.21.15.150:44768) 2024-09-17T04:49:53.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd98b7fe640 1 -- 172.21.15.150:0/1044895126 learned_addr learned my addr 172.21.15.150:0/1044895126 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbabdc06640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbab80aa320 0x7fbab8152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f115cb88640 1 -- start start 2024-09-17T04:49:53.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d293f7640 1 -- start start 2024-09-17T04:49:53.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b6b524640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640c3000 0x7f5b640a6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40c916b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40c40a5cd0 0x7f40c40c03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40c916b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40c40a5cd0 0x7f40c40c03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33164/0 (socket says 172.21.15.150:33164) 2024-09-17T04:49:53.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40c916b640 1 -- 172.21.15.150:0/386061938 learned_addr learned my addr 172.21.15.150:0/386061938 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbabec08640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbab8156200 0x7fbab8149ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbabdc06640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbab80aa320 0x7fbab8152240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:47264/0 (socket says 172.21.15.150:47264) 2024-09-17T04:49:53.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbabdc06640 1 -- 172.21.15.150:0/1050542587 learned_addr learned my addr 172.21.15.150:0/1050542587 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbabe407640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbab8070aa0 0x7fbab80aab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd98bfff640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c0703e0 0x7fd98c151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d293f7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d0c0042d0 0x7f0d24152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40c3fff640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c40a9be0 0x7f40c40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b6b524640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b641534f0 0x7f5b640a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d293f7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d241deee0 0x7f0d241dcae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b6b524640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b64155d30 0x7f5b640a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd990bd4640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd98c0c3fd0 0x7fd98c14eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40c896a640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40c4075040 0x7f40c41f7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d293f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d241dd020 0x7f0d241498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b6b524640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5b64156850 con 0x7f5b64155d30 2024-09-17T04:49:53.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b6b524640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5b640c4ec0 con 0x7f5b641534f0 2024-09-17T04:49:53.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b6b524640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5b6415f880 con 0x7f5b640c3000 2024-09-17T04:49:53.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd98b7fe640 1 -- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd98c0c3fd0 msgr2=0x7fd98c14eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd98b7fe640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd98c0c3fd0 0x7fd98c14eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd98b7fe640 1 -- 172.21.15.150:0/1044895126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c0703e0 msgr2=0x7fd98c151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d293f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2406e4c0 con 0x7f0d241dd020 2024-09-17T04:49:53.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b6a522640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640c3000 0x7f5b640a6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d293f7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0d240c4180 con 0x7f0d241deee0 2024-09-17T04:49:53.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f115cb88640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1158075040 0x7f11581538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b6a522640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640c3000 0x7f5b640a6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:47286/0 (socket says 172.21.15.150:47286) 2024-09-17T04:49:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b6a522640 1 -- 172.21.15.150:0/1635772773 learned_addr learned my addr 172.21.15.150:0/1635772773 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b69d21640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b641534f0 0x7f5b640a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d22ffd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d0c0042d0 0x7f0d24152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d241dd020 0x7f0d241498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d241dd020 0x7f0d241498e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33200/0 (socket says 172.21.15.150:33200) 2024-09-17T04:49:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d237fe640 1 -- 172.21.15.150:0/3003713965 learned_addr learned my addr 172.21.15.150:0/3003713965 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd98b7fe640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c0703e0 0x7fd98c151830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd98b7fe640 1 -- 172.21.15.150:0/1044895126 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c14cda0 con 0x7fd98c0707f0 2024-09-17T04:49:53.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d227fc640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d241deee0 0x7f0d241dcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f0d293f7640 1 -- 172.21.15.150:0/3003713965 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0d24075220 con 0x7f0d0c0042d0 2024-09-17T04:49:53.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f115cb88640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11580a82a0 0x7f1158152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f115cb88640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11580a9be0 0x7f11580a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f115659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1158075040 0x7f11581538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f115659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1158075040 0x7f11581538f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33214/0 (socket says 172.21.15.150:33214) 2024-09-17T04:49:53.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbabe407640 1 -- 172.21.15.150:0/1050542587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbab80aa320 msgr2=0x7fbab8152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbabe407640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbab80aa320 0x7fbab8152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbabe407640 1 -- 172.21.15.150:0/1050542587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbab8156200 msgr2=0x7fbab8149ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbabe407640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbab8156200 0x7fbab8149ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbabe407640 1 -- 172.21.15.150:0/1050542587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab80bd2f0 con 0x7fbab8070aa0 2024-09-17T04:49:53.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fd98bfff640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c0703e0 0x7fd98c151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:53.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fae6f320640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae68075040 0x7fae680c2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fae6f320640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae680a82a0 0x7fae6814f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f5b6ad23640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b64155d30 0x7f5b640a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fbabe407640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbab8070aa0 0x7fbab80aab20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbaa800f6b0 tx=0x7fbaa800fb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fae6f320640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae680a9be0 0x7fae6814fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fae6f320640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fae680c3ea0 con 0x7fae68075040 2024-09-17T04:49:53.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fae6f320640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fae680c3d20 con 0x7fae680a9be0 2024-09-17T04:49:53.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7fae6f320640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fae68161570 con 0x7fae680a82a0 2024-09-17T04:49:53.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f115659c640 1 -- 172.21.15.150:0/727521340 learned_addr learned my addr 172.21.15.150:0/727521340 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f115cb88640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f11580c3d20 con 0x7f1158075040 2024-09-17T04:49:53.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1158073f30 con 0x7f11580a9be0 2024-09-17T04:49:53.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f11580c3ea0 con 0x7f11580a82a0 2024-09-17T04:49:53.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40c916b640 1 -- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40c4075040 msgr2=0x7f40c41f7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40c916b640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f40c4075040 0x7f40c41f7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40c916b640 1 -- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c40a9be0 msgr2=0x7f40c40a5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40c916b640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40c40a9be0 0x7f40c40a5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.012+0000 7f40c916b640 1 -- 172.21.15.150:0/386061938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c40a4ff0 con 0x7f40c40a5cd0 2024-09-17T04:49:53.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fae6d0bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae68075040 0x7fae680c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fae6d0bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae68075040 0x7fae680c2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33220/0 (socket says 172.21.15.150:33220) 2024-09-17T04:49:53.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fae6d0bc640 1 -- 172.21.15.150:0/287468337 learned_addr learned my addr 172.21.15.150:0/287468337 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fd98b7fe640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd98c0707f0 0x7fd98c14fa30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd984001f10 tx=0x7fd984002000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f40c916b640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40c40a5cd0 0x7f40c40c03d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00ef20 tx=0x7f40bc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f8967ac0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f895802e8e0 0x7f89580cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f115659c640 1 -- 172.21.15.150:0/727521340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11580a82a0 msgr2=0x7f1158152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f115659c640 1 --2- 172.21.15.150:0/727521340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11580a82a0 0x7f1158152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f115659c640 1 -- 172.21.15.150:0/727521340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11580a9be0 msgr2=0x7f11580a5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f115659c640 1 --2- 172.21.15.150:0/727521340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11580a9be0 0x7f11580a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f115659c640 1 -- 172.21.15.150:0/727521340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11581f4980 con 0x7f1158075040 2024-09-17T04:49:53.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f8967ac0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f895802ecf0 0x7f8958043fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f8967ac0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89580478a0 0x7f8958045050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fbaaeffd640 1 -- 172.21.15.150:0/1050542587 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8010690 con 0x7fbab8070aa0 2024-09-17T04:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f0d237fe640 1 -- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d0c0042d0 msgr2=0x7f0d24152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f0d237fe640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d0c0042d0 0x7f0d24152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f0d237fe640 1 -- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d241deee0 msgr2=0x7f0d241dcae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f0d237fe640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d241deee0 0x7f0d241dcae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f0d237fe640 1 -- 172.21.15.150:0/3003713965 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d240c2e20 con 0x7f0d241dd020 2024-09-17T04:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f0d22ffd640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d0c0042d0 0x7f0d24152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f5b69d21640 1 -- 172.21.15.150:0/1635772773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640c3000 msgr2=0x7f5b640a6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f5b69d21640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b640c3000 0x7f5b640a6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f5b69d21640 1 -- 172.21.15.150:0/1635772773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b64155d30 msgr2=0x7f5b640a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f5b69d21640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b64155d30 0x7f5b640a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f5b69d21640 1 -- 172.21.15.150:0/1635772773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b641f4060 con 0x7f5b641534f0 2024-09-17T04:49:53.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f8967ac0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8958006bc0 con 0x7f89580478a0 2024-09-17T04:49:53.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f8967ac0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8958006a40 con 0x7f895802ecf0 2024-09-17T04:49:53.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f8967ac0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8958006d40 con 0x7f895802e8e0 2024-09-17T04:49:53.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fbaaeffd640 1 -- 172.21.15.150:0/1050542587 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbaa8010940 con 0x7fbab8070aa0 2024-09-17T04:49:53.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fbaaeffd640 1 -- 172.21.15.150:0/1050542587 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8010d90 con 0x7fbab8070aa0 2024-09-17T04:49:53.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fbac066b640 1 -- 172.21.15.150:0/1050542587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbab8155f50 con 0x7fbab8070aa0 2024-09-17T04:49:53.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fbac066b640 1 -- 172.21.15.150:0/1050542587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbab814b510 con 0x7fbab8070aa0 2024-09-17T04:49:53.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f896605d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89580478a0 0x7f8958045050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f896605d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89580478a0 0x7f8958045050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33232/0 (socket says 172.21.15.150:33232) 2024-09-17T04:49:53.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f896605d640 1 -- 172.21.15.150:0/694586631 learned_addr learned my addr 172.21.15.150:0/694586631 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f0d237fe640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d241dd020 0x7f0d241498e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0d1801b000 tx=0x7f0d18001320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f40c17fa640 1 -- 172.21.15.150:0/386061938 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc003980 con 0x7f40c40a5cd0 2024-09-17T04:49:53.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fae6d8bd640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae680a9be0 0x7fae6814fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fae6d0bc640 1 -- 172.21.15.150:0/287468337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae680a82a0 msgr2=0x7fae6814f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fae6d0bc640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae680a82a0 0x7fae6814f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fae6d0bc640 1 -- 172.21.15.150:0/287468337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae680a9be0 msgr2=0x7fae6814fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fae6d0bc640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae680a9be0 0x7fae6814fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fae6d0bc640 1 -- 172.21.15.150:0/287468337 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae681f4290 con 0x7fae68075040 2024-09-17T04:49:53.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fae6d8bd640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae680a9be0 0x7fae6814fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:53.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f40cabce640 1 -- 172.21.15.150:0/386061938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c40a38a0 con 0x7f40c40a5cd0 2024-09-17T04:49:53.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f5b69d21640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b641534f0 0x7f5b640a3ba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5b540ab000 tx=0x7f5b540056f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f40cabce640 1 -- 172.21.15.150:0/386061938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c41f8230 con 0x7f40c40a5cd0 2024-09-17T04:49:53.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fae6d0bc640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae68075040 0x7fae680c2500 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fae6400c3f0 tx=0x7fae6400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f115659c640 1 --2- 172.21.15.150:0/727521340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1158075040 0x7f11581538f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f114800c400 tx=0x7f114800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fd988ff9640 1 -- 172.21.15.150:0/1044895126 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984016020 con 0x7fd98c0707f0 2024-09-17T04:49:53.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fbac066b640 1 -- 172.21.15.150:0/1050542587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbaa0001040 con 0x7fbab8070aa0 2024-09-17T04:49:53.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fd988ff9640 1 -- 172.21.15.150:0/1044895126 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd98400ab40 con 0x7fd98c0707f0 2024-09-17T04:49:53.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7fd988ff9640 1 -- 172.21.15.150:0/1044895126 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984012420 con 0x7fd98c0707f0 2024-09-17T04:49:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f5b5affd640 1 -- 172.21.15.150:0/1635772773 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b540ba070 con 0x7f5b641534f0 2024-09-17T04:49:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f5b5affd640 1 -- 172.21.15.150:0/1635772773 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b54007d40 con 0x7f5b641534f0 2024-09-17T04:49:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f5b5affd640 1 -- 172.21.15.150:0/1635772773 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b540b1040 con 0x7f5b641534f0 2024-09-17T04:49:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f5b6b524640 1 -- 172.21.15.150:0/1635772773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b640a79a0 con 0x7f5b641534f0 2024-09-17T04:49:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f40cabce640 1 -- 172.21.15.150:0/386061938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f40c41508f0 con 0x7f40c40a5cd0 2024-09-17T04:49:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f5b6b524640 1 -- 172.21.15.150:0/1635772773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b640a7f60 con 0x7f5b641534f0 2024-09-17T04:49:53.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f896505b640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f895802ecf0 0x7f8958043fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f896605d640 1 -- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f895802e8e0 msgr2=0x7f89580cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f896605d640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f895802e8e0 0x7f89580cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f896605d640 1 -- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f895802ecf0 msgr2=0x7f8958043fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f896605d640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f895802ecf0 0x7f8958043fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f896605d640 1 -- 172.21.15.150:0/694586631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895803f680 con 0x7f89580478a0 2024-09-17T04:49:53.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f896605d640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89580478a0 0x7f8958045050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f896002ebe0 tx=0x7f8960076ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7fd992637640 1 -- 172.21.15.150:0/1044895126 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98c156750 con 0x7fd98c0707f0 2024-09-17T04:49:53.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f1146ffd640 1 -- 172.21.15.150:0/727521340 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148017070 con 0x7f1158075040 2024-09-17T04:49:53.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f1146ffd640 1 -- 172.21.15.150:0/727521340 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1148003600 con 0x7f1158075040 2024-09-17T04:49:53.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f1146ffd640 1 -- 172.21.15.150:0/727521340 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148003a30 con 0x7f1158075040 2024-09-17T04:49:53.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11581f4c60 con 0x7f1158075040 2024-09-17T04:49:53.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1158151b10 con 0x7f1158075040 2024-09-17T04:49:53.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.016+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f113c001040 con 0x7f1158075040 2024-09-17T04:49:53.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f5b6b524640 1 -- 172.21.15.150:0/1635772773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5b640a7790 con 0x7f5b641534f0 2024-09-17T04:49:53.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7fd992637640 1 -- 172.21.15.150:0/1044895126 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98c14de30 con 0x7fd98c0707f0 2024-09-17T04:49:53.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f5b5affd640 1 -- 172.21.15.150:0/1635772773 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5b540bd070 con 0x7f5b641534f0 2024-09-17T04:49:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f40c17fa640 1 -- 172.21.15.150:0/386061938 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40bc0098a0 con 0x7f40c40a5cd0 2024-09-17T04:49:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f40c17fa640 1 -- 172.21.15.150:0/386061938 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc01a030 con 0x7f40c40a5cd0 2024-09-17T04:49:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7fbaaeffd640 1 -- 172.21.15.150:0/1050542587 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbaa80213f0 con 0x7fbab8070aa0 2024-09-17T04:49:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7fbaaeffd640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba9c073200 0x7fba9c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f5b5affd640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b48073280 0x7f5b48075790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7fbabdc06640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba9c073200 0x7fba9c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f5b5affd640 1 -- 172.21.15.150:0/1635772773 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f5b5412d5a0 con 0x7f5b641534f0 2024-09-17T04:49:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f5b5affd640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] conn(0x7f5b4807daf0 0x7f5b4807ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f5b5affd640 1 -- 172.21.15.150:0/1635772773 --> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b540b9ec0 con 0x7f5b4807daf0 2024-09-17T04:49:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7fd992637640 1 -- 172.21.15.150:0/1044895126 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd98c156220 con 0x7fd98c0707f0 2024-09-17T04:49:53.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f5b6ad23640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] conn(0x7f5b4807daf0 0x7f5b4807ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f1146ffd640 1 -- 172.21.15.150:0/727521340 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f114801a070 con 0x7f1158075040 2024-09-17T04:49:53.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f5b6ad23640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] conn(0x7f5b4807daf0 0x7f5b4807ff40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f1146ffd640 1 --2- 172.21.15.150:0/727521340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1138073200 0x7f1138075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7fae5dffb640 1 -- 172.21.15.150:0/287468337 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae64017070 con 0x7fae68075040 2024-09-17T04:49:53.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7fae5dffb640 1 -- 172.21.15.150:0/287468337 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae640053c0 con 0x7fae68075040 2024-09-17T04:49:53.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7fae5dffb640 1 -- 172.21.15.150:0/287468337 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae64003860 con 0x7fae68075040 2024-09-17T04:49:53.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7fae6f320640 1 -- 172.21.15.150:0/287468337 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae681f4530 con 0x7fae68075040 2024-09-17T04:49:53.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7fae6f320640 1 -- 172.21.15.150:0/287468337 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae681f4af0 con 0x7fae68075040 2024-09-17T04:49:53.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f1146ffd640 1 -- 172.21.15.150:0/727521340 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f1148093700 con 0x7f1158075040 2024-09-17T04:49:53.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f1146ffd640 1 --2- 172.21.15.150:0/727521340 >> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] conn(0x7f113807da60 0x7f113807feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f1146ffd640 1 -- 172.21.15.150:0/727521340 --> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1138080580 con 0x7f113807da60 2024-09-17T04:49:53.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f40c17fa640 1 -- 172.21.15.150:0/386061938 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f40bc01d060 con 0x7f40c40a5cd0 2024-09-17T04:49:53.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f40c17fa640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f40b0073200 0x7f40b0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f40c17fa640 1 -- 172.21.15.150:0/386061938 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f40bc094dd0 con 0x7f40c40a5cd0 2024-09-17T04:49:53.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f40c17fa640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] conn(0x7f40b007da60 0x7f40b007feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f40c17fa640 1 -- 172.21.15.150:0/386061938 --> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40b0080580 con 0x7f40b007da60 2024-09-17T04:49:53.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f40c896a640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f40b0073200 0x7f40b0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7fd988ff9640 1 -- 172.21.15.150:0/1044895126 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd984019020 con 0x7fd98c0707f0 2024-09-17T04:49:53.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f5b6a522640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b48073280 0x7f5b48075790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f5b5affd640 1 -- 172.21.15.150:0/1635772773 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5b54134050 con 0x7f5b641534f0 2024-09-17T04:49:53.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f5b5affd640 1 -- 172.21.15.150:0/1635772773 <== osd.0 v2:172.21.15.150:6800/437062879 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5b540b9ec0 con 0x7f5b4807daf0 2024-09-17T04:49:53.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f40c3fff640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] conn(0x7f40b007da60 0x7f40b007feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f89567fc640 1 -- 172.21.15.150:0/694586631 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960091070 con 0x7f89580478a0 2024-09-17T04:49:53.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f89567fc640 1 -- 172.21.15.150:0/694586631 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f896008d560 con 0x7f89580478a0 2024-09-17T04:49:53.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f89567fc640 1 -- 172.21.15.150:0/694586631 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896008d9d0 con 0x7f89580478a0 2024-09-17T04:49:53.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f8967ac0640 1 -- 172.21.15.150:0/694586631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89580d3cd0 con 0x7f89580478a0 2024-09-17T04:49:53.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f8967ac0640 1 -- 172.21.15.150:0/694586631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8958045ab0 con 0x7f89580478a0 2024-09-17T04:49:53.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f1156d9d640 1 --2- 172.21.15.150:0/727521340 >> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] conn(0x7f113807da60 0x7f113807feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.020+0000 7f1155d9b640 1 --2- 172.21.15.150:0/727521340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1138073200 0x7f1138075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f8967ac0640 1 -- 172.21.15.150:0/694586631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f895817b5a0 con 0x7f89580478a0 2024-09-17T04:49:53.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fae6f320640 1 -- 172.21.15.150:0/287468337 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fae681f4ed0 con 0x7fae68075040 2024-09-17T04:49:53.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fd988ff9640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd978073200 0x7fd978075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fd988ff9640 1 -- 172.21.15.150:0/1044895126 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fd984094a70 con 0x7fd98c0707f0 2024-09-17T04:49:53.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f0d0b7fe640 1 -- 172.21.15.150:0/3003713965 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d180014e0 con 0x7f0d241dd020 2024-09-17T04:49:53.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f0d0b7fe640 1 -- 172.21.15.150:0/3003713965 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d180b5070 con 0x7f0d241dd020 2024-09-17T04:49:53.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f0d0b7fe640 1 -- 172.21.15.150:0/3003713965 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d18003950 con 0x7f0d241dd020 2024-09-17T04:49:53.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fbaaeffd640 1 -- 172.21.15.150:0/1050542587 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fbaa8091fe0 con 0x7fbab8070aa0 2024-09-17T04:49:53.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fbabdc06640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba9c073200 0x7fba9c075710 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbab401d7f0 tx=0x7fbab40a8040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fbaaeffd640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] conn(0x7fba9c07da60 0x7fba9c07feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fbaaeffd640 1 -- 172.21.15.150:0/1050542587 --> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba9c080580 con 0x7fba9c07da60 2024-09-17T04:49:53.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fbaaeffd640 1 -- 172.21.15.150:0/1050542587 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbaa805ac20 con 0x7fbab8070aa0 2024-09-17T04:49:53.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fd988ff9640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] conn(0x7fd97807da60 0x7fd97807feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fd988ff9640 1 -- 172.21.15.150:0/1044895126 --> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9840058f0 con 0x7fd97807da60 2024-09-17T04:49:53.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fd988ff9640 1 -- 172.21.15.150:0/1044895126 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd984094f00 con 0x7fd98c0707f0 2024-09-17T04:49:53.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fd98bfff640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd978073200 0x7fd978075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fbabec08640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] conn(0x7fba9c07da60 0x7fba9c07feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f5b6a522640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b48073280 0x7f5b48075790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009c00 tx=0x7f5b600091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f40c17fa640 1 -- 172.21.15.150:0/386061938 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f40bc05ba20 con 0x7f40c40a5cd0 2024-09-17T04:49:53.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fd990bd4640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] conn(0x7fd97807da60 0x7fd97807feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f1155d9b640 1 --2- 172.21.15.150:0/727521340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1138073200 0x7f1138075710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f11500af000 tx=0x7f1150019800 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f0d293f7640 1 -- 172.21.15.150:0/3003713965 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d241dda80 con 0x7f0d241dd020 2024-09-17T04:49:53.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f0d293f7640 1 -- 172.21.15.150:0/3003713965 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d241ddd00 con 0x7f0d241dd020 2024-09-17T04:49:53.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fbabec08640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] conn(0x7fba9c07da60 0x7fba9c07feb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fd98bfff640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd978073200 0x7fd978075710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd97c0ae000 tx=0x7fd97c01aef0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f1156d9d640 1 --2- 172.21.15.150:0/727521340 >> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] conn(0x7f113807da60 0x7f113807feb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f40c896a640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f40b0073200 0x7f40b0075710 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f40b8006fd0 tx=0x7f40b8009060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f0d293f7640 1 -- 172.21.15.150:0/3003713965 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0d04001040 con 0x7f0d241dd020 2024-09-17T04:49:53.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fae5dffb640 1 -- 172.21.15.150:0/287468337 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fae6401a070 con 0x7fae68075040 2024-09-17T04:49:53.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fbaaeffd640 1 -- 172.21.15.150:0/1050542587 <== osd.1 v2:172.21.15.150:6808/4273227163 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba9c080580 con 0x7fba9c07da60 2024-09-17T04:49:53.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f1146ffd640 1 -- 172.21.15.150:0/727521340 <== osd.3 v2:172.21.15.150:6824/158177050 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1138080580 con 0x7f113807da60 2024-09-17T04:49:53.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fd990bd4640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] conn(0x7fd97807da60 0x7fd97807feb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fae5dffb640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae4c073250 0x7fae4c075760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7f0d0b7fe640 1 -- 172.21.15.150:0/3003713965 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0d18003af0 con 0x7f0d241dd020 2024-09-17T04:49:53.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fae5dffb640 1 -- 172.21.15.150:0/287468337 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fae64093b80 con 0x7fae68075040 2024-09-17T04:49:53.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fae5dffb640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] conn(0x7fae4c07db30 0x7fae4c07ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fae5dffb640 1 -- 172.21.15.150:0/287468337 --> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae4c080650 con 0x7fae4c07db30 2024-09-17T04:49:53.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.024+0000 7fae5dffb640 1 -- 172.21.15.150:0/287468337 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fae64094060 con 0x7fae68075040 2024-09-17T04:49:53.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f1146ffd640 1 -- 172.21.15.150:0/727521340 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f114809a050 con 0x7f1158075040 2024-09-17T04:49:53.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f40c3fff640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] conn(0x7f40b007da60 0x7f40b007feb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7fae6c8bb640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae4c073250 0x7fae4c075760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f89567fc640 1 -- 172.21.15.150:0/694586631 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8960094070 con 0x7f89580478a0 2024-09-17T04:49:53.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f0d0b7fe640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0d10073250 0x7f0d10075760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f0d0b7fe640 1 -- 172.21.15.150:0/3003713965 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f0d1812e220 con 0x7f0d241dd020 2024-09-17T04:49:53.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f0d22ffd640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0d10073250 0x7f0d10075760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f0d0b7fe640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] conn(0x7f0d1007db40 0x7f0d1007ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f0d0b7fe640 1 -- 172.21.15.150:0/3003713965 --> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d180bde80 con 0x7f0d1007db40 2024-09-17T04:49:53.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f0d0b7fe640 1 -- 172.21.15.150:0/3003713965 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0d18135050 con 0x7f0d241dd020 2024-09-17T04:49:53.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f0d227fc640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] conn(0x7f0d1007db40 0x7f0d1007ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f89567fc640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8944073200 0x7f8944075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7fae6d8bd640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] conn(0x7fae4c07db30 0x7fae4c07ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f89567fc640 1 -- 172.21.15.150:0/694586631 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f8960110a90 con 0x7f89580478a0 2024-09-17T04:49:53.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f89567fc640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] conn(0x7f894407da60 0x7f894407feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f896585c640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8944073200 0x7f8944075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f89567fc640 1 -- 172.21.15.150:0/694586631 --> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8960094e70 con 0x7f894407da60 2024-09-17T04:49:53.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f89567fc640 1 -- 172.21.15.150:0/694586631 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8960110f20 con 0x7f89580478a0 2024-09-17T04:49:53.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f0d22ffd640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0d10073250 0x7f0d10075760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0d14001700 tx=0x7f0d14001790 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7fae6d8bd640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] conn(0x7fae4c07db30 0x7fae4c07ff80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7fae6c8bb640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae4c073250 0x7fae4c075760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fae600af000 tx=0x7fae60019a20 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7fd988ff9640 1 -- 172.21.15.150:0/1044895126 <== osd.4 v2:172.21.15.176:6826/4247631702 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9840058f0 con 0x7fd97807da60 2024-09-17T04:49:53.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f896505b640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] conn(0x7f894407da60 0x7f894407feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f0d227fc640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] conn(0x7f0d1007db40 0x7f0d1007ff90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f896585c640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8944073200 0x7f8944075710 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f895c005fd0 tx=0x7f895c0095a0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.028+0000 7f40c17fa640 1 -- 172.21.15.150:0/386061938 <== osd.5 v2:172.21.15.176:6810/2266081607 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f40b0080580 con 0x7f40b007da60 2024-09-17T04:49:53.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.032+0000 7f896505b640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] conn(0x7f894407da60 0x7f894407feb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.032+0000 7f89567fc640 1 -- 172.21.15.150:0/694586631 <== osd.6 v2:172.21.15.176:6802/4084217765 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8960094e70 con 0x7f894407da60 2024-09-17T04:49:53.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.032+0000 7f0d0b7fe640 1 -- 172.21.15.150:0/3003713965 <== osd.7 v2:172.21.15.176:6818/621582613 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d180bde80 con 0x7f0d1007db40 2024-09-17T04:49:53.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.036+0000 7fae5dffb640 1 -- 172.21.15.150:0/287468337 <== osd.2 v2:172.21.15.150:6816/1456340777 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae4c080650 con 0x7fae4c07db30 2024-09-17T04:49:53.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b6b524640 1 -- 172.21.15.150:0/1635772773 --> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5b640c3410 con 0x7f5b4807daf0 2024-09-17T04:49:53.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7fd992637640 1 -- 172.21.15.150:0/1044895126 --> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd98c0c3160 con 0x7fd97807da60 2024-09-17T04:49:53.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b5affd640 1 -- 172.21.15.150:0/1635772773 <== osd.0 v2:172.21.15.150:6800/437062879 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5b640c3410 con 0x7f5b4807daf0 2024-09-17T04:49:53.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b6b524640 1 -- 172.21.15.150:0/1635772773 >> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] conn(0x7f5b4807daf0 msgr2=0x7f5b4807ff40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b6b524640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] conn(0x7f5b4807daf0 0x7f5b4807ff40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b6b524640 1 -- 172.21.15.150:0/1635772773 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b48073280 msgr2=0x7f5b48075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b6b524640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b48073280 0x7f5b48075790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009c00 tx=0x7f5b600091e0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b6b524640 1 -- 172.21.15.150:0/1635772773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b641534f0 msgr2=0x7f5b640a3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b6b524640 1 --2- 172.21.15.150:0/1635772773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b641534f0 0x7f5b640a3ba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5b540ab000 tx=0x7f5b540056f0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b6ad23640 1 -- 172.21.15.150:0/1635772773 reap_dead start 2024-09-17T04:49:53.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b6b524640 1 -- 172.21.15.150:0/1635772773 shutdown_connections 2024-09-17T04:49:53.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b6b524640 1 -- 172.21.15.150:0/1635772773 >> 172.21.15.150:0/1635772773 conn(0x7f5b6406ce50 msgr2=0x7f5b641522f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7fd988ff9640 1 -- 172.21.15.150:0/1044895126 <== osd.4 v2:172.21.15.176:6826/4247631702 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd98c0c3160 con 0x7fd97807da60 2024-09-17T04:49:53.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b6b524640 1 -- 172.21.15.150:0/1635772773 shutdown_connections 2024-09-17T04:49:53.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7fbac066b640 1 -- 172.21.15.150:0/1050542587 --> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbaa0002de0 con 0x7fba9c07da60 2024-09-17T04:49:53.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7f5b6b524640 1 -- 172.21.15.150:0/1635772773 wait complete. 2024-09-17T04:49:53.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.052+0000 7fbaaeffd640 1 -- 172.21.15.150:0/1050542587 <== osd.1 v2:172.21.15.150:6808/4273227163 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbaa0002de0 con 0x7fba9c07da60 2024-09-17T04:49:53.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 --> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f113c002de0 con 0x7f113807da60 2024-09-17T04:49:53.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f1146ffd640 1 -- 172.21.15.150:0/727521340 <== osd.3 v2:172.21.15.150:6824/158177050 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f113c002de0 con 0x7f113807da60 2024-09-17T04:49:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7fbac066b640 1 -- 172.21.15.150:0/1050542587 >> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] conn(0x7fba9c07da60 msgr2=0x7fba9c07feb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7fbac066b640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] conn(0x7fba9c07da60 0x7fba9c07feb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7fbac066b640 1 -- 172.21.15.150:0/1050542587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba9c073200 msgr2=0x7fba9c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7fbac066b640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba9c073200 0x7fba9c075710 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbab401d7f0 tx=0x7fbab40a8040 comp rx=0 tx=0).stop 2024-09-17T04:49:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7fbac066b640 1 -- 172.21.15.150:0/1050542587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbab8070aa0 msgr2=0x7fbab80aab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7fbac066b640 1 --2- 172.21.15.150:0/1050542587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbab8070aa0 0x7fbab80aab20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbaa800f6b0 tx=0x7fbaa800fb70 comp rx=0 tx=0).stop 2024-09-17T04:49:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7fbabec08640 1 -- 172.21.15.150:0/1050542587 reap_dead start 2024-09-17T04:49:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7fbac066b640 1 -- 172.21.15.150:0/1050542587 shutdown_connections 2024-09-17T04:49:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7fbac066b640 1 -- 172.21.15.150:0/1050542587 >> 172.21.15.150:0/1050542587 conn(0x7fbab80c7cd0 msgr2=0x7fbab80c80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7fbac066b640 1 -- 172.21.15.150:0/1050542587 shutdown_connections 2024-09-17T04:49:53.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7fbac066b640 1 -- 172.21.15.150:0/1050542587 wait complete. 2024-09-17T04:49:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 >> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] conn(0x7f113807da60 msgr2=0x7f113807feb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f115cb88640 1 --2- 172.21.15.150:0/727521340 >> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] conn(0x7f113807da60 0x7f113807feb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1138073200 msgr2=0x7f1138075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f115cb88640 1 --2- 172.21.15.150:0/727521340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1138073200 0x7f1138075710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f11500af000 tx=0x7f1150019800 comp rx=0 tx=0).stop 2024-09-17T04:49:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1158075040 msgr2=0x7f11581538f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f115cb88640 1 --2- 172.21.15.150:0/727521340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1158075040 0x7f11581538f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f114800c400 tx=0x7f114800c8c0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f1156d9d640 1 -- 172.21.15.150:0/727521340 reap_dead start 2024-09-17T04:49:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 shutdown_connections 2024-09-17T04:49:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 >> 172.21.15.150:0/727521340 conn(0x7f11580c7cd0 msgr2=0x7f1158157b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 shutdown_connections 2024-09-17T04:49:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.056+0000 7f115cb88640 1 -- 172.21.15.150:0/727521340 wait complete. 2024-09-17T04:49:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7fae6f320640 1 -- 172.21.15.150:0/287468337 --> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fae681508f0 con 0x7fae4c07db30 2024-09-17T04:49:53.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d293f7640 1 -- 172.21.15.150:0/3003713965 --> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0d04002de0 con 0x7f0d1007db40 2024-09-17T04:49:53.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7fd992637640 1 -- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] conn(0x7fd97807da60 msgr2=0x7fd97807feb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7fd992637640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] conn(0x7fd97807da60 0x7fd97807feb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f8967ac0640 1 -- 172.21.15.150:0/694586631 --> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8958044cd0 con 0x7f894407da60 2024-09-17T04:49:53.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f40cabce640 1 -- 172.21.15.150:0/386061938 --> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f40c40a3650 con 0x7f40b007da60 2024-09-17T04:49:53.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d0b7fe640 1 -- 172.21.15.150:0/3003713965 <== osd.7 v2:172.21.15.176:6818/621582613 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0d04002de0 con 0x7f0d1007db40 2024-09-17T04:49:53.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d293f7640 1 -- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] conn(0x7f0d1007db40 msgr2=0x7f0d1007ff90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d293f7640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] conn(0x7f0d1007db40 0x7f0d1007ff90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d293f7640 1 -- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0d10073250 msgr2=0x7f0d10075760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d293f7640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0d10073250 0x7f0d10075760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0d14001700 tx=0x7f0d14001790 comp rx=0 tx=0).stop 2024-09-17T04:49:53.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d293f7640 1 -- 172.21.15.150:0/3003713965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d241dd020 msgr2=0x7f0d241498e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d293f7640 1 --2- 172.21.15.150:0/3003713965 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d241dd020 0x7f0d241498e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0d1801b000 tx=0x7f0d18001320 comp rx=0 tx=0).stop 2024-09-17T04:49:53.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d237fe640 1 -- 172.21.15.150:0/3003713965 reap_dead start 2024-09-17T04:49:53.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d293f7640 1 -- 172.21.15.150:0/3003713965 shutdown_connections 2024-09-17T04:49:53.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d293f7640 1 -- 172.21.15.150:0/3003713965 >> 172.21.15.150:0/3003713965 conn(0x7f0d24070ba0 msgr2=0x7f0d2414f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d293f7640 1 -- 172.21.15.150:0/3003713965 shutdown_connections 2024-09-17T04:49:53.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f40c17fa640 1 -- 172.21.15.150:0/386061938 <== osd.5 v2:172.21.15.176:6810/2266081607 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f40c40a3650 con 0x7f40b007da60 2024-09-17T04:49:53.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f0d293f7640 1 -- 172.21.15.150:0/3003713965 wait complete. 2024-09-17T04:49:53.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f89567fc640 1 -- 172.21.15.150:0/694586631 <== osd.6 v2:172.21.15.176:6802/4084217765 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8958044cd0 con 0x7f894407da60 2024-09-17T04:49:53.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f8967ac0640 1 -- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] conn(0x7f894407da60 msgr2=0x7f894407feb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f8967ac0640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] conn(0x7f894407da60 0x7f894407feb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f8967ac0640 1 -- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8944073200 msgr2=0x7f8944075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f8967ac0640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8944073200 0x7f8944075710 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f895c005fd0 tx=0x7f895c0095a0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f8967ac0640 1 -- 172.21.15.150:0/694586631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89580478a0 msgr2=0x7f8958045050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f8967ac0640 1 --2- 172.21.15.150:0/694586631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89580478a0 0x7f8958045050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f896002ebe0 tx=0x7f8960076ea0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f40cabce640 1 -- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] conn(0x7f40b007da60 msgr2=0x7f40b007feb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f40cabce640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] conn(0x7f40b007da60 0x7f40b007feb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f40cabce640 1 -- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f40b0073200 msgr2=0x7f40b0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f40cabce640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f40b0073200 0x7f40b0075710 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f40b8006fd0 tx=0x7f40b8009060 comp rx=0 tx=0).stop 2024-09-17T04:49:53.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f40cabce640 1 -- 172.21.15.150:0/386061938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40c40a5cd0 msgr2=0x7f40c40c03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f40cabce640 1 --2- 172.21.15.150:0/386061938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40c40a5cd0 0x7f40c40c03d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00ef20 tx=0x7f40bc00c800 comp rx=0 tx=0).stop 2024-09-17T04:49:53.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f40c916b640 1 -- 172.21.15.150:0/386061938 reap_dead start 2024-09-17T04:49:53.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f40cabce640 1 -- 172.21.15.150:0/386061938 shutdown_connections 2024-09-17T04:49:53.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.064+0000 7f40cabce640 1 -- 172.21.15.150:0/386061938 >> 172.21.15.150:0/386061938 conn(0x7f40c40c7cd0 msgr2=0x7f40c41547b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7f40cabce640 1 -- 172.21.15.150:0/386061938 shutdown_connections 2024-09-17T04:49:53.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7f40cabce640 1 -- 172.21.15.150:0/386061938 wait complete. 2024-09-17T04:49:53.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fd992637640 1 -- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd978073200 msgr2=0x7fd978075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fd992637640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd978073200 0x7fd978075710 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd97c0ae000 tx=0x7fd97c01aef0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fd992637640 1 -- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd98c0707f0 msgr2=0x7fd98c14fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7f896605d640 1 -- 172.21.15.150:0/694586631 reap_dead start 2024-09-17T04:49:53.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7f8967ac0640 1 -- 172.21.15.150:0/694586631 shutdown_connections 2024-09-17T04:49:53.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7f8967ac0640 1 -- 172.21.15.150:0/694586631 >> 172.21.15.150:0/694586631 conn(0x7f8958049590 msgr2=0x7f895804afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fd992637640 1 --2- 172.21.15.150:0/1044895126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd98c0707f0 0x7fd98c14fa30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd984001f10 tx=0x7fd984002000 comp rx=0 tx=0).stop 2024-09-17T04:49:53.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fd990bd4640 1 -- 172.21.15.150:0/1044895126 reap_dead start 2024-09-17T04:49:53.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fd992637640 1 -- 172.21.15.150:0/1044895126 shutdown_connections 2024-09-17T04:49:53.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fd992637640 1 -- 172.21.15.150:0/1044895126 >> 172.21.15.150:0/1044895126 conn(0x7fd98c0c7cd0 msgr2=0x7fd98c074b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fae5dffb640 1 -- 172.21.15.150:0/287468337 <== osd.2 v2:172.21.15.150:6816/1456340777 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fae681508f0 con 0x7fae4c07db30 2024-09-17T04:49:53.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fae6f320640 1 -- 172.21.15.150:0/287468337 >> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] conn(0x7fae4c07db30 msgr2=0x7fae4c07ff80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fae6f320640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] conn(0x7fae4c07db30 0x7fae4c07ff80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fae6f320640 1 -- 172.21.15.150:0/287468337 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae4c073250 msgr2=0x7fae4c075760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fae6f320640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae4c073250 0x7fae4c075760 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fae600af000 tx=0x7fae60019a20 comp rx=0 tx=0).stop 2024-09-17T04:49:53.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fae6f320640 1 -- 172.21.15.150:0/287468337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae68075040 msgr2=0x7fae680c2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fae6f320640 1 --2- 172.21.15.150:0/287468337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae68075040 0x7fae680c2500 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fae6400c3f0 tx=0x7fae6400c8b0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7f8967ac0640 1 -- 172.21.15.150:0/694586631 shutdown_connections 2024-09-17T04:49:53.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fae6d8bd640 1 -- 172.21.15.150:0/287468337 reap_dead start 2024-09-17T04:49:53.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fae6f320640 1 -- 172.21.15.150:0/287468337 shutdown_connections 2024-09-17T04:49:53.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fae6f320640 1 -- 172.21.15.150:0/287468337 >> 172.21.15.150:0/287468337 conn(0x7fae680c7cd0 msgr2=0x7fae68151230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fae6f320640 1 -- 172.21.15.150:0/287468337 shutdown_connections 2024-09-17T04:49:53.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7f8967ac0640 1 -- 172.21.15.150:0/694586631 wait complete. 2024-09-17T04:49:53.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.068+0000 7fae6f320640 1 -- 172.21.15.150:0/287468337 wait complete. 2024-09-17T04:49:53.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.072+0000 7fd992637640 1 -- 172.21.15.150:0/1044895126 shutdown_connections 2024-09-17T04:49:53.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.072+0000 7fd992637640 1 -- 172.21.15.150:0/1044895126 wait complete. 2024-09-17T04:49:53.105 INFO:teuthology.orchestra.run.smithi150.stdout:64424509450 2024-09-17T04:49:53.105 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T04:49:53.106 INFO:teuthology.orchestra.run.smithi150.stdout:64424509451 2024-09-17T04:49:53.107 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T04:49:53.107 INFO:teuthology.orchestra.run.smithi150.stdout:64424509450 2024-09-17T04:49:53.108 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T04:49:53.109 INFO:teuthology.orchestra.run.smithi150.stdout:64424509451 2024-09-17T04:49:53.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T04:49:53.113 INFO:teuthology.orchestra.run.smithi150.stdout:64424509450 2024-09-17T04:49:53.113 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T04:49:53.116 INFO:teuthology.orchestra.run.smithi150.stdout:64424509450 2024-09-17T04:49:53.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-17T04:49:53.117 INFO:teuthology.orchestra.run.smithi150.stdout:60129542154 2024-09-17T04:49:53.117 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-17T04:49:53.119 INFO:teuthology.orchestra.run.smithi150.stdout:64424509450 2024-09-17T04:49:53.119 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T04:49:53.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7fa3fa5b6640 1 -- 172.21.15.150:0/1268067051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f4073540 msgr2=0x7fa3f406d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7fa3fa5b6640 1 --2- 172.21.15.150:0/1268067051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f4073540 0x7fa3f406d560 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4072ef0 tx=0x7fa3ec01caf0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7fa3fa5b6640 1 -- 172.21.15.150:0/1268067051 shutdown_connections 2024-09-17T04:49:53.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7fa3fa5b6640 1 --2- 172.21.15.150:0/1268067051 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f4073540 0x7fa3f406d560 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7fa3fa5b6640 1 --2- 172.21.15.150:0/1268067051 >> v2:172.21.15.176:3300/0 conn(0x7fa3f40a72a0 0x7fa3f4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7fa3fa5b6640 1 -- 172.21.15.150:0/1268067051 >> 172.21.15.150:0/1268067051 conn(0x7fa3f40c4e30 msgr2=0x7fa3f406ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7fa3fa5b6640 1 -- 172.21.15.150:0/1268067051 shutdown_connections 2024-09-17T04:49:53.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7fa3fa5b6640 1 -- 172.21.15.150:0/1268067051 wait complete. 2024-09-17T04:49:53.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7f62360d5640 1 -- 172.21.15.150:0/3385318648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6230074f70 msgr2=0x7f62300734d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7f62360d5640 1 --2- 172.21.15.150:0/3385318648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6230074f70 0x7f62300734d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6230075620 tx=0x7f622801ccb0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7f62360d5640 1 -- 172.21.15.150:0/3385318648 shutdown_connections 2024-09-17T04:49:53.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7f62360d5640 1 --2- 172.21.15.150:0/3385318648 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6230156c50 0x7f62301659d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7f62360d5640 1 --2- 172.21.15.150:0/3385318648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6230074f70 0x7f62300734d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7f62360d5640 1 -- 172.21.15.150:0/3385318648 >> 172.21.15.150:0/3385318648 conn(0x7f62300c4e30 msgr2=0x7f623006ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7f62360d5640 1 -- 172.21.15.150:0/3385318648 shutdown_connections 2024-09-17T04:49:53.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.312+0000 7f62360d5640 1 -- 172.21.15.150:0/3385318648 wait complete. 2024-09-17T04:49:53.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3fa5b6640 1 Processor -- start 2024-09-17T04:49:53.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3fa5b6640 1 -- start start 2024-09-17T04:49:53.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7f9c1b67c640 1 -- 172.21.15.150:0/2636882062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c0c02d5f0 msgr2=0x7f9c0c02da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7f9c1b67c640 1 --2- 172.21.15.150:0/2636882062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c0c02d5f0 0x7f9c0c02da00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00d000 tx=0x7f9c0401d8a0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7f9c1b67c640 1 -- 172.21.15.150:0/2636882062 shutdown_connections 2024-09-17T04:49:53.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7f9c1b67c640 1 --2- 172.21.15.150:0/2636882062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c0c02d5f0 0x7f9c0c02da00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7f9c1b67c640 1 --2- 172.21.15.150:0/2636882062 >> v2:172.21.15.176:3300/0 conn(0x7f9c0c030cc0 0x7f9c0c0d5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7f9c1b67c640 1 -- 172.21.15.150:0/2636882062 >> 172.21.15.150:0/2636882062 conn(0x7f9c0c048630 msgr2=0x7f9c0c02cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7f9c1b67c640 1 -- 172.21.15.150:0/2636882062 shutdown_connections 2024-09-17T04:49:53.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7f9c1b67c640 1 -- 172.21.15.150:0/2636882062 wait complete. 2024-09-17T04:49:53.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7f62360d5640 1 Processor -- start 2024-09-17T04:49:53.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3fa5b6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3f4071b20 0x7fa3f41f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3fa5b6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3f4073540 0x7fa3f41536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3fa5b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f40a72a0 0x7fa3f40ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3f8b53640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f40a72a0 0x7fa3f40ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3f8b53640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f40a72a0 0x7fa3f40ad2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33276/0 (socket says 172.21.15.150:33276) 2024-09-17T04:49:53.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3f37fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3f4073540 0x7fa3f41536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3f37fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3f4073540 0x7fa3f41536b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:47366/0 (socket says 172.21.15.150:47366) 2024-09-17T04:49:53.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3f3fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3f4071b20 0x7fa3f41f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3f3fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3f4071b20 0x7fa3f41f3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:44854/0 (socket says 172.21.15.150:44854) 2024-09-17T04:49:53.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3f8b53640 1 -- 172.21.15.150:0/4037602783 learned_addr learned my addr 172.21.15.150:0/4037602783 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3fa5b6640 1 -- 172.21.15.150:0/4037602783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f4163d70 con 0x7fa3f40a72a0 2024-09-17T04:49:53.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3fa5b6640 1 -- 172.21.15.150:0/4037602783 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f4163bf0 con 0x7fa3f4071b20 2024-09-17T04:49:53.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3fa5b6640 1 -- 172.21.15.150:0/4037602783 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa3f4163ef0 con 0x7fa3f4073540 2024-09-17T04:49:53.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7f62360d5640 1 -- start start 2024-09-17T04:49:53.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3f3fff640 1 -- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3f4073540 msgr2=0x7fa3f41536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3f3fff640 1 --2- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3f4073540 0x7fa3f41536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3f3fff640 1 -- 172.21.15.150:0/4037602783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f40a72a0 msgr2=0x7fa3f40ad2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3f3fff640 1 --2- 172.21.15.150:0/4037602783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f40a72a0 0x7fa3f40ad2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7fa3f3fff640 1 -- 172.21.15.150:0/4037602783 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f40adfd0 con 0x7fa3f4071b20 2024-09-17T04:49:53.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7f62360d5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6230074f70 0x7f623014c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.316+0000 7f62360d5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6230156840 0x7f623014f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f62360d5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6230156c50 0x7f6230152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f62360d5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6230071c50 con 0x7f6230156c50 2024-09-17T04:49:53.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c1b67c640 1 Processor -- start 2024-09-17T04:49:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f622f7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6230074f70 0x7f623014c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f622f7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6230074f70 0x7f623014c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:44860/0 (socket says 172.21.15.150:44860) 2024-09-17T04:49:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f622f7fe640 1 -- 172.21.15.150:0/378858577 learned_addr learned my addr 172.21.15.150:0/378858577 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c1b67c640 1 -- start start 2024-09-17T04:49:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7fa3f3fff640 1 --2- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3f4071b20 0x7fa3f41f3400 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800e6f0 tx=0x7fa3e800ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f62360d5640 1 -- 172.21.15.150:0/378858577 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f62300c2400 con 0x7f6230074f70 2024-09-17T04:49:53.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f62360d5640 1 -- 172.21.15.150:0/378858577 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6230071dd0 con 0x7f6230156840 2024-09-17T04:49:53.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f622effd640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6230156840 0x7f623014f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f622ffff640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6230156c50 0x7f6230152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7fa3f0ff9640 1 -- 172.21.15.150:0/4037602783 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e800cab0 con 0x7fa3f4071b20 2024-09-17T04:49:53.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7fa3f0ff9640 1 -- 172.21.15.150:0/4037602783 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3e800cd40 con 0x7fa3f4071b20 2024-09-17T04:49:53.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7fa3f0ff9640 1 -- 172.21.15.150:0/4037602783 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8010690 con 0x7fa3f4071b20 2024-09-17T04:49:53.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f622ffff640 1 -- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6230156840 msgr2=0x7f623014f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f622ffff640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6230156840 0x7f623014f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f622ffff640 1 -- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6230074f70 msgr2=0x7f623014c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f622ffff640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6230074f70 0x7f623014c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f622ffff640 1 -- 172.21.15.150:0/378858577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62301f7690 con 0x7f6230156c50 2024-09-17T04:49:53.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7fa3fa5b6640 1 -- 172.21.15.150:0/4037602783 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3f414fdd0 con 0x7fa3f4071b20 2024-09-17T04:49:53.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7fa3fa5b6640 1 -- 172.21.15.150:0/4037602783 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3f41f3d70 con 0x7fa3f4071b20 2024-09-17T04:49:53.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f622ffff640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6230156c50 0x7f6230152630 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f62300a7060 tx=0x7f622000ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c1b67c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c0c02d5f0 0x7f9c0c044070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c1b67c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c0c030cc0 0x7f9c0c03f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f6217fff640 1 -- 172.21.15.150:0/378858577 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220004220 con 0x7f6230156c50 2024-09-17T04:49:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f6217fff640 1 -- 172.21.15.150:0/378858577 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6220005020 con 0x7f6230156c50 2024-09-17T04:49:53.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f62360d5640 1 -- 172.21.15.150:0/378858577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6230153260 con 0x7f6230156c50 2024-09-17T04:49:53.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c19418640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c0c02d5f0 0x7f9c0c044070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c19418640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c0c02d5f0 0x7f9c0c044070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:44866/0 (socket says 172.21.15.150:44866) 2024-09-17T04:49:53.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c19418640 1 -- 172.21.15.150:0/1056047346 learned_addr learned my addr 172.21.15.150:0/1056047346 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c1b67c640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c0c045d70 0x7f9c0c1726b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7fa3fa5b6640 1 -- 172.21.15.150:0/4037602783 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3f406d560 con 0x7fa3f4071b20 2024-09-17T04:49:53.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c0d6070 con 0x7f9c0c030cc0 2024-09-17T04:49:53.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9c0c0d6c80 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9c0c0da250 con 0x7f9c0c045d70 2024-09-17T04:49:53.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c19c19640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c0c045d70 0x7f9c0c1726b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c19418640 1 -- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c0c045d70 msgr2=0x7f9c0c1726b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c19418640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c0c045d70 0x7f9c0c1726b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c19418640 1 -- 172.21.15.150:0/1056047346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c0c030cc0 msgr2=0x7f9c0c03f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c19418640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c0c030cc0 0x7f9c0c03f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c19418640 1 -- 172.21.15.150:0/1056047346 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c0c0438a0 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.320+0000 7f9c19418640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c0c02d5f0 0x7f9c0c044070 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000e760 tx=0x7f9c1000ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.324+0000 7fa3f0ff9640 1 -- 172.21.15.150:0/4037602783 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa3e8010830 con 0x7fa3f4071b20 2024-09-17T04:49:53.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.324+0000 7fa3f0ff9640 1 --2- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa3dc073350 0x7fa3dc075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.324+0000 7fa3f0ff9640 1 -- 172.21.15.150:0/4037602783 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fa3e8095260 con 0x7fa3f4071b20 2024-09-17T04:49:53.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.324+0000 7fa3f37fe640 1 --2- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa3dc073350 0x7fa3dc075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.324+0000 7f6217fff640 1 -- 172.21.15.150:0/378858577 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220005470 con 0x7f6230156c50 2024-09-17T04:49:53.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.324+0000 7f6217fff640 1 -- 172.21.15.150:0/378858577 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f622000c7e0 con 0x7f6230156c50 2024-09-17T04:49:53.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.324+0000 7f6217fff640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f621c073280 0x7f621c075790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.324+0000 7f62360d5640 1 -- 172.21.15.150:0/378858577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f623014fa40 con 0x7f6230156c50 2024-09-17T04:49:53.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.324+0000 7fa3f37fe640 1 --2- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa3dc073350 0x7fa3dc075860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40046c0 tx=0x7fa3e4007ea0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.324+0000 7f6217fff640 1 -- 172.21.15.150:0/378858577 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f6220005610 con 0x7f6230156c50 2024-09-17T04:49:53.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.324+0000 7f622f7fe640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f621c073280 0x7f621c075790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.328+0000 7f622f7fe640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f621c073280 0x7f621c075790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f622801d550 tx=0x7f622801b550 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.328+0000 7f6215ffb640 1 -- 172.21.15.150:0/378858577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62080052a0 con 0x7f6230156c50 2024-09-17T04:49:53.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.332+0000 7f6217fff640 1 -- 172.21.15.150:0/378858577 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f62301f7690 con 0x7f6230156c50 2024-09-17T04:49:53.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.336+0000 7fa3f0ff9640 1 -- 172.21.15.150:0/4037602783 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa3e805f970 con 0x7fa3f4071b20 2024-09-17T04:49:53.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.336+0000 7f9c09ffb640 1 -- 172.21.15.150:0/1056047346 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c10018020 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.336+0000 7f9c09ffb640 1 -- 172.21.15.150:0/1056047346 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c1000ca30 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.336+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c0c1730e0 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.336+0000 7f9c09ffb640 1 -- 172.21.15.150:0/1056047346 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c1000ce40 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.336+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c0c0d97a0 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.340+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c0c02da00 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.340+0000 7f9c09ffb640 1 -- 172.21.15.150:0/1056047346 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9c1001b020 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.340+0000 7f9c09ffb640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9bf8073200 0x7f9bf8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7f31818a8640 1 -- 172.21.15.150:0/2246072401 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31740489b0 msgr2=0x7f31740d5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7f31818a8640 1 --2- 172.21.15.150:0/2246072401 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31740489b0 0x7f31740d5860 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3168009930 tx=0x7f316801c800 comp rx=0 tx=0).stop 2024-09-17T04:49:53.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7f9c09ffb640 1 -- 172.21.15.150:0/1056047346 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f9c10094b70 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7f9c18c17640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9bf8073200 0x7f9bf8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7f9c18c17640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9bf8073200 0x7f9bf8075710 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9c040b0000 tx=0x7f9c0401a640 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7f31818a8640 1 -- 172.21.15.150:0/2246072401 shutdown_connections 2024-09-17T04:49:53.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7f31818a8640 1 --2- 172.21.15.150:0/2246072401 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31740489b0 0x7f31740d5860 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7f31818a8640 1 -- 172.21.15.150:0/2246072401 >> 172.21.15.150:0/2246072401 conn(0x7f317404d620 msgr2=0x7f317404da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7f31818a8640 1 -- 172.21.15.150:0/2246072401 shutdown_connections 2024-09-17T04:49:53.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7f31818a8640 1 -- 172.21.15.150:0/2246072401 wait complete. 2024-09-17T04:49:53.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7fa55b655640 1 -- 172.21.15.150:0/2450265829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5540c4250 msgr2=0x7fa554153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7fa55b655640 1 --2- 172.21.15.150:0/2450265829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5540c4250 0x7fa554153ee0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa550009900 tx=0x7fa55001c8e0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7fa55b655640 1 -- 172.21.15.150:0/2450265829 shutdown_connections 2024-09-17T04:49:53.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7fa55b655640 1 --2- 172.21.15.150:0/2450265829 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5540c3d10 0x7fa55415df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7fa55b655640 1 --2- 172.21.15.150:0/2450265829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5540c4250 0x7fa554153ee0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7fa55b655640 1 -- 172.21.15.150:0/2450265829 >> 172.21.15.150:0/2450265829 conn(0x7fa55406e360 msgr2=0x7fa55406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7fa55b655640 1 -- 172.21.15.150:0/2450265829 shutdown_connections 2024-09-17T04:49:53.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.344+0000 7fa55b655640 1 -- 172.21.15.150:0/2450265829 wait complete. 2024-09-17T04:49:53.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31818a8640 1 Processor -- start 2024-09-17T04:49:53.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31818a8640 1 -- start start 2024-09-17T04:49:53.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7fa55b655640 1 Processor -- start 2024-09-17T04:49:53.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7fa55b655640 1 -- start start 2024-09-17T04:49:53.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31818a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31740489b0 0x7f3174009070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31818a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f317404c390 0x7f3174180900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31818a8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31740db9b0 0x7f31740d4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31808a6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31740489b0 0x7f3174009070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31810a7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31740db9b0 0x7f31740d4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31810a7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31740db9b0 0x7f31740d4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33310/0 (socket says 172.21.15.150:33310) 2024-09-17T04:49:53.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31808a6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31740489b0 0x7f3174009070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:44882/0 (socket says 172.21.15.150:44882) 2024-09-17T04:49:53.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31810a7640 1 -- 172.21.15.150:0/2022370438 learned_addr learned my addr 172.21.15.150:0/2022370438 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f317404cf50 con 0x7f31740db9b0 2024-09-17T04:49:53.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f317404a9d0 con 0x7f31740489b0 2024-09-17T04:49:53.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.348+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f317404d0d0 con 0x7f317404c390 2024-09-17T04:49:53.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7f317bfff640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f317404c390 0x7f3174180900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7f31810a7640 1 -- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f317404c390 msgr2=0x7f3174180900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7f31810a7640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f317404c390 0x7f3174180900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7f31810a7640 1 -- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31740489b0 msgr2=0x7f3174009070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7f31810a7640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31740489b0 0x7f3174009070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7f31810a7640 1 -- 172.21.15.150:0/2022370438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3174181360 con 0x7f31740db9b0 2024-09-17T04:49:53.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7f31808a6640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31740489b0 0x7f3174009070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-09-17T04:49:53.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa55b655640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5540c3900 0x7fa5540bf720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa55b655640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5540c3d10 0x7fa5540be3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa55b655640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5540c4250 0x7fa5540c0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa5593f1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5540c3900 0x7fa5540bf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa5593f1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5540c3900 0x7fa5540bf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33318/0 (socket says 172.21.15.150:33318) 2024-09-17T04:49:53.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa5593f1640 1 -- 172.21.15.150:0/955985010 learned_addr learned my addr 172.21.15.150:0/955985010 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa55b655640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa554071120 con 0x7fa5540c3900 2024-09-17T04:49:53.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7f31810a7640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31740db9b0 0x7f31740d4b60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f317c076440 tx=0x7f317c076ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa55b655640 1 -- 172.21.15.150:0/955985010 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa554070fa0 con 0x7fa5540c3d10 2024-09-17T04:49:53.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa55b655640 1 -- 172.21.15.150:0/955985010 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa554072db0 con 0x7fa5540c4250 2024-09-17T04:49:53.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa559bf2640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5540c4250 0x7fa5540c0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa558bf0640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5540c3d10 0x7fa5540be3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa5593f1640 1 -- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5540c4250 msgr2=0x7fa5540c0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa5593f1640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5540c4250 0x7fa5540c0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa5593f1640 1 -- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5540c3d10 msgr2=0x7fa5540be3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa5593f1640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5540c3d10 0x7fa5540be3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa5593f1640 1 -- 172.21.15.150:0/955985010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5541efb40 con 0x7fa5540c3900 2024-09-17T04:49:53.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa5593f1640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5540c3900 0x7fa5540bf720 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa55001c710 tx=0x7fa550005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa549ffb640 1 -- 172.21.15.150:0/955985010 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5500b2040 con 0x7fa5540c3900 2024-09-17T04:49:53.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa549ffb640 1 -- 172.21.15.150:0/955985010 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa550004340 con 0x7fa5540c3900 2024-09-17T04:49:53.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7f31797fa640 1 -- 172.21.15.150:0/2022370438 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c091070 con 0x7f31740db9b0 2024-09-17T04:49:53.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7f31797fa640 1 -- 172.21.15.150:0/2022370438 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f317c08d470 con 0x7f31740db9b0 2024-09-17T04:49:53.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f317402d000 con 0x7f31740db9b0 2024-09-17T04:49:53.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.352+0000 7fa549ffb640 1 -- 172.21.15.150:0/955985010 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550003860 con 0x7fa5540c3900 2024-09-17T04:49:53.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.356+0000 7fa55b655640 1 -- 172.21.15.150:0/955985010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5540c2d70 con 0x7fa5540c3900 2024-09-17T04:49:53.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.356+0000 7f9c09ffb640 1 -- 172.21.15.150:0/1056047346 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9c10022030 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.356+0000 7fa55b655640 1 -- 172.21.15.150:0/955985010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5541ef060 con 0x7fa5540c3900 2024-09-17T04:49:53.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.356+0000 7fa5377fe640 1 -- 172.21.15.150:0/955985010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5300052a0 con 0x7fa5540c3900 2024-09-17T04:49:53.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.356+0000 7f31797fa640 1 -- 172.21.15.150:0/2022370438 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c08d940 con 0x7f31740db9b0 2024-09-17T04:49:53.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.356+0000 7f31797fa640 1 -- 172.21.15.150:0/2022370438 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f317c094070 con 0x7f31740db9b0 2024-09-17T04:49:53.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.356+0000 7f31797fa640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31600731e0 0x7f31600756f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.356+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31740db3c0 con 0x7f31740db9b0 2024-09-17T04:49:53.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.360+0000 7f31808a6640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31600731e0 0x7f31600756f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.360+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31740d5860 con 0x7f31740db9b0 2024-09-17T04:49:53.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.360+0000 7f31808a6640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31600731e0 0x7f31600756f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3170001040 tx=0x7f3170009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.368+0000 7fa549ffb640 1 -- 172.21.15.150:0/955985010 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa550003a00 con 0x7fa5540c3900 2024-09-17T04:49:53.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.368+0000 7fa549ffb640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa538073230 0x7fa538075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.368+0000 7fa549ffb640 1 -- 172.21.15.150:0/955985010 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fa55012df50 con 0x7fa5540c3900 2024-09-17T04:49:53.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.368+0000 7fa558bf0640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa538073230 0x7fa538075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.368+0000 7fa558bf0640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa538073230 0x7fa538075740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa554150290 tx=0x7fa544009060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.372+0000 7f31797fa640 1 -- 172.21.15.150:0/2022370438 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f317c0d6480 con 0x7f31740db9b0 2024-09-17T04:49:53.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.372+0000 7fa549ffb640 1 -- 172.21.15.150:0/955985010 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa5500164d0 con 0x7fa5540c3900 2024-09-17T04:49:53.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.376+0000 7f31797fa640 1 -- 172.21.15.150:0/2022370438 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f317c0da910 con 0x7f31740db9b0 2024-09-17T04:49:53.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.376+0000 7fd98f59e640 1 -- 172.21.15.150:0/4179762430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9880478a0 msgr2=0x7fd9880d79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.376+0000 7fd98f59e640 1 --2- 172.21.15.150:0/4179762430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9880478a0 0x7fd9880d79a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd984005fd0 tx=0x7fd98401c820 comp rx=0 tx=0).stop 2024-09-17T04:49:53.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 -- 172.21.15.150:0/4179762430 shutdown_connections 2024-09-17T04:49:53.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 --2- 172.21.15.150:0/4179762430 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9880478a0 0x7fd9880d79a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 -- 172.21.15.150:0/4179762430 >> 172.21.15.150:0/4179762430 conn(0x7fd988049590 msgr2=0x7fd98802e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 -- 172.21.15.150:0/4179762430 shutdown_connections 2024-09-17T04:49:53.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 -- 172.21.15.150:0/4179762430 wait complete. 2024-09-17T04:49:53.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 Processor -- start 2024-09-17T04:49:53.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 -- start start 2024-09-17T04:49:53.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd98802e8e0 0x7fd98800b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd98802ecf0 0x7fd988044000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9880478a0 0x7fd9880c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd9880e5660 con 0x7fd9880478a0 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd9880e54e0 con 0x7fd98802e8e0 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98f59e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd9880e57e0 con 0x7fd98802ecf0 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9880478a0 0x7fd9880c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9880478a0 0x7fd9880c9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33338/0 (socket says 172.21.15.150:33338) 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.380+0000 7fd98ed9d640 1 -- 172.21.15.150:0/755976639 learned_addr learned my addr 172.21.15.150:0/755976639 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd98dd9b640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd98802ecf0 0x7fd988044000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd98e59c640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd98802e8e0 0x7fd98800b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd98ed9d640 1 -- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd98802ecf0 msgr2=0x7fd988044000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd98ed9d640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd98802ecf0 0x7fd988044000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd98ed9d640 1 -- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd98802e8e0 msgr2=0x7fd98800b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd98ed9d640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd98802e8e0 0x7fd98800b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd98ed9d640 1 -- 172.21.15.150:0/755976639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9880d85d0 con 0x7fd9880478a0 2024-09-17T04:49:53.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd98dd9b640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd98802ecf0 0x7fd988044000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:53.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd98ed9d640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9880478a0 0x7fd9880c9990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd99006b590 tx=0x7fd990076ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd97effd640 1 -- 172.21.15.150:0/755976639 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99008e5e0 con 0x7fd9880478a0 2024-09-17T04:49:53.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd97effd640 1 -- 172.21.15.150:0/755976639 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd99008e8b0 con 0x7fd9880478a0 2024-09-17T04:49:53.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd97effd640 1 -- 172.21.15.150:0/755976639 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99008ed00 con 0x7fd9880478a0 2024-09-17T04:49:53.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd98f59e640 1 -- 172.21.15.150:0/755976639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9880450d0 con 0x7fd9880478a0 2024-09-17T04:49:53.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.384+0000 7fd98f59e640 1 -- 172.21.15.150:0/755976639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9880ca810 con 0x7fd9880478a0 2024-09-17T04:49:53.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.388+0000 7fd97effd640 1 -- 172.21.15.150:0/755976639 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd990096400 con 0x7fd9880478a0 2024-09-17T04:49:53.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.388+0000 7fd97effd640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd970073200 0x7fd970075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.388+0000 7fd97effd640 1 -- 172.21.15.150:0/755976639 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fd9901128f0 con 0x7fd9880478a0 2024-09-17T04:49:53.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.388+0000 7fd98e59c640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd970073200 0x7fd970075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.388+0000 7fd98f59e640 1 -- 172.21.15.150:0/755976639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9880d79a0 con 0x7fd9880478a0 2024-09-17T04:49:53.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.388+0000 7fd98e59c640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd970073200 0x7fd970075710 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd9840ae000 tx=0x7fd98401b060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.392+0000 7f3e09573640 1 -- 172.21.15.150:0/1367481962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e040ad7f0 msgr2=0x7f3e040c6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.392+0000 7f3e09573640 1 --2- 172.21.15.150:0/1367481962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e040ad7f0 0x7f3e040c6230 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009840 tx=0x7f3df401c240 comp rx=0 tx=0).stop 2024-09-17T04:49:53.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.392+0000 7f3e09573640 1 -- 172.21.15.150:0/1367481962 shutdown_connections 2024-09-17T04:49:53.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.392+0000 7f3e09573640 1 --2- 172.21.15.150:0/1367481962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e040ad7f0 0x7f3e040c6230 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.392+0000 7f3e09573640 1 -- 172.21.15.150:0/1367481962 >> 172.21.15.150:0/1367481962 conn(0x7f3e040c9de0 msgr2=0x7f3e040ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.392+0000 7f3e09573640 1 -- 172.21.15.150:0/1367481962 shutdown_connections 2024-09-17T04:49:53.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.392+0000 7f3e09573640 1 -- 172.21.15.150:0/1367481962 wait complete. 2024-09-17T04:49:53.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.392+0000 7f3e09573640 1 Processor -- start 2024-09-17T04:49:53.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.392+0000 7f3e09573640 1 -- start start 2024-09-17T04:49:53.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e09573640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e040ad7f0 0x7f3e04072ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e09573640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e04151e10 0x7f3e04145e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e09573640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e04157ca0 0x7f3e0408b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e09573640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3e040ad210 con 0x7f3e04157ca0 2024-09-17T04:49:53.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e09573640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3e040c5a40 con 0x7f3e040ad7f0 2024-09-17T04:49:53.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e09573640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3e04162860 con 0x7f3e04151e10 2024-09-17T04:49:53.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e037fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e04157ca0 0x7f3e0408b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e037fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e04157ca0 0x7f3e0408b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33348/0 (socket says 172.21.15.150:33348) 2024-09-17T04:49:53.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e037fe640 1 -- 172.21.15.150:0/808188963 learned_addr learned my addr 172.21.15.150:0/808188963 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e027fc640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e04151e10 0x7f3e04145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e02ffd640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e040ad7f0 0x7f3e04072ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e037fe640 1 -- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e04151e10 msgr2=0x7f3e04145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e037fe640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e04151e10 0x7f3e04145e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e037fe640 1 -- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e040ad7f0 msgr2=0x7f3e04072ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e037fe640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e040ad7f0 0x7f3e04072ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e037fe640 1 -- 172.21.15.150:0/808188963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e041505d0 con 0x7f3e04157ca0 2024-09-17T04:49:53.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.396+0000 7f3e037fe640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e04157ca0 0x7f3e0408b490 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc00c8c0 tx=0x7f3dfc00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.404+0000 7f3deb7fe640 1 -- 172.21.15.150:0/808188963 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc017070 con 0x7f3e04157ca0 2024-09-17T04:49:53.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.404+0000 7f3e09573640 1 -- 172.21.15.150:0/808188963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e041502f0 con 0x7f3e04157ca0 2024-09-17T04:49:53.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.404+0000 7f3e09573640 1 -- 172.21.15.150:0/808188963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e04135a70 con 0x7f3e04157ca0 2024-09-17T04:49:53.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.404+0000 7fd97effd640 1 -- 172.21.15.150:0/755976639 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd990092030 con 0x7fd9880478a0 2024-09-17T04:49:53.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.404+0000 7f3deb7fe640 1 -- 172.21.15.150:0/808188963 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dfc003db0 con 0x7f3e04157ca0 2024-09-17T04:49:53.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.404+0000 7f3deb7fe640 1 -- 172.21.15.150:0/808188963 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc021060 con 0x7f3e04157ca0 2024-09-17T04:49:53.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.404+0000 7f82c2a74640 1 -- 172.21.15.150:0/142125335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82b40fb370 msgr2=0x7f82b40fb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.404+0000 7f82c2a74640 1 --2- 172.21.15.150:0/142125335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82b40fb370 0x7f82b40fb780 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f82bc076440 tx=0x7f82bc089800 comp rx=0 tx=0).stop 2024-09-17T04:49:53.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.404+0000 7f82c2a74640 1 -- 172.21.15.150:0/142125335 shutdown_connections 2024-09-17T04:49:53.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.404+0000 7f82c2a74640 1 --2- 172.21.15.150:0/142125335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82b40fb370 0x7f82b40fb780 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.404+0000 7f82c2a74640 1 --2- 172.21.15.150:0/142125335 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82b400a850 0x7f82b400ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.404+0000 7f82c2a74640 1 -- 172.21.15.150:0/142125335 >> 172.21.15.150:0/142125335 conn(0x7f82b4048270 msgr2=0x7f82b4048680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.408+0000 7f82c2a74640 1 -- 172.21.15.150:0/142125335 shutdown_connections 2024-09-17T04:49:53.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.408+0000 7f82c2a74640 1 -- 172.21.15.150:0/142125335 wait complete. 2024-09-17T04:49:53.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.408+0000 7f82c2a74640 1 Processor -- start 2024-09-17T04:49:53.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.408+0000 7f3e09573640 1 -- 172.21.15.150:0/808188963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de40052a0 con 0x7f3e04157ca0 2024-09-17T04:49:53.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.408+0000 7f3deb7fe640 1 -- 172.21.15.150:0/808188963 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3dfc01a070 con 0x7f3e04157ca0 2024-09-17T04:49:53.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f3deb7fe640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3df0073200 0x7f3df0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f3deb7fe640 1 -- 172.21.15.150:0/808188963 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f3dfc093c30 con 0x7f3e04157ca0 2024-09-17T04:49:53.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f82c2a74640 1 -- start start 2024-09-17T04:49:53.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f3e02ffd640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3df0073200 0x7f3df0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f3e02ffd640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3df0073200 0x7f3df0075710 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3df40ad000 tx=0x7f3df401aa80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f82c2a74640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82b400a850 0x7f82b4043d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f82c2a74640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82b40463b0 0x7f82b40e9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f82c1a72640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82b400a850 0x7f82b4043d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f82c1271640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82b40463b0 0x7f82b40e9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f82c1271640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82b40463b0 0x7f82b40e9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33354/0 (socket says 172.21.15.150:33354) 2024-09-17T04:49:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f82c1a72640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82b400a850 0x7f82b4043d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:47448/0 (socket says 172.21.15.150:47448) 2024-09-17T04:49:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f82c1271640 1 -- 172.21.15.150:0/2123214531 learned_addr learned my addr 172.21.15.150:0/2123214531 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.412+0000 7f82c2a74640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82b40fb370 0x7f82b402e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.416+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f82b40fbcc0 con 0x7f82b40463b0 2024-09-17T04:49:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.416+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f82b404a430 con 0x7f82b40fb370 2024-09-17T04:49:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.416+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f82b4046970 con 0x7f82b400a850 2024-09-17T04:49:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.416+0000 7f82c2273640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82b40fb370 0x7f82b402e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.416+0000 7f82c1271640 1 -- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82b400a850 msgr2=0x7f82b4043d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.416+0000 7f82c1271640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82b400a850 0x7f82b4043d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.416+0000 7f82c1271640 1 -- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82b40fb370 msgr2=0x7f82b402e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.416+0000 7f82c1271640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82b40fb370 0x7f82b402e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.416+0000 7f82c1271640 1 -- 172.21.15.150:0/2123214531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b402f3c0 con 0x7f82b40463b0 2024-09-17T04:49:53.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.416+0000 7f82c1271640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82b40463b0 0x7f82b40e9bf0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f82b800d9b0 tx=0x7f82b800de70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.420+0000 7f3deb7fe640 1 -- 172.21.15.150:0/808188963 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3dfc05e2a0 con 0x7f3e04157ca0 2024-09-17T04:49:53.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.420+0000 7f82b27fc640 1 -- 172.21.15.150:0/2123214531 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b8017070 con 0x7f82b40463b0 2024-09-17T04:49:53.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.420+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82b4042680 con 0x7f82b40463b0 2024-09-17T04:49:53.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.420+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82b40ea130 con 0x7f82b40463b0 2024-09-17T04:49:53.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.424+0000 7f82b27fc640 1 -- 172.21.15.150:0/2123214531 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82b8004310 con 0x7f82b40463b0 2024-09-17T04:49:53.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.424+0000 7f82b27fc640 1 -- 172.21.15.150:0/2123214531 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b8009c30 con 0x7f82b40463b0 2024-09-17T04:49:53.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.424+0000 7f82b27fc640 1 -- 172.21.15.150:0/2123214531 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f82b8005020 con 0x7f82b40463b0 2024-09-17T04:49:53.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.424+0000 7f82b27fc640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82a0073300 0x7f82a0075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:53.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.428+0000 7f82b27fc640 1 -- 172.21.15.150:0/2123214531 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f82b8094aa0 con 0x7f82b40463b0 2024-09-17T04:49:53.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.428+0000 7f82c1a72640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82a0073300 0x7f82a0075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:53.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.428+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82a40052a0 con 0x7f82b40463b0 2024-09-17T04:49:53.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.436+0000 7f82b27fc640 1 -- 172.21.15.150:0/2123214531 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f82b8020020 con 0x7f82b40463b0 2024-09-17T04:49:53.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.436+0000 7f82c1a72640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82a0073300 0x7f82a0075810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f82ac007fd0 tx=0x7f82ac009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:53.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.648+0000 7f6215ffb640 1 -- 172.21.15.150:0/378858577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f6208005860 con 0x7f6230156c50 2024-09-17T04:49:53.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.652+0000 7f6217fff640 1 -- 172.21.15.150:0/378858577 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f622005e7c0 con 0x7f6230156c50 2024-09-17T04:49:53.655 INFO:teuthology.orchestra.run.smithi150.stdout:64424509449 2024-09-17T04:49:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.652+0000 7f62360d5640 1 -- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f621c073280 msgr2=0x7f621c075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.652+0000 7f62360d5640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f621c073280 0x7f621c075790 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f622801d550 tx=0x7f622801b550 comp rx=0 tx=0).stop 2024-09-17T04:49:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.652+0000 7f62360d5640 1 -- 172.21.15.150:0/378858577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6230156c50 msgr2=0x7f6230152630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.652+0000 7f62360d5640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6230156c50 0x7f6230152630 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f62300a7060 tx=0x7f622000ead0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.656+0000 7f62360d5640 1 -- 172.21.15.150:0/378858577 shutdown_connections 2024-09-17T04:49:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.656+0000 7f62360d5640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f621c073280 0x7f621c075790 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.656+0000 7f62360d5640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6230156c50 0x7f6230152630 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.656+0000 7f62360d5640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6230156840 0x7f623014f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.656+0000 7f62360d5640 1 --2- 172.21.15.150:0/378858577 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6230074f70 0x7f623014c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.656+0000 7f62360d5640 1 -- 172.21.15.150:0/378858577 >> 172.21.15.150:0/378858577 conn(0x7f62300c4e30 msgr2=0x7f62300a7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.656+0000 7f62360d5640 1 -- 172.21.15.150:0/378858577 shutdown_connections 2024-09-17T04:49:53.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.656+0000 7f62360d5640 1 -- 172.21.15.150:0/378858577 wait complete. 2024-09-17T04:49:53.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.672+0000 7fa3fa5b6640 1 -- 172.21.15.150:0/4037602783 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fa3f40c20d0 con 0x7fa3f4071b20 2024-09-17T04:49:53.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.672+0000 7fa3f0ff9640 1 -- 172.21.15.150:0/4037602783 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa3e80643a0 con 0x7fa3f4071b20 2024-09-17T04:49:53.677 INFO:teuthology.orchestra.run.smithi150.stdout:64424509450 2024-09-17T04:49:53.684 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.2 2024-09-17T04:49:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.680+0000 7fa3da7fc640 1 -- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa3dc073350 msgr2=0x7fa3dc075860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.680+0000 7fa3da7fc640 1 --2- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa3dc073350 0x7fa3dc075860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40046c0 tx=0x7fa3e4007ea0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.680+0000 7fa3da7fc640 1 -- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3f4071b20 msgr2=0x7fa3f41f3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.680+0000 7fa3da7fc640 1 --2- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3f4071b20 0x7fa3f41f3400 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800e6f0 tx=0x7fa3e800ebb0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.684+0000 7fa3da7fc640 1 -- 172.21.15.150:0/4037602783 shutdown_connections 2024-09-17T04:49:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.684+0000 7fa3da7fc640 1 --2- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa3dc073350 0x7fa3dc075860 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.684+0000 7fa3da7fc640 1 --2- 172.21.15.150:0/4037602783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3f40a72a0 0x7fa3f40ad2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.684+0000 7fa3da7fc640 1 --2- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa3f4073540 0x7fa3f41536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.684+0000 7fa3da7fc640 1 --2- 172.21.15.150:0/4037602783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3f4071b20 0x7fa3f41f3400 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.684+0000 7fa3da7fc640 1 -- 172.21.15.150:0/4037602783 >> 172.21.15.150:0/4037602783 conn(0x7fa3f40c4e30 msgr2=0x7fa3f406f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.684+0000 7fa3da7fc640 1 -- 172.21.15.150:0/4037602783 shutdown_connections 2024-09-17T04:49:53.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.684+0000 7fa3da7fc640 1 -- 172.21.15.150:0/4037602783 wait complete. 2024-09-17T04:49:53.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.684+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f9c0c04a720 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.688+0000 7f9c09ffb640 1 -- 172.21.15.150:0/1056047346 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9c10010040 con 0x7f9c0c02d5f0 2024-09-17T04:49:53.688 INFO:teuthology.orchestra.run.smithi150.stdout:64424509449 2024-09-17T04:49:53.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.688+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9bf8073200 msgr2=0x7f9bf8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.688+0000 7f9c1b67c640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9bf8073200 0x7f9bf8075710 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9c040b0000 tx=0x7f9c0401a640 comp rx=0 tx=0).stop 2024-09-17T04:49:53.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.688+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c0c02d5f0 msgr2=0x7f9c0c044070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.688+0000 7f9c1b67c640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c0c02d5f0 0x7f9c0c044070 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000e760 tx=0x7f9c1000ec20 comp rx=0 tx=0).stop 2024-09-17T04:49:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.692+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 shutdown_connections 2024-09-17T04:49:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.692+0000 7f9c1b67c640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9bf8073200 0x7f9bf8075710 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.692+0000 7f9c1b67c640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c0c045d70 0x7f9c0c1726b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.692+0000 7f9c1b67c640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c0c030cc0 0x7f9c0c03f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.692+0000 7f9c1b67c640 1 --2- 172.21.15.150:0/1056047346 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c0c02d5f0 0x7f9c0c044070 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.692+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 >> 172.21.15.150:0/1056047346 conn(0x7f9c0c048630 msgr2=0x7f9c0c0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.692+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 shutdown_connections 2024-09-17T04:49:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.692+0000 7f9c1b67c640 1 -- 172.21.15.150:0/1056047346 wait complete. 2024-09-17T04:49:53.718 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509450 for osd.1 2024-09-17T04:49:53.720 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.3 2024-09-17T04:49:53.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.720+0000 7fa5377fe640 1 -- 172.21.15.150:0/955985010 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa530005860 con 0x7fa5540c3900 2024-09-17T04:49:53.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa549ffb640 1 -- 172.21.15.150:0/955985010 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa5500f8670 con 0x7fa5540c3900 2024-09-17T04:49:53.725 INFO:teuthology.orchestra.run.smithi150.stdout:64424509450 2024-09-17T04:49:53.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa5377fe640 1 -- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa538073230 msgr2=0x7fa538075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa5377fe640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa538073230 0x7fa538075740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa554150290 tx=0x7fa544009060 comp rx=0 tx=0).stop 2024-09-17T04:49:53.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa5377fe640 1 -- 172.21.15.150:0/955985010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5540c3900 msgr2=0x7fa5540bf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa5377fe640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5540c3900 0x7fa5540bf720 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa55001c710 tx=0x7fa550005e00 comp rx=0 tx=0).stop 2024-09-17T04:49:53.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa5377fe640 1 -- 172.21.15.150:0/955985010 shutdown_connections 2024-09-17T04:49:53.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa5377fe640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa538073230 0x7fa538075740 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa5377fe640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5540c3d10 0x7fa5540be3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa5377fe640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5540c4250 0x7fa5540c0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa5377fe640 1 --2- 172.21.15.150:0/955985010 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5540c3900 0x7fa5540bf720 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa5377fe640 1 -- 172.21.15.150:0/955985010 >> 172.21.15.150:0/955985010 conn(0x7fa55406e360 msgr2=0x7fa554153890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa5377fe640 1 -- 172.21.15.150:0/955985010 shutdown_connections 2024-09-17T04:49:53.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.724+0000 7fa5377fe640 1 -- 172.21.15.150:0/955985010 wait complete. 2024-09-17T04:49:53.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.728+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3174048690 con 0x7f31740db9b0 2024-09-17T04:49:53.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.728+0000 7f31797fa640 1 -- 172.21.15.150:0/2022370438 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f317c0da210 con 0x7f31740db9b0 2024-09-17T04:49:53.730 INFO:teuthology.orchestra.run.smithi150.stdout:64424509449 2024-09-17T04:49:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.732+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31600731e0 msgr2=0x7f31600756f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.732+0000 7f31818a8640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31600731e0 0x7f31600756f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3170001040 tx=0x7f3170009040 comp rx=0 tx=0).stop 2024-09-17T04:49:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.732+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31740db9b0 msgr2=0x7f31740d4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.732+0000 7f31818a8640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31740db9b0 0x7f31740d4b60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f317c076440 tx=0x7f317c076ea0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.732+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 shutdown_connections 2024-09-17T04:49:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.732+0000 7f31818a8640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31600731e0 0x7f31600756f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.732+0000 7f31818a8640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31740db9b0 0x7f31740d4b60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.732+0000 7f31818a8640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f317404c390 0x7f3174180900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.732+0000 7f31818a8640 1 --2- 172.21.15.150:0/2022370438 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31740489b0 0x7f3174009070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.732+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 >> 172.21.15.150:0/2022370438 conn(0x7f317404d620 msgr2=0x7f31740d6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.732+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 shutdown_connections 2024-09-17T04:49:53.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.732+0000 7f31818a8640 1 -- 172.21.15.150:0/2022370438 wait complete. 2024-09-17T04:49:53.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f82a4005590 con 0x7f82b40463b0 2024-09-17T04:49:53.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82b27fc640 1 -- 172.21.15.150:0/2123214531 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f82b805f1b0 con 0x7f82b40463b0 2024-09-17T04:49:53.737 INFO:teuthology.orchestra.run.smithi150.stdout:64424509449 2024-09-17T04:49:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82a0073300 msgr2=0x7f82a0075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82a0073300 0x7f82a0075810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f82ac007fd0 tx=0x7f82ac009040 comp rx=0 tx=0).stop 2024-09-17T04:49:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82b40463b0 msgr2=0x7f82b40e9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82b40463b0 0x7f82b40e9bf0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f82b800d9b0 tx=0x7f82b800de70 comp rx=0 tx=0).stop 2024-09-17T04:49:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 shutdown_connections 2024-09-17T04:49:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82a0073300 0x7f82a0075810 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82b40fb370 0x7f82b402e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82b40463b0 0x7f82b40e9bf0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 --2- 172.21.15.150:0/2123214531 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82b400a850 0x7f82b4043d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 >> 172.21.15.150:0/2123214531 conn(0x7f82b4048270 msgr2=0x7f82b4005270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 shutdown_connections 2024-09-17T04:49:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.736+0000 7f82c2a74640 1 -- 172.21.15.150:0/2123214531 wait complete. 2024-09-17T04:49:53.759 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509450 for osd.0 2024-09-17T04:49:53.760 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.7 2024-09-17T04:49:53.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.4 2024-09-17T04:49:53.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.760+0000 7fd98f59e640 1 -- 172.21.15.150:0/755976639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd98800bf80 con 0x7fd9880478a0 2024-09-17T04:49:53.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.760+0000 7fd97effd640 1 -- 172.21.15.150:0/755976639 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd9900dd540 con 0x7fd9880478a0 2024-09-17T04:49:53.763 INFO:teuthology.orchestra.run.smithi150.stdout:64424509449 2024-09-17T04:49:53.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.764+0000 7fd98f59e640 1 -- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd970073200 msgr2=0x7fd970075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.764+0000 7fd98f59e640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd970073200 0x7fd970075710 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd9840ae000 tx=0x7fd98401b060 comp rx=0 tx=0).stop 2024-09-17T04:49:53.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.764+0000 7fd98f59e640 1 -- 172.21.15.150:0/755976639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9880478a0 msgr2=0x7fd9880c9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.764+0000 7fd98f59e640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9880478a0 0x7fd9880c9990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd99006b590 tx=0x7fd990076ea0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.764+0000 7fd98f59e640 1 -- 172.21.15.150:0/755976639 shutdown_connections 2024-09-17T04:49:53.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.764+0000 7fd98f59e640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd970073200 0x7fd970075710 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.764+0000 7fd98f59e640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9880478a0 0x7fd9880c9990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.764+0000 7fd98f59e640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd98802ecf0 0x7fd988044000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.764+0000 7fd98f59e640 1 --2- 172.21.15.150:0/755976639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd98802e8e0 0x7fd98800b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.764+0000 7fd98f59e640 1 -- 172.21.15.150:0/755976639 >> 172.21.15.150:0/755976639 conn(0x7fd988049590 msgr2=0x7fd9880d0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.764+0000 7fd98f59e640 1 -- 172.21.15.150:0/755976639 shutdown_connections 2024-09-17T04:49:53.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.764+0000 7fd98f59e640 1 -- 172.21.15.150:0/755976639 wait complete. 2024-09-17T04:49:53.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.768+0000 7f3e09573640 1 -- 172.21.15.150:0/808188963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f3de4005860 con 0x7f3e04157ca0 2024-09-17T04:49:53.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.768+0000 7f3deb7fe640 1 -- 172.21.15.150:0/808188963 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3dfc062cd0 con 0x7f3e04157ca0 2024-09-17T04:49:53.769 INFO:teuthology.orchestra.run.smithi150.stdout:60129542153 2024-09-17T04:49:53.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.772+0000 7f3e09573640 1 -- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3df0073200 msgr2=0x7f3df0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.772+0000 7f3e09573640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3df0073200 0x7f3df0075710 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3df40ad000 tx=0x7f3df401aa80 comp rx=0 tx=0).stop 2024-09-17T04:49:53.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.772+0000 7f3e09573640 1 -- 172.21.15.150:0/808188963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e04157ca0 msgr2=0x7f3e0408b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:53.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.772+0000 7f3e09573640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e04157ca0 0x7f3e0408b490 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc00c8c0 tx=0x7f3dfc00cd80 comp rx=0 tx=0).stop 2024-09-17T04:49:53.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.772+0000 7f3e09573640 1 -- 172.21.15.150:0/808188963 shutdown_connections 2024-09-17T04:49:53.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.772+0000 7f3e09573640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3df0073200 0x7f3df0075710 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.772+0000 7f3e09573640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e04157ca0 0x7f3e0408b490 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.772+0000 7f3e09573640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3e04151e10 0x7f3e04145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.772+0000 7f3e09573640 1 --2- 172.21.15.150:0/808188963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e040ad7f0 0x7f3e04072ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:53.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.772+0000 7f3e09573640 1 -- 172.21.15.150:0/808188963 >> 172.21.15.150:0/808188963 conn(0x7f3e040c9de0 msgr2=0x7f3e040a9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:53.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.772+0000 7f3e09573640 1 -- 172.21.15.150:0/808188963 shutdown_connections 2024-09-17T04:49:53.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:53.772+0000 7f3e09573640 1 -- 172.21.15.150:0/808188963 wait complete. 2024-09-17T04:49:53.788 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.5 2024-09-17T04:49:53.792 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.6 2024-09-17T04:49:54.685 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T04:49:54.719 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T04:49:54.722 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T04:49:54.760 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T04:49:54.761 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-17T04:49:54.763 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T04:49:54.789 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T04:49:54.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-17T04:49:54.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.908+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2931438001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1b4003a40 msgr2=0x7ff1b4003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:54.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.908+0000 7ff1d1aa2640 1 --2- 172.21.15.150:0/2931438001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1b4003a40 0x7ff1b4003eb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400c640 tx=0x7ff1c401d2e0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.908+0000 7ff1cbfff640 1 -- 172.21.15.150:0/2931438001 reap_dead start 2024-09-17T04:49:54.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.908+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2931438001 shutdown_connections 2024-09-17T04:49:54.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.908+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2931438001 >> 172.21.15.150:0/2931438001 conn(0x7ff1cc0c7f30 msgr2=0x7ff1cc0c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:54.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.908+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2931438001 shutdown_connections 2024-09-17T04:49:54.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.908+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2931438001 wait complete. 2024-09-17T04:49:54.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.912+0000 7ff1d1aa2640 1 Processor -- start 2024-09-17T04:49:54.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.912+0000 7ff1d1aa2640 1 -- start start 2024-09-17T04:49:54.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1d1aa2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff1cc155b10 0x7ff1cc0c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1d1aa2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc149c60 0x7ff1cc14a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1d1aa2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1cc0ad420 0x7ff1cc0ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1d1aa2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc075350 con 0x7ff1cc149c60 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1d1aa2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc0751d0 con 0x7ff1cc0ad420 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1d1aa2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff1cc153390 con 0x7ff1cc155b10 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1caffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc149c60 0x7ff1cc14a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1caffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc149c60 0x7ff1cc14a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33362/0 (socket says 172.21.15.150:33362) 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1caffd640 1 -- 172.21.15.150:0/2090689525 learned_addr learned my addr 172.21.15.150:0/2090689525 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1cb7fe640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff1cc155b10 0x7ff1cc0c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1cbfff640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1cc0ad420 0x7ff1cc0ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1caffd640 1 -- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff1cc155b10 msgr2=0x7ff1cc0c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1caffd640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff1cc155b10 0x7ff1cc0c58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1caffd640 1 -- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1cc0ad420 msgr2=0x7ff1cc0ad830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:54.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1caffd640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1cc0ad420 0x7ff1cc0ad830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1caffd640 1 -- 172.21.15.150:0/2090689525 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1cc15ab30 con 0x7ff1cc149c60 2024-09-17T04:49:54.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.916+0000 7ff1caffd640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc149c60 0x7ff1cc14a070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc004ee0 tx=0x7ff1bc001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:54.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.920+0000 7ff1b3fff640 1 -- 172.21.15.150:0/2090689525 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc015020 con 0x7ff1cc149c60 2024-09-17T04:49:54.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.920+0000 7ff1b3fff640 1 -- 172.21.15.150:0/2090689525 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1bc011960 con 0x7ff1cc149c60 2024-09-17T04:49:54.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.920+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2090689525 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1cc0c63a0 con 0x7ff1cc149c60 2024-09-17T04:49:54.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.920+0000 7ff1b3fff640 1 -- 172.21.15.150:0/2090689525 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc011d90 con 0x7ff1cc149c60 2024-09-17T04:49:54.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.920+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2090689525 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1cc1555f0 con 0x7ff1cc149c60 2024-09-17T04:49:54.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.920+0000 7ff1b1ffb640 1 -- 172.21.15.150:0/2090689525 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1b80052a0 con 0x7ff1cc149c60 2024-09-17T04:49:54.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.924+0000 7ff1b3fff640 1 -- 172.21.15.150:0/2090689525 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff1bc018020 con 0x7ff1cc149c60 2024-09-17T04:49:54.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.924+0000 7ff1b3fff640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff1b4075120 0x7ff1b4077630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:54.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.924+0000 7ff1b3fff640 1 -- 172.21.15.150:0/2090689525 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7ff1bc094db0 con 0x7ff1cc149c60 2024-09-17T04:49:54.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.924+0000 7ff1cb7fe640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff1b4075120 0x7ff1b4077630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:54.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.924+0000 7ff1cb7fe640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff1b4075120 0x7ff1b4077630 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40ac000 tx=0x7ff1c401a120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:54.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.936+0000 7f08a446d640 1 -- 172.21.15.150:0/1543002923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f089c074140 msgr2=0x7f089c153ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:54.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.936+0000 7f08a446d640 1 --2- 172.21.15.150:0/1543002923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f089c074140 0x7f089c153ec0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f089c0c4370 tx=0x7f088c01c6d0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.936+0000 7ff1b3fff640 1 -- 172.21.15.150:0/2090689525 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff1bc01f020 con 0x7ff1cc149c60 2024-09-17T04:49:54.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.936+0000 7f08a446d640 1 -- 172.21.15.150:0/1543002923 shutdown_connections 2024-09-17T04:49:54.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.936+0000 7f08a446d640 1 --2- 172.21.15.150:0/1543002923 >> v2:172.21.15.176:3300/0 conn(0x7f089c0715b0 0x7f089c0719c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.936+0000 7f08a446d640 1 --2- 172.21.15.150:0/1543002923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f089c074140 0x7f089c153ec0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.936+0000 7f08a446d640 1 -- 172.21.15.150:0/1543002923 >> 172.21.15.150:0/1543002923 conn(0x7f089c0c4e30 msgr2=0x7f089c06ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:54.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.936+0000 7f08a446d640 1 -- 172.21.15.150:0/1543002923 shutdown_connections 2024-09-17T04:49:54.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.940+0000 7f85f5583640 1 -- 172.21.15.150:0/2211682451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85f00707e0 msgr2=0x7f85f0070bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:54.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.940+0000 7f85f5583640 1 --2- 172.21.15.150:0/2211682451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85f00707e0 0x7f85f0070bf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f85f00ade70 tx=0x7f85e801cbc0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.940+0000 7f85f5583640 1 -- 172.21.15.150:0/2211682451 shutdown_connections 2024-09-17T04:49:54.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.940+0000 7f85f5583640 1 --2- 172.21.15.150:0/2211682451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85f00707e0 0x7f85f0070bf0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.940+0000 7f85f5583640 1 --2- 172.21.15.150:0/2211682451 >> v2:172.21.15.150:3300/0 conn(0x7f85f014f490 0x7f85f014f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.940+0000 7f85f5583640 1 -- 172.21.15.150:0/2211682451 >> 172.21.15.150:0/2211682451 conn(0x7f85f00c7f30 msgr2=0x7f85f00c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:54.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.940+0000 7f85f5583640 1 -- 172.21.15.150:0/2211682451 shutdown_connections 2024-09-17T04:49:54.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.944+0000 7f85f5583640 1 -- 172.21.15.150:0/2211682451 wait complete. 2024-09-17T04:49:54.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.944+0000 7f85f5583640 1 Processor -- start 2024-09-17T04:49:54.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.944+0000 7f85f5583640 1 -- start start 2024-09-17T04:49:54.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f08a446d640 1 -- 172.21.15.150:0/1543002923 wait complete. 2024-09-17T04:49:54.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85f5583640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85f00707e0 0x7f85f00ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:54.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85f5583640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85f00c2500 0x7f85f01554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:54.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85f5583640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85f014f490 0x7f85f0142ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:54.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85f5583640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f85f00751d0 con 0x7f85f00c2500 2024-09-17T04:49:54.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85f5583640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f85f01502e0 con 0x7f85f014f490 2024-09-17T04:49:54.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85f5583640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f85f0075350 con 0x7f85f00707e0 2024-09-17T04:49:54.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85ee7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85f00c2500 0x7f85f01554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:54.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85ee7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85f00c2500 0x7f85f01554d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33368/0 (socket says 172.21.15.150:33368) 2024-09-17T04:49:54.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85ee7fc640 1 -- 172.21.15.150:0/960620669 learned_addr learned my addr 172.21.15.150:0/960620669 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:54.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85eeffd640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85f00707e0 0x7f85f00ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:54.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85ef7fe640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85f014f490 0x7f85f0142ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:54.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85ee7fc640 1 -- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85f00707e0 msgr2=0x7f85f00ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:54.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85ee7fc640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85f00707e0 0x7f85f00ad940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85ee7fc640 1 -- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85f014f490 msgr2=0x7f85f0142ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:54.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85ee7fc640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85f014f490 0x7f85f0142ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85ee7fc640 1 -- 172.21.15.150:0/960620669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f0147fe0 con 0x7f85f00c2500 2024-09-17T04:49:54.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.948+0000 7f85ee7fc640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85f00c2500 0x7f85f01554d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009b10 tx=0x7f85e8009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:54.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.952+0000 7f85d77fe640 1 -- 172.21.15.150:0/960620669 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e80bb070 con 0x7f85f00c2500 2024-09-17T04:49:54.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.952+0000 7f85f5583640 1 -- 172.21.15.150:0/960620669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f0145180 con 0x7f85f00c2500 2024-09-17T04:49:54.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.952+0000 7f85f5583640 1 -- 172.21.15.150:0/960620669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f00ade80 con 0x7f85f00c2500 2024-09-17T04:49:54.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.952+0000 7f85d77fe640 1 -- 172.21.15.150:0/960620669 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85e80a5b60 con 0x7f85f00c2500 2024-09-17T04:49:54.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.952+0000 7f85d77fe640 1 -- 172.21.15.150:0/960620669 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e80c5060 con 0x7f85f00c2500 2024-09-17T04:49:54.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.952+0000 7f08a446d640 1 Processor -- start 2024-09-17T04:49:54.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.952+0000 7f08a446d640 1 -- start start 2024-09-17T04:49:54.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.952+0000 7f85d77fe640 1 -- 172.21.15.150:0/960620669 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f85e80be070 con 0x7f85f00c2500 2024-09-17T04:49:54.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.956+0000 7f85d77fe640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85dc073230 0x7f85dc075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:54.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.956+0000 7f85d77fe640 1 -- 172.21.15.150:0/960620669 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f85e812e1d0 con 0x7f85f00c2500 2024-09-17T04:49:54.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.956+0000 7f85eeffd640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85dc073230 0x7f85dc075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:54.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.956+0000 7f85eeffd640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85dc073230 0x7f85dc075740 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f85e00046c0 tx=0x7f85e0007ea0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:54.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.960+0000 7f85f5583640 1 -- 172.21.15.150:0/960620669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85d00052a0 con 0x7f85f00c2500 2024-09-17T04:49:54.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a446d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f089c0715b0 0x7f089c0ac3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:54.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a446d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f089c074140 0x7f089c1ec7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:54.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a446d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f089c153450 0x7f089c149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:54.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a446d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f089c073e50 con 0x7f089c0715b0 2024-09-17T04:49:54.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a446d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f089c151000 con 0x7f089c153450 2024-09-17T04:49:54.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a446d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f089c06dc70 con 0x7f089c074140 2024-09-17T04:49:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a2209640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f089c0715b0 0x7f089c0ac3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a2209640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f089c0715b0 0x7f089c0ac3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33384/0 (socket says 172.21.15.150:33384) 2024-09-17T04:49:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a2209640 1 -- 172.21.15.150:0/1056681768 learned_addr learned my addr 172.21.15.150:0/1056681768 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a2a0a640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f089c153450 0x7f089c149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a1a08640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f089c074140 0x7f089c1ec7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a2209640 1 -- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f089c074140 msgr2=0x7f089c1ec7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a2209640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f089c074140 0x7f089c1ec7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a2209640 1 -- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f089c153450 msgr2=0x7f089c149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a2209640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f089c153450 0x7f089c149e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a2209640 1 -- 172.21.15.150:0/1056681768 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089c14d940 con 0x7f089c0715b0 2024-09-17T04:49:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a2a0a640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f089c153450 0x7f089c149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:49:54.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.964+0000 7f08a2209640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f089c0715b0 0x7f089c0ac3d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f088c01a300 tx=0x7f088c0a5de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.968+0000 7f85d77fe640 1 -- 172.21.15.150:0/960620669 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f85f0147fe0 con 0x7f85f00c2500 2024-09-17T04:49:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.968+0000 7f0892ffd640 1 -- 172.21.15.150:0/1056681768 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c0b2040 con 0x7f089c0715b0 2024-09-17T04:49:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.968+0000 7f08a446d640 1 -- 172.21.15.150:0/1056681768 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f089c0c1b70 con 0x7f089c0715b0 2024-09-17T04:49:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.968+0000 7f08a446d640 1 -- 172.21.15.150:0/1056681768 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f089c0ace40 con 0x7f089c0715b0 2024-09-17T04:49:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.968+0000 7f0892ffd640 1 -- 172.21.15.150:0/1056681768 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f088c0043e0 con 0x7f089c0715b0 2024-09-17T04:49:54.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.972+0000 7f0892ffd640 1 -- 172.21.15.150:0/1056681768 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c005270 con 0x7f089c0715b0 2024-09-17T04:49:54.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.972+0000 7f08a446d640 1 -- 172.21.15.150:0/1056681768 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08840052a0 con 0x7f089c0715b0 2024-09-17T04:49:54.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.972+0000 7f0892ffd640 1 -- 172.21.15.150:0/1056681768 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f088c005410 con 0x7f089c0715b0 2024-09-17T04:49:54.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.972+0000 7f0892ffd640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0880073230 0x7f0880075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:54.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.976+0000 7f0892ffd640 1 -- 172.21.15.150:0/1056681768 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f088c130250 con 0x7f089c0715b0 2024-09-17T04:49:54.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.976+0000 7f08a1a08640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0880073230 0x7f0880075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:54.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.976+0000 7f08a1a08640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0880073230 0x7f0880075740 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f089c15d0d0 tx=0x7f0898009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:54.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:54.984+0000 7f0892ffd640 1 -- 172.21.15.150:0/1056681768 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f089c14d940 con 0x7f089c0715b0 2024-09-17T04:49:55.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.000+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/3504139209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00da170 msgr2=0x7fe0c00d4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.000+0000 7fe0cf0d0640 1 --2- 172.21.15.150:0/3504139209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00da170 0x7fe0c00d4220 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe0c000d000 tx=0x7fe0c80996d0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.000+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/3504139209 shutdown_connections 2024-09-17T04:49:55.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.000+0000 7fe0cf0d0640 1 --2- 172.21.15.150:0/3504139209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00da170 0x7fe0c00d4220 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.000+0000 7fe0cf0d0640 1 --2- 172.21.15.150:0/3504139209 >> v2:172.21.15.176:3300/0 conn(0x7fe0c0031110 0x7fe0c000a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.000+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/3504139209 >> 172.21.15.150:0/3504139209 conn(0x7fe0c0048270 msgr2=0x7fe0c0048680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.000+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/3504139209 shutdown_connections 2024-09-17T04:49:55.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.000+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/3504139209 wait complete. 2024-09-17T04:49:55.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cf0d0640 1 Processor -- start 2024-09-17T04:49:55.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cf0d0640 1 -- start start 2024-09-17T04:49:55.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cf0d0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0c000ad90 0x7fe0c00e7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cf0d0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0c0031110 0x7fe0c0178650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cf0d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00da170 0x7fe0c00cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cf0d0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c00ebb20 con 0x7fe0c00da170 2024-09-17T04:49:55.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cf0d0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c0049e10 con 0x7fe0c000ad90 2024-09-17T04:49:55.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cf0d0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe0c00ebca0 con 0x7fe0c0031110 2024-09-17T04:49:55.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cce6c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0c000ad90 0x7fe0c00e7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cce6c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0c000ad90 0x7fe0c00e7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:44992/0 (socket says 172.21.15.150:44992) 2024-09-17T04:49:55.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cce6c640 1 -- 172.21.15.150:0/2169543868 learned_addr learned my addr 172.21.15.150:0/2169543868 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:55.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cd66d640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00da170 0x7fe0c00cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cce6c640 1 -- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0c0031110 msgr2=0x7fe0c0178650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cce6c640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0c0031110 0x7fe0c0178650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cce6c640 1 -- 172.21.15.150:0/2169543868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00da170 msgr2=0x7fe0c00cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cce6c640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00da170 0x7fe0c00cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cce6c640 1 -- 172.21.15.150:0/2169543868 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c00cd220 con 0x7fe0c000ad90 2024-09-17T04:49:55.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cce6c640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0c000ad90 0x7fe0c00e7ca0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0058d0 tx=0x7fe0bc00dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:55.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0c57fa640 1 -- 172.21.15.150:0/2169543868 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc017070 con 0x7fe0c000ad90 2024-09-17T04:49:55.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/2169543868 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c002d680 con 0x7fe0c000ad90 2024-09-17T04:49:55.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.004+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/2169543868 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c002b020 con 0x7fe0c000ad90 2024-09-17T04:49:55.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7fe0c57fa640 1 -- 172.21.15.150:0/2169543868 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0bc009d20 con 0x7fe0c000ad90 2024-09-17T04:49:55.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7fe0c57fa640 1 -- 172.21.15.150:0/2169543868 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc0106c0 con 0x7fe0c000ad90 2024-09-17T04:49:55.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7f54be14b640 1 -- 172.21.15.150:0/3814236711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54b8071b20 msgr2=0x7f54b8073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7f54be14b640 1 --2- 172.21.15.150:0/3814236711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54b8071b20 0x7f54b8073540 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f54ac009840 tx=0x7f54ac01c790 comp rx=0 tx=0).stop 2024-09-17T04:49:55.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7f54be14b640 1 -- 172.21.15.150:0/3814236711 shutdown_connections 2024-09-17T04:49:55.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7f54be14b640 1 --2- 172.21.15.150:0/3814236711 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54b8071b20 0x7f54b8073540 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7f54be14b640 1 --2- 172.21.15.150:0/3814236711 >> v2:172.21.15.176:3300/0 conn(0x7f54b80a72a0 0x7f54b8074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7f54be14b640 1 -- 172.21.15.150:0/3814236711 >> 172.21.15.150:0/3814236711 conn(0x7f54b80c4e30 msgr2=0x7f54b806ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7f54be14b640 1 -- 172.21.15.150:0/3814236711 shutdown_connections 2024-09-17T04:49:55.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7f54be14b640 1 -- 172.21.15.150:0/3814236711 wait complete. 2024-09-17T04:49:55.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/2169543868 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c00e9110 con 0x7fe0c000ad90 2024-09-17T04:49:55.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7f54be14b640 1 Processor -- start 2024-09-17T04:49:55.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7fe0c57fa640 1 -- 172.21.15.150:0/2169543868 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe0bc01a070 con 0x7fe0c000ad90 2024-09-17T04:49:55.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7fe0c57fa640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe0ac073200 0x7fe0ac075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7fe0c57fa640 1 -- 172.21.15.150:0/2169543868 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fe0bc094a30 con 0x7fe0c000ad90 2024-09-17T04:49:55.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.008+0000 7f54be14b640 1 -- start start 2024-09-17T04:49:55.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54be14b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54b8071b20 0x7f54b8151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54be14b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54b80a72a0 0x7f54b8150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54be14b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54b8156840 0x7f54b81533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7fe0c7fff640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe0ac073200 0x7fe0ac075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54be14b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f54b8163d70 con 0x7f54b8156840 2024-09-17T04:49:55.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54be14b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f54b8163bf0 con 0x7f54b8071b20 2024-09-17T04:49:55.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54be14b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f54b8163ef0 con 0x7f54b80a72a0 2024-09-17T04:49:55.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54b77fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54b8071b20 0x7f54b8151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54b77fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54b8071b20 0x7f54b8151140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:44994/0 (socket says 172.21.15.150:44994) 2024-09-17T04:49:55.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54b77fe640 1 -- 172.21.15.150:0/1833078483 learned_addr learned my addr 172.21.15.150:0/1833078483 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:55.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54b7fff640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54b8156840 0x7f54b81533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54b6ffd640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54b80a72a0 0x7f54b8150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54b7fff640 1 -- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54b80a72a0 msgr2=0x7f54b8150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54b7fff640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54b80a72a0 0x7f54b8150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54b7fff640 1 -- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54b8071b20 msgr2=0x7f54b8151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54b7fff640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54b8071b20 0x7f54b8151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54b7fff640 1 -- 172.21.15.150:0/1833078483 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b81f4890 con 0x7f54b8156840 2024-09-17T04:49:55.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54b7fff640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54b8156840 0x7f54b81533c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f54a800c3c0 tx=0x7f54a800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:55.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f549ffff640 1 -- 172.21.15.150:0/1833078483 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8011790 con 0x7f54b8156840 2024-09-17T04:49:55.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54be14b640 1 -- 172.21.15.150:0/1833078483 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b80adbe0 con 0x7f54b8156840 2024-09-17T04:49:55.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f54be14b640 1 -- 172.21.15.150:0/1833078483 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b806d830 con 0x7f54b8156840 2024-09-17T04:49:55.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f549ffff640 1 -- 172.21.15.150:0/1833078483 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54a8017070 con 0x7f54b8156840 2024-09-17T04:49:55.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f38ac818640 1 -- 172.21.15.150:0/1711738158 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38a80c72a0 msgr2=0x7f38a8150740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7fe0c7fff640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe0ac073200 0x7fe0ac075710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0006d80 tx=0x7fe0c806cbf0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:55.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f38ac818640 1 --2- 172.21.15.150:0/1711738158 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38a80c72a0 0x7f38a8150740 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38a0009c80 tx=0x7f38a001cdc0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f38ac818640 1 -- 172.21.15.150:0/1711738158 shutdown_connections 2024-09-17T04:49:55.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f38ac818640 1 --2- 172.21.15.150:0/1711738158 >> v2:172.21.15.150:3300/0 conn(0x7f38a80c8540 0x7f38a80c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f38ac818640 1 --2- 172.21.15.150:0/1711738158 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38a80c72a0 0x7f38a8150740 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f38ac818640 1 -- 172.21.15.150:0/1711738158 >> 172.21.15.150:0/1711738158 conn(0x7f38a8074580 msgr2=0x7f38a8074990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f38ac818640 1 -- 172.21.15.150:0/1711738158 shutdown_connections 2024-09-17T04:49:55.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.012+0000 7f38ac818640 1 -- 172.21.15.150:0/1711738158 wait complete. 2024-09-17T04:49:55.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.016+0000 7f549ffff640 1 -- 172.21.15.150:0/1833078483 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8011ce0 con 0x7f54b8156840 2024-09-17T04:49:55.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.016+0000 7f549ffff640 1 -- 172.21.15.150:0/1833078483 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f54a801c460 con 0x7f54b8156840 2024-09-17T04:49:55.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.016+0000 7f549ffff640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f54a4073200 0x7f54a4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.016+0000 7f549ffff640 1 -- 172.21.15.150:0/1833078483 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f54a8098060 con 0x7f54b8156840 2024-09-17T04:49:55.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.016+0000 7f54b77fe640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f54a4073200 0x7f54a4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.016+0000 7f54be14b640 1 -- 172.21.15.150:0/1833078483 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54980052a0 con 0x7f54b8156840 2024-09-17T04:49:55.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.016+0000 7f38ac818640 1 Processor -- start 2024-09-17T04:49:55.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.016+0000 7f38ac818640 1 -- start start 2024-09-17T04:49:55.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38ac818640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38a80c72a0 0x7f38a814c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38ac818640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a80c8540 0x7f38a80cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38ac818640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38a80c8e90 0x7f38a80c21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38ac818640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f38a81633f0 con 0x7f38a80c8540 2024-09-17T04:49:55.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38ac818640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f38a80c3100 con 0x7f38a80c72a0 2024-09-17T04:49:55.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38ac818640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f38a8163570 con 0x7f38a80c8e90 2024-09-17T04:49:55.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38a659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a80c8540 0x7f38a80cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38a659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a80c8540 0x7f38a80cb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33428/0 (socket says 172.21.15.150:33428) 2024-09-17T04:49:55.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38a659c640 1 -- 172.21.15.150:0/2755184709 learned_addr learned my addr 172.21.15.150:0/2755184709 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:55.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38a6d9d640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38a80c72a0 0x7f38a814c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38a759e640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38a80c8e90 0x7f38a80c21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38a659c640 1 -- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38a80c8e90 msgr2=0x7f38a80c21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38a659c640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38a80c8e90 0x7f38a80c21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38a659c640 1 -- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38a80c72a0 msgr2=0x7f38a814c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38a659c640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38a80c72a0 0x7f38a814c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38a659c640 1 -- 172.21.15.150:0/2755184709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a814d660 con 0x7f38a80c8540 2024-09-17T04:49:55.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f38a659c640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a80c8540 0x7f38a80cb590 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f389c00c940 tx=0x7f389c00ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:55.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.020+0000 7f54b77fe640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f54a4073200 0x7f54a4075710 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009a00 tx=0x7f54b0009500 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:55.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.024+0000 7f549ffff640 1 -- 172.21.15.150:0/1833078483 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f54a805dde0 con 0x7f54b8156840 2024-09-17T04:49:55.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.024+0000 7f388f7fe640 1 -- 172.21.15.150:0/2755184709 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c004220 con 0x7f38a80c8540 2024-09-17T04:49:55.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.024+0000 7f38ac818640 1 -- 172.21.15.150:0/2755184709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a814d920 con 0x7f38a80c8540 2024-09-17T04:49:55.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.024+0000 7f38ac818640 1 -- 172.21.15.150:0/2755184709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a80cca50 con 0x7f38a80c8540 2024-09-17T04:49:55.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.024+0000 7f388f7fe640 1 -- 172.21.15.150:0/2755184709 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f389c0043c0 con 0x7f38a80c8540 2024-09-17T04:49:55.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.024+0000 7fe0c57fa640 1 -- 172.21.15.150:0/2169543868 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe0bc05f1d0 con 0x7fe0c000ad90 2024-09-17T04:49:55.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.028+0000 7f388f7fe640 1 -- 172.21.15.150:0/2755184709 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c01d060 con 0x7f38a80c8540 2024-09-17T04:49:55.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.028+0000 7f388f7fe640 1 -- 172.21.15.150:0/2755184709 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f389c007c90 con 0x7f38a80c8540 2024-09-17T04:49:55.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.028+0000 7f38ac818640 1 -- 172.21.15.150:0/2755184709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38880052a0 con 0x7f38a80c8540 2024-09-17T04:49:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.028+0000 7f388f7fe640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f38940731b0 0x7f38940756c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.028+0000 7f388f7fe640 1 -- 172.21.15.150:0/2755184709 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f389c094800 con 0x7f38a80c8540 2024-09-17T04:49:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.028+0000 7f38a6d9d640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f38940731b0 0x7f38940756c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.028+0000 7f38a6d9d640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f38940731b0 0x7f38940756c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f38a00af000 tx=0x7f38a001b600 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.040+0000 7fc80b59e640 1 -- 172.21.15.150:0/3894912293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc804046780 msgr2=0x7fc8040492f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.040+0000 7fc80b59e640 1 --2- 172.21.15.150:0/3894912293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc804046780 0x7fc8040492f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc800009840 tx=0x7fc80001c630 comp rx=0 tx=0).stop 2024-09-17T04:49:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.040+0000 7fc80b59e640 1 -- 172.21.15.150:0/3894912293 shutdown_connections 2024-09-17T04:49:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.040+0000 7fc80b59e640 1 --2- 172.21.15.150:0/3894912293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc804046780 0x7fc8040492f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.040+0000 7fc80b59e640 1 --2- 172.21.15.150:0/3894912293 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8040da170 0x7fc8040d4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.040+0000 7fc80b59e640 1 -- 172.21.15.150:0/3894912293 >> 172.21.15.150:0/3894912293 conn(0x7fc804048270 msgr2=0x7fc804048680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.040+0000 7fc80b59e640 1 -- 172.21.15.150:0/3894912293 shutdown_connections 2024-09-17T04:49:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.040+0000 7fc80b59e640 1 -- 172.21.15.150:0/3894912293 wait complete. 2024-09-17T04:49:55.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.040+0000 7fc80b59e640 1 Processor -- start 2024-09-17T04:49:55.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.044+0000 7f388f7fe640 1 -- 172.21.15.150:0/2755184709 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f389c014030 con 0x7f38a80c8540 2024-09-17T04:49:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.040+0000 7fc80b59e640 1 -- start start 2024-09-17T04:49:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.044+0000 7fc80b59e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc804031110 0x7fc80402ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.044+0000 7fc80b59e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc804046780 0x7fc80417b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.044+0000 7fc80b59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8040da170 0x7fc804044550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.044+0000 7fc80b59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc80400b0e0 con 0x7fc8040da170 2024-09-17T04:49:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.044+0000 7fc80b59e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc80400af60 con 0x7fc804031110 2024-09-17T04:49:55.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.044+0000 7fc80b59e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc8040eb310 con 0x7fc804046780 2024-09-17T04:49:55.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.048+0000 7fc80a59c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc804031110 0x7fc80402ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.048+0000 7fc809d9b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc804046780 0x7fc80417b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.048+0000 7fc809d9b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc804046780 0x7fc80417b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:47568/0 (socket says 172.21.15.150:47568) 2024-09-17T04:49:55.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.048+0000 7fc809d9b640 1 -- 172.21.15.150:0/3880169444 learned_addr learned my addr 172.21.15.150:0/3880169444 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:55.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.048+0000 7fa0641bf640 1 -- 172.21.15.150:0/1214047706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa05c071430 msgr2=0x7fa05c13da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.048+0000 7fa0641bf640 1 --2- 172.21.15.150:0/1214047706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa05c071430 0x7fa05c13da80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa05c1522d0 tx=0x7fa05401cd80 comp rx=0 tx=0).stop 2024-09-17T04:49:55.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.048+0000 7fc809d9b640 1 -- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc804031110 msgr2=0x7fc80402ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.048+0000 7fc809d9b640 1 --2- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc804031110 0x7fc80402ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.048+0000 7fc809d9b640 1 -- 172.21.15.150:0/3880169444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8040da170 msgr2=0x7fc804044550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.048+0000 7fc809d9b640 1 --2- 172.21.15.150:0/3880169444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8040da170 0x7fc804044550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.048+0000 7fc809d9b640 1 -- 172.21.15.150:0/3880169444 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8040d7c60 con 0x7fc804046780 2024-09-17T04:49:55.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.048+0000 7fc809d9b640 1 --2- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc804046780 0x7fc80417b460 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0668e0 tx=0x7fc80c076ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:55.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fc7faffd640 1 -- 172.21.15.150:0/3880169444 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c06de60 con 0x7fc804046780 2024-09-17T04:49:55.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fc7faffd640 1 -- 172.21.15.150:0/3880169444 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc80c08d7c0 con 0x7fc804046780 2024-09-17T04:49:55.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fa0641bf640 1 -- 172.21.15.150:0/1214047706 shutdown_connections 2024-09-17T04:49:55.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fa0641bf640 1 --2- 172.21.15.150:0/1214047706 >> v2:172.21.15.176:3300/0 conn(0x7fa05c0c6c60 0x7fa05c1586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fa0641bf640 1 --2- 172.21.15.150:0/1214047706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa05c071430 0x7fa05c13da80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fa0641bf640 1 -- 172.21.15.150:0/1214047706 >> 172.21.15.150:0/1214047706 conn(0x7fa05c0abb30 msgr2=0x7fa05c0abf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fa0641bf640 1 -- 172.21.15.150:0/1214047706 shutdown_connections 2024-09-17T04:49:55.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fc7faffd640 1 -- 172.21.15.150:0/3880169444 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c08db90 con 0x7fc804046780 2024-09-17T04:49:55.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fc80b59e640 1 -- 172.21.15.150:0/3880169444 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80417af40 con 0x7fc804046780 2024-09-17T04:49:55.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fc80b59e640 1 -- 172.21.15.150:0/3880169444 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8040d3360 con 0x7fc804046780 2024-09-17T04:49:55.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fc80b59e640 1 -- 172.21.15.150:0/3880169444 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8040492f0 con 0x7fc804046780 2024-09-17T04:49:55.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fa0641bf640 1 -- 172.21.15.150:0/1214047706 wait complete. 2024-09-17T04:49:55.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fa0641bf640 1 Processor -- start 2024-09-17T04:49:55.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.052+0000 7fa0641bf640 1 -- start start 2024-09-17T04:49:55.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fc7faffd640 1 -- 172.21.15.150:0/3880169444 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc80c09f3f0 con 0x7fc804046780 2024-09-17T04:49:55.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fc7faffd640 1 --2- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7ec073250 0x7fc7ec075760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fc7faffd640 1 -- 172.21.15.150:0/3880169444 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fc80c111ad0 con 0x7fc804046780 2024-09-17T04:49:55.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa0641bf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa05c071430 0x7fa05c153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa0641bf640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa05c0c6850 0x7fa05c0ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa0641bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa05c0c6c60 0x7fa05c0acf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa0641bf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa05c0c42b0 con 0x7fa05c0c6c60 2024-09-17T04:49:55.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa06275c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa05c0c6c60 0x7fa05c0acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa06275c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa05c0c6c60 0x7fa05c0acf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33454/0 (socket says 172.21.15.150:33454) 2024-09-17T04:49:55.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa06275c640 1 -- 172.21.15.150:0/2610476820 learned_addr learned my addr 172.21.15.150:0/2610476820 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:55.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fc80a59c640 1 --2- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7ec073250 0x7fc7ec075760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa0641bf640 1 -- 172.21.15.150:0/2610476820 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa05c0c4430 con 0x7fa05c0c6850 2024-09-17T04:49:55.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa061f5b640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa05c071430 0x7fa05c153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa0641bf640 1 -- 172.21.15.150:0/2610476820 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa05c0c9520 con 0x7fa05c071430 2024-09-17T04:49:55.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa06175a640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa05c0c6850 0x7fa05c0ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa06275c640 1 -- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa05c071430 msgr2=0x7fa05c153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa06275c640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa05c071430 0x7fa05c153790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa06275c640 1 -- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa05c0c6850 msgr2=0x7fa05c0ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa06275c640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa05c0c6850 0x7fa05c0ae670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa06275c640 1 -- 172.21.15.150:0/2610476820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05c0c8470 con 0x7fa05c0c6c60 2024-09-17T04:49:55.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fa06275c640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa05c0c6c60 0x7fa05c0acf00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa05800e770 tx=0x7fa05800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:55.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.056+0000 7fc80a59c640 1 --2- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7ec073250 0x7fc7ec075760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc001570 tx=0x7fc7fc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:55.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.064+0000 7fa0527fc640 1 -- 172.21.15.150:0/2610476820 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05800cc60 con 0x7fa05c0c6c60 2024-09-17T04:49:55.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.064+0000 7fa0527fc640 1 -- 172.21.15.150:0/2610476820 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa05800ce00 con 0x7fa05c0c6c60 2024-09-17T04:49:55.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.064+0000 7fa0641bf640 1 -- 172.21.15.150:0/2610476820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa05c154100 con 0x7fa05c0c6c60 2024-09-17T04:49:55.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.064+0000 7fa0527fc640 1 -- 172.21.15.150:0/2610476820 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058005280 con 0x7fa05c0c6c60 2024-09-17T04:49:55.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.064+0000 7fa0527fc640 1 -- 172.21.15.150:0/2610476820 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa058005420 con 0x7fa05c0c6c60 2024-09-17T04:49:55.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.064+0000 7fa0641bf640 1 -- 172.21.15.150:0/2610476820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa05c0ad960 con 0x7fa05c0c6c60 2024-09-17T04:49:55.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.064+0000 7fa0527fc640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa040073280 0x7fa040075790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.068+0000 7fa0527fc640 1 -- 172.21.15.150:0/2610476820 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fa0580943e0 con 0x7fa05c0c6c60 2024-09-17T04:49:55.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.068+0000 7fa061f5b640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa040073280 0x7fa040075790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.068+0000 7fa0641bf640 1 -- 172.21.15.150:0/2610476820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa05c0adbc0 con 0x7fa05c0c6c60 2024-09-17T04:49:55.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.068+0000 7fa061f5b640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa040073280 0x7fa040075790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa0540af000 tx=0x7fa05401b5c0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:55.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.068+0000 7fc7faffd640 1 -- 172.21.15.150:0/3880169444 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc80c0712d0 con 0x7fc804046780 2024-09-17T04:49:55.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.076+0000 7fa0527fc640 1 -- 172.21.15.150:0/2610476820 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa05805eca0 con 0x7fa05c0c6c60 2024-09-17T04:49:55.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.272+0000 7ff1b1ffb640 1 -- 172.21.15.150:0/2090689525 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ff1b8005590 con 0x7ff1cc149c60 2024-09-17T04:49:55.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.272+0000 7ff1b3fff640 1 -- 172.21.15.150:0/2090689525 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff1cc15ab30 con 0x7ff1cc149c60 2024-09-17T04:49:55.275 INFO:teuthology.orchestra.run.smithi150.stdout:64424509450 2024-09-17T04:49:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.280+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff1b4075120 msgr2=0x7ff1b4077630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.280+0000 7ff1d1aa2640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff1b4075120 0x7ff1b4077630 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40ac000 tx=0x7ff1c401a120 comp rx=0 tx=0).stop 2024-09-17T04:49:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.280+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2090689525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc149c60 msgr2=0x7ff1cc14a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.280+0000 7ff1d1aa2640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc149c60 0x7ff1cc14a070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc004ee0 tx=0x7ff1bc001640 comp rx=0 tx=0).stop 2024-09-17T04:49:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.280+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2090689525 shutdown_connections 2024-09-17T04:49:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.280+0000 7ff1d1aa2640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff1b4075120 0x7ff1b4077630 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.280+0000 7ff1d1aa2640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1cc0ad420 0x7ff1cc0ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.280+0000 7ff1d1aa2640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1cc149c60 0x7ff1cc14a070 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.280+0000 7ff1d1aa2640 1 --2- 172.21.15.150:0/2090689525 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff1cc155b10 0x7ff1cc0c58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.280+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2090689525 >> 172.21.15.150:0/2090689525 conn(0x7ff1cc0c7f30 msgr2=0x7ff1cc153830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.280+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2090689525 shutdown_connections 2024-09-17T04:49:55.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.280+0000 7ff1d1aa2640 1 -- 172.21.15.150:0/2090689525 wait complete. 2024-09-17T04:49:55.308 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.2 2024-09-17T04:49:55.309 DEBUG:teuthology.parallel:result is None 2024-09-17T04:49:55.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.328+0000 7f85f5583640 1 -- 172.21.15.150:0/960620669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f85d0005860 con 0x7f85f00c2500 2024-09-17T04:49:55.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.328+0000 7f85d77fe640 1 -- 172.21.15.150:0/960620669 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f85e80167b0 con 0x7f85f00c2500 2024-09-17T04:49:55.332 INFO:teuthology.orchestra.run.smithi150.stdout:64424509451 2024-09-17T04:49:55.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.332+0000 7f85f5583640 1 -- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85dc073230 msgr2=0x7f85dc075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.332+0000 7f85f5583640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85dc073230 0x7f85dc075740 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f85e00046c0 tx=0x7f85e0007ea0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.332+0000 7f85f5583640 1 -- 172.21.15.150:0/960620669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85f00c2500 msgr2=0x7f85f01554d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.332+0000 7f85f5583640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85f00c2500 0x7f85f01554d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009b10 tx=0x7f85e8009280 comp rx=0 tx=0).stop 2024-09-17T04:49:55.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.332+0000 7f85f5583640 1 -- 172.21.15.150:0/960620669 shutdown_connections 2024-09-17T04:49:55.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.332+0000 7f85f5583640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f85dc073230 0x7f85dc075740 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.332+0000 7f85f5583640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85f014f490 0x7f85f0142ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.332+0000 7f85f5583640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85f00c2500 0x7f85f01554d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.332+0000 7f85f5583640 1 --2- 172.21.15.150:0/960620669 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85f00707e0 0x7f85f00ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.332+0000 7f85f5583640 1 -- 172.21.15.150:0/960620669 >> 172.21.15.150:0/960620669 conn(0x7f85f00c7f30 msgr2=0x7f85f0149300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.332+0000 7f85f5583640 1 -- 172.21.15.150:0/960620669 shutdown_connections 2024-09-17T04:49:55.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.332+0000 7f85f5583640 1 -- 172.21.15.150:0/960620669 wait complete. 2024-09-17T04:49:55.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.344+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/2169543868 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fe0c000a850 con 0x7fe0c000ad90 2024-09-17T04:49:55.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.344+0000 7fe0c57fa640 1 -- 172.21.15.150:0/2169543868 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe0bc063c00 con 0x7fe0c000ad90 2024-09-17T04:49:55.346 INFO:teuthology.orchestra.run.smithi150.stdout:64424509450 2024-09-17T04:49:55.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7f08a446d640 1 -- 172.21.15.150:0/1056681768 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0884005590 con 0x7f089c0715b0 2024-09-17T04:49:55.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe0ac073200 msgr2=0x7fe0ac075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7fe0cf0d0640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe0ac073200 0x7fe0ac075710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0006d80 tx=0x7fe0c806cbf0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0c000ad90 msgr2=0x7fe0c00e7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7fe0cf0d0640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0c000ad90 0x7fe0c00e7ca0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0058d0 tx=0x7fe0bc00dde0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7f0892ffd640 1 -- 172.21.15.150:0/1056681768 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f088c016410 con 0x7f089c0715b0 2024-09-17T04:49:55.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/2169543868 shutdown_connections 2024-09-17T04:49:55.351 INFO:teuthology.orchestra.run.smithi150.stdout:64424509450 2024-09-17T04:49:55.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7fe0cf0d0640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe0ac073200 0x7fe0ac075710 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7fe0cf0d0640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00da170 0x7fe0c00cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7fe0cf0d0640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0c0031110 0x7fe0c0178650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7fe0cf0d0640 1 --2- 172.21.15.150:0/2169543868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0c000ad90 0x7fe0c00e7ca0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/2169543868 >> 172.21.15.150:0/2169543868 conn(0x7fe0c0048270 msgr2=0x7fe0c0004a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/2169543868 shutdown_connections 2024-09-17T04:49:55.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.348+0000 7fe0cf0d0640 1 -- 172.21.15.150:0/2169543868 wait complete. 2024-09-17T04:49:55.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.352+0000 7f08a446d640 1 -- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0880073230 msgr2=0x7f0880075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.352+0000 7f08a446d640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0880073230 0x7f0880075740 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f089c15d0d0 tx=0x7f0898009040 comp rx=0 tx=0).stop 2024-09-17T04:49:55.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.352+0000 7f08a446d640 1 -- 172.21.15.150:0/1056681768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f089c0715b0 msgr2=0x7f089c0ac3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.352+0000 7f08a446d640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f089c0715b0 0x7f089c0ac3d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f088c01a300 tx=0x7f088c0a5de0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.352+0000 7f08a446d640 1 -- 172.21.15.150:0/1056681768 shutdown_connections 2024-09-17T04:49:55.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.352+0000 7f08a446d640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0880073230 0x7f0880075740 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.352+0000 7f08a446d640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f089c153450 0x7f089c149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.352+0000 7f08a446d640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f089c074140 0x7f089c1ec7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.352+0000 7f08a446d640 1 --2- 172.21.15.150:0/1056681768 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f089c0715b0 0x7f089c0ac3d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.352+0000 7f08a446d640 1 -- 172.21.15.150:0/1056681768 >> 172.21.15.150:0/1056681768 conn(0x7f089c0c4e30 msgr2=0x7f089c06ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.352+0000 7f08a446d640 1 -- 172.21.15.150:0/1056681768 shutdown_connections 2024-09-17T04:49:55.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.352+0000 7f08a446d640 1 -- 172.21.15.150:0/1056681768 wait complete. 2024-09-17T04:49:55.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.360+0000 7fc80b59e640 1 -- 172.21.15.150:0/3880169444 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc8040ce240 con 0x7fc804046780 2024-09-17T04:49:55.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.360+0000 7fc7faffd640 1 -- 172.21.15.150:0/3880169444 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc80c0dc9c0 con 0x7fc804046780 2024-09-17T04:49:55.362 INFO:teuthology.orchestra.run.smithi150.stdout:64424509450 2024-09-17T04:49:55.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509451 for osd.1 2024-09-17T04:49:55.363 DEBUG:teuthology.parallel:result is None 2024-09-17T04:49:55.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.364+0000 7fc80b59e640 1 -- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7ec073250 msgr2=0x7fc7ec075760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.364+0000 7fc80b59e640 1 --2- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7ec073250 0x7fc7ec075760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc001570 tx=0x7fc7fc009040 comp rx=0 tx=0).stop 2024-09-17T04:49:55.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.364+0000 7fc80b59e640 1 -- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc804046780 msgr2=0x7fc80417b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.364+0000 7fc80b59e640 1 --2- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc804046780 0x7fc80417b460 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0668e0 tx=0x7fc80c076ea0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.364+0000 7fc80b59e640 1 -- 172.21.15.150:0/3880169444 shutdown_connections 2024-09-17T04:49:55.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.364+0000 7fc80b59e640 1 --2- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7ec073250 0x7fc7ec075760 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.364+0000 7fc80b59e640 1 --2- 172.21.15.150:0/3880169444 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8040da170 0x7fc804044550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.364+0000 7fc80b59e640 1 --2- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc804046780 0x7fc80417b460 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.364+0000 7fc80b59e640 1 --2- 172.21.15.150:0/3880169444 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc804031110 0x7fc80402ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.364+0000 7fc80b59e640 1 -- 172.21.15.150:0/3880169444 >> 172.21.15.150:0/3880169444 conn(0x7fc804048270 msgr2=0x7fc8040cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.364+0000 7fc80b59e640 1 -- 172.21.15.150:0/3880169444 shutdown_connections 2024-09-17T04:49:55.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.364+0000 7fc80b59e640 1 -- 172.21.15.150:0/3880169444 wait complete. 2024-09-17T04:49:55.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.372+0000 7f38ac818640 1 -- 172.21.15.150:0/2755184709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3888005590 con 0x7f38a80c8540 2024-09-17T04:49:55.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.372+0000 7f388f7fe640 1 -- 172.21.15.150:0/2755184709 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f389c05ef20 con 0x7f38a80c8540 2024-09-17T04:49:55.376 INFO:teuthology.orchestra.run.smithi150.stdout:64424509450 2024-09-17T04:49:55.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f54be14b640 1 -- 172.21.15.150:0/1833078483 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f5498005860 con 0x7f54b8156840 2024-09-17T04:49:55.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f549ffff640 1 -- 172.21.15.150:0/1833078483 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f54a8062810 con 0x7f54b8156840 2024-09-17T04:49:55.377 INFO:teuthology.orchestra.run.smithi150.stdout:64424509451 2024-09-17T04:49:55.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f38ac818640 1 -- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f38940731b0 msgr2=0x7f38940756c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f38ac818640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f38940731b0 0x7f38940756c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f38a00af000 tx=0x7f38a001b600 comp rx=0 tx=0).stop 2024-09-17T04:49:55.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f38ac818640 1 -- 172.21.15.150:0/2755184709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a80c8540 msgr2=0x7f38a80cb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f38ac818640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a80c8540 0x7f38a80cb590 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f389c00c940 tx=0x7f389c00ce00 comp rx=0 tx=0).stop 2024-09-17T04:49:55.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f38ac818640 1 -- 172.21.15.150:0/2755184709 shutdown_connections 2024-09-17T04:49:55.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f38ac818640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f38940731b0 0x7f38940756c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f38ac818640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38a80c8e90 0x7f38a80c21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f38ac818640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38a80c8540 0x7f38a80cb590 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f38ac818640 1 --2- 172.21.15.150:0/2755184709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38a80c72a0 0x7f38a814c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f38ac818640 1 -- 172.21.15.150:0/2755184709 >> 172.21.15.150:0/2755184709 conn(0x7f38a8074580 msgr2=0x7f38a80c4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f38ac818640 1 -- 172.21.15.150:0/2755184709 shutdown_connections 2024-09-17T04:49:55.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.376+0000 7f38ac818640 1 -- 172.21.15.150:0/2755184709 wait complete. 2024-09-17T04:49:55.381 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.7 2024-09-17T04:49:55.381 DEBUG:teuthology.parallel:result is None 2024-09-17T04:49:55.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.380+0000 7f54be14b640 1 -- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f54a4073200 msgr2=0x7f54a4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.380+0000 7f54be14b640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f54a4073200 0x7f54a4075710 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009a00 tx=0x7f54b0009500 comp rx=0 tx=0).stop 2024-09-17T04:49:55.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.380+0000 7f54be14b640 1 -- 172.21.15.150:0/1833078483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54b8156840 msgr2=0x7f54b81533c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.380+0000 7f54be14b640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54b8156840 0x7f54b81533c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f54a800c3c0 tx=0x7f54a800c880 comp rx=0 tx=0).stop 2024-09-17T04:49:55.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.380+0000 7f54be14b640 1 -- 172.21.15.150:0/1833078483 shutdown_connections 2024-09-17T04:49:55.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.380+0000 7f54be14b640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f54a4073200 0x7f54a4075710 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.380+0000 7f54be14b640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54b8156840 0x7f54b81533c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.380+0000 7f54be14b640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54b80a72a0 0x7f54b8150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.380+0000 7f54be14b640 1 --2- 172.21.15.150:0/1833078483 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54b8071b20 0x7f54b8151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.380+0000 7f54be14b640 1 -- 172.21.15.150:0/1833078483 >> 172.21.15.150:0/1833078483 conn(0x7f54b80c4e30 msgr2=0x7f54b80703f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.380+0000 7f54be14b640 1 -- 172.21.15.150:0/1833078483 shutdown_connections 2024-09-17T04:49:55.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.380+0000 7f54be14b640 1 -- 172.21.15.150:0/1833078483 wait complete. 2024-09-17T04:49:55.388 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.3 2024-09-17T04:49:55.388 DEBUG:teuthology.parallel:result is None 2024-09-17T04:49:55.392 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.5 2024-09-17T04:49:55.392 DEBUG:teuthology.parallel:result is None 2024-09-17T04:49:55.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.4 2024-09-17T04:49:55.404 DEBUG:teuthology.parallel:result is None 2024-09-17T04:49:55.411 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509451 for osd.0 2024-09-17T04:49:55.411 DEBUG:teuthology.parallel:result is None 2024-09-17T04:49:55.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.412+0000 7fa0641bf640 1 -- 172.21.15.150:0/2610476820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa05c13da80 con 0x7fa05c0c6c60 2024-09-17T04:49:55.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.412+0000 7fa0527fc640 1 -- 172.21.15.150:0/2610476820 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa0580636d0 con 0x7fa05c0c6c60 2024-09-17T04:49:55.414 INFO:teuthology.orchestra.run.smithi150.stdout:60129542155 2024-09-17T04:49:55.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.416+0000 7fa0641bf640 1 -- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa040073280 msgr2=0x7fa040075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.416+0000 7fa0641bf640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa040073280 0x7fa040075790 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa0540af000 tx=0x7fa05401b5c0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.416+0000 7fa0641bf640 1 -- 172.21.15.150:0/2610476820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa05c0c6c60 msgr2=0x7fa05c0acf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.416+0000 7fa0641bf640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa05c0c6c60 0x7fa05c0acf00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa05800e770 tx=0x7fa05800ec30 comp rx=0 tx=0).stop 2024-09-17T04:49:55.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.416+0000 7fa0641bf640 1 -- 172.21.15.150:0/2610476820 shutdown_connections 2024-09-17T04:49:55.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.416+0000 7fa0641bf640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa040073280 0x7fa040075790 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.416+0000 7fa0641bf640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa05c0c6850 0x7fa05c0ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.416+0000 7fa0641bf640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa05c0c6c60 0x7fa05c0acf00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.416+0000 7fa0641bf640 1 --2- 172.21.15.150:0/2610476820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa05c071430 0x7fa05c153790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.416+0000 7fa0641bf640 1 -- 172.21.15.150:0/2610476820 >> 172.21.15.150:0/2610476820 conn(0x7fa05c0abb30 msgr2=0x7fa05c0c5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.416+0000 7fa0641bf640 1 -- 172.21.15.150:0/2610476820 shutdown_connections 2024-09-17T04:49:55.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.416+0000 7fa0641bf640 1 -- 172.21.15.150:0/2610476820 wait complete. 2024-09-17T04:49:55.434 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542155 for osd.6 2024-09-17T04:49:55.434 DEBUG:teuthology.parallel:result is None 2024-09-17T04:49:55.434 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:49:55.434 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:49:55.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.608+0000 7fd6af92e640 1 -- 172.21.15.150:0/2575340639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd690003a60 msgr2=0x7fd690003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.608+0000 7fd6af92e640 1 --2- 172.21.15.150:0/2575340639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd690003a60 0x7fd690003ed0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd69800bb10 tx=0x7fd69801cde0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.608+0000 7fd6adecb640 1 -- 172.21.15.150:0/2575340639 reap_dead start 2024-09-17T04:49:55.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.608+0000 7fd6af92e640 1 -- 172.21.15.150:0/2575340639 shutdown_connections 2024-09-17T04:49:55.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.608+0000 7fd6af92e640 1 -- 172.21.15.150:0/2575340639 >> 172.21.15.150:0/2575340639 conn(0x7fd6a80c4a00 msgr2=0x7fd6a80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.608+0000 7fd6af92e640 1 -- 172.21.15.150:0/2575340639 shutdown_connections 2024-09-17T04:49:55.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.608+0000 7fd6af92e640 1 -- 172.21.15.150:0/2575340639 wait complete. 2024-09-17T04:49:55.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6af92e640 1 Processor -- start 2024-09-17T04:49:55.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6af92e640 1 -- start start 2024-09-17T04:49:55.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6af92e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6a80a9060 0x7fd6a8148760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6af92e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6a80a9470 0x7fd6a8148ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6af92e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6a80ac5f0 0x7fd6a80aca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6af92e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a80710d0 con 0x7fd6a80a9470 2024-09-17T04:49:55.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6af92e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a81536e0 con 0x7fd6a80a9060 2024-09-17T04:49:55.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6af92e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a815d1c0 con 0x7fd6a80ac5f0 2024-09-17T04:49:55.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6acec9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6a80a9470 0x7fd6a8148ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6acec9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6a80a9470 0x7fd6a8148ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33464/0 (socket says 172.21.15.150:33464) 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6acec9640 1 -- 172.21.15.150:0/1680077626 learned_addr learned my addr 172.21.15.150:0/1680077626 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6ad6ca640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6a80a9060 0x7fd6a8148760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6adecb640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6a80ac5f0 0x7fd6a80aca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6acec9640 1 -- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6a80ac5f0 msgr2=0x7fd6a80aca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6acec9640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6a80ac5f0 0x7fd6a80aca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6acec9640 1 -- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6a80a9060 msgr2=0x7fd6a8148760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6acec9640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6a80a9060 0x7fd6a8148760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6acec9640 1 -- 172.21.15.150:0/1680077626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a8149900 con 0x7fd6a80a9470 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6adecb640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6a80ac5f0 0x7fd6a80aca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6acec9640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6a80a9470 0x7fd6a8148ca0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd69c002240 tx=0x7fd69c00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6a5ffb640 1 -- 172.21.15.150:0/1680077626 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c015070 con 0x7fd6a80a9470 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6a5ffb640 1 -- 172.21.15.150:0/1680077626 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd69c00cda0 con 0x7fd6a80a9470 2024-09-17T04:49:55.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6af92e640 1 -- 172.21.15.150:0/1680077626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a8149460 con 0x7fd6a80a9470 2024-09-17T04:49:55.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6a5ffb640 1 -- 172.21.15.150:0/1680077626 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c0116c0 con 0x7fd6a80a9470 2024-09-17T04:49:55.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6af92e640 1 -- 172.21.15.150:0/1680077626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a81518d0 con 0x7fd6a80a9470 2024-09-17T04:49:55.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.612+0000 7fd6af92e640 1 -- 172.21.15.150:0/1680077626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6a81501c0 con 0x7fd6a80a9470 2024-09-17T04:49:55.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.616+0000 7fd6a5ffb640 1 -- 172.21.15.150:0/1680077626 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd69c018070 con 0x7fd6a80a9470 2024-09-17T04:49:55.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.616+0000 7fd6a5ffb640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd694073200 0x7fd694075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:55.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.616+0000 7fd6a5ffb640 1 -- 172.21.15.150:0/1680077626 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fd69c094d10 con 0x7fd6a80a9470 2024-09-17T04:49:55.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.616+0000 7fd6ad6ca640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd694073200 0x7fd694075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:55.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.616+0000 7fd6ad6ca640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd694073200 0x7fd694075710 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd6980b0000 tx=0x7fd6980a7040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:55.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.620+0000 7fd6a5ffb640 1 -- 172.21.15.150:0/1680077626 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd69c05fb20 con 0x7fd6a80a9470 2024-09-17T04:49:55.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.772+0000 7fd6af92e640 1 -- 172.21.15.150:0/1680077626 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd6a80c3290 con 0x7fd694073200 2024-09-17T04:49:55.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.792+0000 7fd6a5ffb640 1 -- 172.21.15.150:0/1680077626 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490297 (secure 0 0 0) 0x7fd6a80c3290 con 0x7fd694073200 2024-09-17T04:49:55.801 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:55.805 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T04:49:55.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.800+0000 7fd6af92e640 1 -- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd694073200 msgr2=0x7fd694075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.800+0000 7fd6af92e640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd694073200 0x7fd694075710 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd6980b0000 tx=0x7fd6980a7040 comp rx=0 tx=0).stop 2024-09-17T04:49:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.800+0000 7fd6af92e640 1 -- 172.21.15.150:0/1680077626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6a80a9470 msgr2=0x7fd6a8148ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.800+0000 7fd6af92e640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6a80a9470 0x7fd6a8148ca0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd69c002240 tx=0x7fd69c00ee00 comp rx=0 tx=0).stop 2024-09-17T04:49:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.800+0000 7fd6af92e640 1 -- 172.21.15.150:0/1680077626 shutdown_connections 2024-09-17T04:49:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.800+0000 7fd6af92e640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd694073200 0x7fd694075710 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.800+0000 7fd6af92e640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd6a80ac5f0 0x7fd6a80aca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.800+0000 7fd6af92e640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6a80a9470 0x7fd6a8148ca0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.800+0000 7fd6af92e640 1 --2- 172.21.15.150:0/1680077626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6a80a9060 0x7fd6a8148760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.800+0000 7fd6af92e640 1 -- 172.21.15.150:0/1680077626 >> 172.21.15.150:0/1680077626 conn(0x7fd6a80c4a00 msgr2=0x7fd6a80c1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.800+0000 7fd6af92e640 1 -- 172.21.15.150:0/1680077626 shutdown_connections 2024-09-17T04:49:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:55.800+0000 7fd6af92e640 1 -- 172.21.15.150:0/1680077626 wait complete. 2024-09-17T04:49:55.821 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-17T04:49:54.951293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":191,"num_write_kb":1656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223812,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750908,"statfs":{"total":773094113280,"available":772864929792,"internally_reserved":0,"allocated":3481600,"data_stored":2027328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005969"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671077+0000","last_change":"2024-09-17T04:49:41.025271+0000","last_active":"2024-09-17T04:49:42.671077+0000","last_peered":"2024-09-17T04:49:42.671077+0000","last_clean":"2024-09-17T04:49:42.671077+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T04:49:42.671077+0000","last_undegraded":"2024-09-17T04:49:42.671077+0000","last_fullsized":"2024-09-17T04:49:42.671077+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:39.715602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054640+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T04:49:42.054640+0000","last_peered":"2024-09-17T04:49:42.054640+0000","last_clean":"2024-09-17T04:49:42.054640+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T04:49:42.054640+0000","last_undegraded":"2024-09-17T04:49:42.054640+0000","last_fullsized":"2024-09-17T04:49:42.054640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671706+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T04:49:42.671706+0000","last_peered":"2024-09-17T04:49:42.671706+0000","last_clean":"2024-09-17T04:49:42.671706+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T04:49:42.671706+0000","last_undegraded":"2024-09-17T04:49:42.671706+0000","last_fullsized":"2024-09-17T04:49:42.671706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668736+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T04:49:42.668736+0000","last_peered":"2024-09-17T04:49:42.668736+0000","last_clean":"2024-09-17T04:49:42.668736+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T04:49:42.668736+0000","last_undegraded":"2024-09-17T04:49:42.668736+0000","last_fullsized":"2024-09-17T04:49:42.668736+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:38.417840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047972+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T04:49:42.047972+0000","last_peered":"2024-09-17T04:49:42.047972+0000","last_clean":"2024-09-17T04:49:42.047972+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T04:49:42.047972+0000","last_undegraded":"2024-09-17T04:49:42.047972+0000","last_fullsized":"2024-09-17T04:49:42.047972+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672300+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T04:49:42.672300+0000","last_peered":"2024-09-17T04:49:42.672300+0000","last_clean":"2024-09-17T04:49:42.672300+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T04:49:42.672300+0000","last_undegraded":"2024-09-17T04:49:42.672300+0000","last_fullsized":"2024-09-17T04:49:42.672300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672014+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T04:49:42.672014+0000","last_peered":"2024-09-17T04:49:42.672014+0000","last_clean":"2024-09-17T04:49:42.672014+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T04:49:42.672014+0000","last_undegraded":"2024-09-17T04:49:42.672014+0000","last_fullsized":"2024-09-17T04:49:42.672014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535134+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T04:49:42.535134+0000","last_peered":"2024-09-17T04:49:42.535134+0000","last_clean":"2024-09-17T04:49:42.535134+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T04:49:42.535134+0000","last_undegraded":"2024-09-17T04:49:42.535134+0000","last_fullsized":"2024-09-17T04:49:42.535134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672206+0000","last_change":"2024-09-17T04:49:41.067317+0000","last_active":"2024-09-17T04:49:42.672206+0000","last_peered":"2024-09-17T04:49:42.672206+0000","last_clean":"2024-09-17T04:49:42.672206+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T04:49:42.672206+0000","last_undegraded":"2024-09-17T04:49:42.672206+0000","last_fullsized":"2024-09-17T04:49:42.672206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:37.637477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670707+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T04:49:42.670707+0000","last_peered":"2024-09-17T04:49:42.670707+0000","last_clean":"2024-09-17T04:49:42.670707+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T04:49:42.670707+0000","last_undegraded":"2024-09-17T04:49:42.670707+0000","last_fullsized":"2024-09-17T04:49:42.670707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674274+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T04:49:42.674274+0000","last_peered":"2024-09-17T04:49:42.674274+0000","last_clean":"2024-09-17T04:49:42.674274+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T04:49:42.674274+0000","last_undegraded":"2024-09-17T04:49:42.674274+0000","last_fullsized":"2024-09-17T04:49:42.674274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535118+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T04:49:42.535118+0000","last_peered":"2024-09-17T04:49:42.535118+0000","last_clean":"2024-09-17T04:49:42.535118+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T04:49:42.535118+0000","last_undegraded":"2024-09-17T04:49:42.535118+0000","last_fullsized":"2024-09-17T04:49:42.535118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535259+0000","last_change":"2024-09-17T04:49:41.043531+0000","last_active":"2024-09-17T04:49:42.535259+0000","last_peered":"2024-09-17T04:49:42.535259+0000","last_clean":"2024-09-17T04:49:42.535259+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T04:49:42.535259+0000","last_undegraded":"2024-09-17T04:49:42.535259+0000","last_fullsized":"2024-09-17T04:49:42.535259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672162+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T04:49:42.672162+0000","last_peered":"2024-09-17T04:49:42.672162+0000","last_clean":"2024-09-17T04:49:42.672162+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T04:49:42.672162+0000","last_undegraded":"2024-09-17T04:49:42.672162+0000","last_fullsized":"2024-09-17T04:49:42.672162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042945+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T04:49:42.042945+0000","last_peered":"2024-09-17T04:49:42.042945+0000","last_clean":"2024-09-17T04:49:42.042945+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T04:49:42.042945+0000","last_undegraded":"2024-09-17T04:49:42.042945+0000","last_fullsized":"2024-09-17T04:49:42.042945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535003+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T04:49:42.535003+0000","last_peered":"2024-09-17T04:49:42.535003+0000","last_clean":"2024-09-17T04:49:42.535003+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T04:49:42.535003+0000","last_undegraded":"2024-09-17T04:49:42.535003+0000","last_fullsized":"2024-09-17T04:49:42.535003+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:16.854365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669588+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T04:49:42.669588+0000","last_peered":"2024-09-17T04:49:42.669588+0000","last_clean":"2024-09-17T04:49:42.669588+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T04:49:42.669588+0000","last_undegraded":"2024-09-17T04:49:42.669588+0000","last_fullsized":"2024-09-17T04:49:42.669588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042526+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T04:49:42.042526+0000","last_peered":"2024-09-17T04:49:42.042526+0000","last_clean":"2024-09-17T04:49:42.042526+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T04:49:42.042526+0000","last_undegraded":"2024-09-17T04:49:42.042526+0000","last_fullsized":"2024-09-17T04:49:42.042526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041407+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T04:49:42.041407+0000","last_peered":"2024-09-17T04:49:42.041407+0000","last_clean":"2024-09-17T04:49:42.041407+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T04:49:42.041407+0000","last_undegraded":"2024-09-17T04:49:42.041407+0000","last_fullsized":"2024-09-17T04:49:42.041407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042319+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T04:49:42.042319+0000","last_peered":"2024-09-17T04:49:42.042319+0000","last_clean":"2024-09-17T04:49:42.042319+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T04:49:42.042319+0000","last_undegraded":"2024-09-17T04:49:42.042319+0000","last_fullsized":"2024-09-17T04:49:42.042319+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:28.365399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669339+0000","last_change":"2024-09-17T04:49:41.038415+0000","last_active":"2024-09-17T04:49:42.669339+0000","last_peered":"2024-09-17T04:49:42.669339+0000","last_clean":"2024-09-17T04:49:42.669339+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T04:49:42.669339+0000","last_undegraded":"2024-09-17T04:49:42.669339+0000","last_fullsized":"2024-09-17T04:49:42.669339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:17.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668616+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T04:49:42.668616+0000","last_peered":"2024-09-17T04:49:42.668616+0000","last_clean":"2024-09-17T04:49:42.668616+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T04:49:42.668616+0000","last_undegraded":"2024-09-17T04:49:42.668616+0000","last_fullsized":"2024-09-17T04:49:42.668616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052248+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T04:49:42.052248+0000","last_peered":"2024-09-17T04:49:42.052248+0000","last_clean":"2024-09-17T04:49:42.052248+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T04:49:42.052248+0000","last_undegraded":"2024-09-17T04:49:42.052248+0000","last_fullsized":"2024-09-17T04:49:42.052248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671586+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T04:49:42.671586+0000","last_peered":"2024-09-17T04:49:42.671586+0000","last_clean":"2024-09-17T04:49:42.671586+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T04:49:42.671586+0000","last_undegraded":"2024-09-17T04:49:42.671586+0000","last_fullsized":"2024-09-17T04:49:42.671586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:17.424265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042028+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T04:49:42.042028+0000","last_peered":"2024-09-17T04:49:42.042028+0000","last_clean":"2024-09-17T04:49:42.042028+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T04:49:42.042028+0000","last_undegraded":"2024-09-17T04:49:42.042028+0000","last_fullsized":"2024-09-17T04:49:42.042028+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:06.516432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048455+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T04:49:42.048455+0000","last_peered":"2024-09-17T04:49:42.048455+0000","last_clean":"2024-09-17T04:49:42.048455+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T04:49:42.048455+0000","last_undegraded":"2024-09-17T04:49:42.048455+0000","last_fullsized":"2024-09-17T04:49:42.048455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048500+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T04:49:42.048500+0000","last_peered":"2024-09-17T04:49:42.048500+0000","last_clean":"2024-09-17T04:49:42.048500+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T04:49:42.048500+0000","last_undegraded":"2024-09-17T04:49:42.048500+0000","last_fullsized":"2024-09-17T04:49:42.048500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668827+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T04:49:42.668827+0000","last_peered":"2024-09-17T04:49:42.668827+0000","last_clean":"2024-09-17T04:49:42.668827+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T04:49:42.668827+0000","last_undegraded":"2024-09-17T04:49:42.668827+0000","last_fullsized":"2024-09-17T04:49:42.668827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:00.010365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675406+0000","last_change":"2024-09-17T04:49:41.036658+0000","last_active":"2024-09-17T04:49:42.675406+0000","last_peered":"2024-09-17T04:49:42.675406+0000","last_clean":"2024-09-17T04:49:42.675406+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T04:49:42.675406+0000","last_undegraded":"2024-09-17T04:49:42.675406+0000","last_fullsized":"2024-09-17T04:49:42.675406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042824+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T04:49:42.042824+0000","last_peered":"2024-09-17T04:49:42.042824+0000","last_clean":"2024-09-17T04:49:42.042824+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T04:49:42.042824+0000","last_undegraded":"2024-09-17T04:49:42.042824+0000","last_fullsized":"2024-09-17T04:49:42.042824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668829+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T04:49:42.668829+0000","last_peered":"2024-09-17T04:49:42.668829+0000","last_clean":"2024-09-17T04:49:42.668829+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T04:49:42.668829+0000","last_undegraded":"2024-09-17T04:49:42.668829+0000","last_fullsized":"2024-09-17T04:49:42.668829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536259+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T04:49:42.536259+0000","last_peered":"2024-09-17T04:49:42.536259+0000","last_clean":"2024-09-17T04:49:42.536259+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T04:49:42.536259+0000","last_undegraded":"2024-09-17T04:49:42.536259+0000","last_fullsized":"2024-09-17T04:49:42.536259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048942+0000","last_change":"2024-09-17T04:49:41.050170+0000","last_active":"2024-09-17T04:49:42.048942+0000","last_peered":"2024-09-17T04:49:42.048942+0000","last_clean":"2024-09-17T04:49:42.048942+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T04:49:42.048942+0000","last_undegraded":"2024-09-17T04:49:42.048942+0000","last_fullsized":"2024-09-17T04:49:42.048942+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051929+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T04:49:42.051929+0000","last_peered":"2024-09-17T04:49:42.051929+0000","last_clean":"2024-09-17T04:49:42.051929+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T04:49:42.051929+0000","last_undegraded":"2024-09-17T04:49:42.051929+0000","last_fullsized":"2024-09-17T04:49:42.051929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042185+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T04:49:42.042185+0000","last_peered":"2024-09-17T04:49:42.042185+0000","last_clean":"2024-09-17T04:49:42.042185+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T04:49:42.042185+0000","last_undegraded":"2024-09-17T04:49:42.042185+0000","last_fullsized":"2024-09-17T04:49:42.042185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535553+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T04:49:42.535553+0000","last_peered":"2024-09-17T04:49:42.535553+0000","last_clean":"2024-09-17T04:49:42.535553+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T04:49:42.535553+0000","last_undegraded":"2024-09-17T04:49:42.535553+0000","last_fullsized":"2024-09-17T04:49:42.535553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041211+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T04:49:42.041211+0000","last_peered":"2024-09-17T04:49:42.041211+0000","last_clean":"2024-09-17T04:49:42.041211+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T04:49:42.041211+0000","last_undegraded":"2024-09-17T04:49:42.041211+0000","last_fullsized":"2024-09-17T04:49:42.041211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054336+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T04:49:42.054336+0000","last_peered":"2024-09-17T04:49:42.054336+0000","last_clean":"2024-09-17T04:49:42.054336+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T04:49:42.054336+0000","last_undegraded":"2024-09-17T04:49:42.054336+0000","last_fullsized":"2024-09-17T04:49:42.054336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052826+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T04:49:42.052826+0000","last_peered":"2024-09-17T04:49:42.052826+0000","last_clean":"2024-09-17T04:49:42.052826+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T04:49:42.052826+0000","last_undegraded":"2024-09-17T04:49:42.052826+0000","last_fullsized":"2024-09-17T04:49:42.052826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048145+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T04:49:42.048145+0000","last_peered":"2024-09-17T04:49:42.048145+0000","last_clean":"2024-09-17T04:49:42.048145+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T04:49:42.048145+0000","last_undegraded":"2024-09-17T04:49:42.048145+0000","last_fullsized":"2024-09-17T04:49:42.048145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669693+0000","last_change":"2024-09-17T04:49:41.042314+0000","last_active":"2024-09-17T04:49:42.669693+0000","last_peered":"2024-09-17T04:49:42.669693+0000","last_clean":"2024-09-17T04:49:42.669693+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T04:49:42.669693+0000","last_undegraded":"2024-09-17T04:49:42.669693+0000","last_fullsized":"2024-09-17T04:49:42.669693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055299+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T04:49:42.055299+0000","last_peered":"2024-09-17T04:49:42.055299+0000","last_clean":"2024-09-17T04:49:42.055299+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T04:49:42.055299+0000","last_undegraded":"2024-09-17T04:49:42.055299+0000","last_fullsized":"2024-09-17T04:49:42.055299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043663+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T04:49:42.043663+0000","last_peered":"2024-09-17T04:49:42.043663+0000","last_clean":"2024-09-17T04:49:42.043663+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T04:49:42.043663+0000","last_undegraded":"2024-09-17T04:49:42.043663+0000","last_fullsized":"2024-09-17T04:49:42.043663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053169+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T04:49:42.053169+0000","last_peered":"2024-09-17T04:49:42.053169+0000","last_clean":"2024-09-17T04:49:42.053169+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T04:49:42.053169+0000","last_undegraded":"2024-09-17T04:49:42.053169+0000","last_fullsized":"2024-09-17T04:49:42.053169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052959+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T04:49:42.052959+0000","last_peered":"2024-09-17T04:49:42.052959+0000","last_clean":"2024-09-17T04:49:42.052959+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T04:49:42.052959+0000","last_undegraded":"2024-09-17T04:49:42.052959+0000","last_fullsized":"2024-09-17T04:49:42.052959+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535986+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T04:49:42.535986+0000","last_peered":"2024-09-17T04:49:42.535986+0000","last_clean":"2024-09-17T04:49:42.535986+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T04:49:42.535986+0000","last_undegraded":"2024-09-17T04:49:42.535986+0000","last_fullsized":"2024-09-17T04:49:42.535986+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670900+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T04:49:42.670900+0000","last_peered":"2024-09-17T04:49:42.670900+0000","last_clean":"2024-09-17T04:49:42.670900+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T04:49:42.670900+0000","last_undegraded":"2024-09-17T04:49:42.670900+0000","last_fullsized":"2024-09-17T04:49:42.670900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050191+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T04:49:42.050191+0000","last_peered":"2024-09-17T04:49:42.050191+0000","last_clean":"2024-09-17T04:49:42.050191+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T04:49:42.050191+0000","last_undegraded":"2024-09-17T04:49:42.050191+0000","last_fullsized":"2024-09-17T04:49:42.050191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670215+0000","last_change":"2024-09-17T04:49:41.069177+0000","last_active":"2024-09-17T04:49:42.670215+0000","last_peered":"2024-09-17T04:49:42.670215+0000","last_clean":"2024-09-17T04:49:42.670215+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T04:49:42.670215+0000","last_undegraded":"2024-09-17T04:49:42.670215+0000","last_fullsized":"2024-09-17T04:49:42.670215+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:16.855206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670381+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T04:49:42.670381+0000","last_peered":"2024-09-17T04:49:42.670381+0000","last_clean":"2024-09-17T04:49:42.670381+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T04:49:42.670381+0000","last_undegraded":"2024-09-17T04:49:42.670381+0000","last_fullsized":"2024-09-17T04:49:42.670381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042706+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T04:49:42.042706+0000","last_peered":"2024-09-17T04:49:42.042706+0000","last_clean":"2024-09-17T04:49:42.042706+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T04:49:42.042706+0000","last_undegraded":"2024-09-17T04:49:42.042706+0000","last_fullsized":"2024-09-17T04:49:42.042706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054900+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T04:49:42.054900+0000","last_peered":"2024-09-17T04:49:42.054900+0000","last_clean":"2024-09-17T04:49:42.054900+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T04:49:42.054900+0000","last_undegraded":"2024-09-17T04:49:42.054900+0000","last_fullsized":"2024-09-17T04:49:42.054900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669308+0000","last_change":"2024-09-17T04:49:41.068537+0000","last_active":"2024-09-17T04:49:42.669308+0000","last_peered":"2024-09-17T04:49:42.669308+0000","last_clean":"2024-09-17T04:49:42.669308+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T04:49:42.669308+0000","last_undegraded":"2024-09-17T04:49:42.669308+0000","last_fullsized":"2024-09-17T04:49:42.669308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667397+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T04:49:42.667397+0000","last_peered":"2024-09-17T04:49:42.667397+0000","last_clean":"2024-09-17T04:49:42.667397+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T04:49:42.667397+0000","last_undegraded":"2024-09-17T04:49:42.667397+0000","last_fullsized":"2024-09-17T04:49:42.667397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668940+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T04:49:42.668940+0000","last_peered":"2024-09-17T04:49:42.668940+0000","last_clean":"2024-09-17T04:49:42.668940+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T04:49:42.668940+0000","last_undegraded":"2024-09-17T04:49:42.668940+0000","last_fullsized":"2024-09-17T04:49:42.668940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054921+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T04:49:42.054921+0000","last_peered":"2024-09-17T04:49:42.054921+0000","last_clean":"2024-09-17T04:49:42.054921+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T04:49:42.054921+0000","last_undegraded":"2024-09-17T04:49:42.054921+0000","last_fullsized":"2024-09-17T04:49:42.054921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041762+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T04:49:42.041762+0000","last_peered":"2024-09-17T04:49:42.041762+0000","last_clean":"2024-09-17T04:49:42.041762+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T04:49:42.041762+0000","last_undegraded":"2024-09-17T04:49:42.041762+0000","last_fullsized":"2024-09-17T04:49:42.041762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050299+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T04:49:42.050299+0000","last_peered":"2024-09-17T04:49:42.050299+0000","last_clean":"2024-09-17T04:49:42.050299+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T04:49:42.050299+0000","last_undegraded":"2024-09-17T04:49:42.050299+0000","last_fullsized":"2024-09-17T04:49:42.050299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041771+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T04:49:42.041771+0000","last_peered":"2024-09-17T04:49:42.041771+0000","last_clean":"2024-09-17T04:49:42.041771+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T04:49:42.041771+0000","last_undegraded":"2024-09-17T04:49:42.041771+0000","last_fullsized":"2024-09-17T04:49:42.041771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672234+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T04:49:42.672234+0000","last_peered":"2024-09-17T04:49:42.672234+0000","last_clean":"2024-09-17T04:49:42.672234+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T04:49:42.672234+0000","last_undegraded":"2024-09-17T04:49:42.672234+0000","last_fullsized":"2024-09-17T04:49:42.672234+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671183+0000","last_change":"2024-09-17T04:49:41.052564+0000","last_active":"2024-09-17T04:49:42.671183+0000","last_peered":"2024-09-17T04:49:42.671183+0000","last_clean":"2024-09-17T04:49:42.671183+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T04:49:42.671183+0000","last_undegraded":"2024-09-17T04:49:42.671183+0000","last_fullsized":"2024-09-17T04:49:42.671183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049438+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T04:49:42.049438+0000","last_peered":"2024-09-17T04:49:42.049438+0000","last_clean":"2024-09-17T04:49:42.049438+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T04:49:42.049438+0000","last_undegraded":"2024-09-17T04:49:42.049438+0000","last_fullsized":"2024-09-17T04:49:42.049438+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041458+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T04:49:42.041458+0000","last_peered":"2024-09-17T04:49:42.041458+0000","last_clean":"2024-09-17T04:49:42.041458+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T04:49:42.041458+0000","last_undegraded":"2024-09-17T04:49:42.041458+0000","last_fullsized":"2024-09-17T04:49:42.041458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671735+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T04:49:42.671735+0000","last_peered":"2024-09-17T04:49:42.671735+0000","last_clean":"2024-09-17T04:49:42.671735+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T04:49:42.671735+0000","last_undegraded":"2024-09-17T04:49:42.671735+0000","last_fullsized":"2024-09-17T04:49:42.671735+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043078+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T04:49:42.043078+0000","last_peered":"2024-09-17T04:49:42.043078+0000","last_clean":"2024-09-17T04:49:42.043078+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T04:49:42.043078+0000","last_undegraded":"2024-09-17T04:49:42.043078+0000","last_fullsized":"2024-09-17T04:49:42.043078+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:21.202398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669578+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T04:49:42.669578+0000","last_peered":"2024-09-17T04:49:42.669578+0000","last_clean":"2024-09-17T04:49:42.669578+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T04:49:42.669578+0000","last_undegraded":"2024-09-17T04:49:42.669578+0000","last_fullsized":"2024-09-17T04:49:42.669578+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:48.128609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043280+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T04:49:42.043280+0000","last_peered":"2024-09-17T04:49:42.043280+0000","last_clean":"2024-09-17T04:49:42.043280+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T04:49:42.043280+0000","last_undegraded":"2024-09-17T04:49:42.043280+0000","last_fullsized":"2024-09-17T04:49:42.043280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055292+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T04:49:42.055292+0000","last_peered":"2024-09-17T04:49:42.055292+0000","last_clean":"2024-09-17T04:49:42.055292+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T04:49:42.055292+0000","last_undegraded":"2024-09-17T04:49:42.055292+0000","last_fullsized":"2024-09-17T04:49:42.055292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536164+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T04:49:42.536164+0000","last_peered":"2024-09-17T04:49:42.536164+0000","last_clean":"2024-09-17T04:49:42.536164+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T04:49:42.536164+0000","last_undegraded":"2024-09-17T04:49:42.536164+0000","last_fullsized":"2024-09-17T04:49:42.536164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050257+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T04:49:42.050257+0000","last_peered":"2024-09-17T04:49:42.050257+0000","last_clean":"2024-09-17T04:49:42.050257+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T04:49:42.050257+0000","last_undegraded":"2024-09-17T04:49:42.050257+0000","last_fullsized":"2024-09-17T04:49:42.050257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671927+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T04:49:42.671927+0000","last_peered":"2024-09-17T04:49:42.671927+0000","last_clean":"2024-09-17T04:49:42.671927+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T04:49:42.671927+0000","last_undegraded":"2024-09-17T04:49:42.671927+0000","last_fullsized":"2024-09-17T04:49:42.671927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671411+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T04:49:42.671411+0000","last_peered":"2024-09-17T04:49:42.671411+0000","last_clean":"2024-09-17T04:49:42.671411+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T04:49:42.671411+0000","last_undegraded":"2024-09-17T04:49:42.671411+0000","last_fullsized":"2024-09-17T04:49:42.671411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670087+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T04:49:42.670087+0000","last_peered":"2024-09-17T04:49:42.670087+0000","last_clean":"2024-09-17T04:49:42.670087+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T04:49:42.670087+0000","last_undegraded":"2024-09-17T04:49:42.670087+0000","last_fullsized":"2024-09-17T04:49:42.670087+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:18.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672287+0000","last_change":"2024-09-17T04:49:41.031036+0000","last_active":"2024-09-17T04:49:42.672287+0000","last_peered":"2024-09-17T04:49:42.672287+0000","last_clean":"2024-09-17T04:49:42.672287+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T04:49:42.672287+0000","last_undegraded":"2024-09-17T04:49:42.672287+0000","last_fullsized":"2024-09-17T04:49:42.672287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:48.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672111+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T04:49:42.672111+0000","last_peered":"2024-09-17T04:49:42.672111+0000","last_clean":"2024-09-17T04:49:42.672111+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T04:49:42.672111+0000","last_undegraded":"2024-09-17T04:49:42.672111+0000","last_fullsized":"2024-09-17T04:49:42.672111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052688+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T04:49:42.052688+0000","last_peered":"2024-09-17T04:49:42.052688+0000","last_clean":"2024-09-17T04:49:42.052688+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T04:49:42.052688+0000","last_undegraded":"2024-09-17T04:49:42.052688+0000","last_fullsized":"2024-09-17T04:49:42.052688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041095+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T04:49:42.041095+0000","last_peered":"2024-09-17T04:49:42.041095+0000","last_clean":"2024-09-17T04:49:42.041095+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T04:49:42.041095+0000","last_undegraded":"2024-09-17T04:49:42.041095+0000","last_fullsized":"2024-09-17T04:49:42.041095+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048488+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T04:49:42.048488+0000","last_peered":"2024-09-17T04:49:42.048488+0000","last_clean":"2024-09-17T04:49:42.048488+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T04:49:42.048488+0000","last_undegraded":"2024-09-17T04:49:42.048488+0000","last_fullsized":"2024-09-17T04:49:42.048488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536967+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T04:49:42.536967+0000","last_peered":"2024-09-17T04:49:42.536967+0000","last_clean":"2024-09-17T04:49:42.536967+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T04:49:42.536967+0000","last_undegraded":"2024-09-17T04:49:42.536967+0000","last_fullsized":"2024-09-17T04:49:42.536967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672404+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T04:49:42.672404+0000","last_peered":"2024-09-17T04:49:42.672404+0000","last_clean":"2024-09-17T04:49:42.672404+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T04:49:42.672404+0000","last_undegraded":"2024-09-17T04:49:42.672404+0000","last_fullsized":"2024-09-17T04:49:42.672404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672966+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T04:49:42.672966+0000","last_peered":"2024-09-17T04:49:42.672966+0000","last_clean":"2024-09-17T04:49:42.672966+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T04:49:42.672966+0000","last_undegraded":"2024-09-17T04:49:42.672966+0000","last_fullsized":"2024-09-17T04:49:42.672966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:09.712272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052007+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T04:49:42.052007+0000","last_peered":"2024-09-17T04:49:42.052007+0000","last_clean":"2024-09-17T04:49:42.052007+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T04:49:42.052007+0000","last_undegraded":"2024-09-17T04:49:42.052007+0000","last_fullsized":"2024-09-17T04:49:42.052007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:43.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042787+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T04:49:42.042787+0000","last_peered":"2024-09-17T04:49:42.042787+0000","last_clean":"2024-09-17T04:49:42.042787+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T04:49:42.042787+0000","last_undegraded":"2024-09-17T04:49:42.042787+0000","last_fullsized":"2024-09-17T04:49:42.042787+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049500+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T04:49:42.049500+0000","last_peered":"2024-09-17T04:49:42.049500+0000","last_clean":"2024-09-17T04:49:42.049500+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T04:49:42.049500+0000","last_undegraded":"2024-09-17T04:49:42.049500+0000","last_fullsized":"2024-09-17T04:49:42.049500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671253+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T04:49:42.671253+0000","last_peered":"2024-09-17T04:49:42.671253+0000","last_clean":"2024-09-17T04:49:42.671253+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T04:49:42.671253+0000","last_undegraded":"2024-09-17T04:49:42.671253+0000","last_fullsized":"2024-09-17T04:49:42.671253+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:52.364568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041504+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T04:49:42.041504+0000","last_peered":"2024-09-17T04:49:42.041504+0000","last_clean":"2024-09-17T04:49:42.041504+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T04:49:42.041504+0000","last_undegraded":"2024-09-17T04:49:42.041504+0000","last_fullsized":"2024-09-17T04:49:42.041504+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:03.537392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671439+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T04:49:42.671439+0000","last_peered":"2024-09-17T04:49:42.671439+0000","last_clean":"2024-09-17T04:49:42.671439+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T04:49:42.671439+0000","last_undegraded":"2024-09-17T04:49:42.671439+0000","last_fullsized":"2024-09-17T04:49:42.671439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670943+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T04:49:42.670943+0000","last_peered":"2024-09-17T04:49:42.670943+0000","last_clean":"2024-09-17T04:49:42.670943+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T04:49:42.670943+0000","last_undegraded":"2024-09-17T04:49:42.670943+0000","last_fullsized":"2024-09-17T04:49:42.670943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671787+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T04:49:42.671787+0000","last_peered":"2024-09-17T04:49:42.671787+0000","last_clean":"2024-09-17T04:49:42.671787+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T04:49:42.671787+0000","last_undegraded":"2024-09-17T04:49:42.671787+0000","last_fullsized":"2024-09-17T04:49:42.671787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:52.120543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041402+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T04:49:42.041402+0000","last_peered":"2024-09-17T04:49:42.041402+0000","last_clean":"2024-09-17T04:49:42.041402+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T04:49:42.041402+0000","last_undegraded":"2024-09-17T04:49:42.041402+0000","last_fullsized":"2024-09-17T04:49:42.041402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:56.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041285+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T04:49:42.041285+0000","last_peered":"2024-09-17T04:49:42.041285+0000","last_clean":"2024-09-17T04:49:42.041285+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T04:49:42.041285+0000","last_undegraded":"2024-09-17T04:49:42.041285+0000","last_fullsized":"2024-09-17T04:49:42.041285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042566+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T04:49:42.042566+0000","last_peered":"2024-09-17T04:49:42.042566+0000","last_clean":"2024-09-17T04:49:42.042566+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T04:49:42.042566+0000","last_undegraded":"2024-09-17T04:49:42.042566+0000","last_fullsized":"2024-09-17T04:49:42.042566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054288+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T04:49:42.054288+0000","last_peered":"2024-09-17T04:49:42.054288+0000","last_clean":"2024-09-17T04:49:42.054288+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T04:49:42.054288+0000","last_undegraded":"2024-09-17T04:49:42.054288+0000","last_fullsized":"2024-09-17T04:49:42.054288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051153+0000","last_change":"2024-09-17T04:49:41.058280+0000","last_active":"2024-09-17T04:49:42.051153+0000","last_peered":"2024-09-17T04:49:42.051153+0000","last_clean":"2024-09-17T04:49:42.051153+0000","last_became_active":"2024-09-17T04:49:41.052009+0000","last_became_peered":"2024-09-17T04:49:41.052009+0000","last_unstale":"2024-09-17T04:49:42.051153+0000","last_undegraded":"2024-09-17T04:49:42.051153+0000","last_fullsized":"2024-09-17T04:49:42.051153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:14.100818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051977+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T04:49:42.051977+0000","last_peered":"2024-09-17T04:49:42.051977+0000","last_clean":"2024-09-17T04:49:42.051977+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T04:49:42.051977+0000","last_undegraded":"2024-09-17T04:49:42.051977+0000","last_fullsized":"2024-09-17T04:49:42.051977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669355+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T04:49:42.669355+0000","last_peered":"2024-09-17T04:49:42.669355+0000","last_clean":"2024-09-17T04:49:42.669355+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T04:49:42.669355+0000","last_undegraded":"2024-09-17T04:49:42.669355+0000","last_fullsized":"2024-09-17T04:49:42.669355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042782+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T04:49:42.042782+0000","last_peered":"2024-09-17T04:49:42.042782+0000","last_clean":"2024-09-17T04:49:42.042782+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T04:49:42.042782+0000","last_undegraded":"2024-09-17T04:49:42.042782+0000","last_fullsized":"2024-09-17T04:49:42.042782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:07.945355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052280+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T04:49:42.052280+0000","last_peered":"2024-09-17T04:49:42.052280+0000","last_clean":"2024-09-17T04:49:42.052280+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T04:49:42.052280+0000","last_undegraded":"2024-09-17T04:49:42.052280+0000","last_fullsized":"2024-09-17T04:49:42.052280+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670412+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T04:49:42.670412+0000","last_peered":"2024-09-17T04:49:42.670412+0000","last_clean":"2024-09-17T04:49:42.670412+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T04:49:42.670412+0000","last_undegraded":"2024-09-17T04:49:42.670412+0000","last_fullsized":"2024-09-17T04:49:42.670412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537054+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T04:49:42.537054+0000","last_peered":"2024-09-17T04:49:42.537054+0000","last_clean":"2024-09-17T04:49:42.537054+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T04:49:42.537054+0000","last_undegraded":"2024-09-17T04:49:42.537054+0000","last_fullsized":"2024-09-17T04:49:42.537054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048913+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T04:49:42.048913+0000","last_peered":"2024-09-17T04:49:42.048913+0000","last_clean":"2024-09-17T04:49:42.048913+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T04:49:42.048913+0000","last_undegraded":"2024-09-17T04:49:42.048913+0000","last_fullsized":"2024-09-17T04:49:42.048913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:16.447835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051373+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T04:49:42.051373+0000","last_peered":"2024-09-17T04:49:42.051373+0000","last_clean":"2024-09-17T04:49:42.051373+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T04:49:42.051373+0000","last_undegraded":"2024-09-17T04:49:42.051373+0000","last_fullsized":"2024-09-17T04:49:42.051373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042063+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T04:49:42.042063+0000","last_peered":"2024-09-17T04:49:42.042063+0000","last_clean":"2024-09-17T04:49:42.042063+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T04:49:42.042063+0000","last_undegraded":"2024-09-17T04:49:42.042063+0000","last_fullsized":"2024-09-17T04:49:42.042063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670596+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T04:49:42.670596+0000","last_peered":"2024-09-17T04:49:42.670596+0000","last_clean":"2024-09-17T04:49:42.670596+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T04:49:42.670596+0000","last_undegraded":"2024-09-17T04:49:42.670596+0000","last_fullsized":"2024-09-17T04:49:42.670596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535510+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T04:49:42.535510+0000","last_peered":"2024-09-17T04:49:42.535510+0000","last_clean":"2024-09-17T04:49:42.535510+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T04:49:42.535510+0000","last_undegraded":"2024-09-17T04:49:42.535510+0000","last_fullsized":"2024-09-17T04:49:42.535510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041322+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T04:49:42.041322+0000","last_peered":"2024-09-17T04:49:42.041322+0000","last_clean":"2024-09-17T04:49:42.041322+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T04:49:42.041322+0000","last_undegraded":"2024-09-17T04:49:42.041322+0000","last_fullsized":"2024-09-17T04:49:42.041322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672182+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T04:49:42.672182+0000","last_peered":"2024-09-17T04:49:42.672182+0000","last_clean":"2024-09-17T04:49:42.672182+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T04:49:42.672182+0000","last_undegraded":"2024-09-17T04:49:42.672182+0000","last_fullsized":"2024-09-17T04:49:42.672182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671261+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T04:49:42.671261+0000","last_peered":"2024-09-17T04:49:42.671261+0000","last_clean":"2024-09-17T04:49:42.671261+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T04:49:42.671261+0000","last_undegraded":"2024-09-17T04:49:42.671261+0000","last_fullsized":"2024-09-17T04:49:42.671261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667575+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T04:49:42.667575+0000","last_peered":"2024-09-17T04:49:42.667575+0000","last_clean":"2024-09-17T04:49:42.667575+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T04:49:42.667575+0000","last_undegraded":"2024-09-17T04:49:42.667575+0000","last_fullsized":"2024-09-17T04:49:42.667575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:40.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534654+0000","last_change":"2024-09-17T04:49:41.040278+0000","last_active":"2024-09-17T04:49:42.534654+0000","last_peered":"2024-09-17T04:49:42.534654+0000","last_clean":"2024-09-17T04:49:42.534654+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T04:49:42.534654+0000","last_undegraded":"2024-09-17T04:49:42.534654+0000","last_fullsized":"2024-09-17T04:49:42.534654+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.895223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536732+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T04:49:42.536732+0000","last_peered":"2024-09-17T04:49:42.536732+0000","last_clean":"2024-09-17T04:49:42.536732+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T04:49:42.536732+0000","last_undegraded":"2024-09-17T04:49:42.536732+0000","last_fullsized":"2024-09-17T04:49:42.536732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675753+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T04:49:42.675753+0000","last_peered":"2024-09-17T04:49:42.675753+0000","last_clean":"2024-09-17T04:49:42.675753+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T04:49:42.675753+0000","last_undegraded":"2024-09-17T04:49:42.675753+0000","last_fullsized":"2024-09-17T04:49:42.675753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047646+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T04:49:42.047646+0000","last_peered":"2024-09-17T04:49:42.047646+0000","last_clean":"2024-09-17T04:49:42.047646+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T04:49:42.047646+0000","last_undegraded":"2024-09-17T04:49:42.047646+0000","last_fullsized":"2024-09-17T04:49:42.047646+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:15.845612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537298+0000","last_change":"2024-09-17T04:49:41.050551+0000","last_active":"2024-09-17T04:49:42.537298+0000","last_peered":"2024-09-17T04:49:42.537298+0000","last_clean":"2024-09-17T04:49:42.537298+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T04:49:42.537298+0000","last_undegraded":"2024-09-17T04:49:42.537298+0000","last_fullsized":"2024-09-17T04:49:42.537298+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:11.088230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051592+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T04:49:42.051592+0000","last_peered":"2024-09-17T04:49:42.051592+0000","last_clean":"2024-09-17T04:49:42.051592+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T04:49:42.051592+0000","last_undegraded":"2024-09-17T04:49:42.051592+0000","last_fullsized":"2024-09-17T04:49:42.051592+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:37.778756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048401+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T04:49:42.048401+0000","last_peered":"2024-09-17T04:49:42.048401+0000","last_clean":"2024-09-17T04:49:42.048401+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T04:49:42.048401+0000","last_undegraded":"2024-09-17T04:49:42.048401+0000","last_fullsized":"2024-09-17T04:49:42.048401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041797+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T04:49:42.041797+0000","last_peered":"2024-09-17T04:49:42.041797+0000","last_clean":"2024-09-17T04:49:42.041797+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T04:49:42.041797+0000","last_undegraded":"2024-09-17T04:49:42.041797+0000","last_fullsized":"2024-09-17T04:49:42.041797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055365+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T04:49:42.055365+0000","last_peered":"2024-09-17T04:49:42.055365+0000","last_clean":"2024-09-17T04:49:42.055365+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T04:49:42.055365+0000","last_undegraded":"2024-09-17T04:49:42.055365+0000","last_fullsized":"2024-09-17T04:49:42.055365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041767+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T04:49:42.041767+0000","last_peered":"2024-09-17T04:49:42.041767+0000","last_clean":"2024-09-17T04:49:42.041767+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T04:49:42.041767+0000","last_undegraded":"2024-09-17T04:49:42.041767+0000","last_fullsized":"2024-09-17T04:49:42.041767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667708+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T04:49:42.667708+0000","last_peered":"2024-09-17T04:49:42.667708+0000","last_clean":"2024-09-17T04:49:42.667708+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T04:49:42.667708+0000","last_undegraded":"2024-09-17T04:49:42.667708+0000","last_fullsized":"2024-09-17T04:49:42.667708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050522+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T04:49:42.050522+0000","last_peered":"2024-09-17T04:49:42.050522+0000","last_clean":"2024-09-17T04:49:42.050522+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T04:49:42.050522+0000","last_undegraded":"2024-09-17T04:49:42.050522+0000","last_fullsized":"2024-09-17T04:49:42.050522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049954+0000","last_change":"2024-09-17T04:49:41.057647+0000","last_active":"2024-09-17T04:49:42.049954+0000","last_peered":"2024-09-17T04:49:42.049954+0000","last_clean":"2024-09-17T04:49:42.049954+0000","last_became_active":"2024-09-17T04:49:41.056213+0000","last_became_peered":"2024-09-17T04:49:41.056213+0000","last_unstale":"2024-09-17T04:49:42.049954+0000","last_undegraded":"2024-09-17T04:49:42.049954+0000","last_fullsized":"2024-09-17T04:49:42.049954+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042498+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T04:49:42.042498+0000","last_peered":"2024-09-17T04:49:42.042498+0000","last_clean":"2024-09-17T04:49:42.042498+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T04:49:42.042498+0000","last_undegraded":"2024-09-17T04:49:42.042498+0000","last_fullsized":"2024-09-17T04:49:42.042498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053714+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T04:49:42.053714+0000","last_peered":"2024-09-17T04:49:42.053714+0000","last_clean":"2024-09-17T04:49:42.053714+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T04:49:42.053714+0000","last_undegraded":"2024-09-17T04:49:42.053714+0000","last_fullsized":"2024-09-17T04:49:42.053714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051282+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T04:49:42.051282+0000","last_peered":"2024-09-17T04:49:42.051282+0000","last_clean":"2024-09-17T04:49:42.051282+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T04:49:42.051282+0000","last_undegraded":"2024-09-17T04:49:42.051282+0000","last_fullsized":"2024-09-17T04:49:42.051282+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:27.230604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536526+0000","last_change":"2024-09-17T04:49:41.055263+0000","last_active":"2024-09-17T04:49:42.536526+0000","last_peered":"2024-09-17T04:49:42.536526+0000","last_clean":"2024-09-17T04:49:42.536526+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T04:49:42.536526+0000","last_undegraded":"2024-09-17T04:49:42.536526+0000","last_fullsized":"2024-09-17T04:49:42.536526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670438+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T04:49:42.670438+0000","last_peered":"2024-09-17T04:49:42.670438+0000","last_clean":"2024-09-17T04:49:42.670438+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T04:49:42.670438+0000","last_undegraded":"2024-09-17T04:49:42.670438+0000","last_fullsized":"2024-09-17T04:49:42.670438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673048+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T04:49:42.673048+0000","last_peered":"2024-09-17T04:49:42.673048+0000","last_clean":"2024-09-17T04:49:42.673048+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T04:49:42.673048+0000","last_undegraded":"2024-09-17T04:49:42.673048+0000","last_fullsized":"2024-09-17T04:49:42.673048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670447+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T04:49:42.670447+0000","last_peered":"2024-09-17T04:49:42.670447+0000","last_clean":"2024-09-17T04:49:42.670447+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T04:49:42.670447+0000","last_undegraded":"2024-09-17T04:49:42.670447+0000","last_fullsized":"2024-09-17T04:49:42.670447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:24.586834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051128+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.051128+0000","last_peered":"2024-09-17T04:49:42.051128+0000","last_clean":"2024-09-17T04:49:42.051128+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T04:49:42.051128+0000","last_undegraded":"2024-09-17T04:49:42.051128+0000","last_fullsized":"2024-09-17T04:49:42.051128+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536163+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T04:49:42.536163+0000","last_peered":"2024-09-17T04:49:42.536163+0000","last_clean":"2024-09-17T04:49:42.536163+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T04:49:42.536163+0000","last_undegraded":"2024-09-17T04:49:42.536163+0000","last_fullsized":"2024-09-17T04:49:42.536163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052121+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T04:49:42.052121+0000","last_peered":"2024-09-17T04:49:42.052121+0000","last_clean":"2024-09-17T04:49:42.052121+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T04:49:42.052121+0000","last_undegraded":"2024-09-17T04:49:42.052121+0000","last_fullsized":"2024-09-17T04:49:42.052121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668467+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T04:49:42.668467+0000","last_peered":"2024-09-17T04:49:42.668467+0000","last_clean":"2024-09-17T04:49:42.668467+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T04:49:42.668467+0000","last_undegraded":"2024-09-17T04:49:42.668467+0000","last_fullsized":"2024-09-17T04:49:42.668467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536096+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T04:49:42.536096+0000","last_peered":"2024-09-17T04:49:42.536096+0000","last_clean":"2024-09-17T04:49:42.536096+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T04:49:42.536096+0000","last_undegraded":"2024-09-17T04:49:42.536096+0000","last_fullsized":"2024-09-17T04:49:42.536096+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042322+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T04:49:42.042322+0000","last_peered":"2024-09-17T04:49:42.042322+0000","last_clean":"2024-09-17T04:49:42.042322+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T04:49:42.042322+0000","last_undegraded":"2024-09-17T04:49:42.042322+0000","last_fullsized":"2024-09-17T04:49:42.042322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671200+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T04:49:42.671200+0000","last_peered":"2024-09-17T04:49:42.671200+0000","last_clean":"2024-09-17T04:49:42.671200+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T04:49:42.671200+0000","last_undegraded":"2024-09-17T04:49:42.671200+0000","last_fullsized":"2024-09-17T04:49:42.671200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670309+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T04:49:42.670309+0000","last_peered":"2024-09-17T04:49:42.670309+0000","last_clean":"2024-09-17T04:49:42.670309+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T04:49:42.670309+0000","last_undegraded":"2024-09-17T04:49:42.670309+0000","last_fullsized":"2024-09-17T04:49:42.670309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668969+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T04:49:42.668969+0000","last_peered":"2024-09-17T04:49:42.668969+0000","last_clean":"2024-09-17T04:49:42.668969+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T04:49:42.668969+0000","last_undegraded":"2024-09-17T04:49:42.668969+0000","last_fullsized":"2024-09-17T04:49:42.668969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054935+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T04:49:42.054935+0000","last_peered":"2024-09-17T04:49:42.054935+0000","last_clean":"2024-09-17T04:49:42.054935+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T04:49:42.054935+0000","last_undegraded":"2024-09-17T04:49:42.054935+0000","last_fullsized":"2024-09-17T04:49:42.054935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669059+0000","last_change":"2024-09-17T04:49:41.073709+0000","last_active":"2024-09-17T04:49:42.669059+0000","last_peered":"2024-09-17T04:49:42.669059+0000","last_clean":"2024-09-17T04:49:42.669059+0000","last_became_active":"2024-09-17T04:49:41.073330+0000","last_became_peered":"2024-09-17T04:49:41.073330+0000","last_unstale":"2024-09-17T04:49:42.669059+0000","last_undegraded":"2024-09-17T04:49:42.669059+0000","last_fullsized":"2024-09-17T04:49:42.669059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054811+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T04:49:42.054811+0000","last_peered":"2024-09-17T04:49:42.054811+0000","last_clean":"2024-09-17T04:49:42.054811+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T04:49:42.054811+0000","last_undegraded":"2024-09-17T04:49:42.054811+0000","last_fullsized":"2024-09-17T04:49:42.054811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041256+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T04:49:42.041256+0000","last_peered":"2024-09-17T04:49:42.041256+0000","last_clean":"2024-09-17T04:49:42.041256+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T04:49:42.041256+0000","last_undegraded":"2024-09-17T04:49:42.041256+0000","last_fullsized":"2024-09-17T04:49:42.041256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536669+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T04:49:42.536669+0000","last_peered":"2024-09-17T04:49:42.536669+0000","last_clean":"2024-09-17T04:49:42.536669+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T04:49:42.536669+0000","last_undegraded":"2024-09-17T04:49:42.536669+0000","last_fullsized":"2024-09-17T04:49:42.536669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534968+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T04:49:42.534968+0000","last_peered":"2024-09-17T04:49:42.534968+0000","last_clean":"2024-09-17T04:49:42.534968+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T04:49:42.534968+0000","last_undegraded":"2024-09-17T04:49:42.534968+0000","last_fullsized":"2024-09-17T04:49:42.534968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672019+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T04:49:42.672019+0000","last_peered":"2024-09-17T04:49:42.672019+0000","last_clean":"2024-09-17T04:49:42.672019+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T04:49:42.672019+0000","last_undegraded":"2024-09-17T04:49:42.672019+0000","last_fullsized":"2024-09-17T04:49:42.672019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668565+0000","last_change":"2024-09-17T04:49:41.045502+0000","last_active":"2024-09-17T04:49:42.668565+0000","last_peered":"2024-09-17T04:49:42.668565+0000","last_clean":"2024-09-17T04:49:42.668565+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T04:49:42.668565+0000","last_undegraded":"2024-09-17T04:49:42.668565+0000","last_fullsized":"2024-09-17T04:49:42.668565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:19.746356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041666+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T04:49:42.041666+0000","last_peered":"2024-09-17T04:49:42.041666+0000","last_clean":"2024-09-17T04:49:42.041666+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T04:49:42.041666+0000","last_undegraded":"2024-09-17T04:49:42.041666+0000","last_fullsized":"2024-09-17T04:49:42.041666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042310+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T04:49:42.042310+0000","last_peered":"2024-09-17T04:49:42.042310+0000","last_clean":"2024-09-17T04:49:42.042310+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T04:49:42.042310+0000","last_undegraded":"2024-09-17T04:49:42.042310+0000","last_fullsized":"2024-09-17T04:49:42.042310+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668045+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T04:49:42.668045+0000","last_peered":"2024-09-17T04:49:42.668045+0000","last_clean":"2024-09-17T04:49:42.668045+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T04:49:42.668045+0000","last_undegraded":"2024-09-17T04:49:42.668045+0000","last_fullsized":"2024-09-17T04:49:42.668045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054699+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T04:49:42.054699+0000","last_peered":"2024-09-17T04:49:42.054699+0000","last_clean":"2024-09-17T04:49:42.054699+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T04:49:42.054699+0000","last_undegraded":"2024-09-17T04:49:42.054699+0000","last_fullsized":"2024-09-17T04:49:42.054699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673038+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T04:49:42.673038+0000","last_peered":"2024-09-17T04:49:42.673038+0000","last_clean":"2024-09-17T04:49:42.673038+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T04:49:42.673038+0000","last_undegraded":"2024-09-17T04:49:42.673038+0000","last_fullsized":"2024-09-17T04:49:42.673038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669104+0000","last_change":"2024-09-17T04:49:41.038265+0000","last_active":"2024-09-17T04:49:42.669104+0000","last_peered":"2024-09-17T04:49:42.669104+0000","last_clean":"2024-09-17T04:49:42.669104+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T04:49:42.669104+0000","last_undegraded":"2024-09-17T04:49:42.669104+0000","last_fullsized":"2024-09-17T04:49:42.669104+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:41.638403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053416+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T04:49:42.053416+0000","last_peered":"2024-09-17T04:49:42.053416+0000","last_clean":"2024-09-17T04:49:42.053416+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T04:49:42.053416+0000","last_undegraded":"2024-09-17T04:49:42.053416+0000","last_fullsized":"2024-09-17T04:49:42.053416+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535001+0000","last_change":"2024-09-17T04:49:41.053443+0000","last_active":"2024-09-17T04:49:42.535001+0000","last_peered":"2024-09-17T04:49:42.535001+0000","last_clean":"2024-09-17T04:49:42.535001+0000","last_became_active":"2024-09-17T04:49:41.053090+0000","last_became_peered":"2024-09-17T04:49:41.053090+0000","last_unstale":"2024-09-17T04:49:42.535001+0000","last_undegraded":"2024-09-17T04:49:42.535001+0000","last_fullsized":"2024-09-17T04:49:42.535001+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671599+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T04:49:42.671599+0000","last_peered":"2024-09-17T04:49:42.671599+0000","last_clean":"2024-09-17T04:49:42.671599+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T04:49:42.671599+0000","last_undegraded":"2024-09-17T04:49:42.671599+0000","last_fullsized":"2024-09-17T04:49:42.671599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669669+0000","last_change":"2024-09-17T04:49:41.038072+0000","last_active":"2024-09-17T04:49:42.669669+0000","last_peered":"2024-09-17T04:49:42.669669+0000","last_clean":"2024-09-17T04:49:42.669669+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T04:49:42.669669+0000","last_undegraded":"2024-09-17T04:49:42.669669+0000","last_fullsized":"2024-09-17T04:49:42.669669+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043492+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T04:49:42.043492+0000","last_peered":"2024-09-17T04:49:42.043492+0000","last_clean":"2024-09-17T04:49:42.043492+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T04:49:42.043492+0000","last_undegraded":"2024-09-17T04:49:42.043492+0000","last_fullsized":"2024-09-17T04:49:42.043492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534462+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T04:49:42.534462+0000","last_peered":"2024-09-17T04:49:42.534462+0000","last_clean":"2024-09-17T04:49:42.534462+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T04:49:42.534462+0000","last_undegraded":"2024-09-17T04:49:42.534462+0000","last_fullsized":"2024-09-17T04:49:42.534462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669602+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T04:49:42.669602+0000","last_peered":"2024-09-17T04:49:42.669602+0000","last_clean":"2024-09-17T04:49:42.669602+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T04:49:42.669602+0000","last_undegraded":"2024-09-17T04:49:42.669602+0000","last_fullsized":"2024-09-17T04:49:42.669602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050316+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T04:49:42.050316+0000","last_peered":"2024-09-17T04:49:42.050316+0000","last_clean":"2024-09-17T04:49:42.050316+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T04:49:42.050316+0000","last_undegraded":"2024-09-17T04:49:42.050316+0000","last_fullsized":"2024-09-17T04:49:42.050316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042123+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T04:49:42.042123+0000","last_peered":"2024-09-17T04:49:42.042123+0000","last_clean":"2024-09-17T04:49:42.042123+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T04:49:42.042123+0000","last_undegraded":"2024-09-17T04:49:42.042123+0000","last_fullsized":"2024-09-17T04:49:42.042123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052378+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T04:49:42.052378+0000","last_peered":"2024-09-17T04:49:42.052378+0000","last_clean":"2024-09-17T04:49:42.052378+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T04:49:42.052378+0000","last_undegraded":"2024-09-17T04:49:42.052378+0000","last_fullsized":"2024-09-17T04:49:42.052378+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041588+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T04:49:42.041588+0000","last_peered":"2024-09-17T04:49:42.041588+0000","last_clean":"2024-09-17T04:49:42.041588+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T04:49:42.041588+0000","last_undegraded":"2024-09-17T04:49:42.041588+0000","last_fullsized":"2024-09-17T04:49:42.041588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674897+0000","last_change":"2024-09-17T04:49:41.067207+0000","last_active":"2024-09-17T04:49:42.674897+0000","last_peered":"2024-09-17T04:49:42.674897+0000","last_clean":"2024-09-17T04:49:42.674897+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T04:49:42.674897+0000","last_undegraded":"2024-09-17T04:49:42.674897+0000","last_fullsized":"2024-09-17T04:49:42.674897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:08.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667484+0000","last_change":"2024-09-17T04:49:31.911036+0000","last_active":"2024-09-17T04:49:42.667484+0000","last_peered":"2024-09-17T04:49:42.667484+0000","last_clean":"2024-09-17T04:49:42.667484+0000","last_became_active":"2024-09-17T04:49:31.910285+0000","last_became_peered":"2024-09-17T04:49:31.910285+0000","last_unstale":"2024-09-17T04:49:42.667484+0000","last_undegraded":"2024-09-17T04:49:42.667484+0000","last_fullsized":"2024-09-17T04:49:42.667484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673474+0000","last_change":"2024-09-17T04:49:41.046170+0000","last_active":"2024-09-17T04:49:42.673474+0000","last_peered":"2024-09-17T04:49:42.673474+0000","last_clean":"2024-09-17T04:49:42.673474+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T04:49:42.673474+0000","last_undegraded":"2024-09-17T04:49:42.673474+0000","last_fullsized":"2024-09-17T04:49:42.673474+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:01.683826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051528+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T04:49:42.051528+0000","last_peered":"2024-09-17T04:49:42.051528+0000","last_clean":"2024-09-17T04:49:42.051528+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T04:49:42.051528+0000","last_undegraded":"2024-09-17T04:49:42.051528+0000","last_fullsized":"2024-09-17T04:49:42.051528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041843+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T04:49:42.041843+0000","last_peered":"2024-09-17T04:49:42.041843+0000","last_clean":"2024-09-17T04:49:42.041843+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T04:49:42.041843+0000","last_undegraded":"2024-09-17T04:49:42.041843+0000","last_fullsized":"2024-09-17T04:49:42.041843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:25.574682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053942+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T04:49:42.053942+0000","last_peered":"2024-09-17T04:49:42.053942+0000","last_clean":"2024-09-17T04:49:42.053942+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T04:49:42.053942+0000","last_undegraded":"2024-09-17T04:49:42.053942+0000","last_fullsized":"2024-09-17T04:49:42.053942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048414+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T04:49:42.048414+0000","last_peered":"2024-09-17T04:49:42.048414+0000","last_clean":"2024-09-17T04:49:42.048414+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T04:49:42.048414+0000","last_undegraded":"2024-09-17T04:49:42.048414+0000","last_fullsized":"2024-09-17T04:49:42.048414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671218+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T04:49:42.671218+0000","last_peered":"2024-09-17T04:49:42.671218+0000","last_clean":"2024-09-17T04:49:42.671218+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T04:49:42.671218+0000","last_undegraded":"2024-09-17T04:49:42.671218+0000","last_fullsized":"2024-09-17T04:49:42.671218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053498+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T04:49:42.053498+0000","last_peered":"2024-09-17T04:49:42.053498+0000","last_clean":"2024-09-17T04:49:42.053498+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T04:49:42.053498+0000","last_undegraded":"2024-09-17T04:49:42.053498+0000","last_fullsized":"2024-09-17T04:49:42.053498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670119+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T04:49:42.670119+0000","last_peered":"2024-09-17T04:49:42.670119+0000","last_clean":"2024-09-17T04:49:42.670119+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T04:49:42.670119+0000","last_undegraded":"2024-09-17T04:49:42.670119+0000","last_fullsized":"2024-09-17T04:49:42.670119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668585+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T04:49:42.668585+0000","last_peered":"2024-09-17T04:49:42.668585+0000","last_clean":"2024-09-17T04:49:42.668585+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T04:49:42.668585+0000","last_undegraded":"2024-09-17T04:49:42.668585+0000","last_fullsized":"2024-09-17T04:49:42.668585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670988+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T04:49:42.670988+0000","last_peered":"2024-09-17T04:49:42.670988+0000","last_clean":"2024-09-17T04:49:42.670988+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T04:49:42.670988+0000","last_undegraded":"2024-09-17T04:49:42.670988+0000","last_fullsized":"2024-09-17T04:49:42.670988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671416+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.671416+0000","last_peered":"2024-09-17T04:49:42.671416+0000","last_clean":"2024-09-17T04:49:42.671416+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T04:49:42.671416+0000","last_undegraded":"2024-09-17T04:49:42.671416+0000","last_fullsized":"2024-09-17T04:49:42.671416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675766+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T04:49:42.675766+0000","last_peered":"2024-09-17T04:49:42.675766+0000","last_clean":"2024-09-17T04:49:42.675766+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T04:49:42.675766+0000","last_undegraded":"2024-09-17T04:49:42.675766+0000","last_fullsized":"2024-09-17T04:49:42.675766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671330+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T04:49:42.671330+0000","last_peered":"2024-09-17T04:49:42.671330+0000","last_clean":"2024-09-17T04:49:42.671330+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T04:49:42.671330+0000","last_undegraded":"2024-09-17T04:49:42.671330+0000","last_fullsized":"2024-09-17T04:49:42.671330+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667310+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T04:49:42.667310+0000","last_peered":"2024-09-17T04:49:42.667310+0000","last_clean":"2024-09-17T04:49:42.667310+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T04:49:42.667310+0000","last_undegraded":"2024-09-17T04:49:42.667310+0000","last_fullsized":"2024-09-17T04:49:42.667310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537133+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T04:49:42.537133+0000","last_peered":"2024-09-17T04:49:42.537133+0000","last_clean":"2024-09-17T04:49:42.537133+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T04:49:42.537133+0000","last_undegraded":"2024-09-17T04:49:42.537133+0000","last_fullsized":"2024-09-17T04:49:42.537133+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669347+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T04:49:42.669347+0000","last_peered":"2024-09-17T04:49:42.669347+0000","last_clean":"2024-09-17T04:49:42.669347+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T04:49:42.669347+0000","last_undegraded":"2024-09-17T04:49:42.669347+0000","last_fullsized":"2024-09-17T04:49:42.669347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050809+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T04:49:42.050809+0000","last_peered":"2024-09-17T04:49:42.050809+0000","last_clean":"2024-09-17T04:49:42.050809+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T04:49:42.050809+0000","last_undegraded":"2024-09-17T04:49:42.050809+0000","last_fullsized":"2024-09-17T04:49:42.050809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535865+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T04:49:42.535865+0000","last_peered":"2024-09-17T04:49:42.535865+0000","last_clean":"2024-09-17T04:49:42.535865+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T04:49:42.535865+0000","last_undegraded":"2024-09-17T04:49:42.535865+0000","last_fullsized":"2024-09-17T04:49:42.535865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673274+0000","last_change":"2024-09-17T04:49:41.041201+0000","last_active":"2024-09-17T04:49:42.673274+0000","last_peered":"2024-09-17T04:49:42.673274+0000","last_clean":"2024-09-17T04:49:42.673274+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T04:49:42.673274+0000","last_undegraded":"2024-09-17T04:49:42.673274+0000","last_fullsized":"2024-09-17T04:49:42.673274+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:02.588786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670272+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T04:49:42.670272+0000","last_peered":"2024-09-17T04:49:42.670272+0000","last_clean":"2024-09-17T04:49:42.670272+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T04:49:42.670272+0000","last_undegraded":"2024-09-17T04:49:42.670272+0000","last_fullsized":"2024-09-17T04:49:42.670272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667767+0000","last_change":"2024-09-17T04:49:41.061092+0000","last_active":"2024-09-17T04:49:42.667767+0000","last_peered":"2024-09-17T04:49:42.667767+0000","last_clean":"2024-09-17T04:49:42.667767+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T04:49:42.667767+0000","last_undegraded":"2024-09-17T04:49:42.667767+0000","last_fullsized":"2024-09-17T04:49:42.667767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:52.714614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536820+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T04:49:42.536820+0000","last_peered":"2024-09-17T04:49:42.536820+0000","last_clean":"2024-09-17T04:49:42.536820+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T04:49:42.536820+0000","last_undegraded":"2024-09-17T04:49:42.536820+0000","last_fullsized":"2024-09-17T04:49:42.536820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669397+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T04:49:42.669397+0000","last_peered":"2024-09-17T04:49:42.669397+0000","last_clean":"2024-09-17T04:49:42.669397+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T04:49:42.669397+0000","last_undegraded":"2024-09-17T04:49:42.669397+0000","last_fullsized":"2024-09-17T04:49:42.669397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671170+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T04:49:42.671170+0000","last_peered":"2024-09-17T04:49:42.671170+0000","last_clean":"2024-09-17T04:49:42.671170+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T04:49:42.671170+0000","last_undegraded":"2024-09-17T04:49:42.671170+0000","last_fullsized":"2024-09-17T04:49:42.671170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536018+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T04:49:42.536018+0000","last_peered":"2024-09-17T04:49:42.536018+0000","last_clean":"2024-09-17T04:49:42.536018+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T04:49:42.536018+0000","last_undegraded":"2024-09-17T04:49:42.536018+0000","last_fullsized":"2024-09-17T04:49:42.536018+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671602+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T04:49:42.671602+0000","last_peered":"2024-09-17T04:49:42.671602+0000","last_clean":"2024-09-17T04:49:42.671602+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T04:49:42.671602+0000","last_undegraded":"2024-09-17T04:49:42.671602+0000","last_fullsized":"2024-09-17T04:49:42.671602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534962+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T04:49:42.534962+0000","last_peered":"2024-09-17T04:49:42.534962+0000","last_clean":"2024-09-17T04:49:42.534962+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T04:49:42.534962+0000","last_undegraded":"2024-09-17T04:49:42.534962+0000","last_fullsized":"2024-09-17T04:49:42.534962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671825+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T04:49:42.671825+0000","last_peered":"2024-09-17T04:49:42.671825+0000","last_clean":"2024-09-17T04:49:42.671825+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T04:49:42.671825+0000","last_undegraded":"2024-09-17T04:49:42.671825+0000","last_fullsized":"2024-09-17T04:49:42.671825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509450,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509450,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509450,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":307200,"data_stored":108537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509450,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509450,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":286720,"data_stored":106265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509451,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509451,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542155,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:49:55.890 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:49:56.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.044+0000 7fe542674640 1 -- 172.21.15.150:0/1591990256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe524004e70 msgr2=0x7fe5240052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.044+0000 7fe542674640 1 --2- 172.21.15.150:0/1591990256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe524004e70 0x7fe5240052c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe53c0c2550 tx=0x7fe53401cee0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.044+0000 7fe540c11640 1 -- 172.21.15.150:0/1591990256 reap_dead start 2024-09-17T04:49:56.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.044+0000 7fe542674640 1 -- 172.21.15.150:0/1591990256 shutdown_connections 2024-09-17T04:49:56.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.044+0000 7fe542674640 1 -- 172.21.15.150:0/1591990256 >> 172.21.15.150:0/1591990256 conn(0x7fe53c0c4a00 msgr2=0x7fe53c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:56.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.044+0000 7fe542674640 1 -- 172.21.15.150:0/1591990256 shutdown_connections 2024-09-17T04:49:56.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.044+0000 7fe542674640 1 -- 172.21.15.150:0/1591990256 wait complete. 2024-09-17T04:49:56.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.044+0000 7fe542674640 1 Processor -- start 2024-09-17T04:49:56.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.044+0000 7fe542674640 1 -- start start 2024-09-17T04:49:56.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe542674640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe524004e70 0x7fe53c1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:56.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe542674640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe53c1f3b40 0x7fe53c070d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:56.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe542674640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53c071280 0x7fe53c157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:56.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe542674640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe53c0c7260 con 0x7fe53c071280 2024-09-17T04:49:56.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe542674640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe53c0c70e0 con 0x7fe53c1f3b40 2024-09-17T04:49:56.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe542674640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe53c162af0 con 0x7fe524004e70 2024-09-17T04:49:56.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe540c11640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53c071280 0x7fe53c157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:56.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe540c11640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53c071280 0x7fe53c157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33478/0 (socket says 172.21.15.150:33478) 2024-09-17T04:49:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe540c11640 1 -- 172.21.15.150:0/2614498497 learned_addr learned my addr 172.21.15.150:0/2614498497 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe53b7fe640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe53c1f3b40 0x7fe53c070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe53bfff640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe524004e70 0x7fe53c1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe540c11640 1 -- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe524004e70 msgr2=0x7fe53c1585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe540c11640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe524004e70 0x7fe53c1585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe540c11640 1 -- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe53c1f3b40 msgr2=0x7fe53c070d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe540c11640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe53c1f3b40 0x7fe53c070d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe540c11640 1 -- 172.21.15.150:0/2614498497 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe53c06f780 con 0x7fe53c071280 2024-09-17T04:49:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe540c11640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53c071280 0x7fe53c157b10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe534098890 tx=0x7fe534001d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe538ff9640 1 -- 172.21.15.150:0/2614498497 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5340a5b70 con 0x7fe53c071280 2024-09-17T04:49:56.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe538ff9640 1 -- 172.21.15.150:0/2614498497 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe534000e50 con 0x7fe53c071280 2024-09-17T04:49:56.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe542674640 1 -- 172.21.15.150:0/2614498497 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe53c1566c0 con 0x7fe53c071280 2024-09-17T04:49:56.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe538ff9640 1 -- 172.21.15.150:0/2614498497 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534001700 con 0x7fe53c071280 2024-09-17T04:49:56.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe542674640 1 -- 172.21.15.150:0/2614498497 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe53c14c600 con 0x7fe53c071280 2024-09-17T04:49:56.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe542674640 1 -- 172.21.15.150:0/2614498497 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe53c0c1330 con 0x7fe53c071280 2024-09-17T04:49:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe538ff9640 1 -- 172.21.15.150:0/2614498497 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe534006070 con 0x7fe53c071280 2024-09-17T04:49:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe538ff9640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe528073230 0x7fe528075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe538ff9640 1 -- 172.21.15.150:0/2614498497 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fe53412d5a0 con 0x7fe53c071280 2024-09-17T04:49:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.048+0000 7fe53bfff640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe528073230 0x7fe528075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.052+0000 7fe53bfff640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe528073230 0x7fe528075740 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe52c001100 tx=0x7fe52c001070 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:56.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.052+0000 7fe538ff9640 1 -- 172.21.15.150:0/2614498497 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe534016c90 con 0x7fe53c071280 2024-09-17T04:49:56.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.208+0000 7fe542674640 1 -- 172.21.15.150:0/2614498497 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe53c1563e0 con 0x7fe528073230 2024-09-17T04:49:56.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.232+0000 7fe538ff9640 1 -- 172.21.15.150:0/2614498497 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490297 (secure 0 0 0) 0x7fe53c1563e0 con 0x7fe528073230 2024-09-17T04:49:56.240 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:56.243 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T04:49:56.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.240+0000 7fe542674640 1 -- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe528073230 msgr2=0x7fe528075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.240+0000 7fe542674640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe528073230 0x7fe528075740 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe52c001100 tx=0x7fe52c001070 comp rx=0 tx=0).stop 2024-09-17T04:49:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.240+0000 7fe542674640 1 -- 172.21.15.150:0/2614498497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53c071280 msgr2=0x7fe53c157b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.240+0000 7fe542674640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53c071280 0x7fe53c157b10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe534098890 tx=0x7fe534001d50 comp rx=0 tx=0).stop 2024-09-17T04:49:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.240+0000 7fe542674640 1 -- 172.21.15.150:0/2614498497 shutdown_connections 2024-09-17T04:49:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.240+0000 7fe542674640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe528073230 0x7fe528075740 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.240+0000 7fe542674640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe53c1f3b40 0x7fe53c070d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.240+0000 7fe542674640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53c071280 0x7fe53c157b10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.240+0000 7fe542674640 1 --2- 172.21.15.150:0/2614498497 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe524004e70 0x7fe53c1585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.240+0000 7fe542674640 1 -- 172.21.15.150:0/2614498497 >> 172.21.15.150:0/2614498497 conn(0x7fe53c0c4a00 msgr2=0x7fe53c0c62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.240+0000 7fe542674640 1 -- 172.21.15.150:0/2614498497 shutdown_connections 2024-09-17T04:49:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.240+0000 7fe542674640 1 -- 172.21.15.150:0/2614498497 wait complete. 2024-09-17T04:49:56.263 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-17T04:49:54.951293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":191,"num_write_kb":1656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223812,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750908,"statfs":{"total":773094113280,"available":772864929792,"internally_reserved":0,"allocated":3481600,"data_stored":2027328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005969"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671077+0000","last_change":"2024-09-17T04:49:41.025271+0000","last_active":"2024-09-17T04:49:42.671077+0000","last_peered":"2024-09-17T04:49:42.671077+0000","last_clean":"2024-09-17T04:49:42.671077+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T04:49:42.671077+0000","last_undegraded":"2024-09-17T04:49:42.671077+0000","last_fullsized":"2024-09-17T04:49:42.671077+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:39.715602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054640+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T04:49:42.054640+0000","last_peered":"2024-09-17T04:49:42.054640+0000","last_clean":"2024-09-17T04:49:42.054640+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T04:49:42.054640+0000","last_undegraded":"2024-09-17T04:49:42.054640+0000","last_fullsized":"2024-09-17T04:49:42.054640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671706+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T04:49:42.671706+0000","last_peered":"2024-09-17T04:49:42.671706+0000","last_clean":"2024-09-17T04:49:42.671706+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T04:49:42.671706+0000","last_undegraded":"2024-09-17T04:49:42.671706+0000","last_fullsized":"2024-09-17T04:49:42.671706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668736+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T04:49:42.668736+0000","last_peered":"2024-09-17T04:49:42.668736+0000","last_clean":"2024-09-17T04:49:42.668736+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T04:49:42.668736+0000","last_undegraded":"2024-09-17T04:49:42.668736+0000","last_fullsized":"2024-09-17T04:49:42.668736+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:38.417840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047972+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T04:49:42.047972+0000","last_peered":"2024-09-17T04:49:42.047972+0000","last_clean":"2024-09-17T04:49:42.047972+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T04:49:42.047972+0000","last_undegraded":"2024-09-17T04:49:42.047972+0000","last_fullsized":"2024-09-17T04:49:42.047972+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672300+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T04:49:42.672300+0000","last_peered":"2024-09-17T04:49:42.672300+0000","last_clean":"2024-09-17T04:49:42.672300+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T04:49:42.672300+0000","last_undegraded":"2024-09-17T04:49:42.672300+0000","last_fullsized":"2024-09-17T04:49:42.672300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672014+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T04:49:42.672014+0000","last_peered":"2024-09-17T04:49:42.672014+0000","last_clean":"2024-09-17T04:49:42.672014+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T04:49:42.672014+0000","last_undegraded":"2024-09-17T04:49:42.672014+0000","last_fullsized":"2024-09-17T04:49:42.672014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535134+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T04:49:42.535134+0000","last_peered":"2024-09-17T04:49:42.535134+0000","last_clean":"2024-09-17T04:49:42.535134+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T04:49:42.535134+0000","last_undegraded":"2024-09-17T04:49:42.535134+0000","last_fullsized":"2024-09-17T04:49:42.535134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672206+0000","last_change":"2024-09-17T04:49:41.067317+0000","last_active":"2024-09-17T04:49:42.672206+0000","last_peered":"2024-09-17T04:49:42.672206+0000","last_clean":"2024-09-17T04:49:42.672206+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T04:49:42.672206+0000","last_undegraded":"2024-09-17T04:49:42.672206+0000","last_fullsized":"2024-09-17T04:49:42.672206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:37.637477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670707+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T04:49:42.670707+0000","last_peered":"2024-09-17T04:49:42.670707+0000","last_clean":"2024-09-17T04:49:42.670707+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T04:49:42.670707+0000","last_undegraded":"2024-09-17T04:49:42.670707+0000","last_fullsized":"2024-09-17T04:49:42.670707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674274+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T04:49:42.674274+0000","last_peered":"2024-09-17T04:49:42.674274+0000","last_clean":"2024-09-17T04:49:42.674274+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T04:49:42.674274+0000","last_undegraded":"2024-09-17T04:49:42.674274+0000","last_fullsized":"2024-09-17T04:49:42.674274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535118+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T04:49:42.535118+0000","last_peered":"2024-09-17T04:49:42.535118+0000","last_clean":"2024-09-17T04:49:42.535118+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T04:49:42.535118+0000","last_undegraded":"2024-09-17T04:49:42.535118+0000","last_fullsized":"2024-09-17T04:49:42.535118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535259+0000","last_change":"2024-09-17T04:49:41.043531+0000","last_active":"2024-09-17T04:49:42.535259+0000","last_peered":"2024-09-17T04:49:42.535259+0000","last_clean":"2024-09-17T04:49:42.535259+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T04:49:42.535259+0000","last_undegraded":"2024-09-17T04:49:42.535259+0000","last_fullsized":"2024-09-17T04:49:42.535259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672162+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T04:49:42.672162+0000","last_peered":"2024-09-17T04:49:42.672162+0000","last_clean":"2024-09-17T04:49:42.672162+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T04:49:42.672162+0000","last_undegraded":"2024-09-17T04:49:42.672162+0000","last_fullsized":"2024-09-17T04:49:42.672162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042945+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T04:49:42.042945+0000","last_peered":"2024-09-17T04:49:42.042945+0000","last_clean":"2024-09-17T04:49:42.042945+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T04:49:42.042945+0000","last_undegraded":"2024-09-17T04:49:42.042945+0000","last_fullsized":"2024-09-17T04:49:42.042945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535003+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T04:49:42.535003+0000","last_peered":"2024-09-17T04:49:42.535003+0000","last_clean":"2024-09-17T04:49:42.535003+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T04:49:42.535003+0000","last_undegraded":"2024-09-17T04:49:42.535003+0000","last_fullsized":"2024-09-17T04:49:42.535003+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:16.854365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669588+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T04:49:42.669588+0000","last_peered":"2024-09-17T04:49:42.669588+0000","last_clean":"2024-09-17T04:49:42.669588+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T04:49:42.669588+0000","last_undegraded":"2024-09-17T04:49:42.669588+0000","last_fullsized":"2024-09-17T04:49:42.669588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042526+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T04:49:42.042526+0000","last_peered":"2024-09-17T04:49:42.042526+0000","last_clean":"2024-09-17T04:49:42.042526+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T04:49:42.042526+0000","last_undegraded":"2024-09-17T04:49:42.042526+0000","last_fullsized":"2024-09-17T04:49:42.042526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041407+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T04:49:42.041407+0000","last_peered":"2024-09-17T04:49:42.041407+0000","last_clean":"2024-09-17T04:49:42.041407+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T04:49:42.041407+0000","last_undegraded":"2024-09-17T04:49:42.041407+0000","last_fullsized":"2024-09-17T04:49:42.041407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042319+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T04:49:42.042319+0000","last_peered":"2024-09-17T04:49:42.042319+0000","last_clean":"2024-09-17T04:49:42.042319+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T04:49:42.042319+0000","last_undegraded":"2024-09-17T04:49:42.042319+0000","last_fullsized":"2024-09-17T04:49:42.042319+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:28.365399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669339+0000","last_change":"2024-09-17T04:49:41.038415+0000","last_active":"2024-09-17T04:49:42.669339+0000","last_peered":"2024-09-17T04:49:42.669339+0000","last_clean":"2024-09-17T04:49:42.669339+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T04:49:42.669339+0000","last_undegraded":"2024-09-17T04:49:42.669339+0000","last_fullsized":"2024-09-17T04:49:42.669339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:17.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668616+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T04:49:42.668616+0000","last_peered":"2024-09-17T04:49:42.668616+0000","last_clean":"2024-09-17T04:49:42.668616+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T04:49:42.668616+0000","last_undegraded":"2024-09-17T04:49:42.668616+0000","last_fullsized":"2024-09-17T04:49:42.668616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052248+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T04:49:42.052248+0000","last_peered":"2024-09-17T04:49:42.052248+0000","last_clean":"2024-09-17T04:49:42.052248+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T04:49:42.052248+0000","last_undegraded":"2024-09-17T04:49:42.052248+0000","last_fullsized":"2024-09-17T04:49:42.052248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671586+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T04:49:42.671586+0000","last_peered":"2024-09-17T04:49:42.671586+0000","last_clean":"2024-09-17T04:49:42.671586+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T04:49:42.671586+0000","last_undegraded":"2024-09-17T04:49:42.671586+0000","last_fullsized":"2024-09-17T04:49:42.671586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:17.424265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042028+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T04:49:42.042028+0000","last_peered":"2024-09-17T04:49:42.042028+0000","last_clean":"2024-09-17T04:49:42.042028+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T04:49:42.042028+0000","last_undegraded":"2024-09-17T04:49:42.042028+0000","last_fullsized":"2024-09-17T04:49:42.042028+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:06.516432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048455+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T04:49:42.048455+0000","last_peered":"2024-09-17T04:49:42.048455+0000","last_clean":"2024-09-17T04:49:42.048455+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T04:49:42.048455+0000","last_undegraded":"2024-09-17T04:49:42.048455+0000","last_fullsized":"2024-09-17T04:49:42.048455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048500+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T04:49:42.048500+0000","last_peered":"2024-09-17T04:49:42.048500+0000","last_clean":"2024-09-17T04:49:42.048500+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T04:49:42.048500+0000","last_undegraded":"2024-09-17T04:49:42.048500+0000","last_fullsized":"2024-09-17T04:49:42.048500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668827+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T04:49:42.668827+0000","last_peered":"2024-09-17T04:49:42.668827+0000","last_clean":"2024-09-17T04:49:42.668827+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T04:49:42.668827+0000","last_undegraded":"2024-09-17T04:49:42.668827+0000","last_fullsized":"2024-09-17T04:49:42.668827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:00.010365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675406+0000","last_change":"2024-09-17T04:49:41.036658+0000","last_active":"2024-09-17T04:49:42.675406+0000","last_peered":"2024-09-17T04:49:42.675406+0000","last_clean":"2024-09-17T04:49:42.675406+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T04:49:42.675406+0000","last_undegraded":"2024-09-17T04:49:42.675406+0000","last_fullsized":"2024-09-17T04:49:42.675406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042824+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T04:49:42.042824+0000","last_peered":"2024-09-17T04:49:42.042824+0000","last_clean":"2024-09-17T04:49:42.042824+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T04:49:42.042824+0000","last_undegraded":"2024-09-17T04:49:42.042824+0000","last_fullsized":"2024-09-17T04:49:42.042824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668829+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T04:49:42.668829+0000","last_peered":"2024-09-17T04:49:42.668829+0000","last_clean":"2024-09-17T04:49:42.668829+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T04:49:42.668829+0000","last_undegraded":"2024-09-17T04:49:42.668829+0000","last_fullsized":"2024-09-17T04:49:42.668829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536259+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T04:49:42.536259+0000","last_peered":"2024-09-17T04:49:42.536259+0000","last_clean":"2024-09-17T04:49:42.536259+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T04:49:42.536259+0000","last_undegraded":"2024-09-17T04:49:42.536259+0000","last_fullsized":"2024-09-17T04:49:42.536259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048942+0000","last_change":"2024-09-17T04:49:41.050170+0000","last_active":"2024-09-17T04:49:42.048942+0000","last_peered":"2024-09-17T04:49:42.048942+0000","last_clean":"2024-09-17T04:49:42.048942+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T04:49:42.048942+0000","last_undegraded":"2024-09-17T04:49:42.048942+0000","last_fullsized":"2024-09-17T04:49:42.048942+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051929+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T04:49:42.051929+0000","last_peered":"2024-09-17T04:49:42.051929+0000","last_clean":"2024-09-17T04:49:42.051929+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T04:49:42.051929+0000","last_undegraded":"2024-09-17T04:49:42.051929+0000","last_fullsized":"2024-09-17T04:49:42.051929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042185+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T04:49:42.042185+0000","last_peered":"2024-09-17T04:49:42.042185+0000","last_clean":"2024-09-17T04:49:42.042185+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T04:49:42.042185+0000","last_undegraded":"2024-09-17T04:49:42.042185+0000","last_fullsized":"2024-09-17T04:49:42.042185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535553+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T04:49:42.535553+0000","last_peered":"2024-09-17T04:49:42.535553+0000","last_clean":"2024-09-17T04:49:42.535553+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T04:49:42.535553+0000","last_undegraded":"2024-09-17T04:49:42.535553+0000","last_fullsized":"2024-09-17T04:49:42.535553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041211+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T04:49:42.041211+0000","last_peered":"2024-09-17T04:49:42.041211+0000","last_clean":"2024-09-17T04:49:42.041211+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T04:49:42.041211+0000","last_undegraded":"2024-09-17T04:49:42.041211+0000","last_fullsized":"2024-09-17T04:49:42.041211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054336+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T04:49:42.054336+0000","last_peered":"2024-09-17T04:49:42.054336+0000","last_clean":"2024-09-17T04:49:42.054336+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T04:49:42.054336+0000","last_undegraded":"2024-09-17T04:49:42.054336+0000","last_fullsized":"2024-09-17T04:49:42.054336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052826+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T04:49:42.052826+0000","last_peered":"2024-09-17T04:49:42.052826+0000","last_clean":"2024-09-17T04:49:42.052826+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T04:49:42.052826+0000","last_undegraded":"2024-09-17T04:49:42.052826+0000","last_fullsized":"2024-09-17T04:49:42.052826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048145+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T04:49:42.048145+0000","last_peered":"2024-09-17T04:49:42.048145+0000","last_clean":"2024-09-17T04:49:42.048145+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T04:49:42.048145+0000","last_undegraded":"2024-09-17T04:49:42.048145+0000","last_fullsized":"2024-09-17T04:49:42.048145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669693+0000","last_change":"2024-09-17T04:49:41.042314+0000","last_active":"2024-09-17T04:49:42.669693+0000","last_peered":"2024-09-17T04:49:42.669693+0000","last_clean":"2024-09-17T04:49:42.669693+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T04:49:42.669693+0000","last_undegraded":"2024-09-17T04:49:42.669693+0000","last_fullsized":"2024-09-17T04:49:42.669693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055299+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T04:49:42.055299+0000","last_peered":"2024-09-17T04:49:42.055299+0000","last_clean":"2024-09-17T04:49:42.055299+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T04:49:42.055299+0000","last_undegraded":"2024-09-17T04:49:42.055299+0000","last_fullsized":"2024-09-17T04:49:42.055299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043663+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T04:49:42.043663+0000","last_peered":"2024-09-17T04:49:42.043663+0000","last_clean":"2024-09-17T04:49:42.043663+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T04:49:42.043663+0000","last_undegraded":"2024-09-17T04:49:42.043663+0000","last_fullsized":"2024-09-17T04:49:42.043663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053169+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T04:49:42.053169+0000","last_peered":"2024-09-17T04:49:42.053169+0000","last_clean":"2024-09-17T04:49:42.053169+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T04:49:42.053169+0000","last_undegraded":"2024-09-17T04:49:42.053169+0000","last_fullsized":"2024-09-17T04:49:42.053169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052959+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T04:49:42.052959+0000","last_peered":"2024-09-17T04:49:42.052959+0000","last_clean":"2024-09-17T04:49:42.052959+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T04:49:42.052959+0000","last_undegraded":"2024-09-17T04:49:42.052959+0000","last_fullsized":"2024-09-17T04:49:42.052959+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535986+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T04:49:42.535986+0000","last_peered":"2024-09-17T04:49:42.535986+0000","last_clean":"2024-09-17T04:49:42.535986+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T04:49:42.535986+0000","last_undegraded":"2024-09-17T04:49:42.535986+0000","last_fullsized":"2024-09-17T04:49:42.535986+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670900+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T04:49:42.670900+0000","last_peered":"2024-09-17T04:49:42.670900+0000","last_clean":"2024-09-17T04:49:42.670900+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T04:49:42.670900+0000","last_undegraded":"2024-09-17T04:49:42.670900+0000","last_fullsized":"2024-09-17T04:49:42.670900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050191+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T04:49:42.050191+0000","last_peered":"2024-09-17T04:49:42.050191+0000","last_clean":"2024-09-17T04:49:42.050191+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T04:49:42.050191+0000","last_undegraded":"2024-09-17T04:49:42.050191+0000","last_fullsized":"2024-09-17T04:49:42.050191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670215+0000","last_change":"2024-09-17T04:49:41.069177+0000","last_active":"2024-09-17T04:49:42.670215+0000","last_peered":"2024-09-17T04:49:42.670215+0000","last_clean":"2024-09-17T04:49:42.670215+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T04:49:42.670215+0000","last_undegraded":"2024-09-17T04:49:42.670215+0000","last_fullsized":"2024-09-17T04:49:42.670215+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:16.855206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670381+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T04:49:42.670381+0000","last_peered":"2024-09-17T04:49:42.670381+0000","last_clean":"2024-09-17T04:49:42.670381+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T04:49:42.670381+0000","last_undegraded":"2024-09-17T04:49:42.670381+0000","last_fullsized":"2024-09-17T04:49:42.670381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042706+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T04:49:42.042706+0000","last_peered":"2024-09-17T04:49:42.042706+0000","last_clean":"2024-09-17T04:49:42.042706+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T04:49:42.042706+0000","last_undegraded":"2024-09-17T04:49:42.042706+0000","last_fullsized":"2024-09-17T04:49:42.042706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054900+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T04:49:42.054900+0000","last_peered":"2024-09-17T04:49:42.054900+0000","last_clean":"2024-09-17T04:49:42.054900+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T04:49:42.054900+0000","last_undegraded":"2024-09-17T04:49:42.054900+0000","last_fullsized":"2024-09-17T04:49:42.054900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669308+0000","last_change":"2024-09-17T04:49:41.068537+0000","last_active":"2024-09-17T04:49:42.669308+0000","last_peered":"2024-09-17T04:49:42.669308+0000","last_clean":"2024-09-17T04:49:42.669308+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T04:49:42.669308+0000","last_undegraded":"2024-09-17T04:49:42.669308+0000","last_fullsized":"2024-09-17T04:49:42.669308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667397+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T04:49:42.667397+0000","last_peered":"2024-09-17T04:49:42.667397+0000","last_clean":"2024-09-17T04:49:42.667397+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T04:49:42.667397+0000","last_undegraded":"2024-09-17T04:49:42.667397+0000","last_fullsized":"2024-09-17T04:49:42.667397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668940+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T04:49:42.668940+0000","last_peered":"2024-09-17T04:49:42.668940+0000","last_clean":"2024-09-17T04:49:42.668940+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T04:49:42.668940+0000","last_undegraded":"2024-09-17T04:49:42.668940+0000","last_fullsized":"2024-09-17T04:49:42.668940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054921+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T04:49:42.054921+0000","last_peered":"2024-09-17T04:49:42.054921+0000","last_clean":"2024-09-17T04:49:42.054921+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T04:49:42.054921+0000","last_undegraded":"2024-09-17T04:49:42.054921+0000","last_fullsized":"2024-09-17T04:49:42.054921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041762+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T04:49:42.041762+0000","last_peered":"2024-09-17T04:49:42.041762+0000","last_clean":"2024-09-17T04:49:42.041762+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T04:49:42.041762+0000","last_undegraded":"2024-09-17T04:49:42.041762+0000","last_fullsized":"2024-09-17T04:49:42.041762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050299+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T04:49:42.050299+0000","last_peered":"2024-09-17T04:49:42.050299+0000","last_clean":"2024-09-17T04:49:42.050299+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T04:49:42.050299+0000","last_undegraded":"2024-09-17T04:49:42.050299+0000","last_fullsized":"2024-09-17T04:49:42.050299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041771+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T04:49:42.041771+0000","last_peered":"2024-09-17T04:49:42.041771+0000","last_clean":"2024-09-17T04:49:42.041771+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T04:49:42.041771+0000","last_undegraded":"2024-09-17T04:49:42.041771+0000","last_fullsized":"2024-09-17T04:49:42.041771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672234+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T04:49:42.672234+0000","last_peered":"2024-09-17T04:49:42.672234+0000","last_clean":"2024-09-17T04:49:42.672234+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T04:49:42.672234+0000","last_undegraded":"2024-09-17T04:49:42.672234+0000","last_fullsized":"2024-09-17T04:49:42.672234+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671183+0000","last_change":"2024-09-17T04:49:41.052564+0000","last_active":"2024-09-17T04:49:42.671183+0000","last_peered":"2024-09-17T04:49:42.671183+0000","last_clean":"2024-09-17T04:49:42.671183+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T04:49:42.671183+0000","last_undegraded":"2024-09-17T04:49:42.671183+0000","last_fullsized":"2024-09-17T04:49:42.671183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049438+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T04:49:42.049438+0000","last_peered":"2024-09-17T04:49:42.049438+0000","last_clean":"2024-09-17T04:49:42.049438+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T04:49:42.049438+0000","last_undegraded":"2024-09-17T04:49:42.049438+0000","last_fullsized":"2024-09-17T04:49:42.049438+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041458+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T04:49:42.041458+0000","last_peered":"2024-09-17T04:49:42.041458+0000","last_clean":"2024-09-17T04:49:42.041458+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T04:49:42.041458+0000","last_undegraded":"2024-09-17T04:49:42.041458+0000","last_fullsized":"2024-09-17T04:49:42.041458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671735+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T04:49:42.671735+0000","last_peered":"2024-09-17T04:49:42.671735+0000","last_clean":"2024-09-17T04:49:42.671735+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T04:49:42.671735+0000","last_undegraded":"2024-09-17T04:49:42.671735+0000","last_fullsized":"2024-09-17T04:49:42.671735+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043078+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T04:49:42.043078+0000","last_peered":"2024-09-17T04:49:42.043078+0000","last_clean":"2024-09-17T04:49:42.043078+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T04:49:42.043078+0000","last_undegraded":"2024-09-17T04:49:42.043078+0000","last_fullsized":"2024-09-17T04:49:42.043078+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:21.202398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669578+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T04:49:42.669578+0000","last_peered":"2024-09-17T04:49:42.669578+0000","last_clean":"2024-09-17T04:49:42.669578+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T04:49:42.669578+0000","last_undegraded":"2024-09-17T04:49:42.669578+0000","last_fullsized":"2024-09-17T04:49:42.669578+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:48.128609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043280+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T04:49:42.043280+0000","last_peered":"2024-09-17T04:49:42.043280+0000","last_clean":"2024-09-17T04:49:42.043280+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T04:49:42.043280+0000","last_undegraded":"2024-09-17T04:49:42.043280+0000","last_fullsized":"2024-09-17T04:49:42.043280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055292+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T04:49:42.055292+0000","last_peered":"2024-09-17T04:49:42.055292+0000","last_clean":"2024-09-17T04:49:42.055292+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T04:49:42.055292+0000","last_undegraded":"2024-09-17T04:49:42.055292+0000","last_fullsized":"2024-09-17T04:49:42.055292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536164+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T04:49:42.536164+0000","last_peered":"2024-09-17T04:49:42.536164+0000","last_clean":"2024-09-17T04:49:42.536164+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T04:49:42.536164+0000","last_undegraded":"2024-09-17T04:49:42.536164+0000","last_fullsized":"2024-09-17T04:49:42.536164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050257+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T04:49:42.050257+0000","last_peered":"2024-09-17T04:49:42.050257+0000","last_clean":"2024-09-17T04:49:42.050257+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T04:49:42.050257+0000","last_undegraded":"2024-09-17T04:49:42.050257+0000","last_fullsized":"2024-09-17T04:49:42.050257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671927+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T04:49:42.671927+0000","last_peered":"2024-09-17T04:49:42.671927+0000","last_clean":"2024-09-17T04:49:42.671927+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T04:49:42.671927+0000","last_undegraded":"2024-09-17T04:49:42.671927+0000","last_fullsized":"2024-09-17T04:49:42.671927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671411+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T04:49:42.671411+0000","last_peered":"2024-09-17T04:49:42.671411+0000","last_clean":"2024-09-17T04:49:42.671411+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T04:49:42.671411+0000","last_undegraded":"2024-09-17T04:49:42.671411+0000","last_fullsized":"2024-09-17T04:49:42.671411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670087+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T04:49:42.670087+0000","last_peered":"2024-09-17T04:49:42.670087+0000","last_clean":"2024-09-17T04:49:42.670087+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T04:49:42.670087+0000","last_undegraded":"2024-09-17T04:49:42.670087+0000","last_fullsized":"2024-09-17T04:49:42.670087+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:18.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672287+0000","last_change":"2024-09-17T04:49:41.031036+0000","last_active":"2024-09-17T04:49:42.672287+0000","last_peered":"2024-09-17T04:49:42.672287+0000","last_clean":"2024-09-17T04:49:42.672287+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T04:49:42.672287+0000","last_undegraded":"2024-09-17T04:49:42.672287+0000","last_fullsized":"2024-09-17T04:49:42.672287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:48.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672111+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T04:49:42.672111+0000","last_peered":"2024-09-17T04:49:42.672111+0000","last_clean":"2024-09-17T04:49:42.672111+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T04:49:42.672111+0000","last_undegraded":"2024-09-17T04:49:42.672111+0000","last_fullsized":"2024-09-17T04:49:42.672111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052688+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T04:49:42.052688+0000","last_peered":"2024-09-17T04:49:42.052688+0000","last_clean":"2024-09-17T04:49:42.052688+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T04:49:42.052688+0000","last_undegraded":"2024-09-17T04:49:42.052688+0000","last_fullsized":"2024-09-17T04:49:42.052688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041095+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T04:49:42.041095+0000","last_peered":"2024-09-17T04:49:42.041095+0000","last_clean":"2024-09-17T04:49:42.041095+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T04:49:42.041095+0000","last_undegraded":"2024-09-17T04:49:42.041095+0000","last_fullsized":"2024-09-17T04:49:42.041095+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048488+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T04:49:42.048488+0000","last_peered":"2024-09-17T04:49:42.048488+0000","last_clean":"2024-09-17T04:49:42.048488+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T04:49:42.048488+0000","last_undegraded":"2024-09-17T04:49:42.048488+0000","last_fullsized":"2024-09-17T04:49:42.048488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536967+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T04:49:42.536967+0000","last_peered":"2024-09-17T04:49:42.536967+0000","last_clean":"2024-09-17T04:49:42.536967+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T04:49:42.536967+0000","last_undegraded":"2024-09-17T04:49:42.536967+0000","last_fullsized":"2024-09-17T04:49:42.536967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672404+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T04:49:42.672404+0000","last_peered":"2024-09-17T04:49:42.672404+0000","last_clean":"2024-09-17T04:49:42.672404+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T04:49:42.672404+0000","last_undegraded":"2024-09-17T04:49:42.672404+0000","last_fullsized":"2024-09-17T04:49:42.672404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672966+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T04:49:42.672966+0000","last_peered":"2024-09-17T04:49:42.672966+0000","last_clean":"2024-09-17T04:49:42.672966+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T04:49:42.672966+0000","last_undegraded":"2024-09-17T04:49:42.672966+0000","last_fullsized":"2024-09-17T04:49:42.672966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:09.712272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052007+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T04:49:42.052007+0000","last_peered":"2024-09-17T04:49:42.052007+0000","last_clean":"2024-09-17T04:49:42.052007+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T04:49:42.052007+0000","last_undegraded":"2024-09-17T04:49:42.052007+0000","last_fullsized":"2024-09-17T04:49:42.052007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:43.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042787+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T04:49:42.042787+0000","last_peered":"2024-09-17T04:49:42.042787+0000","last_clean":"2024-09-17T04:49:42.042787+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T04:49:42.042787+0000","last_undegraded":"2024-09-17T04:49:42.042787+0000","last_fullsized":"2024-09-17T04:49:42.042787+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049500+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T04:49:42.049500+0000","last_peered":"2024-09-17T04:49:42.049500+0000","last_clean":"2024-09-17T04:49:42.049500+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T04:49:42.049500+0000","last_undegraded":"2024-09-17T04:49:42.049500+0000","last_fullsized":"2024-09-17T04:49:42.049500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671253+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T04:49:42.671253+0000","last_peered":"2024-09-17T04:49:42.671253+0000","last_clean":"2024-09-17T04:49:42.671253+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T04:49:42.671253+0000","last_undegraded":"2024-09-17T04:49:42.671253+0000","last_fullsized":"2024-09-17T04:49:42.671253+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:52.364568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041504+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T04:49:42.041504+0000","last_peered":"2024-09-17T04:49:42.041504+0000","last_clean":"2024-09-17T04:49:42.041504+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T04:49:42.041504+0000","last_undegraded":"2024-09-17T04:49:42.041504+0000","last_fullsized":"2024-09-17T04:49:42.041504+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:03.537392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671439+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T04:49:42.671439+0000","last_peered":"2024-09-17T04:49:42.671439+0000","last_clean":"2024-09-17T04:49:42.671439+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T04:49:42.671439+0000","last_undegraded":"2024-09-17T04:49:42.671439+0000","last_fullsized":"2024-09-17T04:49:42.671439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670943+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T04:49:42.670943+0000","last_peered":"2024-09-17T04:49:42.670943+0000","last_clean":"2024-09-17T04:49:42.670943+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T04:49:42.670943+0000","last_undegraded":"2024-09-17T04:49:42.670943+0000","last_fullsized":"2024-09-17T04:49:42.670943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671787+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T04:49:42.671787+0000","last_peered":"2024-09-17T04:49:42.671787+0000","last_clean":"2024-09-17T04:49:42.671787+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T04:49:42.671787+0000","last_undegraded":"2024-09-17T04:49:42.671787+0000","last_fullsized":"2024-09-17T04:49:42.671787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:52.120543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041402+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T04:49:42.041402+0000","last_peered":"2024-09-17T04:49:42.041402+0000","last_clean":"2024-09-17T04:49:42.041402+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T04:49:42.041402+0000","last_undegraded":"2024-09-17T04:49:42.041402+0000","last_fullsized":"2024-09-17T04:49:42.041402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:56.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041285+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T04:49:42.041285+0000","last_peered":"2024-09-17T04:49:42.041285+0000","last_clean":"2024-09-17T04:49:42.041285+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T04:49:42.041285+0000","last_undegraded":"2024-09-17T04:49:42.041285+0000","last_fullsized":"2024-09-17T04:49:42.041285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042566+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T04:49:42.042566+0000","last_peered":"2024-09-17T04:49:42.042566+0000","last_clean":"2024-09-17T04:49:42.042566+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T04:49:42.042566+0000","last_undegraded":"2024-09-17T04:49:42.042566+0000","last_fullsized":"2024-09-17T04:49:42.042566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054288+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T04:49:42.054288+0000","last_peered":"2024-09-17T04:49:42.054288+0000","last_clean":"2024-09-17T04:49:42.054288+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T04:49:42.054288+0000","last_undegraded":"2024-09-17T04:49:42.054288+0000","last_fullsized":"2024-09-17T04:49:42.054288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051153+0000","last_change":"2024-09-17T04:49:41.058280+0000","last_active":"2024-09-17T04:49:42.051153+0000","last_peered":"2024-09-17T04:49:42.051153+0000","last_clean":"2024-09-17T04:49:42.051153+0000","last_became_active":"2024-09-17T04:49:41.052009+0000","last_became_peered":"2024-09-17T04:49:41.052009+0000","last_unstale":"2024-09-17T04:49:42.051153+0000","last_undegraded":"2024-09-17T04:49:42.051153+0000","last_fullsized":"2024-09-17T04:49:42.051153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:14.100818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051977+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T04:49:42.051977+0000","last_peered":"2024-09-17T04:49:42.051977+0000","last_clean":"2024-09-17T04:49:42.051977+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T04:49:42.051977+0000","last_undegraded":"2024-09-17T04:49:42.051977+0000","last_fullsized":"2024-09-17T04:49:42.051977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669355+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T04:49:42.669355+0000","last_peered":"2024-09-17T04:49:42.669355+0000","last_clean":"2024-09-17T04:49:42.669355+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T04:49:42.669355+0000","last_undegraded":"2024-09-17T04:49:42.669355+0000","last_fullsized":"2024-09-17T04:49:42.669355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042782+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T04:49:42.042782+0000","last_peered":"2024-09-17T04:49:42.042782+0000","last_clean":"2024-09-17T04:49:42.042782+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T04:49:42.042782+0000","last_undegraded":"2024-09-17T04:49:42.042782+0000","last_fullsized":"2024-09-17T04:49:42.042782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:07.945355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052280+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T04:49:42.052280+0000","last_peered":"2024-09-17T04:49:42.052280+0000","last_clean":"2024-09-17T04:49:42.052280+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T04:49:42.052280+0000","last_undegraded":"2024-09-17T04:49:42.052280+0000","last_fullsized":"2024-09-17T04:49:42.052280+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670412+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T04:49:42.670412+0000","last_peered":"2024-09-17T04:49:42.670412+0000","last_clean":"2024-09-17T04:49:42.670412+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T04:49:42.670412+0000","last_undegraded":"2024-09-17T04:49:42.670412+0000","last_fullsized":"2024-09-17T04:49:42.670412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537054+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T04:49:42.537054+0000","last_peered":"2024-09-17T04:49:42.537054+0000","last_clean":"2024-09-17T04:49:42.537054+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T04:49:42.537054+0000","last_undegraded":"2024-09-17T04:49:42.537054+0000","last_fullsized":"2024-09-17T04:49:42.537054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048913+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T04:49:42.048913+0000","last_peered":"2024-09-17T04:49:42.048913+0000","last_clean":"2024-09-17T04:49:42.048913+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T04:49:42.048913+0000","last_undegraded":"2024-09-17T04:49:42.048913+0000","last_fullsized":"2024-09-17T04:49:42.048913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:16.447835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051373+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T04:49:42.051373+0000","last_peered":"2024-09-17T04:49:42.051373+0000","last_clean":"2024-09-17T04:49:42.051373+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T04:49:42.051373+0000","last_undegraded":"2024-09-17T04:49:42.051373+0000","last_fullsized":"2024-09-17T04:49:42.051373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042063+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T04:49:42.042063+0000","last_peered":"2024-09-17T04:49:42.042063+0000","last_clean":"2024-09-17T04:49:42.042063+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T04:49:42.042063+0000","last_undegraded":"2024-09-17T04:49:42.042063+0000","last_fullsized":"2024-09-17T04:49:42.042063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670596+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T04:49:42.670596+0000","last_peered":"2024-09-17T04:49:42.670596+0000","last_clean":"2024-09-17T04:49:42.670596+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T04:49:42.670596+0000","last_undegraded":"2024-09-17T04:49:42.670596+0000","last_fullsized":"2024-09-17T04:49:42.670596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535510+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T04:49:42.535510+0000","last_peered":"2024-09-17T04:49:42.535510+0000","last_clean":"2024-09-17T04:49:42.535510+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T04:49:42.535510+0000","last_undegraded":"2024-09-17T04:49:42.535510+0000","last_fullsized":"2024-09-17T04:49:42.535510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041322+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T04:49:42.041322+0000","last_peered":"2024-09-17T04:49:42.041322+0000","last_clean":"2024-09-17T04:49:42.041322+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T04:49:42.041322+0000","last_undegraded":"2024-09-17T04:49:42.041322+0000","last_fullsized":"2024-09-17T04:49:42.041322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672182+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T04:49:42.672182+0000","last_peered":"2024-09-17T04:49:42.672182+0000","last_clean":"2024-09-17T04:49:42.672182+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T04:49:42.672182+0000","last_undegraded":"2024-09-17T04:49:42.672182+0000","last_fullsized":"2024-09-17T04:49:42.672182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671261+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T04:49:42.671261+0000","last_peered":"2024-09-17T04:49:42.671261+0000","last_clean":"2024-09-17T04:49:42.671261+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T04:49:42.671261+0000","last_undegraded":"2024-09-17T04:49:42.671261+0000","last_fullsized":"2024-09-17T04:49:42.671261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667575+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T04:49:42.667575+0000","last_peered":"2024-09-17T04:49:42.667575+0000","last_clean":"2024-09-17T04:49:42.667575+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T04:49:42.667575+0000","last_undegraded":"2024-09-17T04:49:42.667575+0000","last_fullsized":"2024-09-17T04:49:42.667575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:40.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534654+0000","last_change":"2024-09-17T04:49:41.040278+0000","last_active":"2024-09-17T04:49:42.534654+0000","last_peered":"2024-09-17T04:49:42.534654+0000","last_clean":"2024-09-17T04:49:42.534654+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T04:49:42.534654+0000","last_undegraded":"2024-09-17T04:49:42.534654+0000","last_fullsized":"2024-09-17T04:49:42.534654+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.895223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536732+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T04:49:42.536732+0000","last_peered":"2024-09-17T04:49:42.536732+0000","last_clean":"2024-09-17T04:49:42.536732+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T04:49:42.536732+0000","last_undegraded":"2024-09-17T04:49:42.536732+0000","last_fullsized":"2024-09-17T04:49:42.536732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675753+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T04:49:42.675753+0000","last_peered":"2024-09-17T04:49:42.675753+0000","last_clean":"2024-09-17T04:49:42.675753+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T04:49:42.675753+0000","last_undegraded":"2024-09-17T04:49:42.675753+0000","last_fullsized":"2024-09-17T04:49:42.675753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047646+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T04:49:42.047646+0000","last_peered":"2024-09-17T04:49:42.047646+0000","last_clean":"2024-09-17T04:49:42.047646+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T04:49:42.047646+0000","last_undegraded":"2024-09-17T04:49:42.047646+0000","last_fullsized":"2024-09-17T04:49:42.047646+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:15.845612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537298+0000","last_change":"2024-09-17T04:49:41.050551+0000","last_active":"2024-09-17T04:49:42.537298+0000","last_peered":"2024-09-17T04:49:42.537298+0000","last_clean":"2024-09-17T04:49:42.537298+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T04:49:42.537298+0000","last_undegraded":"2024-09-17T04:49:42.537298+0000","last_fullsized":"2024-09-17T04:49:42.537298+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:11.088230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051592+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T04:49:42.051592+0000","last_peered":"2024-09-17T04:49:42.051592+0000","last_clean":"2024-09-17T04:49:42.051592+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T04:49:42.051592+0000","last_undegraded":"2024-09-17T04:49:42.051592+0000","last_fullsized":"2024-09-17T04:49:42.051592+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:37.778756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048401+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T04:49:42.048401+0000","last_peered":"2024-09-17T04:49:42.048401+0000","last_clean":"2024-09-17T04:49:42.048401+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T04:49:42.048401+0000","last_undegraded":"2024-09-17T04:49:42.048401+0000","last_fullsized":"2024-09-17T04:49:42.048401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041797+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T04:49:42.041797+0000","last_peered":"2024-09-17T04:49:42.041797+0000","last_clean":"2024-09-17T04:49:42.041797+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T04:49:42.041797+0000","last_undegraded":"2024-09-17T04:49:42.041797+0000","last_fullsized":"2024-09-17T04:49:42.041797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055365+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T04:49:42.055365+0000","last_peered":"2024-09-17T04:49:42.055365+0000","last_clean":"2024-09-17T04:49:42.055365+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T04:49:42.055365+0000","last_undegraded":"2024-09-17T04:49:42.055365+0000","last_fullsized":"2024-09-17T04:49:42.055365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041767+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T04:49:42.041767+0000","last_peered":"2024-09-17T04:49:42.041767+0000","last_clean":"2024-09-17T04:49:42.041767+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T04:49:42.041767+0000","last_undegraded":"2024-09-17T04:49:42.041767+0000","last_fullsized":"2024-09-17T04:49:42.041767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667708+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T04:49:42.667708+0000","last_peered":"2024-09-17T04:49:42.667708+0000","last_clean":"2024-09-17T04:49:42.667708+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T04:49:42.667708+0000","last_undegraded":"2024-09-17T04:49:42.667708+0000","last_fullsized":"2024-09-17T04:49:42.667708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050522+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T04:49:42.050522+0000","last_peered":"2024-09-17T04:49:42.050522+0000","last_clean":"2024-09-17T04:49:42.050522+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T04:49:42.050522+0000","last_undegraded":"2024-09-17T04:49:42.050522+0000","last_fullsized":"2024-09-17T04:49:42.050522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049954+0000","last_change":"2024-09-17T04:49:41.057647+0000","last_active":"2024-09-17T04:49:42.049954+0000","last_peered":"2024-09-17T04:49:42.049954+0000","last_clean":"2024-09-17T04:49:42.049954+0000","last_became_active":"2024-09-17T04:49:41.056213+0000","last_became_peered":"2024-09-17T04:49:41.056213+0000","last_unstale":"2024-09-17T04:49:42.049954+0000","last_undegraded":"2024-09-17T04:49:42.049954+0000","last_fullsized":"2024-09-17T04:49:42.049954+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042498+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T04:49:42.042498+0000","last_peered":"2024-09-17T04:49:42.042498+0000","last_clean":"2024-09-17T04:49:42.042498+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T04:49:42.042498+0000","last_undegraded":"2024-09-17T04:49:42.042498+0000","last_fullsized":"2024-09-17T04:49:42.042498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053714+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T04:49:42.053714+0000","last_peered":"2024-09-17T04:49:42.053714+0000","last_clean":"2024-09-17T04:49:42.053714+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T04:49:42.053714+0000","last_undegraded":"2024-09-17T04:49:42.053714+0000","last_fullsized":"2024-09-17T04:49:42.053714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051282+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T04:49:42.051282+0000","last_peered":"2024-09-17T04:49:42.051282+0000","last_clean":"2024-09-17T04:49:42.051282+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T04:49:42.051282+0000","last_undegraded":"2024-09-17T04:49:42.051282+0000","last_fullsized":"2024-09-17T04:49:42.051282+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:27.230604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536526+0000","last_change":"2024-09-17T04:49:41.055263+0000","last_active":"2024-09-17T04:49:42.536526+0000","last_peered":"2024-09-17T04:49:42.536526+0000","last_clean":"2024-09-17T04:49:42.536526+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T04:49:42.536526+0000","last_undegraded":"2024-09-17T04:49:42.536526+0000","last_fullsized":"2024-09-17T04:49:42.536526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670438+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T04:49:42.670438+0000","last_peered":"2024-09-17T04:49:42.670438+0000","last_clean":"2024-09-17T04:49:42.670438+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T04:49:42.670438+0000","last_undegraded":"2024-09-17T04:49:42.670438+0000","last_fullsized":"2024-09-17T04:49:42.670438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673048+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T04:49:42.673048+0000","last_peered":"2024-09-17T04:49:42.673048+0000","last_clean":"2024-09-17T04:49:42.673048+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T04:49:42.673048+0000","last_undegraded":"2024-09-17T04:49:42.673048+0000","last_fullsized":"2024-09-17T04:49:42.673048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670447+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T04:49:42.670447+0000","last_peered":"2024-09-17T04:49:42.670447+0000","last_clean":"2024-09-17T04:49:42.670447+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T04:49:42.670447+0000","last_undegraded":"2024-09-17T04:49:42.670447+0000","last_fullsized":"2024-09-17T04:49:42.670447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:24.586834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051128+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.051128+0000","last_peered":"2024-09-17T04:49:42.051128+0000","last_clean":"2024-09-17T04:49:42.051128+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T04:49:42.051128+0000","last_undegraded":"2024-09-17T04:49:42.051128+0000","last_fullsized":"2024-09-17T04:49:42.051128+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536163+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T04:49:42.536163+0000","last_peered":"2024-09-17T04:49:42.536163+0000","last_clean":"2024-09-17T04:49:42.536163+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T04:49:42.536163+0000","last_undegraded":"2024-09-17T04:49:42.536163+0000","last_fullsized":"2024-09-17T04:49:42.536163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052121+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T04:49:42.052121+0000","last_peered":"2024-09-17T04:49:42.052121+0000","last_clean":"2024-09-17T04:49:42.052121+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T04:49:42.052121+0000","last_undegraded":"2024-09-17T04:49:42.052121+0000","last_fullsized":"2024-09-17T04:49:42.052121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668467+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T04:49:42.668467+0000","last_peered":"2024-09-17T04:49:42.668467+0000","last_clean":"2024-09-17T04:49:42.668467+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T04:49:42.668467+0000","last_undegraded":"2024-09-17T04:49:42.668467+0000","last_fullsized":"2024-09-17T04:49:42.668467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536096+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T04:49:42.536096+0000","last_peered":"2024-09-17T04:49:42.536096+0000","last_clean":"2024-09-17T04:49:42.536096+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T04:49:42.536096+0000","last_undegraded":"2024-09-17T04:49:42.536096+0000","last_fullsized":"2024-09-17T04:49:42.536096+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042322+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T04:49:42.042322+0000","last_peered":"2024-09-17T04:49:42.042322+0000","last_clean":"2024-09-17T04:49:42.042322+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T04:49:42.042322+0000","last_undegraded":"2024-09-17T04:49:42.042322+0000","last_fullsized":"2024-09-17T04:49:42.042322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671200+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T04:49:42.671200+0000","last_peered":"2024-09-17T04:49:42.671200+0000","last_clean":"2024-09-17T04:49:42.671200+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T04:49:42.671200+0000","last_undegraded":"2024-09-17T04:49:42.671200+0000","last_fullsized":"2024-09-17T04:49:42.671200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670309+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T04:49:42.670309+0000","last_peered":"2024-09-17T04:49:42.670309+0000","last_clean":"2024-09-17T04:49:42.670309+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T04:49:42.670309+0000","last_undegraded":"2024-09-17T04:49:42.670309+0000","last_fullsized":"2024-09-17T04:49:42.670309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668969+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T04:49:42.668969+0000","last_peered":"2024-09-17T04:49:42.668969+0000","last_clean":"2024-09-17T04:49:42.668969+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T04:49:42.668969+0000","last_undegraded":"2024-09-17T04:49:42.668969+0000","last_fullsized":"2024-09-17T04:49:42.668969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054935+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T04:49:42.054935+0000","last_peered":"2024-09-17T04:49:42.054935+0000","last_clean":"2024-09-17T04:49:42.054935+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T04:49:42.054935+0000","last_undegraded":"2024-09-17T04:49:42.054935+0000","last_fullsized":"2024-09-17T04:49:42.054935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669059+0000","last_change":"2024-09-17T04:49:41.073709+0000","last_active":"2024-09-17T04:49:42.669059+0000","last_peered":"2024-09-17T04:49:42.669059+0000","last_clean":"2024-09-17T04:49:42.669059+0000","last_became_active":"2024-09-17T04:49:41.073330+0000","last_became_peered":"2024-09-17T04:49:41.073330+0000","last_unstale":"2024-09-17T04:49:42.669059+0000","last_undegraded":"2024-09-17T04:49:42.669059+0000","last_fullsized":"2024-09-17T04:49:42.669059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054811+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T04:49:42.054811+0000","last_peered":"2024-09-17T04:49:42.054811+0000","last_clean":"2024-09-17T04:49:42.054811+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T04:49:42.054811+0000","last_undegraded":"2024-09-17T04:49:42.054811+0000","last_fullsized":"2024-09-17T04:49:42.054811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041256+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T04:49:42.041256+0000","last_peered":"2024-09-17T04:49:42.041256+0000","last_clean":"2024-09-17T04:49:42.041256+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T04:49:42.041256+0000","last_undegraded":"2024-09-17T04:49:42.041256+0000","last_fullsized":"2024-09-17T04:49:42.041256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536669+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T04:49:42.536669+0000","last_peered":"2024-09-17T04:49:42.536669+0000","last_clean":"2024-09-17T04:49:42.536669+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T04:49:42.536669+0000","last_undegraded":"2024-09-17T04:49:42.536669+0000","last_fullsized":"2024-09-17T04:49:42.536669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534968+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T04:49:42.534968+0000","last_peered":"2024-09-17T04:49:42.534968+0000","last_clean":"2024-09-17T04:49:42.534968+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T04:49:42.534968+0000","last_undegraded":"2024-09-17T04:49:42.534968+0000","last_fullsized":"2024-09-17T04:49:42.534968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672019+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T04:49:42.672019+0000","last_peered":"2024-09-17T04:49:42.672019+0000","last_clean":"2024-09-17T04:49:42.672019+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T04:49:42.672019+0000","last_undegraded":"2024-09-17T04:49:42.672019+0000","last_fullsized":"2024-09-17T04:49:42.672019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668565+0000","last_change":"2024-09-17T04:49:41.045502+0000","last_active":"2024-09-17T04:49:42.668565+0000","last_peered":"2024-09-17T04:49:42.668565+0000","last_clean":"2024-09-17T04:49:42.668565+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T04:49:42.668565+0000","last_undegraded":"2024-09-17T04:49:42.668565+0000","last_fullsized":"2024-09-17T04:49:42.668565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:19.746356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041666+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T04:49:42.041666+0000","last_peered":"2024-09-17T04:49:42.041666+0000","last_clean":"2024-09-17T04:49:42.041666+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T04:49:42.041666+0000","last_undegraded":"2024-09-17T04:49:42.041666+0000","last_fullsized":"2024-09-17T04:49:42.041666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042310+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T04:49:42.042310+0000","last_peered":"2024-09-17T04:49:42.042310+0000","last_clean":"2024-09-17T04:49:42.042310+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T04:49:42.042310+0000","last_undegraded":"2024-09-17T04:49:42.042310+0000","last_fullsized":"2024-09-17T04:49:42.042310+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668045+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T04:49:42.668045+0000","last_peered":"2024-09-17T04:49:42.668045+0000","last_clean":"2024-09-17T04:49:42.668045+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T04:49:42.668045+0000","last_undegraded":"2024-09-17T04:49:42.668045+0000","last_fullsized":"2024-09-17T04:49:42.668045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054699+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T04:49:42.054699+0000","last_peered":"2024-09-17T04:49:42.054699+0000","last_clean":"2024-09-17T04:49:42.054699+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T04:49:42.054699+0000","last_undegraded":"2024-09-17T04:49:42.054699+0000","last_fullsized":"2024-09-17T04:49:42.054699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673038+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T04:49:42.673038+0000","last_peered":"2024-09-17T04:49:42.673038+0000","last_clean":"2024-09-17T04:49:42.673038+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T04:49:42.673038+0000","last_undegraded":"2024-09-17T04:49:42.673038+0000","last_fullsized":"2024-09-17T04:49:42.673038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669104+0000","last_change":"2024-09-17T04:49:41.038265+0000","last_active":"2024-09-17T04:49:42.669104+0000","last_peered":"2024-09-17T04:49:42.669104+0000","last_clean":"2024-09-17T04:49:42.669104+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T04:49:42.669104+0000","last_undegraded":"2024-09-17T04:49:42.669104+0000","last_fullsized":"2024-09-17T04:49:42.669104+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:41.638403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053416+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T04:49:42.053416+0000","last_peered":"2024-09-17T04:49:42.053416+0000","last_clean":"2024-09-17T04:49:42.053416+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T04:49:42.053416+0000","last_undegraded":"2024-09-17T04:49:42.053416+0000","last_fullsized":"2024-09-17T04:49:42.053416+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535001+0000","last_change":"2024-09-17T04:49:41.053443+0000","last_active":"2024-09-17T04:49:42.535001+0000","last_peered":"2024-09-17T04:49:42.535001+0000","last_clean":"2024-09-17T04:49:42.535001+0000","last_became_active":"2024-09-17T04:49:41.053090+0000","last_became_peered":"2024-09-17T04:49:41.053090+0000","last_unstale":"2024-09-17T04:49:42.535001+0000","last_undegraded":"2024-09-17T04:49:42.535001+0000","last_fullsized":"2024-09-17T04:49:42.535001+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671599+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T04:49:42.671599+0000","last_peered":"2024-09-17T04:49:42.671599+0000","last_clean":"2024-09-17T04:49:42.671599+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T04:49:42.671599+0000","last_undegraded":"2024-09-17T04:49:42.671599+0000","last_fullsized":"2024-09-17T04:49:42.671599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669669+0000","last_change":"2024-09-17T04:49:41.038072+0000","last_active":"2024-09-17T04:49:42.669669+0000","last_peered":"2024-09-17T04:49:42.669669+0000","last_clean":"2024-09-17T04:49:42.669669+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T04:49:42.669669+0000","last_undegraded":"2024-09-17T04:49:42.669669+0000","last_fullsized":"2024-09-17T04:49:42.669669+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043492+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T04:49:42.043492+0000","last_peered":"2024-09-17T04:49:42.043492+0000","last_clean":"2024-09-17T04:49:42.043492+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T04:49:42.043492+0000","last_undegraded":"2024-09-17T04:49:42.043492+0000","last_fullsized":"2024-09-17T04:49:42.043492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534462+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T04:49:42.534462+0000","last_peered":"2024-09-17T04:49:42.534462+0000","last_clean":"2024-09-17T04:49:42.534462+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T04:49:42.534462+0000","last_undegraded":"2024-09-17T04:49:42.534462+0000","last_fullsized":"2024-09-17T04:49:42.534462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669602+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T04:49:42.669602+0000","last_peered":"2024-09-17T04:49:42.669602+0000","last_clean":"2024-09-17T04:49:42.669602+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T04:49:42.669602+0000","last_undegraded":"2024-09-17T04:49:42.669602+0000","last_fullsized":"2024-09-17T04:49:42.669602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050316+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T04:49:42.050316+0000","last_peered":"2024-09-17T04:49:42.050316+0000","last_clean":"2024-09-17T04:49:42.050316+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T04:49:42.050316+0000","last_undegraded":"2024-09-17T04:49:42.050316+0000","last_fullsized":"2024-09-17T04:49:42.050316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042123+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T04:49:42.042123+0000","last_peered":"2024-09-17T04:49:42.042123+0000","last_clean":"2024-09-17T04:49:42.042123+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T04:49:42.042123+0000","last_undegraded":"2024-09-17T04:49:42.042123+0000","last_fullsized":"2024-09-17T04:49:42.042123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052378+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T04:49:42.052378+0000","last_peered":"2024-09-17T04:49:42.052378+0000","last_clean":"2024-09-17T04:49:42.052378+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T04:49:42.052378+0000","last_undegraded":"2024-09-17T04:49:42.052378+0000","last_fullsized":"2024-09-17T04:49:42.052378+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041588+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T04:49:42.041588+0000","last_peered":"2024-09-17T04:49:42.041588+0000","last_clean":"2024-09-17T04:49:42.041588+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T04:49:42.041588+0000","last_undegraded":"2024-09-17T04:49:42.041588+0000","last_fullsized":"2024-09-17T04:49:42.041588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674897+0000","last_change":"2024-09-17T04:49:41.067207+0000","last_active":"2024-09-17T04:49:42.674897+0000","last_peered":"2024-09-17T04:49:42.674897+0000","last_clean":"2024-09-17T04:49:42.674897+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T04:49:42.674897+0000","last_undegraded":"2024-09-17T04:49:42.674897+0000","last_fullsized":"2024-09-17T04:49:42.674897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:08.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667484+0000","last_change":"2024-09-17T04:49:31.911036+0000","last_active":"2024-09-17T04:49:42.667484+0000","last_peered":"2024-09-17T04:49:42.667484+0000","last_clean":"2024-09-17T04:49:42.667484+0000","last_became_active":"2024-09-17T04:49:31.910285+0000","last_became_peered":"2024-09-17T04:49:31.910285+0000","last_unstale":"2024-09-17T04:49:42.667484+0000","last_undegraded":"2024-09-17T04:49:42.667484+0000","last_fullsized":"2024-09-17T04:49:42.667484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673474+0000","last_change":"2024-09-17T04:49:41.046170+0000","last_active":"2024-09-17T04:49:42.673474+0000","last_peered":"2024-09-17T04:49:42.673474+0000","last_clean":"2024-09-17T04:49:42.673474+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T04:49:42.673474+0000","last_undegraded":"2024-09-17T04:49:42.673474+0000","last_fullsized":"2024-09-17T04:49:42.673474+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:01.683826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051528+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T04:49:42.051528+0000","last_peered":"2024-09-17T04:49:42.051528+0000","last_clean":"2024-09-17T04:49:42.051528+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T04:49:42.051528+0000","last_undegraded":"2024-09-17T04:49:42.051528+0000","last_fullsized":"2024-09-17T04:49:42.051528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041843+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T04:49:42.041843+0000","last_peered":"2024-09-17T04:49:42.041843+0000","last_clean":"2024-09-17T04:49:42.041843+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T04:49:42.041843+0000","last_undegraded":"2024-09-17T04:49:42.041843+0000","last_fullsized":"2024-09-17T04:49:42.041843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:25.574682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053942+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T04:49:42.053942+0000","last_peered":"2024-09-17T04:49:42.053942+0000","last_clean":"2024-09-17T04:49:42.053942+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T04:49:42.053942+0000","last_undegraded":"2024-09-17T04:49:42.053942+0000","last_fullsized":"2024-09-17T04:49:42.053942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048414+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T04:49:42.048414+0000","last_peered":"2024-09-17T04:49:42.048414+0000","last_clean":"2024-09-17T04:49:42.048414+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T04:49:42.048414+0000","last_undegraded":"2024-09-17T04:49:42.048414+0000","last_fullsized":"2024-09-17T04:49:42.048414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671218+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T04:49:42.671218+0000","last_peered":"2024-09-17T04:49:42.671218+0000","last_clean":"2024-09-17T04:49:42.671218+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T04:49:42.671218+0000","last_undegraded":"2024-09-17T04:49:42.671218+0000","last_fullsized":"2024-09-17T04:49:42.671218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053498+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T04:49:42.053498+0000","last_peered":"2024-09-17T04:49:42.053498+0000","last_clean":"2024-09-17T04:49:42.053498+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T04:49:42.053498+0000","last_undegraded":"2024-09-17T04:49:42.053498+0000","last_fullsized":"2024-09-17T04:49:42.053498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670119+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T04:49:42.670119+0000","last_peered":"2024-09-17T04:49:42.670119+0000","last_clean":"2024-09-17T04:49:42.670119+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T04:49:42.670119+0000","last_undegraded":"2024-09-17T04:49:42.670119+0000","last_fullsized":"2024-09-17T04:49:42.670119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668585+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T04:49:42.668585+0000","last_peered":"2024-09-17T04:49:42.668585+0000","last_clean":"2024-09-17T04:49:42.668585+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T04:49:42.668585+0000","last_undegraded":"2024-09-17T04:49:42.668585+0000","last_fullsized":"2024-09-17T04:49:42.668585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670988+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T04:49:42.670988+0000","last_peered":"2024-09-17T04:49:42.670988+0000","last_clean":"2024-09-17T04:49:42.670988+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T04:49:42.670988+0000","last_undegraded":"2024-09-17T04:49:42.670988+0000","last_fullsized":"2024-09-17T04:49:42.670988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671416+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.671416+0000","last_peered":"2024-09-17T04:49:42.671416+0000","last_clean":"2024-09-17T04:49:42.671416+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T04:49:42.671416+0000","last_undegraded":"2024-09-17T04:49:42.671416+0000","last_fullsized":"2024-09-17T04:49:42.671416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675766+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T04:49:42.675766+0000","last_peered":"2024-09-17T04:49:42.675766+0000","last_clean":"2024-09-17T04:49:42.675766+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T04:49:42.675766+0000","last_undegraded":"2024-09-17T04:49:42.675766+0000","last_fullsized":"2024-09-17T04:49:42.675766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671330+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T04:49:42.671330+0000","last_peered":"2024-09-17T04:49:42.671330+0000","last_clean":"2024-09-17T04:49:42.671330+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T04:49:42.671330+0000","last_undegraded":"2024-09-17T04:49:42.671330+0000","last_fullsized":"2024-09-17T04:49:42.671330+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667310+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T04:49:42.667310+0000","last_peered":"2024-09-17T04:49:42.667310+0000","last_clean":"2024-09-17T04:49:42.667310+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T04:49:42.667310+0000","last_undegraded":"2024-09-17T04:49:42.667310+0000","last_fullsized":"2024-09-17T04:49:42.667310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537133+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T04:49:42.537133+0000","last_peered":"2024-09-17T04:49:42.537133+0000","last_clean":"2024-09-17T04:49:42.537133+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T04:49:42.537133+0000","last_undegraded":"2024-09-17T04:49:42.537133+0000","last_fullsized":"2024-09-17T04:49:42.537133+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669347+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T04:49:42.669347+0000","last_peered":"2024-09-17T04:49:42.669347+0000","last_clean":"2024-09-17T04:49:42.669347+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T04:49:42.669347+0000","last_undegraded":"2024-09-17T04:49:42.669347+0000","last_fullsized":"2024-09-17T04:49:42.669347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050809+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T04:49:42.050809+0000","last_peered":"2024-09-17T04:49:42.050809+0000","last_clean":"2024-09-17T04:49:42.050809+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T04:49:42.050809+0000","last_undegraded":"2024-09-17T04:49:42.050809+0000","last_fullsized":"2024-09-17T04:49:42.050809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535865+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T04:49:42.535865+0000","last_peered":"2024-09-17T04:49:42.535865+0000","last_clean":"2024-09-17T04:49:42.535865+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T04:49:42.535865+0000","last_undegraded":"2024-09-17T04:49:42.535865+0000","last_fullsized":"2024-09-17T04:49:42.535865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673274+0000","last_change":"2024-09-17T04:49:41.041201+0000","last_active":"2024-09-17T04:49:42.673274+0000","last_peered":"2024-09-17T04:49:42.673274+0000","last_clean":"2024-09-17T04:49:42.673274+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T04:49:42.673274+0000","last_undegraded":"2024-09-17T04:49:42.673274+0000","last_fullsized":"2024-09-17T04:49:42.673274+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:02.588786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670272+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T04:49:42.670272+0000","last_peered":"2024-09-17T04:49:42.670272+0000","last_clean":"2024-09-17T04:49:42.670272+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T04:49:42.670272+0000","last_undegraded":"2024-09-17T04:49:42.670272+0000","last_fullsized":"2024-09-17T04:49:42.670272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667767+0000","last_change":"2024-09-17T04:49:41.061092+0000","last_active":"2024-09-17T04:49:42.667767+0000","last_peered":"2024-09-17T04:49:42.667767+0000","last_clean":"2024-09-17T04:49:42.667767+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T04:49:42.667767+0000","last_undegraded":"2024-09-17T04:49:42.667767+0000","last_fullsized":"2024-09-17T04:49:42.667767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:52.714614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536820+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T04:49:42.536820+0000","last_peered":"2024-09-17T04:49:42.536820+0000","last_clean":"2024-09-17T04:49:42.536820+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T04:49:42.536820+0000","last_undegraded":"2024-09-17T04:49:42.536820+0000","last_fullsized":"2024-09-17T04:49:42.536820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669397+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T04:49:42.669397+0000","last_peered":"2024-09-17T04:49:42.669397+0000","last_clean":"2024-09-17T04:49:42.669397+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T04:49:42.669397+0000","last_undegraded":"2024-09-17T04:49:42.669397+0000","last_fullsized":"2024-09-17T04:49:42.669397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671170+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T04:49:42.671170+0000","last_peered":"2024-09-17T04:49:42.671170+0000","last_clean":"2024-09-17T04:49:42.671170+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T04:49:42.671170+0000","last_undegraded":"2024-09-17T04:49:42.671170+0000","last_fullsized":"2024-09-17T04:49:42.671170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536018+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T04:49:42.536018+0000","last_peered":"2024-09-17T04:49:42.536018+0000","last_clean":"2024-09-17T04:49:42.536018+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T04:49:42.536018+0000","last_undegraded":"2024-09-17T04:49:42.536018+0000","last_fullsized":"2024-09-17T04:49:42.536018+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671602+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T04:49:42.671602+0000","last_peered":"2024-09-17T04:49:42.671602+0000","last_clean":"2024-09-17T04:49:42.671602+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T04:49:42.671602+0000","last_undegraded":"2024-09-17T04:49:42.671602+0000","last_fullsized":"2024-09-17T04:49:42.671602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534962+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T04:49:42.534962+0000","last_peered":"2024-09-17T04:49:42.534962+0000","last_clean":"2024-09-17T04:49:42.534962+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T04:49:42.534962+0000","last_undegraded":"2024-09-17T04:49:42.534962+0000","last_fullsized":"2024-09-17T04:49:42.534962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671825+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T04:49:42.671825+0000","last_peered":"2024-09-17T04:49:42.671825+0000","last_clean":"2024-09-17T04:49:42.671825+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T04:49:42.671825+0000","last_undegraded":"2024-09-17T04:49:42.671825+0000","last_fullsized":"2024-09-17T04:49:42.671825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509450,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509450,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509450,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":307200,"data_stored":108537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509450,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509450,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":286720,"data_stored":106265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509451,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509451,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542155,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:49:56.271 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:49:56.271 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T04:49:56.271 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-17T04:49:56.271 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-17T04:49:56.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.452+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/3013982885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0a40053b0 msgr2=0x7fd0a400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.452+0000 7fd0c2c4c640 1 --2- 172.21.15.150:0/3013982885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0a40053b0 0x7fd0a400ba60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd0b40016c0 tx=0x7fd0b401c790 comp rx=0 tx=0).stop 2024-09-17T04:49:56.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.452+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/3013982885 shutdown_connections 2024-09-17T04:49:56.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.452+0000 7fd0c2c4c640 1 --2- 172.21.15.150:0/3013982885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0a40053b0 0x7fd0a400ba60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.452+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/3013982885 >> 172.21.15.150:0/3013982885 conn(0x7fd0bc06d450 msgr2=0x7fd0bc0a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:56.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.452+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/3013982885 shutdown_connections 2024-09-17T04:49:56.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.452+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/3013982885 wait complete. 2024-09-17T04:49:56.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0c2c4c640 1 Processor -- start 2024-09-17T04:49:56.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0c2c4c640 1 -- start start 2024-09-17T04:49:56.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0c2c4c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0a40053b0 0x7fd0bc0bd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:56.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0c2c4c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0bc151ac0 0x7fd0bc154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:56.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0c2c4c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd0bc151ed0 0x7fd0bc149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:56.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0c2c4c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd0bc070b20 con 0x7fd0bc151ac0 2024-09-17T04:49:56.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0c2c4c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd0bc0aa8e0 con 0x7fd0a40053b0 2024-09-17T04:49:56.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0c2c4c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd0bc070ca0 con 0x7fd0bc151ed0 2024-09-17T04:49:56.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0bbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0bc151ac0 0x7fd0bc154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:56.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0bbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0bc151ac0 0x7fd0bc154430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33494/0 (socket says 172.21.15.150:33494) 2024-09-17T04:49:56.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0bbfff640 1 -- 172.21.15.150:0/1334668667 learned_addr learned my addr 172.21.15.150:0/1334668667 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:56.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0bbfff640 1 -- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd0bc151ed0 msgr2=0x7fd0bc149090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0bbfff640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd0bc151ed0 0x7fd0bc149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0c09e8640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0a40053b0 0x7fd0bc0bd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:56.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0bbfff640 1 -- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0a40053b0 msgr2=0x7fd0bc0bd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0bbfff640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0a40053b0 0x7fd0bc0bd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0bbfff640 1 -- 172.21.15.150:0/1334668667 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0bc158910 con 0x7fd0bc151ac0 2024-09-17T04:49:56.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0bbfff640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0bc151ac0 0x7fd0bc154430 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0011750 tx=0x7fd0b0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:56.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0b97fa640 1 -- 172.21.15.150:0/1334668667 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b000fcf0 con 0x7fd0bc151ac0 2024-09-17T04:49:56.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0b97fa640 1 -- 172.21.15.150:0/1334668667 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0b0016070 con 0x7fd0bc151ac0 2024-09-17T04:49:56.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/1334668667 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0bc14c3c0 con 0x7fd0bc151ac0 2024-09-17T04:49:56.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0b97fa640 1 -- 172.21.15.150:0/1334668667 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b00126b0 con 0x7fd0bc151ac0 2024-09-17T04:49:56.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/1334668667 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0bc1f1610 con 0x7fd0bc151ac0 2024-09-17T04:49:56.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.456+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/1334668667 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0bc158b80 con 0x7fd0bc151ac0 2024-09-17T04:49:56.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.460+0000 7fd0b97fa640 1 -- 172.21.15.150:0/1334668667 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd0b0018030 con 0x7fd0bc151ac0 2024-09-17T04:49:56.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.460+0000 7fd0b97fa640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd0a8073230 0x7fd0a8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:56.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.460+0000 7fd0b97fa640 1 -- 172.21.15.150:0/1334668667 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fd0b0094b10 con 0x7fd0bc151ac0 2024-09-17T04:49:56.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.460+0000 7fd0c09e8640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd0a8073230 0x7fd0a8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:56.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.460+0000 7fd0c09e8640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd0a8073230 0x7fd0a8075740 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0c3d30 tx=0x7fd0ac001410 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:56.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.464+0000 7fd0b97fa640 1 -- 172.21.15.150:0/1334668667 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd0b0011e60 con 0x7fd0bc151ac0 2024-09-17T04:49:56.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.652+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/1334668667 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd0bc1f1400 con 0x7fd0bc151ac0 2024-09-17T04:49:56.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.652+0000 7fd0b97fa640 1 -- 172.21.15.150:0/1334668667 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd0b005f920 con 0x7fd0bc151ac0 2024-09-17T04:49:56.656 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:56.656 INFO:teuthology.orchestra.run.smithi150.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T04:49:56.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.656+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd0a8073230 msgr2=0x7fd0a8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.656+0000 7fd0c2c4c640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd0a8073230 0x7fd0a8075740 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0c3d30 tx=0x7fd0ac001410 comp rx=0 tx=0).stop 2024-09-17T04:49:56.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.656+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/1334668667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0bc151ac0 msgr2=0x7fd0bc154430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.656+0000 7fd0c2c4c640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0bc151ac0 0x7fd0bc154430 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0011750 tx=0x7fd0b0011c10 comp rx=0 tx=0).stop 2024-09-17T04:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.656+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/1334668667 shutdown_connections 2024-09-17T04:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.656+0000 7fd0c2c4c640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd0a8073230 0x7fd0a8075740 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.656+0000 7fd0c2c4c640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd0bc151ed0 0x7fd0bc149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.656+0000 7fd0c2c4c640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0bc151ac0 0x7fd0bc154430 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.656+0000 7fd0c2c4c640 1 --2- 172.21.15.150:0/1334668667 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0a40053b0 0x7fd0bc0bd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.656+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/1334668667 >> 172.21.15.150:0/1334668667 conn(0x7fd0bc06d450 msgr2=0x7fd0bc15f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.656+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/1334668667 shutdown_connections 2024-09-17T04:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.656+0000 7fd0c2c4c640 1 -- 172.21.15.150:0/1334668667 wait complete. 2024-09-17T04:49:56.677 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-17T04:49:56.677 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:49:56.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.824+0000 7f2d7ee82640 1 -- 172.21.15.150:0/848033644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d7806fff0 msgr2=0x7f2d78070400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.824+0000 7f2d7ee82640 1 --2- 172.21.15.150:0/848033644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d7806fff0 0x7f2d78070400 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2d780c4740 tx=0x7f2d7001cda0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.824+0000 7f2d7ee82640 1 -- 172.21.15.150:0/848033644 shutdown_connections 2024-09-17T04:49:56.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.824+0000 7f2d7ee82640 1 --2- 172.21.15.150:0/848033644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d7806fff0 0x7f2d78070400 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.824+0000 7f2d7ee82640 1 --2- 172.21.15.150:0/848033644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d780c41c0 0x7f2d7806fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.824+0000 7f2d7ee82640 1 -- 172.21.15.150:0/848033644 >> 172.21.15.150:0/848033644 conn(0x7f2d780c4a00 msgr2=0x7f2d780c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:56.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.824+0000 7f2d7ee82640 1 -- 172.21.15.150:0/848033644 shutdown_connections 2024-09-17T04:49:56.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.824+0000 7f2d7ee82640 1 -- 172.21.15.150:0/848033644 wait complete. 2024-09-17T04:49:56.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.824+0000 7f2d7ee82640 1 Processor -- start 2024-09-17T04:49:56.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.824+0000 7f2d7ee82640 1 -- start start 2024-09-17T04:49:56.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7ee82640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d7806fff0 0x7f2d7814bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:56.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7ee82640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d780c3db0 0x7f2d7814b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:56.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7ee82640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780c41c0 0x7f2d7814d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:56.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7ee82640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2d780adb30 con 0x7f2d780c41c0 2024-09-17T04:49:56.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7ee82640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2d780ad9b0 con 0x7f2d780c3db0 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7ee82640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2d780a75f0 con 0x7f2d7806fff0 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7d41f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780c41c0 0x7f2d7814d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7d41f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780c41c0 0x7f2d7814d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33504/0 (socket says 172.21.15.150:33504) 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7d41f640 1 -- 172.21.15.150:0/2324906199 learned_addr learned my addr 172.21.15.150:0/2324906199 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7cc1e640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d7806fff0 0x7f2d7814bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7d41f640 1 -- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d7806fff0 msgr2=0x7f2d7814bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d77fff640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d780c3db0 0x7f2d7814b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7d41f640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d7806fff0 0x7f2d7814bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7d41f640 1 -- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d780c3db0 msgr2=0x7f2d7814b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7d41f640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d780c3db0 0x7f2d7814b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7d41f640 1 -- 172.21.15.150:0/2324906199 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d780c5920 con 0x7f2d780c41c0 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7d41f640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780c41c0 0x7f2d7814d700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2d7001d640 tx=0x7f2d700a5b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d757fa640 1 -- 172.21.15.150:0/2324906199 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d700bb070 con 0x7f2d780c41c0 2024-09-17T04:49:56.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d757fa640 1 -- 172.21.15.150:0/2324906199 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d70004d40 con 0x7f2d780c41c0 2024-09-17T04:49:56.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7ee82640 1 -- 172.21.15.150:0/2324906199 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d78146780 con 0x7f2d780c41c0 2024-09-17T04:49:56.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d757fa640 1 -- 172.21.15.150:0/2324906199 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d700b2040 con 0x7f2d780c41c0 2024-09-17T04:49:56.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7ee82640 1 -- 172.21.15.150:0/2324906199 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d781464a0 con 0x7f2d780c41c0 2024-09-17T04:49:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d7ee82640 1 -- 172.21.15.150:0/2324906199 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d78070400 con 0x7f2d780c41c0 2024-09-17T04:49:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d757fa640 1 -- 172.21.15.150:0/2324906199 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2d700be070 con 0x7f2d780c41c0 2024-09-17T04:49:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.828+0000 7f2d757fa640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d64073230 0x7f2d64075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.832+0000 7f2d757fa640 1 -- 172.21.15.150:0/2324906199 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f2d7012e430 con 0x7f2d780c41c0 2024-09-17T04:49:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.832+0000 7f2d7cc1e640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d64073230 0x7f2d64075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.832+0000 7f2d7cc1e640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d64073230 0x7f2d64075740 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2d780c2500 tx=0x7f2d68008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:56.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:56.836+0000 7f2d757fa640 1 -- 172.21.15.150:0/2324906199 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2d70016990 con 0x7f2d780c41c0 2024-09-17T04:49:57.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.024+0000 7f2d7ee82640 1 -- 172.21.15.150:0/2324906199 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d780ad550 con 0x7f2d780c41c0 2024-09-17T04:49:57.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.024+0000 7f2d757fa640 1 -- 172.21.15.150:0/2324906199 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f2d700f91b0 con 0x7f2d780c41c0 2024-09-17T04:49:57.027 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:57.027 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:49:57.027 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:49:57.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.028+0000 7f2d7ee82640 1 -- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d64073230 msgr2=0x7f2d64075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.028+0000 7f2d7ee82640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d64073230 0x7f2d64075740 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2d780c2500 tx=0x7f2d68008040 comp rx=0 tx=0).stop 2024-09-17T04:49:57.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.028+0000 7f2d7ee82640 1 -- 172.21.15.150:0/2324906199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780c41c0 msgr2=0x7f2d7814d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.028+0000 7f2d7ee82640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780c41c0 0x7f2d7814d700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2d7001d640 tx=0x7f2d700a5b00 comp rx=0 tx=0).stop 2024-09-17T04:49:57.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.028+0000 7f2d7ee82640 1 -- 172.21.15.150:0/2324906199 shutdown_connections 2024-09-17T04:49:57.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.028+0000 7f2d7ee82640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d64073230 0x7f2d64075740 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.028+0000 7f2d7ee82640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d780c3db0 0x7f2d7814b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.028+0000 7f2d7ee82640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780c41c0 0x7f2d7814d700 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.028+0000 7f2d7ee82640 1 --2- 172.21.15.150:0/2324906199 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d7806fff0 0x7f2d7814bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.028+0000 7f2d7ee82640 1 -- 172.21.15.150:0/2324906199 >> 172.21.15.150:0/2324906199 conn(0x7f2d780c4a00 msgr2=0x7f2d780c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:57.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.028+0000 7f2d7ee82640 1 -- 172.21.15.150:0/2324906199 shutdown_connections 2024-09-17T04:49:57.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.028+0000 7f2d7ee82640 1 -- 172.21.15.150:0/2324906199 wait complete. 2024-09-17T04:49:57.049 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:49:57.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.184+0000 7fafc01ca640 1 -- 172.21.15.150:0/2010500613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafb80a98a0 msgr2=0x7fafb80a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.184+0000 7fafc01ca640 1 --2- 172.21.15.150:0/2010500613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafb80a98a0 0x7fafb80a9cb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fafb8072ba0 tx=0x7fafb001cc60 comp rx=0 tx=0).stop 2024-09-17T04:49:57.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.184+0000 7fafc01ca640 1 -- 172.21.15.150:0/2010500613 shutdown_connections 2024-09-17T04:49:57.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.184+0000 7fafc01ca640 1 --2- 172.21.15.150:0/2010500613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafb80a98a0 0x7fafb80a9cb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.184+0000 7fafc01ca640 1 --2- 172.21.15.150:0/2010500613 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fafb80699b0 0x7fafb8151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.184+0000 7fafc01ca640 1 --2- 172.21.15.150:0/2010500613 >> v2:172.21.15.176:3300/0 conn(0x7fafb806e590 0x7fafb80719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.184+0000 7fafc01ca640 1 -- 172.21.15.150:0/2010500613 >> 172.21.15.150:0/2010500613 conn(0x7fafb8070b50 msgr2=0x7fafb8070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:57.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.184+0000 7fafc01ca640 1 -- 172.21.15.150:0/2010500613 shutdown_connections 2024-09-17T04:49:57.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.184+0000 7fafc01ca640 1 -- 172.21.15.150:0/2010500613 wait complete. 2024-09-17T04:49:57.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafc01ca640 1 Processor -- start 2024-09-17T04:49:57.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafc01ca640 1 -- start start 2024-09-17T04:49:57.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafc01ca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fafb80699b0 0x7fafb8146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:57.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafc01ca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafb806e590 0x7fafb81439c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:57.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafc01ca640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fafb80a98a0 0x7fafb81568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:57.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafc01ca640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fafb8158120 con 0x7fafb806e590 2024-09-17T04:49:57.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafc01ca640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fafb8157fa0 con 0x7fafb80a98a0 2024-09-17T04:49:57.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafc01ca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fafb81582a0 con 0x7fafb80699b0 2024-09-17T04:49:57.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbd765640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafb806e590 0x7fafb81439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:57.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbd765640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafb806e590 0x7fafb81439c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33528/0 (socket says 172.21.15.150:33528) 2024-09-17T04:49:57.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbd765640 1 -- 172.21.15.150:0/3595128351 learned_addr learned my addr 172.21.15.150:0/3595128351 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:57.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbdf66640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fafb80699b0 0x7fafb8146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:57.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbe767640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fafb80a98a0 0x7fafb81568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:57.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbd765640 1 -- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fafb80699b0 msgr2=0x7fafb8146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbd765640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fafb80699b0 0x7fafb8146c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbd765640 1 -- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fafb80a98a0 msgr2=0x7fafb81568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbd765640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fafb80a98a0 0x7fafb81568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbd765640 1 -- 172.21.15.150:0/3595128351 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb8151250 con 0x7fafb806e590 2024-09-17T04:49:57.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbdf66640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fafb80699b0 0x7fafb8146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:57.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbe767640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fafb80a98a0 0x7fafb81568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:57.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafbd765640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafb806e590 0x7fafb81439c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fafa800d690 tx=0x7fafa800db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:57.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafae7fc640 1 -- 172.21.15.150:0/3595128351 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8017070 con 0x7fafb806e590 2024-09-17T04:49:57.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafae7fc640 1 -- 172.21.15.150:0/3595128351 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafa8005310 con 0x7fafb806e590 2024-09-17T04:49:57.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafae7fc640 1 -- 172.21.15.150:0/3595128351 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8009820 con 0x7fafb806e590 2024-09-17T04:49:57.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafc01ca640 1 -- 172.21.15.150:0/3595128351 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb815e230 con 0x7fafb806e590 2024-09-17T04:49:57.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafc01ca640 1 -- 172.21.15.150:0/3595128351 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb8150d30 con 0x7fafb806e590 2024-09-17T04:49:57.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.188+0000 7fafc01ca640 1 -- 172.21.15.150:0/3595128351 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafb80719e0 con 0x7fafb806e590 2024-09-17T04:49:57.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.192+0000 7fafae7fc640 1 -- 172.21.15.150:0/3595128351 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fafa80099c0 con 0x7fafb806e590 2024-09-17T04:49:57.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.192+0000 7fafae7fc640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faf9c073230 0x7faf9c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:57.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.192+0000 7fafae7fc640 1 -- 172.21.15.150:0/3595128351 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fafa80948e0 con 0x7fafb806e590 2024-09-17T04:49:57.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.192+0000 7fafbdf66640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faf9c073230 0x7faf9c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:57.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.192+0000 7fafbdf66640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faf9c073230 0x7faf9c075740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fafb4005fd0 tx=0x7fafb4005dc0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:57.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.196+0000 7fafae7fc640 1 -- 172.21.15.150:0/3595128351 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fafa805f7d0 con 0x7fafb806e590 2024-09-17T04:49:57.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.384+0000 7fafc01ca640 1 -- 172.21.15.150:0/3595128351 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafb81ee080 con 0x7fafb806e590 2024-09-17T04:49:57.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafae7fc640 1 -- 172.21.15.150:0/3595128351 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fafa80660d0 con 0x7fafb806e590 2024-09-17T04:49:57.389 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:57.389 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:49:57.389 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:49:57.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafc01ca640 1 -- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faf9c073230 msgr2=0x7faf9c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafc01ca640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faf9c073230 0x7faf9c075740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fafb4005fd0 tx=0x7fafb4005dc0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafc01ca640 1 -- 172.21.15.150:0/3595128351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafb806e590 msgr2=0x7fafb81439c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafc01ca640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafb806e590 0x7fafb81439c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fafa800d690 tx=0x7fafa800db50 comp rx=0 tx=0).stop 2024-09-17T04:49:57.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafc01ca640 1 -- 172.21.15.150:0/3595128351 shutdown_connections 2024-09-17T04:49:57.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafc01ca640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faf9c073230 0x7faf9c075740 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafc01ca640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fafb80a98a0 0x7fafb81568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafc01ca640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafb806e590 0x7fafb81439c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafc01ca640 1 --2- 172.21.15.150:0/3595128351 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fafb80699b0 0x7fafb8146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafc01ca640 1 -- 172.21.15.150:0/3595128351 >> 172.21.15.150:0/3595128351 conn(0x7fafb8070b50 msgr2=0x7fafb8070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:57.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafc01ca640 1 -- 172.21.15.150:0/3595128351 shutdown_connections 2024-09-17T04:49:57.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.388+0000 7fafc01ca640 1 -- 172.21.15.150:0/3595128351 wait complete. 2024-09-17T04:49:57.410 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:57.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.560+0000 7f312e59e640 1 -- 172.21.15.150:0/2086859328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31280717a0 msgr2=0x7f3128074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.560+0000 7f312e59e640 1 --2- 172.21.15.150:0/2086859328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31280717a0 0x7f3128074450 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3120009bc0 tx=0x7f312001cb10 comp rx=0 tx=0).stop 2024-09-17T04:49:57.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.560+0000 7f312e59e640 1 -- 172.21.15.150:0/2086859328 shutdown_connections 2024-09-17T04:49:57.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.560+0000 7f312e59e640 1 --2- 172.21.15.150:0/2086859328 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3128070e50 0x7f3128071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.560+0000 7f312e59e640 1 --2- 172.21.15.150:0/2086859328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31280717a0 0x7f3128074450 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.560+0000 7f312e59e640 1 -- 172.21.15.150:0/2086859328 >> 172.21.15.150:0/2086859328 conn(0x7f31280c4a00 msgr2=0x7f31280c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:57.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.560+0000 7f312e59e640 1 -- 172.21.15.150:0/2086859328 shutdown_connections 2024-09-17T04:49:57.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.560+0000 7f312e59e640 1 -- 172.21.15.150:0/2086859328 wait complete. 2024-09-17T04:49:57.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.560+0000 7f312e59e640 1 Processor -- start 2024-09-17T04:49:57.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.560+0000 7f312e59e640 1 -- start start 2024-09-17T04:49:57.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f312e59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3128070e50 0x7f31281f0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:57.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f312e59e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31280717a0 0x7f3128147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:57.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f312e59e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3128073440 0x7f3128156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f312e59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f31280ad190 con 0x7f3128070e50 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f312e59e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f31280ad010 con 0x7f3128073440 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f312e59e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f31280ad310 con 0x7f31280717a0 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f3127fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3128070e50 0x7f31281f0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f3127fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3128070e50 0x7f31281f0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33538/0 (socket says 172.21.15.150:33538) 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f3127fff640 1 -- 172.21.15.150:0/2953903689 learned_addr learned my addr 172.21.15.150:0/2953903689 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f311f7fe640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31280717a0 0x7f3128147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f312cb3b640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3128073440 0x7f3128156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f3127fff640 1 -- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31280717a0 msgr2=0x7f3128147a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f3127fff640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31280717a0 0x7f3128147a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f3127fff640 1 -- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3128073440 msgr2=0x7f3128156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f3127fff640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3128073440 0x7f3128156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f3127fff640 1 -- 172.21.15.150:0/2953903689 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312814efc0 con 0x7f3128070e50 2024-09-17T04:49:57.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f3127fff640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3128070e50 0x7f31281f0800 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f311800c400 tx=0x7f311800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:57.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f31257fa640 1 -- 172.21.15.150:0/2953903689 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3118011790 con 0x7f3128070e50 2024-09-17T04:49:57.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f31257fa640 1 -- 172.21.15.150:0/2953903689 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3118017070 con 0x7f3128070e50 2024-09-17T04:49:57.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f31257fa640 1 -- 172.21.15.150:0/2953903689 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3118011cc0 con 0x7f3128070e50 2024-09-17T04:49:57.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f312e59e640 1 -- 172.21.15.150:0/2953903689 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31281edeb0 con 0x7f3128070e50 2024-09-17T04:49:57.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f312e59e640 1 -- 172.21.15.150:0/2953903689 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31281ee190 con 0x7f3128070e50 2024-09-17T04:49:57.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f312e59e640 1 -- 172.21.15.150:0/2953903689 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31280ad800 con 0x7f3128070e50 2024-09-17T04:49:57.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.564+0000 7f31257fa640 1 -- 172.21.15.150:0/2953903689 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f311801c460 con 0x7f3128070e50 2024-09-17T04:49:57.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.568+0000 7f31257fa640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3108073200 0x7f3108075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:57.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.568+0000 7f31257fa640 1 -- 172.21.15.150:0/2953903689 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f3118094290 con 0x7f3128070e50 2024-09-17T04:49:57.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.568+0000 7f311f7fe640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3108073200 0x7f3108075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:57.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.568+0000 7f311f7fe640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3108073200 0x7f3108075710 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f31281564c0 tx=0x7f311000b390 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:57.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.572+0000 7f31257fa640 1 -- 172.21.15.150:0/2953903689 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f311805ef80 con 0x7f3128070e50 2024-09-17T04:49:57.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.736+0000 7f312e59e640 1 -- 172.21.15.150:0/2953903689 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31281577a0 con 0x7f3128070e50 2024-09-17T04:49:57.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.740+0000 7f31257fa640 1 -- 172.21.15.150:0/2953903689 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20431 (secure 0 0 0) 0x7f311805e880 con 0x7f3128070e50 2024-09-17T04:49:57.743 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:57.743 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:42.029799+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:57.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.744+0000 7f312e59e640 1 -- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3108073200 msgr2=0x7f3108075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.744+0000 7f312e59e640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3108073200 0x7f3108075710 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f31281564c0 tx=0x7f311000b390 comp rx=0 tx=0).stop 2024-09-17T04:49:57.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.744+0000 7f312e59e640 1 -- 172.21.15.150:0/2953903689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3128070e50 msgr2=0x7f31281f0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.744+0000 7f312e59e640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3128070e50 0x7f31281f0800 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f311800c400 tx=0x7f311800c8c0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.744+0000 7f312e59e640 1 -- 172.21.15.150:0/2953903689 shutdown_connections 2024-09-17T04:49:57.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.744+0000 7f312e59e640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3108073200 0x7f3108075710 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.744+0000 7f312e59e640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3128073440 0x7f3128156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.744+0000 7f312e59e640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31280717a0 0x7f3128147a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.744+0000 7f312e59e640 1 --2- 172.21.15.150:0/2953903689 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3128070e50 0x7f31281f0800 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.744+0000 7f312e59e640 1 -- 172.21.15.150:0/2953903689 >> 172.21.15.150:0/2953903689 conn(0x7f31280c4a00 msgr2=0x7f31280c35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:57.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.744+0000 7f312e59e640 1 -- 172.21.15.150:0/2953903689 shutdown_connections 2024-09-17T04:49:57.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.744+0000 7f312e59e640 1 -- 172.21.15.150:0/2953903689 wait complete. 2024-09-17T04:49:57.764 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:49:57.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.912+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2492171805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0a40c3d80 msgr2=0x7fc0a4158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.912+0000 7fc0a8cca640 1 --2- 172.21.15.150:0/2492171805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0a40c3d80 0x7fc0a4158340 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc09c009c80 tx=0x7fc09c01cd40 comp rx=0 tx=0).stop 2024-09-17T04:49:57.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.912+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2492171805 shutdown_connections 2024-09-17T04:49:57.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.912+0000 7fc0a8cca640 1 --2- 172.21.15.150:0/2492171805 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0a40c3d80 0x7fc0a4158340 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.912+0000 7fc0a8cca640 1 --2- 172.21.15.150:0/2492171805 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0a4151d00 0x7fc0a40c3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.912+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2492171805 >> 172.21.15.150:0/2492171805 conn(0x7fc0a40c4a00 msgr2=0x7fc0a40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:57.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.912+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2492171805 shutdown_connections 2024-09-17T04:49:57.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.912+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2492171805 wait complete. 2024-09-17T04:49:57.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.912+0000 7fc0a8cca640 1 Processor -- start 2024-09-17T04:49:57.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.912+0000 7fc0a8cca640 1 -- start start 2024-09-17T04:49:57.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a8cca640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0a40c3970 0x7fc0a4152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:57.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a8cca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0a40c3d80 0x7fc0a414fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:57.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a8cca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0a4151d00 0x7fc0a41f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:57.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a8cca640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a40c44b0 con 0x7fc0a40c3d80 2024-09-17T04:49:57.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a8cca640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a40ade60 con 0x7fc0a40c3970 2024-09-17T04:49:57.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a8cca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc0a40c4630 con 0x7fc0a4151d00 2024-09-17T04:49:57.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a1d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0a40c3d80 0x7fc0a414fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a1d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0a40c3d80 0x7fc0a414fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33548/0 (socket says 172.21.15.150:33548) 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a1d9b640 1 -- 172.21.15.150:0/2991799625 learned_addr learned my addr 172.21.15.150:0/2991799625 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a2d9d640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0a4151d00 0x7fc0a41f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a259c640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0a40c3970 0x7fc0a4152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a1d9b640 1 -- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0a4151d00 msgr2=0x7fc0a41f0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a1d9b640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0a4151d00 0x7fc0a41f0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a1d9b640 1 -- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0a40c3970 msgr2=0x7fc0a4152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a1d9b640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0a40c3970 0x7fc0a4152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a1d9b640 1 -- 172.21.15.150:0/2991799625 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0a4156490 con 0x7fc0a40c3d80 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a259c640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0a40c3970 0x7fc0a4152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a1d9b640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0a40c3d80 0x7fc0a414fd80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc09800c400 tx=0x7fc09800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc092ffd640 1 -- 172.21.15.150:0/2991799625 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc098017070 con 0x7fc0a40c3d80 2024-09-17T04:49:57.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc092ffd640 1 -- 172.21.15.150:0/2991799625 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc098003600 con 0x7fc0a40c3d80 2024-09-17T04:49:57.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc092ffd640 1 -- 172.21.15.150:0/2991799625 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc098003a50 con 0x7fc0a40c3d80 2024-09-17T04:49:57.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2991799625 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0a4149390 con 0x7fc0a40c3d80 2024-09-17T04:49:57.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2991799625 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0a40a8f90 con 0x7fc0a40c3d80 2024-09-17T04:49:57.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2991799625 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0a414ba90 con 0x7fc0a40c3d80 2024-09-17T04:49:57.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc092ffd640 1 -- 172.21.15.150:0/2991799625 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc09801a070 con 0x7fc0a40c3d80 2024-09-17T04:49:57.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc092ffd640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc084073200 0x7fc084075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:57.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc092ffd640 1 -- 172.21.15.150:0/2991799625 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fc098093720 con 0x7fc0a40c3d80 2024-09-17T04:49:57.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.916+0000 7fc0a259c640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc084073200 0x7fc084075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:57.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.920+0000 7fc0a259c640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc084073200 0x7fc084075710 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc094005fd0 tx=0x7fc094005dc0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:57.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:57.920+0000 7fc092ffd640 1 -- 172.21.15.150:0/2991799625 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc09805e530 con 0x7fc0a40c3d80 2024-09-17T04:49:57.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:49:57.969 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:49:58.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.112+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2991799625 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0a40c3430 con 0x7fc0a40c3d80 2024-09-17T04:49:58.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.112+0000 7fc092ffd640 1 -- 172.21.15.150:0/2991799625 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fc09805de30 con 0x7fc0a40c3d80 2024-09-17T04:49:58.116 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:58.117 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:49:58.117 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:49:58.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.116+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc084073200 msgr2=0x7fc084075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.116+0000 7fc0a8cca640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc084073200 0x7fc084075710 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc094005fd0 tx=0x7fc094005dc0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.116+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2991799625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0a40c3d80 msgr2=0x7fc0a414fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.116+0000 7fc0a8cca640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0a40c3d80 0x7fc0a414fd80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc09800c400 tx=0x7fc09800c8c0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.116+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2991799625 shutdown_connections 2024-09-17T04:49:58.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.116+0000 7fc0a8cca640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc084073200 0x7fc084075710 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.116+0000 7fc0a8cca640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0a4151d00 0x7fc0a41f0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.116+0000 7fc0a8cca640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0a40c3d80 0x7fc0a414fd80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.116+0000 7fc0a8cca640 1 --2- 172.21.15.150:0/2991799625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0a40c3970 0x7fc0a4152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.116+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2991799625 >> 172.21.15.150:0/2991799625 conn(0x7fc0a40c4a00 msgr2=0x7fc0a40c2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:58.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.116+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2991799625 shutdown_connections 2024-09-17T04:49:58.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.116+0000 7fc0a8cca640 1 -- 172.21.15.150:0/2991799625 wait complete. 2024-09-17T04:49:58.540 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:49:31.253633+0000', 'modified': '2024-09-17T04:49:35.350600+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4404}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4404': {'gid': 4404, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6835/2613040311', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6833', 'nonce': 2613040311}, {'type': 'v1', 'addr': '172.21.15.150:6835', 'nonce': 2613040311}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'a', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4404, 'qdb_cluster': [4404]} 2024-09-17T04:49:58.541 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:49:58.541 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T04:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 -- 172.21.15.150:0/3448453797 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6400c41c0 msgr2=0x7fe64006fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 --2- 172.21.15.150:0/3448453797 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6400c41c0 0x7fe64006fab0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe630009840 tx=0x7fe63001c820 comp rx=0 tx=0).stop 2024-09-17T04:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 -- 172.21.15.150:0/3448453797 shutdown_connections 2024-09-17T04:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 --2- 172.21.15.150:0/3448453797 >> v2:172.21.15.176:3300/0 conn(0x7fe64006fff0 0x7fe640070400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 --2- 172.21.15.150:0/3448453797 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6400c41c0 0x7fe64006fab0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 -- 172.21.15.150:0/3448453797 >> 172.21.15.150:0/3448453797 conn(0x7fe6400c4a00 msgr2=0x7fe6400c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 -- 172.21.15.150:0/3448453797 shutdown_connections 2024-09-17T04:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 -- 172.21.15.150:0/3448453797 wait complete. 2024-09-17T04:49:58.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 Processor -- start 2024-09-17T04:49:58.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 -- start start 2024-09-17T04:49:58.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe64006fff0 0x7fe6401483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6400c3db0 0x7fe640144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6400c41c0 0x7fe640156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe6400adb30 con 0x7fe6400c3db0 2024-09-17T04:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.708+0000 7fe647450640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe6400ad9b0 con 0x7fe64006fff0 2024-09-17T04:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe647450640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe6400a75f0 con 0x7fe6400c41c0 2024-09-17T04:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe6449eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6400c3db0 0x7fe640144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe6449eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6400c3db0 0x7fe640144d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33562/0 (socket says 172.21.15.150:33562) 2024-09-17T04:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe6449eb640 1 -- 172.21.15.150:0/3867226107 learned_addr learned my addr 172.21.15.150:0/3867226107 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe6459ed640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6400c41c0 0x7fe640156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe6451ec640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe64006fff0 0x7fe6401483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe6449eb640 1 -- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6400c41c0 msgr2=0x7fe640156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe6449eb640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6400c41c0 0x7fe640156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe6449eb640 1 -- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe64006fff0 msgr2=0x7fe6401483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe6449eb640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe64006fff0 0x7fe6401483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe6449eb640 1 -- 172.21.15.150:0/3867226107 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6401f1460 con 0x7fe6400c3db0 2024-09-17T04:49:58.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe6449eb640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6400c3db0 0x7fe640144d10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe630009790 tx=0x7fe6300a5670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:58.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe635ffb640 1 -- 172.21.15.150:0/3867226107 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6300bc070 con 0x7fe6400c3db0 2024-09-17T04:49:58.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe635ffb640 1 -- 172.21.15.150:0/3867226107 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe630004470 con 0x7fe6400c3db0 2024-09-17T04:49:58.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe647450640 1 -- 172.21.15.150:0/3867226107 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe640146410 con 0x7fe6400c3db0 2024-09-17T04:49:58.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe635ffb640 1 -- 172.21.15.150:0/3867226107 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6300b2040 con 0x7fe6400c3db0 2024-09-17T04:49:58.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe647450640 1 -- 172.21.15.150:0/3867226107 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe64014bbe0 con 0x7fe6400c3db0 2024-09-17T04:49:58.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe635ffb640 1 -- 172.21.15.150:0/3867226107 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe630003860 con 0x7fe6400c3db0 2024-09-17T04:49:58.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe635ffb640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe624073230 0x7fe624075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe635ffb640 1 -- 172.21.15.150:0/3867226107 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fe63012e0a0 con 0x7fe6400c3db0 2024-09-17T04:49:58.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe6451ec640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe624073230 0x7fe624075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe64e71b000 1 Processor -- start 2024-09-17T04:49:58.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.712+0000 7fe64e71b000 1 -- start start 2024-09-17T04:49:58.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6451ec640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe624073230 0x7fe624075740 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe6400c2500 tx=0x7fe63c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:58.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5b8f460 0x55cfd60c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5f7f950 0x55cfd6084690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd6084bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55cfd5fa5410 con 0x55cfd5f68ca0 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55cfd5ee2b40 con 0x55cfd5f7f950 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55cfd5ed1f70 con 0x55cfd5b8f460 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6459ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd6084bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6459ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd6084bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33570/0 (socket says 172.21.15.150:33570) 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6459ed640 1 -- 172.21.15.150:0/833650721 learned_addr learned my addr 172.21.15.150:0/833650721 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6451ec640 1 --2- 172.21.15.150:0/833650721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5b8f460 0x55cfd60c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6449eb640 1 --2- 172.21.15.150:0/833650721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5f7f950 0x55cfd6084690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6459ed640 1 -- 172.21.15.150:0/833650721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5b8f460 msgr2=0x55cfd60c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6459ed640 1 --2- 172.21.15.150:0/833650721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5b8f460 0x55cfd60c1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6459ed640 1 -- 172.21.15.150:0/833650721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5f7f950 msgr2=0x55cfd6084690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6459ed640 1 --2- 172.21.15.150:0/833650721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5f7f950 0x55cfd6084690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6459ed640 1 -- 172.21.15.150:0/833650721 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfd5e995b0 con 0x55cfd5f68ca0 2024-09-17T04:49:58.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6449eb640 1 --2- 172.21.15.150:0/833650721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5f7f950 0x55cfd6084690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:58.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe6459ed640 1 --2- 172.21.15.150:0/833650721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd6084bd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe638009f00 tx=0x7fe6380107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409eb8bde66ba828 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:58.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe62affd640 1 -- 172.21.15.150:0/833650721 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63801c680 con 0x55cfd5f68ca0 2024-09-17T04:49:58.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe62affd640 1 -- 172.21.15.150:0/833650721 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe63801c910 con 0x55cfd5f68ca0 2024-09-17T04:49:58.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe62affd640 1 -- 172.21.15.150:0/833650721 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63801cd40 con 0x55cfd5f68ca0 2024-09-17T04:49:58.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 -- 172.21.15.150:0/833650721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 msgr2=0x55cfd6084bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 --2- 172.21.15.150:0/833650721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd6084bd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe638009f00 tx=0x7fe6380107a0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 -- 172.21.15.150:0/833650721 shutdown_connections 2024-09-17T04:49:58.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 --2- 172.21.15.150:0/833650721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5f7f950 0x55cfd6084690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 --2- 172.21.15.150:0/833650721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd6084bd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 --2- 172.21.15.150:0/833650721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5b8f460 0x55cfd60c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 -- 172.21.15.150:0/833650721 >> 172.21.15.150:0/833650721 conn(0x55cfd6055d90 msgr2=0x55cfd60c49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:58.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 -- 172.21.15.150:0/833650721 shutdown_connections 2024-09-17T04:49:58.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 -- 172.21.15.150:0/833650721 wait complete. 2024-09-17T04:49:58.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 -- start start 2024-09-17T04:49:58.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.716+0000 7fe64e71b000 1 Processor -- start 2024-09-17T04:49:58.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5b8f460 0x55cfd60c19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd6088e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f7f950 0x55cfd60893a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55cfd5e6c5f0 con 0x55cfd5f68ca0 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55cfd5ed5d80 con 0x55cfd5b8f460 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55cfd5e78520 con 0x55cfd5f7f950 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 10 client.0 _ll_register_callbacks cb 0x55cfd5e7c830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6449eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd6088e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6449eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd6088e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33586/0 (socket says 172.21.15.150:33586) 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6449eb640 1 -- 172.21.15.150:0/3948962272 learned_addr learned my addr 172.21.15.150:0/3948962272 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6459ed640 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f7f950 0x55cfd60893a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6451ec640 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5b8f460 0x55cfd60c19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6449eb640 1 -- 172.21.15.150:0/3948962272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f7f950 msgr2=0x55cfd60893a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6449eb640 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f7f950 0x55cfd60893a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6449eb640 1 -- 172.21.15.150:0/3948962272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5b8f460 msgr2=0x55cfd60c19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6449eb640 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5b8f460 0x55cfd60c19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6449eb640 1 -- 172.21.15.150:0/3948962272 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfd5e70190 con 0x55cfd5f68ca0 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6451ec640 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5b8f460 0x55cfd60c19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6449eb640 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd6088e60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe6300b0040 tx=0x7fe6300f85c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbba1ba67ff4ff61 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe61ffff640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe61ffff640 1 -- 172.21.15.150:0/3948962272 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6300f8b90 con 0x55cfd5f68ca0 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe61ffff640 1 -- 172.21.15.150:0/3948962272 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fe630101c30 con 0x55cfd5f68ca0 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 -- 172.21.15.150:0/3948962272 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cfd5e6a110 con 0x55cfd5f68ca0 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe61ffff640 1 -- 172.21.15.150:0/3948962272 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fe6300f8e30 con 0x55cfd5f68ca0 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 10 client.4842 fetch_fsmap learned FSMap version 15 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 -- 172.21.15.150:0/3948962272 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x55cfd5f56f10 con 0x55cfd5f68ca0 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe61ffff640 1 -- 172.21.15.150:0/3948962272 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55cfd5e6a110 con 0x55cfd5f68ca0 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe61ffff640 1 -- 172.21.15.150:0/3948962272 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 15) ==== 4136+0+0 (secure 0 0 0) 0x7fe6300b5070 con 0x55cfd5f68ca0 2024-09-17T04:49:58.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 10 client.4842 fetch_fsmap finished waiting for FSMap version 15 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 10 client.4842 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 20 client.4842 populate_metadata read hostname 'smithi150' 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.150:6833/2613040311,v1:172.21.15.150:6835/2613040311] conn(0x55cfd5b747d0 0x55cfd608a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 4 client.4842 mds_command: new command op to 4404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 -- 172.21.15.150:0/3948962272 --> [v2:172.21.15.150:6833/2613040311,v1:172.21.15.150:6835/2613040311] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfd5ee4a90 con 0x55cfd5b747d0 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6451ec640 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.150:6833/2613040311,v1:172.21.15.150:6835/2613040311] conn(0x55cfd5b747d0 0x55cfd608a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe61ffff640 10 client.4842 ms_handle_connect on v2:172.21.15.150:6833/2613040311 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe6451ec640 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.150:6833/2613040311,v1:172.21.15.150:6835/2613040311] conn(0x55cfd5b747d0 0x55cfd608a0e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cf0d19e98e3617d server_cookie=faae5d0d3008a1ad in_seq=0 out_seq=0 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe61ffff640 1 -- 172.21.15.150:0/3948962272 <== mds.0 v2:172.21.15.150:6833/2613040311 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cfd5ee4a90 con 0x55cfd5b747d0 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe61ffff640 10 client.4842 handle_command_reply: tid=0 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 client.4842 shutdown 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 -- 172.21.15.150:0/3948962272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 msgr2=0x55cfd6088e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd6088e60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe6300b0040 tx=0x7fe6300f85c0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 -- 172.21.15.150:0/3948962272 shutdown_connections 2024-09-17T04:49:58.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.150:6833/2613040311,v1:172.21.15.150:6835/2613040311] conn(0x55cfd5b747d0 0x55cfd608a0e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f7f950 0x55cfd60893a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd6088e60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 --2- 172.21.15.150:0/3948962272 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5b8f460 0x55cfd60c19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe61ffff640 0 client.4842 ms_handle_reset on v2:172.21.15.150:6833/2613040311 2024-09-17T04:49:58.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 -- 172.21.15.150:0/3948962272 >> 172.21.15.150:0/3948962272 conn(0x55cfd6055d90 msgr2=0x55cfd60c3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:58.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 -- 172.21.15.150:0/3948962272 shutdown_connections 2024-09-17T04:49:58.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 1 -- 172.21.15.150:0/3948962272 wait complete. 2024-09-17T04:49:58.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.720+0000 7fe64e71b000 20 client.4842 trim_cache size 0 max 16384 2024-09-17T04:49:58.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.740+0000 7fe64e71b000 1 Processor -- start 2024-09-17T04:49:58.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.740+0000 7fe64e71b000 1 -- start start 2024-09-17T04:49:58.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.740+0000 7fe64e71b000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5b8f460 0x55cfd6088e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.740+0000 7fe64e71b000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd60853e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f7f950 0x55cfd6085920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401fc560 con 0x55cfd5f68ca0 2024-09-17T04:49:58.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401fd5f0 con 0x55cfd5b8f460 2024-09-17T04:49:58.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe64014af70 con 0x55cfd5f7f950 2024-09-17T04:49:58.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6449eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd60853e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6449eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd60853e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33590/0 (socket says 172.21.15.150:33590) 2024-09-17T04:49:58.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6449eb640 1 -- 172.21.15.150:0/3395930108 learned_addr learned my addr 172.21.15.150:0/3395930108 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:58.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6449eb640 1 -- 172.21.15.150:0/3395930108 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f7f950 msgr2=0x55cfd6085920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:49:58.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6459ed640 1 --2- 172.21.15.150:0/3395930108 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f7f950 0x55cfd6085920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6449eb640 1 --2- 172.21.15.150:0/3395930108 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f7f950 0x55cfd6085920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6449eb640 1 -- 172.21.15.150:0/3395930108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5b8f460 msgr2=0x55cfd6088e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:49:58.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6451ec640 1 --2- 172.21.15.150:0/3395930108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5b8f460 0x55cfd6088e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6449eb640 1 --2- 172.21.15.150:0/3395930108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5b8f460 0x55cfd6088e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6449eb640 1 -- 172.21.15.150:0/3395930108 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfd6086f10 con 0x55cfd5f68ca0 2024-09-17T04:49:58.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6449eb640 1 --2- 172.21.15.150:0/3395930108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd60853e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe630005c50 tx=0x7fe63001c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0e91d67cd84e7a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:58.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe62a7fc640 1 -- 172.21.15.150:0/3395930108 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6300f8780 con 0x55cfd5f68ca0 2024-09-17T04:49:58.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe62a7fc640 1 -- 172.21.15.150:0/3395930108 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6300fd210 con 0x55cfd5f68ca0 2024-09-17T04:49:58.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe62a7fc640 1 -- 172.21.15.150:0/3395930108 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6300f7ea0 con 0x55cfd5f68ca0 2024-09-17T04:49:58.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 -- 172.21.15.150:0/3395930108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 msgr2=0x55cfd60853e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 --2- 172.21.15.150:0/3395930108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd60853e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe630005c50 tx=0x7fe63001c820 comp rx=0 tx=0).stop 2024-09-17T04:49:58.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 -- 172.21.15.150:0/3395930108 shutdown_connections 2024-09-17T04:49:58.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 --2- 172.21.15.150:0/3395930108 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f7f950 0x55cfd6085920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 --2- 172.21.15.150:0/3395930108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5f68ca0 0x55cfd60853e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 --2- 172.21.15.150:0/3395930108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5b8f460 0x55cfd6088e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 -- 172.21.15.150:0/3395930108 >> 172.21.15.150:0/3395930108 conn(0x55cfd6055d90 msgr2=0x55cfd6084cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:58.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 -- 172.21.15.150:0/3395930108 shutdown_connections 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 -- 172.21.15.150:0/3395930108 wait complete. 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 -- start start 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 Processor -- start 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5b8f460 0x55cfd60d9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f68ca0 0x55cfd60d9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5f7f950 0x55cfd60d9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55cfd5e6c5f0 con 0x55cfd5b8f460 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401fb680 con 0x55cfd5f7f950 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55cfd5ed5d80 con 0x55cfd5f68ca0 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe64e71b000 10 client.0 _ll_register_callbacks cb 0x7fe630005f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6451ec640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5b8f460 0x55cfd60d9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6451ec640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5b8f460 0x55cfd60d9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33598/0 (socket says 172.21.15.150:33598) 2024-09-17T04:49:58.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.744+0000 7fe6451ec640 1 -- 172.21.15.150:0/826207177 learned_addr learned my addr 172.21.15.150:0/826207177 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe6449eb640 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f68ca0 0x55cfd60d9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe6451ec640 1 -- 172.21.15.150:0/826207177 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f68ca0 msgr2=0x55cfd60d9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe6451ec640 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f68ca0 0x55cfd60d9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe6459ed640 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5f7f950 0x55cfd60d9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe6451ec640 1 -- 172.21.15.150:0/826207177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5f7f950 msgr2=0x55cfd60d9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe6451ec640 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5f7f950 0x55cfd60d9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe6451ec640 1 -- 172.21.15.150:0/826207177 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfd5f2ddb0 con 0x55cfd5b8f460 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe6451ec640 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5b8f460 0x55cfd60d9400 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe63c015620 tx=0x7fe63c000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f635d5a4b7e6ff62 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe61ffff640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe61ffff640 1 -- 172.21.15.150:0/826207177 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c030060 con 0x55cfd5b8f460 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe61ffff640 1 -- 172.21.15.150:0/826207177 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fe63c02bdd0 con 0x55cfd5b8f460 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 -- 172.21.15.150:0/826207177 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6300fd210 con 0x55cfd5b8f460 2024-09-17T04:49:58.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe61ffff640 1 -- 172.21.15.150:0/826207177 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fe63c01e040 con 0x55cfd5b8f460 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 10 client.4848 fetch_fsmap learned FSMap version 15 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 -- 172.21.15.150:0/826207177 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x55cfd5e98ab0 con 0x55cfd5b8f460 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe61ffff640 1 -- 172.21.15.150:0/826207177 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6300fd210 con 0x55cfd5b8f460 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe61ffff640 1 -- 172.21.15.150:0/826207177 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 15) ==== 4136+0+0 (secure 0 0 0) 0x7fe63c029070 con 0x55cfd5b8f460 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 10 client.4848 fetch_fsmap finished waiting for FSMap version 15 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 10 client.4848 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 20 client.4848 populate_metadata read hostname 'smithi150' 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.150:6833/2613040311,v1:172.21.15.150:6835/2613040311] conn(0x55cfd5b62830 0x55cfd614e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 4 client.4848 mds_command: new command op to 4404 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 -- 172.21.15.150:0/826207177 --> [v2:172.21.15.150:6833/2613040311,v1:172.21.15.150:6835/2613040311] -- command(tid 0: {"prefix": "status"}) -- 0x55cfd5ee0540 con 0x55cfd5b62830 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe6449eb640 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.150:6833/2613040311,v1:172.21.15.150:6835/2613040311] conn(0x55cfd5b62830 0x55cfd614e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe61ffff640 10 client.4848 ms_handle_connect on v2:172.21.15.150:6833/2613040311 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe6449eb640 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.150:6833/2613040311,v1:172.21.15.150:6835/2613040311] conn(0x55cfd5b62830 0x55cfd614e9f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5626be553a9a242f server_cookie=8a5526b28f6ce089 in_seq=0 out_seq=0 2024-09-17T04:49:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe61ffff640 1 -- 172.21.15.150:0/826207177 <== mds.0 v2:172.21.15.150:6833/2613040311 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55cfd5ee0540 con 0x55cfd5b62830 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe61ffff640 10 client.4848 handle_command_reply: tid=0 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 client.4848 shutdown 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 -- 172.21.15.150:0/826207177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5b8f460 msgr2=0x55cfd60d9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5b8f460 0x55cfd60d9400 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe63c015620 tx=0x7fe63c000bd0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 -- 172.21.15.150:0/826207177 shutdown_connections 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.150:6833/2613040311,v1:172.21.15.150:6835/2613040311] conn(0x55cfd5b62830 0x55cfd614e9f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55cfd5f7f950 0x55cfd60d9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe61ffff640 0 client.4848 ms_handle_reset on v2:172.21.15.150:6833/2613040311 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55cfd5f68ca0 0x55cfd60d9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 --2- 172.21.15.150:0/826207177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55cfd5b8f460 0x55cfd60d9400 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 -- 172.21.15.150:0/826207177 >> 172.21.15.150:0/826207177 conn(0x55cfd6055d90 msgr2=0x55cfd6085690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 -- 172.21.15.150:0/826207177 shutdown_connections 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 1 -- 172.21.15.150:0/826207177 wait complete. 2024-09-17T04:49:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe64e71b000 20 client.4848 trim_cache size 0 max 16384 2024-09-17T04:49:58.756 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-17T04:49:58.756 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "172984ad-e98f-497a-a83c-2299d42da6ea", 2024-09-17T04:49:58.756 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-09-17T04:49:58.756 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 4404, 2024-09-17T04:49:58.756 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-09-17T04:49:58.756 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-09-17T04:49:58.756 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "a", 2024-09-17T04:49:58.756 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 26.868065427000001, 2024-09-17T04:49:58.756 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 9, 2024-09-17T04:49:58.757 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 26, 2024-09-17T04:49:58.757 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 22, 2024-09-17T04:49:58.757 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 32.627915506999997 2024-09-17T04:49:58.757 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-17T04:49:58.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe647450640 1 -- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe624073230 msgr2=0x7fe624075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe647450640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe624073230 0x7fe624075740 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe6400c2500 tx=0x7fe63c009040 comp rx=0 tx=0).stop 2024-09-17T04:49:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe647450640 1 -- 172.21.15.150:0/3867226107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6400c3db0 msgr2=0x7fe640144d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe647450640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6400c3db0 0x7fe640144d10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe630009790 tx=0x7fe6300a5670 comp rx=0 tx=0).stop 2024-09-17T04:49:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe647450640 1 -- 172.21.15.150:0/3867226107 shutdown_connections 2024-09-17T04:49:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe647450640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe624073230 0x7fe624075740 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe647450640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6400c41c0 0x7fe640156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe647450640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6400c3db0 0x7fe640144d10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe647450640 1 --2- 172.21.15.150:0/3867226107 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe64006fff0 0x7fe6401483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe647450640 1 -- 172.21.15.150:0/3867226107 >> 172.21.15.150:0/3867226107 conn(0x7fe6400c4a00 msgr2=0x7fe64015e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe647450640 1 -- 172.21.15.150:0/3867226107 shutdown_connections 2024-09-17T04:49:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.748+0000 7fe647450640 1 -- 172.21.15.150:0/3867226107 wait complete. 2024-09-17T04:49:58.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:49:58.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.900+0000 7f5ce218b640 1 -- 172.21.15.150:0/152424255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc4004d50 msgr2=0x7f5cc4005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.900+0000 7f5ce218b640 1 --2- 172.21.15.150:0/152424255 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc4004d50 0x7f5cc4005160 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009eb0 tx=0x7f5cd401d5f0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.900+0000 7f5cdbfff640 1 -- 172.21.15.150:0/152424255 reap_dead start 2024-09-17T04:49:58.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.900+0000 7f5ce218b640 1 -- 172.21.15.150:0/152424255 shutdown_connections 2024-09-17T04:49:58.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.900+0000 7f5ce218b640 1 -- 172.21.15.150:0/152424255 >> 172.21.15.150:0/152424255 conn(0x7f5cdc0c4a00 msgr2=0x7f5cdc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:58.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.900+0000 7f5ce218b640 1 -- 172.21.15.150:0/152424255 shutdown_connections 2024-09-17T04:49:58.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.900+0000 7f5ce218b640 1 -- 172.21.15.150:0/152424255 wait complete. 2024-09-17T04:49:58.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5ce218b640 1 Processor -- start 2024-09-17T04:49:58.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5ce218b640 1 -- start start 2024-09-17T04:49:58.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5ce218b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cdc0ad730 0x7f5cdc151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5ce218b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5cdc147b90 0x7f5cdc147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5ce218b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5cdc144a60 0x7f5cdc144e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5ce218b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc0adce0 con 0x7f5cdc0ad730 2024-09-17T04:49:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5ce218b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc06f810 con 0x7f5cdc147b90 2024-09-17T04:49:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5ce218b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5cdc0ade60 con 0x7f5cdc144a60 2024-09-17T04:49:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5cdb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cdc0ad730 0x7f5cdc151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5cdb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cdc0ad730 0x7f5cdc151560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33620/0 (socket says 172.21.15.150:33620) 2024-09-17T04:49:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5cdb7fe640 1 -- 172.21.15.150:0/3337205877 learned_addr learned my addr 172.21.15.150:0/3337205877 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5cdb7fe640 1 -- 172.21.15.150:0/3337205877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5cdc144a60 msgr2=0x7f5cdc144e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5cdb7fe640 1 --2- 172.21.15.150:0/3337205877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5cdc144a60 0x7f5cdc144e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5cdb7fe640 1 -- 172.21.15.150:0/3337205877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5cdc147b90 msgr2=0x7f5cdc147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5cdb7fe640 1 --2- 172.21.15.150:0/3337205877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5cdc147b90 0x7f5cdc147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5cdb7fe640 1 -- 172.21.15.150:0/3337205877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cdc1e8530 con 0x7f5cdc0ad730 2024-09-17T04:49:58.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5cdb7fe640 1 --2- 172.21.15.150:0/3337205877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cdc0ad730 0x7f5cdc151560 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00f590 tx=0x7f5ccc00fa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:58.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5cc3fff640 1 -- 172.21.15.150:0/3337205877 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc017070 con 0x7f5cdc0ad730 2024-09-17T04:49:58.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5cc3fff640 1 -- 172.21.15.150:0/3337205877 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ccc011990 con 0x7f5cdc0ad730 2024-09-17T04:49:58.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5ce218b640 1 -- 172.21.15.150:0/3337205877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cdc1dea60 con 0x7f5cdc0ad730 2024-09-17T04:49:58.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5cc3fff640 1 -- 172.21.15.150:0/3337205877 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc011da0 con 0x7f5cdc0ad730 2024-09-17T04:49:58.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5ce218b640 1 -- 172.21.15.150:0/3337205877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cdc148af0 con 0x7f5cdc0ad730 2024-09-17T04:49:58.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.904+0000 7f5ce218b640 1 -- 172.21.15.150:0/3337205877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cdc154020 con 0x7f5cdc0ad730 2024-09-17T04:49:58.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.908+0000 7f5cc3fff640 1 -- 172.21.15.150:0/3337205877 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5ccc01a070 con 0x7f5cdc0ad730 2024-09-17T04:49:58.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.908+0000 7f5cc3fff640 1 --2- 172.21.15.150:0/3337205877 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5cc8073200 0x7f5cc8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:58.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.908+0000 7f5cc3fff640 1 -- 172.21.15.150:0/3337205877 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f5ccc0975a0 con 0x7f5cdc0ad730 2024-09-17T04:49:58.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.908+0000 7f5cdaffd640 1 --2- 172.21.15.150:0/3337205877 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5cc8073200 0x7f5cc8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:58.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.908+0000 7f5cdaffd640 1 --2- 172.21.15.150:0/3337205877 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5cc8073200 0x7f5cc8075710 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009aa0 tx=0x7f5cd0009560 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:58.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:58.912+0000 7f5cc3fff640 1 -- 172.21.15.150:0/3337205877 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5ccc0622a0 con 0x7f5cdc0ad730 2024-09-17T04:49:59.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.100+0000 7f5ce218b640 1 -- 172.21.15.150:0/3337205877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5cdc1dec30 con 0x7f5cdc0ad730 2024-09-17T04:49:59.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.100+0000 7f5cc3fff640 1 -- 172.21.15.150:0/3337205877 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f5ccc061ba0 con 0x7f5cdc0ad730 2024-09-17T04:49:59.104 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:59.104 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:49:59.104 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:49:59.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.104+0000 7f5ce218b640 1 -- 172.21.15.150:0/3337205877 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5cc8073200 msgr2=0x7f5cc8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:59.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.104+0000 7f5ce218b640 1 --2- 172.21.15.150:0/3337205877 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5cc8073200 0x7f5cc8075710 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0009aa0 tx=0x7f5cd0009560 comp rx=0 tx=0).stop 2024-09-17T04:49:59.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.104+0000 7f5ce218b640 1 -- 172.21.15.150:0/3337205877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cdc0ad730 msgr2=0x7f5cdc151560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:59.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.104+0000 7f5ce218b640 1 --2- 172.21.15.150:0/3337205877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cdc0ad730 0x7f5cdc151560 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00f590 tx=0x7f5ccc00fa50 comp rx=0 tx=0).stop 2024-09-17T04:49:59.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.104+0000 7f5ce218b640 1 -- 172.21.15.150:0/3337205877 shutdown_connections 2024-09-17T04:49:59.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.104+0000 7f5ce218b640 1 --2- 172.21.15.150:0/3337205877 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5cc8073200 0x7f5cc8075710 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.104+0000 7f5ce218b640 1 --2- 172.21.15.150:0/3337205877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5cdc147b90 0x7f5cdc147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.104+0000 7f5ce218b640 1 --2- 172.21.15.150:0/3337205877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5cdc144a60 0x7f5cdc144e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.104+0000 7f5ce218b640 1 --2- 172.21.15.150:0/3337205877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cdc0ad730 0x7f5cdc151560 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.104+0000 7f5ce218b640 1 -- 172.21.15.150:0/3337205877 >> 172.21.15.150:0/3337205877 conn(0x7f5cdc0c4a00 msgr2=0x7f5cdc0a8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:59.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.104+0000 7f5ce218b640 1 -- 172.21.15.150:0/3337205877 shutdown_connections 2024-09-17T04:49:59.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.104+0000 7f5ce218b640 1 -- 172.21.15.150:0/3337205877 wait complete. 2024-09-17T04:49:59.125 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:49:59.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.260+0000 7f09cac49640 1 -- 172.21.15.150:0/4277291321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c40701c0 msgr2=0x7f09c4160060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:59.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.260+0000 7f09cac49640 1 --2- 172.21.15.150:0/4277291321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c40701c0 0x7f09c4160060 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f09c4072820 tx=0x7f09bc01cba0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.260+0000 7f09cac49640 1 -- 172.21.15.150:0/4277291321 shutdown_connections 2024-09-17T04:49:59.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.260+0000 7f09cac49640 1 --2- 172.21.15.150:0/4277291321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c40701c0 0x7f09c4160060 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.260+0000 7f09cac49640 1 -- 172.21.15.150:0/4277291321 >> 172.21.15.150:0/4277291321 conn(0x7f09c40c4a00 msgr2=0x7f09c40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:59.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.260+0000 7f09cac49640 1 -- 172.21.15.150:0/4277291321 shutdown_connections 2024-09-17T04:49:59.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.260+0000 7f09cac49640 1 -- 172.21.15.150:0/4277291321 wait complete. 2024-09-17T04:49:59.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.260+0000 7f09cac49640 1 Processor -- start 2024-09-17T04:49:59.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.260+0000 7f09cac49640 1 -- start start 2024-09-17T04:49:59.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09cac49640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09c40701c0 0x7f09c4151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:59.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09cac49640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c40c3db0 0x7f09c4155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:59.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09cac49640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09c40c41c0 0x7f09c41500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:59.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09cac49640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f09c4070750 con 0x7f09c40c3db0 2024-09-17T04:49:59.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09cac49640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f09c40705d0 con 0x7f09c40c41c0 2024-09-17T04:49:59.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09cac49640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f09c40ad9b0 con 0x7f09c40701c0 2024-09-17T04:49:59.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09bbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c40c3db0 0x7f09c4155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:59.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09bbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c40c3db0 0x7f09c4155ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58296/0 (socket says 172.21.15.150:58296) 2024-09-17T04:49:59.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09bbfff640 1 -- 172.21.15.150:0/3514441048 learned_addr learned my addr 172.21.15.150:0/3514441048 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:59.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09c91e6640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09c40c41c0 0x7f09c41500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09c89e5640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09c40701c0 0x7f09c4151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09bbfff640 1 -- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09c40701c0 msgr2=0x7f09c4151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09bbfff640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09c40701c0 0x7f09c4151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09bbfff640 1 -- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09c40c41c0 msgr2=0x7f09c41500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09bbfff640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09c40c41c0 0x7f09c41500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09bbfff640 1 -- 172.21.15.150:0/3514441048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c414f950 con 0x7f09c40c3db0 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09c89e5640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09c40701c0 0x7f09c4151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09c91e6640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09c40c41c0 0x7f09c41500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09bbfff640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c40c3db0 0x7f09c4155ac0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f09b400b3e0 tx=0x7f09b400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09b97fa640 1 -- 172.21.15.150:0/3514441048 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4015020 con 0x7f09c40c3db0 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09cac49640 1 -- 172.21.15.150:0/3514441048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c4150bc0 con 0x7f09c40c3db0 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09b97fa640 1 -- 172.21.15.150:0/3514441048 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09b4011920 con 0x7f09c40c3db0 2024-09-17T04:49:59.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09b97fa640 1 -- 172.21.15.150:0/3514441048 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4011db0 con 0x7f09c40c3db0 2024-09-17T04:49:59.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09cac49640 1 -- 172.21.15.150:0/3514441048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c4149c20 con 0x7f09c40c3db0 2024-09-17T04:49:59.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09cac49640 1 -- 172.21.15.150:0/3514441048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c40a7780 con 0x7f09c40c3db0 2024-09-17T04:49:59.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09b97fa640 1 -- 172.21.15.150:0/3514441048 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f09b4018020 con 0x7f09c40c3db0 2024-09-17T04:49:59.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09b97fa640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f09b0073230 0x7f09b0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:59.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09b97fa640 1 -- 172.21.15.150:0/3514441048 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f09b40944f0 con 0x7f09c40c3db0 2024-09-17T04:49:59.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.264+0000 7f09c89e5640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f09b0073230 0x7f09b0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:59.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.268+0000 7f09c89e5640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f09b0073230 0x7f09b0075740 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f09c0000f70 tx=0x7f09c0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:59.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.268+0000 7f09b97fa640 1 -- 172.21.15.150:0/3514441048 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f09b401f020 con 0x7f09c40c3db0 2024-09-17T04:49:59.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.428+0000 7f09cac49640 1 -- 172.21.15.150:0/3514441048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09c40a9450 con 0x7f09c40c3db0 2024-09-17T04:49:59.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.432+0000 7f09b97fa640 1 -- 172.21.15.150:0/3514441048 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20431 (secure 0 0 0) 0x7f09b405f1f0 con 0x7f09c40c3db0 2024-09-17T04:49:59.435 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:59.435 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:42.029799+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:49:59.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.436+0000 7f09cac49640 1 -- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f09b0073230 msgr2=0x7f09b0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:59.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.436+0000 7f09cac49640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f09b0073230 0x7f09b0075740 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f09c0000f70 tx=0x7f09c0009040 comp rx=0 tx=0).stop 2024-09-17T04:49:59.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.436+0000 7f09cac49640 1 -- 172.21.15.150:0/3514441048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c40c3db0 msgr2=0x7f09c4155ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:59.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.436+0000 7f09cac49640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c40c3db0 0x7f09c4155ac0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f09b400b3e0 tx=0x7f09b400b8a0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.436+0000 7f09cac49640 1 -- 172.21.15.150:0/3514441048 shutdown_connections 2024-09-17T04:49:59.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.436+0000 7f09cac49640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f09b0073230 0x7f09b0075740 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.436+0000 7f09cac49640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09c40c41c0 0x7f09c41500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.436+0000 7f09cac49640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09c40c3db0 0x7f09c4155ac0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.436+0000 7f09cac49640 1 --2- 172.21.15.150:0/3514441048 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09c40701c0 0x7f09c4151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.436+0000 7f09cac49640 1 -- 172.21.15.150:0/3514441048 >> 172.21.15.150:0/3514441048 conn(0x7f09c40c4a00 msgr2=0x7f09c40c5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:59.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.436+0000 7f09cac49640 1 -- 172.21.15.150:0/3514441048 shutdown_connections 2024-09-17T04:49:59.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.436+0000 7f09cac49640 1 -- 172.21.15.150:0/3514441048 wait complete. 2024-09-17T04:49:59.455 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:49:59.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.600+0000 7fb181f69640 1 -- 172.21.15.150:0/3704806305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c075150 msgr2=0x7fb17c157d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:59.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.600+0000 7fb181f69640 1 --2- 172.21.15.150:0/3704806305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c075150 0x7fb17c157d60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0c7060 tx=0x7fb17401c770 comp rx=0 tx=0).stop 2024-09-17T04:49:59.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.600+0000 7fb181f69640 1 -- 172.21.15.150:0/3704806305 shutdown_connections 2024-09-17T04:49:59.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.600+0000 7fb181f69640 1 --2- 172.21.15.150:0/3704806305 >> v2:172.21.15.176:3300/0 conn(0x7fb17c0ade20 0x7fb17c074c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.600+0000 7fb181f69640 1 --2- 172.21.15.150:0/3704806305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c075150 0x7fb17c157d60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.600+0000 7fb181f69640 1 -- 172.21.15.150:0/3704806305 >> 172.21.15.150:0/3704806305 conn(0x7fb17c0aadc0 msgr2=0x7fb17c0ab1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:59.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.600+0000 7fb181f69640 1 -- 172.21.15.150:0/3704806305 shutdown_connections 2024-09-17T04:49:59.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.600+0000 7fb181f69640 1 -- 172.21.15.150:0/3704806305 wait complete. 2024-09-17T04:49:59.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.600+0000 7fb181f69640 1 Processor -- start 2024-09-17T04:49:59.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.600+0000 7fb181f69640 1 -- start start 2024-09-17T04:49:59.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb181f69640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb17c075150 0x7fb17c070430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:59.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb181f69640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c0ad8d0 0x7fb17c14ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:59.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb181f69640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb17c0ade20 0x7fb17c14c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:59.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb181f69640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c161400 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb181f69640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c161280 con 0x7fb17c0ade20 2024-09-17T04:49:59.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb181f69640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb17c161580 con 0x7fb17c075150 2024-09-17T04:49:59.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb17affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c0ad8d0 0x7fb17c14ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:59.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb17affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c0ad8d0 0x7fb17c14ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58320/0 (socket says 172.21.15.150:58320) 2024-09-17T04:49:59.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb17affd640 1 -- 172.21.15.150:0/962195085 learned_addr learned my addr 172.21.15.150:0/962195085 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:49:59.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb17b7fe640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb17c075150 0x7fb17c070430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:59.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb17bfff640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb17c0ade20 0x7fb17c14c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:59.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb17affd640 1 -- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb17c075150 msgr2=0x7fb17c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:59.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb17affd640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb17c075150 0x7fb17c070430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb17affd640 1 -- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb17c0ade20 msgr2=0x7fb17c14c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:59.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb17affd640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb17c0ade20 0x7fb17c14c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb17affd640 1 -- 172.21.15.150:0/962195085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17c070ae0 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb17affd640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c0ad8d0 0x7fb17c14ea00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00eaa0 tx=0x7fb16c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:59.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb173fff640 1 -- 172.21.15.150:0/962195085 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c00cdb0 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb173fff640 1 -- 172.21.15.150:0/962195085 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb16c011470 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb173fff640 1 -- 172.21.15.150:0/962195085 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c0118a0 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb181f69640 1 -- 172.21.15.150:0/962195085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17c1f3780 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb181f69640 1 -- 172.21.15.150:0/962195085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb17c1ef680 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb181f69640 1 -- 172.21.15.150:0/962195085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb17c1f35b0 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.604+0000 7fb173fff640 1 -- 172.21.15.150:0/962195085 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb16c01a400 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.608+0000 7fb173fff640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb15c073200 0x7fb15c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:49:59.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.608+0000 7fb173fff640 1 -- 172.21.15.150:0/962195085 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fb16c095660 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.608+0000 7fb17b7fe640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb15c073200 0x7fb15c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:49:59.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.608+0000 7fb17b7fe640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb15c073200 0x7fb15c075710 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1640015f0 tx=0x7fb164009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:49:59.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.612+0000 7fb173fff640 1 -- 172.21.15.150:0/962195085 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb16c028020 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.812+0000 7fb181f69640 1 -- 172.21.15.150:0/962195085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb17c157d60 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.812+0000 7fb173fff640 1 -- 172.21.15.150:0/962195085 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fb16c016030 con 0x7fb17c0ad8d0 2024-09-17T04:49:59.814 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:49:59.814 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:49:59.815 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:49:59.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.812+0000 7fb181f69640 1 -- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb15c073200 msgr2=0x7fb15c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:59.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.812+0000 7fb181f69640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb15c073200 0x7fb15c075710 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1640015f0 tx=0x7fb164009040 comp rx=0 tx=0).stop 2024-09-17T04:49:59.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.812+0000 7fb181f69640 1 -- 172.21.15.150:0/962195085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c0ad8d0 msgr2=0x7fb17c14ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:49:59.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.812+0000 7fb181f69640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c0ad8d0 0x7fb17c14ea00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00eaa0 tx=0x7fb16c00ef60 comp rx=0 tx=0).stop 2024-09-17T04:49:59.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.812+0000 7fb181f69640 1 -- 172.21.15.150:0/962195085 shutdown_connections 2024-09-17T04:49:59.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.812+0000 7fb181f69640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb15c073200 0x7fb15c075710 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.812+0000 7fb181f69640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb17c0ade20 0x7fb17c14c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.812+0000 7fb181f69640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb17c0ad8d0 0x7fb17c14ea00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.816+0000 7fb181f69640 1 --2- 172.21.15.150:0/962195085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb17c075150 0x7fb17c070430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:49:59.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.816+0000 7fb181f69640 1 -- 172.21.15.150:0/962195085 >> 172.21.15.150:0/962195085 conn(0x7fb17c0aadc0 msgr2=0x7fb17c15e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:49:59.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.816+0000 7fb181f69640 1 -- 172.21.15.150:0/962195085 shutdown_connections 2024-09-17T04:49:59.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:49:59.816+0000 7fb181f69640 1 -- 172.21.15.150:0/962195085 wait complete. 2024-09-17T04:50:00.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:49:40.378290+0000', 'modified': '2024-09-17T04:49:44.467846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4401}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4401': {'gid': 4401, 'name': 'a', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.150:6834/480168496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6832', 'nonce': 480168496}, {'type': 'v1', 'addr': '172.21.15.150:6834', 'nonce': 480168496}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'b', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4401, 'qdb_cluster': [4401]} 2024-09-17T04:50:00.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:50:00.240 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-17T04:50:00.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.404+0000 7f69cc254640 1 -- 172.21.15.150:0/2638569938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69c40717a0 msgr2=0x7f69c4074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:00.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.404+0000 7f69cc254640 1 --2- 172.21.15.150:0/2638569938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69c40717a0 0x7f69c4074450 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009c80 tx=0x7f69bc01cd10 comp rx=0 tx=0).stop 2024-09-17T04:50:00.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.404+0000 7f69cc254640 1 -- 172.21.15.150:0/2638569938 shutdown_connections 2024-09-17T04:50:00.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.404+0000 7f69cc254640 1 --2- 172.21.15.150:0/2638569938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69c40717a0 0x7f69c4074450 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.404+0000 7f69cc254640 1 --2- 172.21.15.150:0/2638569938 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69c4070e50 0x7f69c4071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.404+0000 7f69cc254640 1 -- 172.21.15.150:0/2638569938 >> 172.21.15.150:0/2638569938 conn(0x7f69c40c4a00 msgr2=0x7f69c40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:00.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.404+0000 7f69cc254640 1 -- 172.21.15.150:0/2638569938 shutdown_connections 2024-09-17T04:50:00.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.404+0000 7f69cc254640 1 -- 172.21.15.150:0/2638569938 wait complete. 2024-09-17T04:50:00.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69cc254640 1 Processor -- start 2024-09-17T04:50:00.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69cc254640 1 -- start start 2024-09-17T04:50:00.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69cc254640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69c4070e50 0x7f69c41f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69cc254640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69c40717a0 0x7f69c41ef190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69cc254640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69c4073440 0x7f69c41f0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69cc254640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f69c40ad240 con 0x7f69c4070e50 2024-09-17T04:50:00.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69cc254640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f69c40ad0c0 con 0x7f69c40717a0 2024-09-17T04:50:00.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69cc254640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f69c40ad3c0 con 0x7f69c4073440 2024-09-17T04:50:00.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69c9ff0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69c4070e50 0x7f69c41f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69c9ff0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69c4070e50 0x7f69c41f11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58324/0 (socket says 172.21.15.150:58324) 2024-09-17T04:50:00.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69c9ff0640 1 -- 172.21.15.150:0/4015371115 learned_addr learned my addr 172.21.15.150:0/4015371115 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:00.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69ca7f1640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69c4073440 0x7f69c41f0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69c97ef640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69c40717a0 0x7f69c41ef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69c9ff0640 1 -- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69c4073440 msgr2=0x7f69c41f0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69c4073440 0x7f69c41f0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69c9ff0640 1 -- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69c40717a0 msgr2=0x7f69c41ef190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69c40717a0 0x7f69c41ef190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69c9ff0640 1 -- 172.21.15.150:0/4015371115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c4157400 con 0x7f69c4070e50 2024-09-17T04:50:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69c97ef640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69c40717a0 0x7f69c41ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:50:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69c4070e50 0x7f69c41f11c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f69b800b410 tx=0x7f69b800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69b67fc640 1 -- 172.21.15.150:0/4015371115 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b8015030 con 0x7f69c4070e50 2024-09-17T04:50:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69b67fc640 1 -- 172.21.15.150:0/4015371115 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69b8004020 con 0x7f69c4070e50 2024-09-17T04:50:00.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69cc254640 1 -- 172.21.15.150:0/4015371115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c41578b0 con 0x7f69c4070e50 2024-09-17T04:50:00.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69b67fc640 1 -- 172.21.15.150:0/4015371115 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b8004470 con 0x7f69c4070e50 2024-09-17T04:50:00.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.408+0000 7f69cc254640 1 -- 172.21.15.150:0/4015371115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c41f1ba0 con 0x7f69c4070e50 2024-09-17T04:50:00.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69b67fc640 1 -- 172.21.15.150:0/4015371115 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f69b8018070 con 0x7f69c4070e50 2024-09-17T04:50:00.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69b67fc640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f69a8073230 0x7f69a8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69b67fc640 1 -- 172.21.15.150:0/4015371115 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f69b8094050 con 0x7f69c4070e50 2024-09-17T04:50:00.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69c97ef640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f69a8073230 0x7f69a8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69d351f000 1 Processor -- start 2024-09-17T04:50:00.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69d351f000 1 -- start start 2024-09-17T04:50:00.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69c97ef640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f69a8073230 0x7f69a8075740 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69c4152180 tx=0x7f69c0005d20 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:00.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69d351f000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 0x55a6ff103f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69d351f000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6fefc1950 0x55a6ff0c6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69d351f000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefaaca0 0x55a6ff0c6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69d351f000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55a6fefe7410 con 0x55a6febd1460 2024-09-17T04:50:00.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69d351f000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55a6fef24b40 con 0x55a6fefc1950 2024-09-17T04:50:00.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69d351f000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55a6fef13f70 con 0x55a6fefaaca0 2024-09-17T04:50:00.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69c9ff0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 0x55a6ff103f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69c9ff0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 0x55a6ff103f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58340/0 (socket says 172.21.15.150:58340) 2024-09-17T04:50:00.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69c9ff0640 1 -- 172.21.15.150:0/1856870855 learned_addr learned my addr 172.21.15.150:0/1856870855 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:00.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69c9ff0640 1 -- 172.21.15.150:0/1856870855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefaaca0 msgr2=0x55a6ff0c6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:50:00.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69ca7f1640 1 --2- 172.21.15.150:0/1856870855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefaaca0 0x55a6ff0c6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/1856870855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefaaca0 0x55a6ff0c6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69c9ff0640 1 -- 172.21.15.150:0/1856870855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6fefc1950 msgr2=0x55a6ff0c6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:50:00.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/1856870855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6fefc1950 0x55a6ff0c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69c9ff0640 1 -- 172.21.15.150:0/1856870855 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6feedb5b0 con 0x55a6febd1460 2024-09-17T04:50:00.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/1856870855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 0x55a6ff103f90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f69b80921f0 tx=0x7f69b805e310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7b76c5757a21a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69a77fe640 1 -- 172.21.15.150:0/1856870855 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b8068060 con 0x55a6febd1460 2024-09-17T04:50:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.412+0000 7f69a77fe640 1 -- 172.21.15.150:0/1856870855 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69b800fb60 con 0x55a6febd1460 2024-09-17T04:50:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69a77fe640 1 -- 172.21.15.150:0/1856870855 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b805d930 con 0x55a6febd1460 2024-09-17T04:50:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 -- 172.21.15.150:0/1856870855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 msgr2=0x55a6ff103f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 --2- 172.21.15.150:0/1856870855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 0x55a6ff103f90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f69b80921f0 tx=0x7f69b805e310 comp rx=0 tx=0).stop 2024-09-17T04:50:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 -- 172.21.15.150:0/1856870855 shutdown_connections 2024-09-17T04:50:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 --2- 172.21.15.150:0/1856870855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6fefc1950 0x55a6ff0c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 --2- 172.21.15.150:0/1856870855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefaaca0 0x55a6ff0c6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 --2- 172.21.15.150:0/1856870855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 0x55a6ff103f90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 -- 172.21.15.150:0/1856870855 >> 172.21.15.150:0/1856870855 conn(0x55a6ff097d90 msgr2=0x55a6ff1069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 -- 172.21.15.150:0/1856870855 shutdown_connections 2024-09-17T04:50:00.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 -- 172.21.15.150:0/1856870855 wait complete. 2024-09-17T04:50:00.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 -- start start 2024-09-17T04:50:00.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 Processor -- start 2024-09-17T04:50:00.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6febd1460 0x55a6ff1039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 0x55a6ff0cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefc1950 0x55a6ff0cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55a6feeae5f0 con 0x55a6fefaaca0 2024-09-17T04:50:00.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55a6fef17d80 con 0x55a6febd1460 2024-09-17T04:50:00.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55a6feeba520 con 0x55a6fefc1950 2024-09-17T04:50:00.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 10 client.0 _ll_register_callbacks cb 0x55a6feebe830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:50:00.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69c97ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 0x55a6ff0cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69c97ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 0x55a6ff0cae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58342/0 (socket says 172.21.15.150:58342) 2024-09-17T04:50:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69c97ef640 1 -- 172.21.15.150:0/237916719 learned_addr learned my addr 172.21.15.150:0/237916719 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69ca7f1640 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefc1950 0x55a6ff0cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6febd1460 0x55a6ff1039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69c97ef640 1 -- 172.21.15.150:0/237916719 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefc1950 msgr2=0x55a6ff0cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:50:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69c97ef640 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefc1950 0x55a6ff0cb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69c97ef640 1 -- 172.21.15.150:0/237916719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6febd1460 msgr2=0x55a6ff1039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:50:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69c97ef640 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6febd1460 0x55a6ff1039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69c97ef640 1 -- 172.21.15.150:0/237916719 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6feeb2190 con 0x55a6fefaaca0 2024-09-17T04:50:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69c97ef640 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 0x55a6ff0cae90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f69c000f9a0 tx=0x7f69c00046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f379aa3da73414 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-17T04:50:00.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69a4ff9640 1 -- 172.21.15.150:0/237916719 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0020070 con 0x55a6fefaaca0 2024-09-17T04:50:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69a4ff9640 1 -- 172.21.15.150:0/237916719 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f69c000ecc0 con 0x55a6fefaaca0 2024-09-17T04:50:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.416+0000 7f69d351f000 1 -- 172.21.15.150:0/237916719 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a6feeac110 con 0x55a6fefaaca0 2024-09-17T04:50:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69a4ff9640 1 -- 172.21.15.150:0/237916719 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f69c002b050 con 0x55a6fefaaca0 2024-09-17T04:50:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 10 client.4884 fetch_fsmap learned FSMap version 15 2024-09-17T04:50:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 -- 172.21.15.150:0/237916719 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x55a6fef98f10 con 0x55a6fefaaca0 2024-09-17T04:50:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69a4ff9640 1 -- 172.21.15.150:0/237916719 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x55a6feeac110 con 0x55a6fefaaca0 2024-09-17T04:50:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69a4ff9640 1 -- 172.21.15.150:0/237916719 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 15) ==== 4136+0+0 (secure 0 0 0) 0x7f69c002e020 con 0x55a6fefaaca0 2024-09-17T04:50:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 10 client.4884 fetch_fsmap finished waiting for FSMap version 15 2024-09-17T04:50:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 10 client.4884 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T04:50:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 20 client.4884 populate_metadata read hostname 'smithi150' 2024-09-17T04:50:00.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.150:6832/480168496,v1:172.21.15.150:6834/480168496] conn(0x55a6febb67d0 0x55a6ff0cc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 4 client.4884 mds_command: new command op to 4401 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:50:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 -- 172.21.15.150:0/237916719 --> [v2:172.21.15.150:6832/480168496,v1:172.21.15.150:6834/480168496] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6fef26a90 con 0x55a6febb67d0 2024-09-17T04:50:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.150:6832/480168496,v1:172.21.15.150:6834/480168496] conn(0x55a6febb67d0 0x55a6ff0cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69a4ff9640 10 client.4884 ms_handle_connect on v2:172.21.15.150:6832/480168496 2024-09-17T04:50:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.150:6832/480168496,v1:172.21.15.150:6834/480168496] conn(0x55a6febb67d0 0x55a6ff0cc110 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f152ce9921abcce6 server_cookie=3797b6103405ce98 in_seq=0 out_seq=0 2024-09-17T04:50:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69a4ff9640 1 -- 172.21.15.150:0/237916719 <== mds.0 v2:172.21.15.150:6832/480168496 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a6fef26a90 con 0x55a6febb67d0 2024-09-17T04:50:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69a4ff9640 10 client.4884 handle_command_reply: tid=0 2024-09-17T04:50:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 client.4884 shutdown 2024-09-17T04:50:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 -- 172.21.15.150:0/237916719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 msgr2=0x55a6ff0cae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 0x55a6ff0cae90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f69c000f9a0 tx=0x7f69c00046d0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 -- 172.21.15.150:0/237916719 shutdown_connections 2024-09-17T04:50:00.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.150:6832/480168496,v1:172.21.15.150:6834/480168496] conn(0x55a6febb67d0 0x55a6ff0cc110 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefc1950 0x55a6ff0cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69a4ff9640 0 client.4884 ms_handle_reset on v2:172.21.15.150:6832/480168496 2024-09-17T04:50:00.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 0x55a6ff0cae90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 --2- 172.21.15.150:0/237916719 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6febd1460 0x55a6ff1039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 -- 172.21.15.150:0/237916719 >> 172.21.15.150:0/237916719 conn(0x55a6ff097d90 msgr2=0x55a6ff105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:00.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 -- 172.21.15.150:0/237916719 shutdown_connections 2024-09-17T04:50:00.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 1 -- 172.21.15.150:0/237916719 wait complete. 2024-09-17T04:50:00.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.420+0000 7f69d351f000 20 client.4884 trim_cache size 0 max 16384 2024-09-17T04:50:00.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 Processor -- start 2024-09-17T04:50:00.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 -- start start 2024-09-17T04:50:00.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 0x55a6ff11f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefaaca0 0x55a6ff11f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6fefc1950 0x55a6ff0c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41fc810 con 0x55a6febd1460 2024-09-17T04:50:00.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41fd8a0 con 0x55a6fefc1950 2024-09-17T04:50:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f69c41fb310 con 0x55a6fefaaca0 2024-09-17T04:50:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69c9ff0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 0x55a6ff11f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69c9ff0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 0x55a6ff11f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58344/0 (socket says 172.21.15.150:58344) 2024-09-17T04:50:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69c9ff0640 1 -- 172.21.15.150:0/2872374432 learned_addr learned my addr 172.21.15.150:0/2872374432 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69c97ef640 1 --2- 172.21.15.150:0/2872374432 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefaaca0 0x55a6ff11f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69c9ff0640 1 -- 172.21.15.150:0/2872374432 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefaaca0 msgr2=0x55a6ff11f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:50:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69ca7f1640 1 --2- 172.21.15.150:0/2872374432 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6fefc1950 0x55a6ff0c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/2872374432 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefaaca0 0x55a6ff11f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69c9ff0640 1 -- 172.21.15.150:0/2872374432 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6fefc1950 msgr2=0x55a6ff0c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:50:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/2872374432 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6fefc1950 0x55a6ff0c9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69c9ff0640 1 -- 172.21.15.150:0/2872374432 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6ff0c8f10 con 0x55a6febd1460 2024-09-17T04:50:00.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69c97ef640 1 --2- 172.21.15.150:0/2872374432 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefaaca0 0x55a6ff11f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:50:00.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/2872374432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 0x55a6ff11f200 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f69b805b940 tx=0x7f69b805e310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af326e5d7310806d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:00.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69a6ffd640 1 -- 172.21.15.150:0/2872374432 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b8078b90 con 0x55a6febd1460 2024-09-17T04:50:00.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69a6ffd640 1 -- 172.21.15.150:0/2872374432 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69b807c070 con 0x55a6febd1460 2024-09-17T04:50:00.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69a6ffd640 1 -- 172.21.15.150:0/2872374432 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b800fb60 con 0x55a6febd1460 2024-09-17T04:50:00.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 -- 172.21.15.150:0/2872374432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 msgr2=0x55a6ff11f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:00.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 --2- 172.21.15.150:0/2872374432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 0x55a6ff11f200 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f69b805b940 tx=0x7f69b805e310 comp rx=0 tx=0).stop 2024-09-17T04:50:00.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 -- 172.21.15.150:0/2872374432 shutdown_connections 2024-09-17T04:50:00.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 --2- 172.21.15.150:0/2872374432 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6fefc1950 0x55a6ff0c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 --2- 172.21.15.150:0/2872374432 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefaaca0 0x55a6ff11f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 --2- 172.21.15.150:0/2872374432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6febd1460 0x55a6ff11f200 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 -- 172.21.15.150:0/2872374432 >> 172.21.15.150:0/2872374432 conn(0x55a6ff097d90 msgr2=0x55a6ff120050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:00.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 -- 172.21.15.150:0/2872374432 shutdown_connections 2024-09-17T04:50:00.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.440+0000 7f69d351f000 1 -- 172.21.15.150:0/2872374432 wait complete. 2024-09-17T04:50:00.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 -- start start 2024-09-17T04:50:00.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 Processor -- start 2024-09-17T04:50:00.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6febd1460 0x55a6ff148e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 0x55a6ff149370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefc1950 0x55a6ff1498b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55a6feeae5f0 con 0x55a6fefaaca0 2024-09-17T04:50:00.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55a6fef17090 con 0x55a6febd1460 2024-09-17T04:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x55a6fef1c3b0 con 0x55a6fefc1950 2024-09-17T04:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 10 client.0 _ll_register_callbacks cb 0x7f69c0013070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69c97ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 0x55a6ff149370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69c97ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 0x55a6ff149370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58356/0 (socket says 172.21.15.150:58356) 2024-09-17T04:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69c97ef640 1 -- 172.21.15.150:0/3624632033 learned_addr learned my addr 172.21.15.150:0/3624632033 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6febd1460 0x55a6ff148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69ca7f1640 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefc1950 0x55a6ff1498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69c97ef640 1 -- 172.21.15.150:0/3624632033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefc1950 msgr2=0x55a6ff1498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69c97ef640 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefc1950 0x55a6ff1498b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69c97ef640 1 -- 172.21.15.150:0/3624632033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6febd1460 msgr2=0x55a6ff148e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69c97ef640 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6febd1460 0x55a6ff148e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69c97ef640 1 -- 172.21.15.150:0/3624632033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6fef6fdb0 con 0x55a6fefaaca0 2024-09-17T04:50:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69c97ef640 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 0x55a6ff149370 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f69c000dee0 tx=0x7f69c000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b91b58c283ebc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-09-17T04:50:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69a4ff9640 1 -- 172.21.15.150:0/3624632033 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0033020 con 0x55a6fefaaca0 2024-09-17T04:50:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69a4ff9640 1 -- 172.21.15.150:0/3624632033 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f69c001ccf0 con 0x55a6fefaaca0 2024-09-17T04:50:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 -- 172.21.15.150:0/3624632033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69b807c070 con 0x55a6fefaaca0 2024-09-17T04:50:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69a4ff9640 1 -- 172.21.15.150:0/3624632033 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f69c003d050 con 0x55a6fefaaca0 2024-09-17T04:50:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 10 client.4890 fetch_fsmap learned FSMap version 15 2024-09-17T04:50:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 -- 172.21.15.150:0/3624632033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x55a6feedaab0 con 0x55a6fefaaca0 2024-09-17T04:50:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69a4ff9640 1 -- 172.21.15.150:0/3624632033 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69b807c070 con 0x55a6fefaaca0 2024-09-17T04:50:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69a4ff9640 1 -- 172.21.15.150:0/3624632033 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 15) ==== 4136+0+0 (secure 0 0 0) 0x7f69c0040020 con 0x55a6fefaaca0 2024-09-17T04:50:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 10 client.4890 fetch_fsmap finished waiting for FSMap version 15 2024-09-17T04:50:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 10 client.4890 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T04:50:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 20 client.4890 populate_metadata read hostname 'smithi150' 2024-09-17T04:50:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.150:6832/480168496,v1:172.21.15.150:6834/480168496] conn(0x55a6feba4830 0x55a6ff14a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 4 client.4890 mds_command: new command op to 4401 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:50:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 -- 172.21.15.150:0/3624632033 --> [v2:172.21.15.150:6832/480168496,v1:172.21.15.150:6834/480168496] -- command(tid 0: {"prefix": "status"}) -- 0x55a6fef22540 con 0x55a6feba4830 2024-09-17T04:50:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.150:6832/480168496,v1:172.21.15.150:6834/480168496] conn(0x55a6feba4830 0x55a6ff14a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69a4ff9640 10 client.4890 ms_handle_connect on v2:172.21.15.150:6832/480168496 2024-09-17T04:50:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69c9ff0640 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.150:6832/480168496,v1:172.21.15.150:6834/480168496] conn(0x55a6feba4830 0x55a6ff14a150 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ffccc1b1b867b82 server_cookie=2e712559d2d28cf4 in_seq=0 out_seq=0 2024-09-17T04:50:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69a4ff9640 1 -- 172.21.15.150:0/3624632033 <== mds.0 v2:172.21.15.150:6832/480168496 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55a6fef22540 con 0x55a6feba4830 2024-09-17T04:50:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69a4ff9640 10 client.4890 handle_command_reply: tid=0 2024-09-17T04:50:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 client.4890 shutdown 2024-09-17T04:50:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 -- 172.21.15.150:0/3624632033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 msgr2=0x55a6ff149370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.444+0000 7f69d351f000 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 0x55a6ff149370 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f69c000dee0 tx=0x7f69c000ed70 comp rx=0 tx=0).stop 2024-09-17T04:50:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69d351f000 1 -- 172.21.15.150:0/3624632033 shutdown_connections 2024-09-17T04:50:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69d351f000 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.150:6832/480168496,v1:172.21.15.150:6834/480168496] conn(0x55a6feba4830 0x55a6ff14a150 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69d351f000 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x55a6fefc1950 0x55a6ff1498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69a4ff9640 0 client.4890 ms_handle_reset on v2:172.21.15.150:6832/480168496 2024-09-17T04:50:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69d351f000 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a6fefaaca0 0x55a6ff149370 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69d351f000 1 --2- 172.21.15.150:0/3624632033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55a6febd1460 0x55a6ff148e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69d351f000 1 -- 172.21.15.150:0/3624632033 >> 172.21.15.150:0/3624632033 conn(0x55a6ff097d90 msgr2=0x55a6ff0c78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69d351f000 1 -- 172.21.15.150:0/3624632033 shutdown_connections 2024-09-17T04:50:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69d351f000 1 -- 172.21.15.150:0/3624632033 wait complete. 2024-09-17T04:50:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69d351f000 20 client.4890 trim_cache size 0 max 16384 2024-09-17T04:50:00.456 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-17T04:50:00.456 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "172984ad-e98f-497a-a83c-2299d42da6ea", 2024-09-17T04:50:00.456 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-09-17T04:50:00.456 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 4401, 2024-09-17T04:50:00.456 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-09-17T04:50:00.456 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-09-17T04:50:00.456 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "b", 2024-09-17T04:50:00.456 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 19.418385840999999, 2024-09-17T04:50:00.456 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 15, 2024-09-17T04:50:00.457 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 26, 2024-09-17T04:50:00.457 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 26, 2024-09-17T04:50:00.457 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 34.325894480999999 2024-09-17T04:50:00.457 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-17T04:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69cc254640 1 -- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f69a8073230 msgr2=0x7f69a8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69cc254640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f69a8073230 0x7f69a8075740 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69c4152180 tx=0x7f69c0005d20 comp rx=0 tx=0).stop 2024-09-17T04:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69cc254640 1 -- 172.21.15.150:0/4015371115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69c4070e50 msgr2=0x7f69c41f11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69cc254640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69c4070e50 0x7f69c41f11c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f69b800b410 tx=0x7f69b800b8d0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69cc254640 1 -- 172.21.15.150:0/4015371115 shutdown_connections 2024-09-17T04:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69cc254640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f69a8073230 0x7f69a8075740 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69cc254640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69c40717a0 0x7f69c41ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69cc254640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f69c4073440 0x7f69c41f0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69cc254640 1 --2- 172.21.15.150:0/4015371115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69c4070e50 0x7f69c41f11c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69cc254640 1 -- 172.21.15.150:0/4015371115 >> 172.21.15.150:0/4015371115 conn(0x7f69c40c4a00 msgr2=0x7f69c406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69cc254640 1 -- 172.21.15.150:0/4015371115 shutdown_connections 2024-09-17T04:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.448+0000 7f69cc254640 1 -- 172.21.15.150:0/4015371115 wait complete. 2024-09-17T04:50:00.467 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-09-17T04:50:00.492 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-09-17T04:50:00.492 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-09-17T04:50:00.493 INFO:tasks.ceph_fuse:client.0 config is {'cephfs_name': 'a'} 2024-09-17T04:50:00.493 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-17T04:50:00.493 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:50:00.493 INFO:tasks.ceph_fuse:client.1 config is {'cephfs_name': 'b'} 2024-09-17T04:50:00.493 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-17T04:50:00.493 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-17T04:50:00.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:00.494 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-09-17T04:50:00.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:00.501 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link delete ceph-brx 2024-09-17T04:50:00.560 INFO:teuthology.orchestra.run.smithi176.stderr:Cannot find device "ceph-brx" 2024-09-17T04:50:00.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:50:00.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:00.562 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-09-17T04:50:00.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:00.568 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-09-17T04:50:00.627 INFO:teuthology.orchestra.run.smithi150.stderr:Cannot find device "ceph-brx" 2024-09-17T04:50:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:50:00.629 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-09-17T04:50:00.629 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-17T04:50:00.629 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:50:00.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.828+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1166129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55bc004d50 msgr2=0x7f55bc005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:00.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.828+0000 7f55dbbd4640 1 --2- 172.21.15.150:0/1166129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55bc004d50 0x7f55bc005160 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009b30 tx=0x7f55c401d270 comp rx=0 tx=0).stop 2024-09-17T04:50:00.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.828+0000 7f55da171640 1 -- 172.21.15.150:0/1166129 reap_dead start 2024-09-17T04:50:00.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.828+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1166129 shutdown_connections 2024-09-17T04:50:00.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.828+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1166129 >> 172.21.15.150:0/1166129 conn(0x7f55d40c4a00 msgr2=0x7f55d40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:00.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.828+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1166129 shutdown_connections 2024-09-17T04:50:00.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.828+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1166129 wait complete. 2024-09-17T04:50:00.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55dbbd4640 1 Processor -- start 2024-09-17T04:50:00.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55dbbd4640 1 -- start start 2024-09-17T04:50:00.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55dbbd4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d40ac810 0x7f55d40ab370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55dbbd4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d40acea0 0x7f55d40c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55dbbd4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55d40c1870 0x7f55d414d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55dbbd4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f55d40add70 con 0x7f55d40ac810 2024-09-17T04:50:00.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55dbbd4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f55d40c3000 con 0x7f55d40acea0 2024-09-17T04:50:00.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55dbbd4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f55d40705d0 con 0x7f55d40c1870 2024-09-17T04:50:00.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d9970640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d40ac810 0x7f55d40ab370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d9970640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d40ac810 0x7f55d40ab370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58366/0 (socket says 172.21.15.150:58366) 2024-09-17T04:50:00.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d9970640 1 -- 172.21.15.150:0/1503864468 learned_addr learned my addr 172.21.15.150:0/1503864468 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:00.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d916f640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d40acea0 0x7f55d40c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55da171640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55d40c1870 0x7f55d414d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d9970640 1 -- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55d40c1870 msgr2=0x7f55d414d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:00.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d9970640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55d40c1870 0x7f55d414d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d9970640 1 -- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d40acea0 msgr2=0x7f55d40c1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:00.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d9970640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d40acea0 0x7f55d40c1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:00.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d9970640 1 -- 172.21.15.150:0/1503864468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d4148980 con 0x7f55d40ac810 2024-09-17T04:50:00.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d9970640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d40ac810 0x7f55d40ab370 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f55cc001830 tx=0x7f55cc00b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:00.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d27fc640 1 -- 172.21.15.150:0/1503864468 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc003a60 con 0x7f55d40ac810 2024-09-17T04:50:00.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d27fc640 1 -- 172.21.15.150:0/1503864468 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55cc015020 con 0x7f55d40ac810 2024-09-17T04:50:00.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1503864468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d41484e0 con 0x7f55d40ac810 2024-09-17T04:50:00.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55d27fc640 1 -- 172.21.15.150:0/1503864468 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0060d0 con 0x7f55d40ac810 2024-09-17T04:50:00.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.832+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1503864468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d41519e0 con 0x7f55d40ac810 2024-09-17T04:50:00.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.836+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1503864468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d4148300 con 0x7f55d40ac810 2024-09-17T04:50:00.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.836+0000 7f55d27fc640 1 -- 172.21.15.150:0/1503864468 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f55cc006270 con 0x7f55d40ac810 2024-09-17T04:50:00.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.836+0000 7f55d27fc640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f55c0073200 0x7f55c0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:00.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.836+0000 7f55d27fc640 1 -- 172.21.15.150:0/1503864468 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f55cc0952b0 con 0x7f55d40ac810 2024-09-17T04:50:00.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.836+0000 7f55d916f640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f55c0073200 0x7f55c0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:00.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.836+0000 7f55d916f640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f55c0073200 0x7f55c0075710 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f55d40acc20 tx=0x7f55c8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:00.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:00.840+0000 7f55d27fc640 1 -- 172.21.15.150:0/1503864468 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f55d4148300 con 0x7f55d40ac810 2024-09-17T04:50:01.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.032+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1503864468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f55d41486b0 con 0x7f55d40ac810 2024-09-17T04:50:01.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55d27fc640 1 -- 172.21.15.150:0/1503864468 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+116 (secure 0 0 0) 0x7f55cc01f020 con 0x7f55d40ac810 2024-09-17T04:50:01.037 INFO:teuthology.orchestra.run.smithi150.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2024-09-17T04:50:01.037 INFO:teuthology.orchestra.run.smithi150.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2024-09-17T04:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f55c0073200 msgr2=0x7f55c0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55dbbd4640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f55c0073200 0x7f55c0075710 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f55d40acc20 tx=0x7f55c8009040 comp rx=0 tx=0).stop 2024-09-17T04:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1503864468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d40ac810 msgr2=0x7f55d40ab370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55dbbd4640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d40ac810 0x7f55d40ab370 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f55cc001830 tx=0x7f55cc00b620 comp rx=0 tx=0).stop 2024-09-17T04:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1503864468 shutdown_connections 2024-09-17T04:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55dbbd4640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f55c0073200 0x7f55c0075710 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55dbbd4640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d40acea0 0x7f55d40c1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55dbbd4640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f55d40c1870 0x7f55d414d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55dbbd4640 1 --2- 172.21.15.150:0/1503864468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55d40ac810 0x7f55d40ab370 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1503864468 >> 172.21.15.150:0/1503864468 conn(0x7f55d40c4a00 msgr2=0x7f55d41ddd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1503864468 shutdown_connections 2024-09-17T04:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01.036+0000 7f55dbbd4640 1 -- 172.21.15.150:0/1503864468 wait complete. 2024-09-17T04:50:01.058 INFO:tasks.cephfs.mount:Mounting Ceph FS a; just confirmed its presence on cluster 2024-09-17T04:50:01.058 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:50:01.058 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-09-17T04:50:01.058 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T04:50:01.058 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:50:01.058 INFO:tasks.cephfs.mount:self.cephfs_name = a 2024-09-17T04:50:01.058 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-17T04:50:01.058 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:50:01.058 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:50:01.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:01.059 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-09-17T04:50:01.065 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:50:01.065 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:50:01.065 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:50:01.065 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:50:01.065 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-09-17T04:50:01.065 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:50:01.065 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42449sec preferred_lft 42449sec 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-09-17T04:50:01.066 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:50:01.067 INFO:teuthology.orchestra.run.smithi150.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-17T04:50:01.067 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 02:42:e5:bf:2a:af brd ff:ff:ff:ff:ff:ff 2024-09-17T04:50:01.067 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-17T04:50:01.067 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:50:01.067 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-17T04:50:01.067 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:50:01.067 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-09-17T04:50:01.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add name ceph-brx type bridge 2024-09-17T04:50:01.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr flush dev ceph-brx 2024-09-17T04:50:01.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set ceph-brx up 2024-09-17T04:50:01.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-17T04:50:01.068 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-09-17T04:50:01.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:50:01.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:50:01.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:01.190 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:50:01.241 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-09-17T04:50:01.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:01.242 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-09-17T04:50:01.334 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-09-17T04:50:01.334 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:50:01.334 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-17T04:50:01.334 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:50:01.334 INFO:teuthology.orchestra.run.smithi150.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-17T04:50:01.334 INFO:teuthology.orchestra.run.smithi150.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-17T04:50:01.334 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:50:01.335 INFO:teuthology.orchestra.run.smithi150.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-17T04:50:01.335 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-17T04:50:01.335 INFO:teuthology.orchestra.run.smithi150.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-17T04:50:01.335 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:50:01.335 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:50:01.335 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-09-17T04:50:01.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:50:01.336 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:50:01.336 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:50:01.336 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-09-17T04:50:01.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:50:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:50:01.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:01.414 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-09-17T04:50:01.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:01.420 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list-id 2024-09-17T04:50:01.465 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:50:01.465 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-09-17T04:50:01.465 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:50:01.465 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-17T04:50:01.465 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-09-17T04:50:01.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:50:01.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:50:01.582 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-17T04:50:01.582 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:50:01.582 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-09-17T04:50:01.582 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-17T04:50:01.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-17T04:50:01.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-17T04:50:01.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-17T04:50:01.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-17T04:50:01.583 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-09-17T04:50:01.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:50:01.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:50:01.690 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:50:01.691 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-09-17T04:50:01.691 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-09-17T04:50:01.691 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set dev brx.0 master ceph-brx 2024-09-17T04:50:01.691 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-09-17T04:50:01.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:50:01.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:50:01.792 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'cephfs_name': 'a'} 2024-09-17T04:50:01.792 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:50:01.792 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T04:50:01.841 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:50:01.841 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:50:01.841 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:50:01.889 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-09-17T04:50:01.946 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-09-17T04:50:01.995 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-09-17T04:50:01.995 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/run/lock 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-09-17T04:50:01.996 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/fs/nfsd 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysusers.service 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/run/rpc_pipefs 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-09-17T04:50:01.997 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-09-17T04:50:01.998 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:50:01.998 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:50:01.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:01.999 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-09-17T04:50:02.045 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-17T04:50:02.046 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_fs=a) 2024-09-17T04:50:02.093 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-09-17T04:50:02.107 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-09-17T04:50:02.153 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/run/lock 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-09-17T04:50:02.154 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/fs/nfsd 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysusers.service 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/run/rpc_pipefs 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-09-17T04:50:02.155 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-09-17T04:50:02.156 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-09-17T04:50:02.156 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-09-17T04:50:02.156 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:50:02.156 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:50:02.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:02.157 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-09-17T04:50:02.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:2024-09-17T04:50:02.196+0000 7f5f02224580 -1 init, newargv = 0x7f5ef0001090 newargc=15 2024-09-17T04:50:02.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[37323]: starting ceph client 2024-09-17T04:50:02.208 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[37323]: starting fuse 2024-09-17T04:50:03.306 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-09-17T04:50:03.323 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-09-17T04:50:03.370 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-09-17T04:50:03.370 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-09-17T04:50:03.370 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/run/lock 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-09-17T04:50:03.371 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/fs/nfsd 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysusers.service 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/run/rpc_pipefs 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-09-17T04:50:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:50:03.373 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:50:03.373 INFO:teuthology.orchestra.run.smithi150.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-17T04:50:03.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:03.374 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-09-17T04:50:03.422 INFO:teuthology.orchestra.run.smithi150.stdout:45 2024-09-17T04:50:03.422 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-17T04:50:03.422 DEBUG:teuthology.orchestra.run.smithi150:> sudo stdin-killer -- python3 -c ' 2024-09-17T04:50:03.422 DEBUG:teuthology.orchestra.run.smithi150:> import glob 2024-09-17T04:50:03.422 DEBUG:teuthology.orchestra.run.smithi150:> import re 2024-09-17T04:50:03.422 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> import subprocess 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> def _find_admin_socket(client_name): 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> files = glob.glob(asok_path) 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> # Given a non-glob path, it better be there 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> if "*" not in asok_path: 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> assert(len(files) == 1) 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> return files[0] 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> for f in files: 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-17T04:50:03.423 DEBUG:teuthology.orchestra.run.smithi150:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-17T04:50:03.424 DEBUG:teuthology.orchestra.run.smithi150:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-17T04:50:03.424 DEBUG:teuthology.orchestra.run.smithi150:> contents = proc_f.read() 2024-09-17T04:50:03.424 DEBUG:teuthology.orchestra.run.smithi150:> if mountpoint in contents: 2024-09-17T04:50:03.424 DEBUG:teuthology.orchestra.run.smithi150:> return f 2024-09-17T04:50:03.424 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-17T04:50:03.424 DEBUG:teuthology.orchestra.run.smithi150:> 2024-09-17T04:50:03.424 DEBUG:teuthology.orchestra.run.smithi150:> print(_find_admin_socket("client.0")) 2024-09-17T04:50:03.424 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-09-17T04:50:03.546 INFO:teuthology.orchestra.run.smithi150.stdout:/var/run/ceph/ceph-client.0.37323.asok 2024-09-17T04:50:03.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:50:03.557 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.37323.asok 2024-09-17T04:50:03.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:03.558 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.37323.asok status 2024-09-17T04:50:03.734 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-17T04:50:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "metadata": { 2024-09-17T04:50:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_sha1": "ce1b342bf8c6349a8c90cd607f83bff2443f21cf", 2024-09-17T04:50:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.3.0-4989-gce1b342b (ce1b342bf8c6349a8c90cd607f83bff2443f21cf) squid (dev)", 2024-09-17T04:50:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "entity_id": "0", 2024-09-17T04:50:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": "smithi150", 2024-09-17T04:50:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-17T04:50:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "pid": "37323", 2024-09-17T04:50:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "root": "/" 2024-09-17T04:50:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-17T04:50:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_count": 0, 2024-09-17T04:50:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_pinned_count": 0, 2024-09-17T04:50:03.735 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 4902, 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "inst": { 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "name": { 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "client", 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "num": 4902 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 4069431801 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-09-17T04:50:03.736 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-09-17T04:50:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-09-17T04:50:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 4069431801 2024-09-17T04:50:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-09-17T04:50:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "inst_str": "client.4902 192.168.0.1:0/4069431801", 2024-09-17T04:50:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "addr_str": "192.168.0.1:0/4069431801", 2024-09-17T04:50:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "inode_count": 1, 2024-09-17T04:50:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_epoch": 9, 2024-09-17T04:50:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch": 26, 2024-09-17T04:50:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch_barrier": 0, 2024-09-17T04:50:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklisted": false, 2024-09-17T04:50:03.737 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "a" 2024-09-17T04:50:03.737 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-17T04:50:03.751 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-17T04:50:03.751 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:50:03.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.904+0000 7fc0163fc640 1 -- 172.21.15.150:0/3294038169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc010074d80 msgr2=0x7fc010157250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:03.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.904+0000 7fc0163fc640 1 --2- 172.21.15.150:0/3294038169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc010074d80 0x7fc010157250 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc0100c4bc0 tx=0x7fc00801c810 comp rx=0 tx=0).stop 2024-09-17T04:50:03.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.904+0000 7fc0163fc640 1 -- 172.21.15.150:0/3294038169 shutdown_connections 2024-09-17T04:50:03.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.904+0000 7fc0163fc640 1 --2- 172.21.15.150:0/3294038169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc010074d80 0x7fc010157250 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:03.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.904+0000 7fc0163fc640 1 --2- 172.21.15.150:0/3294038169 >> v2:172.21.15.176:3300/0 conn(0x7fc010070580 0x7fc01006dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:03.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.904+0000 7fc0163fc640 1 -- 172.21.15.150:0/3294038169 >> 172.21.15.150:0/3294038169 conn(0x7fc0100c5970 msgr2=0x7fc010073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:03.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.904+0000 7fc0163fc640 1 -- 172.21.15.150:0/3294038169 shutdown_connections 2024-09-17T04:50:03.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.904+0000 7fc0163fc640 1 -- 172.21.15.150:0/3294038169 wait complete. 2024-09-17T04:50:03.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0163fc640 1 Processor -- start 2024-09-17T04:50:03.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0163fc640 1 -- start start 2024-09-17T04:50:03.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0163fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc01006e420 0x7fc0101f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:03.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0163fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc010070580 0x7fc01014a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:03.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0163fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc010074d80 0x7fc01013a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:03.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0163fc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc010160ad0 con 0x7fc010070580 2024-09-17T04:50:03.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0163fc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc010160950 con 0x7fc010074d80 2024-09-17T04:50:03.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0163fc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc010160c50 con 0x7fc01006e420 2024-09-17T04:50:03.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0077fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc010070580 0x7fc01014a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:03.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0077fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc010070580 0x7fc01014a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58394/0 (socket says 172.21.15.150:58394) 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0077fe640 1 -- 172.21.15.150:0/4171956105 learned_addr learned my addr 172.21.15.150:0/4171956105 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc014999640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc010074d80 0x7fc01013a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc00ffff640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc01006e420 0x7fc0101f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0077fe640 1 -- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc01006e420 msgr2=0x7fc0101f16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0077fe640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc01006e420 0x7fc0101f16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0077fe640 1 -- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc010074d80 msgr2=0x7fc01013a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0077fe640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc010074d80 0x7fc01013a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0077fe640 1 -- 172.21.15.150:0/4171956105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0100709c0 con 0x7fc010070580 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc00ffff640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc01006e420 0x7fc0101f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc014999640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc010074d80 0x7fc01013a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0077fe640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc010070580 0x7fc01014a2c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbff800c3c0 tx=0x7fbff800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc00d7fa640 1 -- 172.21.15.150:0/4171956105 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8017070 con 0x7fc010070580 2024-09-17T04:50:03.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc00d7fa640 1 -- 172.21.15.150:0/4171956105 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbff80053c0 con 0x7fc010070580 2024-09-17T04:50:03.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0163fc640 1 -- 172.21.15.150:0/4171956105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0101f2030 con 0x7fc010070580 2024-09-17T04:50:03.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc00d7fa640 1 -- 172.21.15.150:0/4171956105 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8003840 con 0x7fc010070580 2024-09-17T04:50:03.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.908+0000 7fc0163fc640 1 -- 172.21.15.150:0/4171956105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01014c860 con 0x7fc010070580 2024-09-17T04:50:03.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.912+0000 7fc0163fc640 1 -- 172.21.15.150:0/4171956105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc010149230 con 0x7fc010070580 2024-09-17T04:50:03.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.912+0000 7fc00d7fa640 1 -- 172.21.15.150:0/4171956105 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbff801a070 con 0x7fc010070580 2024-09-17T04:50:03.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.912+0000 7fc00d7fa640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbff0073230 0x7fbff0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:03.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.912+0000 7fc00d7fa640 1 -- 172.21.15.150:0/4171956105 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fbff8093940 con 0x7fc010070580 2024-09-17T04:50:03.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.912+0000 7fc00ffff640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbff0073230 0x7fbff0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:03.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.912+0000 7fc00ffff640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbff0073230 0x7fbff0075740 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc000009e70 tx=0x7fc000009850 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:03.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:03.916+0000 7fc00d7fa640 1 -- 172.21.15.150:0/4171956105 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbff8021080 con 0x7fc010070580 2024-09-17T04:50:04.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.124+0000 7fc0163fc640 1 -- 172.21.15.150:0/4171956105 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc01006dee0 con 0x7fc010070580 2024-09-17T04:50:04.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.124+0000 7fc00d7fa640 1 -- 172.21.15.150:0/4171956105 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+116 (secure 0 0 0) 0x7fbff805e710 con 0x7fc010070580 2024-09-17T04:50:04.127 INFO:teuthology.orchestra.run.smithi150.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2024-09-17T04:50:04.127 INFO:teuthology.orchestra.run.smithi150.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2024-09-17T04:50:04.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.128+0000 7fc0163fc640 1 -- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbff0073230 msgr2=0x7fbff0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:04.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.128+0000 7fc0163fc640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbff0073230 0x7fbff0075740 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc000009e70 tx=0x7fc000009850 comp rx=0 tx=0).stop 2024-09-17T04:50:04.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.128+0000 7fc0163fc640 1 -- 172.21.15.150:0/4171956105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc010070580 msgr2=0x7fc01014a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:04.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.128+0000 7fc0163fc640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc010070580 0x7fc01014a2c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbff800c3c0 tx=0x7fbff800c880 comp rx=0 tx=0).stop 2024-09-17T04:50:04.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.128+0000 7fc0163fc640 1 -- 172.21.15.150:0/4171956105 shutdown_connections 2024-09-17T04:50:04.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.128+0000 7fc0163fc640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbff0073230 0x7fbff0075740 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:04.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.128+0000 7fc0163fc640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc010074d80 0x7fc01013a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:04.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.128+0000 7fc0163fc640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc010070580 0x7fc01014a2c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:04.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.128+0000 7fc0163fc640 1 --2- 172.21.15.150:0/4171956105 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc01006e420 0x7fc0101f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:04.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.128+0000 7fc0163fc640 1 -- 172.21.15.150:0/4171956105 >> 172.21.15.150:0/4171956105 conn(0x7fc0100c5970 msgr2=0x7fc010069b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:04.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.128+0000 7fc0163fc640 1 -- 172.21.15.150:0/4171956105 shutdown_connections 2024-09-17T04:50:04.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:04.128+0000 7fc0163fc640 1 -- 172.21.15.150:0/4171956105 wait complete. 2024-09-17T04:50:04.149 INFO:tasks.cephfs.mount:Mounting Ceph FS b; just confirmed its presence on cluster 2024-09-17T04:50:04.149 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:50:04.149 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi176.front.sepia.ceph.com 2024-09-17T04:50:04.149 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-17T04:50:04.149 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-17T04:50:04.149 INFO:tasks.cephfs.mount:self.cephfs_name = b 2024-09-17T04:50:04.149 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-17T04:50:04.149 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:50:04.149 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-17T04:50:04.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:04.150 DEBUG:teuthology.orchestra.run.smithi176:> ip addr 2024-09-17T04:50:04.156 INFO:teuthology.orchestra.run.smithi176.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:50:04.157 INFO:teuthology.orchestra.run.smithi176.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:50:04.157 INFO:teuthology.orchestra.run.smithi176.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:50:04.157 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:50:04.157 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 ::1/128 scope host 2024-09-17T04:50:04.157 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:50:04.157 INFO:teuthology.orchestra.run.smithi176.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-17T04:50:04.157 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:80 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:50:04.157 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f0 2024-09-17T04:50:04.157 INFO:teuthology.orchestra.run.smithi176.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-17T04:50:04.157 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:d9:36:81 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:50:04.157 INFO:teuthology.orchestra.run.smithi176.stdout: altname enp7s0f1 2024-09-17T04:50:04.157 INFO:teuthology.orchestra.run.smithi176.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-17T04:50:04.158 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:50:04.158 INFO:teuthology.orchestra.run.smithi176.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:50:04.158 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 0c:c4:7a:88:6f:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:50:04.158 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.21.15.176/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-17T04:50:04.158 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft 42431sec preferred_lft 42431sec 2024-09-17T04:50:04.158 INFO:teuthology.orchestra.run.smithi176.stdout: inet6 fe80::ec4:7aff:fe88:6f61/64 scope link 2024-09-17T04:50:04.158 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:50:04.158 INFO:teuthology.orchestra.run.smithi176.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-17T04:50:04.158 INFO:teuthology.orchestra.run.smithi176.stdout: link/ether 02:42:1c:78:cc:5b brd ff:ff:ff:ff:ff:ff 2024-09-17T04:50:04.158 INFO:teuthology.orchestra.run.smithi176.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-17T04:50:04.158 INFO:teuthology.orchestra.run.smithi176.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:50:04.158 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-17T04:50:04.159 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:50:04.159 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-09-17T04:50:04.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link add name ceph-brx type bridge 2024-09-17T04:50:04.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip addr flush dev ceph-brx 2024-09-17T04:50:04.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set ceph-brx up 2024-09-17T04:50:04.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-17T04:50:04.159 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-09-17T04:50:04.247 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:50:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:50:04.280 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:50:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:50:04.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:04.287 DEBUG:teuthology.orchestra.run.smithi176:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:50:04.298 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-09-17T04:50:04.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:04.299 DEBUG:teuthology.orchestra.run.smithi176:> route 2024-09-17T04:50:04.557 INFO:teuthology.orchestra.run.smithi176.stdout:Kernel IP routing table 2024-09-17T04:50:04.558 INFO:teuthology.orchestra.run.smithi176.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:50:04.558 INFO:teuthology.orchestra.run.smithi176.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-17T04:50:04.558 INFO:teuthology.orchestra.run.smithi176.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:50:04.558 INFO:teuthology.orchestra.run.smithi176.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-09-17T04:50:04.558 INFO:teuthology.orchestra.run.smithi176.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-17T04:50:04.558 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:50:04.558 INFO:teuthology.orchestra.run.smithi176.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-17T04:50:04.558 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-17T04:50:04.558 INFO:teuthology.orchestra.run.smithi176.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-17T04:50:04.558 INFO:teuthology.orchestra.run.smithi176.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:50:04.559 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:50:04.559 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-09-17T04:50:04.559 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:50:04.559 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:50:04.559 DEBUG:teuthology.orchestra.run.smithi176:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:50:04.559 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-09-17T04:50:04.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:50:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:50:04.636 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:50:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:50:04.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:04.644 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-09-17T04:50:04.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:04.650 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list-id 2024-09-17T04:50:04.696 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:50:04.696 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-09-17T04:50:04.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-17T04:50:04.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-17T04:50:04.696 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-09-17T04:50:04.787 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:50:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:50:04.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:50:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:50:04.812 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-17T04:50:04.812 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:50:04.812 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-09-17T04:50:04.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-17T04:50:04.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-17T04:50:04.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-17T04:50:04.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-17T04:50:04.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-17T04:50:04.813 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-09-17T04:50:04.853 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:50:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:50:04.913 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:50:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:50:04.920 DEBUG:teuthology.orchestra.run.smithi176:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:50:04.920 DEBUG:teuthology.orchestra.run.smithi176:> set -e 2024-09-17T04:50:04.920 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set brx.0 up 2024-09-17T04:50:04.921 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link set dev brx.0 master ceph-brx 2024-09-17T04:50:04.921 DEBUG:teuthology.orchestra.run.smithi176:> ') 2024-09-17T04:50:05.005 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:50:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:50:05.022 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:50:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:50:05.028 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {'cephfs_name': 'b'} 2024-09-17T04:50:05.029 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:50:05.029 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-17T04:50:05.075 INFO:teuthology.orchestra.run.smithi176.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-17T04:50:05.075 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:50:05.075 DEBUG:teuthology.orchestra.run.smithi176:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-17T04:50:05.123 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2024-09-17T04:50:05.181 DEBUG:teuthology.orchestra.run.smithi176:> cat /proc/self/mounts | awk '{print $2}' 2024-09-17T04:50:05.230 INFO:teuthology.orchestra.run.smithi176.stdout:/sys 2024-09-17T04:50:05.230 INFO:teuthology.orchestra.run.smithi176.stdout:/proc 2024-09-17T04:50:05.230 INFO:teuthology.orchestra.run.smithi176.stdout:/dev 2024-09-17T04:50:05.230 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/pts 2024-09-17T04:50:05.230 INFO:teuthology.orchestra.run.smithi176.stdout:/run 2024-09-17T04:50:05.230 INFO:teuthology.orchestra.run.smithi176.stdout:/ 2024-09-17T04:50:05.231 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/kernel/security 2024-09-17T04:50:05.231 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/shm 2024-09-17T04:50:05.231 INFO:teuthology.orchestra.run.smithi176.stdout:/run/lock 2024-09-17T04:50:05.231 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/fs/cgroup 2024-09-17T04:50:05.231 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/fs/pstore 2024-09-17T04:50:05.231 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/fs/bpf 2024-09-17T04:50:05.231 INFO:teuthology.orchestra.run.smithi176.stdout:/proc/sys/fs/binfmt_misc 2024-09-17T04:50:05.231 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/hugepages 2024-09-17T04:50:05.231 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/mqueue 2024-09-17T04:50:05.232 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/kernel/debug 2024-09-17T04:50:05.232 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/kernel/tracing 2024-09-17T04:50:05.232 INFO:teuthology.orchestra.run.smithi176.stdout:/proc/fs/nfsd 2024-09-17T04:50:05.232 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/fs/fuse/connections 2024-09-17T04:50:05.232 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/kernel/config 2024-09-17T04:50:05.232 INFO:teuthology.orchestra.run.smithi176.stdout:/run/credentials/systemd-sysusers.service 2024-09-17T04:50:05.232 INFO:teuthology.orchestra.run.smithi176.stdout:/run/rpc_pipefs 2024-09-17T04:50:05.232 INFO:teuthology.orchestra.run.smithi176.stdout:/run/user/1000 2024-09-17T04:50:05.233 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph 2024-09-17T04:50:05.233 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph/osd/ceph-4 2024-09-17T04:50:05.233 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph/osd/ceph-5 2024-09-17T04:50:05.233 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph/osd/ceph-6 2024-09-17T04:50:05.233 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph/osd/ceph-7 2024-09-17T04:50:05.233 INFO:teuthology.orchestra.run.smithi176.stdout:/run/netns 2024-09-17T04:50:05.233 INFO:teuthology.orchestra.run.smithi176.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-17T04:50:05.233 INFO:teuthology.orchestra.run.smithi176.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-17T04:50:05.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:05.234 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2024-09-17T04:50:05.283 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-17T04:50:05.284 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_fs=b) 2024-09-17T04:50:05.329 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2024-09-17T04:50:05.343 DEBUG:teuthology.orchestra.run.smithi176:> cat /proc/self/mounts | awk '{print $2}' 2024-09-17T04:50:05.391 INFO:teuthology.orchestra.run.smithi176.stdout:/sys 2024-09-17T04:50:05.392 INFO:teuthology.orchestra.run.smithi176.stdout:/proc 2024-09-17T04:50:05.392 INFO:teuthology.orchestra.run.smithi176.stdout:/dev 2024-09-17T04:50:05.392 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/pts 2024-09-17T04:50:05.392 INFO:teuthology.orchestra.run.smithi176.stdout:/run 2024-09-17T04:50:05.392 INFO:teuthology.orchestra.run.smithi176.stdout:/ 2024-09-17T04:50:05.392 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/kernel/security 2024-09-17T04:50:05.392 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/shm 2024-09-17T04:50:05.392 INFO:teuthology.orchestra.run.smithi176.stdout:/run/lock 2024-09-17T04:50:05.392 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/fs/cgroup 2024-09-17T04:50:05.392 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/fs/pstore 2024-09-17T04:50:05.392 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/fs/bpf 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/proc/sys/fs/binfmt_misc 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/hugepages 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/mqueue 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/kernel/debug 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/kernel/tracing 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/proc/fs/nfsd 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/fs/fuse/connections 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/kernel/config 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/run/credentials/systemd-sysusers.service 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/run/rpc_pipefs 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/run/user/1000 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph 2024-09-17T04:50:05.393 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph/osd/ceph-4 2024-09-17T04:50:05.394 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph/osd/ceph-5 2024-09-17T04:50:05.394 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph/osd/ceph-6 2024-09-17T04:50:05.394 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph/osd/ceph-7 2024-09-17T04:50:05.394 INFO:teuthology.orchestra.run.smithi176.stdout:/run/netns 2024-09-17T04:50:05.394 INFO:teuthology.orchestra.run.smithi176.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-17T04:50:05.394 INFO:teuthology.orchestra.run.smithi176.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-17T04:50:05.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:05.395 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2024-09-17T04:50:05.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:2024-09-17T04:50:05.434+0000 7f5092c4b580 -1 init, newargv = 0x7f50800ad020 newargc=15 2024-09-17T04:50:05.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[34616]: starting ceph client 2024-09-17T04:50:05.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[34616]: starting fuse 2024-09-17T04:50:06.543 DEBUG:teuthology.orchestra.run.smithi176:> sudo modprobe fuse 2024-09-17T04:50:06.560 DEBUG:teuthology.orchestra.run.smithi176:> cat /proc/self/mounts | awk '{print $2}' 2024-09-17T04:50:06.609 INFO:teuthology.orchestra.run.smithi176.stdout:/sys 2024-09-17T04:50:06.609 INFO:teuthology.orchestra.run.smithi176.stdout:/proc 2024-09-17T04:50:06.609 INFO:teuthology.orchestra.run.smithi176.stdout:/dev 2024-09-17T04:50:06.609 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/pts 2024-09-17T04:50:06.609 INFO:teuthology.orchestra.run.smithi176.stdout:/run 2024-09-17T04:50:06.609 INFO:teuthology.orchestra.run.smithi176.stdout:/ 2024-09-17T04:50:06.609 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/kernel/security 2024-09-17T04:50:06.609 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/shm 2024-09-17T04:50:06.609 INFO:teuthology.orchestra.run.smithi176.stdout:/run/lock 2024-09-17T04:50:06.609 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/fs/cgroup 2024-09-17T04:50:06.609 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/fs/pstore 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/fs/bpf 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/proc/sys/fs/binfmt_misc 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/hugepages 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/mqueue 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/kernel/debug 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/kernel/tracing 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/proc/fs/nfsd 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/fs/fuse/connections 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/sys/kernel/config 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/run/credentials/systemd-sysusers.service 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/run/rpc_pipefs 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/run/user/1000 2024-09-17T04:50:06.610 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph 2024-09-17T04:50:06.611 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph/osd/ceph-4 2024-09-17T04:50:06.611 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph/osd/ceph-5 2024-09-17T04:50:06.611 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph/osd/ceph-6 2024-09-17T04:50:06.611 INFO:teuthology.orchestra.run.smithi176.stdout:/var/lib/ceph/osd/ceph-7 2024-09-17T04:50:06.611 INFO:teuthology.orchestra.run.smithi176.stdout:/run/netns 2024-09-17T04:50:06.611 INFO:teuthology.orchestra.run.smithi176.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-17T04:50:06.611 INFO:teuthology.orchestra.run.smithi176.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-17T04:50:06.611 INFO:teuthology.orchestra.run.smithi176.stdout:/home/ubuntu/cephtest/mnt.1 2024-09-17T04:50:06.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:06.612 DEBUG:teuthology.orchestra.run.smithi176:> ls /sys/fs/fuse/connections 2024-09-17T04:50:06.660 INFO:teuthology.orchestra.run.smithi176.stdout:45 2024-09-17T04:50:06.660 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-09-17T04:50:06.660 DEBUG:teuthology.orchestra.run.smithi176:> sudo stdin-killer -- python3 -c ' 2024-09-17T04:50:06.660 DEBUG:teuthology.orchestra.run.smithi176:> import glob 2024-09-17T04:50:06.660 DEBUG:teuthology.orchestra.run.smithi176:> import re 2024-09-17T04:50:06.660 DEBUG:teuthology.orchestra.run.smithi176:> import os 2024-09-17T04:50:06.660 DEBUG:teuthology.orchestra.run.smithi176:> import subprocess 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> def _find_admin_socket(client_name): 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> files = glob.glob(asok_path) 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> # Given a non-glob path, it better be there 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> if "*" not in asok_path: 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> assert(len(files) == 1) 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> return files[0] 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> for f in files: 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-17T04:50:06.661 DEBUG:teuthology.orchestra.run.smithi176:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-17T04:50:06.662 DEBUG:teuthology.orchestra.run.smithi176:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-17T04:50:06.662 DEBUG:teuthology.orchestra.run.smithi176:> contents = proc_f.read() 2024-09-17T04:50:06.662 DEBUG:teuthology.orchestra.run.smithi176:> if mountpoint in contents: 2024-09-17T04:50:06.662 DEBUG:teuthology.orchestra.run.smithi176:> return f 2024-09-17T04:50:06.662 DEBUG:teuthology.orchestra.run.smithi176:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-17T04:50:06.662 DEBUG:teuthology.orchestra.run.smithi176:> 2024-09-17T04:50:06.662 DEBUG:teuthology.orchestra.run.smithi176:> print(_find_admin_socket("client.1")) 2024-09-17T04:50:06.662 DEBUG:teuthology.orchestra.run.smithi176:> ' 2024-09-17T04:50:06.783 INFO:teuthology.orchestra.run.smithi176.stdout:/var/run/ceph/ceph-client.1.34616.asok 2024-09-17T04:50:06.787 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-17T04:50:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:50:06.794 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34616.asok 2024-09-17T04:50:06.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:06.795 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34616.asok status 2024-09-17T04:50:06.980 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-09-17T04:50:06.980 INFO:teuthology.orchestra.run.smithi176.stdout: "metadata": { 2024-09-17T04:50:06.980 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_sha1": "ce1b342bf8c6349a8c90cd607f83bff2443f21cf", 2024-09-17T04:50:06.980 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 19.3.0-4989-gce1b342b (ce1b342bf8c6349a8c90cd607f83bff2443f21cf) squid (dev)", 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: "entity_id": "1", 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: "hostname": "smithi176", 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: "pid": "34616", 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: "root": "/" 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_count": 0, 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: "dentry_pinned_count": 0, 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: "id": 4786, 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: "inst": { 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: "name": { 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "client", 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: "num": 4786 2024-09-17T04:50:06.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "192.168.0.1:0", 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 965921890 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": { 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: "type": "v1", 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: "addr": "192.168.0.1:0", 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: "nonce": 965921890 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: "inst_str": "client.4786 192.168.0.1:0/965921890", 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: "addr_str": "192.168.0.1:0/965921890", 2024-09-17T04:50:06.982 INFO:teuthology.orchestra.run.smithi176.stdout: "inode_count": 1, 2024-09-17T04:50:06.983 INFO:teuthology.orchestra.run.smithi176.stdout: "mds_epoch": 15, 2024-09-17T04:50:06.983 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch": 26, 2024-09-17T04:50:06.983 INFO:teuthology.orchestra.run.smithi176.stdout: "osd_epoch_barrier": 0, 2024-09-17T04:50:06.983 INFO:teuthology.orchestra.run.smithi176.stdout: "blocklisted": false, 2024-09-17T04:50:06.983 INFO:teuthology.orchestra.run.smithi176.stdout: "fs_name": "b" 2024-09-17T04:50:06.983 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-09-17T04:50:06.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:06.996 DEBUG:teuthology.orchestra.run.smithi150:> stat --file-system '--printf=%T 2024-09-17T04:50:06.996 DEBUG:teuthology.orchestra.run.smithi150:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-17T04:50:07.003 INFO:teuthology.orchestra.run.smithi150.stdout:fuseblk 2024-09-17T04:50:07.004 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-17T04:50:07.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:07.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:50:07.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:07.060 DEBUG:teuthology.orchestra.run.smithi176:> stat --file-system '--printf=%T 2024-09-17T04:50:07.060 DEBUG:teuthology.orchestra.run.smithi176:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-17T04:50:07.066 INFO:teuthology.orchestra.run.smithi176.stdout:fuseblk 2024-09-17T04:50:07.067 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-17T04:50:07.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:50:07.067 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-17T04:50:07.125 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-09-17T04:50:07.137 INFO:tasks.mds_thrash:mds thrasher using random seed: 1726548607 2024-09-17T04:50:07.138 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:50:07.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.295+0000 7f9b6f626640 1 -- 172.21.15.150:0/2480444401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b680717a0 msgr2=0x7f9b68074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:07.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.295+0000 7f9b6f626640 1 --2- 172.21.15.150:0/2480444401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b680717a0 0x7f9b68074450 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9b60009900 tx=0x7f9b6001c850 comp rx=0 tx=0).stop 2024-09-17T04:50:07.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.295+0000 7f9b6f626640 1 -- 172.21.15.150:0/2480444401 shutdown_connections 2024-09-17T04:50:07.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.295+0000 7f9b6f626640 1 --2- 172.21.15.150:0/2480444401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b680717a0 0x7f9b68074450 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.295+0000 7f9b6f626640 1 --2- 172.21.15.150:0/2480444401 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b68070e50 0x7f9b68071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.295+0000 7f9b6f626640 1 -- 172.21.15.150:0/2480444401 >> 172.21.15.150:0/2480444401 conn(0x7f9b680c4a00 msgr2=0x7f9b680c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:07.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.295+0000 7f9b6f626640 1 -- 172.21.15.150:0/2480444401 shutdown_connections 2024-09-17T04:50:07.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.295+0000 7f9b6f626640 1 -- 172.21.15.150:0/2480444401 wait complete. 2024-09-17T04:50:07.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6f626640 1 Processor -- start 2024-09-17T04:50:07.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6f626640 1 -- start start 2024-09-17T04:50:07.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6f626640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b68070e50 0x7f9b68156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:07.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6f626640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b680717a0 0x7f9b681573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6f626640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b68073440 0x7f9b6814a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6f626640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9b680ad210 con 0x7f9b68070e50 2024-09-17T04:50:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6f626640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9b680ad090 con 0x7f9b680717a0 2024-09-17T04:50:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6f626640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9b680ad390 con 0x7f9b68073440 2024-09-17T04:50:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6d3c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b68070e50 0x7f9b68156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6d3c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b68070e50 0x7f9b68156250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58404/0 (socket says 172.21.15.150:58404) 2024-09-17T04:50:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6d3c2640 1 -- 172.21.15.150:0/3157641423 learned_addr learned my addr 172.21.15.150:0/3157641423 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6cbc1640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b680717a0 0x7f9b681573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6dbc3640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b68073440 0x7f9b6814a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6d3c2640 1 -- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b68073440 msgr2=0x7f9b6814a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6d3c2640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b68073440 0x7f9b6814a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6d3c2640 1 -- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b680717a0 msgr2=0x7f9b681573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:07.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6d3c2640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b680717a0 0x7f9b681573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6d3c2640 1 -- 172.21.15.150:0/3157641423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b68157910 con 0x7f9b68070e50 2024-09-17T04:50:07.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6d3c2640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b68070e50 0x7f9b68156250 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800c740 tx=0x7f9b5800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:07.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b65ffb640 1 -- 172.21.15.150:0/3157641423 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b58003b80 con 0x7f9b68070e50 2024-09-17T04:50:07.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b65ffb640 1 -- 172.21.15.150:0/3157641423 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b5800ce50 con 0x7f9b68070e50 2024-09-17T04:50:07.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6f626640 1 -- 172.21.15.150:0/3157641423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b68156ef0 con 0x7f9b68070e50 2024-09-17T04:50:07.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b65ffb640 1 -- 172.21.15.150:0/3157641423 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5800a850 con 0x7f9b68070e50 2024-09-17T04:50:07.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6f626640 1 -- 172.21.15.150:0/3157641423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b68149c80 con 0x7f9b68070e50 2024-09-17T04:50:07.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.299+0000 7f9b6f626640 1 -- 172.21.15.150:0/3157641423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b680ad860 con 0x7f9b68070e50 2024-09-17T04:50:07.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.303+0000 7f9b65ffb640 1 -- 172.21.15.150:0/3157641423 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9b5800a9f0 con 0x7f9b68070e50 2024-09-17T04:50:07.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.303+0000 7f9b65ffb640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b54073200 0x7f9b54075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:07.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.303+0000 7f9b65ffb640 1 -- 172.21.15.150:0/3157641423 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f9b58094350 con 0x7f9b68070e50 2024-09-17T04:50:07.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.303+0000 7f9b6cbc1640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b54073200 0x7f9b54075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:07.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.303+0000 7f9b6cbc1640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b54073200 0x7f9b54075710 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c005fd0 tx=0x7f9b5c005d10 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:07.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.307+0000 7f9b65ffb640 1 -- 172.21.15.150:0/3157641423 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9b5805f0e0 con 0x7f9b68070e50 2024-09-17T04:50:07.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.479+0000 7f9b6f626640 1 -- 172.21.15.150:0/3157641423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b680c2380 con 0x7f9b68070e50 2024-09-17T04:50:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b65ffb640 1 -- 172.21.15.150:0/3157641423 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20431 (secure 0 0 0) 0x7f9b5805e9e0 con 0x7f9b68070e50 2024-09-17T04:50:07.486 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:50:07.486 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:49:42.029799+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:50:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b6f626640 1 -- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b54073200 msgr2=0x7f9b54075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b6f626640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b54073200 0x7f9b54075710 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c005fd0 tx=0x7f9b5c005d10 comp rx=0 tx=0).stop 2024-09-17T04:50:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b6f626640 1 -- 172.21.15.150:0/3157641423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b68070e50 msgr2=0x7f9b68156250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b6f626640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b68070e50 0x7f9b68156250 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800c740 tx=0x7f9b5800cc00 comp rx=0 tx=0).stop 2024-09-17T04:50:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b6f626640 1 -- 172.21.15.150:0/3157641423 shutdown_connections 2024-09-17T04:50:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b6f626640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b54073200 0x7f9b54075710 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b6f626640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b680717a0 0x7f9b681573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b6f626640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b68073440 0x7f9b6814a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b6f626640 1 --2- 172.21.15.150:0/3157641423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b68070e50 0x7f9b68156250 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b6f626640 1 -- 172.21.15.150:0/3157641423 >> 172.21.15.150:0/3157641423 conn(0x7f9b680c4a00 msgr2=0x7f9b680c38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b6f626640 1 -- 172.21.15.150:0/3157641423 shutdown_connections 2024-09-17T04:50:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.483+0000 7f9b6f626640 1 -- 172.21.15.150:0/3157641423 wait complete. 2024-09-17T04:50:07.508 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T04:49:20.992684+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-09-17T04:49:29.488488+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-09-17T04:49:30.232367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-09-17T04:49:38.219844+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-09-17T04:49:39.346549+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T04:50:07.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T04:50:07.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.651+0000 7f95acadb640 1 -- 172.21.15.150:0/952392288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9594004810 msgr2=0x7f9594004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:07.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.651+0000 7f95acadb640 1 --2- 172.21.15.150:0/952392288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9594004810 0x7f9594004c20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f959000c260 tx=0x7f959001cf00 comp rx=0 tx=0).stop 2024-09-17T04:50:07.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.651+0000 7f95acadb640 1 -- 172.21.15.150:0/952392288 shutdown_connections 2024-09-17T04:50:07.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.651+0000 7f95acadb640 1 --2- 172.21.15.150:0/952392288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9594004810 0x7f9594004c20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.651+0000 7f95acadb640 1 -- 172.21.15.150:0/952392288 >> 172.21.15.150:0/952392288 conn(0x7f95a80c4a00 msgr2=0x7f95a80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:07.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.651+0000 7f95acadb640 1 -- 172.21.15.150:0/952392288 shutdown_connections 2024-09-17T04:50:07.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.651+0000 7f95acadb640 1 -- 172.21.15.150:0/952392288 wait complete. 2024-09-17T04:50:07.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95acadb640 1 Processor -- start 2024-09-17T04:50:07.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95acadb640 1 -- start start 2024-09-17T04:50:07.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95acadb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9594004810 0x7f95a80a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:07.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95acadb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95a8150d10 0x7f95a81eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:07.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95acadb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95a8155b50 0x7f95a8155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:07.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95acadb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f95a80749e0 con 0x7f95a8155b50 2024-09-17T04:50:07.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95acadb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f95a8074860 con 0x7f95a8150d10 2024-09-17T04:50:07.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95acadb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f95a8074b60 con 0x7f9594004810 2024-09-17T04:50:07.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95a6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95a8155b50 0x7f95a8155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:07.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95a6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95a8155b50 0x7f95a8155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58414/0 (socket says 172.21.15.150:58414) 2024-09-17T04:50:07.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95a6d9d640 1 -- 172.21.15.150:0/636138445 learned_addr learned my addr 172.21.15.150:0/636138445 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:07.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95a5d9b640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95a8150d10 0x7f95a81eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:07.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95a659c640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9594004810 0x7f95a80a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:07.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95a6d9d640 1 -- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9594004810 msgr2=0x7f95a80a7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:07.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95a6d9d640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9594004810 0x7f95a80a7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95a6d9d640 1 -- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95a8150d10 msgr2=0x7f95a81eeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:07.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95a6d9d640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95a8150d10 0x7f95a81eeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95a6d9d640 1 -- 172.21.15.150:0/636138445 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a81ec3a0 con 0x7f95a8155b50 2024-09-17T04:50:07.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95a6d9d640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95a8155b50 0x7f95a8155f60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f95a8073240 tx=0x7f9598010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:07.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f959effd640 1 -- 172.21.15.150:0/636138445 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959800ea60 con 0x7f95a8155b50 2024-09-17T04:50:07.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f959effd640 1 -- 172.21.15.150:0/636138445 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f959800ed10 con 0x7f95a8155b50 2024-09-17T04:50:07.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f959effd640 1 -- 172.21.15.150:0/636138445 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9598004280 con 0x7f95a8155b50 2024-09-17T04:50:07.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95acadb640 1 -- 172.21.15.150:0/636138445 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a80c6e10 con 0x7f95a8155b50 2024-09-17T04:50:07.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.655+0000 7f95acadb640 1 -- 172.21.15.150:0/636138445 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a81ec640 con 0x7f95a8155b50 2024-09-17T04:50:07.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.659+0000 7f95acadb640 1 -- 172.21.15.150:0/636138445 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a8075070 con 0x7f95a8155b50 2024-09-17T04:50:07.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.659+0000 7f959effd640 1 -- 172.21.15.150:0/636138445 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f959801d020 con 0x7f95a8155b50 2024-09-17T04:50:07.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.659+0000 7f959effd640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9588073200 0x7f9588075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:07.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.659+0000 7f959effd640 1 -- 172.21.15.150:0/636138445 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f95980943c0 con 0x7f95a8155b50 2024-09-17T04:50:07.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.659+0000 7f95a659c640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9588073200 0x7f9588075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:07.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.659+0000 7f95a659c640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9588073200 0x7f9588075710 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f95a8052280 tx=0x7f95a0006dd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:07.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.663+0000 7f959effd640 1 -- 172.21.15.150:0/636138445 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9598024020 con 0x7f95a8155b50 2024-09-17T04:50:07.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.819+0000 7f95acadb640 1 -- 172.21.15.150:0/636138445 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f95a80c60f0 con 0x7f95a8155b50 2024-09-17T04:50:07.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.819+0000 7f959effd640 1 -- 172.21.15.150:0/636138445 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f959805f0a0 con 0x7f95a8155b50 2024-09-17T04:50:07.823 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-09-17T04:50:07.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.823+0000 7f95acadb640 1 -- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9588073200 msgr2=0x7f9588075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:07.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.823+0000 7f95acadb640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9588073200 0x7f9588075710 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f95a8052280 tx=0x7f95a0006dd0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.823+0000 7f95acadb640 1 -- 172.21.15.150:0/636138445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95a8155b50 msgr2=0x7f95a8155f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:07.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.823+0000 7f95acadb640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95a8155b50 0x7f95a8155f60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f95a8073240 tx=0x7f9598010da0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.823+0000 7f95acadb640 1 -- 172.21.15.150:0/636138445 shutdown_connections 2024-09-17T04:50:07.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.823+0000 7f95acadb640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9588073200 0x7f9588075710 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.823+0000 7f95acadb640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95a8150d10 0x7f95a81eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.823+0000 7f95acadb640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95a8155b50 0x7f95a8155f60 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.823+0000 7f95acadb640 1 --2- 172.21.15.150:0/636138445 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9594004810 0x7f95a80a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:07.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.823+0000 7f95acadb640 1 -- 172.21.15.150:0/636138445 >> 172.21.15.150:0/636138445 conn(0x7f95a80c4a00 msgr2=0x7f95a80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:07.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.823+0000 7f95acadb640 1 -- 172.21.15.150:0/636138445 shutdown_connections 2024-09-17T04:50:07.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.823+0000 7f95acadb640 1 -- 172.21.15.150:0/636138445 wait complete. 2024-09-17T04:50:07.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-09-17T04:50:08.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.999+0000 7f4279cf4640 1 -- 172.21.15.150:0/3888147065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4260005f80 msgr2=0x7f42600063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.999+0000 7f4279cf4640 1 --2- 172.21.15.150:0/3888147065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4260005f80 0x7f42600063d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f42740ae070 tx=0x7f426c01cad0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.999+0000 7f4273fff640 1 -- 172.21.15.150:0/3888147065 reap_dead start 2024-09-17T04:50:08.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.999+0000 7f4279cf4640 1 -- 172.21.15.150:0/3888147065 shutdown_connections 2024-09-17T04:50:08.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.999+0000 7f4279cf4640 1 -- 172.21.15.150:0/3888147065 >> 172.21.15.150:0/3888147065 conn(0x7f42740c4a00 msgr2=0x7f42740c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:08.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.999+0000 7f4279cf4640 1 -- 172.21.15.150:0/3888147065 shutdown_connections 2024-09-17T04:50:08.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.999+0000 7f4279cf4640 1 -- 172.21.15.150:0/3888147065 wait complete. 2024-09-17T04:50:08.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.999+0000 7f4279cf4640 1 Processor -- start 2024-09-17T04:50:08.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:07.999+0000 7f4279cf4640 1 -- start start 2024-09-17T04:50:08.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4279cf4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4260005f80 0x7f42740ad360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:08.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4279cf4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f427414b1e0 0x7f42740ac080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:08.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4279cf4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4274073440 0x7f4274073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:08.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4279cf4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4274070ac0 con 0x7f4260005f80 2024-09-17T04:50:08.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4279cf4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f42740c22c0 con 0x7f427414b1e0 2024-09-17T04:50:08.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4279cf4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4274070da0 con 0x7f4274073440 2024-09-17T04:50:08.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f42737fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4260005f80 0x7f42740ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:08.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f42737fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4260005f80 0x7f42740ad360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58432/0 (socket says 172.21.15.150:58432) 2024-09-17T04:50:08.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f42737fe640 1 -- 172.21.15.150:0/1029373216 learned_addr learned my addr 172.21.15.150:0/1029373216 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f426affd640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f427414b1e0 0x7f42740ac080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4273fff640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4274073440 0x7f4274073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f42737fe640 1 -- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4274073440 msgr2=0x7f4274073850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f42737fe640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4274073440 0x7f4274073850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f42737fe640 1 -- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f427414b1e0 msgr2=0x7f42740ac080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f42737fe640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f427414b1e0 0x7f42740ac080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f42737fe640 1 -- 172.21.15.150:0/1029373216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42740adbb0 con 0x7f4260005f80 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f42737fe640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4260005f80 0x7f42740ad360 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f426400e990 tx=0x7f426400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4270ff9640 1 -- 172.21.15.150:0/1029373216 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426400cb80 con 0x7f4260005f80 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4270ff9640 1 -- 172.21.15.150:0/1029373216 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f426400ce10 con 0x7f4260005f80 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4279cf4640 1 -- 172.21.15.150:0/1029373216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427414b9c0 con 0x7f4260005f80 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4270ff9640 1 -- 172.21.15.150:0/1029373216 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4264005260 con 0x7f4260005f80 2024-09-17T04:50:08.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4279cf4640 1 -- 172.21.15.150:0/1029373216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4274156ba0 con 0x7f4260005f80 2024-09-17T04:50:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4279cf4640 1 -- 172.21.15.150:0/1029373216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4274156de0 con 0x7f4260005f80 2024-09-17T04:50:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4270ff9640 1 -- 172.21.15.150:0/1029373216 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4264005400 con 0x7f4260005f80 2024-09-17T04:50:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4270ff9640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4254073200 0x7f4254075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f4270ff9640 1 -- 172.21.15.150:0/1029373216 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f4264094df0 con 0x7f4260005f80 2024-09-17T04:50:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.003+0000 7f426affd640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4254073200 0x7f4254075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.007+0000 7f426affd640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4254073200 0x7f4254075710 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f425c001870 tx=0x7f425c001d20 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.011+0000 7f4270ff9640 1 -- 172.21.15.150:0/1029373216 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4274156de0 con 0x7f4260005f80 2024-09-17T04:50:08.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.171+0000 7f4279cf4640 1 -- 172.21.15.150:0/1029373216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f4274150060 con 0x7f4260005f80 2024-09-17T04:50:08.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4270ff9640 1 -- 172.21.15.150:0/1029373216 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v26) ==== 99+0+11 (secure 0 0 0) 0x7f42740adbb0 con 0x7f4260005f80 2024-09-17T04:50:08.177 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-09-17T04:50:08.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4279cf4640 1 -- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4254073200 msgr2=0x7f4254075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4279cf4640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4254073200 0x7f4254075710 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f425c001870 tx=0x7f425c001d20 comp rx=0 tx=0).stop 2024-09-17T04:50:08.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4279cf4640 1 -- 172.21.15.150:0/1029373216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4260005f80 msgr2=0x7f42740ad360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4279cf4640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4260005f80 0x7f42740ad360 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f426400e990 tx=0x7f426400ee50 comp rx=0 tx=0).stop 2024-09-17T04:50:08.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4279cf4640 1 -- 172.21.15.150:0/1029373216 shutdown_connections 2024-09-17T04:50:08.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4279cf4640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4254073200 0x7f4254075710 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4279cf4640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f427414b1e0 0x7f42740ac080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4279cf4640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4274073440 0x7f4274073850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4279cf4640 1 --2- 172.21.15.150:0/1029373216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4260005f80 0x7f42740ad360 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4279cf4640 1 -- 172.21.15.150:0/1029373216 >> 172.21.15.150:0/1029373216 conn(0x7f42740c4a00 msgr2=0x7f42740c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:08.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4279cf4640 1 -- 172.21.15.150:0/1029373216 shutdown_connections 2024-09-17T04:50:08.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.175+0000 7f4279cf4640 1 -- 172.21.15.150:0/1029373216 wait complete. 2024-09-17T04:50:08.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-09-17T04:50:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.355+0000 7f23b9b42640 1 -- 172.21.15.150:0/585207676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23b41568a0 msgr2=0x7f23b40c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.355+0000 7f23b9b42640 1 --2- 172.21.15.150:0/585207676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23b41568a0 0x7f23b40c5810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009900 tx=0x7f23a401c8b0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.355+0000 7f23b9b42640 1 -- 172.21.15.150:0/585207676 shutdown_connections 2024-09-17T04:50:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.355+0000 7f23b9b42640 1 --2- 172.21.15.150:0/585207676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23b40c20a0 0x7f23b40c24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.355+0000 7f23b9b42640 1 --2- 172.21.15.150:0/585207676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23b41568a0 0x7f23b40c5810 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.355+0000 7f23b9b42640 1 -- 172.21.15.150:0/585207676 >> 172.21.15.150:0/585207676 conn(0x7f23b4074b80 msgr2=0x7f23b406f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.355+0000 7f23b9b42640 1 -- 172.21.15.150:0/585207676 shutdown_connections 2024-09-17T04:50:08.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.355+0000 7f23b9b42640 1 -- 172.21.15.150:0/585207676 wait complete. 2024-09-17T04:50:08.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b9b42640 1 Processor -- start 2024-09-17T04:50:08.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b9b42640 1 -- start start 2024-09-17T04:50:08.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b9b42640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23b40c20a0 0x7f23b4147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:08.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b9b42640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23b40c35b0 0x7f23b40cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b9b42640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23b41568a0 0x7f23b41f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b9b42640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f23b40c3b40 con 0x7f23b40c35b0 2024-09-17T04:50:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b9b42640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f23b40c39c0 con 0x7f23b40c20a0 2024-09-17T04:50:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b9b42640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f23b40c69f0 con 0x7f23b41568a0 2024-09-17T04:50:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23b40c35b0 0x7f23b40cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23b40c35b0 0x7f23b40cb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58438/0 (socket says 172.21.15.150:58438) 2024-09-17T04:50:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b2ffd640 1 -- 172.21.15.150:0/88808505 learned_addr learned my addr 172.21.15.150:0/88808505 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b3fff640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23b41568a0 0x7f23b41f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b37fe640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23b40c20a0 0x7f23b4147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b2ffd640 1 -- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23b41568a0 msgr2=0x7f23b41f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b2ffd640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23b41568a0 0x7f23b41f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b2ffd640 1 -- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23b40c20a0 msgr2=0x7f23b4147870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b2ffd640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23b40c20a0 0x7f23b4147870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b2ffd640 1 -- 172.21.15.150:0/88808505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b41511b0 con 0x7f23b40c35b0 2024-09-17T04:50:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b37fe640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23b40c20a0 0x7f23b4147870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:50:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b2ffd640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23b40c35b0 0x7f23b40cb730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f23a800c3c0 tx=0x7f23a800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f239bfff640 1 -- 172.21.15.150:0/88808505 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a80052d0 con 0x7f23b40c35b0 2024-09-17T04:50:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f239bfff640 1 -- 172.21.15.150:0/88808505 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f23a8007020 con 0x7f23b40c35b0 2024-09-17T04:50:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b9b42640 1 -- 172.21.15.150:0/88808505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23b414a8a0 con 0x7f23b40c35b0 2024-09-17T04:50:08.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f239bfff640 1 -- 172.21.15.150:0/88808505 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a8007470 con 0x7f23b40c35b0 2024-09-17T04:50:08.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b9b42640 1 -- 172.21.15.150:0/88808505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23b41f3d10 con 0x7f23b40c35b0 2024-09-17T04:50:08.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.359+0000 7f23b9b42640 1 -- 172.21.15.150:0/88808505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23b4150f20 con 0x7f23b40c35b0 2024-09-17T04:50:08.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.363+0000 7f239bfff640 1 -- 172.21.15.150:0/88808505 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f23a80067e0 con 0x7f23b40c35b0 2024-09-17T04:50:08.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.363+0000 7f239bfff640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f23a0073200 0x7f23a0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:08.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.363+0000 7f239bfff640 1 -- 172.21.15.150:0/88808505 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f23a8093970 con 0x7f23b40c35b0 2024-09-17T04:50:08.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.363+0000 7f23b37fe640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f23a0073200 0x7f23a0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:08.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.363+0000 7f23b37fe640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f23a0073200 0x7f23a0075710 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f23a40af000 tx=0x7f23a401b0f0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:08.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.367+0000 7f239bfff640 1 -- 172.21.15.150:0/88808505 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f23a805e670 con 0x7f23b40c35b0 2024-09-17T04:50:08.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.527+0000 7f23b9b42640 1 -- 172.21.15.150:0/88808505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7f23b40c24b0 con 0x7f23b40c35b0 2024-09-17T04:50:08.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f239bfff640 1 -- 172.21.15.150:0/88808505 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v26) ==== 95+0+11 (secure 0 0 0) 0x7f23a805df70 con 0x7f23b40c35b0 2024-09-17T04:50:08.533 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-09-17T04:50:08.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f23b9b42640 1 -- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f23a0073200 msgr2=0x7f23a0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f23b9b42640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f23a0073200 0x7f23a0075710 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f23a40af000 tx=0x7f23a401b0f0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f23b9b42640 1 -- 172.21.15.150:0/88808505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23b40c35b0 msgr2=0x7f23b40cb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f23b9b42640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23b40c35b0 0x7f23b40cb730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f23a800c3c0 tx=0x7f23a800c880 comp rx=0 tx=0).stop 2024-09-17T04:50:08.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f23b9b42640 1 -- 172.21.15.150:0/88808505 shutdown_connections 2024-09-17T04:50:08.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f23b9b42640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f23a0073200 0x7f23a0075710 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f23b9b42640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23b41568a0 0x7f23b41f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f23b9b42640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23b40c35b0 0x7f23b40cb730 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f23b9b42640 1 --2- 172.21.15.150:0/88808505 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23b40c20a0 0x7f23b4147870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f23b9b42640 1 -- 172.21.15.150:0/88808505 >> 172.21.15.150:0/88808505 conn(0x7f23b4074b80 msgr2=0x7f23b415a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:08.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f23b9b42640 1 -- 172.21.15.150:0/88808505 shutdown_connections 2024-09-17T04:50:08.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.531+0000 7f23b9b42640 1 -- 172.21.15.150:0/88808505 wait complete. 2024-09-17T04:50:08.554 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-09-17T04:50:08.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 -- 172.21.15.150:0/903438803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb581535b0 msgr2=0x7ffb58157440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 --2- 172.21.15.150:0/903438803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb581535b0 0x7ffb58157440 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ffb580c6860 tx=0x7ffb5401c770 comp rx=0 tx=0).stop 2024-09-17T04:50:08.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 -- 172.21.15.150:0/903438803 shutdown_connections 2024-09-17T04:50:08.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 --2- 172.21.15.150:0/903438803 >> v2:172.21.15.176:3300/0 conn(0x7ffb580c5760 0x7ffb580c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 --2- 172.21.15.150:0/903438803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb581535b0 0x7ffb58157440 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 -- 172.21.15.150:0/903438803 >> 172.21.15.150:0/903438803 conn(0x7ffb580aaa80 msgr2=0x7ffb580aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:08.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 -- 172.21.15.150:0/903438803 shutdown_connections 2024-09-17T04:50:08.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 -- 172.21.15.150:0/903438803 wait complete. 2024-09-17T04:50:08.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 Processor -- start 2024-09-17T04:50:08.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 -- start start 2024-09-17T04:50:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb580a7f10 0x7ffb5814efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb580c5760 0x7ffb58150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb581535b0 0x7ffb5814f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffb580718d0 con 0x7ffb580a7f10 2024-09-17T04:50:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffb580a8730 con 0x7ffb581535b0 2024-09-17T04:50:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.707+0000 7ffb60146640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffb580c5370 con 0x7ffb580c5760 2024-09-17T04:50:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5dee2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb580a7f10 0x7ffb5814efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5dee2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb580a7f10 0x7ffb5814efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58462/0 (socket says 172.21.15.150:58462) 2024-09-17T04:50:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5dee2640 1 -- 172.21.15.150:0/147542510 learned_addr learned my addr 172.21.15.150:0/147542510 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5e6e3640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb581535b0 0x7ffb5814f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5d6e1640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb580c5760 0x7ffb58150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5dee2640 1 -- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb580c5760 msgr2=0x7ffb58150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5dee2640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb580c5760 0x7ffb58150910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5dee2640 1 -- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb581535b0 msgr2=0x7ffb5814f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5dee2640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb581535b0 0x7ffb5814f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5dee2640 1 -- 172.21.15.150:0/147542510 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb58150e50 con 0x7ffb580a7f10 2024-09-17T04:50:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5d6e1640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb580c5760 0x7ffb58150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:50:08.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5dee2640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb580a7f10 0x7ffb5814efc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ffb54098260 tx=0x7ffb540a5540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:08.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb4e7fc640 1 -- 172.21.15.150:0/147542510 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb540b2040 con 0x7ffb580a7f10 2024-09-17T04:50:08.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb4e7fc640 1 -- 172.21.15.150:0/147542510 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb540bb070 con 0x7ffb580a7f10 2024-09-17T04:50:08.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb60146640 1 -- 172.21.15.150:0/147542510 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb58149020 con 0x7ffb580a7f10 2024-09-17T04:50:08.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb4e7fc640 1 -- 172.21.15.150:0/147542510 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54003600 con 0x7ffb580a7f10 2024-09-17T04:50:08.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb60146640 1 -- 172.21.15.150:0/147542510 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb5814aaf0 con 0x7ffb580a7f10 2024-09-17T04:50:08.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb60146640 1 -- 172.21.15.150:0/147542510 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb5814ad50 con 0x7ffb580a7f10 2024-09-17T04:50:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb4e7fc640 1 -- 172.21.15.150:0/147542510 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ffb540037a0 con 0x7ffb580a7f10 2024-09-17T04:50:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb4e7fc640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffb3c073200 0x7ffb3c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb4e7fc640 1 -- 172.21.15.150:0/147542510 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7ffb5412ea70 con 0x7ffb580a7f10 2024-09-17T04:50:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.711+0000 7ffb5d6e1640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffb3c073200 0x7ffb3c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.715+0000 7ffb5d6e1640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffb3c073200 0x7ffb3c075710 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffb5815df30 tx=0x7ffb48005d20 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.715+0000 7ffb4e7fc640 1 -- 172.21.15.150:0/147542510 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ffb540163f0 con 0x7ffb580a7f10 2024-09-17T04:50:08.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.871+0000 7ffb60146640 1 -- 172.21.15.150:0/147542510 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7ffb581539c0 con 0x7ffb580a7f10 2024-09-17T04:50:08.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.875+0000 7ffb4e7fc640 1 -- 172.21.15.150:0/147542510 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v26) ==== 99+0+11 (secure 0 0 0) 0x7ffb540f9880 con 0x7ffb580a7f10 2024-09-17T04:50:08.877 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-09-17T04:50:08.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.879+0000 7ffb60146640 1 -- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffb3c073200 msgr2=0x7ffb3c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.879+0000 7ffb60146640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffb3c073200 0x7ffb3c075710 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffb5815df30 tx=0x7ffb48005d20 comp rx=0 tx=0).stop 2024-09-17T04:50:08.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.879+0000 7ffb60146640 1 -- 172.21.15.150:0/147542510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb580a7f10 msgr2=0x7ffb5814efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:08.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.879+0000 7ffb60146640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb580a7f10 0x7ffb5814efc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ffb54098260 tx=0x7ffb540a5540 comp rx=0 tx=0).stop 2024-09-17T04:50:08.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.879+0000 7ffb60146640 1 -- 172.21.15.150:0/147542510 shutdown_connections 2024-09-17T04:50:08.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.879+0000 7ffb60146640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffb3c073200 0x7ffb3c075710 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.879+0000 7ffb60146640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb581535b0 0x7ffb5814f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.879+0000 7ffb60146640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb580c5760 0x7ffb58150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.879+0000 7ffb60146640 1 --2- 172.21.15.150:0/147542510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb580a7f10 0x7ffb5814efc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:08.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.879+0000 7ffb60146640 1 -- 172.21.15.150:0/147542510 >> 172.21.15.150:0/147542510 conn(0x7ffb580aaa80 msgr2=0x7ffb5815e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:08.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.879+0000 7ffb60146640 1 -- 172.21.15.150:0/147542510 shutdown_connections 2024-09-17T04:50:08.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:08.879+0000 7ffb60146640 1 -- 172.21.15.150:0/147542510 wait complete. 2024-09-17T04:50:08.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-09-17T04:50:09.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.043+0000 7f57a7859640 1 -- 172.21.15.150:0/3216762136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57a00c8fc0 msgr2=0x7f57a00c20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.043+0000 7f57a7859640 1 --2- 172.21.15.150:0/3216762136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57a00c8fc0 0x7f57a00c20a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5798009c80 tx=0x7f579801cd70 comp rx=0 tx=0).stop 2024-09-17T04:50:09.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.043+0000 7f57a7859640 1 -- 172.21.15.150:0/3216762136 shutdown_connections 2024-09-17T04:50:09.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.043+0000 7f57a7859640 1 --2- 172.21.15.150:0/3216762136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57a00c8fc0 0x7f57a00c20a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.043+0000 7f57a7859640 1 --2- 172.21.15.150:0/3216762136 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57a00c35b0 0x7f57a00c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.043+0000 7f57a7859640 1 -- 172.21.15.150:0/3216762136 >> 172.21.15.150:0/3216762136 conn(0x7f57a0074b80 msgr2=0x7f57a006f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:09.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.043+0000 7f57a7859640 1 -- 172.21.15.150:0/3216762136 shutdown_connections 2024-09-17T04:50:09.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.043+0000 7f57a7859640 1 -- 172.21.15.150:0/3216762136 wait complete. 2024-09-17T04:50:09.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.043+0000 7f57a7859640 1 Processor -- start 2024-09-17T04:50:09.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.043+0000 7f57a7859640 1 -- start start 2024-09-17T04:50:09.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a7859640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57a00c35b0 0x7f57a01527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:09.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a7859640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57a00c8fc0 0x7f57a01f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:09.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a7859640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57a01568a0 0x7f57a0147970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:09.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a7859640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f57a00c6b70 con 0x7f57a00c8fc0 2024-09-17T04:50:09.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a7859640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f57a00c69f0 con 0x7f57a01568a0 2024-09-17T04:50:09.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a7859640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f57a00b0fd0 con 0x7f57a00c35b0 2024-09-17T04:50:09.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a4df4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57a00c8fc0 0x7f57a01f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:09.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a4df4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57a00c8fc0 0x7f57a01f32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58472/0 (socket says 172.21.15.150:58472) 2024-09-17T04:50:09.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a4df4640 1 -- 172.21.15.150:0/2170004026 learned_addr learned my addr 172.21.15.150:0/2170004026 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:09.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a55f5640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57a00c35b0 0x7f57a01527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:09.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a5df6640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57a01568a0 0x7f57a0147970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:09.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a4df4640 1 -- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57a00c35b0 msgr2=0x7f57a01527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a4df4640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57a00c35b0 0x7f57a01527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a4df4640 1 -- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57a01568a0 msgr2=0x7f57a0147970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a4df4640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57a01568a0 0x7f57a0147970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a4df4640 1 -- 172.21.15.150:0/2170004026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a0149d80 con 0x7f57a00c8fc0 2024-09-17T04:50:09.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a5df6640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57a01568a0 0x7f57a0147970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:50:09.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a4df4640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57a00c8fc0 0x7f57a01f32e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f579000b440 tx=0x7f579000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:09.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f5795ffb640 1 -- 172.21.15.150:0/2170004026 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5790015020 con 0x7f57a00c8fc0 2024-09-17T04:50:09.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f5795ffb640 1 -- 172.21.15.150:0/2170004026 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57900119a0 con 0x7f57a00c8fc0 2024-09-17T04:50:09.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a7859640 1 -- 172.21.15.150:0/2170004026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57a0153530 con 0x7f57a00c8fc0 2024-09-17T04:50:09.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f5795ffb640 1 -- 172.21.15.150:0/2170004026 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5790011df0 con 0x7f57a00c8fc0 2024-09-17T04:50:09.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a7859640 1 -- 172.21.15.150:0/2170004026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57a0148600 con 0x7f57a00c8fc0 2024-09-17T04:50:09.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a7859640 1 -- 172.21.15.150:0/2170004026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57a0153320 con 0x7f57a00c8fc0 2024-09-17T04:50:09.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f5795ffb640 1 -- 172.21.15.150:0/2170004026 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5790018020 con 0x7f57a00c8fc0 2024-09-17T04:50:09.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f5795ffb640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5784073200 0x7f5784075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:09.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f5795ffb640 1 -- 172.21.15.150:0/2170004026 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f5790094670 con 0x7f57a00c8fc0 2024-09-17T04:50:09.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.047+0000 7f57a55f5640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5784073200 0x7f5784075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:09.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.051+0000 7f57a55f5640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5784073200 0x7f5784075710 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f579c0098e0 tx=0x7f579c009060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:09.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.051+0000 7f5795ffb640 1 -- 172.21.15.150:0/2170004026 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f579001f020 con 0x7f57a00c8fc0 2024-09-17T04:50:09.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.207+0000 7f57a7859640 1 -- 172.21.15.150:0/2170004026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7f57a0148d90 con 0x7f57a00c8fc0 2024-09-17T04:50:09.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f5795ffb640 1 -- 172.21.15.150:0/2170004026 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v26) ==== 95+0+11 (secure 0 0 0) 0x7f57a0148d90 con 0x7f57a00c8fc0 2024-09-17T04:50:09.213 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-09-17T04:50:09.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f57a7859640 1 -- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5784073200 msgr2=0x7f5784075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f57a7859640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5784073200 0x7f5784075710 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f579c0098e0 tx=0x7f579c009060 comp rx=0 tx=0).stop 2024-09-17T04:50:09.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f57a7859640 1 -- 172.21.15.150:0/2170004026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57a00c8fc0 msgr2=0x7f57a01f32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f57a7859640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57a00c8fc0 0x7f57a01f32e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f579000b440 tx=0x7f579000b900 comp rx=0 tx=0).stop 2024-09-17T04:50:09.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f57a7859640 1 -- 172.21.15.150:0/2170004026 shutdown_connections 2024-09-17T04:50:09.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f57a7859640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5784073200 0x7f5784075710 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f57a7859640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57a01568a0 0x7f57a0147970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f57a7859640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57a00c8fc0 0x7f57a01f32e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f57a7859640 1 --2- 172.21.15.150:0/2170004026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57a00c35b0 0x7f57a01527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f57a7859640 1 -- 172.21.15.150:0/2170004026 >> 172.21.15.150:0/2170004026 conn(0x7f57a0074b80 msgr2=0x7f57a01560b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:09.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f57a7859640 1 -- 172.21.15.150:0/2170004026 shutdown_connections 2024-09-17T04:50:09.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.211+0000 7f57a7859640 1 -- 172.21.15.150:0/2170004026 wait complete. 2024-09-17T04:50:09.232 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-09-17T04:50:09.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:50:09.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.383+0000 7f4b26ae1640 1 -- 172.21.15.150:0/560845557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b200c5490 msgr2=0x7f4b20153c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.383+0000 7f4b26ae1640 1 --2- 172.21.15.150:0/560845557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b200c5490 0x7f4b20153c50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4b08009bc0 tx=0x7f4b0801ca80 comp rx=0 tx=0).stop 2024-09-17T04:50:09.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.383+0000 7f4b26ae1640 1 -- 172.21.15.150:0/560845557 shutdown_connections 2024-09-17T04:50:09.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.383+0000 7f4b26ae1640 1 --2- 172.21.15.150:0/560845557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b200c5490 0x7f4b20153c50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.383+0000 7f4b26ae1640 1 --2- 172.21.15.150:0/560845557 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b2006f920 0x7f4b200c6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.383+0000 7f4b26ae1640 1 -- 172.21.15.150:0/560845557 >> 172.21.15.150:0/560845557 conn(0x7f4b200abb00 msgr2=0x7f4b200abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:09.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.383+0000 7f4b26ae1640 1 -- 172.21.15.150:0/560845557 shutdown_connections 2024-09-17T04:50:09.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.383+0000 7f4b26ae1640 1 -- 172.21.15.150:0/560845557 wait complete. 2024-09-17T04:50:09.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.383+0000 7f4b26ae1640 1 Processor -- start 2024-09-17T04:50:09.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b26ae1640 1 -- start start 2024-09-17T04:50:09.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b26ae1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b2006f920 0x7f4b201487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:09.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b26ae1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b200c5490 0x7f4b2014ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:09.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b26ae1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b20154190 0x7f4b200aa140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:09.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b26ae1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4b200c6d50 con 0x7f4b20154190 2024-09-17T04:50:09.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b26ae1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4b200c71d0 con 0x7f4b200c5490 2024-09-17T04:50:09.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b26ae1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4b201639d0 con 0x7f4b2006f920 2024-09-17T04:50:09.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b2507e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b20154190 0x7f4b200aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:09.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b2507e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b20154190 0x7f4b200aa140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37558/0 (socket says 172.21.15.150:37558) 2024-09-17T04:50:09.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b2507e640 1 -- 172.21.15.150:0/1481993315 learned_addr learned my addr 172.21.15.150:0/1481993315 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:09.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b2487d640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b2006f920 0x7f4b201487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:09.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b1ffff640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b200c5490 0x7f4b2014ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:09.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b2507e640 1 -- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b2006f920 msgr2=0x7f4b201487c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b2507e640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b2006f920 0x7f4b201487c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b2507e640 1 -- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b200c5490 msgr2=0x7f4b2014ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b2507e640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b200c5490 0x7f4b2014ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b2507e640 1 -- 172.21.15.150:0/1481993315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2014d110 con 0x7f4b20154190 2024-09-17T04:50:09.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b1ffff640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b200c5490 0x7f4b2014ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:50:09.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b2507e640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b20154190 0x7f4b200aa140 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800c3c0 tx=0x7f4b1800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:09.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b1d7fa640 1 -- 172.21.15.150:0/1481993315 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b18011790 con 0x7f4b20154190 2024-09-17T04:50:09.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b1d7fa640 1 -- 172.21.15.150:0/1481993315 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b18017070 con 0x7f4b20154190 2024-09-17T04:50:09.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b1d7fa640 1 -- 172.21.15.150:0/1481993315 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b18011ce0 con 0x7f4b20154190 2024-09-17T04:50:09.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b26ae1640 1 -- 172.21.15.150:0/1481993315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b20155c10 con 0x7f4b20154190 2024-09-17T04:50:09.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b26ae1640 1 -- 172.21.15.150:0/1481993315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b2014bf70 con 0x7f4b20154190 2024-09-17T04:50:09.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.387+0000 7f4b26ae1640 1 -- 172.21.15.150:0/1481993315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b200c6650 con 0x7f4b20154190 2024-09-17T04:50:09.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.391+0000 7f4b1d7fa640 1 -- 172.21.15.150:0/1481993315 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4b1801c460 con 0x7f4b20154190 2024-09-17T04:50:09.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.391+0000 7f4b1d7fa640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4b00073200 0x7f4b00075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:09.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.391+0000 7f4b1d7fa640 1 -- 172.21.15.150:0/1481993315 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f4b18094250 con 0x7f4b20154190 2024-09-17T04:50:09.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.391+0000 7f4b2487d640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4b00073200 0x7f4b00075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:09.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.391+0000 7f4b2487d640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4b00073200 0x7f4b00075710 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4b10009ad0 tx=0x7f4b100091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:09.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.395+0000 7f4b1d7fa640 1 -- 172.21.15.150:0/1481993315 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4b1805efc0 con 0x7f4b20154190 2024-09-17T04:50:09.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.603+0000 7f4b26ae1640 1 -- 172.21.15.150:0/1481993315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b200c4e20 con 0x7f4b20154190 2024-09-17T04:50:09.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b1d7fa640 1 -- 172.21.15.150:0/1481993315 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f4b1805e8c0 con 0x7f4b20154190 2024-09-17T04:50:09.609 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:50:09.610 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:50:09.610 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:50:09.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b26ae1640 1 -- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4b00073200 msgr2=0x7f4b00075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b26ae1640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4b00073200 0x7f4b00075710 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4b10009ad0 tx=0x7f4b100091e0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b26ae1640 1 -- 172.21.15.150:0/1481993315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b20154190 msgr2=0x7f4b200aa140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b26ae1640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b20154190 0x7f4b200aa140 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800c3c0 tx=0x7f4b1800c880 comp rx=0 tx=0).stop 2024-09-17T04:50:09.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b26ae1640 1 -- 172.21.15.150:0/1481993315 shutdown_connections 2024-09-17T04:50:09.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b26ae1640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4b00073200 0x7f4b00075710 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b26ae1640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b200c5490 0x7f4b2014ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b26ae1640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b20154190 0x7f4b200aa140 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b26ae1640 1 --2- 172.21.15.150:0/1481993315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b2006f920 0x7f4b201487c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b26ae1640 1 -- 172.21.15.150:0/1481993315 >> 172.21.15.150:0/1481993315 conn(0x7f4b200abb00 msgr2=0x7f4b200abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:09.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b26ae1640 1 -- 172.21.15.150:0/1481993315 shutdown_connections 2024-09-17T04:50:09.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.607+0000 7f4b26ae1640 1 -- 172.21.15.150:0/1481993315 wait complete. 2024-09-17T04:50:09.629 INFO:tasks.mds_thrash:Ready to start thrashing 2024-09-17T04:50:09.630 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-09-17T04:50:09.630 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:50:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.763+0000 7fa98b13f640 1 -- 172.21.15.150:0/271207626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa984158170 msgr2=0x7fa984158580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.763+0000 7fa98b13f640 1 --2- 172.21.15.150:0/271207626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa984158170 0x7fa984158580 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa9840adcf0 tx=0x7fa97c01cd80 comp rx=0 tx=0).stop 2024-09-17T04:50:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.763+0000 7fa98b13f640 1 -- 172.21.15.150:0/271207626 shutdown_connections 2024-09-17T04:50:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.763+0000 7fa98b13f640 1 --2- 172.21.15.150:0/271207626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa984158170 0x7fa984158580 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.763+0000 7fa98b13f640 1 --2- 172.21.15.150:0/271207626 >> v2:172.21.15.176:3300/0 conn(0x7fa9840c3b40 0x7fa984157c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.763+0000 7fa98b13f640 1 -- 172.21.15.150:0/271207626 >> 172.21.15.150:0/271207626 conn(0x7fa9840c4a00 msgr2=0x7fa9840c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:09.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.763+0000 7fa98b13f640 1 -- 172.21.15.150:0/271207626 shutdown_connections 2024-09-17T04:50:09.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.763+0000 7fa98b13f640 1 -- 172.21.15.150:0/271207626 wait complete. 2024-09-17T04:50:09.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa98b13f640 1 Processor -- start 2024-09-17T04:50:09.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa98b13f640 1 -- start start 2024-09-17T04:50:09.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa98b13f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9840c3b40 0x7fa984073580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:09.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa98b13f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa984151d00 0x7fa9841f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:09.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa98b13f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa984158170 0x7fa98414b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:09.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa98b13f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa9840c3f50 con 0x7fa984151d00 2024-09-17T04:50:09.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa98b13f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa9840ade90 con 0x7fa984158170 2024-09-17T04:50:09.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa98b13f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa9840c40d0 con 0x7fa9840c3b40 2024-09-17T04:50:09.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa983fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa984151d00 0x7fa9841f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:09.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa983fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa984151d00 0x7fa9841f3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37576/0 (socket says 172.21.15.150:37576) 2024-09-17T04:50:09.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa983fff640 1 -- 172.21.15.150:0/3871077399 learned_addr learned my addr 172.21.15.150:0/3871077399 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:09.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa988edb640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9840c3b40 0x7fa984073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:09.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa9896dc640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa984158170 0x7fa98414b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:09.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa983fff640 1 -- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9840c3b40 msgr2=0x7fa984073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa983fff640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9840c3b40 0x7fa984073580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa983fff640 1 -- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa984158170 msgr2=0x7fa98414b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa983fff640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa984158170 0x7fa98414b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa983fff640 1 -- 172.21.15.150:0/3871077399 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9841f3250 con 0x7fa984151d00 2024-09-17T04:50:09.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa988edb640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9840c3b40 0x7fa984073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:50:09.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa983fff640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa984151d00 0x7fa9841f3bc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa97400c970 tx=0x7fa97400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:09.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa9817fa640 1 -- 172.21.15.150:0/3871077399 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa974004220 con 0x7fa984151d00 2024-09-17T04:50:09.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa9817fa640 1 -- 172.21.15.150:0/3871077399 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa9740079b0 con 0x7fa984151d00 2024-09-17T04:50:09.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa9817fa640 1 -- 172.21.15.150:0/3871077399 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9740044b0 con 0x7fa984151d00 2024-09-17T04:50:09.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa98b13f640 1 -- 172.21.15.150:0/3871077399 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa984149570 con 0x7fa984151d00 2024-09-17T04:50:09.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa98b13f640 1 -- 172.21.15.150:0/3871077399 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9841f4100 con 0x7fa984151d00 2024-09-17T04:50:09.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa98b13f640 1 -- 172.21.15.150:0/3871077399 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa98414ae90 con 0x7fa984151d00 2024-09-17T04:50:09.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.767+0000 7fa9817fa640 1 -- 172.21.15.150:0/3871077399 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa974003600 con 0x7fa984151d00 2024-09-17T04:50:09.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.771+0000 7fa9817fa640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa970073200 0x7fa970075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:09.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.771+0000 7fa9817fa640 1 -- 172.21.15.150:0/3871077399 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fa9740945c0 con 0x7fa984151d00 2024-09-17T04:50:09.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.771+0000 7fa988edb640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa970073200 0x7fa970075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:09.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.771+0000 7fa988edb640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa970073200 0x7fa970075710 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa97800a850 tx=0x7fa978008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:09.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.775+0000 7fa9817fa640 1 -- 172.21.15.150:0/3871077399 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa98414ae90 con 0x7fa984151d00 2024-09-17T04:50:09.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.927+0000 7fa98b13f640 1 -- 172.21.15.150:0/3871077399 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa984157c30 con 0x7fa970073200 2024-09-17T04:50:09.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.951+0000 7fa9817fa640 1 -- 172.21.15.150:0/3871077399 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490314 (secure 0 0 0) 0x7fa984157c30 con 0x7fa970073200 2024-09-17T04:50:09.958 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:50:09.961 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T04:50:09.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.955+0000 7fa98b13f640 1 -- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa970073200 msgr2=0x7fa970075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.955+0000 7fa98b13f640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa970073200 0x7fa970075710 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa97800a850 tx=0x7fa978008040 comp rx=0 tx=0).stop 2024-09-17T04:50:09.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.955+0000 7fa98b13f640 1 -- 172.21.15.150:0/3871077399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa984151d00 msgr2=0x7fa9841f3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:09.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.955+0000 7fa98b13f640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa984151d00 0x7fa9841f3bc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa97400c970 tx=0x7fa97400ce30 comp rx=0 tx=0).stop 2024-09-17T04:50:09.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.955+0000 7fa98b13f640 1 -- 172.21.15.150:0/3871077399 shutdown_connections 2024-09-17T04:50:09.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.955+0000 7fa98b13f640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa970073200 0x7fa970075710 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.955+0000 7fa98b13f640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa984158170 0x7fa98414b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.955+0000 7fa98b13f640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa984151d00 0x7fa9841f3bc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.955+0000 7fa98b13f640 1 --2- 172.21.15.150:0/3871077399 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9840c3b40 0x7fa984073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:09.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.955+0000 7fa98b13f640 1 -- 172.21.15.150:0/3871077399 >> 172.21.15.150:0/3871077399 conn(0x7fa9840c4a00 msgr2=0x7fa984156440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:09.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.955+0000 7fa98b13f640 1 -- 172.21.15.150:0/3871077399 shutdown_connections 2024-09-17T04:50:09.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:09.955+0000 7fa98b13f640 1 -- 172.21.15.150:0/3871077399 wait complete. 2024-09-17T04:50:09.975 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-17T04:50:08.959982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595629,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":193,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223792,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750928,"statfs":{"total":773094113280,"available":772864950272,"internally_reserved":0,"allocated":3481600,"data_stored":2028210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":441,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006952"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671077+0000","last_change":"2024-09-17T04:49:41.025271+0000","last_active":"2024-09-17T04:49:42.671077+0000","last_peered":"2024-09-17T04:49:42.671077+0000","last_clean":"2024-09-17T04:49:42.671077+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T04:49:42.671077+0000","last_undegraded":"2024-09-17T04:49:42.671077+0000","last_fullsized":"2024-09-17T04:49:42.671077+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:39.715602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054640+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T04:49:42.054640+0000","last_peered":"2024-09-17T04:49:42.054640+0000","last_clean":"2024-09-17T04:49:42.054640+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T04:49:42.054640+0000","last_undegraded":"2024-09-17T04:49:42.054640+0000","last_fullsized":"2024-09-17T04:49:42.054640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671706+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T04:49:42.671706+0000","last_peered":"2024-09-17T04:49:42.671706+0000","last_clean":"2024-09-17T04:49:42.671706+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T04:49:42.671706+0000","last_undegraded":"2024-09-17T04:49:42.671706+0000","last_fullsized":"2024-09-17T04:49:42.671706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668736+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T04:49:42.668736+0000","last_peered":"2024-09-17T04:49:42.668736+0000","last_clean":"2024-09-17T04:49:42.668736+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T04:49:42.668736+0000","last_undegraded":"2024-09-17T04:49:42.668736+0000","last_fullsized":"2024-09-17T04:49:42.668736+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:38.417840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047972+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T04:49:42.047972+0000","last_peered":"2024-09-17T04:49:42.047972+0000","last_clean":"2024-09-17T04:49:42.047972+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T04:49:42.047972+0000","last_undegraded":"2024-09-17T04:49:42.047972+0000","last_fullsized":"2024-09-17T04:49:42.047972+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672300+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T04:49:42.672300+0000","last_peered":"2024-09-17T04:49:42.672300+0000","last_clean":"2024-09-17T04:49:42.672300+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T04:49:42.672300+0000","last_undegraded":"2024-09-17T04:49:42.672300+0000","last_fullsized":"2024-09-17T04:49:42.672300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672014+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T04:49:42.672014+0000","last_peered":"2024-09-17T04:49:42.672014+0000","last_clean":"2024-09-17T04:49:42.672014+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T04:49:42.672014+0000","last_undegraded":"2024-09-17T04:49:42.672014+0000","last_fullsized":"2024-09-17T04:49:42.672014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535134+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T04:49:42.535134+0000","last_peered":"2024-09-17T04:49:42.535134+0000","last_clean":"2024-09-17T04:49:42.535134+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T04:49:42.535134+0000","last_undegraded":"2024-09-17T04:49:42.535134+0000","last_fullsized":"2024-09-17T04:49:42.535134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672206+0000","last_change":"2024-09-17T04:49:41.067317+0000","last_active":"2024-09-17T04:49:42.672206+0000","last_peered":"2024-09-17T04:49:42.672206+0000","last_clean":"2024-09-17T04:49:42.672206+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T04:49:42.672206+0000","last_undegraded":"2024-09-17T04:49:42.672206+0000","last_fullsized":"2024-09-17T04:49:42.672206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:37.637477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670707+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T04:49:42.670707+0000","last_peered":"2024-09-17T04:49:42.670707+0000","last_clean":"2024-09-17T04:49:42.670707+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T04:49:42.670707+0000","last_undegraded":"2024-09-17T04:49:42.670707+0000","last_fullsized":"2024-09-17T04:49:42.670707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674274+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T04:49:42.674274+0000","last_peered":"2024-09-17T04:49:42.674274+0000","last_clean":"2024-09-17T04:49:42.674274+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T04:49:42.674274+0000","last_undegraded":"2024-09-17T04:49:42.674274+0000","last_fullsized":"2024-09-17T04:49:42.674274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535118+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T04:49:42.535118+0000","last_peered":"2024-09-17T04:49:42.535118+0000","last_clean":"2024-09-17T04:49:42.535118+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T04:49:42.535118+0000","last_undegraded":"2024-09-17T04:49:42.535118+0000","last_fullsized":"2024-09-17T04:49:42.535118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535259+0000","last_change":"2024-09-17T04:49:41.043531+0000","last_active":"2024-09-17T04:49:42.535259+0000","last_peered":"2024-09-17T04:49:42.535259+0000","last_clean":"2024-09-17T04:49:42.535259+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T04:49:42.535259+0000","last_undegraded":"2024-09-17T04:49:42.535259+0000","last_fullsized":"2024-09-17T04:49:42.535259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672162+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T04:49:42.672162+0000","last_peered":"2024-09-17T04:49:42.672162+0000","last_clean":"2024-09-17T04:49:42.672162+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T04:49:42.672162+0000","last_undegraded":"2024-09-17T04:49:42.672162+0000","last_fullsized":"2024-09-17T04:49:42.672162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042945+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T04:49:42.042945+0000","last_peered":"2024-09-17T04:49:42.042945+0000","last_clean":"2024-09-17T04:49:42.042945+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T04:49:42.042945+0000","last_undegraded":"2024-09-17T04:49:42.042945+0000","last_fullsized":"2024-09-17T04:49:42.042945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535003+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T04:49:42.535003+0000","last_peered":"2024-09-17T04:49:42.535003+0000","last_clean":"2024-09-17T04:49:42.535003+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T04:49:42.535003+0000","last_undegraded":"2024-09-17T04:49:42.535003+0000","last_fullsized":"2024-09-17T04:49:42.535003+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:16.854365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669588+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T04:49:42.669588+0000","last_peered":"2024-09-17T04:49:42.669588+0000","last_clean":"2024-09-17T04:49:42.669588+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T04:49:42.669588+0000","last_undegraded":"2024-09-17T04:49:42.669588+0000","last_fullsized":"2024-09-17T04:49:42.669588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042526+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T04:49:42.042526+0000","last_peered":"2024-09-17T04:49:42.042526+0000","last_clean":"2024-09-17T04:49:42.042526+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T04:49:42.042526+0000","last_undegraded":"2024-09-17T04:49:42.042526+0000","last_fullsized":"2024-09-17T04:49:42.042526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041407+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T04:49:42.041407+0000","last_peered":"2024-09-17T04:49:42.041407+0000","last_clean":"2024-09-17T04:49:42.041407+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T04:49:42.041407+0000","last_undegraded":"2024-09-17T04:49:42.041407+0000","last_fullsized":"2024-09-17T04:49:42.041407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042319+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T04:49:42.042319+0000","last_peered":"2024-09-17T04:49:42.042319+0000","last_clean":"2024-09-17T04:49:42.042319+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T04:49:42.042319+0000","last_undegraded":"2024-09-17T04:49:42.042319+0000","last_fullsized":"2024-09-17T04:49:42.042319+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:28.365399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669339+0000","last_change":"2024-09-17T04:49:41.038415+0000","last_active":"2024-09-17T04:49:42.669339+0000","last_peered":"2024-09-17T04:49:42.669339+0000","last_clean":"2024-09-17T04:49:42.669339+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T04:49:42.669339+0000","last_undegraded":"2024-09-17T04:49:42.669339+0000","last_fullsized":"2024-09-17T04:49:42.669339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:17.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668616+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T04:49:42.668616+0000","last_peered":"2024-09-17T04:49:42.668616+0000","last_clean":"2024-09-17T04:49:42.668616+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T04:49:42.668616+0000","last_undegraded":"2024-09-17T04:49:42.668616+0000","last_fullsized":"2024-09-17T04:49:42.668616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052248+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T04:49:42.052248+0000","last_peered":"2024-09-17T04:49:42.052248+0000","last_clean":"2024-09-17T04:49:42.052248+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T04:49:42.052248+0000","last_undegraded":"2024-09-17T04:49:42.052248+0000","last_fullsized":"2024-09-17T04:49:42.052248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671586+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T04:49:42.671586+0000","last_peered":"2024-09-17T04:49:42.671586+0000","last_clean":"2024-09-17T04:49:42.671586+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T04:49:42.671586+0000","last_undegraded":"2024-09-17T04:49:42.671586+0000","last_fullsized":"2024-09-17T04:49:42.671586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:17.424265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042028+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T04:49:42.042028+0000","last_peered":"2024-09-17T04:49:42.042028+0000","last_clean":"2024-09-17T04:49:42.042028+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T04:49:42.042028+0000","last_undegraded":"2024-09-17T04:49:42.042028+0000","last_fullsized":"2024-09-17T04:49:42.042028+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:06.516432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048455+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T04:49:42.048455+0000","last_peered":"2024-09-17T04:49:42.048455+0000","last_clean":"2024-09-17T04:49:42.048455+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T04:49:42.048455+0000","last_undegraded":"2024-09-17T04:49:42.048455+0000","last_fullsized":"2024-09-17T04:49:42.048455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048500+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T04:49:42.048500+0000","last_peered":"2024-09-17T04:49:42.048500+0000","last_clean":"2024-09-17T04:49:42.048500+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T04:49:42.048500+0000","last_undegraded":"2024-09-17T04:49:42.048500+0000","last_fullsized":"2024-09-17T04:49:42.048500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668827+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T04:49:42.668827+0000","last_peered":"2024-09-17T04:49:42.668827+0000","last_clean":"2024-09-17T04:49:42.668827+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T04:49:42.668827+0000","last_undegraded":"2024-09-17T04:49:42.668827+0000","last_fullsized":"2024-09-17T04:49:42.668827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:00.010365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675406+0000","last_change":"2024-09-17T04:49:41.036658+0000","last_active":"2024-09-17T04:49:42.675406+0000","last_peered":"2024-09-17T04:49:42.675406+0000","last_clean":"2024-09-17T04:49:42.675406+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T04:49:42.675406+0000","last_undegraded":"2024-09-17T04:49:42.675406+0000","last_fullsized":"2024-09-17T04:49:42.675406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042824+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T04:49:42.042824+0000","last_peered":"2024-09-17T04:49:42.042824+0000","last_clean":"2024-09-17T04:49:42.042824+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T04:49:42.042824+0000","last_undegraded":"2024-09-17T04:49:42.042824+0000","last_fullsized":"2024-09-17T04:49:42.042824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668829+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T04:49:42.668829+0000","last_peered":"2024-09-17T04:49:42.668829+0000","last_clean":"2024-09-17T04:49:42.668829+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T04:49:42.668829+0000","last_undegraded":"2024-09-17T04:49:42.668829+0000","last_fullsized":"2024-09-17T04:49:42.668829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536259+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T04:49:42.536259+0000","last_peered":"2024-09-17T04:49:42.536259+0000","last_clean":"2024-09-17T04:49:42.536259+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T04:49:42.536259+0000","last_undegraded":"2024-09-17T04:49:42.536259+0000","last_fullsized":"2024-09-17T04:49:42.536259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048942+0000","last_change":"2024-09-17T04:49:41.050170+0000","last_active":"2024-09-17T04:49:42.048942+0000","last_peered":"2024-09-17T04:49:42.048942+0000","last_clean":"2024-09-17T04:49:42.048942+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T04:49:42.048942+0000","last_undegraded":"2024-09-17T04:49:42.048942+0000","last_fullsized":"2024-09-17T04:49:42.048942+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051929+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T04:49:42.051929+0000","last_peered":"2024-09-17T04:49:42.051929+0000","last_clean":"2024-09-17T04:49:42.051929+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T04:49:42.051929+0000","last_undegraded":"2024-09-17T04:49:42.051929+0000","last_fullsized":"2024-09-17T04:49:42.051929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042185+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T04:49:42.042185+0000","last_peered":"2024-09-17T04:49:42.042185+0000","last_clean":"2024-09-17T04:49:42.042185+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T04:49:42.042185+0000","last_undegraded":"2024-09-17T04:49:42.042185+0000","last_fullsized":"2024-09-17T04:49:42.042185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535553+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T04:49:42.535553+0000","last_peered":"2024-09-17T04:49:42.535553+0000","last_clean":"2024-09-17T04:49:42.535553+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T04:49:42.535553+0000","last_undegraded":"2024-09-17T04:49:42.535553+0000","last_fullsized":"2024-09-17T04:49:42.535553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041211+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T04:49:42.041211+0000","last_peered":"2024-09-17T04:49:42.041211+0000","last_clean":"2024-09-17T04:49:42.041211+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T04:49:42.041211+0000","last_undegraded":"2024-09-17T04:49:42.041211+0000","last_fullsized":"2024-09-17T04:49:42.041211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054336+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T04:49:42.054336+0000","last_peered":"2024-09-17T04:49:42.054336+0000","last_clean":"2024-09-17T04:49:42.054336+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T04:49:42.054336+0000","last_undegraded":"2024-09-17T04:49:42.054336+0000","last_fullsized":"2024-09-17T04:49:42.054336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052826+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T04:49:42.052826+0000","last_peered":"2024-09-17T04:49:42.052826+0000","last_clean":"2024-09-17T04:49:42.052826+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T04:49:42.052826+0000","last_undegraded":"2024-09-17T04:49:42.052826+0000","last_fullsized":"2024-09-17T04:49:42.052826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048145+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T04:49:42.048145+0000","last_peered":"2024-09-17T04:49:42.048145+0000","last_clean":"2024-09-17T04:49:42.048145+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T04:49:42.048145+0000","last_undegraded":"2024-09-17T04:49:42.048145+0000","last_fullsized":"2024-09-17T04:49:42.048145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669693+0000","last_change":"2024-09-17T04:49:41.042314+0000","last_active":"2024-09-17T04:49:42.669693+0000","last_peered":"2024-09-17T04:49:42.669693+0000","last_clean":"2024-09-17T04:49:42.669693+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T04:49:42.669693+0000","last_undegraded":"2024-09-17T04:49:42.669693+0000","last_fullsized":"2024-09-17T04:49:42.669693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055299+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T04:49:42.055299+0000","last_peered":"2024-09-17T04:49:42.055299+0000","last_clean":"2024-09-17T04:49:42.055299+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T04:49:42.055299+0000","last_undegraded":"2024-09-17T04:49:42.055299+0000","last_fullsized":"2024-09-17T04:49:42.055299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043663+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T04:49:42.043663+0000","last_peered":"2024-09-17T04:49:42.043663+0000","last_clean":"2024-09-17T04:49:42.043663+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T04:49:42.043663+0000","last_undegraded":"2024-09-17T04:49:42.043663+0000","last_fullsized":"2024-09-17T04:49:42.043663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053169+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T04:49:42.053169+0000","last_peered":"2024-09-17T04:49:42.053169+0000","last_clean":"2024-09-17T04:49:42.053169+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T04:49:42.053169+0000","last_undegraded":"2024-09-17T04:49:42.053169+0000","last_fullsized":"2024-09-17T04:49:42.053169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052959+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T04:49:42.052959+0000","last_peered":"2024-09-17T04:49:42.052959+0000","last_clean":"2024-09-17T04:49:42.052959+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T04:49:42.052959+0000","last_undegraded":"2024-09-17T04:49:42.052959+0000","last_fullsized":"2024-09-17T04:49:42.052959+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535986+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T04:49:42.535986+0000","last_peered":"2024-09-17T04:49:42.535986+0000","last_clean":"2024-09-17T04:49:42.535986+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T04:49:42.535986+0000","last_undegraded":"2024-09-17T04:49:42.535986+0000","last_fullsized":"2024-09-17T04:49:42.535986+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670900+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T04:49:42.670900+0000","last_peered":"2024-09-17T04:49:42.670900+0000","last_clean":"2024-09-17T04:49:42.670900+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T04:49:42.670900+0000","last_undegraded":"2024-09-17T04:49:42.670900+0000","last_fullsized":"2024-09-17T04:49:42.670900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050191+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T04:49:42.050191+0000","last_peered":"2024-09-17T04:49:42.050191+0000","last_clean":"2024-09-17T04:49:42.050191+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T04:49:42.050191+0000","last_undegraded":"2024-09-17T04:49:42.050191+0000","last_fullsized":"2024-09-17T04:49:42.050191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670215+0000","last_change":"2024-09-17T04:49:41.069177+0000","last_active":"2024-09-17T04:49:42.670215+0000","last_peered":"2024-09-17T04:49:42.670215+0000","last_clean":"2024-09-17T04:49:42.670215+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T04:49:42.670215+0000","last_undegraded":"2024-09-17T04:49:42.670215+0000","last_fullsized":"2024-09-17T04:49:42.670215+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:16.855206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670381+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T04:49:42.670381+0000","last_peered":"2024-09-17T04:49:42.670381+0000","last_clean":"2024-09-17T04:49:42.670381+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T04:49:42.670381+0000","last_undegraded":"2024-09-17T04:49:42.670381+0000","last_fullsized":"2024-09-17T04:49:42.670381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042706+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T04:49:42.042706+0000","last_peered":"2024-09-17T04:49:42.042706+0000","last_clean":"2024-09-17T04:49:42.042706+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T04:49:42.042706+0000","last_undegraded":"2024-09-17T04:49:42.042706+0000","last_fullsized":"2024-09-17T04:49:42.042706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054900+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T04:49:42.054900+0000","last_peered":"2024-09-17T04:49:42.054900+0000","last_clean":"2024-09-17T04:49:42.054900+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T04:49:42.054900+0000","last_undegraded":"2024-09-17T04:49:42.054900+0000","last_fullsized":"2024-09-17T04:49:42.054900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669308+0000","last_change":"2024-09-17T04:49:41.068537+0000","last_active":"2024-09-17T04:49:42.669308+0000","last_peered":"2024-09-17T04:49:42.669308+0000","last_clean":"2024-09-17T04:49:42.669308+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T04:49:42.669308+0000","last_undegraded":"2024-09-17T04:49:42.669308+0000","last_fullsized":"2024-09-17T04:49:42.669308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667397+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T04:49:42.667397+0000","last_peered":"2024-09-17T04:49:42.667397+0000","last_clean":"2024-09-17T04:49:42.667397+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T04:49:42.667397+0000","last_undegraded":"2024-09-17T04:49:42.667397+0000","last_fullsized":"2024-09-17T04:49:42.667397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668940+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T04:49:42.668940+0000","last_peered":"2024-09-17T04:49:42.668940+0000","last_clean":"2024-09-17T04:49:42.668940+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T04:49:42.668940+0000","last_undegraded":"2024-09-17T04:49:42.668940+0000","last_fullsized":"2024-09-17T04:49:42.668940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054921+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T04:49:42.054921+0000","last_peered":"2024-09-17T04:49:42.054921+0000","last_clean":"2024-09-17T04:49:42.054921+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T04:49:42.054921+0000","last_undegraded":"2024-09-17T04:49:42.054921+0000","last_fullsized":"2024-09-17T04:49:42.054921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041762+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T04:49:42.041762+0000","last_peered":"2024-09-17T04:49:42.041762+0000","last_clean":"2024-09-17T04:49:42.041762+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T04:49:42.041762+0000","last_undegraded":"2024-09-17T04:49:42.041762+0000","last_fullsized":"2024-09-17T04:49:42.041762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050299+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T04:49:42.050299+0000","last_peered":"2024-09-17T04:49:42.050299+0000","last_clean":"2024-09-17T04:49:42.050299+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T04:49:42.050299+0000","last_undegraded":"2024-09-17T04:49:42.050299+0000","last_fullsized":"2024-09-17T04:49:42.050299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041771+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T04:49:42.041771+0000","last_peered":"2024-09-17T04:49:42.041771+0000","last_clean":"2024-09-17T04:49:42.041771+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T04:49:42.041771+0000","last_undegraded":"2024-09-17T04:49:42.041771+0000","last_fullsized":"2024-09-17T04:49:42.041771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672234+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T04:49:42.672234+0000","last_peered":"2024-09-17T04:49:42.672234+0000","last_clean":"2024-09-17T04:49:42.672234+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T04:49:42.672234+0000","last_undegraded":"2024-09-17T04:49:42.672234+0000","last_fullsized":"2024-09-17T04:49:42.672234+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671183+0000","last_change":"2024-09-17T04:49:41.052564+0000","last_active":"2024-09-17T04:49:42.671183+0000","last_peered":"2024-09-17T04:49:42.671183+0000","last_clean":"2024-09-17T04:49:42.671183+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T04:49:42.671183+0000","last_undegraded":"2024-09-17T04:49:42.671183+0000","last_fullsized":"2024-09-17T04:49:42.671183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049438+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T04:49:42.049438+0000","last_peered":"2024-09-17T04:49:42.049438+0000","last_clean":"2024-09-17T04:49:42.049438+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T04:49:42.049438+0000","last_undegraded":"2024-09-17T04:49:42.049438+0000","last_fullsized":"2024-09-17T04:49:42.049438+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041458+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T04:49:42.041458+0000","last_peered":"2024-09-17T04:49:42.041458+0000","last_clean":"2024-09-17T04:49:42.041458+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T04:49:42.041458+0000","last_undegraded":"2024-09-17T04:49:42.041458+0000","last_fullsized":"2024-09-17T04:49:42.041458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671735+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T04:49:42.671735+0000","last_peered":"2024-09-17T04:49:42.671735+0000","last_clean":"2024-09-17T04:49:42.671735+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T04:49:42.671735+0000","last_undegraded":"2024-09-17T04:49:42.671735+0000","last_fullsized":"2024-09-17T04:49:42.671735+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043078+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T04:49:42.043078+0000","last_peered":"2024-09-17T04:49:42.043078+0000","last_clean":"2024-09-17T04:49:42.043078+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T04:49:42.043078+0000","last_undegraded":"2024-09-17T04:49:42.043078+0000","last_fullsized":"2024-09-17T04:49:42.043078+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:21.202398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669578+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T04:49:42.669578+0000","last_peered":"2024-09-17T04:49:42.669578+0000","last_clean":"2024-09-17T04:49:42.669578+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T04:49:42.669578+0000","last_undegraded":"2024-09-17T04:49:42.669578+0000","last_fullsized":"2024-09-17T04:49:42.669578+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:48.128609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043280+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T04:49:42.043280+0000","last_peered":"2024-09-17T04:49:42.043280+0000","last_clean":"2024-09-17T04:49:42.043280+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T04:49:42.043280+0000","last_undegraded":"2024-09-17T04:49:42.043280+0000","last_fullsized":"2024-09-17T04:49:42.043280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055292+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T04:49:42.055292+0000","last_peered":"2024-09-17T04:49:42.055292+0000","last_clean":"2024-09-17T04:49:42.055292+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T04:49:42.055292+0000","last_undegraded":"2024-09-17T04:49:42.055292+0000","last_fullsized":"2024-09-17T04:49:42.055292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536164+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T04:49:42.536164+0000","last_peered":"2024-09-17T04:49:42.536164+0000","last_clean":"2024-09-17T04:49:42.536164+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T04:49:42.536164+0000","last_undegraded":"2024-09-17T04:49:42.536164+0000","last_fullsized":"2024-09-17T04:49:42.536164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050257+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T04:49:42.050257+0000","last_peered":"2024-09-17T04:49:42.050257+0000","last_clean":"2024-09-17T04:49:42.050257+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T04:49:42.050257+0000","last_undegraded":"2024-09-17T04:49:42.050257+0000","last_fullsized":"2024-09-17T04:49:42.050257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671927+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T04:49:42.671927+0000","last_peered":"2024-09-17T04:49:42.671927+0000","last_clean":"2024-09-17T04:49:42.671927+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T04:49:42.671927+0000","last_undegraded":"2024-09-17T04:49:42.671927+0000","last_fullsized":"2024-09-17T04:49:42.671927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671411+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T04:49:42.671411+0000","last_peered":"2024-09-17T04:49:42.671411+0000","last_clean":"2024-09-17T04:49:42.671411+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T04:49:42.671411+0000","last_undegraded":"2024-09-17T04:49:42.671411+0000","last_fullsized":"2024-09-17T04:49:42.671411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670087+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T04:49:42.670087+0000","last_peered":"2024-09-17T04:49:42.670087+0000","last_clean":"2024-09-17T04:49:42.670087+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T04:49:42.670087+0000","last_undegraded":"2024-09-17T04:49:42.670087+0000","last_fullsized":"2024-09-17T04:49:42.670087+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:18.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672287+0000","last_change":"2024-09-17T04:49:41.031036+0000","last_active":"2024-09-17T04:49:42.672287+0000","last_peered":"2024-09-17T04:49:42.672287+0000","last_clean":"2024-09-17T04:49:42.672287+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T04:49:42.672287+0000","last_undegraded":"2024-09-17T04:49:42.672287+0000","last_fullsized":"2024-09-17T04:49:42.672287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:48.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672111+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T04:49:42.672111+0000","last_peered":"2024-09-17T04:49:42.672111+0000","last_clean":"2024-09-17T04:49:42.672111+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T04:49:42.672111+0000","last_undegraded":"2024-09-17T04:49:42.672111+0000","last_fullsized":"2024-09-17T04:49:42.672111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052688+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T04:49:42.052688+0000","last_peered":"2024-09-17T04:49:42.052688+0000","last_clean":"2024-09-17T04:49:42.052688+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T04:49:42.052688+0000","last_undegraded":"2024-09-17T04:49:42.052688+0000","last_fullsized":"2024-09-17T04:49:42.052688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041095+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T04:49:42.041095+0000","last_peered":"2024-09-17T04:49:42.041095+0000","last_clean":"2024-09-17T04:49:42.041095+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T04:49:42.041095+0000","last_undegraded":"2024-09-17T04:49:42.041095+0000","last_fullsized":"2024-09-17T04:49:42.041095+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048488+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T04:49:42.048488+0000","last_peered":"2024-09-17T04:49:42.048488+0000","last_clean":"2024-09-17T04:49:42.048488+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T04:49:42.048488+0000","last_undegraded":"2024-09-17T04:49:42.048488+0000","last_fullsized":"2024-09-17T04:49:42.048488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536967+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T04:49:42.536967+0000","last_peered":"2024-09-17T04:49:42.536967+0000","last_clean":"2024-09-17T04:49:42.536967+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T04:49:42.536967+0000","last_undegraded":"2024-09-17T04:49:42.536967+0000","last_fullsized":"2024-09-17T04:49:42.536967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672404+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T04:49:42.672404+0000","last_peered":"2024-09-17T04:49:42.672404+0000","last_clean":"2024-09-17T04:49:42.672404+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T04:49:42.672404+0000","last_undegraded":"2024-09-17T04:49:42.672404+0000","last_fullsized":"2024-09-17T04:49:42.672404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672966+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T04:49:42.672966+0000","last_peered":"2024-09-17T04:49:42.672966+0000","last_clean":"2024-09-17T04:49:42.672966+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T04:49:42.672966+0000","last_undegraded":"2024-09-17T04:49:42.672966+0000","last_fullsized":"2024-09-17T04:49:42.672966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:09.712272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052007+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T04:49:42.052007+0000","last_peered":"2024-09-17T04:49:42.052007+0000","last_clean":"2024-09-17T04:49:42.052007+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T04:49:42.052007+0000","last_undegraded":"2024-09-17T04:49:42.052007+0000","last_fullsized":"2024-09-17T04:49:42.052007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:43.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042787+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T04:49:42.042787+0000","last_peered":"2024-09-17T04:49:42.042787+0000","last_clean":"2024-09-17T04:49:42.042787+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T04:49:42.042787+0000","last_undegraded":"2024-09-17T04:49:42.042787+0000","last_fullsized":"2024-09-17T04:49:42.042787+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049500+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T04:49:42.049500+0000","last_peered":"2024-09-17T04:49:42.049500+0000","last_clean":"2024-09-17T04:49:42.049500+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T04:49:42.049500+0000","last_undegraded":"2024-09-17T04:49:42.049500+0000","last_fullsized":"2024-09-17T04:49:42.049500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671253+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T04:49:42.671253+0000","last_peered":"2024-09-17T04:49:42.671253+0000","last_clean":"2024-09-17T04:49:42.671253+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T04:49:42.671253+0000","last_undegraded":"2024-09-17T04:49:42.671253+0000","last_fullsized":"2024-09-17T04:49:42.671253+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:52.364568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041504+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T04:49:42.041504+0000","last_peered":"2024-09-17T04:49:42.041504+0000","last_clean":"2024-09-17T04:49:42.041504+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T04:49:42.041504+0000","last_undegraded":"2024-09-17T04:49:42.041504+0000","last_fullsized":"2024-09-17T04:49:42.041504+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:03.537392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671439+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T04:49:42.671439+0000","last_peered":"2024-09-17T04:49:42.671439+0000","last_clean":"2024-09-17T04:49:42.671439+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T04:49:42.671439+0000","last_undegraded":"2024-09-17T04:49:42.671439+0000","last_fullsized":"2024-09-17T04:49:42.671439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670943+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T04:49:42.670943+0000","last_peered":"2024-09-17T04:49:42.670943+0000","last_clean":"2024-09-17T04:49:42.670943+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T04:49:42.670943+0000","last_undegraded":"2024-09-17T04:49:42.670943+0000","last_fullsized":"2024-09-17T04:49:42.670943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671787+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T04:49:42.671787+0000","last_peered":"2024-09-17T04:49:42.671787+0000","last_clean":"2024-09-17T04:49:42.671787+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T04:49:42.671787+0000","last_undegraded":"2024-09-17T04:49:42.671787+0000","last_fullsized":"2024-09-17T04:49:42.671787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:52.120543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041402+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T04:49:42.041402+0000","last_peered":"2024-09-17T04:49:42.041402+0000","last_clean":"2024-09-17T04:49:42.041402+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T04:49:42.041402+0000","last_undegraded":"2024-09-17T04:49:42.041402+0000","last_fullsized":"2024-09-17T04:49:42.041402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:56.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041285+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T04:49:42.041285+0000","last_peered":"2024-09-17T04:49:42.041285+0000","last_clean":"2024-09-17T04:49:42.041285+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T04:49:42.041285+0000","last_undegraded":"2024-09-17T04:49:42.041285+0000","last_fullsized":"2024-09-17T04:49:42.041285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042566+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T04:49:42.042566+0000","last_peered":"2024-09-17T04:49:42.042566+0000","last_clean":"2024-09-17T04:49:42.042566+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T04:49:42.042566+0000","last_undegraded":"2024-09-17T04:49:42.042566+0000","last_fullsized":"2024-09-17T04:49:42.042566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054288+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T04:49:42.054288+0000","last_peered":"2024-09-17T04:49:42.054288+0000","last_clean":"2024-09-17T04:49:42.054288+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T04:49:42.054288+0000","last_undegraded":"2024-09-17T04:49:42.054288+0000","last_fullsized":"2024-09-17T04:49:42.054288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051153+0000","last_change":"2024-09-17T04:49:41.058280+0000","last_active":"2024-09-17T04:49:42.051153+0000","last_peered":"2024-09-17T04:49:42.051153+0000","last_clean":"2024-09-17T04:49:42.051153+0000","last_became_active":"2024-09-17T04:49:41.052009+0000","last_became_peered":"2024-09-17T04:49:41.052009+0000","last_unstale":"2024-09-17T04:49:42.051153+0000","last_undegraded":"2024-09-17T04:49:42.051153+0000","last_fullsized":"2024-09-17T04:49:42.051153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:14.100818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051977+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T04:49:42.051977+0000","last_peered":"2024-09-17T04:49:42.051977+0000","last_clean":"2024-09-17T04:49:42.051977+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T04:49:42.051977+0000","last_undegraded":"2024-09-17T04:49:42.051977+0000","last_fullsized":"2024-09-17T04:49:42.051977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669355+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T04:49:42.669355+0000","last_peered":"2024-09-17T04:49:42.669355+0000","last_clean":"2024-09-17T04:49:42.669355+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T04:49:42.669355+0000","last_undegraded":"2024-09-17T04:49:42.669355+0000","last_fullsized":"2024-09-17T04:49:42.669355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042782+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T04:49:42.042782+0000","last_peered":"2024-09-17T04:49:42.042782+0000","last_clean":"2024-09-17T04:49:42.042782+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T04:49:42.042782+0000","last_undegraded":"2024-09-17T04:49:42.042782+0000","last_fullsized":"2024-09-17T04:49:42.042782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:07.945355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052280+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T04:49:42.052280+0000","last_peered":"2024-09-17T04:49:42.052280+0000","last_clean":"2024-09-17T04:49:42.052280+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T04:49:42.052280+0000","last_undegraded":"2024-09-17T04:49:42.052280+0000","last_fullsized":"2024-09-17T04:49:42.052280+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670412+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T04:49:42.670412+0000","last_peered":"2024-09-17T04:49:42.670412+0000","last_clean":"2024-09-17T04:49:42.670412+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T04:49:42.670412+0000","last_undegraded":"2024-09-17T04:49:42.670412+0000","last_fullsized":"2024-09-17T04:49:42.670412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:50:02.209562+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T04:50:02.209562+0000","last_peered":"2024-09-17T04:50:02.209562+0000","last_clean":"2024-09-17T04:50:02.209562+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T04:50:02.209562+0000","last_undegraded":"2024-09-17T04:50:02.209562+0000","last_fullsized":"2024-09-17T04:50:02.209562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048913+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T04:49:42.048913+0000","last_peered":"2024-09-17T04:49:42.048913+0000","last_clean":"2024-09-17T04:49:42.048913+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T04:49:42.048913+0000","last_undegraded":"2024-09-17T04:49:42.048913+0000","last_fullsized":"2024-09-17T04:49:42.048913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:16.447835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051373+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T04:49:42.051373+0000","last_peered":"2024-09-17T04:49:42.051373+0000","last_clean":"2024-09-17T04:49:42.051373+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T04:49:42.051373+0000","last_undegraded":"2024-09-17T04:49:42.051373+0000","last_fullsized":"2024-09-17T04:49:42.051373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042063+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T04:49:42.042063+0000","last_peered":"2024-09-17T04:49:42.042063+0000","last_clean":"2024-09-17T04:49:42.042063+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T04:49:42.042063+0000","last_undegraded":"2024-09-17T04:49:42.042063+0000","last_fullsized":"2024-09-17T04:49:42.042063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670596+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T04:49:42.670596+0000","last_peered":"2024-09-17T04:49:42.670596+0000","last_clean":"2024-09-17T04:49:42.670596+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T04:49:42.670596+0000","last_undegraded":"2024-09-17T04:49:42.670596+0000","last_fullsized":"2024-09-17T04:49:42.670596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535510+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T04:49:42.535510+0000","last_peered":"2024-09-17T04:49:42.535510+0000","last_clean":"2024-09-17T04:49:42.535510+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T04:49:42.535510+0000","last_undegraded":"2024-09-17T04:49:42.535510+0000","last_fullsized":"2024-09-17T04:49:42.535510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041322+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T04:49:42.041322+0000","last_peered":"2024-09-17T04:49:42.041322+0000","last_clean":"2024-09-17T04:49:42.041322+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T04:49:42.041322+0000","last_undegraded":"2024-09-17T04:49:42.041322+0000","last_fullsized":"2024-09-17T04:49:42.041322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672182+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T04:49:42.672182+0000","last_peered":"2024-09-17T04:49:42.672182+0000","last_clean":"2024-09-17T04:49:42.672182+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T04:49:42.672182+0000","last_undegraded":"2024-09-17T04:49:42.672182+0000","last_fullsized":"2024-09-17T04:49:42.672182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671261+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T04:49:42.671261+0000","last_peered":"2024-09-17T04:49:42.671261+0000","last_clean":"2024-09-17T04:49:42.671261+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T04:49:42.671261+0000","last_undegraded":"2024-09-17T04:49:42.671261+0000","last_fullsized":"2024-09-17T04:49:42.671261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667575+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T04:49:42.667575+0000","last_peered":"2024-09-17T04:49:42.667575+0000","last_clean":"2024-09-17T04:49:42.667575+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T04:49:42.667575+0000","last_undegraded":"2024-09-17T04:49:42.667575+0000","last_fullsized":"2024-09-17T04:49:42.667575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:40.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534654+0000","last_change":"2024-09-17T04:49:41.040278+0000","last_active":"2024-09-17T04:49:42.534654+0000","last_peered":"2024-09-17T04:49:42.534654+0000","last_clean":"2024-09-17T04:49:42.534654+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T04:49:42.534654+0000","last_undegraded":"2024-09-17T04:49:42.534654+0000","last_fullsized":"2024-09-17T04:49:42.534654+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.895223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536732+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T04:49:42.536732+0000","last_peered":"2024-09-17T04:49:42.536732+0000","last_clean":"2024-09-17T04:49:42.536732+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T04:49:42.536732+0000","last_undegraded":"2024-09-17T04:49:42.536732+0000","last_fullsized":"2024-09-17T04:49:42.536732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675753+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T04:49:42.675753+0000","last_peered":"2024-09-17T04:49:42.675753+0000","last_clean":"2024-09-17T04:49:42.675753+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T04:49:42.675753+0000","last_undegraded":"2024-09-17T04:49:42.675753+0000","last_fullsized":"2024-09-17T04:49:42.675753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047646+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T04:49:42.047646+0000","last_peered":"2024-09-17T04:49:42.047646+0000","last_clean":"2024-09-17T04:49:42.047646+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T04:49:42.047646+0000","last_undegraded":"2024-09-17T04:49:42.047646+0000","last_fullsized":"2024-09-17T04:49:42.047646+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:15.845612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537298+0000","last_change":"2024-09-17T04:49:41.050551+0000","last_active":"2024-09-17T04:49:42.537298+0000","last_peered":"2024-09-17T04:49:42.537298+0000","last_clean":"2024-09-17T04:49:42.537298+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T04:49:42.537298+0000","last_undegraded":"2024-09-17T04:49:42.537298+0000","last_fullsized":"2024-09-17T04:49:42.537298+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:11.088230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051592+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T04:49:42.051592+0000","last_peered":"2024-09-17T04:49:42.051592+0000","last_clean":"2024-09-17T04:49:42.051592+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T04:49:42.051592+0000","last_undegraded":"2024-09-17T04:49:42.051592+0000","last_fullsized":"2024-09-17T04:49:42.051592+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:37.778756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048401+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T04:49:42.048401+0000","last_peered":"2024-09-17T04:49:42.048401+0000","last_clean":"2024-09-17T04:49:42.048401+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T04:49:42.048401+0000","last_undegraded":"2024-09-17T04:49:42.048401+0000","last_fullsized":"2024-09-17T04:49:42.048401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041797+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T04:49:42.041797+0000","last_peered":"2024-09-17T04:49:42.041797+0000","last_clean":"2024-09-17T04:49:42.041797+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T04:49:42.041797+0000","last_undegraded":"2024-09-17T04:49:42.041797+0000","last_fullsized":"2024-09-17T04:49:42.041797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055365+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T04:49:42.055365+0000","last_peered":"2024-09-17T04:49:42.055365+0000","last_clean":"2024-09-17T04:49:42.055365+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T04:49:42.055365+0000","last_undegraded":"2024-09-17T04:49:42.055365+0000","last_fullsized":"2024-09-17T04:49:42.055365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041767+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T04:49:42.041767+0000","last_peered":"2024-09-17T04:49:42.041767+0000","last_clean":"2024-09-17T04:49:42.041767+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T04:49:42.041767+0000","last_undegraded":"2024-09-17T04:49:42.041767+0000","last_fullsized":"2024-09-17T04:49:42.041767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667708+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T04:49:42.667708+0000","last_peered":"2024-09-17T04:49:42.667708+0000","last_clean":"2024-09-17T04:49:42.667708+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T04:49:42.667708+0000","last_undegraded":"2024-09-17T04:49:42.667708+0000","last_fullsized":"2024-09-17T04:49:42.667708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050522+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T04:49:42.050522+0000","last_peered":"2024-09-17T04:49:42.050522+0000","last_clean":"2024-09-17T04:49:42.050522+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T04:49:42.050522+0000","last_undegraded":"2024-09-17T04:49:42.050522+0000","last_fullsized":"2024-09-17T04:49:42.050522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049954+0000","last_change":"2024-09-17T04:49:41.057647+0000","last_active":"2024-09-17T04:49:42.049954+0000","last_peered":"2024-09-17T04:49:42.049954+0000","last_clean":"2024-09-17T04:49:42.049954+0000","last_became_active":"2024-09-17T04:49:41.056213+0000","last_became_peered":"2024-09-17T04:49:41.056213+0000","last_unstale":"2024-09-17T04:49:42.049954+0000","last_undegraded":"2024-09-17T04:49:42.049954+0000","last_fullsized":"2024-09-17T04:49:42.049954+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042498+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T04:49:42.042498+0000","last_peered":"2024-09-17T04:49:42.042498+0000","last_clean":"2024-09-17T04:49:42.042498+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T04:49:42.042498+0000","last_undegraded":"2024-09-17T04:49:42.042498+0000","last_fullsized":"2024-09-17T04:49:42.042498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053714+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T04:49:42.053714+0000","last_peered":"2024-09-17T04:49:42.053714+0000","last_clean":"2024-09-17T04:49:42.053714+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T04:49:42.053714+0000","last_undegraded":"2024-09-17T04:49:42.053714+0000","last_fullsized":"2024-09-17T04:49:42.053714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051282+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T04:49:42.051282+0000","last_peered":"2024-09-17T04:49:42.051282+0000","last_clean":"2024-09-17T04:49:42.051282+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T04:49:42.051282+0000","last_undegraded":"2024-09-17T04:49:42.051282+0000","last_fullsized":"2024-09-17T04:49:42.051282+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:27.230604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536526+0000","last_change":"2024-09-17T04:49:41.055263+0000","last_active":"2024-09-17T04:49:42.536526+0000","last_peered":"2024-09-17T04:49:42.536526+0000","last_clean":"2024-09-17T04:49:42.536526+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T04:49:42.536526+0000","last_undegraded":"2024-09-17T04:49:42.536526+0000","last_fullsized":"2024-09-17T04:49:42.536526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670438+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T04:49:42.670438+0000","last_peered":"2024-09-17T04:49:42.670438+0000","last_clean":"2024-09-17T04:49:42.670438+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T04:49:42.670438+0000","last_undegraded":"2024-09-17T04:49:42.670438+0000","last_fullsized":"2024-09-17T04:49:42.670438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673048+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T04:49:42.673048+0000","last_peered":"2024-09-17T04:49:42.673048+0000","last_clean":"2024-09-17T04:49:42.673048+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T04:49:42.673048+0000","last_undegraded":"2024-09-17T04:49:42.673048+0000","last_fullsized":"2024-09-17T04:49:42.673048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670447+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T04:49:42.670447+0000","last_peered":"2024-09-17T04:49:42.670447+0000","last_clean":"2024-09-17T04:49:42.670447+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T04:49:42.670447+0000","last_undegraded":"2024-09-17T04:49:42.670447+0000","last_fullsized":"2024-09-17T04:49:42.670447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:24.586834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051128+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.051128+0000","last_peered":"2024-09-17T04:49:42.051128+0000","last_clean":"2024-09-17T04:49:42.051128+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T04:49:42.051128+0000","last_undegraded":"2024-09-17T04:49:42.051128+0000","last_fullsized":"2024-09-17T04:49:42.051128+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536163+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T04:49:42.536163+0000","last_peered":"2024-09-17T04:49:42.536163+0000","last_clean":"2024-09-17T04:49:42.536163+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T04:49:42.536163+0000","last_undegraded":"2024-09-17T04:49:42.536163+0000","last_fullsized":"2024-09-17T04:49:42.536163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052121+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T04:49:42.052121+0000","last_peered":"2024-09-17T04:49:42.052121+0000","last_clean":"2024-09-17T04:49:42.052121+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T04:49:42.052121+0000","last_undegraded":"2024-09-17T04:49:42.052121+0000","last_fullsized":"2024-09-17T04:49:42.052121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668467+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T04:49:42.668467+0000","last_peered":"2024-09-17T04:49:42.668467+0000","last_clean":"2024-09-17T04:49:42.668467+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T04:49:42.668467+0000","last_undegraded":"2024-09-17T04:49:42.668467+0000","last_fullsized":"2024-09-17T04:49:42.668467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536096+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T04:49:42.536096+0000","last_peered":"2024-09-17T04:49:42.536096+0000","last_clean":"2024-09-17T04:49:42.536096+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T04:49:42.536096+0000","last_undegraded":"2024-09-17T04:49:42.536096+0000","last_fullsized":"2024-09-17T04:49:42.536096+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042322+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T04:49:42.042322+0000","last_peered":"2024-09-17T04:49:42.042322+0000","last_clean":"2024-09-17T04:49:42.042322+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T04:49:42.042322+0000","last_undegraded":"2024-09-17T04:49:42.042322+0000","last_fullsized":"2024-09-17T04:49:42.042322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671200+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T04:49:42.671200+0000","last_peered":"2024-09-17T04:49:42.671200+0000","last_clean":"2024-09-17T04:49:42.671200+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T04:49:42.671200+0000","last_undegraded":"2024-09-17T04:49:42.671200+0000","last_fullsized":"2024-09-17T04:49:42.671200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670309+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T04:49:42.670309+0000","last_peered":"2024-09-17T04:49:42.670309+0000","last_clean":"2024-09-17T04:49:42.670309+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T04:49:42.670309+0000","last_undegraded":"2024-09-17T04:49:42.670309+0000","last_fullsized":"2024-09-17T04:49:42.670309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668969+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T04:49:42.668969+0000","last_peered":"2024-09-17T04:49:42.668969+0000","last_clean":"2024-09-17T04:49:42.668969+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T04:49:42.668969+0000","last_undegraded":"2024-09-17T04:49:42.668969+0000","last_fullsized":"2024-09-17T04:49:42.668969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054935+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T04:49:42.054935+0000","last_peered":"2024-09-17T04:49:42.054935+0000","last_clean":"2024-09-17T04:49:42.054935+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T04:49:42.054935+0000","last_undegraded":"2024-09-17T04:49:42.054935+0000","last_fullsized":"2024-09-17T04:49:42.054935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669059+0000","last_change":"2024-09-17T04:49:41.073709+0000","last_active":"2024-09-17T04:49:42.669059+0000","last_peered":"2024-09-17T04:49:42.669059+0000","last_clean":"2024-09-17T04:49:42.669059+0000","last_became_active":"2024-09-17T04:49:41.073330+0000","last_became_peered":"2024-09-17T04:49:41.073330+0000","last_unstale":"2024-09-17T04:49:42.669059+0000","last_undegraded":"2024-09-17T04:49:42.669059+0000","last_fullsized":"2024-09-17T04:49:42.669059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054811+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T04:49:42.054811+0000","last_peered":"2024-09-17T04:49:42.054811+0000","last_clean":"2024-09-17T04:49:42.054811+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T04:49:42.054811+0000","last_undegraded":"2024-09-17T04:49:42.054811+0000","last_fullsized":"2024-09-17T04:49:42.054811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041256+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T04:49:42.041256+0000","last_peered":"2024-09-17T04:49:42.041256+0000","last_clean":"2024-09-17T04:49:42.041256+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T04:49:42.041256+0000","last_undegraded":"2024-09-17T04:49:42.041256+0000","last_fullsized":"2024-09-17T04:49:42.041256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536669+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T04:49:42.536669+0000","last_peered":"2024-09-17T04:49:42.536669+0000","last_clean":"2024-09-17T04:49:42.536669+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T04:49:42.536669+0000","last_undegraded":"2024-09-17T04:49:42.536669+0000","last_fullsized":"2024-09-17T04:49:42.536669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534968+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T04:49:42.534968+0000","last_peered":"2024-09-17T04:49:42.534968+0000","last_clean":"2024-09-17T04:49:42.534968+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T04:49:42.534968+0000","last_undegraded":"2024-09-17T04:49:42.534968+0000","last_fullsized":"2024-09-17T04:49:42.534968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672019+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T04:49:42.672019+0000","last_peered":"2024-09-17T04:49:42.672019+0000","last_clean":"2024-09-17T04:49:42.672019+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T04:49:42.672019+0000","last_undegraded":"2024-09-17T04:49:42.672019+0000","last_fullsized":"2024-09-17T04:49:42.672019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668565+0000","last_change":"2024-09-17T04:49:41.045502+0000","last_active":"2024-09-17T04:49:42.668565+0000","last_peered":"2024-09-17T04:49:42.668565+0000","last_clean":"2024-09-17T04:49:42.668565+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T04:49:42.668565+0000","last_undegraded":"2024-09-17T04:49:42.668565+0000","last_fullsized":"2024-09-17T04:49:42.668565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:19.746356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041666+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T04:49:42.041666+0000","last_peered":"2024-09-17T04:49:42.041666+0000","last_clean":"2024-09-17T04:49:42.041666+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T04:49:42.041666+0000","last_undegraded":"2024-09-17T04:49:42.041666+0000","last_fullsized":"2024-09-17T04:49:42.041666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042310+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T04:49:42.042310+0000","last_peered":"2024-09-17T04:49:42.042310+0000","last_clean":"2024-09-17T04:49:42.042310+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T04:49:42.042310+0000","last_undegraded":"2024-09-17T04:49:42.042310+0000","last_fullsized":"2024-09-17T04:49:42.042310+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668045+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T04:49:42.668045+0000","last_peered":"2024-09-17T04:49:42.668045+0000","last_clean":"2024-09-17T04:49:42.668045+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T04:49:42.668045+0000","last_undegraded":"2024-09-17T04:49:42.668045+0000","last_fullsized":"2024-09-17T04:49:42.668045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054699+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T04:49:42.054699+0000","last_peered":"2024-09-17T04:49:42.054699+0000","last_clean":"2024-09-17T04:49:42.054699+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T04:49:42.054699+0000","last_undegraded":"2024-09-17T04:49:42.054699+0000","last_fullsized":"2024-09-17T04:49:42.054699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673038+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T04:49:42.673038+0000","last_peered":"2024-09-17T04:49:42.673038+0000","last_clean":"2024-09-17T04:49:42.673038+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T04:49:42.673038+0000","last_undegraded":"2024-09-17T04:49:42.673038+0000","last_fullsized":"2024-09-17T04:49:42.673038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669104+0000","last_change":"2024-09-17T04:49:41.038265+0000","last_active":"2024-09-17T04:49:42.669104+0000","last_peered":"2024-09-17T04:49:42.669104+0000","last_clean":"2024-09-17T04:49:42.669104+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T04:49:42.669104+0000","last_undegraded":"2024-09-17T04:49:42.669104+0000","last_fullsized":"2024-09-17T04:49:42.669104+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:41.638403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053416+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T04:49:42.053416+0000","last_peered":"2024-09-17T04:49:42.053416+0000","last_clean":"2024-09-17T04:49:42.053416+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T04:49:42.053416+0000","last_undegraded":"2024-09-17T04:49:42.053416+0000","last_fullsized":"2024-09-17T04:49:42.053416+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535001+0000","last_change":"2024-09-17T04:49:41.053443+0000","last_active":"2024-09-17T04:49:42.535001+0000","last_peered":"2024-09-17T04:49:42.535001+0000","last_clean":"2024-09-17T04:49:42.535001+0000","last_became_active":"2024-09-17T04:49:41.053090+0000","last_became_peered":"2024-09-17T04:49:41.053090+0000","last_unstale":"2024-09-17T04:49:42.535001+0000","last_undegraded":"2024-09-17T04:49:42.535001+0000","last_fullsized":"2024-09-17T04:49:42.535001+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671599+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T04:49:42.671599+0000","last_peered":"2024-09-17T04:49:42.671599+0000","last_clean":"2024-09-17T04:49:42.671599+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T04:49:42.671599+0000","last_undegraded":"2024-09-17T04:49:42.671599+0000","last_fullsized":"2024-09-17T04:49:42.671599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669669+0000","last_change":"2024-09-17T04:49:41.038072+0000","last_active":"2024-09-17T04:49:42.669669+0000","last_peered":"2024-09-17T04:49:42.669669+0000","last_clean":"2024-09-17T04:49:42.669669+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T04:49:42.669669+0000","last_undegraded":"2024-09-17T04:49:42.669669+0000","last_fullsized":"2024-09-17T04:49:42.669669+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043492+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T04:49:42.043492+0000","last_peered":"2024-09-17T04:49:42.043492+0000","last_clean":"2024-09-17T04:49:42.043492+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T04:49:42.043492+0000","last_undegraded":"2024-09-17T04:49:42.043492+0000","last_fullsized":"2024-09-17T04:49:42.043492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534462+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T04:49:42.534462+0000","last_peered":"2024-09-17T04:49:42.534462+0000","last_clean":"2024-09-17T04:49:42.534462+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T04:49:42.534462+0000","last_undegraded":"2024-09-17T04:49:42.534462+0000","last_fullsized":"2024-09-17T04:49:42.534462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669602+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T04:49:42.669602+0000","last_peered":"2024-09-17T04:49:42.669602+0000","last_clean":"2024-09-17T04:49:42.669602+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T04:49:42.669602+0000","last_undegraded":"2024-09-17T04:49:42.669602+0000","last_fullsized":"2024-09-17T04:49:42.669602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050316+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T04:49:42.050316+0000","last_peered":"2024-09-17T04:49:42.050316+0000","last_clean":"2024-09-17T04:49:42.050316+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T04:49:42.050316+0000","last_undegraded":"2024-09-17T04:49:42.050316+0000","last_fullsized":"2024-09-17T04:49:42.050316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042123+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T04:49:42.042123+0000","last_peered":"2024-09-17T04:49:42.042123+0000","last_clean":"2024-09-17T04:49:42.042123+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T04:49:42.042123+0000","last_undegraded":"2024-09-17T04:49:42.042123+0000","last_fullsized":"2024-09-17T04:49:42.042123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052378+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T04:49:42.052378+0000","last_peered":"2024-09-17T04:49:42.052378+0000","last_clean":"2024-09-17T04:49:42.052378+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T04:49:42.052378+0000","last_undegraded":"2024-09-17T04:49:42.052378+0000","last_fullsized":"2024-09-17T04:49:42.052378+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041588+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T04:49:42.041588+0000","last_peered":"2024-09-17T04:49:42.041588+0000","last_clean":"2024-09-17T04:49:42.041588+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T04:49:42.041588+0000","last_undegraded":"2024-09-17T04:49:42.041588+0000","last_fullsized":"2024-09-17T04:49:42.041588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674897+0000","last_change":"2024-09-17T04:49:41.067207+0000","last_active":"2024-09-17T04:49:42.674897+0000","last_peered":"2024-09-17T04:49:42.674897+0000","last_clean":"2024-09-17T04:49:42.674897+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T04:49:42.674897+0000","last_undegraded":"2024-09-17T04:49:42.674897+0000","last_fullsized":"2024-09-17T04:49:42.674897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:08.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667484+0000","last_change":"2024-09-17T04:49:31.911036+0000","last_active":"2024-09-17T04:49:42.667484+0000","last_peered":"2024-09-17T04:49:42.667484+0000","last_clean":"2024-09-17T04:49:42.667484+0000","last_became_active":"2024-09-17T04:49:31.910285+0000","last_became_peered":"2024-09-17T04:49:31.910285+0000","last_unstale":"2024-09-17T04:49:42.667484+0000","last_undegraded":"2024-09-17T04:49:42.667484+0000","last_fullsized":"2024-09-17T04:49:42.667484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673474+0000","last_change":"2024-09-17T04:49:41.046170+0000","last_active":"2024-09-17T04:49:42.673474+0000","last_peered":"2024-09-17T04:49:42.673474+0000","last_clean":"2024-09-17T04:49:42.673474+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T04:49:42.673474+0000","last_undegraded":"2024-09-17T04:49:42.673474+0000","last_fullsized":"2024-09-17T04:49:42.673474+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:01.683826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051528+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T04:49:42.051528+0000","last_peered":"2024-09-17T04:49:42.051528+0000","last_clean":"2024-09-17T04:49:42.051528+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T04:49:42.051528+0000","last_undegraded":"2024-09-17T04:49:42.051528+0000","last_fullsized":"2024-09-17T04:49:42.051528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041843+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T04:49:42.041843+0000","last_peered":"2024-09-17T04:49:42.041843+0000","last_clean":"2024-09-17T04:49:42.041843+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T04:49:42.041843+0000","last_undegraded":"2024-09-17T04:49:42.041843+0000","last_fullsized":"2024-09-17T04:49:42.041843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:25.574682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053942+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T04:49:42.053942+0000","last_peered":"2024-09-17T04:49:42.053942+0000","last_clean":"2024-09-17T04:49:42.053942+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T04:49:42.053942+0000","last_undegraded":"2024-09-17T04:49:42.053942+0000","last_fullsized":"2024-09-17T04:49:42.053942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048414+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T04:49:42.048414+0000","last_peered":"2024-09-17T04:49:42.048414+0000","last_clean":"2024-09-17T04:49:42.048414+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T04:49:42.048414+0000","last_undegraded":"2024-09-17T04:49:42.048414+0000","last_fullsized":"2024-09-17T04:49:42.048414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671218+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T04:49:42.671218+0000","last_peered":"2024-09-17T04:49:42.671218+0000","last_clean":"2024-09-17T04:49:42.671218+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T04:49:42.671218+0000","last_undegraded":"2024-09-17T04:49:42.671218+0000","last_fullsized":"2024-09-17T04:49:42.671218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053498+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T04:49:42.053498+0000","last_peered":"2024-09-17T04:49:42.053498+0000","last_clean":"2024-09-17T04:49:42.053498+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T04:49:42.053498+0000","last_undegraded":"2024-09-17T04:49:42.053498+0000","last_fullsized":"2024-09-17T04:49:42.053498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670119+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T04:49:42.670119+0000","last_peered":"2024-09-17T04:49:42.670119+0000","last_clean":"2024-09-17T04:49:42.670119+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T04:49:42.670119+0000","last_undegraded":"2024-09-17T04:49:42.670119+0000","last_fullsized":"2024-09-17T04:49:42.670119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668585+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T04:49:42.668585+0000","last_peered":"2024-09-17T04:49:42.668585+0000","last_clean":"2024-09-17T04:49:42.668585+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T04:49:42.668585+0000","last_undegraded":"2024-09-17T04:49:42.668585+0000","last_fullsized":"2024-09-17T04:49:42.668585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670988+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T04:49:42.670988+0000","last_peered":"2024-09-17T04:49:42.670988+0000","last_clean":"2024-09-17T04:49:42.670988+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T04:49:42.670988+0000","last_undegraded":"2024-09-17T04:49:42.670988+0000","last_fullsized":"2024-09-17T04:49:42.670988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671416+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.671416+0000","last_peered":"2024-09-17T04:49:42.671416+0000","last_clean":"2024-09-17T04:49:42.671416+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T04:49:42.671416+0000","last_undegraded":"2024-09-17T04:49:42.671416+0000","last_fullsized":"2024-09-17T04:49:42.671416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675766+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T04:49:42.675766+0000","last_peered":"2024-09-17T04:49:42.675766+0000","last_clean":"2024-09-17T04:49:42.675766+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T04:49:42.675766+0000","last_undegraded":"2024-09-17T04:49:42.675766+0000","last_fullsized":"2024-09-17T04:49:42.675766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671330+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T04:49:42.671330+0000","last_peered":"2024-09-17T04:49:42.671330+0000","last_clean":"2024-09-17T04:49:42.671330+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T04:49:42.671330+0000","last_undegraded":"2024-09-17T04:49:42.671330+0000","last_fullsized":"2024-09-17T04:49:42.671330+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667310+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T04:49:42.667310+0000","last_peered":"2024-09-17T04:49:42.667310+0000","last_clean":"2024-09-17T04:49:42.667310+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T04:49:42.667310+0000","last_undegraded":"2024-09-17T04:49:42.667310+0000","last_fullsized":"2024-09-17T04:49:42.667310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537133+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T04:49:42.537133+0000","last_peered":"2024-09-17T04:49:42.537133+0000","last_clean":"2024-09-17T04:49:42.537133+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T04:49:42.537133+0000","last_undegraded":"2024-09-17T04:49:42.537133+0000","last_fullsized":"2024-09-17T04:49:42.537133+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669347+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T04:49:42.669347+0000","last_peered":"2024-09-17T04:49:42.669347+0000","last_clean":"2024-09-17T04:49:42.669347+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T04:49:42.669347+0000","last_undegraded":"2024-09-17T04:49:42.669347+0000","last_fullsized":"2024-09-17T04:49:42.669347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050809+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T04:49:42.050809+0000","last_peered":"2024-09-17T04:49:42.050809+0000","last_clean":"2024-09-17T04:49:42.050809+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T04:49:42.050809+0000","last_undegraded":"2024-09-17T04:49:42.050809+0000","last_fullsized":"2024-09-17T04:49:42.050809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535865+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T04:49:42.535865+0000","last_peered":"2024-09-17T04:49:42.535865+0000","last_clean":"2024-09-17T04:49:42.535865+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T04:49:42.535865+0000","last_undegraded":"2024-09-17T04:49:42.535865+0000","last_fullsized":"2024-09-17T04:49:42.535865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673274+0000","last_change":"2024-09-17T04:49:41.041201+0000","last_active":"2024-09-17T04:49:42.673274+0000","last_peered":"2024-09-17T04:49:42.673274+0000","last_clean":"2024-09-17T04:49:42.673274+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T04:49:42.673274+0000","last_undegraded":"2024-09-17T04:49:42.673274+0000","last_fullsized":"2024-09-17T04:49:42.673274+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:02.588786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670272+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T04:49:42.670272+0000","last_peered":"2024-09-17T04:49:42.670272+0000","last_clean":"2024-09-17T04:49:42.670272+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T04:49:42.670272+0000","last_undegraded":"2024-09-17T04:49:42.670272+0000","last_fullsized":"2024-09-17T04:49:42.670272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667767+0000","last_change":"2024-09-17T04:49:41.061092+0000","last_active":"2024-09-17T04:49:42.667767+0000","last_peered":"2024-09-17T04:49:42.667767+0000","last_clean":"2024-09-17T04:49:42.667767+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T04:49:42.667767+0000","last_undegraded":"2024-09-17T04:49:42.667767+0000","last_fullsized":"2024-09-17T04:49:42.667767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:52.714614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536820+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T04:49:42.536820+0000","last_peered":"2024-09-17T04:49:42.536820+0000","last_clean":"2024-09-17T04:49:42.536820+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T04:49:42.536820+0000","last_undegraded":"2024-09-17T04:49:42.536820+0000","last_fullsized":"2024-09-17T04:49:42.536820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669397+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T04:49:42.669397+0000","last_peered":"2024-09-17T04:49:42.669397+0000","last_clean":"2024-09-17T04:49:42.669397+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T04:49:42.669397+0000","last_undegraded":"2024-09-17T04:49:42.669397+0000","last_fullsized":"2024-09-17T04:49:42.669397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671170+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T04:49:42.671170+0000","last_peered":"2024-09-17T04:49:42.671170+0000","last_clean":"2024-09-17T04:49:42.671170+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T04:49:42.671170+0000","last_undegraded":"2024-09-17T04:49:42.671170+0000","last_fullsized":"2024-09-17T04:49:42.671170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536018+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T04:49:42.536018+0000","last_peered":"2024-09-17T04:49:42.536018+0000","last_clean":"2024-09-17T04:49:42.536018+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T04:49:42.536018+0000","last_undegraded":"2024-09-17T04:49:42.536018+0000","last_fullsized":"2024-09-17T04:49:42.536018+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671602+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T04:49:42.671602+0000","last_peered":"2024-09-17T04:49:42.671602+0000","last_clean":"2024-09-17T04:49:42.671602+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T04:49:42.671602+0000","last_undegraded":"2024-09-17T04:49:42.671602+0000","last_fullsized":"2024-09-17T04:49:42.671602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534962+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T04:49:42.534962+0000","last_peered":"2024-09-17T04:49:42.534962+0000","last_clean":"2024-09-17T04:49:42.534962+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T04:49:42.534962+0000","last_undegraded":"2024-09-17T04:49:42.534962+0000","last_fullsized":"2024-09-17T04:49:42.534962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671825+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T04:49:42.671825+0000","last_peered":"2024-09-17T04:49:42.671825+0000","last_clean":"2024-09-17T04:49:42.671825+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T04:49:42.671825+0000","last_undegraded":"2024-09-17T04:49:42.671825+0000","last_fullsized":"2024-09-17T04:49:42.671825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2851,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509453,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509453,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509453,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":108978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509453,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509453,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":106706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509454,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509454,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:50:09.984 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:50:10.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.167+0000 7f443a6f6640 1 -- 172.21.15.150:0/2197279299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44340c3db0 msgr2=0x7f44340ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.167+0000 7f443a6f6640 1 --2- 172.21.15.150:0/2197279299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44340c3db0 0x7f44340ad2a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f443406f7a0 tx=0x7f442c01cba0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.167+0000 7f443a6f6640 1 -- 172.21.15.150:0/2197279299 shutdown_connections 2024-09-17T04:50:10.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.167+0000 7f443a6f6640 1 --2- 172.21.15.150:0/2197279299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44340c41c0 0x7f44340ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.167+0000 7f443a6f6640 1 --2- 172.21.15.150:0/2197279299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44340c3db0 0x7f44340ad2a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.167+0000 7f443a6f6640 1 -- 172.21.15.150:0/2197279299 >> 172.21.15.150:0/2197279299 conn(0x7f44340c4a00 msgr2=0x7f44340c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:10.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.167+0000 7f443a6f6640 1 -- 172.21.15.150:0/2197279299 shutdown_connections 2024-09-17T04:50:10.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.167+0000 7f443a6f6640 1 -- 172.21.15.150:0/2197279299 wait complete. 2024-09-17T04:50:10.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.167+0000 7f443a6f6640 1 Processor -- start 2024-09-17T04:50:10.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.167+0000 7f443a6f6640 1 -- start start 2024-09-17T04:50:10.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f443a6f6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f443406fab0 0x7f4434148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:10.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f443a6f6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44340c3db0 0x7f4434158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:10.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f443a6f6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44340c41c0 0x7f443414ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:10.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f443a6f6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4434070750 con 0x7f44340c41c0 2024-09-17T04:50:10.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f443a6f6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f44340705d0 con 0x7f443406fab0 2024-09-17T04:50:10.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f443a6f6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f44340a75f0 con 0x7f44340c3db0 2024-09-17T04:50:10.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4438c93640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44340c41c0 0x7f443414ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:10.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4438c93640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44340c41c0 0x7f443414ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37588/0 (socket says 172.21.15.150:37588) 2024-09-17T04:50:10.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4438c93640 1 -- 172.21.15.150:0/2924256849 learned_addr learned my addr 172.21.15.150:0/2924256849 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:10.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4433fff640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f443406fab0 0x7f4434148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:10.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f44337fe640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44340c3db0 0x7f4434158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:10.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4438c93640 1 -- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44340c3db0 msgr2=0x7f4434158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4438c93640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44340c3db0 0x7f4434158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4438c93640 1 -- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f443406fab0 msgr2=0x7f4434148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4438c93640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f443406fab0 0x7f4434148410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4438c93640 1 -- 172.21.15.150:0/2924256849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44340c23c0 con 0x7f44340c41c0 2024-09-17T04:50:10.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f44337fe640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44340c3db0 0x7f4434158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:50:10.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4438c93640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44340c41c0 0x7f443414ed00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f442400c3f0 tx=0x7f442400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:10.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4430ff9640 1 -- 172.21.15.150:0/2924256849 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424017070 con 0x7f44340c41c0 2024-09-17T04:50:10.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f443a6f6640 1 -- 172.21.15.150:0/2924256849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4434152da0 con 0x7f44340c41c0 2024-09-17T04:50:10.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4430ff9640 1 -- 172.21.15.150:0/2924256849 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4424003600 con 0x7f44340c41c0 2024-09-17T04:50:10.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4430ff9640 1 -- 172.21.15.150:0/2924256849 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424003a50 con 0x7f44340c41c0 2024-09-17T04:50:10.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f443a6f6640 1 -- 172.21.15.150:0/2924256849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4434157210 con 0x7f44340c41c0 2024-09-17T04:50:10.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f443a6f6640 1 -- 172.21.15.150:0/2924256849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f443406fec0 con 0x7f44340c41c0 2024-09-17T04:50:10.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.171+0000 7f4430ff9640 1 -- 172.21.15.150:0/2924256849 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f442401a070 con 0x7f44340c41c0 2024-09-17T04:50:10.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.175+0000 7f4430ff9640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4414073200 0x7f4414075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:10.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.175+0000 7f4430ff9640 1 -- 172.21.15.150:0/2924256849 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f4424093bf0 con 0x7f44340c41c0 2024-09-17T04:50:10.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.175+0000 7f4433fff640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4414073200 0x7f4414075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:10.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.175+0000 7f4433fff640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4414073200 0x7f4414075710 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f442c0af000 tx=0x7f442c01b3e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:10.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.175+0000 7f4430ff9640 1 -- 172.21.15.150:0/2924256849 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4424021020 con 0x7f44340c41c0 2024-09-17T04:50:10.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.343+0000 7f443a6f6640 1 -- 172.21.15.150:0/2924256849 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4434148950 con 0x7f4414073200 2024-09-17T04:50:10.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.367+0000 7f4430ff9640 1 -- 172.21.15.150:0/2924256849 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490314 (secure 0 0 0) 0x7f442c004a50 con 0x7f4414073200 2024-09-17T04:50:10.374 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:50:10.377 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T04:50:10.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.371+0000 7f443a6f6640 1 -- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4414073200 msgr2=0x7f4414075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.371+0000 7f443a6f6640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4414073200 0x7f4414075710 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f442c0af000 tx=0x7f442c01b3e0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.371+0000 7f443a6f6640 1 -- 172.21.15.150:0/2924256849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44340c41c0 msgr2=0x7f443414ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.371+0000 7f443a6f6640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44340c41c0 0x7f443414ed00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f442400c3f0 tx=0x7f442400c8b0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.371+0000 7f443a6f6640 1 -- 172.21.15.150:0/2924256849 shutdown_connections 2024-09-17T04:50:10.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.371+0000 7f443a6f6640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4414073200 0x7f4414075710 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.371+0000 7f443a6f6640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44340c41c0 0x7f443414ed00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.371+0000 7f443a6f6640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44340c3db0 0x7f4434158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.371+0000 7f443a6f6640 1 --2- 172.21.15.150:0/2924256849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f443406fab0 0x7f4434148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.371+0000 7f443a6f6640 1 -- 172.21.15.150:0/2924256849 >> 172.21.15.150:0/2924256849 conn(0x7f44340c4a00 msgr2=0x7f44340c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:10.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.371+0000 7f443a6f6640 1 -- 172.21.15.150:0/2924256849 shutdown_connections 2024-09-17T04:50:10.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.371+0000 7f443a6f6640 1 -- 172.21.15.150:0/2924256849 wait complete. 2024-09-17T04:50:10.395 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-17T04:50:08.959982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595629,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":193,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223792,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750928,"statfs":{"total":773094113280,"available":772864950272,"internally_reserved":0,"allocated":3481600,"data_stored":2028210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":441,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006952"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671077+0000","last_change":"2024-09-17T04:49:41.025271+0000","last_active":"2024-09-17T04:49:42.671077+0000","last_peered":"2024-09-17T04:49:42.671077+0000","last_clean":"2024-09-17T04:49:42.671077+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T04:49:42.671077+0000","last_undegraded":"2024-09-17T04:49:42.671077+0000","last_fullsized":"2024-09-17T04:49:42.671077+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:39.715602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054640+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T04:49:42.054640+0000","last_peered":"2024-09-17T04:49:42.054640+0000","last_clean":"2024-09-17T04:49:42.054640+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T04:49:42.054640+0000","last_undegraded":"2024-09-17T04:49:42.054640+0000","last_fullsized":"2024-09-17T04:49:42.054640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671706+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T04:49:42.671706+0000","last_peered":"2024-09-17T04:49:42.671706+0000","last_clean":"2024-09-17T04:49:42.671706+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T04:49:42.671706+0000","last_undegraded":"2024-09-17T04:49:42.671706+0000","last_fullsized":"2024-09-17T04:49:42.671706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668736+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T04:49:42.668736+0000","last_peered":"2024-09-17T04:49:42.668736+0000","last_clean":"2024-09-17T04:49:42.668736+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T04:49:42.668736+0000","last_undegraded":"2024-09-17T04:49:42.668736+0000","last_fullsized":"2024-09-17T04:49:42.668736+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:38.417840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047972+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T04:49:42.047972+0000","last_peered":"2024-09-17T04:49:42.047972+0000","last_clean":"2024-09-17T04:49:42.047972+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T04:49:42.047972+0000","last_undegraded":"2024-09-17T04:49:42.047972+0000","last_fullsized":"2024-09-17T04:49:42.047972+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672300+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T04:49:42.672300+0000","last_peered":"2024-09-17T04:49:42.672300+0000","last_clean":"2024-09-17T04:49:42.672300+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T04:49:42.672300+0000","last_undegraded":"2024-09-17T04:49:42.672300+0000","last_fullsized":"2024-09-17T04:49:42.672300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672014+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T04:49:42.672014+0000","last_peered":"2024-09-17T04:49:42.672014+0000","last_clean":"2024-09-17T04:49:42.672014+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T04:49:42.672014+0000","last_undegraded":"2024-09-17T04:49:42.672014+0000","last_fullsized":"2024-09-17T04:49:42.672014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535134+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T04:49:42.535134+0000","last_peered":"2024-09-17T04:49:42.535134+0000","last_clean":"2024-09-17T04:49:42.535134+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T04:49:42.535134+0000","last_undegraded":"2024-09-17T04:49:42.535134+0000","last_fullsized":"2024-09-17T04:49:42.535134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672206+0000","last_change":"2024-09-17T04:49:41.067317+0000","last_active":"2024-09-17T04:49:42.672206+0000","last_peered":"2024-09-17T04:49:42.672206+0000","last_clean":"2024-09-17T04:49:42.672206+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T04:49:42.672206+0000","last_undegraded":"2024-09-17T04:49:42.672206+0000","last_fullsized":"2024-09-17T04:49:42.672206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:37.637477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670707+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T04:49:42.670707+0000","last_peered":"2024-09-17T04:49:42.670707+0000","last_clean":"2024-09-17T04:49:42.670707+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T04:49:42.670707+0000","last_undegraded":"2024-09-17T04:49:42.670707+0000","last_fullsized":"2024-09-17T04:49:42.670707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674274+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T04:49:42.674274+0000","last_peered":"2024-09-17T04:49:42.674274+0000","last_clean":"2024-09-17T04:49:42.674274+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T04:49:42.674274+0000","last_undegraded":"2024-09-17T04:49:42.674274+0000","last_fullsized":"2024-09-17T04:49:42.674274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535118+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T04:49:42.535118+0000","last_peered":"2024-09-17T04:49:42.535118+0000","last_clean":"2024-09-17T04:49:42.535118+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T04:49:42.535118+0000","last_undegraded":"2024-09-17T04:49:42.535118+0000","last_fullsized":"2024-09-17T04:49:42.535118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535259+0000","last_change":"2024-09-17T04:49:41.043531+0000","last_active":"2024-09-17T04:49:42.535259+0000","last_peered":"2024-09-17T04:49:42.535259+0000","last_clean":"2024-09-17T04:49:42.535259+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T04:49:42.535259+0000","last_undegraded":"2024-09-17T04:49:42.535259+0000","last_fullsized":"2024-09-17T04:49:42.535259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672162+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T04:49:42.672162+0000","last_peered":"2024-09-17T04:49:42.672162+0000","last_clean":"2024-09-17T04:49:42.672162+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T04:49:42.672162+0000","last_undegraded":"2024-09-17T04:49:42.672162+0000","last_fullsized":"2024-09-17T04:49:42.672162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042945+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T04:49:42.042945+0000","last_peered":"2024-09-17T04:49:42.042945+0000","last_clean":"2024-09-17T04:49:42.042945+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T04:49:42.042945+0000","last_undegraded":"2024-09-17T04:49:42.042945+0000","last_fullsized":"2024-09-17T04:49:42.042945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535003+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T04:49:42.535003+0000","last_peered":"2024-09-17T04:49:42.535003+0000","last_clean":"2024-09-17T04:49:42.535003+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T04:49:42.535003+0000","last_undegraded":"2024-09-17T04:49:42.535003+0000","last_fullsized":"2024-09-17T04:49:42.535003+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:16.854365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669588+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T04:49:42.669588+0000","last_peered":"2024-09-17T04:49:42.669588+0000","last_clean":"2024-09-17T04:49:42.669588+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T04:49:42.669588+0000","last_undegraded":"2024-09-17T04:49:42.669588+0000","last_fullsized":"2024-09-17T04:49:42.669588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042526+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T04:49:42.042526+0000","last_peered":"2024-09-17T04:49:42.042526+0000","last_clean":"2024-09-17T04:49:42.042526+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T04:49:42.042526+0000","last_undegraded":"2024-09-17T04:49:42.042526+0000","last_fullsized":"2024-09-17T04:49:42.042526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041407+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T04:49:42.041407+0000","last_peered":"2024-09-17T04:49:42.041407+0000","last_clean":"2024-09-17T04:49:42.041407+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T04:49:42.041407+0000","last_undegraded":"2024-09-17T04:49:42.041407+0000","last_fullsized":"2024-09-17T04:49:42.041407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042319+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T04:49:42.042319+0000","last_peered":"2024-09-17T04:49:42.042319+0000","last_clean":"2024-09-17T04:49:42.042319+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T04:49:42.042319+0000","last_undegraded":"2024-09-17T04:49:42.042319+0000","last_fullsized":"2024-09-17T04:49:42.042319+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:28.365399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669339+0000","last_change":"2024-09-17T04:49:41.038415+0000","last_active":"2024-09-17T04:49:42.669339+0000","last_peered":"2024-09-17T04:49:42.669339+0000","last_clean":"2024-09-17T04:49:42.669339+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T04:49:42.669339+0000","last_undegraded":"2024-09-17T04:49:42.669339+0000","last_fullsized":"2024-09-17T04:49:42.669339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:17.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668616+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T04:49:42.668616+0000","last_peered":"2024-09-17T04:49:42.668616+0000","last_clean":"2024-09-17T04:49:42.668616+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T04:49:42.668616+0000","last_undegraded":"2024-09-17T04:49:42.668616+0000","last_fullsized":"2024-09-17T04:49:42.668616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052248+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T04:49:42.052248+0000","last_peered":"2024-09-17T04:49:42.052248+0000","last_clean":"2024-09-17T04:49:42.052248+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T04:49:42.052248+0000","last_undegraded":"2024-09-17T04:49:42.052248+0000","last_fullsized":"2024-09-17T04:49:42.052248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671586+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T04:49:42.671586+0000","last_peered":"2024-09-17T04:49:42.671586+0000","last_clean":"2024-09-17T04:49:42.671586+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T04:49:42.671586+0000","last_undegraded":"2024-09-17T04:49:42.671586+0000","last_fullsized":"2024-09-17T04:49:42.671586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:17.424265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042028+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T04:49:42.042028+0000","last_peered":"2024-09-17T04:49:42.042028+0000","last_clean":"2024-09-17T04:49:42.042028+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T04:49:42.042028+0000","last_undegraded":"2024-09-17T04:49:42.042028+0000","last_fullsized":"2024-09-17T04:49:42.042028+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:06.516432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048455+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T04:49:42.048455+0000","last_peered":"2024-09-17T04:49:42.048455+0000","last_clean":"2024-09-17T04:49:42.048455+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T04:49:42.048455+0000","last_undegraded":"2024-09-17T04:49:42.048455+0000","last_fullsized":"2024-09-17T04:49:42.048455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048500+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T04:49:42.048500+0000","last_peered":"2024-09-17T04:49:42.048500+0000","last_clean":"2024-09-17T04:49:42.048500+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T04:49:42.048500+0000","last_undegraded":"2024-09-17T04:49:42.048500+0000","last_fullsized":"2024-09-17T04:49:42.048500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668827+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T04:49:42.668827+0000","last_peered":"2024-09-17T04:49:42.668827+0000","last_clean":"2024-09-17T04:49:42.668827+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T04:49:42.668827+0000","last_undegraded":"2024-09-17T04:49:42.668827+0000","last_fullsized":"2024-09-17T04:49:42.668827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:00.010365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675406+0000","last_change":"2024-09-17T04:49:41.036658+0000","last_active":"2024-09-17T04:49:42.675406+0000","last_peered":"2024-09-17T04:49:42.675406+0000","last_clean":"2024-09-17T04:49:42.675406+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T04:49:42.675406+0000","last_undegraded":"2024-09-17T04:49:42.675406+0000","last_fullsized":"2024-09-17T04:49:42.675406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042824+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T04:49:42.042824+0000","last_peered":"2024-09-17T04:49:42.042824+0000","last_clean":"2024-09-17T04:49:42.042824+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T04:49:42.042824+0000","last_undegraded":"2024-09-17T04:49:42.042824+0000","last_fullsized":"2024-09-17T04:49:42.042824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668829+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T04:49:42.668829+0000","last_peered":"2024-09-17T04:49:42.668829+0000","last_clean":"2024-09-17T04:49:42.668829+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T04:49:42.668829+0000","last_undegraded":"2024-09-17T04:49:42.668829+0000","last_fullsized":"2024-09-17T04:49:42.668829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536259+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T04:49:42.536259+0000","last_peered":"2024-09-17T04:49:42.536259+0000","last_clean":"2024-09-17T04:49:42.536259+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T04:49:42.536259+0000","last_undegraded":"2024-09-17T04:49:42.536259+0000","last_fullsized":"2024-09-17T04:49:42.536259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048942+0000","last_change":"2024-09-17T04:49:41.050170+0000","last_active":"2024-09-17T04:49:42.048942+0000","last_peered":"2024-09-17T04:49:42.048942+0000","last_clean":"2024-09-17T04:49:42.048942+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T04:49:42.048942+0000","last_undegraded":"2024-09-17T04:49:42.048942+0000","last_fullsized":"2024-09-17T04:49:42.048942+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051929+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T04:49:42.051929+0000","last_peered":"2024-09-17T04:49:42.051929+0000","last_clean":"2024-09-17T04:49:42.051929+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T04:49:42.051929+0000","last_undegraded":"2024-09-17T04:49:42.051929+0000","last_fullsized":"2024-09-17T04:49:42.051929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042185+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T04:49:42.042185+0000","last_peered":"2024-09-17T04:49:42.042185+0000","last_clean":"2024-09-17T04:49:42.042185+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T04:49:42.042185+0000","last_undegraded":"2024-09-17T04:49:42.042185+0000","last_fullsized":"2024-09-17T04:49:42.042185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535553+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T04:49:42.535553+0000","last_peered":"2024-09-17T04:49:42.535553+0000","last_clean":"2024-09-17T04:49:42.535553+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T04:49:42.535553+0000","last_undegraded":"2024-09-17T04:49:42.535553+0000","last_fullsized":"2024-09-17T04:49:42.535553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041211+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T04:49:42.041211+0000","last_peered":"2024-09-17T04:49:42.041211+0000","last_clean":"2024-09-17T04:49:42.041211+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T04:49:42.041211+0000","last_undegraded":"2024-09-17T04:49:42.041211+0000","last_fullsized":"2024-09-17T04:49:42.041211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054336+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T04:49:42.054336+0000","last_peered":"2024-09-17T04:49:42.054336+0000","last_clean":"2024-09-17T04:49:42.054336+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T04:49:42.054336+0000","last_undegraded":"2024-09-17T04:49:42.054336+0000","last_fullsized":"2024-09-17T04:49:42.054336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052826+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T04:49:42.052826+0000","last_peered":"2024-09-17T04:49:42.052826+0000","last_clean":"2024-09-17T04:49:42.052826+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T04:49:42.052826+0000","last_undegraded":"2024-09-17T04:49:42.052826+0000","last_fullsized":"2024-09-17T04:49:42.052826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048145+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T04:49:42.048145+0000","last_peered":"2024-09-17T04:49:42.048145+0000","last_clean":"2024-09-17T04:49:42.048145+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T04:49:42.048145+0000","last_undegraded":"2024-09-17T04:49:42.048145+0000","last_fullsized":"2024-09-17T04:49:42.048145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669693+0000","last_change":"2024-09-17T04:49:41.042314+0000","last_active":"2024-09-17T04:49:42.669693+0000","last_peered":"2024-09-17T04:49:42.669693+0000","last_clean":"2024-09-17T04:49:42.669693+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T04:49:42.669693+0000","last_undegraded":"2024-09-17T04:49:42.669693+0000","last_fullsized":"2024-09-17T04:49:42.669693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055299+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T04:49:42.055299+0000","last_peered":"2024-09-17T04:49:42.055299+0000","last_clean":"2024-09-17T04:49:42.055299+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T04:49:42.055299+0000","last_undegraded":"2024-09-17T04:49:42.055299+0000","last_fullsized":"2024-09-17T04:49:42.055299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043663+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T04:49:42.043663+0000","last_peered":"2024-09-17T04:49:42.043663+0000","last_clean":"2024-09-17T04:49:42.043663+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T04:49:42.043663+0000","last_undegraded":"2024-09-17T04:49:42.043663+0000","last_fullsized":"2024-09-17T04:49:42.043663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053169+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T04:49:42.053169+0000","last_peered":"2024-09-17T04:49:42.053169+0000","last_clean":"2024-09-17T04:49:42.053169+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T04:49:42.053169+0000","last_undegraded":"2024-09-17T04:49:42.053169+0000","last_fullsized":"2024-09-17T04:49:42.053169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052959+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T04:49:42.052959+0000","last_peered":"2024-09-17T04:49:42.052959+0000","last_clean":"2024-09-17T04:49:42.052959+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T04:49:42.052959+0000","last_undegraded":"2024-09-17T04:49:42.052959+0000","last_fullsized":"2024-09-17T04:49:42.052959+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535986+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T04:49:42.535986+0000","last_peered":"2024-09-17T04:49:42.535986+0000","last_clean":"2024-09-17T04:49:42.535986+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T04:49:42.535986+0000","last_undegraded":"2024-09-17T04:49:42.535986+0000","last_fullsized":"2024-09-17T04:49:42.535986+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670900+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T04:49:42.670900+0000","last_peered":"2024-09-17T04:49:42.670900+0000","last_clean":"2024-09-17T04:49:42.670900+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T04:49:42.670900+0000","last_undegraded":"2024-09-17T04:49:42.670900+0000","last_fullsized":"2024-09-17T04:49:42.670900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050191+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T04:49:42.050191+0000","last_peered":"2024-09-17T04:49:42.050191+0000","last_clean":"2024-09-17T04:49:42.050191+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T04:49:42.050191+0000","last_undegraded":"2024-09-17T04:49:42.050191+0000","last_fullsized":"2024-09-17T04:49:42.050191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670215+0000","last_change":"2024-09-17T04:49:41.069177+0000","last_active":"2024-09-17T04:49:42.670215+0000","last_peered":"2024-09-17T04:49:42.670215+0000","last_clean":"2024-09-17T04:49:42.670215+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T04:49:42.670215+0000","last_undegraded":"2024-09-17T04:49:42.670215+0000","last_fullsized":"2024-09-17T04:49:42.670215+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:16.855206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670381+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T04:49:42.670381+0000","last_peered":"2024-09-17T04:49:42.670381+0000","last_clean":"2024-09-17T04:49:42.670381+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T04:49:42.670381+0000","last_undegraded":"2024-09-17T04:49:42.670381+0000","last_fullsized":"2024-09-17T04:49:42.670381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042706+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T04:49:42.042706+0000","last_peered":"2024-09-17T04:49:42.042706+0000","last_clean":"2024-09-17T04:49:42.042706+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T04:49:42.042706+0000","last_undegraded":"2024-09-17T04:49:42.042706+0000","last_fullsized":"2024-09-17T04:49:42.042706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054900+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T04:49:42.054900+0000","last_peered":"2024-09-17T04:49:42.054900+0000","last_clean":"2024-09-17T04:49:42.054900+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T04:49:42.054900+0000","last_undegraded":"2024-09-17T04:49:42.054900+0000","last_fullsized":"2024-09-17T04:49:42.054900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669308+0000","last_change":"2024-09-17T04:49:41.068537+0000","last_active":"2024-09-17T04:49:42.669308+0000","last_peered":"2024-09-17T04:49:42.669308+0000","last_clean":"2024-09-17T04:49:42.669308+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T04:49:42.669308+0000","last_undegraded":"2024-09-17T04:49:42.669308+0000","last_fullsized":"2024-09-17T04:49:42.669308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667397+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T04:49:42.667397+0000","last_peered":"2024-09-17T04:49:42.667397+0000","last_clean":"2024-09-17T04:49:42.667397+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T04:49:42.667397+0000","last_undegraded":"2024-09-17T04:49:42.667397+0000","last_fullsized":"2024-09-17T04:49:42.667397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668940+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T04:49:42.668940+0000","last_peered":"2024-09-17T04:49:42.668940+0000","last_clean":"2024-09-17T04:49:42.668940+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T04:49:42.668940+0000","last_undegraded":"2024-09-17T04:49:42.668940+0000","last_fullsized":"2024-09-17T04:49:42.668940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054921+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T04:49:42.054921+0000","last_peered":"2024-09-17T04:49:42.054921+0000","last_clean":"2024-09-17T04:49:42.054921+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T04:49:42.054921+0000","last_undegraded":"2024-09-17T04:49:42.054921+0000","last_fullsized":"2024-09-17T04:49:42.054921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041762+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T04:49:42.041762+0000","last_peered":"2024-09-17T04:49:42.041762+0000","last_clean":"2024-09-17T04:49:42.041762+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T04:49:42.041762+0000","last_undegraded":"2024-09-17T04:49:42.041762+0000","last_fullsized":"2024-09-17T04:49:42.041762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050299+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T04:49:42.050299+0000","last_peered":"2024-09-17T04:49:42.050299+0000","last_clean":"2024-09-17T04:49:42.050299+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T04:49:42.050299+0000","last_undegraded":"2024-09-17T04:49:42.050299+0000","last_fullsized":"2024-09-17T04:49:42.050299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041771+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T04:49:42.041771+0000","last_peered":"2024-09-17T04:49:42.041771+0000","last_clean":"2024-09-17T04:49:42.041771+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T04:49:42.041771+0000","last_undegraded":"2024-09-17T04:49:42.041771+0000","last_fullsized":"2024-09-17T04:49:42.041771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672234+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T04:49:42.672234+0000","last_peered":"2024-09-17T04:49:42.672234+0000","last_clean":"2024-09-17T04:49:42.672234+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T04:49:42.672234+0000","last_undegraded":"2024-09-17T04:49:42.672234+0000","last_fullsized":"2024-09-17T04:49:42.672234+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671183+0000","last_change":"2024-09-17T04:49:41.052564+0000","last_active":"2024-09-17T04:49:42.671183+0000","last_peered":"2024-09-17T04:49:42.671183+0000","last_clean":"2024-09-17T04:49:42.671183+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T04:49:42.671183+0000","last_undegraded":"2024-09-17T04:49:42.671183+0000","last_fullsized":"2024-09-17T04:49:42.671183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049438+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T04:49:42.049438+0000","last_peered":"2024-09-17T04:49:42.049438+0000","last_clean":"2024-09-17T04:49:42.049438+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T04:49:42.049438+0000","last_undegraded":"2024-09-17T04:49:42.049438+0000","last_fullsized":"2024-09-17T04:49:42.049438+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041458+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T04:49:42.041458+0000","last_peered":"2024-09-17T04:49:42.041458+0000","last_clean":"2024-09-17T04:49:42.041458+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T04:49:42.041458+0000","last_undegraded":"2024-09-17T04:49:42.041458+0000","last_fullsized":"2024-09-17T04:49:42.041458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671735+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T04:49:42.671735+0000","last_peered":"2024-09-17T04:49:42.671735+0000","last_clean":"2024-09-17T04:49:42.671735+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T04:49:42.671735+0000","last_undegraded":"2024-09-17T04:49:42.671735+0000","last_fullsized":"2024-09-17T04:49:42.671735+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043078+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T04:49:42.043078+0000","last_peered":"2024-09-17T04:49:42.043078+0000","last_clean":"2024-09-17T04:49:42.043078+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T04:49:42.043078+0000","last_undegraded":"2024-09-17T04:49:42.043078+0000","last_fullsized":"2024-09-17T04:49:42.043078+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:21.202398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669578+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T04:49:42.669578+0000","last_peered":"2024-09-17T04:49:42.669578+0000","last_clean":"2024-09-17T04:49:42.669578+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T04:49:42.669578+0000","last_undegraded":"2024-09-17T04:49:42.669578+0000","last_fullsized":"2024-09-17T04:49:42.669578+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:48.128609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043280+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T04:49:42.043280+0000","last_peered":"2024-09-17T04:49:42.043280+0000","last_clean":"2024-09-17T04:49:42.043280+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T04:49:42.043280+0000","last_undegraded":"2024-09-17T04:49:42.043280+0000","last_fullsized":"2024-09-17T04:49:42.043280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055292+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T04:49:42.055292+0000","last_peered":"2024-09-17T04:49:42.055292+0000","last_clean":"2024-09-17T04:49:42.055292+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T04:49:42.055292+0000","last_undegraded":"2024-09-17T04:49:42.055292+0000","last_fullsized":"2024-09-17T04:49:42.055292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536164+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T04:49:42.536164+0000","last_peered":"2024-09-17T04:49:42.536164+0000","last_clean":"2024-09-17T04:49:42.536164+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T04:49:42.536164+0000","last_undegraded":"2024-09-17T04:49:42.536164+0000","last_fullsized":"2024-09-17T04:49:42.536164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050257+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T04:49:42.050257+0000","last_peered":"2024-09-17T04:49:42.050257+0000","last_clean":"2024-09-17T04:49:42.050257+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T04:49:42.050257+0000","last_undegraded":"2024-09-17T04:49:42.050257+0000","last_fullsized":"2024-09-17T04:49:42.050257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671927+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T04:49:42.671927+0000","last_peered":"2024-09-17T04:49:42.671927+0000","last_clean":"2024-09-17T04:49:42.671927+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T04:49:42.671927+0000","last_undegraded":"2024-09-17T04:49:42.671927+0000","last_fullsized":"2024-09-17T04:49:42.671927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671411+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T04:49:42.671411+0000","last_peered":"2024-09-17T04:49:42.671411+0000","last_clean":"2024-09-17T04:49:42.671411+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T04:49:42.671411+0000","last_undegraded":"2024-09-17T04:49:42.671411+0000","last_fullsized":"2024-09-17T04:49:42.671411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670087+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T04:49:42.670087+0000","last_peered":"2024-09-17T04:49:42.670087+0000","last_clean":"2024-09-17T04:49:42.670087+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T04:49:42.670087+0000","last_undegraded":"2024-09-17T04:49:42.670087+0000","last_fullsized":"2024-09-17T04:49:42.670087+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:18.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672287+0000","last_change":"2024-09-17T04:49:41.031036+0000","last_active":"2024-09-17T04:49:42.672287+0000","last_peered":"2024-09-17T04:49:42.672287+0000","last_clean":"2024-09-17T04:49:42.672287+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T04:49:42.672287+0000","last_undegraded":"2024-09-17T04:49:42.672287+0000","last_fullsized":"2024-09-17T04:49:42.672287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:48.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672111+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T04:49:42.672111+0000","last_peered":"2024-09-17T04:49:42.672111+0000","last_clean":"2024-09-17T04:49:42.672111+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T04:49:42.672111+0000","last_undegraded":"2024-09-17T04:49:42.672111+0000","last_fullsized":"2024-09-17T04:49:42.672111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052688+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T04:49:42.052688+0000","last_peered":"2024-09-17T04:49:42.052688+0000","last_clean":"2024-09-17T04:49:42.052688+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T04:49:42.052688+0000","last_undegraded":"2024-09-17T04:49:42.052688+0000","last_fullsized":"2024-09-17T04:49:42.052688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041095+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T04:49:42.041095+0000","last_peered":"2024-09-17T04:49:42.041095+0000","last_clean":"2024-09-17T04:49:42.041095+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T04:49:42.041095+0000","last_undegraded":"2024-09-17T04:49:42.041095+0000","last_fullsized":"2024-09-17T04:49:42.041095+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048488+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T04:49:42.048488+0000","last_peered":"2024-09-17T04:49:42.048488+0000","last_clean":"2024-09-17T04:49:42.048488+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T04:49:42.048488+0000","last_undegraded":"2024-09-17T04:49:42.048488+0000","last_fullsized":"2024-09-17T04:49:42.048488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536967+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T04:49:42.536967+0000","last_peered":"2024-09-17T04:49:42.536967+0000","last_clean":"2024-09-17T04:49:42.536967+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T04:49:42.536967+0000","last_undegraded":"2024-09-17T04:49:42.536967+0000","last_fullsized":"2024-09-17T04:49:42.536967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672404+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T04:49:42.672404+0000","last_peered":"2024-09-17T04:49:42.672404+0000","last_clean":"2024-09-17T04:49:42.672404+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T04:49:42.672404+0000","last_undegraded":"2024-09-17T04:49:42.672404+0000","last_fullsized":"2024-09-17T04:49:42.672404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672966+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T04:49:42.672966+0000","last_peered":"2024-09-17T04:49:42.672966+0000","last_clean":"2024-09-17T04:49:42.672966+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T04:49:42.672966+0000","last_undegraded":"2024-09-17T04:49:42.672966+0000","last_fullsized":"2024-09-17T04:49:42.672966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:09.712272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052007+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T04:49:42.052007+0000","last_peered":"2024-09-17T04:49:42.052007+0000","last_clean":"2024-09-17T04:49:42.052007+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T04:49:42.052007+0000","last_undegraded":"2024-09-17T04:49:42.052007+0000","last_fullsized":"2024-09-17T04:49:42.052007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:43.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042787+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T04:49:42.042787+0000","last_peered":"2024-09-17T04:49:42.042787+0000","last_clean":"2024-09-17T04:49:42.042787+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T04:49:42.042787+0000","last_undegraded":"2024-09-17T04:49:42.042787+0000","last_fullsized":"2024-09-17T04:49:42.042787+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049500+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T04:49:42.049500+0000","last_peered":"2024-09-17T04:49:42.049500+0000","last_clean":"2024-09-17T04:49:42.049500+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T04:49:42.049500+0000","last_undegraded":"2024-09-17T04:49:42.049500+0000","last_fullsized":"2024-09-17T04:49:42.049500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671253+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T04:49:42.671253+0000","last_peered":"2024-09-17T04:49:42.671253+0000","last_clean":"2024-09-17T04:49:42.671253+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T04:49:42.671253+0000","last_undegraded":"2024-09-17T04:49:42.671253+0000","last_fullsized":"2024-09-17T04:49:42.671253+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:52.364568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041504+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T04:49:42.041504+0000","last_peered":"2024-09-17T04:49:42.041504+0000","last_clean":"2024-09-17T04:49:42.041504+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T04:49:42.041504+0000","last_undegraded":"2024-09-17T04:49:42.041504+0000","last_fullsized":"2024-09-17T04:49:42.041504+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:03.537392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671439+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T04:49:42.671439+0000","last_peered":"2024-09-17T04:49:42.671439+0000","last_clean":"2024-09-17T04:49:42.671439+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T04:49:42.671439+0000","last_undegraded":"2024-09-17T04:49:42.671439+0000","last_fullsized":"2024-09-17T04:49:42.671439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670943+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T04:49:42.670943+0000","last_peered":"2024-09-17T04:49:42.670943+0000","last_clean":"2024-09-17T04:49:42.670943+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T04:49:42.670943+0000","last_undegraded":"2024-09-17T04:49:42.670943+0000","last_fullsized":"2024-09-17T04:49:42.670943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671787+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T04:49:42.671787+0000","last_peered":"2024-09-17T04:49:42.671787+0000","last_clean":"2024-09-17T04:49:42.671787+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T04:49:42.671787+0000","last_undegraded":"2024-09-17T04:49:42.671787+0000","last_fullsized":"2024-09-17T04:49:42.671787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:52.120543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041402+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T04:49:42.041402+0000","last_peered":"2024-09-17T04:49:42.041402+0000","last_clean":"2024-09-17T04:49:42.041402+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T04:49:42.041402+0000","last_undegraded":"2024-09-17T04:49:42.041402+0000","last_fullsized":"2024-09-17T04:49:42.041402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:56.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041285+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T04:49:42.041285+0000","last_peered":"2024-09-17T04:49:42.041285+0000","last_clean":"2024-09-17T04:49:42.041285+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T04:49:42.041285+0000","last_undegraded":"2024-09-17T04:49:42.041285+0000","last_fullsized":"2024-09-17T04:49:42.041285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042566+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T04:49:42.042566+0000","last_peered":"2024-09-17T04:49:42.042566+0000","last_clean":"2024-09-17T04:49:42.042566+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T04:49:42.042566+0000","last_undegraded":"2024-09-17T04:49:42.042566+0000","last_fullsized":"2024-09-17T04:49:42.042566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054288+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T04:49:42.054288+0000","last_peered":"2024-09-17T04:49:42.054288+0000","last_clean":"2024-09-17T04:49:42.054288+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T04:49:42.054288+0000","last_undegraded":"2024-09-17T04:49:42.054288+0000","last_fullsized":"2024-09-17T04:49:42.054288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051153+0000","last_change":"2024-09-17T04:49:41.058280+0000","last_active":"2024-09-17T04:49:42.051153+0000","last_peered":"2024-09-17T04:49:42.051153+0000","last_clean":"2024-09-17T04:49:42.051153+0000","last_became_active":"2024-09-17T04:49:41.052009+0000","last_became_peered":"2024-09-17T04:49:41.052009+0000","last_unstale":"2024-09-17T04:49:42.051153+0000","last_undegraded":"2024-09-17T04:49:42.051153+0000","last_fullsized":"2024-09-17T04:49:42.051153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:14.100818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051977+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T04:49:42.051977+0000","last_peered":"2024-09-17T04:49:42.051977+0000","last_clean":"2024-09-17T04:49:42.051977+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T04:49:42.051977+0000","last_undegraded":"2024-09-17T04:49:42.051977+0000","last_fullsized":"2024-09-17T04:49:42.051977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669355+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T04:49:42.669355+0000","last_peered":"2024-09-17T04:49:42.669355+0000","last_clean":"2024-09-17T04:49:42.669355+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T04:49:42.669355+0000","last_undegraded":"2024-09-17T04:49:42.669355+0000","last_fullsized":"2024-09-17T04:49:42.669355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042782+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T04:49:42.042782+0000","last_peered":"2024-09-17T04:49:42.042782+0000","last_clean":"2024-09-17T04:49:42.042782+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T04:49:42.042782+0000","last_undegraded":"2024-09-17T04:49:42.042782+0000","last_fullsized":"2024-09-17T04:49:42.042782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:07.945355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052280+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T04:49:42.052280+0000","last_peered":"2024-09-17T04:49:42.052280+0000","last_clean":"2024-09-17T04:49:42.052280+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T04:49:42.052280+0000","last_undegraded":"2024-09-17T04:49:42.052280+0000","last_fullsized":"2024-09-17T04:49:42.052280+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670412+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T04:49:42.670412+0000","last_peered":"2024-09-17T04:49:42.670412+0000","last_clean":"2024-09-17T04:49:42.670412+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T04:49:42.670412+0000","last_undegraded":"2024-09-17T04:49:42.670412+0000","last_fullsized":"2024-09-17T04:49:42.670412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:50:02.209562+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T04:50:02.209562+0000","last_peered":"2024-09-17T04:50:02.209562+0000","last_clean":"2024-09-17T04:50:02.209562+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T04:50:02.209562+0000","last_undegraded":"2024-09-17T04:50:02.209562+0000","last_fullsized":"2024-09-17T04:50:02.209562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048913+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T04:49:42.048913+0000","last_peered":"2024-09-17T04:49:42.048913+0000","last_clean":"2024-09-17T04:49:42.048913+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T04:49:42.048913+0000","last_undegraded":"2024-09-17T04:49:42.048913+0000","last_fullsized":"2024-09-17T04:49:42.048913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:16.447835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051373+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T04:49:42.051373+0000","last_peered":"2024-09-17T04:49:42.051373+0000","last_clean":"2024-09-17T04:49:42.051373+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T04:49:42.051373+0000","last_undegraded":"2024-09-17T04:49:42.051373+0000","last_fullsized":"2024-09-17T04:49:42.051373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042063+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T04:49:42.042063+0000","last_peered":"2024-09-17T04:49:42.042063+0000","last_clean":"2024-09-17T04:49:42.042063+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T04:49:42.042063+0000","last_undegraded":"2024-09-17T04:49:42.042063+0000","last_fullsized":"2024-09-17T04:49:42.042063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670596+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T04:49:42.670596+0000","last_peered":"2024-09-17T04:49:42.670596+0000","last_clean":"2024-09-17T04:49:42.670596+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T04:49:42.670596+0000","last_undegraded":"2024-09-17T04:49:42.670596+0000","last_fullsized":"2024-09-17T04:49:42.670596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535510+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T04:49:42.535510+0000","last_peered":"2024-09-17T04:49:42.535510+0000","last_clean":"2024-09-17T04:49:42.535510+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T04:49:42.535510+0000","last_undegraded":"2024-09-17T04:49:42.535510+0000","last_fullsized":"2024-09-17T04:49:42.535510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041322+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T04:49:42.041322+0000","last_peered":"2024-09-17T04:49:42.041322+0000","last_clean":"2024-09-17T04:49:42.041322+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T04:49:42.041322+0000","last_undegraded":"2024-09-17T04:49:42.041322+0000","last_fullsized":"2024-09-17T04:49:42.041322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672182+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T04:49:42.672182+0000","last_peered":"2024-09-17T04:49:42.672182+0000","last_clean":"2024-09-17T04:49:42.672182+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T04:49:42.672182+0000","last_undegraded":"2024-09-17T04:49:42.672182+0000","last_fullsized":"2024-09-17T04:49:42.672182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671261+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T04:49:42.671261+0000","last_peered":"2024-09-17T04:49:42.671261+0000","last_clean":"2024-09-17T04:49:42.671261+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T04:49:42.671261+0000","last_undegraded":"2024-09-17T04:49:42.671261+0000","last_fullsized":"2024-09-17T04:49:42.671261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667575+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T04:49:42.667575+0000","last_peered":"2024-09-17T04:49:42.667575+0000","last_clean":"2024-09-17T04:49:42.667575+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T04:49:42.667575+0000","last_undegraded":"2024-09-17T04:49:42.667575+0000","last_fullsized":"2024-09-17T04:49:42.667575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:40.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534654+0000","last_change":"2024-09-17T04:49:41.040278+0000","last_active":"2024-09-17T04:49:42.534654+0000","last_peered":"2024-09-17T04:49:42.534654+0000","last_clean":"2024-09-17T04:49:42.534654+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T04:49:42.534654+0000","last_undegraded":"2024-09-17T04:49:42.534654+0000","last_fullsized":"2024-09-17T04:49:42.534654+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.895223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536732+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T04:49:42.536732+0000","last_peered":"2024-09-17T04:49:42.536732+0000","last_clean":"2024-09-17T04:49:42.536732+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T04:49:42.536732+0000","last_undegraded":"2024-09-17T04:49:42.536732+0000","last_fullsized":"2024-09-17T04:49:42.536732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675753+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T04:49:42.675753+0000","last_peered":"2024-09-17T04:49:42.675753+0000","last_clean":"2024-09-17T04:49:42.675753+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T04:49:42.675753+0000","last_undegraded":"2024-09-17T04:49:42.675753+0000","last_fullsized":"2024-09-17T04:49:42.675753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047646+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T04:49:42.047646+0000","last_peered":"2024-09-17T04:49:42.047646+0000","last_clean":"2024-09-17T04:49:42.047646+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T04:49:42.047646+0000","last_undegraded":"2024-09-17T04:49:42.047646+0000","last_fullsized":"2024-09-17T04:49:42.047646+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:15.845612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537298+0000","last_change":"2024-09-17T04:49:41.050551+0000","last_active":"2024-09-17T04:49:42.537298+0000","last_peered":"2024-09-17T04:49:42.537298+0000","last_clean":"2024-09-17T04:49:42.537298+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T04:49:42.537298+0000","last_undegraded":"2024-09-17T04:49:42.537298+0000","last_fullsized":"2024-09-17T04:49:42.537298+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:11.088230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051592+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T04:49:42.051592+0000","last_peered":"2024-09-17T04:49:42.051592+0000","last_clean":"2024-09-17T04:49:42.051592+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T04:49:42.051592+0000","last_undegraded":"2024-09-17T04:49:42.051592+0000","last_fullsized":"2024-09-17T04:49:42.051592+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:37.778756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048401+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T04:49:42.048401+0000","last_peered":"2024-09-17T04:49:42.048401+0000","last_clean":"2024-09-17T04:49:42.048401+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T04:49:42.048401+0000","last_undegraded":"2024-09-17T04:49:42.048401+0000","last_fullsized":"2024-09-17T04:49:42.048401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041797+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T04:49:42.041797+0000","last_peered":"2024-09-17T04:49:42.041797+0000","last_clean":"2024-09-17T04:49:42.041797+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T04:49:42.041797+0000","last_undegraded":"2024-09-17T04:49:42.041797+0000","last_fullsized":"2024-09-17T04:49:42.041797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055365+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T04:49:42.055365+0000","last_peered":"2024-09-17T04:49:42.055365+0000","last_clean":"2024-09-17T04:49:42.055365+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T04:49:42.055365+0000","last_undegraded":"2024-09-17T04:49:42.055365+0000","last_fullsized":"2024-09-17T04:49:42.055365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041767+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T04:49:42.041767+0000","last_peered":"2024-09-17T04:49:42.041767+0000","last_clean":"2024-09-17T04:49:42.041767+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T04:49:42.041767+0000","last_undegraded":"2024-09-17T04:49:42.041767+0000","last_fullsized":"2024-09-17T04:49:42.041767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667708+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T04:49:42.667708+0000","last_peered":"2024-09-17T04:49:42.667708+0000","last_clean":"2024-09-17T04:49:42.667708+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T04:49:42.667708+0000","last_undegraded":"2024-09-17T04:49:42.667708+0000","last_fullsized":"2024-09-17T04:49:42.667708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050522+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T04:49:42.050522+0000","last_peered":"2024-09-17T04:49:42.050522+0000","last_clean":"2024-09-17T04:49:42.050522+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T04:49:42.050522+0000","last_undegraded":"2024-09-17T04:49:42.050522+0000","last_fullsized":"2024-09-17T04:49:42.050522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049954+0000","last_change":"2024-09-17T04:49:41.057647+0000","last_active":"2024-09-17T04:49:42.049954+0000","last_peered":"2024-09-17T04:49:42.049954+0000","last_clean":"2024-09-17T04:49:42.049954+0000","last_became_active":"2024-09-17T04:49:41.056213+0000","last_became_peered":"2024-09-17T04:49:41.056213+0000","last_unstale":"2024-09-17T04:49:42.049954+0000","last_undegraded":"2024-09-17T04:49:42.049954+0000","last_fullsized":"2024-09-17T04:49:42.049954+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042498+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T04:49:42.042498+0000","last_peered":"2024-09-17T04:49:42.042498+0000","last_clean":"2024-09-17T04:49:42.042498+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T04:49:42.042498+0000","last_undegraded":"2024-09-17T04:49:42.042498+0000","last_fullsized":"2024-09-17T04:49:42.042498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053714+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T04:49:42.053714+0000","last_peered":"2024-09-17T04:49:42.053714+0000","last_clean":"2024-09-17T04:49:42.053714+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T04:49:42.053714+0000","last_undegraded":"2024-09-17T04:49:42.053714+0000","last_fullsized":"2024-09-17T04:49:42.053714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051282+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T04:49:42.051282+0000","last_peered":"2024-09-17T04:49:42.051282+0000","last_clean":"2024-09-17T04:49:42.051282+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T04:49:42.051282+0000","last_undegraded":"2024-09-17T04:49:42.051282+0000","last_fullsized":"2024-09-17T04:49:42.051282+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:27.230604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536526+0000","last_change":"2024-09-17T04:49:41.055263+0000","last_active":"2024-09-17T04:49:42.536526+0000","last_peered":"2024-09-17T04:49:42.536526+0000","last_clean":"2024-09-17T04:49:42.536526+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T04:49:42.536526+0000","last_undegraded":"2024-09-17T04:49:42.536526+0000","last_fullsized":"2024-09-17T04:49:42.536526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670438+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T04:49:42.670438+0000","last_peered":"2024-09-17T04:49:42.670438+0000","last_clean":"2024-09-17T04:49:42.670438+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T04:49:42.670438+0000","last_undegraded":"2024-09-17T04:49:42.670438+0000","last_fullsized":"2024-09-17T04:49:42.670438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673048+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T04:49:42.673048+0000","last_peered":"2024-09-17T04:49:42.673048+0000","last_clean":"2024-09-17T04:49:42.673048+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T04:49:42.673048+0000","last_undegraded":"2024-09-17T04:49:42.673048+0000","last_fullsized":"2024-09-17T04:49:42.673048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670447+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T04:49:42.670447+0000","last_peered":"2024-09-17T04:49:42.670447+0000","last_clean":"2024-09-17T04:49:42.670447+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T04:49:42.670447+0000","last_undegraded":"2024-09-17T04:49:42.670447+0000","last_fullsized":"2024-09-17T04:49:42.670447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:24.586834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051128+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.051128+0000","last_peered":"2024-09-17T04:49:42.051128+0000","last_clean":"2024-09-17T04:49:42.051128+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T04:49:42.051128+0000","last_undegraded":"2024-09-17T04:49:42.051128+0000","last_fullsized":"2024-09-17T04:49:42.051128+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536163+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T04:49:42.536163+0000","last_peered":"2024-09-17T04:49:42.536163+0000","last_clean":"2024-09-17T04:49:42.536163+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T04:49:42.536163+0000","last_undegraded":"2024-09-17T04:49:42.536163+0000","last_fullsized":"2024-09-17T04:49:42.536163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052121+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T04:49:42.052121+0000","last_peered":"2024-09-17T04:49:42.052121+0000","last_clean":"2024-09-17T04:49:42.052121+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T04:49:42.052121+0000","last_undegraded":"2024-09-17T04:49:42.052121+0000","last_fullsized":"2024-09-17T04:49:42.052121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668467+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T04:49:42.668467+0000","last_peered":"2024-09-17T04:49:42.668467+0000","last_clean":"2024-09-17T04:49:42.668467+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T04:49:42.668467+0000","last_undegraded":"2024-09-17T04:49:42.668467+0000","last_fullsized":"2024-09-17T04:49:42.668467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536096+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T04:49:42.536096+0000","last_peered":"2024-09-17T04:49:42.536096+0000","last_clean":"2024-09-17T04:49:42.536096+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T04:49:42.536096+0000","last_undegraded":"2024-09-17T04:49:42.536096+0000","last_fullsized":"2024-09-17T04:49:42.536096+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042322+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T04:49:42.042322+0000","last_peered":"2024-09-17T04:49:42.042322+0000","last_clean":"2024-09-17T04:49:42.042322+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T04:49:42.042322+0000","last_undegraded":"2024-09-17T04:49:42.042322+0000","last_fullsized":"2024-09-17T04:49:42.042322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671200+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T04:49:42.671200+0000","last_peered":"2024-09-17T04:49:42.671200+0000","last_clean":"2024-09-17T04:49:42.671200+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T04:49:42.671200+0000","last_undegraded":"2024-09-17T04:49:42.671200+0000","last_fullsized":"2024-09-17T04:49:42.671200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670309+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T04:49:42.670309+0000","last_peered":"2024-09-17T04:49:42.670309+0000","last_clean":"2024-09-17T04:49:42.670309+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T04:49:42.670309+0000","last_undegraded":"2024-09-17T04:49:42.670309+0000","last_fullsized":"2024-09-17T04:49:42.670309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668969+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T04:49:42.668969+0000","last_peered":"2024-09-17T04:49:42.668969+0000","last_clean":"2024-09-17T04:49:42.668969+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T04:49:42.668969+0000","last_undegraded":"2024-09-17T04:49:42.668969+0000","last_fullsized":"2024-09-17T04:49:42.668969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054935+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T04:49:42.054935+0000","last_peered":"2024-09-17T04:49:42.054935+0000","last_clean":"2024-09-17T04:49:42.054935+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T04:49:42.054935+0000","last_undegraded":"2024-09-17T04:49:42.054935+0000","last_fullsized":"2024-09-17T04:49:42.054935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669059+0000","last_change":"2024-09-17T04:49:41.073709+0000","last_active":"2024-09-17T04:49:42.669059+0000","last_peered":"2024-09-17T04:49:42.669059+0000","last_clean":"2024-09-17T04:49:42.669059+0000","last_became_active":"2024-09-17T04:49:41.073330+0000","last_became_peered":"2024-09-17T04:49:41.073330+0000","last_unstale":"2024-09-17T04:49:42.669059+0000","last_undegraded":"2024-09-17T04:49:42.669059+0000","last_fullsized":"2024-09-17T04:49:42.669059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054811+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T04:49:42.054811+0000","last_peered":"2024-09-17T04:49:42.054811+0000","last_clean":"2024-09-17T04:49:42.054811+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T04:49:42.054811+0000","last_undegraded":"2024-09-17T04:49:42.054811+0000","last_fullsized":"2024-09-17T04:49:42.054811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041256+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T04:49:42.041256+0000","last_peered":"2024-09-17T04:49:42.041256+0000","last_clean":"2024-09-17T04:49:42.041256+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T04:49:42.041256+0000","last_undegraded":"2024-09-17T04:49:42.041256+0000","last_fullsized":"2024-09-17T04:49:42.041256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536669+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T04:49:42.536669+0000","last_peered":"2024-09-17T04:49:42.536669+0000","last_clean":"2024-09-17T04:49:42.536669+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T04:49:42.536669+0000","last_undegraded":"2024-09-17T04:49:42.536669+0000","last_fullsized":"2024-09-17T04:49:42.536669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534968+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T04:49:42.534968+0000","last_peered":"2024-09-17T04:49:42.534968+0000","last_clean":"2024-09-17T04:49:42.534968+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T04:49:42.534968+0000","last_undegraded":"2024-09-17T04:49:42.534968+0000","last_fullsized":"2024-09-17T04:49:42.534968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672019+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T04:49:42.672019+0000","last_peered":"2024-09-17T04:49:42.672019+0000","last_clean":"2024-09-17T04:49:42.672019+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T04:49:42.672019+0000","last_undegraded":"2024-09-17T04:49:42.672019+0000","last_fullsized":"2024-09-17T04:49:42.672019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668565+0000","last_change":"2024-09-17T04:49:41.045502+0000","last_active":"2024-09-17T04:49:42.668565+0000","last_peered":"2024-09-17T04:49:42.668565+0000","last_clean":"2024-09-17T04:49:42.668565+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T04:49:42.668565+0000","last_undegraded":"2024-09-17T04:49:42.668565+0000","last_fullsized":"2024-09-17T04:49:42.668565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:19.746356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041666+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T04:49:42.041666+0000","last_peered":"2024-09-17T04:49:42.041666+0000","last_clean":"2024-09-17T04:49:42.041666+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T04:49:42.041666+0000","last_undegraded":"2024-09-17T04:49:42.041666+0000","last_fullsized":"2024-09-17T04:49:42.041666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042310+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T04:49:42.042310+0000","last_peered":"2024-09-17T04:49:42.042310+0000","last_clean":"2024-09-17T04:49:42.042310+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T04:49:42.042310+0000","last_undegraded":"2024-09-17T04:49:42.042310+0000","last_fullsized":"2024-09-17T04:49:42.042310+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668045+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T04:49:42.668045+0000","last_peered":"2024-09-17T04:49:42.668045+0000","last_clean":"2024-09-17T04:49:42.668045+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T04:49:42.668045+0000","last_undegraded":"2024-09-17T04:49:42.668045+0000","last_fullsized":"2024-09-17T04:49:42.668045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054699+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T04:49:42.054699+0000","last_peered":"2024-09-17T04:49:42.054699+0000","last_clean":"2024-09-17T04:49:42.054699+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T04:49:42.054699+0000","last_undegraded":"2024-09-17T04:49:42.054699+0000","last_fullsized":"2024-09-17T04:49:42.054699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673038+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T04:49:42.673038+0000","last_peered":"2024-09-17T04:49:42.673038+0000","last_clean":"2024-09-17T04:49:42.673038+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T04:49:42.673038+0000","last_undegraded":"2024-09-17T04:49:42.673038+0000","last_fullsized":"2024-09-17T04:49:42.673038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669104+0000","last_change":"2024-09-17T04:49:41.038265+0000","last_active":"2024-09-17T04:49:42.669104+0000","last_peered":"2024-09-17T04:49:42.669104+0000","last_clean":"2024-09-17T04:49:42.669104+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T04:49:42.669104+0000","last_undegraded":"2024-09-17T04:49:42.669104+0000","last_fullsized":"2024-09-17T04:49:42.669104+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:41.638403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053416+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T04:49:42.053416+0000","last_peered":"2024-09-17T04:49:42.053416+0000","last_clean":"2024-09-17T04:49:42.053416+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T04:49:42.053416+0000","last_undegraded":"2024-09-17T04:49:42.053416+0000","last_fullsized":"2024-09-17T04:49:42.053416+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535001+0000","last_change":"2024-09-17T04:49:41.053443+0000","last_active":"2024-09-17T04:49:42.535001+0000","last_peered":"2024-09-17T04:49:42.535001+0000","last_clean":"2024-09-17T04:49:42.535001+0000","last_became_active":"2024-09-17T04:49:41.053090+0000","last_became_peered":"2024-09-17T04:49:41.053090+0000","last_unstale":"2024-09-17T04:49:42.535001+0000","last_undegraded":"2024-09-17T04:49:42.535001+0000","last_fullsized":"2024-09-17T04:49:42.535001+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671599+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T04:49:42.671599+0000","last_peered":"2024-09-17T04:49:42.671599+0000","last_clean":"2024-09-17T04:49:42.671599+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T04:49:42.671599+0000","last_undegraded":"2024-09-17T04:49:42.671599+0000","last_fullsized":"2024-09-17T04:49:42.671599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669669+0000","last_change":"2024-09-17T04:49:41.038072+0000","last_active":"2024-09-17T04:49:42.669669+0000","last_peered":"2024-09-17T04:49:42.669669+0000","last_clean":"2024-09-17T04:49:42.669669+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T04:49:42.669669+0000","last_undegraded":"2024-09-17T04:49:42.669669+0000","last_fullsized":"2024-09-17T04:49:42.669669+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043492+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T04:49:42.043492+0000","last_peered":"2024-09-17T04:49:42.043492+0000","last_clean":"2024-09-17T04:49:42.043492+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T04:49:42.043492+0000","last_undegraded":"2024-09-17T04:49:42.043492+0000","last_fullsized":"2024-09-17T04:49:42.043492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534462+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T04:49:42.534462+0000","last_peered":"2024-09-17T04:49:42.534462+0000","last_clean":"2024-09-17T04:49:42.534462+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T04:49:42.534462+0000","last_undegraded":"2024-09-17T04:49:42.534462+0000","last_fullsized":"2024-09-17T04:49:42.534462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669602+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T04:49:42.669602+0000","last_peered":"2024-09-17T04:49:42.669602+0000","last_clean":"2024-09-17T04:49:42.669602+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T04:49:42.669602+0000","last_undegraded":"2024-09-17T04:49:42.669602+0000","last_fullsized":"2024-09-17T04:49:42.669602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050316+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T04:49:42.050316+0000","last_peered":"2024-09-17T04:49:42.050316+0000","last_clean":"2024-09-17T04:49:42.050316+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T04:49:42.050316+0000","last_undegraded":"2024-09-17T04:49:42.050316+0000","last_fullsized":"2024-09-17T04:49:42.050316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042123+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T04:49:42.042123+0000","last_peered":"2024-09-17T04:49:42.042123+0000","last_clean":"2024-09-17T04:49:42.042123+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T04:49:42.042123+0000","last_undegraded":"2024-09-17T04:49:42.042123+0000","last_fullsized":"2024-09-17T04:49:42.042123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052378+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T04:49:42.052378+0000","last_peered":"2024-09-17T04:49:42.052378+0000","last_clean":"2024-09-17T04:49:42.052378+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T04:49:42.052378+0000","last_undegraded":"2024-09-17T04:49:42.052378+0000","last_fullsized":"2024-09-17T04:49:42.052378+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041588+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T04:49:42.041588+0000","last_peered":"2024-09-17T04:49:42.041588+0000","last_clean":"2024-09-17T04:49:42.041588+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T04:49:42.041588+0000","last_undegraded":"2024-09-17T04:49:42.041588+0000","last_fullsized":"2024-09-17T04:49:42.041588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674897+0000","last_change":"2024-09-17T04:49:41.067207+0000","last_active":"2024-09-17T04:49:42.674897+0000","last_peered":"2024-09-17T04:49:42.674897+0000","last_clean":"2024-09-17T04:49:42.674897+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T04:49:42.674897+0000","last_undegraded":"2024-09-17T04:49:42.674897+0000","last_fullsized":"2024-09-17T04:49:42.674897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:08.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667484+0000","last_change":"2024-09-17T04:49:31.911036+0000","last_active":"2024-09-17T04:49:42.667484+0000","last_peered":"2024-09-17T04:49:42.667484+0000","last_clean":"2024-09-17T04:49:42.667484+0000","last_became_active":"2024-09-17T04:49:31.910285+0000","last_became_peered":"2024-09-17T04:49:31.910285+0000","last_unstale":"2024-09-17T04:49:42.667484+0000","last_undegraded":"2024-09-17T04:49:42.667484+0000","last_fullsized":"2024-09-17T04:49:42.667484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673474+0000","last_change":"2024-09-17T04:49:41.046170+0000","last_active":"2024-09-17T04:49:42.673474+0000","last_peered":"2024-09-17T04:49:42.673474+0000","last_clean":"2024-09-17T04:49:42.673474+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T04:49:42.673474+0000","last_undegraded":"2024-09-17T04:49:42.673474+0000","last_fullsized":"2024-09-17T04:49:42.673474+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:01.683826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051528+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T04:49:42.051528+0000","last_peered":"2024-09-17T04:49:42.051528+0000","last_clean":"2024-09-17T04:49:42.051528+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T04:49:42.051528+0000","last_undegraded":"2024-09-17T04:49:42.051528+0000","last_fullsized":"2024-09-17T04:49:42.051528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041843+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T04:49:42.041843+0000","last_peered":"2024-09-17T04:49:42.041843+0000","last_clean":"2024-09-17T04:49:42.041843+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T04:49:42.041843+0000","last_undegraded":"2024-09-17T04:49:42.041843+0000","last_fullsized":"2024-09-17T04:49:42.041843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:25.574682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053942+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T04:49:42.053942+0000","last_peered":"2024-09-17T04:49:42.053942+0000","last_clean":"2024-09-17T04:49:42.053942+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T04:49:42.053942+0000","last_undegraded":"2024-09-17T04:49:42.053942+0000","last_fullsized":"2024-09-17T04:49:42.053942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048414+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T04:49:42.048414+0000","last_peered":"2024-09-17T04:49:42.048414+0000","last_clean":"2024-09-17T04:49:42.048414+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T04:49:42.048414+0000","last_undegraded":"2024-09-17T04:49:42.048414+0000","last_fullsized":"2024-09-17T04:49:42.048414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671218+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T04:49:42.671218+0000","last_peered":"2024-09-17T04:49:42.671218+0000","last_clean":"2024-09-17T04:49:42.671218+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T04:49:42.671218+0000","last_undegraded":"2024-09-17T04:49:42.671218+0000","last_fullsized":"2024-09-17T04:49:42.671218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053498+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T04:49:42.053498+0000","last_peered":"2024-09-17T04:49:42.053498+0000","last_clean":"2024-09-17T04:49:42.053498+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T04:49:42.053498+0000","last_undegraded":"2024-09-17T04:49:42.053498+0000","last_fullsized":"2024-09-17T04:49:42.053498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670119+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T04:49:42.670119+0000","last_peered":"2024-09-17T04:49:42.670119+0000","last_clean":"2024-09-17T04:49:42.670119+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T04:49:42.670119+0000","last_undegraded":"2024-09-17T04:49:42.670119+0000","last_fullsized":"2024-09-17T04:49:42.670119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668585+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T04:49:42.668585+0000","last_peered":"2024-09-17T04:49:42.668585+0000","last_clean":"2024-09-17T04:49:42.668585+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T04:49:42.668585+0000","last_undegraded":"2024-09-17T04:49:42.668585+0000","last_fullsized":"2024-09-17T04:49:42.668585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670988+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T04:49:42.670988+0000","last_peered":"2024-09-17T04:49:42.670988+0000","last_clean":"2024-09-17T04:49:42.670988+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T04:49:42.670988+0000","last_undegraded":"2024-09-17T04:49:42.670988+0000","last_fullsized":"2024-09-17T04:49:42.670988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671416+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.671416+0000","last_peered":"2024-09-17T04:49:42.671416+0000","last_clean":"2024-09-17T04:49:42.671416+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T04:49:42.671416+0000","last_undegraded":"2024-09-17T04:49:42.671416+0000","last_fullsized":"2024-09-17T04:49:42.671416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675766+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T04:49:42.675766+0000","last_peered":"2024-09-17T04:49:42.675766+0000","last_clean":"2024-09-17T04:49:42.675766+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T04:49:42.675766+0000","last_undegraded":"2024-09-17T04:49:42.675766+0000","last_fullsized":"2024-09-17T04:49:42.675766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671330+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T04:49:42.671330+0000","last_peered":"2024-09-17T04:49:42.671330+0000","last_clean":"2024-09-17T04:49:42.671330+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T04:49:42.671330+0000","last_undegraded":"2024-09-17T04:49:42.671330+0000","last_fullsized":"2024-09-17T04:49:42.671330+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667310+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T04:49:42.667310+0000","last_peered":"2024-09-17T04:49:42.667310+0000","last_clean":"2024-09-17T04:49:42.667310+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T04:49:42.667310+0000","last_undegraded":"2024-09-17T04:49:42.667310+0000","last_fullsized":"2024-09-17T04:49:42.667310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537133+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T04:49:42.537133+0000","last_peered":"2024-09-17T04:49:42.537133+0000","last_clean":"2024-09-17T04:49:42.537133+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T04:49:42.537133+0000","last_undegraded":"2024-09-17T04:49:42.537133+0000","last_fullsized":"2024-09-17T04:49:42.537133+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669347+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T04:49:42.669347+0000","last_peered":"2024-09-17T04:49:42.669347+0000","last_clean":"2024-09-17T04:49:42.669347+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T04:49:42.669347+0000","last_undegraded":"2024-09-17T04:49:42.669347+0000","last_fullsized":"2024-09-17T04:49:42.669347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050809+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T04:49:42.050809+0000","last_peered":"2024-09-17T04:49:42.050809+0000","last_clean":"2024-09-17T04:49:42.050809+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T04:49:42.050809+0000","last_undegraded":"2024-09-17T04:49:42.050809+0000","last_fullsized":"2024-09-17T04:49:42.050809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535865+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T04:49:42.535865+0000","last_peered":"2024-09-17T04:49:42.535865+0000","last_clean":"2024-09-17T04:49:42.535865+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T04:49:42.535865+0000","last_undegraded":"2024-09-17T04:49:42.535865+0000","last_fullsized":"2024-09-17T04:49:42.535865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673274+0000","last_change":"2024-09-17T04:49:41.041201+0000","last_active":"2024-09-17T04:49:42.673274+0000","last_peered":"2024-09-17T04:49:42.673274+0000","last_clean":"2024-09-17T04:49:42.673274+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T04:49:42.673274+0000","last_undegraded":"2024-09-17T04:49:42.673274+0000","last_fullsized":"2024-09-17T04:49:42.673274+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:02.588786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670272+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T04:49:42.670272+0000","last_peered":"2024-09-17T04:49:42.670272+0000","last_clean":"2024-09-17T04:49:42.670272+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T04:49:42.670272+0000","last_undegraded":"2024-09-17T04:49:42.670272+0000","last_fullsized":"2024-09-17T04:49:42.670272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667767+0000","last_change":"2024-09-17T04:49:41.061092+0000","last_active":"2024-09-17T04:49:42.667767+0000","last_peered":"2024-09-17T04:49:42.667767+0000","last_clean":"2024-09-17T04:49:42.667767+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T04:49:42.667767+0000","last_undegraded":"2024-09-17T04:49:42.667767+0000","last_fullsized":"2024-09-17T04:49:42.667767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:52.714614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536820+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T04:49:42.536820+0000","last_peered":"2024-09-17T04:49:42.536820+0000","last_clean":"2024-09-17T04:49:42.536820+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T04:49:42.536820+0000","last_undegraded":"2024-09-17T04:49:42.536820+0000","last_fullsized":"2024-09-17T04:49:42.536820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669397+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T04:49:42.669397+0000","last_peered":"2024-09-17T04:49:42.669397+0000","last_clean":"2024-09-17T04:49:42.669397+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T04:49:42.669397+0000","last_undegraded":"2024-09-17T04:49:42.669397+0000","last_fullsized":"2024-09-17T04:49:42.669397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671170+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T04:49:42.671170+0000","last_peered":"2024-09-17T04:49:42.671170+0000","last_clean":"2024-09-17T04:49:42.671170+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T04:49:42.671170+0000","last_undegraded":"2024-09-17T04:49:42.671170+0000","last_fullsized":"2024-09-17T04:49:42.671170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536018+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T04:49:42.536018+0000","last_peered":"2024-09-17T04:49:42.536018+0000","last_clean":"2024-09-17T04:49:42.536018+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T04:49:42.536018+0000","last_undegraded":"2024-09-17T04:49:42.536018+0000","last_fullsized":"2024-09-17T04:49:42.536018+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671602+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T04:49:42.671602+0000","last_peered":"2024-09-17T04:49:42.671602+0000","last_clean":"2024-09-17T04:49:42.671602+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T04:49:42.671602+0000","last_undegraded":"2024-09-17T04:49:42.671602+0000","last_fullsized":"2024-09-17T04:49:42.671602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534962+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T04:49:42.534962+0000","last_peered":"2024-09-17T04:49:42.534962+0000","last_clean":"2024-09-17T04:49:42.534962+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T04:49:42.534962+0000","last_undegraded":"2024-09-17T04:49:42.534962+0000","last_fullsized":"2024-09-17T04:49:42.534962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671825+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T04:49:42.671825+0000","last_peered":"2024-09-17T04:49:42.671825+0000","last_clean":"2024-09-17T04:49:42.671825+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T04:49:42.671825+0000","last_undegraded":"2024-09-17T04:49:42.671825+0000","last_fullsized":"2024-09-17T04:49:42.671825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2851,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509453,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509453,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509453,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":108978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509453,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509453,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":106706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509454,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509454,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:50:10.404 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-09-17T04:50:10.404 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:50:10.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.591+0000 7f184ab5b640 1 -- 172.21.15.150:0/592085218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18440c3db0 msgr2=0x7f18440ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.591+0000 7f184ab5b640 1 --2- 172.21.15.150:0/592085218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18440c3db0 0x7f18440ad2a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f18440c4740 tx=0x7f183c01cd80 comp rx=0 tx=0).stop 2024-09-17T04:50:10.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.591+0000 7f184ab5b640 1 -- 172.21.15.150:0/592085218 shutdown_connections 2024-09-17T04:50:10.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.591+0000 7f184ab5b640 1 --2- 172.21.15.150:0/592085218 >> v2:172.21.15.176:3300/0 conn(0x7f184406fab0 0x7f184406fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.591+0000 7f184ab5b640 1 --2- 172.21.15.150:0/592085218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18440c41c0 0x7f18440ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.591+0000 7f184ab5b640 1 --2- 172.21.15.150:0/592085218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18440c3db0 0x7f18440ad2a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.591+0000 7f184ab5b640 1 -- 172.21.15.150:0/592085218 >> 172.21.15.150:0/592085218 conn(0x7f18440c4a00 msgr2=0x7f18440c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:10.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.591+0000 7f184ab5b640 1 -- 172.21.15.150:0/592085218 shutdown_connections 2024-09-17T04:50:10.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.591+0000 7f184ab5b640 1 -- 172.21.15.150:0/592085218 wait complete. 2024-09-17T04:50:10.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.591+0000 7f184ab5b640 1 Processor -- start 2024-09-17T04:50:10.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.591+0000 7f184ab5b640 1 -- start start 2024-09-17T04:50:10.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f184ab5b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f184406fab0 0x7f184414e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:10.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f184ab5b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18440c3db0 0x7f18440bcd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:10.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f184ab5b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18440c41c0 0x7f18441570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f184ab5b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f18440a7770 con 0x7f184406fab0 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f184ab5b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f18440a75f0 con 0x7f18440c41c0 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f184ab5b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1844158480 con 0x7f18440c3db0 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18488f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f184406fab0 0x7f184414e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18488f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f184406fab0 0x7f184414e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37604/0 (socket says 172.21.15.150:37604) 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18488f7640 1 -- 172.21.15.150:0/3012051638 learned_addr learned my addr 172.21.15.150:0/3012051638 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18490f8640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18440c41c0 0x7f18441570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f1843fff640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18440c3db0 0x7f18440bcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18488f7640 1 -- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18440c3db0 msgr2=0x7f18440bcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18488f7640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18440c3db0 0x7f18440bcd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18488f7640 1 -- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18440c41c0 msgr2=0x7f18441570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18488f7640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18440c41c0 0x7f18441570b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18488f7640 1 -- 172.21.15.150:0/3012051638 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18441516f0 con 0x7f184406fab0 2024-09-17T04:50:10.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18488f7640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f184406fab0 0x7f184414e470 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f183c01d620 tx=0x7f183c009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:10.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18417fa640 1 -- 172.21.15.150:0/3012051638 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c0bb070 con 0x7f184406fab0 2024-09-17T04:50:10.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18417fa640 1 -- 172.21.15.150:0/3012051638 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f183c004d40 con 0x7f184406fab0 2024-09-17T04:50:10.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f184ab5b640 1 -- 172.21.15.150:0/3012051638 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18441f0660 con 0x7f184406fab0 2024-09-17T04:50:10.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18417fa640 1 -- 172.21.15.150:0/3012051638 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c0b2040 con 0x7f184406fab0 2024-09-17T04:50:10.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f184ab5b640 1 -- 172.21.15.150:0/3012051638 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1844150cd0 con 0x7f184406fab0 2024-09-17T04:50:10.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f184ab5b640 1 -- 172.21.15.150:0/3012051638 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1844150f30 con 0x7f184406fab0 2024-09-17T04:50:10.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.595+0000 7f18417fa640 1 -- 172.21.15.150:0/3012051638 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f183c0be070 con 0x7f184406fab0 2024-09-17T04:50:10.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.599+0000 7f18417fa640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1824073230 0x7f1824075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:10.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.599+0000 7f18417fa640 1 -- 172.21.15.150:0/3012051638 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f183c12e620 con 0x7f184406fab0 2024-09-17T04:50:10.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.599+0000 7f1843fff640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1824073230 0x7f1824075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:10.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.599+0000 7f1843fff640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1824073230 0x7f1824075740 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f182c0058d0 tx=0x7f182c005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:10.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.603+0000 7f18417fa640 1 -- 172.21.15.150:0/3012051638 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f183c016970 con 0x7f184406fab0 2024-09-17T04:50:10.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.755+0000 7f184ab5b640 1 -- 172.21.15.150:0/3012051638 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f184414e9b0 con 0x7f1824073230 2024-09-17T04:50:10.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.779+0000 7f18417fa640 1 -- 172.21.15.150:0/3012051638 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490314 (secure 0 0 0) 0x7f182c00fad0 con 0x7f1824073230 2024-09-17T04:50:10.786 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:50:10.789 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T04:50:10.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.783+0000 7f184ab5b640 1 -- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1824073230 msgr2=0x7f1824075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.783+0000 7f184ab5b640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1824073230 0x7f1824075740 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f182c0058d0 tx=0x7f182c005860 comp rx=0 tx=0).stop 2024-09-17T04:50:10.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.783+0000 7f184ab5b640 1 -- 172.21.15.150:0/3012051638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f184406fab0 msgr2=0x7f184414e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.783+0000 7f184ab5b640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f184406fab0 0x7f184414e470 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f183c01d620 tx=0x7f183c009360 comp rx=0 tx=0).stop 2024-09-17T04:50:10.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.783+0000 7f184ab5b640 1 -- 172.21.15.150:0/3012051638 shutdown_connections 2024-09-17T04:50:10.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.783+0000 7f184ab5b640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1824073230 0x7f1824075740 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.783+0000 7f184ab5b640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18440c41c0 0x7f18441570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.783+0000 7f184ab5b640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18440c3db0 0x7f18440bcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.783+0000 7f184ab5b640 1 --2- 172.21.15.150:0/3012051638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f184406fab0 0x7f184414e470 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.783+0000 7f184ab5b640 1 -- 172.21.15.150:0/3012051638 >> 172.21.15.150:0/3012051638 conn(0x7f18440c4a00 msgr2=0x7f1844153ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:10.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.783+0000 7f184ab5b640 1 -- 172.21.15.150:0/3012051638 shutdown_connections 2024-09-17T04:50:10.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.783+0000 7f184ab5b640 1 -- 172.21.15.150:0/3012051638 wait complete. 2024-09-17T04:50:10.805 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-17T04:50:08.959982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595629,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":193,"num_write_kb":1658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223792,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750928,"statfs":{"total":773094113280,"available":772864950272,"internally_reserved":0,"allocated":3481600,"data_stored":2028210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":441,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006952"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671077+0000","last_change":"2024-09-17T04:49:41.025271+0000","last_active":"2024-09-17T04:49:42.671077+0000","last_peered":"2024-09-17T04:49:42.671077+0000","last_clean":"2024-09-17T04:49:42.671077+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T04:49:42.671077+0000","last_undegraded":"2024-09-17T04:49:42.671077+0000","last_fullsized":"2024-09-17T04:49:42.671077+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:39.715602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054640+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T04:49:42.054640+0000","last_peered":"2024-09-17T04:49:42.054640+0000","last_clean":"2024-09-17T04:49:42.054640+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T04:49:42.054640+0000","last_undegraded":"2024-09-17T04:49:42.054640+0000","last_fullsized":"2024-09-17T04:49:42.054640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671706+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T04:49:42.671706+0000","last_peered":"2024-09-17T04:49:42.671706+0000","last_clean":"2024-09-17T04:49:42.671706+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T04:49:42.671706+0000","last_undegraded":"2024-09-17T04:49:42.671706+0000","last_fullsized":"2024-09-17T04:49:42.671706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668736+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T04:49:42.668736+0000","last_peered":"2024-09-17T04:49:42.668736+0000","last_clean":"2024-09-17T04:49:42.668736+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T04:49:42.668736+0000","last_undegraded":"2024-09-17T04:49:42.668736+0000","last_fullsized":"2024-09-17T04:49:42.668736+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:38.417840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047972+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T04:49:42.047972+0000","last_peered":"2024-09-17T04:49:42.047972+0000","last_clean":"2024-09-17T04:49:42.047972+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T04:49:42.047972+0000","last_undegraded":"2024-09-17T04:49:42.047972+0000","last_fullsized":"2024-09-17T04:49:42.047972+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672300+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T04:49:42.672300+0000","last_peered":"2024-09-17T04:49:42.672300+0000","last_clean":"2024-09-17T04:49:42.672300+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T04:49:42.672300+0000","last_undegraded":"2024-09-17T04:49:42.672300+0000","last_fullsized":"2024-09-17T04:49:42.672300+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672014+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T04:49:42.672014+0000","last_peered":"2024-09-17T04:49:42.672014+0000","last_clean":"2024-09-17T04:49:42.672014+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T04:49:42.672014+0000","last_undegraded":"2024-09-17T04:49:42.672014+0000","last_fullsized":"2024-09-17T04:49:42.672014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535134+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T04:49:42.535134+0000","last_peered":"2024-09-17T04:49:42.535134+0000","last_clean":"2024-09-17T04:49:42.535134+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T04:49:42.535134+0000","last_undegraded":"2024-09-17T04:49:42.535134+0000","last_fullsized":"2024-09-17T04:49:42.535134+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672206+0000","last_change":"2024-09-17T04:49:41.067317+0000","last_active":"2024-09-17T04:49:42.672206+0000","last_peered":"2024-09-17T04:49:42.672206+0000","last_clean":"2024-09-17T04:49:42.672206+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T04:49:42.672206+0000","last_undegraded":"2024-09-17T04:49:42.672206+0000","last_fullsized":"2024-09-17T04:49:42.672206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:37.637477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670707+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T04:49:42.670707+0000","last_peered":"2024-09-17T04:49:42.670707+0000","last_clean":"2024-09-17T04:49:42.670707+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T04:49:42.670707+0000","last_undegraded":"2024-09-17T04:49:42.670707+0000","last_fullsized":"2024-09-17T04:49:42.670707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674274+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T04:49:42.674274+0000","last_peered":"2024-09-17T04:49:42.674274+0000","last_clean":"2024-09-17T04:49:42.674274+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T04:49:42.674274+0000","last_undegraded":"2024-09-17T04:49:42.674274+0000","last_fullsized":"2024-09-17T04:49:42.674274+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535118+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T04:49:42.535118+0000","last_peered":"2024-09-17T04:49:42.535118+0000","last_clean":"2024-09-17T04:49:42.535118+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T04:49:42.535118+0000","last_undegraded":"2024-09-17T04:49:42.535118+0000","last_fullsized":"2024-09-17T04:49:42.535118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535259+0000","last_change":"2024-09-17T04:49:41.043531+0000","last_active":"2024-09-17T04:49:42.535259+0000","last_peered":"2024-09-17T04:49:42.535259+0000","last_clean":"2024-09-17T04:49:42.535259+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T04:49:42.535259+0000","last_undegraded":"2024-09-17T04:49:42.535259+0000","last_fullsized":"2024-09-17T04:49:42.535259+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672162+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T04:49:42.672162+0000","last_peered":"2024-09-17T04:49:42.672162+0000","last_clean":"2024-09-17T04:49:42.672162+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T04:49:42.672162+0000","last_undegraded":"2024-09-17T04:49:42.672162+0000","last_fullsized":"2024-09-17T04:49:42.672162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042945+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T04:49:42.042945+0000","last_peered":"2024-09-17T04:49:42.042945+0000","last_clean":"2024-09-17T04:49:42.042945+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T04:49:42.042945+0000","last_undegraded":"2024-09-17T04:49:42.042945+0000","last_fullsized":"2024-09-17T04:49:42.042945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535003+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T04:49:42.535003+0000","last_peered":"2024-09-17T04:49:42.535003+0000","last_clean":"2024-09-17T04:49:42.535003+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T04:49:42.535003+0000","last_undegraded":"2024-09-17T04:49:42.535003+0000","last_fullsized":"2024-09-17T04:49:42.535003+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:16.854365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669588+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T04:49:42.669588+0000","last_peered":"2024-09-17T04:49:42.669588+0000","last_clean":"2024-09-17T04:49:42.669588+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T04:49:42.669588+0000","last_undegraded":"2024-09-17T04:49:42.669588+0000","last_fullsized":"2024-09-17T04:49:42.669588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042526+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T04:49:42.042526+0000","last_peered":"2024-09-17T04:49:42.042526+0000","last_clean":"2024-09-17T04:49:42.042526+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T04:49:42.042526+0000","last_undegraded":"2024-09-17T04:49:42.042526+0000","last_fullsized":"2024-09-17T04:49:42.042526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041407+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T04:49:42.041407+0000","last_peered":"2024-09-17T04:49:42.041407+0000","last_clean":"2024-09-17T04:49:42.041407+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T04:49:42.041407+0000","last_undegraded":"2024-09-17T04:49:42.041407+0000","last_fullsized":"2024-09-17T04:49:42.041407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042319+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T04:49:42.042319+0000","last_peered":"2024-09-17T04:49:42.042319+0000","last_clean":"2024-09-17T04:49:42.042319+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T04:49:42.042319+0000","last_undegraded":"2024-09-17T04:49:42.042319+0000","last_fullsized":"2024-09-17T04:49:42.042319+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:28.365399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669339+0000","last_change":"2024-09-17T04:49:41.038415+0000","last_active":"2024-09-17T04:49:42.669339+0000","last_peered":"2024-09-17T04:49:42.669339+0000","last_clean":"2024-09-17T04:49:42.669339+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T04:49:42.669339+0000","last_undegraded":"2024-09-17T04:49:42.669339+0000","last_fullsized":"2024-09-17T04:49:42.669339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:17.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668616+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T04:49:42.668616+0000","last_peered":"2024-09-17T04:49:42.668616+0000","last_clean":"2024-09-17T04:49:42.668616+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T04:49:42.668616+0000","last_undegraded":"2024-09-17T04:49:42.668616+0000","last_fullsized":"2024-09-17T04:49:42.668616+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052248+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T04:49:42.052248+0000","last_peered":"2024-09-17T04:49:42.052248+0000","last_clean":"2024-09-17T04:49:42.052248+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T04:49:42.052248+0000","last_undegraded":"2024-09-17T04:49:42.052248+0000","last_fullsized":"2024-09-17T04:49:42.052248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671586+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T04:49:42.671586+0000","last_peered":"2024-09-17T04:49:42.671586+0000","last_clean":"2024-09-17T04:49:42.671586+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T04:49:42.671586+0000","last_undegraded":"2024-09-17T04:49:42.671586+0000","last_fullsized":"2024-09-17T04:49:42.671586+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:17.424265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042028+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T04:49:42.042028+0000","last_peered":"2024-09-17T04:49:42.042028+0000","last_clean":"2024-09-17T04:49:42.042028+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T04:49:42.042028+0000","last_undegraded":"2024-09-17T04:49:42.042028+0000","last_fullsized":"2024-09-17T04:49:42.042028+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:06.516432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048455+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T04:49:42.048455+0000","last_peered":"2024-09-17T04:49:42.048455+0000","last_clean":"2024-09-17T04:49:42.048455+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T04:49:42.048455+0000","last_undegraded":"2024-09-17T04:49:42.048455+0000","last_fullsized":"2024-09-17T04:49:42.048455+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048500+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T04:49:42.048500+0000","last_peered":"2024-09-17T04:49:42.048500+0000","last_clean":"2024-09-17T04:49:42.048500+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T04:49:42.048500+0000","last_undegraded":"2024-09-17T04:49:42.048500+0000","last_fullsized":"2024-09-17T04:49:42.048500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668827+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T04:49:42.668827+0000","last_peered":"2024-09-17T04:49:42.668827+0000","last_clean":"2024-09-17T04:49:42.668827+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T04:49:42.668827+0000","last_undegraded":"2024-09-17T04:49:42.668827+0000","last_fullsized":"2024-09-17T04:49:42.668827+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:00.010365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675406+0000","last_change":"2024-09-17T04:49:41.036658+0000","last_active":"2024-09-17T04:49:42.675406+0000","last_peered":"2024-09-17T04:49:42.675406+0000","last_clean":"2024-09-17T04:49:42.675406+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T04:49:42.675406+0000","last_undegraded":"2024-09-17T04:49:42.675406+0000","last_fullsized":"2024-09-17T04:49:42.675406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042824+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T04:49:42.042824+0000","last_peered":"2024-09-17T04:49:42.042824+0000","last_clean":"2024-09-17T04:49:42.042824+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T04:49:42.042824+0000","last_undegraded":"2024-09-17T04:49:42.042824+0000","last_fullsized":"2024-09-17T04:49:42.042824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668829+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T04:49:42.668829+0000","last_peered":"2024-09-17T04:49:42.668829+0000","last_clean":"2024-09-17T04:49:42.668829+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T04:49:42.668829+0000","last_undegraded":"2024-09-17T04:49:42.668829+0000","last_fullsized":"2024-09-17T04:49:42.668829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536259+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T04:49:42.536259+0000","last_peered":"2024-09-17T04:49:42.536259+0000","last_clean":"2024-09-17T04:49:42.536259+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T04:49:42.536259+0000","last_undegraded":"2024-09-17T04:49:42.536259+0000","last_fullsized":"2024-09-17T04:49:42.536259+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048942+0000","last_change":"2024-09-17T04:49:41.050170+0000","last_active":"2024-09-17T04:49:42.048942+0000","last_peered":"2024-09-17T04:49:42.048942+0000","last_clean":"2024-09-17T04:49:42.048942+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T04:49:42.048942+0000","last_undegraded":"2024-09-17T04:49:42.048942+0000","last_fullsized":"2024-09-17T04:49:42.048942+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051929+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T04:49:42.051929+0000","last_peered":"2024-09-17T04:49:42.051929+0000","last_clean":"2024-09-17T04:49:42.051929+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T04:49:42.051929+0000","last_undegraded":"2024-09-17T04:49:42.051929+0000","last_fullsized":"2024-09-17T04:49:42.051929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042185+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T04:49:42.042185+0000","last_peered":"2024-09-17T04:49:42.042185+0000","last_clean":"2024-09-17T04:49:42.042185+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T04:49:42.042185+0000","last_undegraded":"2024-09-17T04:49:42.042185+0000","last_fullsized":"2024-09-17T04:49:42.042185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535553+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T04:49:42.535553+0000","last_peered":"2024-09-17T04:49:42.535553+0000","last_clean":"2024-09-17T04:49:42.535553+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T04:49:42.535553+0000","last_undegraded":"2024-09-17T04:49:42.535553+0000","last_fullsized":"2024-09-17T04:49:42.535553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041211+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T04:49:42.041211+0000","last_peered":"2024-09-17T04:49:42.041211+0000","last_clean":"2024-09-17T04:49:42.041211+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T04:49:42.041211+0000","last_undegraded":"2024-09-17T04:49:42.041211+0000","last_fullsized":"2024-09-17T04:49:42.041211+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054336+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T04:49:42.054336+0000","last_peered":"2024-09-17T04:49:42.054336+0000","last_clean":"2024-09-17T04:49:42.054336+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T04:49:42.054336+0000","last_undegraded":"2024-09-17T04:49:42.054336+0000","last_fullsized":"2024-09-17T04:49:42.054336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052826+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T04:49:42.052826+0000","last_peered":"2024-09-17T04:49:42.052826+0000","last_clean":"2024-09-17T04:49:42.052826+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T04:49:42.052826+0000","last_undegraded":"2024-09-17T04:49:42.052826+0000","last_fullsized":"2024-09-17T04:49:42.052826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048145+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T04:49:42.048145+0000","last_peered":"2024-09-17T04:49:42.048145+0000","last_clean":"2024-09-17T04:49:42.048145+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T04:49:42.048145+0000","last_undegraded":"2024-09-17T04:49:42.048145+0000","last_fullsized":"2024-09-17T04:49:42.048145+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669693+0000","last_change":"2024-09-17T04:49:41.042314+0000","last_active":"2024-09-17T04:49:42.669693+0000","last_peered":"2024-09-17T04:49:42.669693+0000","last_clean":"2024-09-17T04:49:42.669693+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T04:49:42.669693+0000","last_undegraded":"2024-09-17T04:49:42.669693+0000","last_fullsized":"2024-09-17T04:49:42.669693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055299+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T04:49:42.055299+0000","last_peered":"2024-09-17T04:49:42.055299+0000","last_clean":"2024-09-17T04:49:42.055299+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T04:49:42.055299+0000","last_undegraded":"2024-09-17T04:49:42.055299+0000","last_fullsized":"2024-09-17T04:49:42.055299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043663+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T04:49:42.043663+0000","last_peered":"2024-09-17T04:49:42.043663+0000","last_clean":"2024-09-17T04:49:42.043663+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T04:49:42.043663+0000","last_undegraded":"2024-09-17T04:49:42.043663+0000","last_fullsized":"2024-09-17T04:49:42.043663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053169+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T04:49:42.053169+0000","last_peered":"2024-09-17T04:49:42.053169+0000","last_clean":"2024-09-17T04:49:42.053169+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T04:49:42.053169+0000","last_undegraded":"2024-09-17T04:49:42.053169+0000","last_fullsized":"2024-09-17T04:49:42.053169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052959+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T04:49:42.052959+0000","last_peered":"2024-09-17T04:49:42.052959+0000","last_clean":"2024-09-17T04:49:42.052959+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T04:49:42.052959+0000","last_undegraded":"2024-09-17T04:49:42.052959+0000","last_fullsized":"2024-09-17T04:49:42.052959+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535986+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T04:49:42.535986+0000","last_peered":"2024-09-17T04:49:42.535986+0000","last_clean":"2024-09-17T04:49:42.535986+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T04:49:42.535986+0000","last_undegraded":"2024-09-17T04:49:42.535986+0000","last_fullsized":"2024-09-17T04:49:42.535986+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670900+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T04:49:42.670900+0000","last_peered":"2024-09-17T04:49:42.670900+0000","last_clean":"2024-09-17T04:49:42.670900+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T04:49:42.670900+0000","last_undegraded":"2024-09-17T04:49:42.670900+0000","last_fullsized":"2024-09-17T04:49:42.670900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050191+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T04:49:42.050191+0000","last_peered":"2024-09-17T04:49:42.050191+0000","last_clean":"2024-09-17T04:49:42.050191+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T04:49:42.050191+0000","last_undegraded":"2024-09-17T04:49:42.050191+0000","last_fullsized":"2024-09-17T04:49:42.050191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670215+0000","last_change":"2024-09-17T04:49:41.069177+0000","last_active":"2024-09-17T04:49:42.670215+0000","last_peered":"2024-09-17T04:49:42.670215+0000","last_clean":"2024-09-17T04:49:42.670215+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T04:49:42.670215+0000","last_undegraded":"2024-09-17T04:49:42.670215+0000","last_fullsized":"2024-09-17T04:49:42.670215+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:16.855206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670381+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T04:49:42.670381+0000","last_peered":"2024-09-17T04:49:42.670381+0000","last_clean":"2024-09-17T04:49:42.670381+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T04:49:42.670381+0000","last_undegraded":"2024-09-17T04:49:42.670381+0000","last_fullsized":"2024-09-17T04:49:42.670381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042706+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T04:49:42.042706+0000","last_peered":"2024-09-17T04:49:42.042706+0000","last_clean":"2024-09-17T04:49:42.042706+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T04:49:42.042706+0000","last_undegraded":"2024-09-17T04:49:42.042706+0000","last_fullsized":"2024-09-17T04:49:42.042706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054900+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T04:49:42.054900+0000","last_peered":"2024-09-17T04:49:42.054900+0000","last_clean":"2024-09-17T04:49:42.054900+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T04:49:42.054900+0000","last_undegraded":"2024-09-17T04:49:42.054900+0000","last_fullsized":"2024-09-17T04:49:42.054900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669308+0000","last_change":"2024-09-17T04:49:41.068537+0000","last_active":"2024-09-17T04:49:42.669308+0000","last_peered":"2024-09-17T04:49:42.669308+0000","last_clean":"2024-09-17T04:49:42.669308+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T04:49:42.669308+0000","last_undegraded":"2024-09-17T04:49:42.669308+0000","last_fullsized":"2024-09-17T04:49:42.669308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667397+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T04:49:42.667397+0000","last_peered":"2024-09-17T04:49:42.667397+0000","last_clean":"2024-09-17T04:49:42.667397+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T04:49:42.667397+0000","last_undegraded":"2024-09-17T04:49:42.667397+0000","last_fullsized":"2024-09-17T04:49:42.667397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668940+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T04:49:42.668940+0000","last_peered":"2024-09-17T04:49:42.668940+0000","last_clean":"2024-09-17T04:49:42.668940+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T04:49:42.668940+0000","last_undegraded":"2024-09-17T04:49:42.668940+0000","last_fullsized":"2024-09-17T04:49:42.668940+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054921+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T04:49:42.054921+0000","last_peered":"2024-09-17T04:49:42.054921+0000","last_clean":"2024-09-17T04:49:42.054921+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T04:49:42.054921+0000","last_undegraded":"2024-09-17T04:49:42.054921+0000","last_fullsized":"2024-09-17T04:49:42.054921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041762+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T04:49:42.041762+0000","last_peered":"2024-09-17T04:49:42.041762+0000","last_clean":"2024-09-17T04:49:42.041762+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T04:49:42.041762+0000","last_undegraded":"2024-09-17T04:49:42.041762+0000","last_fullsized":"2024-09-17T04:49:42.041762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050299+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T04:49:42.050299+0000","last_peered":"2024-09-17T04:49:42.050299+0000","last_clean":"2024-09-17T04:49:42.050299+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T04:49:42.050299+0000","last_undegraded":"2024-09-17T04:49:42.050299+0000","last_fullsized":"2024-09-17T04:49:42.050299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041771+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T04:49:42.041771+0000","last_peered":"2024-09-17T04:49:42.041771+0000","last_clean":"2024-09-17T04:49:42.041771+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T04:49:42.041771+0000","last_undegraded":"2024-09-17T04:49:42.041771+0000","last_fullsized":"2024-09-17T04:49:42.041771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672234+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T04:49:42.672234+0000","last_peered":"2024-09-17T04:49:42.672234+0000","last_clean":"2024-09-17T04:49:42.672234+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T04:49:42.672234+0000","last_undegraded":"2024-09-17T04:49:42.672234+0000","last_fullsized":"2024-09-17T04:49:42.672234+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671183+0000","last_change":"2024-09-17T04:49:41.052564+0000","last_active":"2024-09-17T04:49:42.671183+0000","last_peered":"2024-09-17T04:49:42.671183+0000","last_clean":"2024-09-17T04:49:42.671183+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T04:49:42.671183+0000","last_undegraded":"2024-09-17T04:49:42.671183+0000","last_fullsized":"2024-09-17T04:49:42.671183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049438+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T04:49:42.049438+0000","last_peered":"2024-09-17T04:49:42.049438+0000","last_clean":"2024-09-17T04:49:42.049438+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T04:49:42.049438+0000","last_undegraded":"2024-09-17T04:49:42.049438+0000","last_fullsized":"2024-09-17T04:49:42.049438+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041458+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T04:49:42.041458+0000","last_peered":"2024-09-17T04:49:42.041458+0000","last_clean":"2024-09-17T04:49:42.041458+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T04:49:42.041458+0000","last_undegraded":"2024-09-17T04:49:42.041458+0000","last_fullsized":"2024-09-17T04:49:42.041458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":119,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671735+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T04:49:42.671735+0000","last_peered":"2024-09-17T04:49:42.671735+0000","last_clean":"2024-09-17T04:49:42.671735+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T04:49:42.671735+0000","last_undegraded":"2024-09-17T04:49:42.671735+0000","last_fullsized":"2024-09-17T04:49:42.671735+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043078+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T04:49:42.043078+0000","last_peered":"2024-09-17T04:49:42.043078+0000","last_clean":"2024-09-17T04:49:42.043078+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T04:49:42.043078+0000","last_undegraded":"2024-09-17T04:49:42.043078+0000","last_fullsized":"2024-09-17T04:49:42.043078+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:21.202398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669578+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T04:49:42.669578+0000","last_peered":"2024-09-17T04:49:42.669578+0000","last_clean":"2024-09-17T04:49:42.669578+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T04:49:42.669578+0000","last_undegraded":"2024-09-17T04:49:42.669578+0000","last_fullsized":"2024-09-17T04:49:42.669578+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:48.128609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043280+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T04:49:42.043280+0000","last_peered":"2024-09-17T04:49:42.043280+0000","last_clean":"2024-09-17T04:49:42.043280+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T04:49:42.043280+0000","last_undegraded":"2024-09-17T04:49:42.043280+0000","last_fullsized":"2024-09-17T04:49:42.043280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055292+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T04:49:42.055292+0000","last_peered":"2024-09-17T04:49:42.055292+0000","last_clean":"2024-09-17T04:49:42.055292+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T04:49:42.055292+0000","last_undegraded":"2024-09-17T04:49:42.055292+0000","last_fullsized":"2024-09-17T04:49:42.055292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536164+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T04:49:42.536164+0000","last_peered":"2024-09-17T04:49:42.536164+0000","last_clean":"2024-09-17T04:49:42.536164+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T04:49:42.536164+0000","last_undegraded":"2024-09-17T04:49:42.536164+0000","last_fullsized":"2024-09-17T04:49:42.536164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050257+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T04:49:42.050257+0000","last_peered":"2024-09-17T04:49:42.050257+0000","last_clean":"2024-09-17T04:49:42.050257+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T04:49:42.050257+0000","last_undegraded":"2024-09-17T04:49:42.050257+0000","last_fullsized":"2024-09-17T04:49:42.050257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671927+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T04:49:42.671927+0000","last_peered":"2024-09-17T04:49:42.671927+0000","last_clean":"2024-09-17T04:49:42.671927+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T04:49:42.671927+0000","last_undegraded":"2024-09-17T04:49:42.671927+0000","last_fullsized":"2024-09-17T04:49:42.671927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671411+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T04:49:42.671411+0000","last_peered":"2024-09-17T04:49:42.671411+0000","last_clean":"2024-09-17T04:49:42.671411+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T04:49:42.671411+0000","last_undegraded":"2024-09-17T04:49:42.671411+0000","last_fullsized":"2024-09-17T04:49:42.671411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670087+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T04:49:42.670087+0000","last_peered":"2024-09-17T04:49:42.670087+0000","last_clean":"2024-09-17T04:49:42.670087+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T04:49:42.670087+0000","last_undegraded":"2024-09-17T04:49:42.670087+0000","last_fullsized":"2024-09-17T04:49:42.670087+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:18.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672287+0000","last_change":"2024-09-17T04:49:41.031036+0000","last_active":"2024-09-17T04:49:42.672287+0000","last_peered":"2024-09-17T04:49:42.672287+0000","last_clean":"2024-09-17T04:49:42.672287+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T04:49:42.672287+0000","last_undegraded":"2024-09-17T04:49:42.672287+0000","last_fullsized":"2024-09-17T04:49:42.672287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:48.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672111+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T04:49:42.672111+0000","last_peered":"2024-09-17T04:49:42.672111+0000","last_clean":"2024-09-17T04:49:42.672111+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T04:49:42.672111+0000","last_undegraded":"2024-09-17T04:49:42.672111+0000","last_fullsized":"2024-09-17T04:49:42.672111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052688+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T04:49:42.052688+0000","last_peered":"2024-09-17T04:49:42.052688+0000","last_clean":"2024-09-17T04:49:42.052688+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T04:49:42.052688+0000","last_undegraded":"2024-09-17T04:49:42.052688+0000","last_fullsized":"2024-09-17T04:49:42.052688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041095+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T04:49:42.041095+0000","last_peered":"2024-09-17T04:49:42.041095+0000","last_clean":"2024-09-17T04:49:42.041095+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T04:49:42.041095+0000","last_undegraded":"2024-09-17T04:49:42.041095+0000","last_fullsized":"2024-09-17T04:49:42.041095+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048488+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T04:49:42.048488+0000","last_peered":"2024-09-17T04:49:42.048488+0000","last_clean":"2024-09-17T04:49:42.048488+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T04:49:42.048488+0000","last_undegraded":"2024-09-17T04:49:42.048488+0000","last_fullsized":"2024-09-17T04:49:42.048488+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536967+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T04:49:42.536967+0000","last_peered":"2024-09-17T04:49:42.536967+0000","last_clean":"2024-09-17T04:49:42.536967+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T04:49:42.536967+0000","last_undegraded":"2024-09-17T04:49:42.536967+0000","last_fullsized":"2024-09-17T04:49:42.536967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672404+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T04:49:42.672404+0000","last_peered":"2024-09-17T04:49:42.672404+0000","last_clean":"2024-09-17T04:49:42.672404+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T04:49:42.672404+0000","last_undegraded":"2024-09-17T04:49:42.672404+0000","last_fullsized":"2024-09-17T04:49:42.672404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672966+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T04:49:42.672966+0000","last_peered":"2024-09-17T04:49:42.672966+0000","last_clean":"2024-09-17T04:49:42.672966+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T04:49:42.672966+0000","last_undegraded":"2024-09-17T04:49:42.672966+0000","last_fullsized":"2024-09-17T04:49:42.672966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:09.712272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052007+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T04:49:42.052007+0000","last_peered":"2024-09-17T04:49:42.052007+0000","last_clean":"2024-09-17T04:49:42.052007+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T04:49:42.052007+0000","last_undegraded":"2024-09-17T04:49:42.052007+0000","last_fullsized":"2024-09-17T04:49:42.052007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:43.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042787+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T04:49:42.042787+0000","last_peered":"2024-09-17T04:49:42.042787+0000","last_clean":"2024-09-17T04:49:42.042787+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T04:49:42.042787+0000","last_undegraded":"2024-09-17T04:49:42.042787+0000","last_fullsized":"2024-09-17T04:49:42.042787+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049500+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T04:49:42.049500+0000","last_peered":"2024-09-17T04:49:42.049500+0000","last_clean":"2024-09-17T04:49:42.049500+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T04:49:42.049500+0000","last_undegraded":"2024-09-17T04:49:42.049500+0000","last_fullsized":"2024-09-17T04:49:42.049500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671253+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T04:49:42.671253+0000","last_peered":"2024-09-17T04:49:42.671253+0000","last_clean":"2024-09-17T04:49:42.671253+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T04:49:42.671253+0000","last_undegraded":"2024-09-17T04:49:42.671253+0000","last_fullsized":"2024-09-17T04:49:42.671253+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:52.364568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041504+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T04:49:42.041504+0000","last_peered":"2024-09-17T04:49:42.041504+0000","last_clean":"2024-09-17T04:49:42.041504+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T04:49:42.041504+0000","last_undegraded":"2024-09-17T04:49:42.041504+0000","last_fullsized":"2024-09-17T04:49:42.041504+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:03.537392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671439+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T04:49:42.671439+0000","last_peered":"2024-09-17T04:49:42.671439+0000","last_clean":"2024-09-17T04:49:42.671439+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T04:49:42.671439+0000","last_undegraded":"2024-09-17T04:49:42.671439+0000","last_fullsized":"2024-09-17T04:49:42.671439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670943+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T04:49:42.670943+0000","last_peered":"2024-09-17T04:49:42.670943+0000","last_clean":"2024-09-17T04:49:42.670943+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T04:49:42.670943+0000","last_undegraded":"2024-09-17T04:49:42.670943+0000","last_fullsized":"2024-09-17T04:49:42.670943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671787+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T04:49:42.671787+0000","last_peered":"2024-09-17T04:49:42.671787+0000","last_clean":"2024-09-17T04:49:42.671787+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T04:49:42.671787+0000","last_undegraded":"2024-09-17T04:49:42.671787+0000","last_fullsized":"2024-09-17T04:49:42.671787+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:52.120543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041402+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T04:49:42.041402+0000","last_peered":"2024-09-17T04:49:42.041402+0000","last_clean":"2024-09-17T04:49:42.041402+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T04:49:42.041402+0000","last_undegraded":"2024-09-17T04:49:42.041402+0000","last_fullsized":"2024-09-17T04:49:42.041402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:56.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041285+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T04:49:42.041285+0000","last_peered":"2024-09-17T04:49:42.041285+0000","last_clean":"2024-09-17T04:49:42.041285+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T04:49:42.041285+0000","last_undegraded":"2024-09-17T04:49:42.041285+0000","last_fullsized":"2024-09-17T04:49:42.041285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042566+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T04:49:42.042566+0000","last_peered":"2024-09-17T04:49:42.042566+0000","last_clean":"2024-09-17T04:49:42.042566+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T04:49:42.042566+0000","last_undegraded":"2024-09-17T04:49:42.042566+0000","last_fullsized":"2024-09-17T04:49:42.042566+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"25'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054288+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T04:49:42.054288+0000","last_peered":"2024-09-17T04:49:42.054288+0000","last_clean":"2024-09-17T04:49:42.054288+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T04:49:42.054288+0000","last_undegraded":"2024-09-17T04:49:42.054288+0000","last_fullsized":"2024-09-17T04:49:42.054288+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051153+0000","last_change":"2024-09-17T04:49:41.058280+0000","last_active":"2024-09-17T04:49:42.051153+0000","last_peered":"2024-09-17T04:49:42.051153+0000","last_clean":"2024-09-17T04:49:42.051153+0000","last_became_active":"2024-09-17T04:49:41.052009+0000","last_became_peered":"2024-09-17T04:49:41.052009+0000","last_unstale":"2024-09-17T04:49:42.051153+0000","last_undegraded":"2024-09-17T04:49:42.051153+0000","last_fullsized":"2024-09-17T04:49:42.051153+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:14.100818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051977+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T04:49:42.051977+0000","last_peered":"2024-09-17T04:49:42.051977+0000","last_clean":"2024-09-17T04:49:42.051977+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T04:49:42.051977+0000","last_undegraded":"2024-09-17T04:49:42.051977+0000","last_fullsized":"2024-09-17T04:49:42.051977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669355+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T04:49:42.669355+0000","last_peered":"2024-09-17T04:49:42.669355+0000","last_clean":"2024-09-17T04:49:42.669355+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T04:49:42.669355+0000","last_undegraded":"2024-09-17T04:49:42.669355+0000","last_fullsized":"2024-09-17T04:49:42.669355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"25'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042782+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T04:49:42.042782+0000","last_peered":"2024-09-17T04:49:42.042782+0000","last_clean":"2024-09-17T04:49:42.042782+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T04:49:42.042782+0000","last_undegraded":"2024-09-17T04:49:42.042782+0000","last_fullsized":"2024-09-17T04:49:42.042782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:07.945355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052280+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T04:49:42.052280+0000","last_peered":"2024-09-17T04:49:42.052280+0000","last_clean":"2024-09-17T04:49:42.052280+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T04:49:42.052280+0000","last_undegraded":"2024-09-17T04:49:42.052280+0000","last_fullsized":"2024-09-17T04:49:42.052280+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670412+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T04:49:42.670412+0000","last_peered":"2024-09-17T04:49:42.670412+0000","last_clean":"2024-09-17T04:49:42.670412+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T04:49:42.670412+0000","last_undegraded":"2024-09-17T04:49:42.670412+0000","last_fullsized":"2024-09-17T04:49:42.670412+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"26'8","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:50:02.209562+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T04:50:02.209562+0000","last_peered":"2024-09-17T04:50:02.209562+0000","last_clean":"2024-09-17T04:50:02.209562+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T04:50:02.209562+0000","last_undegraded":"2024-09-17T04:50:02.209562+0000","last_fullsized":"2024-09-17T04:50:02.209562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048913+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T04:49:42.048913+0000","last_peered":"2024-09-17T04:49:42.048913+0000","last_clean":"2024-09-17T04:49:42.048913+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T04:49:42.048913+0000","last_undegraded":"2024-09-17T04:49:42.048913+0000","last_fullsized":"2024-09-17T04:49:42.048913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:16.447835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051373+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T04:49:42.051373+0000","last_peered":"2024-09-17T04:49:42.051373+0000","last_clean":"2024-09-17T04:49:42.051373+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T04:49:42.051373+0000","last_undegraded":"2024-09-17T04:49:42.051373+0000","last_fullsized":"2024-09-17T04:49:42.051373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042063+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T04:49:42.042063+0000","last_peered":"2024-09-17T04:49:42.042063+0000","last_clean":"2024-09-17T04:49:42.042063+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T04:49:42.042063+0000","last_undegraded":"2024-09-17T04:49:42.042063+0000","last_fullsized":"2024-09-17T04:49:42.042063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670596+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T04:49:42.670596+0000","last_peered":"2024-09-17T04:49:42.670596+0000","last_clean":"2024-09-17T04:49:42.670596+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T04:49:42.670596+0000","last_undegraded":"2024-09-17T04:49:42.670596+0000","last_fullsized":"2024-09-17T04:49:42.670596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535510+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T04:49:42.535510+0000","last_peered":"2024-09-17T04:49:42.535510+0000","last_clean":"2024-09-17T04:49:42.535510+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T04:49:42.535510+0000","last_undegraded":"2024-09-17T04:49:42.535510+0000","last_fullsized":"2024-09-17T04:49:42.535510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041322+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T04:49:42.041322+0000","last_peered":"2024-09-17T04:49:42.041322+0000","last_clean":"2024-09-17T04:49:42.041322+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T04:49:42.041322+0000","last_undegraded":"2024-09-17T04:49:42.041322+0000","last_fullsized":"2024-09-17T04:49:42.041322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672182+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T04:49:42.672182+0000","last_peered":"2024-09-17T04:49:42.672182+0000","last_clean":"2024-09-17T04:49:42.672182+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T04:49:42.672182+0000","last_undegraded":"2024-09-17T04:49:42.672182+0000","last_fullsized":"2024-09-17T04:49:42.672182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671261+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T04:49:42.671261+0000","last_peered":"2024-09-17T04:49:42.671261+0000","last_clean":"2024-09-17T04:49:42.671261+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T04:49:42.671261+0000","last_undegraded":"2024-09-17T04:49:42.671261+0000","last_fullsized":"2024-09-17T04:49:42.671261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"25'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667575+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T04:49:42.667575+0000","last_peered":"2024-09-17T04:49:42.667575+0000","last_clean":"2024-09-17T04:49:42.667575+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T04:49:42.667575+0000","last_undegraded":"2024-09-17T04:49:42.667575+0000","last_fullsized":"2024-09-17T04:49:42.667575+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:40.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534654+0000","last_change":"2024-09-17T04:49:41.040278+0000","last_active":"2024-09-17T04:49:42.534654+0000","last_peered":"2024-09-17T04:49:42.534654+0000","last_clean":"2024-09-17T04:49:42.534654+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T04:49:42.534654+0000","last_undegraded":"2024-09-17T04:49:42.534654+0000","last_fullsized":"2024-09-17T04:49:42.534654+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.895223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536732+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T04:49:42.536732+0000","last_peered":"2024-09-17T04:49:42.536732+0000","last_clean":"2024-09-17T04:49:42.536732+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T04:49:42.536732+0000","last_undegraded":"2024-09-17T04:49:42.536732+0000","last_fullsized":"2024-09-17T04:49:42.536732+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675753+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T04:49:42.675753+0000","last_peered":"2024-09-17T04:49:42.675753+0000","last_clean":"2024-09-17T04:49:42.675753+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T04:49:42.675753+0000","last_undegraded":"2024-09-17T04:49:42.675753+0000","last_fullsized":"2024-09-17T04:49:42.675753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.047646+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T04:49:42.047646+0000","last_peered":"2024-09-17T04:49:42.047646+0000","last_clean":"2024-09-17T04:49:42.047646+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T04:49:42.047646+0000","last_undegraded":"2024-09-17T04:49:42.047646+0000","last_fullsized":"2024-09-17T04:49:42.047646+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:15.845612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537298+0000","last_change":"2024-09-17T04:49:41.050551+0000","last_active":"2024-09-17T04:49:42.537298+0000","last_peered":"2024-09-17T04:49:42.537298+0000","last_clean":"2024-09-17T04:49:42.537298+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T04:49:42.537298+0000","last_undegraded":"2024-09-17T04:49:42.537298+0000","last_fullsized":"2024-09-17T04:49:42.537298+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:11.088230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"25'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051592+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T04:49:42.051592+0000","last_peered":"2024-09-17T04:49:42.051592+0000","last_clean":"2024-09-17T04:49:42.051592+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T04:49:42.051592+0000","last_undegraded":"2024-09-17T04:49:42.051592+0000","last_fullsized":"2024-09-17T04:49:42.051592+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:37.778756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048401+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T04:49:42.048401+0000","last_peered":"2024-09-17T04:49:42.048401+0000","last_clean":"2024-09-17T04:49:42.048401+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T04:49:42.048401+0000","last_undegraded":"2024-09-17T04:49:42.048401+0000","last_fullsized":"2024-09-17T04:49:42.048401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041797+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T04:49:42.041797+0000","last_peered":"2024-09-17T04:49:42.041797+0000","last_clean":"2024-09-17T04:49:42.041797+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T04:49:42.041797+0000","last_undegraded":"2024-09-17T04:49:42.041797+0000","last_fullsized":"2024-09-17T04:49:42.041797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.055365+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T04:49:42.055365+0000","last_peered":"2024-09-17T04:49:42.055365+0000","last_clean":"2024-09-17T04:49:42.055365+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T04:49:42.055365+0000","last_undegraded":"2024-09-17T04:49:42.055365+0000","last_fullsized":"2024-09-17T04:49:42.055365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"25'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041767+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T04:49:42.041767+0000","last_peered":"2024-09-17T04:49:42.041767+0000","last_clean":"2024-09-17T04:49:42.041767+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T04:49:42.041767+0000","last_undegraded":"2024-09-17T04:49:42.041767+0000","last_fullsized":"2024-09-17T04:49:42.041767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667708+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T04:49:42.667708+0000","last_peered":"2024-09-17T04:49:42.667708+0000","last_clean":"2024-09-17T04:49:42.667708+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T04:49:42.667708+0000","last_undegraded":"2024-09-17T04:49:42.667708+0000","last_fullsized":"2024-09-17T04:49:42.667708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050522+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T04:49:42.050522+0000","last_peered":"2024-09-17T04:49:42.050522+0000","last_clean":"2024-09-17T04:49:42.050522+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T04:49:42.050522+0000","last_undegraded":"2024-09-17T04:49:42.050522+0000","last_fullsized":"2024-09-17T04:49:42.050522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.049954+0000","last_change":"2024-09-17T04:49:41.057647+0000","last_active":"2024-09-17T04:49:42.049954+0000","last_peered":"2024-09-17T04:49:42.049954+0000","last_clean":"2024-09-17T04:49:42.049954+0000","last_became_active":"2024-09-17T04:49:41.056213+0000","last_became_peered":"2024-09-17T04:49:41.056213+0000","last_unstale":"2024-09-17T04:49:42.049954+0000","last_undegraded":"2024-09-17T04:49:42.049954+0000","last_fullsized":"2024-09-17T04:49:42.049954+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042498+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T04:49:42.042498+0000","last_peered":"2024-09-17T04:49:42.042498+0000","last_clean":"2024-09-17T04:49:42.042498+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T04:49:42.042498+0000","last_undegraded":"2024-09-17T04:49:42.042498+0000","last_fullsized":"2024-09-17T04:49:42.042498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053714+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T04:49:42.053714+0000","last_peered":"2024-09-17T04:49:42.053714+0000","last_clean":"2024-09-17T04:49:42.053714+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T04:49:42.053714+0000","last_undegraded":"2024-09-17T04:49:42.053714+0000","last_fullsized":"2024-09-17T04:49:42.053714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051282+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T04:49:42.051282+0000","last_peered":"2024-09-17T04:49:42.051282+0000","last_clean":"2024-09-17T04:49:42.051282+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T04:49:42.051282+0000","last_undegraded":"2024-09-17T04:49:42.051282+0000","last_fullsized":"2024-09-17T04:49:42.051282+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:27.230604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536526+0000","last_change":"2024-09-17T04:49:41.055263+0000","last_active":"2024-09-17T04:49:42.536526+0000","last_peered":"2024-09-17T04:49:42.536526+0000","last_clean":"2024-09-17T04:49:42.536526+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T04:49:42.536526+0000","last_undegraded":"2024-09-17T04:49:42.536526+0000","last_fullsized":"2024-09-17T04:49:42.536526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670438+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T04:49:42.670438+0000","last_peered":"2024-09-17T04:49:42.670438+0000","last_clean":"2024-09-17T04:49:42.670438+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T04:49:42.670438+0000","last_undegraded":"2024-09-17T04:49:42.670438+0000","last_fullsized":"2024-09-17T04:49:42.670438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673048+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T04:49:42.673048+0000","last_peered":"2024-09-17T04:49:42.673048+0000","last_clean":"2024-09-17T04:49:42.673048+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T04:49:42.673048+0000","last_undegraded":"2024-09-17T04:49:42.673048+0000","last_fullsized":"2024-09-17T04:49:42.673048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"25'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670447+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T04:49:42.670447+0000","last_peered":"2024-09-17T04:49:42.670447+0000","last_clean":"2024-09-17T04:49:42.670447+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T04:49:42.670447+0000","last_undegraded":"2024-09-17T04:49:42.670447+0000","last_fullsized":"2024-09-17T04:49:42.670447+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:24.586834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051128+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.051128+0000","last_peered":"2024-09-17T04:49:42.051128+0000","last_clean":"2024-09-17T04:49:42.051128+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T04:49:42.051128+0000","last_undegraded":"2024-09-17T04:49:42.051128+0000","last_fullsized":"2024-09-17T04:49:42.051128+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536163+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T04:49:42.536163+0000","last_peered":"2024-09-17T04:49:42.536163+0000","last_clean":"2024-09-17T04:49:42.536163+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T04:49:42.536163+0000","last_undegraded":"2024-09-17T04:49:42.536163+0000","last_fullsized":"2024-09-17T04:49:42.536163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052121+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T04:49:42.052121+0000","last_peered":"2024-09-17T04:49:42.052121+0000","last_clean":"2024-09-17T04:49:42.052121+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T04:49:42.052121+0000","last_undegraded":"2024-09-17T04:49:42.052121+0000","last_fullsized":"2024-09-17T04:49:42.052121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668467+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T04:49:42.668467+0000","last_peered":"2024-09-17T04:49:42.668467+0000","last_clean":"2024-09-17T04:49:42.668467+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T04:49:42.668467+0000","last_undegraded":"2024-09-17T04:49:42.668467+0000","last_fullsized":"2024-09-17T04:49:42.668467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536096+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T04:49:42.536096+0000","last_peered":"2024-09-17T04:49:42.536096+0000","last_clean":"2024-09-17T04:49:42.536096+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T04:49:42.536096+0000","last_undegraded":"2024-09-17T04:49:42.536096+0000","last_fullsized":"2024-09-17T04:49:42.536096+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042322+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T04:49:42.042322+0000","last_peered":"2024-09-17T04:49:42.042322+0000","last_clean":"2024-09-17T04:49:42.042322+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T04:49:42.042322+0000","last_undegraded":"2024-09-17T04:49:42.042322+0000","last_fullsized":"2024-09-17T04:49:42.042322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671200+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T04:49:42.671200+0000","last_peered":"2024-09-17T04:49:42.671200+0000","last_clean":"2024-09-17T04:49:42.671200+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T04:49:42.671200+0000","last_undegraded":"2024-09-17T04:49:42.671200+0000","last_fullsized":"2024-09-17T04:49:42.671200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670309+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T04:49:42.670309+0000","last_peered":"2024-09-17T04:49:42.670309+0000","last_clean":"2024-09-17T04:49:42.670309+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T04:49:42.670309+0000","last_undegraded":"2024-09-17T04:49:42.670309+0000","last_fullsized":"2024-09-17T04:49:42.670309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668969+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T04:49:42.668969+0000","last_peered":"2024-09-17T04:49:42.668969+0000","last_clean":"2024-09-17T04:49:42.668969+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T04:49:42.668969+0000","last_undegraded":"2024-09-17T04:49:42.668969+0000","last_fullsized":"2024-09-17T04:49:42.668969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054935+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T04:49:42.054935+0000","last_peered":"2024-09-17T04:49:42.054935+0000","last_clean":"2024-09-17T04:49:42.054935+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T04:49:42.054935+0000","last_undegraded":"2024-09-17T04:49:42.054935+0000","last_fullsized":"2024-09-17T04:49:42.054935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669059+0000","last_change":"2024-09-17T04:49:41.073709+0000","last_active":"2024-09-17T04:49:42.669059+0000","last_peered":"2024-09-17T04:49:42.669059+0000","last_clean":"2024-09-17T04:49:42.669059+0000","last_became_active":"2024-09-17T04:49:41.073330+0000","last_became_peered":"2024-09-17T04:49:41.073330+0000","last_unstale":"2024-09-17T04:49:42.669059+0000","last_undegraded":"2024-09-17T04:49:42.669059+0000","last_fullsized":"2024-09-17T04:49:42.669059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054811+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T04:49:42.054811+0000","last_peered":"2024-09-17T04:49:42.054811+0000","last_clean":"2024-09-17T04:49:42.054811+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T04:49:42.054811+0000","last_undegraded":"2024-09-17T04:49:42.054811+0000","last_fullsized":"2024-09-17T04:49:42.054811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041256+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T04:49:42.041256+0000","last_peered":"2024-09-17T04:49:42.041256+0000","last_clean":"2024-09-17T04:49:42.041256+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T04:49:42.041256+0000","last_undegraded":"2024-09-17T04:49:42.041256+0000","last_fullsized":"2024-09-17T04:49:42.041256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536669+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T04:49:42.536669+0000","last_peered":"2024-09-17T04:49:42.536669+0000","last_clean":"2024-09-17T04:49:42.536669+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T04:49:42.536669+0000","last_undegraded":"2024-09-17T04:49:42.536669+0000","last_fullsized":"2024-09-17T04:49:42.536669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534968+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T04:49:42.534968+0000","last_peered":"2024-09-17T04:49:42.534968+0000","last_clean":"2024-09-17T04:49:42.534968+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T04:49:42.534968+0000","last_undegraded":"2024-09-17T04:49:42.534968+0000","last_fullsized":"2024-09-17T04:49:42.534968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.672019+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T04:49:42.672019+0000","last_peered":"2024-09-17T04:49:42.672019+0000","last_clean":"2024-09-17T04:49:42.672019+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T04:49:42.672019+0000","last_undegraded":"2024-09-17T04:49:42.672019+0000","last_fullsized":"2024-09-17T04:49:42.672019+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668565+0000","last_change":"2024-09-17T04:49:41.045502+0000","last_active":"2024-09-17T04:49:42.668565+0000","last_peered":"2024-09-17T04:49:42.668565+0000","last_clean":"2024-09-17T04:49:42.668565+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T04:49:42.668565+0000","last_undegraded":"2024-09-17T04:49:42.668565+0000","last_fullsized":"2024-09-17T04:49:42.668565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:19.746356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041666+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T04:49:42.041666+0000","last_peered":"2024-09-17T04:49:42.041666+0000","last_clean":"2024-09-17T04:49:42.041666+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T04:49:42.041666+0000","last_undegraded":"2024-09-17T04:49:42.041666+0000","last_fullsized":"2024-09-17T04:49:42.041666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042310+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T04:49:42.042310+0000","last_peered":"2024-09-17T04:49:42.042310+0000","last_clean":"2024-09-17T04:49:42.042310+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T04:49:42.042310+0000","last_undegraded":"2024-09-17T04:49:42.042310+0000","last_fullsized":"2024-09-17T04:49:42.042310+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668045+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T04:49:42.668045+0000","last_peered":"2024-09-17T04:49:42.668045+0000","last_clean":"2024-09-17T04:49:42.668045+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T04:49:42.668045+0000","last_undegraded":"2024-09-17T04:49:42.668045+0000","last_fullsized":"2024-09-17T04:49:42.668045+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.054699+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T04:49:42.054699+0000","last_peered":"2024-09-17T04:49:42.054699+0000","last_clean":"2024-09-17T04:49:42.054699+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T04:49:42.054699+0000","last_undegraded":"2024-09-17T04:49:42.054699+0000","last_fullsized":"2024-09-17T04:49:42.054699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673038+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T04:49:42.673038+0000","last_peered":"2024-09-17T04:49:42.673038+0000","last_clean":"2024-09-17T04:49:42.673038+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T04:49:42.673038+0000","last_undegraded":"2024-09-17T04:49:42.673038+0000","last_fullsized":"2024-09-17T04:49:42.673038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669104+0000","last_change":"2024-09-17T04:49:41.038265+0000","last_active":"2024-09-17T04:49:42.669104+0000","last_peered":"2024-09-17T04:49:42.669104+0000","last_clean":"2024-09-17T04:49:42.669104+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T04:49:42.669104+0000","last_undegraded":"2024-09-17T04:49:42.669104+0000","last_fullsized":"2024-09-17T04:49:42.669104+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:41.638403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053416+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T04:49:42.053416+0000","last_peered":"2024-09-17T04:49:42.053416+0000","last_clean":"2024-09-17T04:49:42.053416+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T04:49:42.053416+0000","last_undegraded":"2024-09-17T04:49:42.053416+0000","last_fullsized":"2024-09-17T04:49:42.053416+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535001+0000","last_change":"2024-09-17T04:49:41.053443+0000","last_active":"2024-09-17T04:49:42.535001+0000","last_peered":"2024-09-17T04:49:42.535001+0000","last_clean":"2024-09-17T04:49:42.535001+0000","last_became_active":"2024-09-17T04:49:41.053090+0000","last_became_peered":"2024-09-17T04:49:41.053090+0000","last_unstale":"2024-09-17T04:49:42.535001+0000","last_undegraded":"2024-09-17T04:49:42.535001+0000","last_fullsized":"2024-09-17T04:49:42.535001+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:39.798951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671599+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T04:49:42.671599+0000","last_peered":"2024-09-17T04:49:42.671599+0000","last_clean":"2024-09-17T04:49:42.671599+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T04:49:42.671599+0000","last_undegraded":"2024-09-17T04:49:42.671599+0000","last_fullsized":"2024-09-17T04:49:42.671599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669669+0000","last_change":"2024-09-17T04:49:41.038072+0000","last_active":"2024-09-17T04:49:42.669669+0000","last_peered":"2024-09-17T04:49:42.669669+0000","last_clean":"2024-09-17T04:49:42.669669+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T04:49:42.669669+0000","last_undegraded":"2024-09-17T04:49:42.669669+0000","last_fullsized":"2024-09-17T04:49:42.669669+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.043492+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T04:49:42.043492+0000","last_peered":"2024-09-17T04:49:42.043492+0000","last_clean":"2024-09-17T04:49:42.043492+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T04:49:42.043492+0000","last_undegraded":"2024-09-17T04:49:42.043492+0000","last_fullsized":"2024-09-17T04:49:42.043492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534462+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T04:49:42.534462+0000","last_peered":"2024-09-17T04:49:42.534462+0000","last_clean":"2024-09-17T04:49:42.534462+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T04:49:42.534462+0000","last_undegraded":"2024-09-17T04:49:42.534462+0000","last_fullsized":"2024-09-17T04:49:42.534462+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669602+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T04:49:42.669602+0000","last_peered":"2024-09-17T04:49:42.669602+0000","last_clean":"2024-09-17T04:49:42.669602+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T04:49:42.669602+0000","last_undegraded":"2024-09-17T04:49:42.669602+0000","last_fullsized":"2024-09-17T04:49:42.669602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050316+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T04:49:42.050316+0000","last_peered":"2024-09-17T04:49:42.050316+0000","last_clean":"2024-09-17T04:49:42.050316+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T04:49:42.050316+0000","last_undegraded":"2024-09-17T04:49:42.050316+0000","last_fullsized":"2024-09-17T04:49:42.050316+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.042123+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T04:49:42.042123+0000","last_peered":"2024-09-17T04:49:42.042123+0000","last_clean":"2024-09-17T04:49:42.042123+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T04:49:42.042123+0000","last_undegraded":"2024-09-17T04:49:42.042123+0000","last_fullsized":"2024-09-17T04:49:42.042123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.052378+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T04:49:42.052378+0000","last_peered":"2024-09-17T04:49:42.052378+0000","last_clean":"2024-09-17T04:49:42.052378+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T04:49:42.052378+0000","last_undegraded":"2024-09-17T04:49:42.052378+0000","last_fullsized":"2024-09-17T04:49:42.052378+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041588+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T04:49:42.041588+0000","last_peered":"2024-09-17T04:49:42.041588+0000","last_clean":"2024-09-17T04:49:42.041588+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T04:49:42.041588+0000","last_undegraded":"2024-09-17T04:49:42.041588+0000","last_fullsized":"2024-09-17T04:49:42.041588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.674897+0000","last_change":"2024-09-17T04:49:41.067207+0000","last_active":"2024-09-17T04:49:42.674897+0000","last_peered":"2024-09-17T04:49:42.674897+0000","last_clean":"2024-09-17T04:49:42.674897+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T04:49:42.674897+0000","last_undegraded":"2024-09-17T04:49:42.674897+0000","last_fullsized":"2024-09-17T04:49:42.674897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:08.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667484+0000","last_change":"2024-09-17T04:49:31.911036+0000","last_active":"2024-09-17T04:49:42.667484+0000","last_peered":"2024-09-17T04:49:42.667484+0000","last_clean":"2024-09-17T04:49:42.667484+0000","last_became_active":"2024-09-17T04:49:31.910285+0000","last_became_peered":"2024-09-17T04:49:31.910285+0000","last_unstale":"2024-09-17T04:49:42.667484+0000","last_undegraded":"2024-09-17T04:49:42.667484+0000","last_fullsized":"2024-09-17T04:49:42.667484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673474+0000","last_change":"2024-09-17T04:49:41.046170+0000","last_active":"2024-09-17T04:49:42.673474+0000","last_peered":"2024-09-17T04:49:42.673474+0000","last_clean":"2024-09-17T04:49:42.673474+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T04:49:42.673474+0000","last_undegraded":"2024-09-17T04:49:42.673474+0000","last_fullsized":"2024-09-17T04:49:42.673474+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:01.683826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.051528+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T04:49:42.051528+0000","last_peered":"2024-09-17T04:49:42.051528+0000","last_clean":"2024-09-17T04:49:42.051528+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T04:49:42.051528+0000","last_undegraded":"2024-09-17T04:49:42.051528+0000","last_fullsized":"2024-09-17T04:49:42.051528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.041843+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T04:49:42.041843+0000","last_peered":"2024-09-17T04:49:42.041843+0000","last_clean":"2024-09-17T04:49:42.041843+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T04:49:42.041843+0000","last_undegraded":"2024-09-17T04:49:42.041843+0000","last_fullsized":"2024-09-17T04:49:42.041843+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:25.574682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053942+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T04:49:42.053942+0000","last_peered":"2024-09-17T04:49:42.053942+0000","last_clean":"2024-09-17T04:49:42.053942+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T04:49:42.053942+0000","last_undegraded":"2024-09-17T04:49:42.053942+0000","last_fullsized":"2024-09-17T04:49:42.053942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.048414+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T04:49:42.048414+0000","last_peered":"2024-09-17T04:49:42.048414+0000","last_clean":"2024-09-17T04:49:42.048414+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T04:49:42.048414+0000","last_undegraded":"2024-09-17T04:49:42.048414+0000","last_fullsized":"2024-09-17T04:49:42.048414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671218+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T04:49:42.671218+0000","last_peered":"2024-09-17T04:49:42.671218+0000","last_clean":"2024-09-17T04:49:42.671218+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T04:49:42.671218+0000","last_undegraded":"2024-09-17T04:49:42.671218+0000","last_fullsized":"2024-09-17T04:49:42.671218+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.053498+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T04:49:42.053498+0000","last_peered":"2024-09-17T04:49:42.053498+0000","last_clean":"2024-09-17T04:49:42.053498+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T04:49:42.053498+0000","last_undegraded":"2024-09-17T04:49:42.053498+0000","last_fullsized":"2024-09-17T04:49:42.053498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670119+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T04:49:42.670119+0000","last_peered":"2024-09-17T04:49:42.670119+0000","last_clean":"2024-09-17T04:49:42.670119+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T04:49:42.670119+0000","last_undegraded":"2024-09-17T04:49:42.670119+0000","last_fullsized":"2024-09-17T04:49:42.670119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.668585+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T04:49:42.668585+0000","last_peered":"2024-09-17T04:49:42.668585+0000","last_clean":"2024-09-17T04:49:42.668585+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T04:49:42.668585+0000","last_undegraded":"2024-09-17T04:49:42.668585+0000","last_fullsized":"2024-09-17T04:49:42.668585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670988+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T04:49:42.670988+0000","last_peered":"2024-09-17T04:49:42.670988+0000","last_clean":"2024-09-17T04:49:42.670988+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T04:49:42.670988+0000","last_undegraded":"2024-09-17T04:49:42.670988+0000","last_fullsized":"2024-09-17T04:49:42.670988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671416+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T04:49:42.671416+0000","last_peered":"2024-09-17T04:49:42.671416+0000","last_clean":"2024-09-17T04:49:42.671416+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T04:49:42.671416+0000","last_undegraded":"2024-09-17T04:49:42.671416+0000","last_fullsized":"2024-09-17T04:49:42.671416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.675766+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T04:49:42.675766+0000","last_peered":"2024-09-17T04:49:42.675766+0000","last_clean":"2024-09-17T04:49:42.675766+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T04:49:42.675766+0000","last_undegraded":"2024-09-17T04:49:42.675766+0000","last_fullsized":"2024-09-17T04:49:42.675766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671330+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T04:49:42.671330+0000","last_peered":"2024-09-17T04:49:42.671330+0000","last_clean":"2024-09-17T04:49:42.671330+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T04:49:42.671330+0000","last_undegraded":"2024-09-17T04:49:42.671330+0000","last_fullsized":"2024-09-17T04:49:42.671330+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667310+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T04:49:42.667310+0000","last_peered":"2024-09-17T04:49:42.667310+0000","last_clean":"2024-09-17T04:49:42.667310+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T04:49:42.667310+0000","last_undegraded":"2024-09-17T04:49:42.667310+0000","last_fullsized":"2024-09-17T04:49:42.667310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.537133+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T04:49:42.537133+0000","last_peered":"2024-09-17T04:49:42.537133+0000","last_clean":"2024-09-17T04:49:42.537133+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T04:49:42.537133+0000","last_undegraded":"2024-09-17T04:49:42.537133+0000","last_fullsized":"2024-09-17T04:49:42.537133+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669347+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T04:49:42.669347+0000","last_peered":"2024-09-17T04:49:42.669347+0000","last_clean":"2024-09-17T04:49:42.669347+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T04:49:42.669347+0000","last_undegraded":"2024-09-17T04:49:42.669347+0000","last_fullsized":"2024-09-17T04:49:42.669347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.050809+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T04:49:42.050809+0000","last_peered":"2024-09-17T04:49:42.050809+0000","last_clean":"2024-09-17T04:49:42.050809+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T04:49:42.050809+0000","last_undegraded":"2024-09-17T04:49:42.050809+0000","last_fullsized":"2024-09-17T04:49:42.050809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.535865+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T04:49:42.535865+0000","last_peered":"2024-09-17T04:49:42.535865+0000","last_clean":"2024-09-17T04:49:42.535865+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T04:49:42.535865+0000","last_undegraded":"2024-09-17T04:49:42.535865+0000","last_fullsized":"2024-09-17T04:49:42.535865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.673274+0000","last_change":"2024-09-17T04:49:41.041201+0000","last_active":"2024-09-17T04:49:42.673274+0000","last_peered":"2024-09-17T04:49:42.673274+0000","last_clean":"2024-09-17T04:49:42.673274+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T04:49:42.673274+0000","last_undegraded":"2024-09-17T04:49:42.673274+0000","last_fullsized":"2024-09-17T04:49:42.673274+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:02.588786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.670272+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T04:49:42.670272+0000","last_peered":"2024-09-17T04:49:42.670272+0000","last_clean":"2024-09-17T04:49:42.670272+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T04:49:42.670272+0000","last_undegraded":"2024-09-17T04:49:42.670272+0000","last_fullsized":"2024-09-17T04:49:42.670272+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.667767+0000","last_change":"2024-09-17T04:49:41.061092+0000","last_active":"2024-09-17T04:49:42.667767+0000","last_peered":"2024-09-17T04:49:42.667767+0000","last_clean":"2024-09-17T04:49:42.667767+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T04:49:42.667767+0000","last_undegraded":"2024-09-17T04:49:42.667767+0000","last_fullsized":"2024-09-17T04:49:42.667767+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:52.714614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536820+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T04:49:42.536820+0000","last_peered":"2024-09-17T04:49:42.536820+0000","last_clean":"2024-09-17T04:49:42.536820+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T04:49:42.536820+0000","last_undegraded":"2024-09-17T04:49:42.536820+0000","last_fullsized":"2024-09-17T04:49:42.536820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.669397+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T04:49:42.669397+0000","last_peered":"2024-09-17T04:49:42.669397+0000","last_clean":"2024-09-17T04:49:42.669397+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T04:49:42.669397+0000","last_undegraded":"2024-09-17T04:49:42.669397+0000","last_fullsized":"2024-09-17T04:49:42.669397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671170+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T04:49:42.671170+0000","last_peered":"2024-09-17T04:49:42.671170+0000","last_clean":"2024-09-17T04:49:42.671170+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T04:49:42.671170+0000","last_undegraded":"2024-09-17T04:49:42.671170+0000","last_fullsized":"2024-09-17T04:49:42.671170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.536018+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T04:49:42.536018+0000","last_peered":"2024-09-17T04:49:42.536018+0000","last_clean":"2024-09-17T04:49:42.536018+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T04:49:42.536018+0000","last_undegraded":"2024-09-17T04:49:42.536018+0000","last_fullsized":"2024-09-17T04:49:42.536018+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671602+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T04:49:42.671602+0000","last_peered":"2024-09-17T04:49:42.671602+0000","last_clean":"2024-09-17T04:49:42.671602+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T04:49:42.671602+0000","last_undegraded":"2024-09-17T04:49:42.671602+0000","last_fullsized":"2024-09-17T04:49:42.671602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.534962+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T04:49:42.534962+0000","last_peered":"2024-09-17T04:49:42.534962+0000","last_clean":"2024-09-17T04:49:42.534962+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T04:49:42.534962+0000","last_undegraded":"2024-09-17T04:49:42.534962+0000","last_fullsized":"2024-09-17T04:49:42.534962+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-17T04:49:42.671825+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T04:49:42.671825+0000","last_peered":"2024-09-17T04:49:42.671825+0000","last_clean":"2024-09-17T04:49:42.671825+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T04:49:42.671825+0000","last_undegraded":"2024-09-17T04:49:42.671825+0000","last_fullsized":"2024-09-17T04:49:42.671825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2851,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509453,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509453,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":876544,"data_stored":695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509453,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":307200,"data_stored":108978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509453,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":290816,"data_stored":105977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509453,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":106706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509454,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":695127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509454,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":282624,"data_stored":104765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542157,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":282624,"data_stored":106231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:50:10.813 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:50:10.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/1663810947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b680a7350 msgr2=0x7f1b68150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 --2- 172.21.15.150:0/1663810947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b680a7350 0x7f1b68150060 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1b680ae070 tx=0x7f1b5001c800 comp rx=0 tx=0).stop 2024-09-17T04:50:10.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/1663810947 shutdown_connections 2024-09-17T04:50:10.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 --2- 172.21.15.150:0/1663810947 >> v2:172.21.15.176:3300/0 conn(0x7f1b681505a0 0x7f1b68152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 --2- 172.21.15.150:0/1663810947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b680a7350 0x7f1b68150060 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/1663810947 >> 172.21.15.150:0/1663810947 conn(0x7f1b680c4a00 msgr2=0x7f1b680c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:10.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/1663810947 shutdown_connections 2024-09-17T04:50:10.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/1663810947 wait complete. 2024-09-17T04:50:10.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 Processor -- start 2024-09-17T04:50:10.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 -- start start 2024-09-17T04:50:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b680a7350 0x7f1b6814a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b680c42e0 0x7f1b68149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b681505a0 0x7f1b681f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1b681537c0 con 0x7f1b680c42e0 2024-09-17T04:50:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1b68153640 con 0x7f1b681505a0 2024-09-17T04:50:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b6cf1c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1b6815d670 con 0x7f1b680a7350 2024-09-17T04:50:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b5dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b680c42e0 0x7f1b68149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b5dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b680c42e0 0x7f1b68149c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37626/0 (socket says 172.21.15.150:37626) 2024-09-17T04:50:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.987+0000 7f1b5dd9b640 1 -- 172.21.15.150:0/2037694605 learned_addr learned my addr 172.21.15.150:0/2037694605 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b66d9d640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b681505a0 0x7f1b681f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b6659c640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b680a7350 0x7f1b6814a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b5dd9b640 1 -- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b680a7350 msgr2=0x7f1b6814a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b5dd9b640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b680a7350 0x7f1b6814a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b5dd9b640 1 -- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b681505a0 msgr2=0x7f1b681f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b5dd9b640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b681505a0 0x7f1b681f10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b5dd9b640 1 -- 172.21.15.150:0/2037694605 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6814a1c0 con 0x7f1b680c42e0 2024-09-17T04:50:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b66d9d640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b681505a0 0x7f1b681f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:50:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b5dd9b640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b680c42e0 0x7f1b68149c80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1b50000c10 tx=0x7f1b500a6620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b5f7fe640 1 -- 172.21.15.150:0/2037694605 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b50004030 con 0x7f1b680c42e0 2024-09-17T04:50:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b5f7fe640 1 -- 172.21.15.150:0/2037694605 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b500041d0 con 0x7f1b680c42e0 2024-09-17T04:50:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/2037694605 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b681f1c10 con 0x7f1b680c42e0 2024-09-17T04:50:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b5f7fe640 1 -- 172.21.15.150:0/2037694605 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b500050c0 con 0x7f1b680c42e0 2024-09-17T04:50:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/2037694605 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b680ac8d0 con 0x7f1b680c42e0 2024-09-17T04:50:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/2037694605 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b68071490 con 0x7f1b680c42e0 2024-09-17T04:50:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b5f7fe640 1 -- 172.21.15.150:0/2037694605 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1b500b8030 con 0x7f1b680c42e0 2024-09-17T04:50:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b5f7fe640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b48073200 0x7f1b48075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b5f7fe640 1 -- 172.21.15.150:0/2037694605 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f1b5012df00 con 0x7f1b680c42e0 2024-09-17T04:50:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b6659c640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b48073200 0x7f1b48075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:10.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.991+0000 7f1b6659c640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b48073200 0x7f1b48075710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1b58000f70 tx=0x7f1b58009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:10.995+0000 7f1b5f7fe640 1 -- 172.21.15.150:0/2037694605 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1b500bf070 con 0x7f1b680c42e0 2024-09-17T04:50:11.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.199+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/2037694605 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b68150060 con 0x7f1b680c42e0 2024-09-17T04:50:11.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b5f7fe640 1 -- 172.21.15.150:0/2037694605 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f1b6814a1c0 con 0x7f1b680c42e0 2024-09-17T04:50:11.206 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:50:11.206 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:50:11.206 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:50:11.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b48073200 msgr2=0x7f1b48075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:11.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b6cf1c640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b48073200 0x7f1b48075710 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1b58000f70 tx=0x7f1b58009040 comp rx=0 tx=0).stop 2024-09-17T04:50:11.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/2037694605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b680c42e0 msgr2=0x7f1b68149c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:11.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b6cf1c640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b680c42e0 0x7f1b68149c80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1b50000c10 tx=0x7f1b500a6620 comp rx=0 tx=0).stop 2024-09-17T04:50:11.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/2037694605 shutdown_connections 2024-09-17T04:50:11.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b6cf1c640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b48073200 0x7f1b48075710 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b6cf1c640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b681505a0 0x7f1b681f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b6cf1c640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b680c42e0 0x7f1b68149c80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b6cf1c640 1 --2- 172.21.15.150:0/2037694605 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b680a7350 0x7f1b6814a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/2037694605 >> 172.21.15.150:0/2037694605 conn(0x7f1b680c4a00 msgr2=0x7f1b680c3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:11.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/2037694605 shutdown_connections 2024-09-17T04:50:11.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.203+0000 7f1b6cf1c640 1 -- 172.21.15.150:0/2037694605 wait complete. 2024-09-17T04:50:11.226 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:50:11.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.371+0000 7f46cbf6e640 1 -- 172.21.15.150:0/3778596327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b0004d50 msgr2=0x7f46b0005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:11.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.371+0000 7f46cbf6e640 1 --2- 172.21.15.150:0/3778596327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b0004d50 0x7f46b0005160 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f46c4157f30 tx=0x7f46c001d250 comp rx=0 tx=0).stop 2024-09-17T04:50:11.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.371+0000 7f46ca50b640 1 -- 172.21.15.150:0/3778596327 reap_dead start 2024-09-17T04:50:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.371+0000 7f46cbf6e640 1 -- 172.21.15.150:0/3778596327 shutdown_connections 2024-09-17T04:50:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.371+0000 7f46cbf6e640 1 -- 172.21.15.150:0/3778596327 >> 172.21.15.150:0/3778596327 conn(0x7f46c40c9150 msgr2=0x7f46c40c9560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.371+0000 7f46cbf6e640 1 -- 172.21.15.150:0/3778596327 shutdown_connections 2024-09-17T04:50:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.371+0000 7f46cbf6e640 1 -- 172.21.15.150:0/3778596327 wait complete. 2024-09-17T04:50:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46cbf6e640 1 Processor -- start 2024-09-17T04:50:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46cbf6e640 1 -- start start 2024-09-17T04:50:11.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46cbf6e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46b0004d50 0x7f46c4150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:11.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46cbf6e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c414a350 0x7f46c414fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:11.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46cbf6e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c41ed4f0 0x7f46c41ed900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:11.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46cbf6e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f46c40c83a0 con 0x7f46c41ed4f0 2024-09-17T04:50:11.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46cbf6e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f46c40c3ca0 con 0x7f46b0004d50 2024-09-17T04:50:11.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46cbf6e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f46c40c7710 con 0x7f46c414a350 2024-09-17T04:50:11.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46ca50b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c41ed4f0 0x7f46c41ed900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:11.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46ca50b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c41ed4f0 0x7f46c41ed900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37652/0 (socket says 172.21.15.150:37652) 2024-09-17T04:50:11.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46ca50b640 1 -- 172.21.15.150:0/2234543208 learned_addr learned my addr 172.21.15.150:0/2234543208 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:11.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46c9d0a640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46b0004d50 0x7f46c4150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:11.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46c9509640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c414a350 0x7f46c414fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:11.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46ca50b640 1 -- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c414a350 msgr2=0x7f46c414fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46ca50b640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c414a350 0x7f46c414fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46ca50b640 1 -- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46b0004d50 msgr2=0x7f46c4150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46ca50b640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46b0004d50 0x7f46c4150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46ca50b640 1 -- 172.21.15.150:0/2234543208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c40c2ff0 con 0x7f46c41ed4f0 2024-09-17T04:50:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46c9d0a640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46b0004d50 0x7f46c4150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:50:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46ca50b640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c41ed4f0 0x7f46c41ed900 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f46c001d080 tx=0x7f46c00adce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46b67fc640 1 -- 172.21.15.150:0/2234543208 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c00b1040 con 0x7f46c41ed4f0 2024-09-17T04:50:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46b67fc640 1 -- 172.21.15.150:0/2234543208 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46c00bb070 con 0x7f46c41ed4f0 2024-09-17T04:50:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46cbf6e640 1 -- 172.21.15.150:0/2234543208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c41ee080 con 0x7f46c41ed4f0 2024-09-17T04:50:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46b67fc640 1 -- 172.21.15.150:0/2234543208 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c0004030 con 0x7f46c41ed4f0 2024-09-17T04:50:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.375+0000 7f46cbf6e640 1 -- 172.21.15.150:0/2234543208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c414daf0 con 0x7f46c41ed4f0 2024-09-17T04:50:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.379+0000 7f46cbf6e640 1 -- 172.21.15.150:0/2234543208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46c40c7c50 con 0x7f46c41ed4f0 2024-09-17T04:50:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.379+0000 7f46b67fc640 1 -- 172.21.15.150:0/2234543208 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f46c00041d0 con 0x7f46c41ed4f0 2024-09-17T04:50:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.379+0000 7f46b67fc640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f46a8073200 0x7f46a8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.379+0000 7f46b67fc640 1 -- 172.21.15.150:0/2234543208 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6666+0+0 (secure 0 0 0) 0x7f46c012f6d0 con 0x7f46c41ed4f0 2024-09-17T04:50:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.379+0000 7f46c9d0a640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f46a8073200 0x7f46a8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.379+0000 7f46c9d0a640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f46a8073200 0x7f46a8075710 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0015b0 tx=0x7f46bc00c040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:11.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.383+0000 7f46b67fc640 1 -- 172.21.15.150:0/2234543208 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f46c0016f60 con 0x7f46c41ed4f0 2024-09-17T04:50:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.543+0000 7f46cbf6e640 1 -- 172.21.15.150:0/2234543208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46c41ede40 con 0x7f46c41ed4f0 2024-09-17T04:50:11.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.543+0000 7f46b67fc640 1 -- 172.21.15.150:0/2234543208 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+20728 (secure 0 0 0) 0x7f46c00fa290 con 0x7f46c41ed4f0 2024-09-17T04:50:11.548 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:50:11.549 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":27,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:50:11.042034+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.31","mappings":[{"from":0,"to":7}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]},{"pgid":"5.2a","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:50:11.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.547+0000 7f46cbf6e640 1 -- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f46a8073200 msgr2=0x7f46a8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:11.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.547+0000 7f46cbf6e640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f46a8073200 0x7f46a8075710 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0015b0 tx=0x7f46bc00c040 comp rx=0 tx=0).stop 2024-09-17T04:50:11.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.547+0000 7f46cbf6e640 1 -- 172.21.15.150:0/2234543208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c41ed4f0 msgr2=0x7f46c41ed900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:11.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.547+0000 7f46cbf6e640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c41ed4f0 0x7f46c41ed900 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f46c001d080 tx=0x7f46c00adce0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.547+0000 7f46cbf6e640 1 -- 172.21.15.150:0/2234543208 shutdown_connections 2024-09-17T04:50:11.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.547+0000 7f46cbf6e640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f46a8073200 0x7f46a8075710 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.547+0000 7f46cbf6e640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46c41ed4f0 0x7f46c41ed900 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.547+0000 7f46cbf6e640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46c414a350 0x7f46c414fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.547+0000 7f46cbf6e640 1 --2- 172.21.15.150:0/2234543208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46b0004d50 0x7f46c4150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.547+0000 7f46cbf6e640 1 -- 172.21.15.150:0/2234543208 >> 172.21.15.150:0/2234543208 conn(0x7f46c40c9150 msgr2=0x7f46c40aa1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:11.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.547+0000 7f46cbf6e640 1 -- 172.21.15.150:0/2234543208 shutdown_connections 2024-09-17T04:50:11.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.547+0000 7f46cbf6e640 1 -- 172.21.15.150:0/2234543208 wait complete. 2024-09-17T04:50:11.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:50:11.571 INFO:tasks.mds_thrash.fs.[a]:starting mds_do_thrash for fs a 2024-09-17T04:50:11.571 INFO:tasks.mds_thrash.fs.[a]:waiting for 58 secs before thrashing 2024-09-17T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 -- 172.21.15.150:0/107762142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba64004810 msgr2=0x7fba64004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 --2- 172.21.15.150:0/107762142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba64004810 0x7fba64004c20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fba7c0c3920 tx=0x7fba7001cf00 comp rx=0 tx=0).stop 2024-09-17T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba80b7b640 1 -- 172.21.15.150:0/107762142 reap_dead start 2024-09-17T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 -- 172.21.15.150:0/107762142 shutdown_connections 2024-09-17T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 -- 172.21.15.150:0/107762142 >> 172.21.15.150:0/107762142 conn(0x7fba7c0c4a00 msgr2=0x7fba7c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 -- 172.21.15.150:0/107762142 shutdown_connections 2024-09-17T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 -- 172.21.15.150:0/107762142 wait complete. 2024-09-17T04:50:11.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 Processor -- start 2024-09-17T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 -- start start 2024-09-17T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba64004810 0x7fba7c1516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba7c1472d0 0x7fba7c1f0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba7c1476e0 0x7fba7c1f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fba7c0a7770 con 0x7fba7c1472d0 2024-09-17T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fba7c0a75f0 con 0x7fba64004810 2024-09-17T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.715+0000 7fba825de640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fba7c160870 con 0x7fba7c1476e0 2024-09-17T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba7b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba7c1472d0 0x7fba7c1f0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba7b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba7c1472d0 0x7fba7c1f0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37680/0 (socket says 172.21.15.150:37680) 2024-09-17T04:50:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba7b7fe640 1 -- 172.21.15.150:0/2680725995 learned_addr learned my addr 172.21.15.150:0/2680725995 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba80b7b640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba7c1476e0 0x7fba7c1f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba7bfff640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba64004810 0x7fba7c1516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba7b7fe640 1 -- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba7c1476e0 msgr2=0x7fba7c1f14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba7b7fe640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba7c1476e0 0x7fba7c1f14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba7b7fe640 1 -- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba64004810 msgr2=0x7fba7c1516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba7b7fe640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba64004810 0x7fba7c1516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba7b7fe640 1 -- 172.21.15.150:0/2680725995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7c157fe0 con 0x7fba7c1472d0 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba80b7b640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba7c1476e0 0x7fba7c1f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba7bfff640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba64004810 0x7fba7c1516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba7b7fe640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba7c1472d0 0x7fba7c1f0f70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fba700988b0 tx=0x7fba70000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba78ff9640 1 -- 172.21.15.150:0/2680725995 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba700a5e50 con 0x7fba7c1472d0 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba78ff9640 1 -- 172.21.15.150:0/2680725995 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba70006090 con 0x7fba7c1472d0 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba78ff9640 1 -- 172.21.15.150:0/2680725995 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba700064c0 con 0x7fba7c1472d0 2024-09-17T04:50:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba825de640 1 -- 172.21.15.150:0/2680725995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba7c1560d0 con 0x7fba7c1472d0 2024-09-17T04:50:11.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba825de640 1 -- 172.21.15.150:0/2680725995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba7c14a070 con 0x7fba7c1472d0 2024-09-17T04:50:11.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba825de640 1 -- 172.21.15.150:0/2680725995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba7c0ad050 con 0x7fba7c1472d0 2024-09-17T04:50:11.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba78ff9640 1 -- 172.21.15.150:0/2680725995 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fba70003620 con 0x7fba7c1472d0 2024-09-17T04:50:11.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba78ff9640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba680733a0 0x7fba680758b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:11.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba78ff9640 1 -- 172.21.15.150:0/2680725995 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6666+0+0 (secure 0 0 0) 0x7fba700b4020 con 0x7fba7c1472d0 2024-09-17T04:50:11.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.719+0000 7fba7bfff640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba680733a0 0x7fba680758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:11.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.723+0000 7fba7bfff640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba680733a0 0x7fba680758b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fba7c072ef0 tx=0x7fba6c005e40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:11.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.723+0000 7fba78ff9640 1 -- 172.21.15.150:0/2680725995 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fba7c0ad050 con 0x7fba7c1472d0 2024-09-17T04:50:11.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.927+0000 7fba825de640 1 -- 172.21.15.150:0/2680725995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba7c155e30 con 0x7fba7c1472d0 2024-09-17T04:50:11.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba78ff9640 1 -- 172.21.15.150:0/2680725995 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fba70016cb0 con 0x7fba7c1472d0 2024-09-17T04:50:11.933 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:50:11.933 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:50:11.934 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:50:11.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba825de640 1 -- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba680733a0 msgr2=0x7fba680758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:11.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba825de640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba680733a0 0x7fba680758b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fba7c072ef0 tx=0x7fba6c005e40 comp rx=0 tx=0).stop 2024-09-17T04:50:11.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba825de640 1 -- 172.21.15.150:0/2680725995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba7c1472d0 msgr2=0x7fba7c1f0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:11.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba825de640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba7c1472d0 0x7fba7c1f0f70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fba700988b0 tx=0x7fba70000ee0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba825de640 1 -- 172.21.15.150:0/2680725995 shutdown_connections 2024-09-17T04:50:11.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba825de640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba680733a0 0x7fba680758b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba825de640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba7c1476e0 0x7fba7c1f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba825de640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba7c1472d0 0x7fba7c1f0f70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba825de640 1 --2- 172.21.15.150:0/2680725995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba64004810 0x7fba7c1516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:11.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba825de640 1 -- 172.21.15.150:0/2680725995 >> 172.21.15.150:0/2680725995 conn(0x7fba7c0c4a00 msgr2=0x7fba7c0a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:11.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba825de640 1 -- 172.21.15.150:0/2680725995 shutdown_connections 2024-09-17T04:50:11.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:11.931+0000 7fba825de640 1 -- 172.21.15.150:0/2680725995 wait complete. 2024-09-17T04:50:11.954 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:50:12.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.099+0000 7f5cebccd640 1 -- 172.21.15.150:0/3642463138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd0004030 msgr2=0x7f5cd0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:12.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.099+0000 7f5cebccd640 1 --2- 172.21.15.150:0/3642463138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd0004030 0x7f5cd0004810 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400bb10 tx=0x7f5cd401cde0 comp rx=0 tx=0).stop 2024-09-17T04:50:12.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.099+0000 7f5cea26a640 1 -- 172.21.15.150:0/3642463138 reap_dead start 2024-09-17T04:50:12.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.099+0000 7f5cebccd640 1 -- 172.21.15.150:0/3642463138 shutdown_connections 2024-09-17T04:50:12.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.099+0000 7f5cebccd640 1 -- 172.21.15.150:0/3642463138 >> 172.21.15.150:0/3642463138 conn(0x7f5ce40c4a00 msgr2=0x7f5ce40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:12.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.099+0000 7f5cebccd640 1 -- 172.21.15.150:0/3642463138 shutdown_connections 2024-09-17T04:50:12.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.099+0000 7f5cebccd640 1 -- 172.21.15.150:0/3642463138 wait complete. 2024-09-17T04:50:12.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cebccd640 1 Processor -- start 2024-09-17T04:50:12.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cebccd640 1 -- start start 2024-09-17T04:50:12.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cebccd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd0004030 0x7f5ce4151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:12.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cebccd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ce4155860 0x7f5ce4155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:12.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cebccd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ce40c1160 0x7f5ce40c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:12.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cebccd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4070ac0 con 0x7f5cd0004030 2024-09-17T04:50:12.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cebccd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce40707e0 con 0x7f5ce40c1160 2024-09-17T04:50:12.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cebccd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce4070da0 con 0x7f5ce4155860 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5ce9a69640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd0004030 0x7f5ce4151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5ce9a69640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd0004030 0x7f5ce4151bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37686/0 (socket says 172.21.15.150:37686) 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5ce9a69640 1 -- 172.21.15.150:0/453800826 learned_addr learned my addr 172.21.15.150:0/453800826 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cea26a640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ce40c1160 0x7f5ce40c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5ce9268640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ce4155860 0x7f5ce4155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5ce9a69640 1 -- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ce4155860 msgr2=0x7f5ce4155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5ce9a69640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ce4155860 0x7f5ce4155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5ce9a69640 1 -- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ce40c1160 msgr2=0x7f5ce40c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5ce9a69640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ce40c1160 0x7f5ce40c1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5ce9a69640 1 -- 172.21.15.150:0/453800826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce41efab0 con 0x7f5cd0004030 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5ce9a69640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd0004030 0x7f5ce4151bb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000d680 tx=0x7f5ce000db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cda7fc640 1 -- 172.21.15.150:0/453800826 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce000a960 con 0x7f5cd0004030 2024-09-17T04:50:12.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cda7fc640 1 -- 172.21.15.150:0/453800826 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ce000abf0 con 0x7f5cd0004030 2024-09-17T04:50:12.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cebccd640 1 -- 172.21.15.150:0/453800826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce41efd90 con 0x7f5cd0004030 2024-09-17T04:50:12.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cda7fc640 1 -- 172.21.15.150:0/453800826 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0012470 con 0x7f5cd0004030 2024-09-17T04:50:12.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cebccd640 1 -- 172.21.15.150:0/453800826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce41f0350 con 0x7f5cd0004030 2024-09-17T04:50:12.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.103+0000 7f5cebccd640 1 -- 172.21.15.150:0/453800826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce4150060 con 0x7f5cd0004030 2024-09-17T04:50:12.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.107+0000 7f5cda7fc640 1 -- 172.21.15.150:0/453800826 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5ce0012610 con 0x7f5cd0004030 2024-09-17T04:50:12.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.107+0000 7f5cda7fc640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5cc8073200 0x7f5cc8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:50:12.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.107+0000 7f5cda7fc640 1 -- 172.21.15.150:0/453800826 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f5ce0097bf0 con 0x7f5cd0004030 2024-09-17T04:50:12.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.107+0000 7f5ce9268640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5cc8073200 0x7f5cc8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:50:12.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.107+0000 7f5ce9268640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5cc8073200 0x7f5cc8075710 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40b0000 tx=0x7f5cd4019b80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:50:12.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.111+0000 7f5cda7fc640 1 -- 172.21.15.150:0/453800826 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5ce0016030 con 0x7f5cd0004030 2024-09-17T04:50:12.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.267+0000 7f5cebccd640 1 -- 172.21.15.150:0/453800826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ce40a8a10 con 0x7f5cd0004030 2024-09-17T04:50:12.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.271+0000 7f5cda7fc640 1 -- 172.21.15.150:0/453800826 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+20728 (secure 0 0 0) 0x7f5ce0062970 con 0x7f5cd0004030 2024-09-17T04:50:12.275 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:50:12.275 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T04:50:12.050672+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.31","mappings":[{"from":0,"to":7}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]},{"pgid":"5.2a","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:50:12.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.275+0000 7f5cebccd640 1 -- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5cc8073200 msgr2=0x7f5cc8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:12.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.275+0000 7f5cebccd640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5cc8073200 0x7f5cc8075710 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40b0000 tx=0x7f5cd4019b80 comp rx=0 tx=0).stop 2024-09-17T04:50:12.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.275+0000 7f5cebccd640 1 -- 172.21.15.150:0/453800826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd0004030 msgr2=0x7f5ce4151bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:50:12.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.275+0000 7f5cebccd640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd0004030 0x7f5ce4151bb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000d680 tx=0x7f5ce000db40 comp rx=0 tx=0).stop 2024-09-17T04:50:12.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.275+0000 7f5cebccd640 1 -- 172.21.15.150:0/453800826 shutdown_connections 2024-09-17T04:50:12.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.275+0000 7f5cebccd640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5cc8073200 0x7f5cc8075710 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:12.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.275+0000 7f5cebccd640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ce40c1160 0x7f5ce40c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:12.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.275+0000 7f5cebccd640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ce4155860 0x7f5ce4155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:12.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.275+0000 7f5cebccd640 1 --2- 172.21.15.150:0/453800826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cd0004030 0x7f5ce4151bb0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:50:12.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.275+0000 7f5cebccd640 1 -- 172.21.15.150:0/453800826 >> 172.21.15.150:0/453800826 conn(0x7f5ce40c4a00 msgr2=0x7f5ce414d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:50:12.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.275+0000 7f5cebccd640 1 -- 172.21.15.150:0/453800826 shutdown_connections 2024-09-17T04:50:12.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:50:12.275+0000 7f5cebccd640 1 -- 172.21.15.150:0/453800826 wait complete. 2024-09-17T04:50:12.296 DEBUG:tasks.mds_thrash:Yielding 2024-09-17T04:50:12.296 INFO:teuthology.run_tasks:Running task workunit... 2024-09-17T04:50:12.310 INFO:tasks.workunit:Pulling workunits from ref ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T04:50:12.310 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-17T04:50:12.310 INFO:tasks.workunit:timeout=3h 2024-09-17T04:50:12.310 INFO:tasks.workunit:cleanup=True 2024-09-17T04:50:12.310 DEBUG:teuthology.orchestra.run.smithi150:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-17T04:50:12.311 INFO:tasks.mds_thrash.fs.[b]:starting mds_do_thrash for fs b 2024-09-17T04:50:12.311 INFO:tasks.mds_thrash.fs.[b]:waiting for 76 secs before thrashing 2024-09-17T04:50:12.317 INFO:teuthology.orchestra.run.smithi150.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-17T04:50:12.317 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-09-17T04:50:12.317 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 2dh/45d Inode: 1 Links: 2 2024-09-17T04:50:12.317 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T04:50:12.317 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-17T04:50:12.317 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-09-17 04:49:31.885696472 +0000 2024-09-17T04:50:12.317 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-09-17 04:50:07.059305959 +0000 2024-09-17T04:50:12.317 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: - 2024-09-17T04:50:12.317 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-17T04:50:12.318 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-17T04:50:12.379 DEBUG:teuthology.orchestra.run.smithi176:> stat -- /home/ubuntu/cephtest/mnt.1 2024-09-17T04:50:12.387 INFO:teuthology.orchestra.run.smithi176.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-09-17T04:50:12.387 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-09-17T04:50:12.388 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 2dh/45d Inode: 1 Links: 2 2024-09-17T04:50:12.388 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T04:50:12.388 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-17T04:50:12.388 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-09-17 04:49:41.033166990 +0000 2024-09-17T04:50:12.388 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-09-17 04:50:07.123273608 +0000 2024-09-17T04:50:12.388 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-09-17T04:50:12.388 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-09-17T04:50:12.388 DEBUG:teuthology.orchestra.run.smithi176:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-09-17T04:50:12.453 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T04:50:12.454 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T04:50:12.461 INFO:tasks.workunit.client.0.smithi150.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-17T04:50:12.502 INFO:tasks.workunit.client.1.smithi176.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-09-17T04:50:28.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:50:28.051 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:50:58.195 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:50:58.198 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:51:09.572 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:09.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 -- 172.21.15.150:0/1090451008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7300157dd0 msgr2=0x7f73001581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:09.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 --2- 172.21.15.150:0/1090451008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7300157dd0 0x7f73001581e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009870 tx=0x7f72f001c790 comp rx=0 tx=0).stop 2024-09-17T04:51:09.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 -- 172.21.15.150:0/1090451008 shutdown_connections 2024-09-17T04:51:09.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 --2- 172.21.15.150:0/1090451008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7300157dd0 0x7f73001581e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:09.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 --2- 172.21.15.150:0/1090451008 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73000c5870 0x7f7300071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:09.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 -- 172.21.15.150:0/1090451008 >> 172.21.15.150:0/1090451008 conn(0x7f73000c4d50 msgr2=0x7f73000c5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:09.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 -- 172.21.15.150:0/1090451008 shutdown_connections 2024-09-17T04:51:09.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 -- 172.21.15.150:0/1090451008 wait complete. 2024-09-17T04:51:09.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 Processor -- start 2024-09-17T04:51:09.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 -- start start 2024-09-17T04:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000708a0 0x7f7300156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73000c5870 0x7f73001443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7300157dd0 0x7f7300142010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7300161b00 con 0x7f73000708a0 2024-09-17T04:51:09.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7300161980 con 0x7f73000c5870 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.718+0000 7f7308643640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7300161c80 con 0x7f7300157dd0 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f73063df640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000708a0 0x7f7300156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f73063df640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000708a0 0x7f7300156b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50730/0 (socket says 172.21.15.150:50730) 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f73063df640 1 -- 172.21.15.150:0/1988742119 learned_addr learned my addr 172.21.15.150:0/1988742119 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f7306be0640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7300157dd0 0x7f7300142010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f7305bde640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73000c5870 0x7f73001443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f73063df640 1 -- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7300157dd0 msgr2=0x7f7300142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f73063df640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7300157dd0 0x7f7300142010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f73063df640 1 -- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73000c5870 msgr2=0x7f73001443e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f73063df640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73000c5870 0x7f73001443e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f73063df640 1 -- 172.21.15.150:0/1988742119 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7300154890 con 0x7f73000708a0 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f7306be0640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7300157dd0 0x7f7300142010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:51:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f7305bde640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73000c5870 0x7f73001443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:51:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f73063df640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000708a0 0x7f7300156b00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f72f800c9a0 tx=0x7f72f800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f72feffd640 1 -- 172.21.15.150:0/1988742119 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f8004220 con 0x7f73000708a0 2024-09-17T04:51:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f72feffd640 1 -- 172.21.15.150:0/1988742119 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72f8007950 con 0x7f73000708a0 2024-09-17T04:51:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f7308643640 1 -- 172.21.15.150:0/1988742119 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7300141780 con 0x7f73000708a0 2024-09-17T04:51:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f72feffd640 1 -- 172.21.15.150:0/1988742119 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f80044b0 con 0x7f73000708a0 2024-09-17T04:51:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f7308643640 1 -- 172.21.15.150:0/1988742119 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73001545b0 con 0x7f73000708a0 2024-09-17T04:51:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f7308643640 1 -- 172.21.15.150:0/1988742119 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7300071da0 con 0x7f73000708a0 2024-09-17T04:51:09.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f72feffd640 1 -- 172.21.15.150:0/1988742119 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f72f8003600 con 0x7f73000708a0 2024-09-17T04:51:09.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f72feffd640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e0073200 0x7f72e0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:09.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f72feffd640 1 -- 172.21.15.150:0/1988742119 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f72f80949c0 con 0x7f73000708a0 2024-09-17T04:51:09.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f7305bde640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e0073200 0x7f72e0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:09.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.722+0000 7f7305bde640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e0073200 0x7f72e0075710 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f72e8007fd0 tx=0x7f72e8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:09.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.726+0000 7f72feffd640 1 -- 172.21.15.150:0/1988742119 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f72f8096e20 con 0x7f73000708a0 2024-09-17T04:51:09.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.918+0000 7f7308643640 1 -- 172.21.15.150:0/1988742119 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73000c42c0 con 0x7f73000708a0 2024-09-17T04:51:09.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.918+0000 7f72feffd640 1 -- 172.21.15.150:0/1988742119 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f7300154890 con 0x7f73000708a0 2024-09-17T04:51:09.919 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:09.919 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:09.919 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:51:09.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.922+0000 7f7308643640 1 -- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e0073200 msgr2=0x7f72e0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:09.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.922+0000 7f7308643640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e0073200 0x7f72e0075710 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f72e8007fd0 tx=0x7f72e8009040 comp rx=0 tx=0).stop 2024-09-17T04:51:09.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.922+0000 7f7308643640 1 -- 172.21.15.150:0/1988742119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000708a0 msgr2=0x7f7300156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:09.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.922+0000 7f7308643640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000708a0 0x7f7300156b00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f72f800c9a0 tx=0x7f72f800ce60 comp rx=0 tx=0).stop 2024-09-17T04:51:09.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.922+0000 7f7308643640 1 -- 172.21.15.150:0/1988742119 shutdown_connections 2024-09-17T04:51:09.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.922+0000 7f7308643640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e0073200 0x7f72e0075710 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:09.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.922+0000 7f7308643640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73000c5870 0x7f73001443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:09.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.922+0000 7f7308643640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7300157dd0 0x7f7300142010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:09.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.922+0000 7f7308643640 1 --2- 172.21.15.150:0/1988742119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000708a0 0x7f7300156b00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:09.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.922+0000 7f7308643640 1 -- 172.21.15.150:0/1988742119 >> 172.21.15.150:0/1988742119 conn(0x7f73000c4d50 msgr2=0x7f730015e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:09.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.922+0000 7f7308643640 1 -- 172.21.15.150:0/1988742119 shutdown_connections 2024-09-17T04:51:09.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:09.922+0000 7f7308643640 1 -- 172.21.15.150:0/1988742119 wait complete. 2024-09-17T04:51:09.939 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-17T04:51:09.940 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:51:09.940 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:51:09.952 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:51:09.952 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:10.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.134+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/550385238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cc80699b0 msgr2=0x7f2cc80719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:10.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.134+0000 7f2ccfaf1640 1 --2- 172.21.15.150:0/550385238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cc80699b0 0x7f2cc80719e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4009840 tx=0x7f2cc401c700 comp rx=0 tx=0).stop 2024-09-17T04:51:10.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.134+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/550385238 shutdown_connections 2024-09-17T04:51:10.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.134+0000 7f2ccfaf1640 1 --2- 172.21.15.150:0/550385238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc8151d10 0x7f2cc80adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:10.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.134+0000 7f2ccfaf1640 1 --2- 172.21.15.150:0/550385238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cc80699b0 0x7f2cc80719e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:10.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.134+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/550385238 >> 172.21.15.150:0/550385238 conn(0x7f2cc8070b50 msgr2=0x7f2cc8070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:10.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.134+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/550385238 shutdown_connections 2024-09-17T04:51:10.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.134+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/550385238 wait complete. 2024-09-17T04:51:10.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccfaf1640 1 Processor -- start 2024-09-17T04:51:10.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccfaf1640 1 -- start start 2024-09-17T04:51:10.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccfaf1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cc80699b0 0x7f2cc8145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:10.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccfaf1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc806e590 0x7f2cc81581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:10.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccfaf1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cc8151d10 0x7f2cc81f0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:10.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccfaf1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc8161990 con 0x7f2cc80699b0 2024-09-17T04:51:10.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccfaf1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc80a9fb0 con 0x7f2cc8151d10 2024-09-17T04:51:10.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccfaf1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc8161b10 con 0x7f2cc806e590 2024-09-17T04:51:10.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccd88d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cc80699b0 0x7f2cc8145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:10.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccd88d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cc80699b0 0x7f2cc8145270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50758/0 (socket says 172.21.15.150:50758) 2024-09-17T04:51:10.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccd88d640 1 -- 172.21.15.150:0/1195607475 learned_addr learned my addr 172.21.15.150:0/1195607475 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:10.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccd08c640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc806e590 0x7f2cc81581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:10.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2cce08e640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cc8151d10 0x7f2cc81f0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:10.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccd88d640 1 -- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc806e590 msgr2=0x7f2cc81581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:10.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccd88d640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc806e590 0x7f2cc81581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:10.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccd88d640 1 -- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cc8151d10 msgr2=0x7f2cc81f0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:10.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccd88d640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cc8151d10 0x7f2cc81f0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:10.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccd88d640 1 -- 172.21.15.150:0/1195607475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc81f1a30 con 0x7f2cc80699b0 2024-09-17T04:51:10.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccd88d640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cc80699b0 0x7f2cc8145270 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00cc20 tx=0x7f2cbc00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:10.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2cba7fc640 1 -- 172.21.15.150:0/1195607475 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc016020 con 0x7f2cc80699b0 2024-09-17T04:51:10.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2cba7fc640 1 -- 172.21.15.150:0/1195607475 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cbc00aae0 con 0x7f2cc80699b0 2024-09-17T04:51:10.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/1195607475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc81457b0 con 0x7f2cc80699b0 2024-09-17T04:51:10.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2cba7fc640 1 -- 172.21.15.150:0/1195607475 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc00ce40 con 0x7f2cc80699b0 2024-09-17T04:51:10.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.138+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/1195607475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc80c7ad0 con 0x7f2cc80699b0 2024-09-17T04:51:10.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.142+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/1195607475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc80719e0 con 0x7f2cc80699b0 2024-09-17T04:51:10.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.142+0000 7f2cba7fc640 1 -- 172.21.15.150:0/1195607475 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2cbc019020 con 0x7f2cc80699b0 2024-09-17T04:51:10.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.142+0000 7f2cba7fc640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cac073230 0x7f2cac075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:10.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.142+0000 7f2cba7fc640 1 -- 172.21.15.150:0/1195607475 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f2cbc094b30 con 0x7f2cc80699b0 2024-09-17T04:51:10.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.142+0000 7f2ccd08c640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cac073230 0x7f2cac075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:10.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.142+0000 7f2ccd08c640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cac073230 0x7f2cac075740 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40af000 tx=0x7f2cc401af40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:10.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.146+0000 7f2cba7fc640 1 -- 172.21.15.150:0/1195607475 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2cbc020020 con 0x7f2cc80699b0 2024-09-17T04:51:10.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.338+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/1195607475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2cc80c4280 con 0x7f2cc80699b0 2024-09-17T04:51:10.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.338+0000 7f2cba7fc640 1 -- 172.21.15.150:0/1195607475 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f2cbc05f870 con 0x7f2cc80699b0 2024-09-17T04:51:10.341 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:10.341 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:10.341 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:51:10.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.342+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cac073230 msgr2=0x7f2cac075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:10.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.342+0000 7f2ccfaf1640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cac073230 0x7f2cac075740 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40af000 tx=0x7f2cc401af40 comp rx=0 tx=0).stop 2024-09-17T04:51:10.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.342+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/1195607475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cc80699b0 msgr2=0x7f2cc8145270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:10.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.342+0000 7f2ccfaf1640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cc80699b0 0x7f2cc8145270 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00cc20 tx=0x7f2cbc00cc50 comp rx=0 tx=0).stop 2024-09-17T04:51:10.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.342+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/1195607475 shutdown_connections 2024-09-17T04:51:10.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.342+0000 7f2ccfaf1640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cac073230 0x7f2cac075740 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:10.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.342+0000 7f2ccfaf1640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cc8151d10 0x7f2cc81f0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:10.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.342+0000 7f2ccfaf1640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cc806e590 0x7f2cc81581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:10.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.342+0000 7f2ccfaf1640 1 --2- 172.21.15.150:0/1195607475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cc80699b0 0x7f2cc8145270 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:10.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.342+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/1195607475 >> 172.21.15.150:0/1195607475 conn(0x7f2cc8070b50 msgr2=0x7f2cc806ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:10.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.342+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/1195607475 shutdown_connections 2024-09-17T04:51:10.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:10.342+0000 7f2ccfaf1640 1 -- 172.21.15.150:0/1195607475 wait complete. 2024-09-17T04:51:10.364 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:51:11.279 INFO:tasks.workunit.client.1.smithi176.stderr:Updating files: 97% (12395/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-17T04:51:11.794 INFO:tasks.workunit.client.0.smithi150.stderr:Note: switching to 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'. 2024-09-17T04:51:11.794 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-17T04:51:11.794 INFO:tasks.workunit.client.0.smithi150.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr:state without impacting any branches by switching back to a branch. 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr: git switch -c 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr:Or undo this operation with: 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr: git switch - 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-17T04:51:11.795 INFO:tasks.workunit.client.0.smithi150.stderr: 2024-09-17T04:51:11.796 INFO:tasks.workunit.client.0.smithi150.stderr:HEAD is now at ce1b342bf8c Merge PR #55006 into wip-pdonnell-testing-20240916.200549-debug 2024-09-17T04:51:11.802 DEBUG:teuthology.orchestra.run.smithi150:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-17T04:51:11.809 INFO:tasks.workunit.client.0.smithi150.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-17T04:51:11.812 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-17T04:51:11.812 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-17T04:51:11.834 INFO:tasks.workunit.client.1.smithi176.stderr:Note: switching to 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'. 2024-09-17T04:51:11.834 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-09-17T04:51:11.834 INFO:tasks.workunit.client.1.smithi176.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr:state without impacting any branches by switching back to a branch. 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr: git switch -c 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr:Or undo this operation with: 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr: git switch - 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-17T04:51:11.835 INFO:tasks.workunit.client.1.smithi176.stderr: 2024-09-17T04:51:11.836 INFO:tasks.workunit.client.1.smithi176.stderr:HEAD is now at ce1b342bf8c Merge PR #55006 into wip-pdonnell-testing-20240916.200549-debug 2024-09-17T04:51:11.841 DEBUG:teuthology.orchestra.run.smithi176:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-09-17T04:51:11.848 INFO:tasks.workunit.client.1.smithi176.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-17T04:51:11.851 INFO:tasks.workunit.client.1.smithi176.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-17T04:51:11.851 INFO:tasks.workunit.client.1.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-17T04:51:12.365 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:12.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.530+0000 7fe9836ca640 1 -- 172.21.15.150:0/2346058960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe97c071670 msgr2=0x7fe97c0c5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:12.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.530+0000 7fe9836ca640 1 --2- 172.21.15.150:0/2346058960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe97c071670 0x7fe97c0c5680 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe97c0753f0 tx=0x7fe97801c890 comp rx=0 tx=0).stop 2024-09-17T04:51:12.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.530+0000 7fe9836ca640 1 -- 172.21.15.150:0/2346058960 shutdown_connections 2024-09-17T04:51:12.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.530+0000 7fe9836ca640 1 --2- 172.21.15.150:0/2346058960 >> v2:172.21.15.176:3300/0 conn(0x7fe97c157c00 0x7fe97c158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:12.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.530+0000 7fe9836ca640 1 --2- 172.21.15.150:0/2346058960 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe97c14e490 0x7fe97c0c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:12.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.530+0000 7fe9836ca640 1 --2- 172.21.15.150:0/2346058960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe97c071670 0x7fe97c0c5680 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:12.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.530+0000 7fe9836ca640 1 -- 172.21.15.150:0/2346058960 >> 172.21.15.150:0/2346058960 conn(0x7fe97c0ca460 msgr2=0x7fe97c0c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:12.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.530+0000 7fe9836ca640 1 -- 172.21.15.150:0/2346058960 shutdown_connections 2024-09-17T04:51:12.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.530+0000 7fe9836ca640 1 -- 172.21.15.150:0/2346058960 wait complete. 2024-09-17T04:51:12.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.530+0000 7fe9836ca640 1 Processor -- start 2024-09-17T04:51:12.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.530+0000 7fe9836ca640 1 -- start start 2024-09-17T04:51:12.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe9836ca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe97c071670 0x7fe97c157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:12.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe9836ca640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe97c14e490 0x7fe97c155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:12.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe9836ca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe97c157c00 0x7fe97c1ed470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:12.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe9836ca640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c0c6740 con 0x7fe97c071670 2024-09-17T04:51:12.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe9836ca640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c0c65c0 con 0x7fe97c14e490 2024-09-17T04:51:12.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe9836ca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe97c14e8a0 con 0x7fe97c157c00 2024-09-17T04:51:12.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe981466640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe97c071670 0x7fe97c157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe981466640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe97c071670 0x7fe97c157300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50776/0 (socket says 172.21.15.150:50776) 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe981466640 1 -- 172.21.15.150:0/150554314 learned_addr learned my addr 172.21.15.150:0/150554314 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe981c67640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe97c157c00 0x7fe97c1ed470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe980c65640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe97c14e490 0x7fe97c155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe981466640 1 -- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe97c157c00 msgr2=0x7fe97c1ed470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe981466640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe97c157c00 0x7fe97c1ed470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe981466640 1 -- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe97c14e490 msgr2=0x7fe97c155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe981466640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe97c14e490 0x7fe97c155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe981466640 1 -- 172.21.15.150:0/150554314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97c142090 con 0x7fe97c071670 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe980c65640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe97c14e490 0x7fe97c155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe981466640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe97c071670 0x7fe97c157300 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe9780b0040 tx=0x7fe9780040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe971ffb640 1 -- 172.21.15.150:0/150554314 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9780b2040 con 0x7fe97c071670 2024-09-17T04:51:12.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe9836ca640 1 -- 172.21.15.150:0/150554314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97c14c070 con 0x7fe97c071670 2024-09-17T04:51:12.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe971ffb640 1 -- 172.21.15.150:0/150554314 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9780044b0 con 0x7fe97c071670 2024-09-17T04:51:12.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe971ffb640 1 -- 172.21.15.150:0/150554314 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe978003840 con 0x7fe97c071670 2024-09-17T04:51:12.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe9836ca640 1 -- 172.21.15.150:0/150554314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe97c142680 con 0x7fe97c071670 2024-09-17T04:51:12.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe9836ca640 1 -- 172.21.15.150:0/150554314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe97c072260 con 0x7fe97c071670 2024-09-17T04:51:12.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe971ffb640 1 -- 172.21.15.150:0/150554314 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe9780039e0 con 0x7fe97c071670 2024-09-17T04:51:12.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe971ffb640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe960073230 0x7fe960075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:12.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe971ffb640 1 -- 172.21.15.150:0/150554314 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7fe97812e9f0 con 0x7fe97c071670 2024-09-17T04:51:12.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.534+0000 7fe980c65640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe960073230 0x7fe960075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:12.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.538+0000 7fe980c65640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe960073230 0x7fe960075740 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe97c06aaa0 tx=0x7fe96c005cc0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:12.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.538+0000 7fe971ffb640 1 -- 172.21.15.150:0/150554314 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe9780164b0 con 0x7fe97c071670 2024-09-17T04:51:12.602 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-17T04:51:12.679 INFO:tasks.workunit.client.1.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-17T04:51:12.683 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-17T04:51:12.728 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-17T04:51:12.729 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-17T04:51:12.729 INFO:tasks.workunit.client.0.smithi150.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-17T04:51:12.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.738+0000 7fe9836ca640 1 -- 172.21.15.150:0/150554314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe97c154140 con 0x7fe97c071670 2024-09-17T04:51:12.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe971ffb640 1 -- 172.21.15.150:0/150554314 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fe9780f9770 con 0x7fe97c071670 2024-09-17T04:51:12.741 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:12.741 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:12.742 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:51:12.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe9836ca640 1 -- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe960073230 msgr2=0x7fe960075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:12.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe9836ca640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe960073230 0x7fe960075740 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe97c06aaa0 tx=0x7fe96c005cc0 comp rx=0 tx=0).stop 2024-09-17T04:51:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe9836ca640 1 -- 172.21.15.150:0/150554314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe97c071670 msgr2=0x7fe97c157300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe9836ca640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe97c071670 0x7fe97c157300 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe9780b0040 tx=0x7fe9780040c0 comp rx=0 tx=0).stop 2024-09-17T04:51:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe9836ca640 1 -- 172.21.15.150:0/150554314 shutdown_connections 2024-09-17T04:51:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe9836ca640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe960073230 0x7fe960075740 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe9836ca640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe97c14e490 0x7fe97c155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe9836ca640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe97c157c00 0x7fe97c1ed470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe9836ca640 1 --2- 172.21.15.150:0/150554314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe97c071670 0x7fe97c157300 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe9836ca640 1 -- 172.21.15.150:0/150554314 >> 172.21.15.150:0/150554314 conn(0x7fe97c0ca460 msgr2=0x7fe97c0c7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe9836ca640 1 -- 172.21.15.150:0/150554314 shutdown_connections 2024-09-17T04:51:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:12.742+0000 7fe9836ca640 1 -- 172.21.15.150:0/150554314 wait complete. 2024-09-17T04:51:12.761 INFO:tasks.workunit.client.1.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-17T04:51:12.764 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:51:12.769 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-17T04:51:12.772 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T04:51:12.772 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-17T04:51:12.810 INFO:tasks.workunit.client.1.smithi176.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-17T04:51:12.811 INFO:tasks.workunit.client.1.smithi176.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-17T04:51:12.811 INFO:tasks.workunit.client.1.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-17T04:51:12.817 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-09-17T04:51:12.818 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-09-17T04:51:12.819 DEBUG:teuthology.orchestra.run.smithi150:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce1b342bf8c6349a8c90cd607f83bff2443f21cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-09-17T04:51:12.855 INFO:tasks.workunit.client.1.smithi176.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-17T04:51:12.859 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T04:51:12.859 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-09-17T04:51:12.908 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.1... 2024-09-17T04:51:12.909 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-09-17T04:51:12.909 DEBUG:teuthology.orchestra.run.smithi176:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ce1b342bf8c6349a8c90cd607f83bff2443f21cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/ffsb.sh 2024-09-17T04:51:12.972 INFO:tasks.workunit.client.1.smithi176.stderr:++ dirname /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/ffsb.sh 2024-09-17T04:51:12.974 INFO:tasks.workunit.client.1.smithi176.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.1/qa/workunits/suites 2024-09-17T04:51:12.974 INFO:tasks.workunit.client.1.smithi176.stderr:+ trap -- retry EXIT 2024-09-17T04:51:12.974 INFO:tasks.workunit.client.1.smithi176.stderr:+ rm -rf ffsb 2024-09-17T04:51:12.975 INFO:tasks.workunit.client.1.smithi176.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-09-17T04:51:12.983 INFO:tasks.workunit.client.1.smithi176.stderr:Cloning into 'ffsb'... 2024-09-17T04:51:14.765 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:14.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.934+0000 7f4f564f5640 1 -- 172.21.15.150:0/3925833833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f500c5680 msgr2=0x7f4f500c7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:14.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.934+0000 7f4f564f5640 1 --2- 172.21.15.150:0/3925833833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f500c5680 0x7f4f500c7d60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009900 tx=0x7f4f4401c8b0 comp rx=0 tx=0).stop 2024-09-17T04:51:14.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.934+0000 7f4f564f5640 1 -- 172.21.15.150:0/3925833833 shutdown_connections 2024-09-17T04:51:14.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.934+0000 7f4f564f5640 1 --2- 172.21.15.150:0/3925833833 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f50157c00 0x7f4f50158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:14.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.934+0000 7f4f564f5640 1 --2- 172.21.15.150:0/3925833833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f500c5680 0x7f4f500c7d60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:14.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.934+0000 7f4f564f5640 1 -- 172.21.15.150:0/3925833833 >> 172.21.15.150:0/3925833833 conn(0x7f4f500ca460 msgr2=0x7f4f500c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:14.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.934+0000 7f4f564f5640 1 -- 172.21.15.150:0/3925833833 shutdown_connections 2024-09-17T04:51:14.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.934+0000 7f4f564f5640 1 -- 172.21.15.150:0/3925833833 wait complete. 2024-09-17T04:51:14.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f564f5640 1 Processor -- start 2024-09-17T04:51:14.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f564f5640 1 -- start start 2024-09-17T04:51:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f564f5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f50071670 0x7f4f50145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f564f5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f500c5680 0x7f4f50156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f564f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f50157c00 0x7f4f5013e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f564f5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4f500c65c0 con 0x7f4f50157c00 2024-09-17T04:51:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f564f5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4f500c62d0 con 0x7f4f500c5680 2024-09-17T04:51:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f564f5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4f500c6740 con 0x7f4f50071670 2024-09-17T04:51:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f54a92640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f50157c00 0x7f4f5013e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f54a92640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f50157c00 0x7f4f5013e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50790/0 (socket says 172.21.15.150:50790) 2024-09-17T04:51:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f54a92640 1 -- 172.21.15.150:0/1529167383 learned_addr learned my addr 172.21.15.150:0/1529167383 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f4f7fe640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f500c5680 0x7f4f50156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f4ffff640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f50071670 0x7f4f50145d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f54a92640 1 -- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f50071670 msgr2=0x7f4f50145d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:14.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f54a92640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f50071670 0x7f4f50145d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:14.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f54a92640 1 -- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f500c5680 msgr2=0x7f4f50156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:14.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f54a92640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f500c5680 0x7f4f50156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:14.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f54a92640 1 -- 172.21.15.150:0/1529167383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f50152c00 con 0x7f4f50157c00 2024-09-17T04:51:14.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f54a92640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f50157c00 0x7f4f5013e640 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800e910 tx=0x7f4f4800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:14.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f4cff9640 1 -- 172.21.15.150:0/1529167383 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f48004d30 con 0x7f4f50157c00 2024-09-17T04:51:14.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f4cff9640 1 -- 172.21.15.150:0/1529167383 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f4800cb20 con 0x7f4f50157c00 2024-09-17T04:51:14.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f564f5640 1 -- 172.21.15.150:0/1529167383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f50143930 con 0x7f4f50157c00 2024-09-17T04:51:14.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f4cff9640 1 -- 172.21.15.150:0/1529167383 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f48005020 con 0x7f4f50157c00 2024-09-17T04:51:14.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.938+0000 7f4f564f5640 1 -- 172.21.15.150:0/1529167383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f50146700 con 0x7f4f50157c00 2024-09-17T04:51:14.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.942+0000 7f4f564f5640 1 -- 172.21.15.150:0/1529167383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f501462b0 con 0x7f4f50157c00 2024-09-17T04:51:14.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.942+0000 7f4f4cff9640 1 -- 172.21.15.150:0/1529167383 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4f480051c0 con 0x7f4f50157c00 2024-09-17T04:51:14.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.942+0000 7f4f4cff9640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f3c073200 0x7f4f3c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:14.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.942+0000 7f4f4cff9640 1 -- 172.21.15.150:0/1529167383 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f4f48094d50 con 0x7f4f50157c00 2024-09-17T04:51:14.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.942+0000 7f4f4ffff640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f3c073200 0x7f4f3c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:14.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.946+0000 7f4f4ffff640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f3c073200 0x7f4f3c075710 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4f400098e0 tx=0x7f4f40009060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:14.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:14.950+0000 7f4f4cff9640 1 -- 172.21.15.150:0/1529167383 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4f501462b0 con 0x7f4f50157c00 2024-09-17T04:51:15.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.198+0000 7f4f564f5640 1 -- 172.21.15.150:0/1529167383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f500c7d60 con 0x7f4f50157c00 2024-09-17T04:51:15.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f4cff9640 1 -- 172.21.15.150:0/1529167383 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f4f48098030 con 0x7f4f50157c00 2024-09-17T04:51:15.202 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:15.202 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:15.202 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:51:15.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f564f5640 1 -- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f3c073200 msgr2=0x7f4f3c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:15.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f564f5640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f3c073200 0x7f4f3c075710 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4f400098e0 tx=0x7f4f40009060 comp rx=0 tx=0).stop 2024-09-17T04:51:15.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f564f5640 1 -- 172.21.15.150:0/1529167383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f50157c00 msgr2=0x7f4f5013e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:15.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f564f5640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f50157c00 0x7f4f5013e640 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4f4800e910 tx=0x7f4f4800edd0 comp rx=0 tx=0).stop 2024-09-17T04:51:15.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f564f5640 1 -- 172.21.15.150:0/1529167383 shutdown_connections 2024-09-17T04:51:15.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f564f5640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f3c073200 0x7f4f3c075710 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:15.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f564f5640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f500c5680 0x7f4f50156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:15.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f564f5640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f50157c00 0x7f4f5013e640 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:15.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f564f5640 1 --2- 172.21.15.150:0/1529167383 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f50071670 0x7f4f50145d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:15.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f564f5640 1 -- 172.21.15.150:0/1529167383 >> 172.21.15.150:0/1529167383 conn(0x7f4f500ca460 msgr2=0x7f4f50144fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:15.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f564f5640 1 -- 172.21.15.150:0/1529167383 shutdown_connections 2024-09-17T04:51:15.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:15.202+0000 7f4f564f5640 1 -- 172.21.15.150:0/1529167383 wait complete. 2024-09-17T04:51:15.226 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:51:16.710 INFO:tasks.workunit.client.1.smithi176.stderr:+ trap - EXIT 2024-09-17T04:51:16.710 INFO:tasks.workunit.client.1.smithi176.stderr:+ cd ffsb 2024-09-17T04:51:16.712 INFO:tasks.workunit.client.1.smithi176.stderr:+ ./configure 2024-09-17T04:51:16.937 INFO:tasks.workunit.client.1.smithi176.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-17T04:51:17.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:17.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.394+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/2489205551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5a414b050 msgr2=0x7ff5a4072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:17.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.394+0000 7ff5aa5f5640 1 --2- 172.21.15.150:0/2489205551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5a414b050 0x7ff5a4072d70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff598009900 tx=0x7ff59801c7c0 comp rx=0 tx=0).stop 2024-09-17T04:51:17.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.394+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/2489205551 shutdown_connections 2024-09-17T04:51:17.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.394+0000 7ff5aa5f5640 1 --2- 172.21.15.150:0/2489205551 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5a40732b0 0x7ff5a415efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:17.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.394+0000 7ff5aa5f5640 1 --2- 172.21.15.150:0/2489205551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5a414b050 0x7ff5a4072d70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:17.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.394+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/2489205551 >> 172.21.15.150:0/2489205551 conn(0x7ff5a40abf10 msgr2=0x7ff5a40ac320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:17.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.394+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/2489205551 shutdown_connections 2024-09-17T04:51:17.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.394+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/2489205551 wait complete. 2024-09-17T04:51:17.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5aa5f5640 1 Processor -- start 2024-09-17T04:51:17.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5aa5f5640 1 -- start start 2024-09-17T04:51:17.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5aa5f5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5a40732b0 0x7ff5a4145120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:17.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5aa5f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5a414b050 0x7ff5a40aa320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:17.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5aa5f5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5a4144460 0x7ff5a41f0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:17.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5aa5f5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a40c9b60 con 0x7ff5a414b050 2024-09-17T04:51:17.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5aa5f5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a40c51b0 con 0x7ff5a4144460 2024-09-17T04:51:17.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5aa5f5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff5a40c9ce0 con 0x7ff5a40732b0 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5a414b050 0x7ff5a40aa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5a414b050 0x7ff5a40aa320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50814/0 (socket says 172.21.15.150:50814) 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a37fe640 1 -- 172.21.15.150:0/3950056260 learned_addr learned my addr 172.21.15.150:0/3950056260 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a8b92640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5a4144460 0x7ff5a41f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a3fff640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5a40732b0 0x7ff5a4145120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a37fe640 1 -- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5a40732b0 msgr2=0x7ff5a4145120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a37fe640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5a40732b0 0x7ff5a4145120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a37fe640 1 -- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5a4144460 msgr2=0x7ff5a41f0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a37fe640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5a4144460 0x7ff5a41f0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a37fe640 1 -- 172.21.15.150:0/3950056260 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a40afde0 con 0x7ff5a414b050 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a37fe640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5a414b050 0x7ff5a40aa320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff59801cca0 tx=0x7ff5980a6510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a0ff9640 1 -- 172.21.15.150:0/3950056260 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5980ac070 con 0x7ff5a414b050 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a0ff9640 1 -- 172.21.15.150:0/3950056260 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff598004340 con 0x7ff5a414b050 2024-09-17T04:51:17.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/3950056260 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a41f1470 con 0x7ff5a414b050 2024-09-17T04:51:17.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5a0ff9640 1 -- 172.21.15.150:0/3950056260 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598005320 con 0x7ff5a414b050 2024-09-17T04:51:17.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.398+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/3950056260 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a40aad80 con 0x7ff5a414b050 2024-09-17T04:51:17.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.402+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/3950056260 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a40c47e0 con 0x7ff5a414b050 2024-09-17T04:51:17.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.402+0000 7ff5a0ff9640 1 -- 172.21.15.150:0/3950056260 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff5980af070 con 0x7ff5a414b050 2024-09-17T04:51:17.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.402+0000 7ff5a0ff9640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff590073200 0x7ff590075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:17.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.402+0000 7ff5a0ff9640 1 -- 172.21.15.150:0/3950056260 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7ff59812eac0 con 0x7ff5a414b050 2024-09-17T04:51:17.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.402+0000 7ff5a3fff640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff590073200 0x7ff590075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:17.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.402+0000 7ff5a3fff640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff590073200 0x7ff590075710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff594001650 tx=0x7ff594009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:17.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.406+0000 7ff5a0ff9640 1 -- 172.21.15.150:0/3950056260 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff5980164d0 con 0x7ff5a414b050 2024-09-17T04:51:17.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.614+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/3950056260 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5a40739d0 con 0x7ff5a414b050 2024-09-17T04:51:17.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.614+0000 7ff5a0ff9640 1 -- 172.21.15.150:0/3950056260 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7ff5980f97c0 con 0x7ff5a414b050 2024-09-17T04:51:17.615 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:17.615 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:17.615 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:51:17.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.618+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff590073200 msgr2=0x7ff590075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:17.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.618+0000 7ff5aa5f5640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff590073200 0x7ff590075710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff594001650 tx=0x7ff594009040 comp rx=0 tx=0).stop 2024-09-17T04:51:17.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.618+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/3950056260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5a414b050 msgr2=0x7ff5a40aa320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:17.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.618+0000 7ff5aa5f5640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5a414b050 0x7ff5a40aa320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff59801cca0 tx=0x7ff5980a6510 comp rx=0 tx=0).stop 2024-09-17T04:51:17.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.618+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/3950056260 shutdown_connections 2024-09-17T04:51:17.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.618+0000 7ff5aa5f5640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff590073200 0x7ff590075710 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:17.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.618+0000 7ff5aa5f5640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5a4144460 0x7ff5a41f0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:17.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.618+0000 7ff5aa5f5640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff5a414b050 0x7ff5a40aa320 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:17.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.618+0000 7ff5aa5f5640 1 --2- 172.21.15.150:0/3950056260 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5a40732b0 0x7ff5a4145120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:17.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.618+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/3950056260 >> 172.21.15.150:0/3950056260 conn(0x7ff5a40abf10 msgr2=0x7ff5a4154320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:17.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.618+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/3950056260 shutdown_connections 2024-09-17T04:51:17.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:17.618+0000 7ff5aa5f5640 1 -- 172.21.15.150:0/3950056260 wait complete. 2024-09-17T04:51:17.637 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:51:17.961 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether build environment is sane... yes 2024-09-17T04:51:17.993 INFO:tasks.workunit.client.1.smithi176.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-17T04:51:17.994 INFO:tasks.workunit.client.1.smithi176.stdout:checking for gawk... gawk 2024-09-17T04:51:18.042 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether make sets $(MAKE)... yes 2024-09-17T04:51:18.099 INFO:tasks.workunit.client.1.smithi176.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-17T04:51:18.099 INFO:tasks.workunit.client.1.smithi176.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-17T04:51:18.140 INFO:tasks.workunit.client.1.smithi176.stdout:checking for style of include used by make... GNU 2024-09-17T04:51:18.154 INFO:tasks.workunit.client.1.smithi176.stdout:checking for gcc... gcc 2024-09-17T04:51:18.284 INFO:tasks.workunit.client.1.smithi176.stdout:checking for C compiler default output file name... a.out 2024-09-17T04:51:18.288 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether the C compiler works... yes 2024-09-17T04:51:18.294 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether we are cross compiling... no 2024-09-17T04:51:18.370 INFO:tasks.workunit.client.1.smithi176.stdout:checking for suffix of executables... 2024-09-17T04:51:18.430 INFO:tasks.workunit.client.1.smithi176.stdout:checking for suffix of object files... o 2024-09-17T04:51:18.527 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether we are using the GNU C compiler... yes 2024-09-17T04:51:18.589 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether gcc accepts -g... yes 2024-09-17T04:51:18.717 INFO:tasks.workunit.client.1.smithi176.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-17T04:51:19.039 INFO:tasks.workunit.client.1.smithi176.stdout:checking dependency style of gcc... gcc3 2024-09-17T04:51:19.152 INFO:tasks.workunit.client.1.smithi176.stdout:checking how to run the C preprocessor... gcc -E 2024-09-17T04:51:19.260 INFO:tasks.workunit.client.1.smithi176.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-17T04:51:19.262 INFO:tasks.workunit.client.1.smithi176.stdout:checking for egrep... /usr/bin/grep -E 2024-09-17T04:51:19.279 INFO:tasks.workunit.client.1.smithi176.stdout:checking for AIX... no 2024-09-17T04:51:19.336 INFO:tasks.workunit.client.1.smithi176.stdout:checking for gcc... (cached) gcc 2024-09-17T04:51:19.352 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-17T04:51:19.354 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-17T04:51:19.354 INFO:tasks.workunit.client.1.smithi176.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-17T04:51:19.355 INFO:tasks.workunit.client.1.smithi176.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-17T04:51:19.474 INFO:tasks.workunit.client.1.smithi176.stdout:checking for main in -lm... yes 2024-09-17T04:51:19.595 INFO:tasks.workunit.client.1.smithi176.stdout:checking for main in -lpthread... yes 2024-09-17T04:51:19.638 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.814+0000 7fae9eee9640 1 -- 172.21.15.150:0/895469387 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae9806ee80 msgr2=0x7fae9806f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.814+0000 7fae9eee9640 1 --2- 172.21.15.150:0/895469387 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae9806ee80 0x7fae9806f290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009870 tx=0x7fae8c01c860 comp rx=0 tx=0).stop 2024-09-17T04:51:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.814+0000 7fae9eee9640 1 -- 172.21.15.150:0/895469387 shutdown_connections 2024-09-17T04:51:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.814+0000 7fae9eee9640 1 --2- 172.21.15.150:0/895469387 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae9806ee80 0x7fae9806f290 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:19.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.814+0000 7fae9eee9640 1 -- 172.21.15.150:0/895469387 >> 172.21.15.150:0/895469387 conn(0x7fae98074b80 msgr2=0x7fae980cbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:19.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.814+0000 7fae9eee9640 1 -- 172.21.15.150:0/895469387 shutdown_connections 2024-09-17T04:51:19.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.814+0000 7fae9eee9640 1 -- 172.21.15.150:0/895469387 wait complete. 2024-09-17T04:51:19.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9eee9640 1 Processor -- start 2024-09-17T04:51:19.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9eee9640 1 -- start start 2024-09-17T04:51:19.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9eee9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae9806ee80 0x7fae981f3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:19.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9eee9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae980c6ef0 0x7fae981445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:19.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9eee9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae98157220 0x7fae98158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:19.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9eee9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fae980c0f20 con 0x7fae98157220 2024-09-17T04:51:19.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9eee9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fae980ccd20 con 0x7fae980c6ef0 2024-09-17T04:51:19.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9eee9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fae980c10a0 con 0x7fae9806ee80 2024-09-17T04:51:19.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9d486640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae98157220 0x7fae98158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:19.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9cc85640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae9806ee80 0x7fae981f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:19.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae97fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae980c6ef0 0x7fae981445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9d486640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae98157220 0x7fae98158a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50590/0 (socket says 172.21.15.150:50590) 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9cc85640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae9806ee80 0x7fae981f3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:41340/0 (socket says 172.21.15.150:41340) 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae97fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae980c6ef0 0x7fae981445b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:46736/0 (socket says 172.21.15.150:46736) 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9d486640 1 -- 172.21.15.150:0/266010258 learned_addr learned my addr 172.21.15.150:0/266010258 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9d486640 1 -- 172.21.15.150:0/266010258 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae9806ee80 msgr2=0x7fae981f3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9d486640 1 --2- 172.21.15.150:0/266010258 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae9806ee80 0x7fae981f3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9d486640 1 -- 172.21.15.150:0/266010258 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae980c6ef0 msgr2=0x7fae981445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9d486640 1 --2- 172.21.15.150:0/266010258 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae980c6ef0 0x7fae981445b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9d486640 1 -- 172.21.15.150:0/266010258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae981f3b10 con 0x7fae98157220 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9d486640 1 --2- 172.21.15.150:0/266010258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae98157220 0x7fae98158a90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c730 tx=0x7fae9000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae957fa640 1 -- 172.21.15.150:0/266010258 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90003b80 con 0x7fae98157220 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9eee9640 1 -- 172.21.15.150:0/266010258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae981f3df0 con 0x7fae98157220 2024-09-17T04:51:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae957fa640 1 -- 172.21.15.150:0/266010258 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae90003d60 con 0x7fae98157220 2024-09-17T04:51:19.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae957fa640 1 -- 172.21.15.150:0/266010258 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90005330 con 0x7fae98157220 2024-09-17T04:51:19.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9eee9640 1 -- 172.21.15.150:0/266010258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae981473b0 con 0x7fae98157220 2024-09-17T04:51:19.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.818+0000 7fae9eee9640 1 -- 172.21.15.150:0/266010258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae9806f290 con 0x7fae98157220 2024-09-17T04:51:19.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.822+0000 7fae957fa640 1 -- 172.21.15.150:0/266010258 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fae900056e0 con 0x7fae98157220 2024-09-17T04:51:19.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.822+0000 7fae957fa640 1 --2- 172.21.15.150:0/266010258 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae84073200 0x7fae84075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:19.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.822+0000 7fae957fa640 1 -- 172.21.15.150:0/266010258 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7fae90094f80 con 0x7fae98157220 2024-09-17T04:51:19.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.822+0000 7fae9cc85640 1 --2- 172.21.15.150:0/266010258 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae84073200 0x7fae84075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:19.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.822+0000 7fae9cc85640 1 --2- 172.21.15.150:0/266010258 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae84073200 0x7fae84075710 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fae88000c10 tx=0x7fae88001590 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:19.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:19.826+0000 7fae957fa640 1 -- 172.21.15.150:0/266010258 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fae9005f620 con 0x7fae98157220 2024-09-17T04:51:19.849 INFO:tasks.workunit.client.1.smithi176.stdout:checking for ANSI C header files... yes 2024-09-17T04:51:19.924 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-17T04:51:20.018 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/types.h... yes 2024-09-17T04:51:20.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.070+0000 7fae9eee9640 1 -- 172.21.15.150:0/266010258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae981583d0 con 0x7fae98157220 2024-09-17T04:51:20.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae957fa640 1 -- 172.21.15.150:0/266010258 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fae90064050 con 0x7fae98157220 2024-09-17T04:51:20.073 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:20.073 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:20.073 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:51:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae9eee9640 1 -- 172.21.15.150:0/266010258 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae84073200 msgr2=0x7fae84075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae9eee9640 1 --2- 172.21.15.150:0/266010258 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae84073200 0x7fae84075710 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fae88000c10 tx=0x7fae88001590 comp rx=0 tx=0).stop 2024-09-17T04:51:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae9eee9640 1 -- 172.21.15.150:0/266010258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae98157220 msgr2=0x7fae98158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae9eee9640 1 --2- 172.21.15.150:0/266010258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae98157220 0x7fae98158a90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c730 tx=0x7fae9000cbf0 comp rx=0 tx=0).stop 2024-09-17T04:51:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae9eee9640 1 -- 172.21.15.150:0/266010258 shutdown_connections 2024-09-17T04:51:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae9eee9640 1 --2- 172.21.15.150:0/266010258 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae84073200 0x7fae84075710 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae9eee9640 1 --2- 172.21.15.150:0/266010258 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae980c6ef0 0x7fae981445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae9eee9640 1 --2- 172.21.15.150:0/266010258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae98157220 0x7fae98158a90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae9eee9640 1 --2- 172.21.15.150:0/266010258 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae9806ee80 0x7fae981f3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:20.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae9eee9640 1 -- 172.21.15.150:0/266010258 >> 172.21.15.150:0/266010258 conn(0x7fae98074b80 msgr2=0x7fae980c5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:20.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae9eee9640 1 -- 172.21.15.150:0/266010258 shutdown_connections 2024-09-17T04:51:20.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:20.074+0000 7fae9eee9640 1 -- 172.21.15.150:0/266010258 wait complete. 2024-09-17T04:51:20.096 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:51:20.130 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/stat.h... yes 2024-09-17T04:51:20.242 INFO:tasks.workunit.client.1.smithi176.stdout:checking for stdlib.h... yes 2024-09-17T04:51:20.346 INFO:tasks.workunit.client.1.smithi176.stdout:checking for string.h... yes 2024-09-17T04:51:20.445 INFO:tasks.workunit.client.1.smithi176.stdout:checking for memory.h... yes 2024-09-17T04:51:20.555 INFO:tasks.workunit.client.1.smithi176.stdout:checking for strings.h... yes 2024-09-17T04:51:20.661 INFO:tasks.workunit.client.1.smithi176.stdout:checking for inttypes.h... yes 2024-09-17T04:51:20.759 INFO:tasks.workunit.client.1.smithi176.stdout:checking for stdint.h... yes 2024-09-17T04:51:20.859 INFO:tasks.workunit.client.1.smithi176.stdout:checking for unistd.h... yes 2024-09-17T04:51:20.978 INFO:tasks.workunit.client.1.smithi176.stdout:checking pthread.h usability... yes 2024-09-17T04:51:21.020 INFO:tasks.workunit.client.1.smithi176.stdout:checking pthread.h presence... yes 2024-09-17T04:51:21.021 INFO:tasks.workunit.client.1.smithi176.stdout:checking for pthread.h... yes 2024-09-17T04:51:21.218 INFO:tasks.workunit.client.1.smithi176.stdout:checking fcntl.h usability... yes 2024-09-17T04:51:21.255 INFO:tasks.workunit.client.1.smithi176.stdout:checking fcntl.h presence... yes 2024-09-17T04:51:21.256 INFO:tasks.workunit.client.1.smithi176.stdout:checking for fcntl.h... yes 2024-09-17T04:51:21.359 INFO:tasks.workunit.client.1.smithi176.stdout:checking limits.h usability... yes 2024-09-17T04:51:21.397 INFO:tasks.workunit.client.1.smithi176.stdout:checking limits.h presence... yes 2024-09-17T04:51:21.397 INFO:tasks.workunit.client.1.smithi176.stdout:checking for limits.h... yes 2024-09-17T04:51:21.440 INFO:tasks.workunit.client.1.smithi176.stdout:checking for stdint.h... (cached) yes 2024-09-17T04:51:21.540 INFO:tasks.workunit.client.1.smithi176.stdout:checking sys/time.h usability... yes 2024-09-17T04:51:21.578 INFO:tasks.workunit.client.1.smithi176.stdout:checking sys/time.h presence... yes 2024-09-17T04:51:21.578 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/time.h... yes 2024-09-17T04:51:21.645 INFO:tasks.workunit.client.1.smithi176.stdout:checking for unistd.h... (cached) yes 2024-09-17T04:51:21.730 INFO:tasks.workunit.client.1.smithi176.stdout:checking sys/vfs.h usability... yes 2024-09-17T04:51:21.768 INFO:tasks.workunit.client.1.smithi176.stdout:checking sys/vfs.h presence... yes 2024-09-17T04:51:21.769 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/vfs.h... yes 2024-09-17T04:51:21.880 INFO:tasks.workunit.client.1.smithi176.stdout:checking sys/limits.h usability... no 2024-09-17T04:51:21.923 INFO:tasks.workunit.client.1.smithi176.stdout:checking sys/limits.h presence... no 2024-09-17T04:51:21.923 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/limits.h... no 2024-09-17T04:51:21.981 INFO:tasks.workunit.client.1.smithi176.stdout:checking for an ANSI C-conforming const... yes 2024-09-17T04:51:22.038 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-17T04:51:22.042 INFO:tasks.workunit.client.1.smithi176.stdout:checking for function prototypes... yes 2024-09-17T04:51:22.097 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:22.103 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether setvbuf arguments are reversed... no 2024-09-17T04:51:22.227 INFO:tasks.workunit.client.1.smithi176.stdout:checking for system... yes 2024-09-17T04:51:22.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.266+0000 7f422d652640 1 -- 172.21.15.150:0/1495912357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4228157ca0 msgr2=0x7f42281580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:22.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.266+0000 7f422d652640 1 --2- 172.21.15.150:0/1495912357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4228157ca0 0x7f42281580b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f42280c7870 tx=0x7f422001cc60 comp rx=0 tx=0).stop 2024-09-17T04:51:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.266+0000 7f422d652640 1 -- 172.21.15.150:0/1495912357 shutdown_connections 2024-09-17T04:51:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.266+0000 7f422d652640 1 --2- 172.21.15.150:0/1495912357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4228157ca0 0x7f42281580b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.266+0000 7f422d652640 1 --2- 172.21.15.150:0/1495912357 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4228151e10 0x7f42280c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.266+0000 7f422d652640 1 --2- 172.21.15.150:0/1495912357 >> v2:172.21.15.176:3300/0 conn(0x7f42280ad7f0 0x7f42280c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.266+0000 7f422d652640 1 -- 172.21.15.150:0/1495912357 >> 172.21.15.150:0/1495912357 conn(0x7f42280c9de0 msgr2=0x7f42280ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.266+0000 7f422d652640 1 -- 172.21.15.150:0/1495912357 shutdown_connections 2024-09-17T04:51:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.266+0000 7f422d652640 1 -- 172.21.15.150:0/1495912357 wait complete. 2024-09-17T04:51:22.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f422d652640 1 Processor -- start 2024-09-17T04:51:22.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f422d652640 1 -- start start 2024-09-17T04:51:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f422d652640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42280ad7f0 0x7f422814f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f422d652640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4228151e10 0x7f422814d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f422d652640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4228157ca0 0x7f42281ef5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f422d652640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f42280c5870 con 0x7f42280ad7f0 2024-09-17T04:51:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f422d652640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4228152330 con 0x7f4228151e10 2024-09-17T04:51:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f422d652640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f42280c59f0 con 0x7f4228157ca0 2024-09-17T04:51:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f4226ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42280ad7f0 0x7f422814f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f4226ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42280ad7f0 0x7f422814f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50602/0 (socket says 172.21.15.150:50602) 2024-09-17T04:51:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f4226ffd640 1 -- 172.21.15.150:0/539694698 learned_addr learned my addr 172.21.15.150:0/539694698 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f42277fe640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4228157ca0 0x7f42281ef5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f421ffff640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4228151e10 0x7f422814d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f42277fe640 1 -- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4228151e10 msgr2=0x7f422814d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f42277fe640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4228151e10 0x7f422814d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f42277fe640 1 -- 172.21.15.150:0/539694698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42280ad7f0 msgr2=0x7f422814f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f42277fe640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42280ad7f0 0x7f422814f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f42277fe640 1 -- 172.21.15.150:0/539694698 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422806f1d0 con 0x7f4228157ca0 2024-09-17T04:51:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f421ffff640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4228151e10 0x7f422814d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:51:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f4226ffd640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42280ad7f0 0x7f422814f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:51:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f42277fe640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4228157ca0 0x7f42281ef5c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f422806dd50 tx=0x7f4220005a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f421f7fe640 1 -- 172.21.15.150:0/539694698 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42200b2040 con 0x7f4228157ca0 2024-09-17T04:51:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f422d652640 1 -- 172.21.15.150:0/539694698 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4228150e50 con 0x7f4228157ca0 2024-09-17T04:51:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f421f7fe640 1 -- 172.21.15.150:0/539694698 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4220003fa0 con 0x7f4228157ca0 2024-09-17T04:51:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f421f7fe640 1 -- 172.21.15.150:0/539694698 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4220004410 con 0x7f4228157ca0 2024-09-17T04:51:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f422d652640 1 -- 172.21.15.150:0/539694698 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f42280c39e0 con 0x7f4228157ca0 2024-09-17T04:51:22.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.270+0000 7f422d652640 1 -- 172.21.15.150:0/539694698 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4228149cc0 con 0x7f4228157ca0 2024-09-17T04:51:22.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.274+0000 7f421f7fe640 1 -- 172.21.15.150:0/539694698 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f42200045b0 con 0x7f4228157ca0 2024-09-17T04:51:22.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.274+0000 7f421f7fe640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4208073230 0x7f4208075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:22.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.274+0000 7f421f7fe640 1 -- 172.21.15.150:0/539694698 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f422012f3f0 con 0x7f4228157ca0 2024-09-17T04:51:22.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.274+0000 7f4226ffd640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4208073230 0x7f4208075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:22.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.274+0000 7f4226ffd640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4208073230 0x7f4208075740 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4218011440 tx=0x7f4218005f40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:22.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.282+0000 7f421f7fe640 1 -- 172.21.15.150:0/539694698 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4220016970 con 0x7f4228157ca0 2024-09-17T04:51:22.341 INFO:tasks.workunit.client.1.smithi176.stdout:checking for gettimeofday... yes 2024-09-17T04:51:22.451 INFO:tasks.workunit.client.1.smithi176.stdout:checking for mkdir... yes 2024-09-17T04:51:22.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.510+0000 7f422d652640 1 -- 172.21.15.150:0/539694698 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42280c5330 con 0x7f4228157ca0 2024-09-17T04:51:22.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.510+0000 7f421f7fe640 1 -- 172.21.15.150:0/539694698 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7f42200f9a80 con 0x7f4228157ca0 2024-09-17T04:51:22.512 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:22.513 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:22.513 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:51:22.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.514+0000 7f422d652640 1 -- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4208073230 msgr2=0x7f4208075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:22.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.514+0000 7f422d652640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4208073230 0x7f4208075740 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4218011440 tx=0x7f4218005f40 comp rx=0 tx=0).stop 2024-09-17T04:51:22.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.514+0000 7f422d652640 1 -- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4228157ca0 msgr2=0x7f42281ef5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:22.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.514+0000 7f422d652640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4228157ca0 0x7f42281ef5c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f422806dd50 tx=0x7f4220005a60 comp rx=0 tx=0).stop 2024-09-17T04:51:22.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.514+0000 7f422d652640 1 -- 172.21.15.150:0/539694698 shutdown_connections 2024-09-17T04:51:22.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.514+0000 7f422d652640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4208073230 0x7f4208075740 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:22.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.514+0000 7f422d652640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4228151e10 0x7f422814d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:22.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.514+0000 7f422d652640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4228157ca0 0x7f42281ef5c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:22.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.514+0000 7f422d652640 1 --2- 172.21.15.150:0/539694698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42280ad7f0 0x7f422814f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:22.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.514+0000 7f422d652640 1 -- 172.21.15.150:0/539694698 >> 172.21.15.150:0/539694698 conn(0x7f42280c9de0 msgr2=0x7f4228068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:22.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.514+0000 7f422d652640 1 -- 172.21.15.150:0/539694698 shutdown_connections 2024-09-17T04:51:22.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:22.514+0000 7f422d652640 1 -- 172.21.15.150:0/539694698 wait complete. 2024-09-17T04:51:22.533 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:51:22.590 INFO:tasks.workunit.client.1.smithi176.stdout:checking for strerror... yes 2024-09-17T04:51:22.733 INFO:tasks.workunit.client.1.smithi176.stdout:checking for open64... yes 2024-09-17T04:51:22.849 INFO:tasks.workunit.client.1.smithi176.stdout:checking for stat64... yes 2024-09-17T04:51:22.972 INFO:tasks.workunit.client.1.smithi176.stdout:checking for fseeko64... yes 2024-09-17T04:51:23.094 INFO:tasks.workunit.client.1.smithi176.stdout:checking for lrand48_r... yes 2024-09-17T04:51:23.219 INFO:tasks.workunit.client.1.smithi176.stdout:checking for srand48_r... yes 2024-09-17T04:51:23.274 INFO:tasks.workunit.client.1.smithi176.stdout:configure: creating ./config.status 2024-09-17T04:51:23.447 INFO:tasks.workunit.client.1.smithi176.stdout:config.status: creating Makefile 2024-09-17T04:51:23.468 INFO:tasks.workunit.client.1.smithi176.stdout:config.status: creating config.h 2024-09-17T04:51:23.503 INFO:tasks.workunit.client.1.smithi176.stdout:config.status: executing depfiles commands 2024-09-17T04:51:23.609 INFO:tasks.workunit.client.1.smithi176.stdout:config.status: executing default commands 2024-09-17T04:51:23.758 INFO:tasks.workunit.client.1.smithi176.stderr:+ make 2024-09-17T04:51:23.812 INFO:tasks.workunit.client.1.smithi176.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run aclocal-1.10 2024-09-17T04:51:23.819 INFO:tasks.workunit.client.1.smithi176.stderr:/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-09-17T04:51:23.820 INFO:tasks.workunit.client.1.smithi176.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-09-17T04:51:23.820 INFO:tasks.workunit.client.1.smithi176.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-09-17T04:51:23.820 INFO:tasks.workunit.client.1.smithi176.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-09-17T04:51:23.820 INFO:tasks.workunit.client.1.smithi176.stderr: any GNU archive site. 2024-09-17T04:51:23.826 INFO:tasks.workunit.client.1.smithi176.stdout: cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run automake-1.10 --gnu 2024-09-17T04:51:23.831 INFO:tasks.workunit.client.1.smithi176.stderr:/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-09-17T04:51:23.832 INFO:tasks.workunit.client.1.smithi176.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-09-17T04:51:23.832 INFO:tasks.workunit.client.1.smithi176.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-09-17T04:51:23.832 INFO:tasks.workunit.client.1.smithi176.stderr: You might want to install the `Automake' and `Perl' packages. 2024-09-17T04:51:23.832 INFO:tasks.workunit.client.1.smithi176.stderr: Grab them from any GNU archive site. 2024-09-17T04:51:24.174 INFO:tasks.workunit.client.1.smithi176.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run autoconf 2024-09-17T04:51:24.324 INFO:tasks.workunit.client.1.smithi176.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-17T04:51:24.324 INFO:tasks.workunit.client.1.smithi176.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-17T04:51:24.324 INFO:tasks.workunit.client.1.smithi176.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-17T04:51:24.324 INFO:tasks.workunit.client.1.smithi176.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-17T04:51:24.453 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:4: warning: The macro `AC_FOREACH' is obsolete. 2024-09-17T04:51:24.453 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:4: You should run autoupdate. 2024-09-17T04:51:24.453 INFO:tasks.workunit.client.1.smithi176.stderr:./lib/autoconf/general.m4:191: AC_FOREACH is expanded from... 2024-09-17T04:51:24.453 INFO:tasks.workunit.client.1.smithi176.stderr:aclocal.m4:684: _AM_SET_OPTIONS is expanded from... 2024-09-17T04:51:24.453 INFO:tasks.workunit.client.1.smithi176.stderr:aclocal.m4:410: AM_INIT_AUTOMAKE is expanded from... 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:4: the top level 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:6: warning: The macro `AM_CONFIG_HEADER' is obsolete. 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:6: You should run autoupdate. 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:aclocal.m4:383: AM_CONFIG_HEADER is expanded from... 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:6: the top level 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:10: warning: The macro `AC_AIX' is obsolete. 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:10: You should run autoupdate. 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:./lib/autoconf/specific.m4:540: AC_AIX is expanded from... 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:10: the top level 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:36: warning: The macro `AC_HEADER_STDC' is obsolete. 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:36: You should run autoupdate. 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from... 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:36: the top level 2024-09-17T04:51:24.454 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:42: warning: The macro `AC_HEADER_TIME' is obsolete. 2024-09-17T04:51:24.455 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:42: You should run autoupdate. 2024-09-17T04:51:24.455 INFO:tasks.workunit.client.1.smithi176.stderr:./lib/autoconf/headers.m4:743: AC_HEADER_TIME is expanded from... 2024-09-17T04:51:24.455 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:42: the top level 2024-09-17T04:51:24.455 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:45: warning: The macro `AC_FUNC_SETVBUF_REVERSED' is obsolete. Remove it and all references to SETVBUF_REVERSED. 2024-09-17T04:51:24.455 INFO:tasks.workunit.client.1.smithi176.stderr:./lib/autoconf/functions.m4:1766: AC_FUNC_SETVBUF_REVERSED is expanded from... 2024-09-17T04:51:24.455 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:45: the top level 2024-09-17T04:51:24.455 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:50: warning: AC_OUTPUT should be used without arguments. 2024-09-17T04:51:24.455 INFO:tasks.workunit.client.1.smithi176.stderr:configure.in:50: You should run autoupdate. 2024-09-17T04:51:24.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:24.557 INFO:tasks.workunit.client.1.smithi176.stdout:/bin/bash ./config.status --recheck 2024-09-17T04:51:24.654 INFO:tasks.workunit.client.1.smithi176.stdout:running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion 2024-09-17T04:51:24.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.702+0000 7fdde6ce2640 1 -- 172.21.15.150:0/3710817579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde00c20a0 msgr2=0x7fdde00c24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:24.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.702+0000 7fdde6ce2640 1 --2- 172.21.15.150:0/3710817579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde00c20a0 0x7fdde00c24b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009840 tx=0x7fddd001c7f0 comp rx=0 tx=0).stop 2024-09-17T04:51:24.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.702+0000 7fdde6ce2640 1 -- 172.21.15.150:0/3710817579 shutdown_connections 2024-09-17T04:51:24.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.702+0000 7fdde6ce2640 1 --2- 172.21.15.150:0/3710817579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde00c20a0 0x7fdde00c24b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:24.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.702+0000 7fdde6ce2640 1 -- 172.21.15.150:0/3710817579 >> 172.21.15.150:0/3710817579 conn(0x7fdde0074b80 msgr2=0x7fdde006f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:24.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.702+0000 7fdde6ce2640 1 -- 172.21.15.150:0/3710817579 shutdown_connections 2024-09-17T04:51:24.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.702+0000 7fdde6ce2640 1 -- 172.21.15.150:0/3710817579 wait complete. 2024-09-17T04:51:24.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde6ce2640 1 Processor -- start 2024-09-17T04:51:24.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde6ce2640 1 -- start start 2024-09-17T04:51:24.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde6ce2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde00c20a0 0x7fdde0151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:24.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde6ce2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde00c35b0 0x7fdde01f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:24.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde6ce2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde01568a0 0x7fdde014b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:24.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde6ce2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdde00c3b40 con 0x7fdde00c20a0 2024-09-17T04:51:24.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde6ce2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdde00c39c0 con 0x7fdde01568a0 2024-09-17T04:51:24.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde6ce2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdde00c69f0 con 0x7fdde00c35b0 2024-09-17T04:51:24.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde4a7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde00c20a0 0x7fdde0151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:24.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde4a7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde00c20a0 0x7fdde0151240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50614/0 (socket says 172.21.15.150:50614) 2024-09-17T04:51:24.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde4a7e640 1 -- 172.21.15.150:0/2019864235 learned_addr learned my addr 172.21.15.150:0/2019864235 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:24.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde527f640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde01568a0 0x7fdde014b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:24.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdddffff640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde00c35b0 0x7fdde01f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:24.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde4a7e640 1 -- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde00c35b0 msgr2=0x7fdde01f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:24.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde4a7e640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde00c35b0 0x7fdde01f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:24.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde4a7e640 1 -- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde01568a0 msgr2=0x7fdde014b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:24.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde4a7e640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde01568a0 0x7fdde014b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:24.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde4a7e640 1 -- 172.21.15.150:0/2019864235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde0069570 con 0x7fdde00c20a0 2024-09-17T04:51:24.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde4a7e640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde00c20a0 0x7fdde0151240 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c7b0 tx=0x7fddd800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:24.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdddd7fa640 1 -- 172.21.15.150:0/2019864235 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8003b80 con 0x7fdde00c20a0 2024-09-17T04:51:24.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdddd7fa640 1 -- 172.21.15.150:0/2019864235 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fddd8003d20 con 0x7fdde00c20a0 2024-09-17T04:51:24.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde6ce2640 1 -- 172.21.15.150:0/2019864235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde01f37b0 con 0x7fdde00c20a0 2024-09-17T04:51:24.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdddd7fa640 1 -- 172.21.15.150:0/2019864235 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd800aab0 con 0x7fdde00c20a0 2024-09-17T04:51:24.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.706+0000 7fdde6ce2640 1 -- 172.21.15.150:0/2019864235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde015a0d0 con 0x7fdde00c20a0 2024-09-17T04:51:24.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.710+0000 7fdddd7fa640 1 -- 172.21.15.150:0/2019864235 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fddd800ac50 con 0x7fdde00c20a0 2024-09-17T04:51:24.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.710+0000 7fdde6ce2640 1 -- 172.21.15.150:0/2019864235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdde00c24b0 con 0x7fdde00c20a0 2024-09-17T04:51:24.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.710+0000 7fdddd7fa640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fddc0073200 0x7fddc0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:24.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.710+0000 7fdddd7fa640 1 -- 172.21.15.150:0/2019864235 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7fddd8094210 con 0x7fdde00c20a0 2024-09-17T04:51:24.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.710+0000 7fdddffff640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fddc0073200 0x7fddc0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:24.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.710+0000 7fdddffff640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fddc0073200 0x7fddc0075710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fddc8007f30 tx=0x7fddc8007ec0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:24.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.714+0000 7fdddd7fa640 1 -- 172.21.15.150:0/2019864235 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fddd805e940 con 0x7fdde00c20a0 2024-09-17T04:51:24.854 INFO:tasks.workunit.client.1.smithi176.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-17T04:51:24.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.938+0000 7fdde6ce2640 1 -- 172.21.15.150:0/2019864235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdde0157790 con 0x7fdde00c20a0 2024-09-17T04:51:24.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.938+0000 7fdddd7fa640 1 -- 172.21.15.150:0/2019864235 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6531 (secure 0 0 0) 0x7fddd8063370 con 0x7fdde00c20a0 2024-09-17T04:51:24.939 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:24.939 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"btime":"2024-09-17T04:49:45:061664+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4358,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:49:35.350600+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4404":{"gid":4404,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6835/2613040311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6833","nonce":2613040311},{"type":"v1","addr":"172.21.15.150:6835","nonce":2613040311}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4404,"qdb_cluster":[4404]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:24.940 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-09-17T04:51:24.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.942+0000 7fdde6ce2640 1 -- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fddc0073200 msgr2=0x7fddc0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:24.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.942+0000 7fdde6ce2640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fddc0073200 0x7fddc0075710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fddc8007f30 tx=0x7fddc8007ec0 comp rx=0 tx=0).stop 2024-09-17T04:51:24.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.942+0000 7fdde6ce2640 1 -- 172.21.15.150:0/2019864235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde00c20a0 msgr2=0x7fdde0151240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:24.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.942+0000 7fdde6ce2640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde00c20a0 0x7fdde0151240 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fddd800c7b0 tx=0x7fddd800cc70 comp rx=0 tx=0).stop 2024-09-17T04:51:24.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.942+0000 7fdde6ce2640 1 -- 172.21.15.150:0/2019864235 shutdown_connections 2024-09-17T04:51:24.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.942+0000 7fdde6ce2640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fddc0073200 0x7fddc0075710 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:24.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.942+0000 7fdde6ce2640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde01568a0 0x7fdde014b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:24.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.942+0000 7fdde6ce2640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde00c35b0 0x7fdde01f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:24.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.942+0000 7fdde6ce2640 1 --2- 172.21.15.150:0/2019864235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdde00c20a0 0x7fdde0151240 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:24.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.942+0000 7fdde6ce2640 1 -- 172.21.15.150:0/2019864235 >> 172.21.15.150:0/2019864235 conn(0x7fdde0074b80 msgr2=0x7fdde0154ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:24.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.942+0000 7fdde6ce2640 1 -- 172.21.15.150:0/2019864235 shutdown_connections 2024-09-17T04:51:24.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:24.942+0000 7fdde6ce2640 1 -- 172.21.15.150:0/2019864235 wait complete. 2024-09-17T04:51:24.960 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:51:25.881 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether build environment is sane... yes 2024-09-17T04:51:25.884 INFO:tasks.workunit.client.1.smithi176.stdout:checking for a race-free mkdir -p... /usr/bin/mkdir -p 2024-09-17T04:51:25.885 INFO:tasks.workunit.client.1.smithi176.stdout:checking for gawk... gawk 2024-09-17T04:51:25.925 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether make sets $(MAKE)... yes 2024-09-17T04:51:25.978 INFO:tasks.workunit.client.1.smithi176.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-17T04:51:25.979 INFO:tasks.workunit.client.1.smithi176.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-17T04:51:26.013 INFO:tasks.workunit.client.1.smithi176.stdout:checking for style of include used by make... GNU 2024-09-17T04:51:26.023 INFO:tasks.workunit.client.1.smithi176.stdout:checking for gcc... gcc 2024-09-17T04:51:26.265 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether the C compiler works... yes 2024-09-17T04:51:26.265 INFO:tasks.workunit.client.1.smithi176.stdout:checking for C compiler default output file name... a.out 2024-09-17T04:51:26.338 INFO:tasks.workunit.client.1.smithi176.stdout:checking for suffix of executables... 2024-09-17T04:51:26.404 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether we are cross compiling... no 2024-09-17T04:51:26.474 INFO:tasks.workunit.client.1.smithi176.stdout:checking for suffix of object files... o 2024-09-17T04:51:26.514 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether the compiler supports GNU C... yes 2024-09-17T04:51:26.555 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether gcc accepts -g... yes 2024-09-17T04:51:26.652 INFO:tasks.workunit.client.1.smithi176.stdout:checking for gcc option to enable C11 features... none needed 2024-09-17T04:51:26.922 INFO:tasks.workunit.client.1.smithi176.stdout:checking dependency style of gcc... gcc3 2024-09-17T04:51:26.961 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:26.973 INFO:tasks.workunit.client.1.smithi176.stdout:checking for stdio.h... yes 2024-09-17T04:51:27.037 INFO:tasks.workunit.client.1.smithi176.stdout:checking for stdlib.h... yes 2024-09-17T04:51:27.088 INFO:tasks.workunit.client.1.smithi176.stdout:checking for string.h... yes 2024-09-17T04:51:27.139 INFO:tasks.workunit.client.1.smithi176.stdout:checking for inttypes.h... yes 2024-09-17T04:51:27.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.142+0000 7fec891e7640 1 -- 172.21.15.150:0/1151166569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec840c4250 msgr2=0x7fec84153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:27.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.142+0000 7fec891e7640 1 --2- 172.21.15.150:0/1151166569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec840c4250 0x7fec84153ee0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fec74009840 tx=0x7fec7401c740 comp rx=0 tx=0).stop 2024-09-17T04:51:27.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.142+0000 7fec891e7640 1 -- 172.21.15.150:0/1151166569 shutdown_connections 2024-09-17T04:51:27.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.142+0000 7fec891e7640 1 --2- 172.21.15.150:0/1151166569 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec840c3d10 0x7fec8415df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.142+0000 7fec891e7640 1 --2- 172.21.15.150:0/1151166569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec840c4250 0x7fec84153ee0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.142+0000 7fec891e7640 1 -- 172.21.15.150:0/1151166569 >> 172.21.15.150:0/1151166569 conn(0x7fec8406e360 msgr2=0x7fec8406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:27.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.142+0000 7fec891e7640 1 -- 172.21.15.150:0/1151166569 shutdown_connections 2024-09-17T04:51:27.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.142+0000 7fec891e7640 1 -- 172.21.15.150:0/1151166569 wait complete. 2024-09-17T04:51:27.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.146+0000 7fec891e7640 1 Processor -- start 2024-09-17T04:51:27.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.146+0000 7fec891e7640 1 -- start start 2024-09-17T04:51:27.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec891e7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec840c3900 0x7fec840c2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:27.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec891e7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec840c3d10 0x7fec841f0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:27.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec891e7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec840c4250 0x7fec8415d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:27.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec891e7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fec84071120 con 0x7fec840c3900 2024-09-17T04:51:27.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec891e7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fec84070fa0 con 0x7fec840c4250 2024-09-17T04:51:27.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec891e7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fec84072db0 con 0x7fec840c3d10 2024-09-17T04:51:27.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec82d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec840c3900 0x7fec840c2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:27.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec82d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec840c3900 0x7fec840c2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50640/0 (socket says 172.21.15.150:50640) 2024-09-17T04:51:27.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec82d9d640 1 -- 172.21.15.150:0/3679375500 learned_addr learned my addr 172.21.15.150:0/3679375500 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:27.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec8359e640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec840c4250 0x7fec8415d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:27.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec8259c640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec840c3d10 0x7fec841f0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:27.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec82d9d640 1 -- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec840c3d10 msgr2=0x7fec841f0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:27.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec82d9d640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec840c3d10 0x7fec841f0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec82d9d640 1 -- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec840c4250 msgr2=0x7fec8415d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:27.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec82d9d640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec840c4250 0x7fec8415d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec82d9d640 1 -- 172.21.15.150:0/3679375500 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8414c870 con 0x7fec840c3900 2024-09-17T04:51:27.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec82d9d640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec840c3900 0x7fec840c2340 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fec7401c570 tx=0x7fec74005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:27.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec6b7fe640 1 -- 172.21.15.150:0/3679375500 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec740ad070 con 0x7fec840c3900 2024-09-17T04:51:27.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec6b7fe640 1 -- 172.21.15.150:0/3679375500 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec740aab70 con 0x7fec840c3900 2024-09-17T04:51:27.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec6b7fe640 1 -- 172.21.15.150:0/3679375500 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec740046d0 con 0x7fec840c3900 2024-09-17T04:51:27.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec891e7640 1 -- 172.21.15.150:0/3679375500 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8414c4c0 con 0x7fec840c3900 2024-09-17T04:51:27.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec891e7640 1 -- 172.21.15.150:0/3679375500 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec84151d00 con 0x7fec840c3900 2024-09-17T04:51:27.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.150+0000 7fec891e7640 1 -- 172.21.15.150:0/3679375500 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec84153ee0 con 0x7fec840c3900 2024-09-17T04:51:27.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.154+0000 7fec6b7fe640 1 -- 172.21.15.150:0/3679375500 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fec740b0070 con 0x7fec840c3900 2024-09-17T04:51:27.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.154+0000 7fec6b7fe640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec70073200 0x7fec70075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:27.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.154+0000 7fec6b7fe640 1 -- 172.21.15.150:0/3679375500 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7fec7412e9f0 con 0x7fec840c3900 2024-09-17T04:51:27.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.154+0000 7fec8259c640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec70073200 0x7fec70075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:27.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.154+0000 7fec8259c640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec70073200 0x7fec70075710 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fec84072570 tx=0x7fec78001e00 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.158+0000 7fec6b7fe640 1 -- 172.21.15.150:0/3679375500 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fec740c1070 con 0x7fec840c3900 2024-09-17T04:51:27.185 INFO:tasks.workunit.client.1.smithi176.stdout:checking for stdint.h... yes 2024-09-17T04:51:27.226 INFO:tasks.workunit.client.1.smithi176.stdout:checking for strings.h... yes 2024-09-17T04:51:27.269 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/stat.h... yes 2024-09-17T04:51:27.311 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/types.h... yes 2024-09-17T04:51:27.363 INFO:tasks.workunit.client.1.smithi176.stdout:checking for unistd.h... yes 2024-09-17T04:51:27.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.406+0000 7fec891e7640 1 -- 172.21.15.150:0/3679375500 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec840725a0 con 0x7fec840c3900 2024-09-17T04:51:27.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec6b7fe640 1 -- 172.21.15.150:0/3679375500 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5704 (secure 0 0 0) 0x7fec740b8070 con 0x7fec840c3900 2024-09-17T04:51:27.409 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:27.409 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":17,"btime":"2024-09-17T04:51:26:821262+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:25.844339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:27.409 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 17 2024-09-17T04:51:27.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec891e7640 1 -- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec70073200 msgr2=0x7fec70075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:27.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec891e7640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec70073200 0x7fec70075710 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fec84072570 tx=0x7fec78001e00 comp rx=0 tx=0).stop 2024-09-17T04:51:27.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec891e7640 1 -- 172.21.15.150:0/3679375500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec840c3900 msgr2=0x7fec840c2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:27.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec891e7640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec840c3900 0x7fec840c2340 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fec7401c570 tx=0x7fec74005d80 comp rx=0 tx=0).stop 2024-09-17T04:51:27.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec891e7640 1 -- 172.21.15.150:0/3679375500 shutdown_connections 2024-09-17T04:51:27.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec891e7640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec70073200 0x7fec70075710 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec891e7640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec840c4250 0x7fec8415d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec891e7640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec840c3d10 0x7fec841f0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec891e7640 1 --2- 172.21.15.150:0/3679375500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec840c3900 0x7fec840c2340 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec891e7640 1 -- 172.21.15.150:0/3679375500 >> 172.21.15.150:0/3679375500 conn(0x7fec8406e360 msgr2=0x7fec84150200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:27.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec891e7640 1 -- 172.21.15.150:0/3679375500 shutdown_connections 2024-09-17T04:51:27.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.410+0000 7fec891e7640 1 -- 172.21.15.150:0/3679375500 wait complete. 2024-09-17T04:51:27.430 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-17T04:51:27.430 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T04:51:27.430 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:27.439 INFO:tasks.workunit.client.1.smithi176.stdout:checking for wchar.h... yes 2024-09-17T04:51:27.523 INFO:tasks.workunit.client.1.smithi176.stdout:checking for minix/config.h... no 2024-09-17T04:51:27.574 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/time.h... yes 2024-09-17T04:51:27.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.590+0000 7fef61ca3640 1 -- 172.21.15.150:0/676877127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef5c0a9ab0 msgr2=0x7fef5c071a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:27.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.590+0000 7fef61ca3640 1 --2- 172.21.15.150:0/676877127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef5c0a9ab0 0x7fef5c071a20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fef5c0707d0 tx=0x7fef4401c890 comp rx=0 tx=0).stop 2024-09-17T04:51:27.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.590+0000 7fef61ca3640 1 -- 172.21.15.150:0/676877127 shutdown_connections 2024-09-17T04:51:27.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.590+0000 7fef61ca3640 1 --2- 172.21.15.150:0/676877127 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c071f60 0x7fef5c153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.590+0000 7fef61ca3640 1 --2- 172.21.15.150:0/676877127 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef5c0a9ab0 0x7fef5c071a20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.590+0000 7fef61ca3640 1 --2- 172.21.15.150:0/676877127 >> v2:172.21.15.176:3300/0 conn(0x7fef5c0c8e20 0x7fef5c14d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.590+0000 7fef61ca3640 1 -- 172.21.15.150:0/676877127 >> 172.21.15.150:0/676877127 conn(0x7fef5c0c8250 msgr2=0x7fef5c0c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:27.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.590+0000 7fef61ca3640 1 -- 172.21.15.150:0/676877127 shutdown_connections 2024-09-17T04:51:27.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.590+0000 7fef61ca3640 1 -- 172.21.15.150:0/676877127 wait complete. 2024-09-17T04:51:27.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef61ca3640 1 Processor -- start 2024-09-17T04:51:27.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef61ca3640 1 -- start start 2024-09-17T04:51:27.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef61ca3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef5c071f60 0x7fef5c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef61ca3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0a9ab0 0x7fef5c148c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef61ca3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c0c8e20 0x7fef5c143b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef61ca3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c0c2dd0 con 0x7fef5c071f60 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef61ca3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c0c2c50 con 0x7fef5c0c8e20 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef61ca3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fef5c0c2f50 con 0x7fef5c0a9ab0 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef5b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef5c071f60 0x7fef5c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef5b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef5c071f60 0x7fef5c0ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50658/0 (socket says 172.21.15.150:50658) 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef5b7fe640 1 -- 172.21.15.150:0/213999950 learned_addr learned my addr 172.21.15.150:0/213999950 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef5bfff640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c0c8e20 0x7fef5c143b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef53fff640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0a9ab0 0x7fef5c148c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef5b7fe640 1 -- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0a9ab0 msgr2=0x7fef5c148c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef5b7fe640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0a9ab0 0x7fef5c148c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef5b7fe640 1 -- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c0c8e20 msgr2=0x7fef5c143b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:27.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef5b7fe640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c0c8e20 0x7fef5c143b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef5b7fe640 1 -- 172.21.15.150:0/213999950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5c1520e0 con 0x7fef5c071f60 2024-09-17T04:51:27.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef5b7fe640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef5c071f60 0x7fef5c0ac8d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00e7d0 tx=0x7fef4c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:27.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef58ff9640 1 -- 172.21.15.150:0/213999950 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c00cab0 con 0x7fef5c071f60 2024-09-17T04:51:27.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef61ca3640 1 -- 172.21.15.150:0/213999950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5c1554c0 con 0x7fef5c071f60 2024-09-17T04:51:27.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef58ff9640 1 -- 172.21.15.150:0/213999950 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef4c00cd40 con 0x7fef5c071f60 2024-09-17T04:51:27.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef58ff9640 1 -- 172.21.15.150:0/213999950 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c010690 con 0x7fef5c071f60 2024-09-17T04:51:27.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef61ca3640 1 -- 172.21.15.150:0/213999950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef5c148140 con 0x7fef5c071f60 2024-09-17T04:51:27.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.594+0000 7fef61ca3640 1 -- 172.21.15.150:0/213999950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef5c073930 con 0x7fef5c071f60 2024-09-17T04:51:27.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.598+0000 7fef58ff9640 1 -- 172.21.15.150:0/213999950 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fef4c010a40 con 0x7fef5c071f60 2024-09-17T04:51:27.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.598+0000 7fef58ff9640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fef3c073230 0x7fef3c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:27.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.598+0000 7fef58ff9640 1 -- 172.21.15.150:0/213999950 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7fef4c095950 con 0x7fef5c071f60 2024-09-17T04:51:27.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.598+0000 7fef53fff640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fef3c073230 0x7fef3c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:27.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.598+0000 7fef53fff640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fef3c073230 0x7fef3c075740 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fef440af000 tx=0x7fef4401b0d0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:27.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.602+0000 7fef58ff9640 1 -- 172.21.15.150:0/213999950 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fef4c060030 con 0x7fef5c071f60 2024-09-17T04:51:27.645 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-09-17T04:51:27.695 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether _XOPEN_SOURCE should be defined... no 2024-09-17T04:51:27.735 INFO:tasks.workunit.client.1.smithi176.stdout:checking for gcc... (cached) gcc 2024-09-17T04:51:27.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.818+0000 7fef61ca3640 1 -- 172.21.15.150:0/213999950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef5c0719e0 con 0x7fef5c071f60 2024-09-17T04:51:27.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef58ff9640 1 -- 172.21.15.150:0/213999950 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5704 (secure 0 0 0) 0x7fef4c064a60 con 0x7fef5c071f60 2024-09-17T04:51:27.822 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:27.822 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":17,"btime":"2024-09-17T04:51:26:821262+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:25.844339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:27.822 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 17 2024-09-17T04:51:27.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef61ca3640 1 -- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fef3c073230 msgr2=0x7fef3c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:27.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef61ca3640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fef3c073230 0x7fef3c075740 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fef440af000 tx=0x7fef4401b0d0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef61ca3640 1 -- 172.21.15.150:0/213999950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef5c071f60 msgr2=0x7fef5c0ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef61ca3640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef5c071f60 0x7fef5c0ac8d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00e7d0 tx=0x7fef4c00ebd0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef61ca3640 1 -- 172.21.15.150:0/213999950 shutdown_connections 2024-09-17T04:51:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef61ca3640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fef3c073230 0x7fef3c075740 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef61ca3640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef5c0c8e20 0x7fef5c143b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef61ca3640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef5c0a9ab0 0x7fef5c148c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef61ca3640 1 --2- 172.21.15.150:0/213999950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef5c071f60 0x7fef5c0ac8d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef61ca3640 1 -- 172.21.15.150:0/213999950 >> 172.21.15.150:0/213999950 conn(0x7fef5c0c8250 msgr2=0x7fef5c151530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef61ca3640 1 -- 172.21.15.150:0/213999950 shutdown_connections 2024-09-17T04:51:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:27.822+0000 7fef61ca3640 1 -- 172.21.15.150:0/213999950 wait complete. 2024-09-17T04:51:27.836 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether the compiler supports GNU C... (cached) yes 2024-09-17T04:51:27.837 INFO:tasks.workunit.client.1.smithi176.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-17T04:51:27.837 INFO:tasks.workunit.client.1.smithi176.stdout:checking for gcc option to enable C11 features... (cached) none needed 2024-09-17T04:51:27.838 INFO:tasks.workunit.client.1.smithi176.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-17T04:51:27.847 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T04:51:26:821262+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4358": { "addr": "172.21.15.176:6837/1045115232", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 1045115232, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 1045115232, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4358, "incarnation": 16, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:51:25.844339+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4358 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4401": { "addr": "172.21.15.150:6834/480168496", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 480168496, "type": "v2" }, { "addr": "172.21.15.150:6834", "nonce": 480168496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4401, "incarnation": 10, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:49:44.467846+0000", "qdb_cluster": [ 4401 ], "qdb_leader": 4401, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4401 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:51:27.906 INFO:tasks.workunit.client.1.smithi176.stdout:checking for main in -lm... yes 2024-09-17T04:51:27.983 INFO:tasks.workunit.client.1.smithi176.stdout:checking for main in -lpthread... yes 2024-09-17T04:51:27.989 INFO:tasks.workunit.client.1.smithi176.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-17T04:51:27.990 INFO:tasks.workunit.client.1.smithi176.stdout:checking for egrep... /usr/bin/grep -E 2024-09-17T04:51:28.048 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-17T04:51:28.113 INFO:tasks.workunit.client.1.smithi176.stdout:checking for pthread.h... yes 2024-09-17T04:51:28.184 INFO:tasks.workunit.client.1.smithi176.stdout:checking for fcntl.h... yes 2024-09-17T04:51:28.245 INFO:tasks.workunit.client.1.smithi176.stdout:checking for limits.h... yes 2024-09-17T04:51:28.248 INFO:tasks.workunit.client.1.smithi176.stdout:checking for stdint.h... (cached) yes 2024-09-17T04:51:28.249 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/time.h... (cached) yes 2024-09-17T04:51:28.250 INFO:tasks.workunit.client.1.smithi176.stdout:checking for unistd.h... (cached) yes 2024-09-17T04:51:28.312 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:28.313 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/vfs.h... yes 2024-09-17T04:51:28.396 INFO:tasks.workunit.client.1.smithi176.stdout:checking for sys/limits.h... no 2024-09-17T04:51:28.399 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:51:28.402 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:51:28.437 INFO:tasks.workunit.client.1.smithi176.stdout:checking for an ANSI C-conforming const... yes 2024-09-17T04:51:28.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.490+0000 7f56ecee3640 1 -- 172.21.15.150:0/3763792611 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e80c35b0 msgr2=0x7f56e80c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:28.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.490+0000 7f56ecee3640 1 --2- 172.21.15.150:0/3763792611 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e80c35b0 0x7f56e80c8a80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f56d0009840 tx=0x7f56d001c7f0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.490+0000 7f56ecee3640 1 -- 172.21.15.150:0/3763792611 shutdown_connections 2024-09-17T04:51:28.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.490+0000 7f56ecee3640 1 --2- 172.21.15.150:0/3763792611 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e80c35b0 0x7f56e80c8a80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.490+0000 7f56ecee3640 1 --2- 172.21.15.150:0/3763792611 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56e81568a0 0x7f56e80c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.490+0000 7f56ecee3640 1 -- 172.21.15.150:0/3763792611 >> 172.21.15.150:0/3763792611 conn(0x7f56e8074b80 msgr2=0x7f56e806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:28.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.490+0000 7f56ecee3640 1 -- 172.21.15.150:0/3763792611 shutdown_connections 2024-09-17T04:51:28.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.490+0000 7f56ecee3640 1 -- 172.21.15.150:0/3763792611 wait complete. 2024-09-17T04:51:28.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56ecee3640 1 Processor -- start 2024-09-17T04:51:28.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56ecee3640 1 -- start start 2024-09-17T04:51:28.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56ecee3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e80c35b0 0x7f56e8145450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:28.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56ecee3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56e80c8fc0 0x7f56e8152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:28.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56ecee3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e81568a0 0x7f56e80cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:28.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56ecee3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f56e80c6b70 con 0x7f56e80c35b0 2024-09-17T04:51:28.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56ecee3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f56e80c69f0 con 0x7f56e81568a0 2024-09-17T04:51:28.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56ecee3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f56e80c27b0 con 0x7f56e80c8fc0 2024-09-17T04:51:28.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e80c35b0 0x7f56e8145450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:28.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e80c35b0 0x7f56e8145450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50662/0 (socket says 172.21.15.150:50662) 2024-09-17T04:51:28.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e5d9b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56e80c8fc0 0x7f56e8152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:28.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e659c640 1 -- 172.21.15.150:0/377657623 learned_addr learned my addr 172.21.15.150:0/377657623 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:28.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e6d9d640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e81568a0 0x7f56e80cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:28.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e5d9b640 1 -- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e81568a0 msgr2=0x7f56e80cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:28.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e5d9b640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e81568a0 0x7f56e80cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e5d9b640 1 -- 172.21.15.150:0/377657623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e80c35b0 msgr2=0x7f56e8145450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:28.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e5d9b640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e80c35b0 0x7f56e8145450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e5d9b640 1 -- 172.21.15.150:0/377657623 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e80cb190 con 0x7f56e80c8fc0 2024-09-17T04:51:28.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e6d9d640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e81568a0 0x7f56e80cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:51:28.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e659c640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e80c35b0 0x7f56e8145450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:51:28.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56e5d9b640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56e80c8fc0 0x7f56e8152140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f56e806ffd0 tx=0x7f56d00a57c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:28.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56deffd640 1 -- 172.21.15.150:0/377657623 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d00b2040 con 0x7f56e80c8fc0 2024-09-17T04:51:28.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56deffd640 1 -- 172.21.15.150:0/377657623 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56d0005020 con 0x7f56e80c8fc0 2024-09-17T04:51:28.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56ecee3640 1 -- 172.21.15.150:0/377657623 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56e81f3340 con 0x7f56e80c8fc0 2024-09-17T04:51:28.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56deffd640 1 -- 172.21.15.150:0/377657623 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d00044f0 con 0x7f56e80c8fc0 2024-09-17T04:51:28.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56ecee3640 1 -- 172.21.15.150:0/377657623 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f56e81f3910 con 0x7f56e80c8fc0 2024-09-17T04:51:28.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.494+0000 7f56ecee3640 1 -- 172.21.15.150:0/377657623 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56e80c8a80 con 0x7f56e80c8fc0 2024-09-17T04:51:28.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.498+0000 7f56deffd640 1 -- 172.21.15.150:0/377657623 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f56d0005470 con 0x7f56e80c8fc0 2024-09-17T04:51:28.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.498+0000 7f56deffd640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f56c8073200 0x7f56c8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:28.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.498+0000 7f56deffd640 1 -- 172.21.15.150:0/377657623 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f56d012dd90 con 0x7f56e80c8fc0 2024-09-17T04:51:28.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.498+0000 7f56e659c640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f56c8073200 0x7f56c8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:28.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.498+0000 7f56e659c640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f56c8073200 0x7f56c8075710 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f56e00117b0 tx=0x7f56e0009500 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:28.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.502+0000 7f56deffd640 1 -- 172.21.15.150:0/377657623 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f56d0016410 con 0x7f56e80c8fc0 2024-09-17T04:51:28.513 INFO:tasks.workunit.client.1.smithi176.stdout:checking for system... yes 2024-09-17T04:51:28.599 INFO:tasks.workunit.client.1.smithi176.stdout:checking for gettimeofday... yes 2024-09-17T04:51:28.678 INFO:tasks.workunit.client.1.smithi176.stdout:checking for mkdir... yes 2024-09-17T04:51:28.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.722+0000 7f56ecee3640 1 -- 172.21.15.150:0/377657623 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56e8069000 con 0x7f56e80c8fc0 2024-09-17T04:51:28.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56deffd640 1 -- 172.21.15.150:0/377657623 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5701 (secure 0 0 0) 0x7f56d00f8350 con 0x7f56e80c8fc0 2024-09-17T04:51:28.726 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:28.726 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":18,"btime":"2024-09-17T04:51:27:823621+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:26.829579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 18 2024-09-17T04:51:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56ecee3640 1 -- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f56c8073200 msgr2=0x7f56c8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56ecee3640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f56c8073200 0x7f56c8075710 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f56e00117b0 tx=0x7f56e0009500 comp rx=0 tx=0).stop 2024-09-17T04:51:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56ecee3640 1 -- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56e80c8fc0 msgr2=0x7f56e8152140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56ecee3640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56e80c8fc0 0x7f56e8152140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f56e806ffd0 tx=0x7f56d00a57c0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56ecee3640 1 -- 172.21.15.150:0/377657623 shutdown_connections 2024-09-17T04:51:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56ecee3640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f56c8073200 0x7f56c8075710 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56ecee3640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56e81568a0 0x7f56e80cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56ecee3640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56e80c8fc0 0x7f56e8152140 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56ecee3640 1 --2- 172.21.15.150:0/377657623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56e80c35b0 0x7f56e8145450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56ecee3640 1 -- 172.21.15.150:0/377657623 >> 172.21.15.150:0/377657623 conn(0x7f56e8074b80 msgr2=0x7f56e8154dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:28.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56ecee3640 1 -- 172.21.15.150:0/377657623 shutdown_connections 2024-09-17T04:51:28.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.726+0000 7f56ecee3640 1 -- 172.21.15.150:0/377657623 wait complete. 2024-09-17T04:51:28.749 INFO:tasks.workunit.client.1.smithi176.stdout:checking for strerror... yes 2024-09-17T04:51:28.751 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-17T04:51:28.751 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:51:28.751 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:51:28.768 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:51:28.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:28.864 INFO:tasks.workunit.client.0.smithi150.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-09-17T04:51:28.865 INFO:tasks.workunit.client.0.smithi150.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-09-17T04:51:28.865 INFO:tasks.workunit.client.0.smithi150.stderr:+ trap -- retry EXIT 2024-09-17T04:51:28.865 INFO:tasks.workunit.client.0.smithi150.stderr:+ rm -rf ffsb 2024-09-17T04:51:28.866 INFO:tasks.workunit.client.0.smithi150.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-09-17T04:51:28.871 INFO:tasks.workunit.client.0.smithi150.stderr:Cloning into 'ffsb'... 2024-09-17T04:51:28.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.978+0000 7f586211e640 1 -- 172.21.15.150:0/2939234338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585c157c00 msgr2=0x7f585c158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:28.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.978+0000 7f586211e640 1 --2- 172.21.15.150:0/2939234338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585c157c00 0x7f585c158010 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f585c0b00f0 tx=0x7f585401cd50 comp rx=0 tx=0).stop 2024-09-17T04:51:28.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.978+0000 7f586211e640 1 -- 172.21.15.150:0/2939234338 shutdown_connections 2024-09-17T04:51:28.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.978+0000 7f586211e640 1 --2- 172.21.15.150:0/2939234338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585c157c00 0x7f585c158010 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.978+0000 7f586211e640 1 --2- 172.21.15.150:0/2939234338 >> v2:172.21.15.176:3300/0 conn(0x7f585c0c5680 0x7f585c0c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.978+0000 7f586211e640 1 -- 172.21.15.150:0/2939234338 >> 172.21.15.150:0/2939234338 conn(0x7f585c0ca460 msgr2=0x7f585c0c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:28.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.978+0000 7f586211e640 1 -- 172.21.15.150:0/2939234338 shutdown_connections 2024-09-17T04:51:28.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.978+0000 7f586211e640 1 -- 172.21.15.150:0/2939234338 wait complete. 2024-09-17T04:51:28.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.978+0000 7f586211e640 1 Processor -- start 2024-09-17T04:51:28.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f586211e640 1 -- start start 2024-09-17T04:51:28.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f586211e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f585c071670 0x7f585c1f2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:28.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f586211e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585c0c5680 0x7f585c06e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:28.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f586211e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f585c157c00 0x7f585c15d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:28.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f586211e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f585c0c65c0 con 0x7f585c0c5680 2024-09-17T04:51:28.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f586211e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f585c0c62d0 con 0x7f585c071670 2024-09-17T04:51:28.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f586211e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f585c0c6740 con 0x7f585c157c00 2024-09-17T04:51:28.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f585affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585c0c5680 0x7f585c06e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:28.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f585affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585c0c5680 0x7f585c06e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50682/0 (socket says 172.21.15.150:50682) 2024-09-17T04:51:28.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f585affd640 1 -- 172.21.15.150:0/1464493528 learned_addr learned my addr 172.21.15.150:0/1464493528 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:28.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f585b7fe640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f585c071670 0x7f585c1f2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:28.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f585bfff640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f585c157c00 0x7f585c15d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:28.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f585affd640 1 -- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f585c157c00 msgr2=0x7f585c15d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:28.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f585affd640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f585c157c00 0x7f585c15d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f585affd640 1 -- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f585c071670 msgr2=0x7f585c1f2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:28.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f585affd640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f585c071670 0x7f585c1f2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:28.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f585affd640 1 -- 172.21.15.150:0/1464493528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f585c13f6c0 con 0x7f585c0c5680 2024-09-17T04:51:28.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f585b7fe640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f585c071670 0x7f585c1f2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:51:28.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f585affd640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585c0c5680 0x7f585c06e220 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f58500058d0 tx=0x7f585000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:28.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f5843fff640 1 -- 172.21.15.150:0/1464493528 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850017070 con 0x7f585c0c5680 2024-09-17T04:51:28.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f5843fff640 1 -- 172.21.15.150:0/1464493528 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5850009cc0 con 0x7f585c0c5680 2024-09-17T04:51:28.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f586211e640 1 -- 172.21.15.150:0/1464493528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f585c06f890 con 0x7f585c0c5680 2024-09-17T04:51:28.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f5843fff640 1 -- 172.21.15.150:0/1464493528 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58500106c0 con 0x7f585c0c5680 2024-09-17T04:51:28.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f586211e640 1 -- 172.21.15.150:0/1464493528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f585c1f3170 con 0x7f585c0c5680 2024-09-17T04:51:28.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.982+0000 7f586211e640 1 -- 172.21.15.150:0/1464493528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f585c06f680 con 0x7f585c0c5680 2024-09-17T04:51:28.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.986+0000 7f5843fff640 1 -- 172.21.15.150:0/1464493528 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f585001a070 con 0x7f585c0c5680 2024-09-17T04:51:28.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.986+0000 7f5843fff640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5848073230 0x7f5848075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:28.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.986+0000 7f5843fff640 1 -- 172.21.15.150:0/1464493528 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f5850094730 con 0x7f585c0c5680 2024-09-17T04:51:28.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.986+0000 7f585b7fe640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5848073230 0x7f5848075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:28.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.986+0000 7f585b7fe640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5848073230 0x7f5848075740 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f584c0098b0 tx=0x7f584c008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:28.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:28.990+0000 7f5843fff640 1 -- 172.21.15.150:0/1464493528 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f585005f350 con 0x7f585c0c5680 2024-09-17T04:51:29.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.210+0000 7f586211e640 1 -- 172.21.15.150:0/1464493528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f585c072280 con 0x7f585c0c5680 2024-09-17T04:51:29.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f5843fff640 1 -- 172.21.15.150:0/1464493528 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5708 (secure 0 0 0) 0x7f585005ec50 con 0x7f585c0c5680 2024-09-17T04:51:29.213 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:29.213 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":19,"btime":"2024-09-17T04:51:28:833986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:29.213 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 19 2024-09-17T04:51:29.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f586211e640 1 -- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5848073230 msgr2=0x7f5848075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:29.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f586211e640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5848073230 0x7f5848075740 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f584c0098b0 tx=0x7f584c008040 comp rx=0 tx=0).stop 2024-09-17T04:51:29.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f586211e640 1 -- 172.21.15.150:0/1464493528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585c0c5680 msgr2=0x7f585c06e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:29.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f586211e640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585c0c5680 0x7f585c06e220 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f58500058d0 tx=0x7f585000dde0 comp rx=0 tx=0).stop 2024-09-17T04:51:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f586211e640 1 -- 172.21.15.150:0/1464493528 shutdown_connections 2024-09-17T04:51:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f586211e640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5848073230 0x7f5848075740 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f586211e640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f585c157c00 0x7f585c15d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f586211e640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f585c0c5680 0x7f585c06e220 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f586211e640 1 --2- 172.21.15.150:0/1464493528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f585c071670 0x7f585c1f2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f586211e640 1 -- 172.21.15.150:0/1464493528 >> 172.21.15.150:0/1464493528 conn(0x7f585c0ca460 msgr2=0x7f585c15e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f586211e640 1 -- 172.21.15.150:0/1464493528 shutdown_connections 2024-09-17T04:51:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:29.214+0000 7f586211e640 1 -- 172.21.15.150:0/1464493528 wait complete. 2024-09-17T04:51:29.237 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:51:29.849 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:30.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.014+0000 7eff6d5d7640 1 -- 172.21.15.150:0/3103435231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff54004e70 msgr2=0x7eff540052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:30.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.014+0000 7eff6d5d7640 1 --2- 172.21.15.150:0/3103435231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff54004e70 0x7eff540052c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7eff6000bb10 tx=0x7eff6001cdc0 comp rx=0 tx=0).stop 2024-09-17T04:51:30.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.014+0000 7eff677fe640 1 -- 172.21.15.150:0/3103435231 reap_dead start 2024-09-17T04:51:30.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.014+0000 7eff6d5d7640 1 -- 172.21.15.150:0/3103435231 shutdown_connections 2024-09-17T04:51:30.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.014+0000 7eff6d5d7640 1 -- 172.21.15.150:0/3103435231 >> 172.21.15.150:0/3103435231 conn(0x7eff680c5ae0 msgr2=0x7eff68073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:30.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.014+0000 7eff6d5d7640 1 -- 172.21.15.150:0/3103435231 shutdown_connections 2024-09-17T04:51:30.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.014+0000 7eff6d5d7640 1 -- 172.21.15.150:0/3103435231 wait complete. 2024-09-17T04:51:30.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff6d5d7640 1 Processor -- start 2024-09-17T04:51:30.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff6d5d7640 1 -- start start 2024-09-17T04:51:30.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff6d5d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff681eb230 0x7eff6814c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:30.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff6d5d7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff68147d50 0x7eff68148160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:30.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff6d5d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff681edda0 0x7eff681ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:30.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff6d5d7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7eff6806e7a0 con 0x7eff681eb230 2024-09-17T04:51:30.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff6d5d7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7eff68157620 con 0x7eff68147d50 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff6d5d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7eff6806e920 con 0x7eff681edda0 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff66ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff681eb230 0x7eff6814c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff66ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff681eb230 0x7eff6814c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42524/0 (socket says 172.21.15.150:42524) 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff66ffd640 1 -- 172.21.15.150:0/2929933082 learned_addr learned my addr 172.21.15.150:0/2929933082 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff677fe640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff681edda0 0x7eff681ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff667fc640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff68147d50 0x7eff68148160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff66ffd640 1 -- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff681edda0 msgr2=0x7eff681ee1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff66ffd640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff681edda0 0x7eff681ee1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff66ffd640 1 -- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff68147d50 msgr2=0x7eff68148160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff66ffd640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff68147d50 0x7eff68148160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff66ffd640 1 -- 172.21.15.150:0/2929933082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff68154400 con 0x7eff681eb230 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff66ffd640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff681eb230 0x7eff6814c490 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7eff58010dd0 tx=0x7eff5800e650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff5f7fe640 1 -- 172.21.15.150:0/2929933082 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58013480 con 0x7eff681eb230 2024-09-17T04:51:30.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff5f7fe640 1 -- 172.21.15.150:0/2929933082 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff58018020 con 0x7eff681eb230 2024-09-17T04:51:30.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff6d5d7640 1 -- 172.21.15.150:0/2929933082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff681eb640 con 0x7eff681eb230 2024-09-17T04:51:30.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff5f7fe640 1 -- 172.21.15.150:0/2929933082 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58013970 con 0x7eff681eb230 2024-09-17T04:51:30.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.018+0000 7eff6d5d7640 1 -- 172.21.15.150:0/2929933082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff68153480 con 0x7eff681eb230 2024-09-17T04:51:30.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.022+0000 7eff6d5d7640 1 -- 172.21.15.150:0/2929933082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff68070610 con 0x7eff681eb230 2024-09-17T04:51:30.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.022+0000 7eff5f7fe640 1 -- 172.21.15.150:0/2929933082 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7eff58013b10 con 0x7eff681eb230 2024-09-17T04:51:30.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.022+0000 7eff5f7fe640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff48073200 0x7eff48075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:30.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.022+0000 7eff5f7fe640 1 -- 172.21.15.150:0/2929933082 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7eff58098f30 con 0x7eff681eb230 2024-09-17T04:51:30.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.022+0000 7eff667fc640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff48073200 0x7eff48075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:30.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.022+0000 7eff667fc640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff48073200 0x7eff48075710 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7eff50001100 tx=0x7eff50001070 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:30.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.026+0000 7eff5f7fe640 1 -- 172.21.15.150:0/2929933082 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7eff58022020 con 0x7eff681eb230 2024-09-17T04:51:30.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.258+0000 7eff6d5d7640 1 -- 172.21.15.150:0/2929933082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff680c2e00 con 0x7eff681eb230 2024-09-17T04:51:30.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.258+0000 7eff5f7fe640 1 -- 172.21.15.150:0/2929933082 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5708 (secure 0 0 0) 0x7eff58063e20 con 0x7eff681eb230 2024-09-17T04:51:30.260 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:30.260 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":19,"btime":"2024-09-17T04:51:28:833986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:30.260 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 19 2024-09-17T04:51:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.262+0000 7eff6d5d7640 1 -- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff48073200 msgr2=0x7eff48075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.262+0000 7eff6d5d7640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff48073200 0x7eff48075710 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7eff50001100 tx=0x7eff50001070 comp rx=0 tx=0).stop 2024-09-17T04:51:30.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.262+0000 7eff6d5d7640 1 -- 172.21.15.150:0/2929933082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff681eb230 msgr2=0x7eff6814c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:30.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.262+0000 7eff6d5d7640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff681eb230 0x7eff6814c490 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7eff58010dd0 tx=0x7eff5800e650 comp rx=0 tx=0).stop 2024-09-17T04:51:30.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.262+0000 7eff6d5d7640 1 -- 172.21.15.150:0/2929933082 shutdown_connections 2024-09-17T04:51:30.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.262+0000 7eff6d5d7640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff48073200 0x7eff48075710 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:30.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.262+0000 7eff6d5d7640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff68147d50 0x7eff68148160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:30.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.262+0000 7eff6d5d7640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff681edda0 0x7eff681ee1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:30.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.262+0000 7eff6d5d7640 1 --2- 172.21.15.150:0/2929933082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff681eb230 0x7eff6814c490 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:30.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.262+0000 7eff6d5d7640 1 -- 172.21.15.150:0/2929933082 >> 172.21.15.150:0/2929933082 conn(0x7eff680c5ae0 msgr2=0x7eff68156d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:30.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.262+0000 7eff6d5d7640 1 -- 172.21.15.150:0/2929933082 shutdown_connections 2024-09-17T04:51:30.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:30.262+0000 7eff6d5d7640 1 -- 172.21.15.150:0/2929933082 wait complete. 2024-09-17T04:51:30.282 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=4404 2024-09-17T04:51:30.282 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before reviving mds.c 2024-09-17T04:51:31.238 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:31.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.406+0000 7fb205655640 1 -- 172.21.15.150:0/2523103566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb200074570 msgr2=0x7fb2000aaa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:31.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.406+0000 7fb205655640 1 --2- 172.21.15.150:0/2523103566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb200074570 0x7fb2000aaa40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb2000c3a90 tx=0x7fb1e801ea40 comp rx=0 tx=0).stop 2024-09-17T04:51:31.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.406+0000 7fb205655640 1 -- 172.21.15.150:0/2523103566 shutdown_connections 2024-09-17T04:51:31.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.406+0000 7fb205655640 1 --2- 172.21.15.150:0/2523103566 >> v2:172.21.15.176:3300/0 conn(0x7fb200074980 0x7fb2000aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:31.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.406+0000 7fb205655640 1 --2- 172.21.15.150:0/2523103566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb200074570 0x7fb2000aaa40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:31.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.406+0000 7fb205655640 1 -- 172.21.15.150:0/2523103566 >> 172.21.15.150:0/2523103566 conn(0x7fb2000c5ae0 msgr2=0x7fb200073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:31.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.406+0000 7fb205655640 1 -- 172.21.15.150:0/2523103566 shutdown_connections 2024-09-17T04:51:31.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.406+0000 7fb205655640 1 -- 172.21.15.150:0/2523103566 wait complete. 2024-09-17T04:51:31.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb205655640 1 Processor -- start 2024-09-17T04:51:31.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb205655640 1 -- start start 2024-09-17T04:51:31.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb205655640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb20006fca0 0x7fb2000abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:31.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb205655640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb200074570 0x7fb2001451e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:31.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb205655640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb200074980 0x7fb2001f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:31.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb205655640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb2000c4510 con 0x7fb20006fca0 2024-09-17T04:51:31.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb205655640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb2000c4390 con 0x7fb200074980 2024-09-17T04:51:31.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb205655640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb2000c4690 con 0x7fb200074570 2024-09-17T04:51:31.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1feffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb20006fca0 0x7fb2000abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:31.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1feffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb20006fca0 0x7fb2000abfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42548/0 (socket says 172.21.15.150:42548) 2024-09-17T04:51:31.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1feffd640 1 -- 172.21.15.150:0/3499603846 learned_addr learned my addr 172.21.15.150:0/3499603846 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:31.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1ff7fe640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb200074980 0x7fb2001f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:31.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1f67fc640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb200074570 0x7fb2001451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:31.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1feffd640 1 -- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb200074570 msgr2=0x7fb2001451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:31.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1feffd640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb200074570 0x7fb2001451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:31.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1feffd640 1 -- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb200074980 msgr2=0x7fb2001f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:31.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1feffd640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb200074980 0x7fb2001f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:31.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1feffd640 1 -- 172.21.15.150:0/3499603846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb20014fd00 con 0x7fb20006fca0 2024-09-17T04:51:31.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1feffd640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb20006fca0 0x7fb2000abfd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb1f000b470 tx=0x7fb1f000b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:31.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1f7fff640 1 -- 172.21.15.150:0/3499603846 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0015030 con 0x7fb20006fca0 2024-09-17T04:51:31.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1f7fff640 1 -- 172.21.15.150:0/3499603846 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1f0004020 con 0x7fb20006fca0 2024-09-17T04:51:31.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb205655640 1 -- 172.21.15.150:0/3499603846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb20014d980 con 0x7fb20006fca0 2024-09-17T04:51:31.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb1f7fff640 1 -- 172.21.15.150:0/3499603846 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0004470 con 0x7fb20006fca0 2024-09-17T04:51:31.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.410+0000 7fb205655640 1 -- 172.21.15.150:0/3499603846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb20014dc20 con 0x7fb20006fca0 2024-09-17T04:51:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.414+0000 7fb205655640 1 -- 172.21.15.150:0/3499603846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2000c4c30 con 0x7fb20006fca0 2024-09-17T04:51:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.414+0000 7fb1f7fff640 1 -- 172.21.15.150:0/3499603846 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb1f0018070 con 0x7fb20006fca0 2024-09-17T04:51:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.414+0000 7fb1f7fff640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1e0073260 0x7fb1e0075770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.414+0000 7fb1f7fff640 1 -- 172.21.15.150:0/3499603846 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7fb1f0094060 con 0x7fb20006fca0 2024-09-17T04:51:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.414+0000 7fb1f67fc640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1e0073260 0x7fb1e0075770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.414+0000 7fb1f67fc640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1e0073260 0x7fb1e0075770 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80b1000 tx=0x7fb1e801d280 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:31.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.418+0000 7fb1f7fff640 1 -- 172.21.15.150:0/3499603846 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb1f005ed90 con 0x7fb20006fca0 2024-09-17T04:51:31.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.638+0000 7fb205655640 1 -- 172.21.15.150:0/3499603846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2000c1dd0 con 0x7fb20006fca0 2024-09-17T04:51:31.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb1f7fff640 1 -- 172.21.15.150:0/3499603846 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5708 (secure 0 0 0) 0x7fb1f005e690 con 0x7fb20006fca0 2024-09-17T04:51:31.641 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:31.641 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":19,"btime":"2024-09-17T04:51:28:833986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:31.641 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 19 2024-09-17T04:51:31.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb205655640 1 -- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1e0073260 msgr2=0x7fb1e0075770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:31.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb205655640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1e0073260 0x7fb1e0075770 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80b1000 tx=0x7fb1e801d280 comp rx=0 tx=0).stop 2024-09-17T04:51:31.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb205655640 1 -- 172.21.15.150:0/3499603846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb20006fca0 msgr2=0x7fb2000abfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:31.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb205655640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb20006fca0 0x7fb2000abfd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb1f000b470 tx=0x7fb1f000b930 comp rx=0 tx=0).stop 2024-09-17T04:51:31.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb205655640 1 -- 172.21.15.150:0/3499603846 shutdown_connections 2024-09-17T04:51:31.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb205655640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1e0073260 0x7fb1e0075770 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:31.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb205655640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb200074980 0x7fb2001f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:31.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb205655640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb200074570 0x7fb2001451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:31.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb205655640 1 --2- 172.21.15.150:0/3499603846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb20006fca0 0x7fb2000abfd0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:31.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb205655640 1 -- 172.21.15.150:0/3499603846 >> 172.21.15.150:0/3499603846 conn(0x7fb2000c5ae0 msgr2=0x7fb20015cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:31.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb205655640 1 -- 172.21.15.150:0/3499603846 shutdown_connections 2024-09-17T04:51:31.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:31.642+0000 7fb205655640 1 -- 172.21.15.150:0/3499603846 wait complete. 2024-09-17T04:51:31.664 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:51:32.173 INFO:tasks.ceph.mds.d.smithi176.stderr:2024-09-17T04:51:32.169+0000 7f93f3047640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:51:32.266 INFO:tasks.workunit.client.0.smithi150.stderr:+ trap - EXIT 2024-09-17T04:51:32.266 INFO:tasks.workunit.client.0.smithi150.stderr:+ cd ffsb 2024-09-17T04:51:32.267 INFO:tasks.workunit.client.0.smithi150.stderr:+ ./configure 2024-09-17T04:51:32.464 INFO:tasks.workunit.client.0.smithi150.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-17T04:51:33.483 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether build environment is sane... yes 2024-09-17T04:51:33.505 INFO:tasks.workunit.client.0.smithi150.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-09-17T04:51:33.506 INFO:tasks.workunit.client.0.smithi150.stdout:checking for gawk... gawk 2024-09-17T04:51:33.548 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether make sets $(MAKE)... yes 2024-09-17T04:51:33.601 INFO:tasks.workunit.client.0.smithi150.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-17T04:51:33.601 INFO:tasks.workunit.client.0.smithi150.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-17T04:51:33.635 INFO:tasks.workunit.client.0.smithi150.stdout:checking for style of include used by make... GNU 2024-09-17T04:51:33.644 INFO:tasks.workunit.client.0.smithi150.stdout:checking for gcc... gcc 2024-09-17T04:51:33.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:33.758 INFO:tasks.workunit.client.0.smithi150.stdout:checking for C compiler default output file name... a.out 2024-09-17T04:51:33.761 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether the C compiler works... yes 2024-09-17T04:51:33.767 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether we are cross compiling... no 2024-09-17T04:51:33.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.810+0000 7fe1a0be8640 1 -- 172.21.15.150:0/1699560837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe188004e70 msgr2=0x7fe1880052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:33.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.810+0000 7fe1a0be8640 1 --2- 172.21.15.150:0/1699560837 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe188004e70 0x7fe1880052a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe194009e90 tx=0x7fe19401d5c0 comp rx=0 tx=0).stop 2024-09-17T04:51:33.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.810+0000 7fe19ad9d640 1 -- 172.21.15.150:0/1699560837 reap_dead start 2024-09-17T04:51:33.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.810+0000 7fe1a0be8640 1 -- 172.21.15.150:0/1699560837 shutdown_connections 2024-09-17T04:51:33.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.810+0000 7fe1a0be8640 1 -- 172.21.15.150:0/1699560837 >> 172.21.15.150:0/1699560837 conn(0x7fe19c0c9de0 msgr2=0x7fe19c0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:33.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.810+0000 7fe1a0be8640 1 -- 172.21.15.150:0/1699560837 shutdown_connections 2024-09-17T04:51:33.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.810+0000 7fe1a0be8640 1 -- 172.21.15.150:0/1699560837 wait complete. 2024-09-17T04:51:33.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.810+0000 7fe1a0be8640 1 Processor -- start 2024-09-17T04:51:33.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.810+0000 7fe1a0be8640 1 -- start start 2024-09-17T04:51:33.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe1a0be8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe188004e70 0x7fe19c148500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:33.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe1a0be8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe19c1f30a0 0x7fe19c1465f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:33.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe1a0be8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe19c146e20 0x7fe19c0c3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:33.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe1a0be8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c0c5a40 con 0x7fe188004e70 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe1a0be8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c160040 con 0x7fe19c146e20 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe1a0be8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe19c1601c0 con 0x7fe19c1f30a0 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe19a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe188004e70 0x7fe19c148500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe19a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe188004e70 0x7fe19c148500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42558/0 (socket says 172.21.15.150:42558) 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe19a59c640 1 -- 172.21.15.150:0/3360016026 learned_addr learned my addr 172.21.15.150:0/3360016026 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe19ad9d640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe19c146e20 0x7fe19c0c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe193fff640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe19c1f30a0 0x7fe19c1465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe19a59c640 1 -- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe19c1f30a0 msgr2=0x7fe19c1465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe19a59c640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe19c1f30a0 0x7fe19c1465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe19a59c640 1 -- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe19c146e20 msgr2=0x7fe19c0c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe19a59c640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe19c146e20 0x7fe19c0c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe19a59c640 1 -- 172.21.15.150:0/3360016026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19c0aa370 con 0x7fe188004e70 2024-09-17T04:51:33.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe19a59c640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe188004e70 0x7fe19c148500 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00c430 tx=0x7fe18c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:33.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe192ffd640 1 -- 172.21.15.150:0/3360016026 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c017070 con 0x7fe188004e70 2024-09-17T04:51:33.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe192ffd640 1 -- 172.21.15.150:0/3360016026 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe18c0053e0 con 0x7fe188004e70 2024-09-17T04:51:33.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe1a0be8640 1 -- 172.21.15.150:0/3360016026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19c157110 con 0x7fe188004e70 2024-09-17T04:51:33.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe192ffd640 1 -- 172.21.15.150:0/3360016026 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c0038b0 con 0x7fe188004e70 2024-09-17T04:51:33.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe1a0be8640 1 -- 172.21.15.150:0/3360016026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe19c149180 con 0x7fe188004e70 2024-09-17T04:51:33.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe1a0be8640 1 -- 172.21.15.150:0/3360016026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe19c148140 con 0x7fe188004e70 2024-09-17T04:51:33.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.814+0000 7fe192ffd640 1 -- 172.21.15.150:0/3360016026 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe18c01a070 con 0x7fe188004e70 2024-09-17T04:51:33.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.818+0000 7fe192ffd640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe17c073200 0x7fe17c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:33.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.818+0000 7fe192ffd640 1 -- 172.21.15.150:0/3360016026 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7fe18c093a70 con 0x7fe188004e70 2024-09-17T04:51:33.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.818+0000 7fe193fff640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe17c073200 0x7fe17c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:33.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.818+0000 7fe193fff640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe17c073200 0x7fe17c075710 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe1840018d0 tx=0x7fe184001090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:33.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:33.822+0000 7fe192ffd640 1 -- 172.21.15.150:0/3360016026 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe18c05e710 con 0x7fe188004e70 2024-09-17T04:51:33.836 INFO:tasks.workunit.client.0.smithi150.stdout:checking for suffix of executables... 2024-09-17T04:51:33.898 INFO:tasks.workunit.client.0.smithi150.stdout:checking for suffix of object files... o 2024-09-17T04:51:33.953 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether we are using the GNU C compiler... yes 2024-09-17T04:51:34.020 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether gcc accepts -g... yes 2024-09-17T04:51:34.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.038+0000 7fe1a0be8640 1 -- 172.21.15.150:0/3360016026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe19c151e10 con 0x7fe188004e70 2024-09-17T04:51:34.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.038+0000 7fe192ffd640 1 -- 172.21.15.150:0/3360016026 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5708 (secure 0 0 0) 0x7fe18c05e010 con 0x7fe188004e70 2024-09-17T04:51:34.039 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:34.040 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":19,"btime":"2024-09-17T04:51:28:833986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:34.040 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 19 2024-09-17T04:51:34.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.042+0000 7fe1a0be8640 1 -- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe17c073200 msgr2=0x7fe17c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:34.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.042+0000 7fe1a0be8640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe17c073200 0x7fe17c075710 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe1840018d0 tx=0x7fe184001090 comp rx=0 tx=0).stop 2024-09-17T04:51:34.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.042+0000 7fe1a0be8640 1 -- 172.21.15.150:0/3360016026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe188004e70 msgr2=0x7fe19c148500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:34.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.042+0000 7fe1a0be8640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe188004e70 0x7fe19c148500 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00c430 tx=0x7fe18c00c8f0 comp rx=0 tx=0).stop 2024-09-17T04:51:34.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.042+0000 7fe1a0be8640 1 -- 172.21.15.150:0/3360016026 shutdown_connections 2024-09-17T04:51:34.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.042+0000 7fe1a0be8640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe17c073200 0x7fe17c075710 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:34.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.042+0000 7fe1a0be8640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe19c146e20 0x7fe19c0c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:34.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.042+0000 7fe1a0be8640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe19c1f30a0 0x7fe19c1465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:34.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.042+0000 7fe1a0be8640 1 --2- 172.21.15.150:0/3360016026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe188004e70 0x7fe19c148500 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:34.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.042+0000 7fe1a0be8640 1 -- 172.21.15.150:0/3360016026 >> 172.21.15.150:0/3360016026 conn(0x7fe19c0c9de0 msgr2=0x7fe19c0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:34.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.042+0000 7fe1a0be8640 1 -- 172.21.15.150:0/3360016026 shutdown_connections 2024-09-17T04:51:34.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:34.042+0000 7fe1a0be8640 1 -- 172.21.15.150:0/3360016026 wait complete. 2024-09-17T04:51:34.060 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:51:34.132 INFO:tasks.workunit.client.0.smithi150.stdout:checking for gcc option to accept ISO C89... none needed 2024-09-17T04:51:34.453 INFO:tasks.workunit.client.0.smithi150.stdout:checking dependency style of gcc... gcc3 2024-09-17T04:51:34.596 INFO:tasks.workunit.client.0.smithi150.stdout:checking how to run the C preprocessor... gcc -E 2024-09-17T04:51:34.722 INFO:tasks.workunit.client.0.smithi150.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-17T04:51:34.725 INFO:tasks.workunit.client.0.smithi150.stdout:checking for egrep... /usr/bin/grep -E 2024-09-17T04:51:34.744 INFO:tasks.workunit.client.0.smithi150.stdout:checking for AIX... no 2024-09-17T04:51:34.804 INFO:tasks.workunit.client.0.smithi150.stdout:checking for gcc... (cached) gcc 2024-09-17T04:51:34.821 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-09-17T04:51:34.822 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-17T04:51:34.823 INFO:tasks.workunit.client.0.smithi150.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-09-17T04:51:34.824 INFO:tasks.workunit.client.0.smithi150.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-17T04:51:34.947 INFO:tasks.workunit.client.0.smithi150.stdout:checking for main in -lm... yes 2024-09-17T04:51:35.079 INFO:tasks.workunit.client.0.smithi150.stdout:checking for main in -lpthread... yes 2024-09-17T04:51:35.283 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-17T04:51:35.283 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:51:35.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:51:35.286 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:51:35.286 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:35.374 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T04:51:35.382 INFO:tasks.workunit.client.0.smithi150.stdout:checking for ANSI C header files... yes 2024-09-17T04:51:35.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.434+0000 7f01cdf15640 1 -- 172.21.15.150:0/2740222391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b0004d50 msgr2=0x7f01b0005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:35.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.434+0000 7f01cdf15640 1 --2- 172.21.15.150:0/2740222391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b0004d50 0x7f01b0005220 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01c80719d0 tx=0x7f01b801caf0 comp rx=0 tx=0).stop 2024-09-17T04:51:35.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.434+0000 7f01c7fff640 1 -- 172.21.15.150:0/2740222391 reap_dead start 2024-09-17T04:51:35.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.434+0000 7f01cdf15640 1 -- 172.21.15.150:0/2740222391 shutdown_connections 2024-09-17T04:51:35.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.434+0000 7f01cdf15640 1 -- 172.21.15.150:0/2740222391 >> 172.21.15.150:0/2740222391 conn(0x7f01c806e360 msgr2=0x7f01c806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:35.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.434+0000 7f01cdf15640 1 -- 172.21.15.150:0/2740222391 shutdown_connections 2024-09-17T04:51:35.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.434+0000 7f01cdf15640 1 -- 172.21.15.150:0/2740222391 wait complete. 2024-09-17T04:51:35.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.434+0000 7f01cdf15640 1 Processor -- start 2024-09-17T04:51:35.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.434+0000 7f01cdf15640 1 -- start start 2024-09-17T04:51:35.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01cdf15640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c80abbe0 0x7f01c81edb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:35.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01cdf15640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01c814aac0 0x7f01c814aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01cdf15640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01c80ad970 0x7f01c80add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01cdf15640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f01c8071120 con 0x7f01c80abbe0 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01cdf15640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f01c8070fa0 con 0x7f01c814aac0 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01cdf15640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f01c8072db0 con 0x7f01c80ad970 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01c77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c80abbe0 0x7f01c81edb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01c77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c80abbe0 0x7f01c81edb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42588/0 (socket says 172.21.15.150:42588) 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01c77fe640 1 -- 172.21.15.150:0/651803137 learned_addr learned my addr 172.21.15.150:0/651803137 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01c7fff640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01c80ad970 0x7f01c80add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01c6ffd640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01c814aac0 0x7f01c814aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01c77fe640 1 -- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01c80ad970 msgr2=0x7f01c80add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01c77fe640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01c80ad970 0x7f01c80add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01c77fe640 1 -- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01c814aac0 msgr2=0x7f01c814aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:35.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01c77fe640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01c814aac0 0x7f01c814aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:35.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01c77fe640 1 -- 172.21.15.150:0/651803137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c81dcf90 con 0x7f01c80abbe0 2024-09-17T04:51:35.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01c7fff640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01c80ad970 0x7f01c80add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:51:35.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01c77fe640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c80abbe0 0x7f01c81edb70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f01c000d820 tx=0x7f01c000dce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:35.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01affff640 1 -- 172.21.15.150:0/651803137 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c0018070 con 0x7f01c80abbe0 2024-09-17T04:51:35.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01affff640 1 -- 172.21.15.150:0/651803137 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01c00052e0 con 0x7f01c80abbe0 2024-09-17T04:51:35.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01cdf15640 1 -- 172.21.15.150:0/651803137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01c81dccf0 con 0x7f01c80abbe0 2024-09-17T04:51:35.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01affff640 1 -- 172.21.15.150:0/651803137 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c00036e0 con 0x7f01c80abbe0 2024-09-17T04:51:35.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01cdf15640 1 -- 172.21.15.150:0/651803137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01c80c7870 con 0x7f01c80abbe0 2024-09-17T04:51:35.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01cdf15640 1 -- 172.21.15.150:0/651803137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c80acfc0 con 0x7f01c80abbe0 2024-09-17T04:51:35.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.438+0000 7f01affff640 1 -- 172.21.15.150:0/651803137 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f01c001b070 con 0x7f01c80abbe0 2024-09-17T04:51:35.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.442+0000 7f01affff640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01b4073200 0x7f01b4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:35.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.442+0000 7f01affff640 1 -- 172.21.15.150:0/651803137 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f01c0093dd0 con 0x7f01c80abbe0 2024-09-17T04:51:35.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.442+0000 7f01c6ffd640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01b4073200 0x7f01b4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:35.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.442+0000 7f01c6ffd640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01b4073200 0x7f01b4075710 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f01c80c71e0 tx=0x7f01bc0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:35.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.446+0000 7f01affff640 1 -- 172.21.15.150:0/651803137 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f01c005e4b0 con 0x7f01c80abbe0 2024-09-17T04:51:35.479 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-17T04:51:35.566 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/types.h... yes 2024-09-17T04:51:35.659 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/stat.h... yes 2024-09-17T04:51:35.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 -- 172.21.15.150:0/651803137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01c80c3d10 con 0x7f01c80abbe0 2024-09-17T04:51:35.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01affff640 1 -- 172.21.15.150:0/651803137 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5708 (secure 0 0 0) 0x7f01c0095020 con 0x7f01c80abbe0 2024-09-17T04:51:35.669 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:35.669 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":19,"btime":"2024-09-17T04:51:28:833986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:35.670 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 19 2024-09-17T04:51:35.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 -- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01b4073200 msgr2=0x7f01b4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:35.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01b4073200 0x7f01b4075710 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f01c80c71e0 tx=0x7f01bc0091e0 comp rx=0 tx=0).stop 2024-09-17T04:51:35.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 -- 172.21.15.150:0/651803137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c80abbe0 msgr2=0x7f01c81edb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:35.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c80abbe0 0x7f01c81edb70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f01c000d820 tx=0x7f01c000dce0 comp rx=0 tx=0).stop 2024-09-17T04:51:35.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 -- 172.21.15.150:0/651803137 shutdown_connections 2024-09-17T04:51:35.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f01b4073200 0x7f01b4075710 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:35.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01c814aac0 0x7f01c814aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:35.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01c80ad970 0x7f01c80add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:35.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 --2- 172.21.15.150:0/651803137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01c80abbe0 0x7f01c81edb70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:35.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 -- 172.21.15.150:0/651803137 >> 172.21.15.150:0/651803137 conn(0x7f01c806e360 msgr2=0x7f01c8150020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:35.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 -- 172.21.15.150:0/651803137 shutdown_connections 2024-09-17T04:51:35.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:35.670+0000 7f01cdf15640 1 -- 172.21.15.150:0/651803137 wait complete. 2024-09-17T04:51:35.690 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-17T04:51:35.770 INFO:tasks.workunit.client.0.smithi150.stdout:checking for stdlib.h... yes 2024-09-17T04:51:35.877 INFO:tasks.workunit.client.0.smithi150.stdout:checking for string.h... yes 2024-09-17T04:51:35.991 INFO:tasks.workunit.client.0.smithi150.stdout:checking for memory.h... yes 2024-09-17T04:51:36.061 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:36.098 INFO:tasks.workunit.client.0.smithi150.stdout:checking for strings.h... yes 2024-09-17T04:51:36.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.202+0000 7eff4f761640 1 -- 172.21.15.150:0/3933084352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff34003a90 msgr2=0x7eff34003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:36.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.202+0000 7eff4f761640 1 --2- 172.21.15.150:0/3933084352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff34003a90 0x7eff34003f00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7eff440016f0 tx=0x7eff4401c790 comp rx=0 tx=0).stop 2024-09-17T04:51:36.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.202+0000 7eff4f761640 1 -- 172.21.15.150:0/3933084352 shutdown_connections 2024-09-17T04:51:36.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.202+0000 7eff4f761640 1 --2- 172.21.15.150:0/3933084352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff34003a90 0x7eff34003f00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:36.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.202+0000 7eff4f761640 1 -- 172.21.15.150:0/3933084352 >> 172.21.15.150:0/3933084352 conn(0x7eff4806e360 msgr2=0x7eff4806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:36.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.202+0000 7eff4f761640 1 -- 172.21.15.150:0/3933084352 shutdown_connections 2024-09-17T04:51:36.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.202+0000 7eff4f761640 1 -- 172.21.15.150:0/3933084352 wait complete. 2024-09-17T04:51:36.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4f761640 1 Processor -- start 2024-09-17T04:51:36.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4f761640 1 -- start start 2024-09-17T04:51:36.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4f761640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff34003a90 0x7eff480bde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:36.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4f761640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff4814d580 0x7eff480adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:36.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4f761640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff481edbf0 0x7eff481ee000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:36.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4f761640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7eff48071120 con 0x7eff481edbf0 2024-09-17T04:51:36.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4f761640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7eff48070fa0 con 0x7eff34003a90 2024-09-17T04:51:36.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4f761640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7eff48072db0 con 0x7eff4814d580 2024-09-17T04:51:36.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4dcfe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff481edbf0 0x7eff481ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:36.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4dcfe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff481edbf0 0x7eff481ee000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42618/0 (socket says 172.21.15.150:42618) 2024-09-17T04:51:36.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4dcfe640 1 -- 172.21.15.150:0/3605890585 learned_addr learned my addr 172.21.15.150:0/3605890585 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:36.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4ccfc640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff4814d580 0x7eff480adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:36.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4d4fd640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff34003a90 0x7eff480bde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4dcfe640 1 -- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff4814d580 msgr2=0x7eff480adfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4dcfe640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff4814d580 0x7eff480adfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4dcfe640 1 -- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff34003a90 msgr2=0x7eff480bde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4dcfe640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff34003a90 0x7eff480bde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4dcfe640 1 -- 172.21.15.150:0/3605890585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff481ed730 con 0x7eff481edbf0 2024-09-17T04:51:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4dcfe640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff481edbf0 0x7eff481ee000 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7eff480c5720 tx=0x7eff4000fed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff3dffb640 1 -- 172.21.15.150:0/3605890585 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4000dda0 con 0x7eff481edbf0 2024-09-17T04:51:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff3dffb640 1 -- 172.21.15.150:0/3605890585 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff40004b30 con 0x7eff481edbf0 2024-09-17T04:51:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff4f761640 1 -- 172.21.15.150:0/3605890585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff481f1900 con 0x7eff481edbf0 2024-09-17T04:51:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.206+0000 7eff3dffb640 1 -- 172.21.15.150:0/3605890585 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff40003980 con 0x7eff481edbf0 2024-09-17T04:51:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.210+0000 7eff4f761640 1 -- 172.21.15.150:0/3605890585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff481df8c0 con 0x7eff481edbf0 2024-09-17T04:51:36.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.210+0000 7eff4f761640 1 -- 172.21.15.150:0/3605890585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff48156be0 con 0x7eff481edbf0 2024-09-17T04:51:36.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.210+0000 7eff3dffb640 1 -- 172.21.15.150:0/3605890585 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7eff40019020 con 0x7eff481edbf0 2024-09-17T04:51:36.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.210+0000 7eff3dffb640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff2c073200 0x7eff2c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:36.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.210+0000 7eff3dffb640 1 -- 172.21.15.150:0/3605890585 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7eff40094ea0 con 0x7eff481edbf0 2024-09-17T04:51:36.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.210+0000 7eff4d4fd640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff2c073200 0x7eff2c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:36.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.210+0000 7eff4d4fd640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff2c073200 0x7eff2c075710 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7eff440016c0 tx=0x7eff44019580 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.214+0000 7eff3dffb640 1 -- 172.21.15.150:0/3605890585 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7eff40020020 con 0x7eff481edbf0 2024-09-17T04:51:36.222 INFO:tasks.workunit.client.0.smithi150.stdout:checking for inttypes.h... yes 2024-09-17T04:51:36.315 INFO:tasks.workunit.client.0.smithi150.stdout:checking for stdint.h... yes 2024-09-17T04:51:36.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.422+0000 7eff4f761640 1 -- 172.21.15.150:0/3605890585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff481e0690 con 0x7eff481edbf0 2024-09-17T04:51:36.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.422+0000 7eff3dffb640 1 -- 172.21.15.150:0/3605890585 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6533 (secure 0 0 0) 0x7eff481ed730 con 0x7eff481edbf0 2024-09-17T04:51:36.424 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:36.424 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":20,"btime":"2024-09-17T04:51:36:054028+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5106,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:36.425 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 20 2024-09-17T04:51:36.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.426+0000 7eff4f761640 1 -- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff2c073200 msgr2=0x7eff2c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:36.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.426+0000 7eff4f761640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff2c073200 0x7eff2c075710 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7eff440016c0 tx=0x7eff44019580 comp rx=0 tx=0).stop 2024-09-17T04:51:36.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.426+0000 7eff4f761640 1 -- 172.21.15.150:0/3605890585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff481edbf0 msgr2=0x7eff481ee000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:36.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.426+0000 7eff4f761640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff481edbf0 0x7eff481ee000 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7eff480c5720 tx=0x7eff4000fed0 comp rx=0 tx=0).stop 2024-09-17T04:51:36.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.426+0000 7eff4f761640 1 -- 172.21.15.150:0/3605890585 shutdown_connections 2024-09-17T04:51:36.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.426+0000 7eff4f761640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff2c073200 0x7eff2c075710 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:36.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.426+0000 7eff4f761640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff481edbf0 0x7eff481ee000 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:36.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.426+0000 7eff4f761640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff4814d580 0x7eff480adfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:36.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.426+0000 7eff4f761640 1 --2- 172.21.15.150:0/3605890585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff34003a90 0x7eff480bde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:36.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.426+0000 7eff4f761640 1 -- 172.21.15.150:0/3605890585 >> 172.21.15.150:0/3605890585 conn(0x7eff4806e360 msgr2=0x7eff4806d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:36.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.426+0000 7eff4f761640 1 -- 172.21.15.150:0/3605890585 shutdown_connections 2024-09-17T04:51:36.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:36.426+0000 7eff4f761640 1 -- 172.21.15.150:0/3605890585 wait complete. 2024-09-17T04:51:36.432 INFO:tasks.workunit.client.0.smithi150.stdout:checking for unistd.h... yes 2024-09-17T04:51:36.445 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:51:36.643 INFO:tasks.workunit.client.0.smithi150.stdout:checking pthread.h usability... yes 2024-09-17T04:51:36.708 INFO:tasks.workunit.client.0.smithi150.stdout:checking pthread.h presence... yes 2024-09-17T04:51:36.709 INFO:tasks.workunit.client.0.smithi150.stdout:checking for pthread.h... yes 2024-09-17T04:51:36.881 INFO:tasks.workunit.client.0.smithi150.stdout:checking fcntl.h usability... yes 2024-09-17T04:51:36.945 INFO:tasks.workunit.client.0.smithi150.stdout:checking fcntl.h presence... yes 2024-09-17T04:51:36.946 INFO:tasks.workunit.client.0.smithi150.stdout:checking for fcntl.h... yes 2024-09-17T04:51:37.129 INFO:tasks.workunit.client.0.smithi150.stdout:checking limits.h usability... yes 2024-09-17T04:51:37.174 INFO:tasks.workunit.client.0.smithi150.stdout:checking limits.h presence... yes 2024-09-17T04:51:37.174 INFO:tasks.workunit.client.0.smithi150.stdout:checking for limits.h... yes 2024-09-17T04:51:37.218 INFO:tasks.workunit.client.0.smithi150.stdout:checking for stdint.h... (cached) yes 2024-09-17T04:51:37.492 INFO:tasks.workunit.client.0.smithi150.stdout:checking sys/time.h usability... yes 2024-09-17T04:51:37.540 INFO:tasks.workunit.client.0.smithi150.stdout:checking sys/time.h presence... yes 2024-09-17T04:51:37.541 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/time.h... yes 2024-09-17T04:51:37.620 INFO:tasks.workunit.client.0.smithi150.stdout:checking for unistd.h... (cached) yes 2024-09-17T04:51:37.691 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:37.779 INFO:tasks.workunit.client.0.smithi150.stdout:checking sys/vfs.h usability... yes 2024-09-17T04:51:37.823 INFO:tasks.workunit.client.0.smithi150.stdout:checking sys/vfs.h presence... yes 2024-09-17T04:51:37.824 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/vfs.h... yes 2024-09-17T04:51:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.846+0000 7f5ddc04b640 1 -- 172.21.15.150:0/634036340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40c97a0 msgr2=0x7f5dd4151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.846+0000 7f5ddc04b640 1 --2- 172.21.15.150:0/634036340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40c97a0 0x7f5dd4151c70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009900 tx=0x7f5dd001c810 comp rx=0 tx=0).stop 2024-09-17T04:51:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.846+0000 7f5ddc04b640 1 -- 172.21.15.150:0/634036340 shutdown_connections 2024-09-17T04:51:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.846+0000 7f5ddc04b640 1 --2- 172.21.15.150:0/634036340 >> v2:172.21.15.176:3300/0 conn(0x7f5dd4072ea0 0x7f5dd40732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.846+0000 7f5ddc04b640 1 --2- 172.21.15.150:0/634036340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40c97a0 0x7f5dd4151c70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.846+0000 7f5ddc04b640 1 -- 172.21.15.150:0/634036340 >> 172.21.15.150:0/634036340 conn(0x7f5dd40abb00 msgr2=0x7f5dd40abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.846+0000 7f5ddc04b640 1 -- 172.21.15.150:0/634036340 shutdown_connections 2024-09-17T04:51:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.846+0000 7f5ddc04b640 1 -- 172.21.15.150:0/634036340 wait complete. 2024-09-17T04:51:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.846+0000 7f5ddc04b640 1 Processor -- start 2024-09-17T04:51:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.846+0000 7f5ddc04b640 1 -- start start 2024-09-17T04:51:37.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5ddc04b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5dd4072ea0 0x7f5dd4150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:37.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5ddc04b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40c97a0 0x7f5dd4156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:37.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5ddc04b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5dd4157be0 0x7f5dd414cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:37.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5ddc04b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd4161790 con 0x7f5dd40c97a0 2024-09-17T04:51:37.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5ddc04b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd40c4df0 con 0x7f5dd4157be0 2024-09-17T04:51:37.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5ddc04b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd4161910 con 0x7f5dd4072ea0 2024-09-17T04:51:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dd95e6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40c97a0 0x7f5dd4156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dd95e6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40c97a0 0x7f5dd4156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42628/0 (socket says 172.21.15.150:42628) 2024-09-17T04:51:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dd95e6640 1 -- 172.21.15.150:0/2986258813 learned_addr learned my addr 172.21.15.150:0/2986258813 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dda5e8640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5dd4157be0 0x7f5dd414cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dd9de7640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5dd4072ea0 0x7f5dd4150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dd95e6640 1 -- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5dd4072ea0 msgr2=0x7f5dd4150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dd95e6640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5dd4072ea0 0x7f5dd4150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dd95e6640 1 -- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5dd4157be0 msgr2=0x7f5dd414cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dd95e6640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5dd4157be0 0x7f5dd414cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dd95e6640 1 -- 172.21.15.150:0/2986258813 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd40aae30 con 0x7f5dd40c97a0 2024-09-17T04:51:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dd95e6640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40c97a0 0x7f5dd4156bc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c970 tx=0x7f5dc400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dca7fc640 1 -- 172.21.15.150:0/2986258813 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4004220 con 0x7f5dd40c97a0 2024-09-17T04:51:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dca7fc640 1 -- 172.21.15.150:0/2986258813 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5dc40079b0 con 0x7f5dd40c97a0 2024-09-17T04:51:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5ddc04b640 1 -- 172.21.15.150:0/2986258813 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd4150390 con 0x7f5dd40c97a0 2024-09-17T04:51:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dca7fc640 1 -- 172.21.15.150:0/2986258813 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc40044b0 con 0x7f5dd40c97a0 2024-09-17T04:51:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5ddc04b640 1 -- 172.21.15.150:0/2986258813 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd414f530 con 0x7f5dd40c97a0 2024-09-17T04:51:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5ddc04b640 1 -- 172.21.15.150:0/2986258813 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd40732b0 con 0x7f5dd40c97a0 2024-09-17T04:51:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dca7fc640 1 -- 172.21.15.150:0/2986258813 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5dc4003600 con 0x7f5dd40c97a0 2024-09-17T04:51:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dca7fc640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5db8073200 0x7f5db8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.850+0000 7f5dca7fc640 1 -- 172.21.15.150:0/2986258813 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f5dc4094de0 con 0x7f5dd40c97a0 2024-09-17T04:51:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.854+0000 7f5dd9de7640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5db8073200 0x7f5db8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.854+0000 7f5dd9de7640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5db8073200 0x7f5db8075710 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00af000 tx=0x7f5dd001b050 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:37.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:37.854+0000 7f5dca7fc640 1 -- 172.21.15.150:0/2986258813 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5dc4014030 con 0x7f5dd40c97a0 2024-09-17T04:51:37.956 INFO:tasks.workunit.client.0.smithi150.stdout:checking sys/limits.h usability... no 2024-09-17T04:51:38.002 INFO:tasks.workunit.client.0.smithi150.stdout:checking sys/limits.h presence... no 2024-09-17T04:51:38.002 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/limits.h... no 2024-09-17T04:51:38.058 INFO:tasks.workunit.client.0.smithi150.stdout:checking for an ANSI C-conforming const... yes 2024-09-17T04:51:38.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.086+0000 7f5ddc04b640 1 -- 172.21.15.150:0/2986258813 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5dd414e980 con 0x7f5dd40c97a0 2024-09-17T04:51:38.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.086+0000 7f5dca7fc640 1 -- 172.21.15.150:0/2986258813 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6533 (secure 0 0 0) 0x7f5dc405fb80 con 0x7f5dd40c97a0 2024-09-17T04:51:38.087 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:38.087 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":20,"btime":"2024-09-17T04:51:36:054028+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5106,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:38.088 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 20 2024-09-17T04:51:38.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.090+0000 7f5ddc04b640 1 -- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5db8073200 msgr2=0x7f5db8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:38.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.090+0000 7f5ddc04b640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5db8073200 0x7f5db8075710 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00af000 tx=0x7f5dd001b050 comp rx=0 tx=0).stop 2024-09-17T04:51:38.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.090+0000 7f5ddc04b640 1 -- 172.21.15.150:0/2986258813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40c97a0 msgr2=0x7f5dd4156bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:38.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.090+0000 7f5ddc04b640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40c97a0 0x7f5dd4156bc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c970 tx=0x7f5dc400ce30 comp rx=0 tx=0).stop 2024-09-17T04:51:38.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.090+0000 7f5ddc04b640 1 -- 172.21.15.150:0/2986258813 shutdown_connections 2024-09-17T04:51:38.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.090+0000 7f5ddc04b640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5db8073200 0x7f5db8075710 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.090+0000 7f5ddc04b640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5dd4157be0 0x7f5dd414cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.090+0000 7f5ddc04b640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd40c97a0 0x7f5dd4156bc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.090+0000 7f5ddc04b640 1 --2- 172.21.15.150:0/2986258813 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5dd4072ea0 0x7f5dd4150f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.090+0000 7f5ddc04b640 1 -- 172.21.15.150:0/2986258813 >> 172.21.15.150:0/2986258813 conn(0x7f5dd40abb00 msgr2=0x7f5dd40a9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:38.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.090+0000 7f5ddc04b640 1 -- 172.21.15.150:0/2986258813 shutdown_connections 2024-09-17T04:51:38.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.090+0000 7f5ddc04b640 1 -- 172.21.15.150:0/2986258813 wait complete. 2024-09-17T04:51:38.108 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-17T04:51:38.109 INFO:tasks.mds_thrash.fs.[a]:waiting for 115 secs before thrashing 2024-09-17T04:51:38.121 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-09-17T04:51:38.125 INFO:tasks.workunit.client.0.smithi150.stdout:checking for function prototypes... yes 2024-09-17T04:51:38.196 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether setvbuf arguments are reversed... no 2024-09-17T04:51:38.330 INFO:tasks.workunit.client.0.smithi150.stdout:checking for system... yes 2024-09-17T04:51:38.446 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:38.447 INFO:tasks.workunit.client.0.smithi150.stdout:checking for gettimeofday... yes 2024-09-17T04:51:38.568 INFO:tasks.workunit.client.0.smithi150.stdout:checking for mkdir... yes 2024-09-17T04:51:38.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.590+0000 7f18efbd6640 1 -- 172.21.15.150:0/475193451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e814a180 msgr2=0x7f18e80c7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:38.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.590+0000 7f18efbd6640 1 --2- 172.21.15.150:0/475193451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e814a180 0x7f18e80c7190 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009bc0 tx=0x7f18e001cb60 comp rx=0 tx=0).stop 2024-09-17T04:51:38.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.590+0000 7f18efbd6640 1 -- 172.21.15.150:0/475193451 shutdown_connections 2024-09-17T04:51:38.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.590+0000 7f18efbd6640 1 --2- 172.21.15.150:0/475193451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e814a180 0x7f18e80c7190 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.590+0000 7f18efbd6640 1 --2- 172.21.15.150:0/475193451 >> v2:172.21.15.176:3300/0 conn(0x7f18e80c7eb0 0x7f18e81540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.590+0000 7f18efbd6640 1 -- 172.21.15.150:0/475193451 >> 172.21.15.150:0/475193451 conn(0x7f18e80aab60 msgr2=0x7f18e80aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:38.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.590+0000 7f18efbd6640 1 -- 172.21.15.150:0/475193451 shutdown_connections 2024-09-17T04:51:38.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.590+0000 7f18efbd6640 1 -- 172.21.15.150:0/475193451 wait complete. 2024-09-17T04:51:38.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.590+0000 7f18efbd6640 1 Processor -- start 2024-09-17T04:51:38.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.590+0000 7f18efbd6640 1 -- start start 2024-09-17T04:51:38.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18efbd6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18e80c7eb0 0x7f18e80c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:38.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18efbd6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18e8149d70 0x7f18e80c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:38.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18efbd6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e814a180 0x7f18e814c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:38.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18efbd6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f18e80c7c60 con 0x7f18e814a180 2024-09-17T04:51:38.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18efbd6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f18e80c7ae0 con 0x7f18e80c7eb0 2024-09-17T04:51:38.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18efbd6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f18e8156540 con 0x7f18e8149d70 2024-09-17T04:51:38.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18ee173640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e814a180 0x7f18e814c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:38.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18ee173640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e814a180 0x7f18e814c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42656/0 (socket says 172.21.15.150:42656) 2024-09-17T04:51:38.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18ee173640 1 -- 172.21.15.150:0/3579605446 learned_addr learned my addr 172.21.15.150:0/3579605446 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18ed972640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18e80c7eb0 0x7f18e80c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18ed171640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18e8149d70 0x7f18e80c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18ee173640 1 -- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18e8149d70 msgr2=0x7f18e80c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18ee173640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18e8149d70 0x7f18e80c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18ee173640 1 -- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18e80c7eb0 msgr2=0x7f18e80c23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18ee173640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18e80c7eb0 0x7f18e80c23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18ee173640 1 -- 172.21.15.150:0/3579605446 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e81f04c0 con 0x7f18e814a180 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18ed972640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18e80c7eb0 0x7f18e80c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18ee173640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e814a180 0x7f18e814c2b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f18e00b0040 tx=0x7f18e0005a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18de7fc640 1 -- 172.21.15.150:0/3579605446 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e00b2040 con 0x7f18e814a180 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18efbd6640 1 -- 172.21.15.150:0/3579605446 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18e80c55f0 con 0x7f18e814a180 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18efbd6640 1 -- 172.21.15.150:0/3579605446 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e80c2d50 con 0x7f18e814a180 2024-09-17T04:51:38.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18de7fc640 1 -- 172.21.15.150:0/3579605446 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18e00a5b20 con 0x7f18e814a180 2024-09-17T04:51:38.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18de7fc640 1 -- 172.21.15.150:0/3579605446 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e00042a0 con 0x7f18e814a180 2024-09-17T04:51:38.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18efbd6640 1 -- 172.21.15.150:0/3579605446 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18d00052a0 con 0x7f18e814a180 2024-09-17T04:51:38.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18de7fc640 1 -- 172.21.15.150:0/3579605446 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f18e00bd030 con 0x7f18e814a180 2024-09-17T04:51:38.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18de7fc640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f18cc073200 0x7f18cc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:38.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.594+0000 7f18de7fc640 1 -- 172.21.15.150:0/3579605446 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6752+0+0 (secure 0 0 0) 0x7f18e012e630 con 0x7f18e814a180 2024-09-17T04:51:38.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.598+0000 7f18ed972640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f18cc073200 0x7f18cc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:38.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.598+0000 7f18ed972640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f18cc073200 0x7f18cc075710 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f18e400bbd0 tx=0x7f18e400b5b0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:38.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.602+0000 7f18de7fc640 1 -- 172.21.15.150:0/3579605446 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f18e00c4020 con 0x7f18e814a180 2024-09-17T04:51:38.708 INFO:tasks.workunit.client.0.smithi150.stdout:checking for strerror... yes 2024-09-17T04:51:38.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.822+0000 7f18efbd6640 1 -- 172.21.15.150:0/3579605446 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18d0005860 con 0x7f18e814a180 2024-09-17T04:51:38.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.822+0000 7f18de7fc640 1 -- 172.21.15.150:0/3579605446 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6533 (secure 0 0 0) 0x7f18e0016750 con 0x7f18e814a180 2024-09-17T04:51:38.823 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:38.823 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":20,"btime":"2024-09-17T04:51:36:054028+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5106,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:49:44.467846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4401":{"gid":4401,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6834/480168496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":480168496},{"type":"v1","addr":"172.21.15.150:6834","nonce":480168496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4401,"qdb_cluster":[4401]},"id":2}]} 2024-09-17T04:51:38.823 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 20 2024-09-17T04:51:38.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.826+0000 7f18efbd6640 1 -- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f18cc073200 msgr2=0x7f18cc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:38.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.826+0000 7f18efbd6640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f18cc073200 0x7f18cc075710 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f18e400bbd0 tx=0x7f18e400b5b0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.826+0000 7f18efbd6640 1 -- 172.21.15.150:0/3579605446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e814a180 msgr2=0x7f18e814c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:38.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.826+0000 7f18efbd6640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e814a180 0x7f18e814c2b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f18e00b0040 tx=0x7f18e0005a60 comp rx=0 tx=0).stop 2024-09-17T04:51:38.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.826+0000 7f18efbd6640 1 -- 172.21.15.150:0/3579605446 shutdown_connections 2024-09-17T04:51:38.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.826+0000 7f18efbd6640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f18cc073200 0x7f18cc075710 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.826+0000 7f18efbd6640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18e814a180 0x7f18e814c2b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.826+0000 7f18efbd6640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18e8149d70 0x7f18e80c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.826+0000 7f18efbd6640 1 --2- 172.21.15.150:0/3579605446 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18e80c7eb0 0x7f18e80c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:38.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.826+0000 7f18efbd6640 1 -- 172.21.15.150:0/3579605446 >> 172.21.15.150:0/3579605446 conn(0x7f18e80aab60 msgr2=0x7f18e806ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:38.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.826+0000 7f18efbd6640 1 -- 172.21.15.150:0/3579605446 shutdown_connections 2024-09-17T04:51:38.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:38.826+0000 7f18efbd6640 1 -- 172.21.15.150:0/3579605446 wait complete. 2024-09-17T04:51:38.838 INFO:tasks.workunit.client.0.smithi150.stdout:checking for open64... yes 2024-09-17T04:51:38.844 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:51:38.963 INFO:tasks.workunit.client.0.smithi150.stdout:checking for stat64... yes 2024-09-17T04:51:39.084 INFO:tasks.workunit.client.0.smithi150.stdout:checking for fseeko64... yes 2024-09-17T04:51:39.198 INFO:tasks.workunit.client.0.smithi150.stdout:checking for lrand48_r... yes 2024-09-17T04:51:39.327 INFO:tasks.workunit.client.0.smithi150.stdout:checking for srand48_r... yes 2024-09-17T04:51:39.387 INFO:tasks.workunit.client.0.smithi150.stdout:configure: creating ./config.status 2024-09-17T04:51:39.582 INFO:tasks.workunit.client.0.smithi150.stdout:config.status: creating Makefile 2024-09-17T04:51:39.604 INFO:tasks.workunit.client.0.smithi150.stdout:config.status: creating config.h 2024-09-17T04:51:39.644 INFO:tasks.workunit.client.0.smithi150.stdout:config.status: executing depfiles commands 2024-09-17T04:51:39.751 INFO:tasks.workunit.client.0.smithi150.stdout:config.status: executing default commands 2024-09-17T04:51:39.905 INFO:tasks.workunit.client.0.smithi150.stderr:+ make 2024-09-17T04:51:39.961 INFO:tasks.workunit.client.0.smithi150.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-09-17T04:51:39.968 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-09-17T04:51:39.969 INFO:tasks.workunit.client.0.smithi150.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-09-17T04:51:39.970 INFO:tasks.workunit.client.0.smithi150.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-09-17T04:51:39.970 INFO:tasks.workunit.client.0.smithi150.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-09-17T04:51:39.970 INFO:tasks.workunit.client.0.smithi150.stderr: any GNU archive site. 2024-09-17T04:51:39.976 INFO:tasks.workunit.client.0.smithi150.stdout: cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-09-17T04:51:39.981 INFO:tasks.workunit.client.0.smithi150.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-09-17T04:51:39.982 INFO:tasks.workunit.client.0.smithi150.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-09-17T04:51:39.982 INFO:tasks.workunit.client.0.smithi150.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-09-17T04:51:39.982 INFO:tasks.workunit.client.0.smithi150.stderr: You might want to install the `Automake' and `Perl' packages. 2024-09-17T04:51:39.982 INFO:tasks.workunit.client.0.smithi150.stderr: Grab them from any GNU archive site. 2024-09-17T04:51:40.326 INFO:tasks.workunit.client.0.smithi150.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-09-17T04:51:40.515 INFO:tasks.workunit.client.0.smithi150.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-17T04:51:40.516 INFO:tasks.workunit.client.0.smithi150.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-17T04:51:40.517 INFO:tasks.workunit.client.0.smithi150.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-17T04:51:40.517 INFO:tasks.workunit.client.0.smithi150.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-17T04:51:40.680 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:4: warning: The macro `AC_FOREACH' is obsolete. 2024-09-17T04:51:40.680 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:4: You should run autoupdate. 2024-09-17T04:51:40.680 INFO:tasks.workunit.client.0.smithi150.stderr:./lib/autoconf/general.m4:191: AC_FOREACH is expanded from... 2024-09-17T04:51:40.680 INFO:tasks.workunit.client.0.smithi150.stderr:aclocal.m4:684: _AM_SET_OPTIONS is expanded from... 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:aclocal.m4:410: AM_INIT_AUTOMAKE is expanded from... 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:4: the top level 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:6: warning: The macro `AM_CONFIG_HEADER' is obsolete. 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:6: You should run autoupdate. 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:aclocal.m4:383: AM_CONFIG_HEADER is expanded from... 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:6: the top level 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:10: warning: The macro `AC_AIX' is obsolete. 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:10: You should run autoupdate. 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:./lib/autoconf/specific.m4:540: AC_AIX is expanded from... 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:10: the top level 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:36: warning: The macro `AC_HEADER_STDC' is obsolete. 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:36: You should run autoupdate. 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from... 2024-09-17T04:51:40.681 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:36: the top level 2024-09-17T04:51:40.682 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:42: warning: The macro `AC_HEADER_TIME' is obsolete. 2024-09-17T04:51:40.682 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:42: You should run autoupdate. 2024-09-17T04:51:40.682 INFO:tasks.workunit.client.0.smithi150.stderr:./lib/autoconf/headers.m4:743: AC_HEADER_TIME is expanded from... 2024-09-17T04:51:40.682 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:42: the top level 2024-09-17T04:51:40.682 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:45: warning: The macro `AC_FUNC_SETVBUF_REVERSED' is obsolete. Remove it and all references to SETVBUF_REVERSED. 2024-09-17T04:51:40.682 INFO:tasks.workunit.client.0.smithi150.stderr:./lib/autoconf/functions.m4:1766: AC_FUNC_SETVBUF_REVERSED is expanded from... 2024-09-17T04:51:40.682 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:45: the top level 2024-09-17T04:51:40.682 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:50: warning: AC_OUTPUT should be used without arguments. 2024-09-17T04:51:40.682 INFO:tasks.workunit.client.0.smithi150.stderr:configure.in:50: You should run autoupdate. 2024-09-17T04:51:40.787 INFO:tasks.workunit.client.0.smithi150.stdout:/bin/bash ./config.status --recheck 2024-09-17T04:51:40.845 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:40.909 INFO:tasks.workunit.client.0.smithi150.stdout:running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion 2024-09-17T04:51:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.038+0000 7fcfac248640 1 -- 172.21.15.150:0/1371838407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfa40c3d10 msgr2=0x7fcfa415df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.038+0000 7fcfac248640 1 --2- 172.21.15.150:0/1371838407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfa40c3d10 0x7fcfa415df70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009c80 tx=0x7fcf9c01cda0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.038+0000 7fcfac248640 1 -- 172.21.15.150:0/1371838407 shutdown_connections 2024-09-17T04:51:41.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.038+0000 7fcfac248640 1 --2- 172.21.15.150:0/1371838407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfa40c3d10 0x7fcfa415df70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.038+0000 7fcfac248640 1 --2- 172.21.15.150:0/1371838407 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfa40c4250 0x7fcfa4153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.038+0000 7fcfac248640 1 -- 172.21.15.150:0/1371838407 >> 172.21.15.150:0/1371838407 conn(0x7fcfa406e360 msgr2=0x7fcfa406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:41.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.038+0000 7fcfac248640 1 -- 172.21.15.150:0/1371838407 shutdown_connections 2024-09-17T04:51:41.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.038+0000 7fcfac248640 1 -- 172.21.15.150:0/1371838407 wait complete. 2024-09-17T04:51:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfac248640 1 Processor -- start 2024-09-17T04:51:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfac248640 1 -- start start 2024-09-17T04:51:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfac248640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfa40c3900 0x7fcfa40a9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfac248640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa40c3d10 0x7fcfa41f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfac248640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfa40c4250 0x7fcfa4149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfac248640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa4071120 con 0x7fcfa40c4250 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfac248640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa4070fa0 con 0x7fcfa40c3d10 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfac248640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa4072db0 con 0x7fcfa40c3900 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfaa7e5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfa40c4250 0x7fcfa4149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfaa7e5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfa40c4250 0x7fcfa4149f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52666/0 (socket says 172.21.15.150:52666) 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfaa7e5640 1 -- 172.21.15.150:0/2023534370 learned_addr learned my addr 172.21.15.150:0/2023534370 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfa97e3640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa40c3d10 0x7fcfa41f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfa9fe4640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfa40c3900 0x7fcfa40a9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfaa7e5640 1 -- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfa40c3900 msgr2=0x7fcfa40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfaa7e5640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfa40c3900 0x7fcfa40a9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfaa7e5640 1 -- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa40c3d10 msgr2=0x7fcfa41f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfaa7e5640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa40c3d10 0x7fcfa41f3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfaa7e5640 1 -- 172.21.15.150:0/2023534370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa40aa5b0 con 0x7fcfa40c4250 2024-09-17T04:51:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.042+0000 7fcfaa7e5640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfa40c4250 0x7fcfa4149f00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0b0040 tx=0x7fcf9c0092c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.046+0000 7fcf9a7fc640 1 -- 172.21.15.150:0/2023534370 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c0bc070 con 0x7fcfa40c4250 2024-09-17T04:51:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.046+0000 7fcfac248640 1 -- 172.21.15.150:0/2023534370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa40aa890 con 0x7fcfa40c4250 2024-09-17T04:51:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.046+0000 7fcf9a7fc640 1 -- 172.21.15.150:0/2023534370 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf9c004240 con 0x7fcfa40c4250 2024-09-17T04:51:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.046+0000 7fcfac248640 1 -- 172.21.15.150:0/2023534370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa41f0300 con 0x7fcfa40c4250 2024-09-17T04:51:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.046+0000 7fcf9a7fc640 1 -- 172.21.15.150:0/2023534370 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c0b2040 con 0x7fcfa40c4250 2024-09-17T04:51:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.046+0000 7fcfac248640 1 -- 172.21.15.150:0/2023534370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa406e9c0 con 0x7fcfa40c4250 2024-09-17T04:51:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.046+0000 7fcf9a7fc640 1 -- 172.21.15.150:0/2023534370 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fcf9c0bf070 con 0x7fcfa40c4250 2024-09-17T04:51:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.046+0000 7fcf9a7fc640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf88073200 0x7fcf88075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.046+0000 7fcf9a7fc640 1 -- 172.21.15.150:0/2023534370 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7fcf9c12e310 con 0x7fcfa40c4250 2024-09-17T04:51:41.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.050+0000 7fcfa9fe4640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf88073200 0x7fcf88075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:41.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.050+0000 7fcfa9fe4640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf88073200 0x7fcf88075710 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009ad0 tx=0x7fcfa00091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.054+0000 7fcf9a7fc640 1 -- 172.21.15.150:0/2023534370 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcf9c016990 con 0x7fcfa40c4250 2024-09-17T04:51:41.160 INFO:tasks.workunit.client.0.smithi150.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-09-17T04:51:41.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.270+0000 7fcfac248640 1 -- 172.21.15.150:0/2023534370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfa4153ee0 con 0x7fcfa40c4250 2024-09-17T04:51:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.270+0000 7fcf9a7fc640 1 -- 172.21.15.150:0/2023534370 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5702 (secure 0 0 0) 0x7fcf9c0f89e0 con 0x7fcfa40c4250 2024-09-17T04:51:41.271 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:41.271 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":21,"btime":"2024-09-17T04:51:40:821887+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:51:40.821881+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"c","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:51:41.271 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 21 2024-09-17T04:51:41.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.274+0000 7fcfac248640 1 -- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf88073200 msgr2=0x7fcf88075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:41.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.274+0000 7fcfac248640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf88073200 0x7fcf88075710 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0009ad0 tx=0x7fcfa00091e0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.274+0000 7fcfac248640 1 -- 172.21.15.150:0/2023534370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfa40c4250 msgr2=0x7fcfa4149f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:41.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.274+0000 7fcfac248640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfa40c4250 0x7fcfa4149f00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0b0040 tx=0x7fcf9c0092c0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.274+0000 7fcfac248640 1 -- 172.21.15.150:0/2023534370 shutdown_connections 2024-09-17T04:51:41.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.274+0000 7fcfac248640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf88073200 0x7fcf88075710 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.274+0000 7fcfac248640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfa40c3d10 0x7fcfa41f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.274+0000 7fcfac248640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfa40c4250 0x7fcfa4149f00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.274+0000 7fcfac248640 1 --2- 172.21.15.150:0/2023534370 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfa40c3900 0x7fcfa40a9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.274+0000 7fcfac248640 1 -- 172.21.15.150:0/2023534370 >> 172.21.15.150:0/2023534370 conn(0x7fcfa406e360 msgr2=0x7fcfa414e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:41.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.274+0000 7fcfac248640 1 -- 172.21.15.150:0/2023534370 shutdown_connections 2024-09-17T04:51:41.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.274+0000 7fcfac248640 1 -- 172.21.15.150:0/2023534370 wait complete. 2024-09-17T04:51:41.291 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-17T04:51:41.291 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T04:51:41.291 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.442+0000 7f9480799640 1 -- 172.21.15.150:0/1974076490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9464004810 msgr2=0x7f9464004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.442+0000 7f9480799640 1 --2- 172.21.15.150:0/1974076490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9464004810 0x7f9464004c20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f94680016f0 tx=0x7f946801c790 comp rx=0 tx=0).stop 2024-09-17T04:51:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.442+0000 7f9480799640 1 -- 172.21.15.150:0/1974076490 shutdown_connections 2024-09-17T04:51:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.442+0000 7f9480799640 1 --2- 172.21.15.150:0/1974076490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9464004810 0x7f9464004c20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.442+0000 7f9480799640 1 -- 172.21.15.150:0/1974076490 >> 172.21.15.150:0/1974076490 conn(0x7f94780c4a00 msgr2=0x7f94780c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:41.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.442+0000 7f9480799640 1 -- 172.21.15.150:0/1974076490 shutdown_connections 2024-09-17T04:51:41.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.442+0000 7f9480799640 1 -- 172.21.15.150:0/1974076490 wait complete. 2024-09-17T04:51:41.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.442+0000 7f9480799640 1 Processor -- start 2024-09-17T04:51:41.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.442+0000 7f9480799640 1 -- start start 2024-09-17T04:51:41.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f9480799640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9464004810 0x7f947814b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:41.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f9480799640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9478154750 0x7f947814e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:41.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f9480799640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f947814e860 0x7f94781492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:41.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f9480799640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9478070ac0 con 0x7f9464004810 2024-09-17T04:51:41.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f9480799640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f94780707e0 con 0x7f9478154750 2024-09-17T04:51:41.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f9480799640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9478070da0 con 0x7f947814e860 2024-09-17T04:51:41.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f947e535640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9464004810 0x7f947814b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:41.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f947e535640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9464004810 0x7f947814b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52690/0 (socket says 172.21.15.150:52690) 2024-09-17T04:51:41.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f947e535640 1 -- 172.21.15.150:0/4240964597 learned_addr learned my addr 172.21.15.150:0/4240964597 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:41.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f947dd34640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9478154750 0x7f947814e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f947ed36640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f947814e860 0x7f94781492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f947e535640 1 -- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f947814e860 msgr2=0x7f94781492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f947e535640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f947814e860 0x7f94781492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f947e535640 1 -- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9478154750 msgr2=0x7f947814e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f947e535640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9478154750 0x7f947814e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f947e535640 1 -- 172.21.15.150:0/4240964597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94781e8d00 con 0x7f9464004810 2024-09-17T04:51:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f947ed36640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f947814e860 0x7f94781492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:51:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f947e535640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9464004810 0x7f947814b450 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f947400f720 tx=0x7f947400fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f946effd640 1 -- 172.21.15.150:0/4240964597 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947401c070 con 0x7f9464004810 2024-09-17T04:51:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f946effd640 1 -- 172.21.15.150:0/4240964597 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f947400de00 con 0x7f9464004810 2024-09-17T04:51:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f9480799640 1 -- 172.21.15.150:0/4240964597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94780ad180 con 0x7f9464004810 2024-09-17T04:51:41.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f946effd640 1 -- 172.21.15.150:0/4240964597 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9474017690 con 0x7f9464004810 2024-09-17T04:51:41.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f9480799640 1 -- 172.21.15.150:0/4240964597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f947814be40 con 0x7f9464004810 2024-09-17T04:51:41.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f9480799640 1 -- 172.21.15.150:0/4240964597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9478153330 con 0x7f9464004810 2024-09-17T04:51:41.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.446+0000 7f946effd640 1 -- 172.21.15.150:0/4240964597 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f947401f070 con 0x7f9464004810 2024-09-17T04:51:41.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.450+0000 7f946effd640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f945c073230 0x7f945c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:41.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.450+0000 7f946effd640 1 -- 172.21.15.150:0/4240964597 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f9474095a20 con 0x7f9464004810 2024-09-17T04:51:41.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.450+0000 7f947dd34640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f945c073230 0x7f945c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:41.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.450+0000 7f947dd34640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f945c073230 0x7f945c075740 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f94680016c0 tx=0x7f94680a7040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.454+0000 7f946effd640 1 -- 172.21.15.150:0/4240964597 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9474060010 con 0x7f9464004810 2024-09-17T04:51:41.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.658+0000 7f9480799640 1 -- 172.21.15.150:0/4240964597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94780a7130 con 0x7f9464004810 2024-09-17T04:51:41.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f946effd640 1 -- 172.21.15.150:0/4240964597 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5702 (secure 0 0 0) 0x7f9474064a40 con 0x7f9464004810 2024-09-17T04:51:41.661 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:41.661 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":21,"btime":"2024-09-17T04:51:40:821887+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:51:40.821881+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"c","rank":0,"incarnation":21,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:51:41.662 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 21 2024-09-17T04:51:41.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f9480799640 1 -- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f945c073230 msgr2=0x7f945c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:41.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f9480799640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f945c073230 0x7f945c075740 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f94680016c0 tx=0x7f94680a7040 comp rx=0 tx=0).stop 2024-09-17T04:51:41.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f9480799640 1 -- 172.21.15.150:0/4240964597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9464004810 msgr2=0x7f947814b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:41.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f9480799640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9464004810 0x7f947814b450 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f947400f720 tx=0x7f947400fbe0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f9480799640 1 -- 172.21.15.150:0/4240964597 shutdown_connections 2024-09-17T04:51:41.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f9480799640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f945c073230 0x7f945c075740 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f9480799640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9478154750 0x7f947814e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f9480799640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f947814e860 0x7f94781492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f9480799640 1 --2- 172.21.15.150:0/4240964597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9464004810 0x7f947814b450 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:41.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f9480799640 1 -- 172.21.15.150:0/4240964597 >> 172.21.15.150:0/4240964597 conn(0x7f94780c4a00 msgr2=0x7f9478073440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:41.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f9480799640 1 -- 172.21.15.150:0/4240964597 shutdown_connections 2024-09-17T04:51:41.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:41.662+0000 7f9480799640 1 -- 172.21.15.150:0/4240964597 wait complete. 2024-09-17T04:51:41.684 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T04:51:40:821887+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 21, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4358": { "addr": "172.21.15.176:6837/1045115232", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 1045115232, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 1045115232, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4358, "incarnation": 16, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:51:28.833974+0000", "qdb_cluster": [ 4358 ], "qdb_leader": 4358, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4358 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 21, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5106": { "addr": "172.21.15.150:6833/961213876", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 961213876, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 961213876, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5106, "incarnation": 21, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:51:40.821881+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5106 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:51:42.179 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether build environment is sane... yes 2024-09-17T04:51:42.193 INFO:tasks.workunit.client.0.smithi150.stdout:checking for a race-free mkdir -p... /usr/bin/mkdir -p 2024-09-17T04:51:42.194 INFO:tasks.workunit.client.0.smithi150.stdout:checking for gawk... gawk 2024-09-17T04:51:42.235 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether make sets $(MAKE)... yes 2024-09-17T04:51:42.298 INFO:tasks.workunit.client.0.smithi150.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-09-17T04:51:42.298 INFO:tasks.workunit.client.0.smithi150.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-09-17T04:51:42.336 INFO:tasks.workunit.client.0.smithi150.stdout:checking for style of include used by make... GNU 2024-09-17T04:51:42.346 INFO:tasks.workunit.client.0.smithi150.stdout:checking for gcc... gcc 2024-09-17T04:51:42.596 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether the C compiler works... yes 2024-09-17T04:51:42.596 INFO:tasks.workunit.client.0.smithi150.stdout:checking for C compiler default output file name... a.out 2024-09-17T04:51:42.678 INFO:tasks.workunit.client.0.smithi150.stdout:checking for suffix of executables... 2024-09-17T04:51:42.753 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether we are cross compiling... no 2024-09-17T04:51:42.832 INFO:tasks.workunit.client.0.smithi150.stdout:checking for suffix of object files... o 2024-09-17T04:51:42.887 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether the compiler supports GNU C... yes 2024-09-17T04:51:42.942 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether gcc accepts -g... yes 2024-09-17T04:51:43.124 INFO:tasks.workunit.client.0.smithi150.stdout:checking for gcc option to enable C11 features... none needed 2024-09-17T04:51:43.480 INFO:tasks.workunit.client.0.smithi150.stdout:checking dependency style of gcc... gcc3 2024-09-17T04:51:43.560 INFO:tasks.workunit.client.0.smithi150.stdout:checking for stdio.h... yes 2024-09-17T04:51:43.685 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:43.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.846+0000 7f25a685a640 1 -- 172.21.15.150:0/2810124690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a00c9040 msgr2=0x7f25a00c55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:43.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.846+0000 7f25a685a640 1 --2- 172.21.15.150:0/2810124690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a00c9040 0x7f25a00c55b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2590009840 tx=0x7f259001c790 comp rx=0 tx=0).stop 2024-09-17T04:51:43.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.846+0000 7f25a685a640 1 -- 172.21.15.150:0/2810124690 shutdown_connections 2024-09-17T04:51:43.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.846+0000 7f25a685a640 1 --2- 172.21.15.150:0/2810124690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a00c9040 0x7f25a00c55b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:43.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.846+0000 7f25a685a640 1 -- 172.21.15.150:0/2810124690 >> 172.21.15.150:0/2810124690 conn(0x7f25a00abf10 msgr2=0x7f25a00ac320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:43.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.846+0000 7f25a685a640 1 -- 172.21.15.150:0/2810124690 shutdown_connections 2024-09-17T04:51:43.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.846+0000 7f25a685a640 1 -- 172.21.15.150:0/2810124690 wait complete. 2024-09-17T04:51:43.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a685a640 1 Processor -- start 2024-09-17T04:51:43.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a685a640 1 -- start start 2024-09-17T04:51:43.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a685a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a00c9040 0x7f25a013ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:43.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a685a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25a013fb50 0x7f25a0158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:43.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a685a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a015a950 0x7f25a00af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:43.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a685a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f25a00c7490 con 0x7f25a015a950 2024-09-17T04:51:43.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a685a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f25a00c7310 con 0x7f25a013fb50 2024-09-17T04:51:43.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a685a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f25a0164b00 con 0x7f25a00c9040 2024-09-17T04:51:43.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a4df7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a015a950 0x7f25a00af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:43.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a4df7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a015a950 0x7f25a00af3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52698/0 (socket says 172.21.15.150:52698) 2024-09-17T04:51:43.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a4df7640 1 -- 172.21.15.150:0/2758350887 learned_addr learned my addr 172.21.15.150:0/2758350887 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:43.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25977fe640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25a013fb50 0x7f25a0158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:43.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f259ffff640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a00c9040 0x7f25a013ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:43.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a4df7640 1 -- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a00c9040 msgr2=0x7f25a013ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:43.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a4df7640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a00c9040 0x7f25a013ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:43.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a4df7640 1 -- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25a013fb50 msgr2=0x7f25a0158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:43.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a4df7640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25a013fb50 0x7f25a0158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:43.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a4df7640 1 -- 172.21.15.150:0/2758350887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a00ad240 con 0x7f25a015a950 2024-09-17T04:51:43.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f259ffff640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a00c9040 0x7f25a013ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:51:43.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a4df7640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a015a950 0x7f25a00af3f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f259800cae0 tx=0x7f259800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:43.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f259d7fa640 1 -- 172.21.15.150:0/2758350887 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2598016020 con 0x7f25a015a950 2024-09-17T04:51:43.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f259d7fa640 1 -- 172.21.15.150:0/2758350887 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f259800aae0 con 0x7f25a015a950 2024-09-17T04:51:43.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f259d7fa640 1 -- 172.21.15.150:0/2758350887 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259800ce20 con 0x7f25a015a950 2024-09-17T04:51:43.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a685a640 1 -- 172.21.15.150:0/2758350887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a01f0ca0 con 0x7f25a015a950 2024-09-17T04:51:43.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a685a640 1 -- 172.21.15.150:0/2758350887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a0158af0 con 0x7f25a015a950 2024-09-17T04:51:43.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.850+0000 7f25a685a640 1 -- 172.21.15.150:0/2758350887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a0158d50 con 0x7f25a015a950 2024-09-17T04:51:43.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.854+0000 7f259d7fa640 1 -- 172.21.15.150:0/2758350887 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2598019020 con 0x7f25a015a950 2024-09-17T04:51:43.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.854+0000 7f259d7fa640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2580073200 0x7f2580075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:43.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.854+0000 7f259d7fa640 1 -- 172.21.15.150:0/2758350887 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f2598094a80 con 0x7f25a015a950 2024-09-17T04:51:43.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.854+0000 7f259ffff640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2580073200 0x7f2580075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:43.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.854+0000 7f259ffff640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2580073200 0x7f2580075710 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f25900af000 tx=0x7f259001afd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:43.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:43.858+0000 7f259d7fa640 1 -- 172.21.15.150:0/2758350887 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2598020030 con 0x7f25a015a950 2024-09-17T04:51:44.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.066+0000 7f25a685a640 1 -- 172.21.15.150:0/2758350887 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25a00c55b0 con 0x7f25a015a950 2024-09-17T04:51:44.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.066+0000 7f259d7fa640 1 -- 172.21.15.150:0/2758350887 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5702 (secure 0 0 0) 0x7f259805f710 con 0x7f25a015a950 2024-09-17T04:51:44.068 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:44.068 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":23,"btime":"2024-09-17T04:51:43:103818+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:51:42.113832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"c","rank":0,"incarnation":21,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:51:44.068 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 23 2024-09-17T04:51:44.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.070+0000 7f25a685a640 1 -- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2580073200 msgr2=0x7f2580075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:44.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.070+0000 7f25a685a640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2580073200 0x7f2580075710 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f25900af000 tx=0x7f259001afd0 comp rx=0 tx=0).stop 2024-09-17T04:51:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.070+0000 7f25a685a640 1 -- 172.21.15.150:0/2758350887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a015a950 msgr2=0x7f25a00af3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.070+0000 7f25a685a640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a015a950 0x7f25a00af3f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f259800cae0 tx=0x7f259800cbd0 comp rx=0 tx=0).stop 2024-09-17T04:51:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.070+0000 7f25a685a640 1 -- 172.21.15.150:0/2758350887 shutdown_connections 2024-09-17T04:51:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.070+0000 7f25a685a640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2580073200 0x7f2580075710 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.070+0000 7f25a685a640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f25a013fb50 0x7f25a0158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.070+0000 7f25a685a640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25a015a950 0x7f25a00af3f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.070+0000 7f25a685a640 1 --2- 172.21.15.150:0/2758350887 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25a00c9040 0x7f25a013ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.070+0000 7f25a685a640 1 -- 172.21.15.150:0/2758350887 >> 172.21.15.150:0/2758350887 conn(0x7f25a00abf10 msgr2=0x7f25a00c57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.070+0000 7f25a685a640 1 -- 172.21.15.150:0/2758350887 shutdown_connections 2024-09-17T04:51:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:44.070+0000 7f25a685a640 1 -- 172.21.15.150:0/2758350887 wait complete. 2024-09-17T04:51:44.090 INFO:tasks.mds_thrash.fs.[b]:no change 2024-09-17T04:51:44.405 INFO:tasks.workunit.client.0.smithi150.stdout:checking for stdlib.h... yes 2024-09-17T04:51:44.486 INFO:tasks.workunit.client.0.smithi150.stdout:checking for string.h... yes 2024-09-17T04:51:44.595 INFO:tasks.workunit.client.0.smithi150.stdout:checking for inttypes.h... yes 2024-09-17T04:51:44.667 INFO:tasks.workunit.client.0.smithi150.stdout:checking for stdint.h... yes 2024-09-17T04:51:44.736 INFO:tasks.workunit.client.0.smithi150.stdout:checking for strings.h... yes 2024-09-17T04:51:44.880 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/stat.h... yes 2024-09-17T04:51:44.955 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/types.h... yes 2024-09-17T04:51:45.028 INFO:tasks.workunit.client.0.smithi150.stdout:checking for unistd.h... yes 2024-09-17T04:51:45.265 INFO:tasks.workunit.client.1.smithi176.stdout:checking for open64... yes 2024-09-17T04:51:45.278 INFO:tasks.workunit.client.0.smithi150.stdout:checking for wchar.h... yes 2024-09-17T04:51:45.352 INFO:tasks.workunit.client.0.smithi150.stdout:checking for minix/config.h... no 2024-09-17T04:51:45.379 INFO:tasks.workunit.client.1.smithi176.stdout:checking for stat64... yes 2024-09-17T04:51:45.409 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/time.h... yes 2024-09-17T04:51:45.482 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-09-17T04:51:45.483 INFO:tasks.workunit.client.1.smithi176.stdout:checking for fseeko64... yes 2024-09-17T04:51:45.530 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether _XOPEN_SOURCE should be defined... no 2024-09-17T04:51:45.552 INFO:tasks.workunit.client.0.smithi150.stdout:checking for gcc... (cached) gcc 2024-09-17T04:51:45.580 INFO:tasks.workunit.client.1.smithi176.stdout:checking for lrand48_r... yes 2024-09-17T04:51:45.628 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether the compiler supports GNU C... (cached) yes 2024-09-17T04:51:45.629 INFO:tasks.workunit.client.0.smithi150.stdout:checking whether gcc accepts -g... (cached) yes 2024-09-17T04:51:45.629 INFO:tasks.workunit.client.0.smithi150.stdout:checking for gcc option to enable C11 features... (cached) none needed 2024-09-17T04:51:45.629 INFO:tasks.workunit.client.0.smithi150.stdout:checking dependency style of gcc... (cached) gcc3 2024-09-17T04:51:45.669 INFO:tasks.workunit.client.1.smithi176.stdout:checking for srand48_r... yes 2024-09-17T04:51:45.690 INFO:tasks.workunit.client.1.smithi176.stdout:configure: creating ./config.status 2024-09-17T04:51:45.701 INFO:tasks.workunit.client.0.smithi150.stdout:checking for main in -lm... yes 2024-09-17T04:51:45.783 INFO:tasks.workunit.client.0.smithi150.stdout:checking for main in -lpthread... yes 2024-09-17T04:51:45.787 INFO:tasks.workunit.client.0.smithi150.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-09-17T04:51:45.788 INFO:tasks.workunit.client.0.smithi150.stdout:checking for egrep... /usr/bin/grep -E 2024-09-17T04:51:45.821 INFO:tasks.workunit.client.1.smithi176.stdout: /bin/bash ./config.status 2024-09-17T04:51:45.850 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-09-17T04:51:45.907 INFO:tasks.workunit.client.1.smithi176.stdout:config.status: creating Makefile 2024-09-17T04:51:45.909 INFO:tasks.workunit.client.0.smithi150.stdout:checking for pthread.h... yes 2024-09-17T04:51:45.931 INFO:tasks.workunit.client.1.smithi176.stdout:config.status: creating config.h 2024-09-17T04:51:45.965 INFO:tasks.workunit.client.1.smithi176.stdout:config.status: executing depfiles commands 2024-09-17T04:51:45.969 INFO:tasks.workunit.client.0.smithi150.stdout:checking for fcntl.h... yes 2024-09-17T04:51:46.005 INFO:tasks.workunit.client.1.smithi176.stdout:config.status: executing default commands 2024-09-17T04:51:46.033 INFO:tasks.workunit.client.0.smithi150.stdout:checking for limits.h... yes 2024-09-17T04:51:46.035 INFO:tasks.workunit.client.0.smithi150.stdout:checking for stdint.h... (cached) yes 2024-09-17T04:51:46.036 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/time.h... (cached) yes 2024-09-17T04:51:46.037 INFO:tasks.workunit.client.0.smithi150.stdout:checking for unistd.h... (cached) yes 2024-09-17T04:51:46.072 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T04:51:46.074+0000 7f42a0ef1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:51:46.074 INFO:tasks.workunit.client.1.smithi176.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run autoheader 2024-09-17T04:51:46.088 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/vfs.h... yes 2024-09-17T04:51:46.091 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:46.141 INFO:tasks.workunit.client.1.smithi176.stderr:autoheader: warning: autoconf input should be named 'configure.ac', not 'configure.in' 2024-09-17T04:51:46.160 INFO:tasks.workunit.client.0.smithi150.stdout:checking for sys/limits.h... no 2024-09-17T04:51:46.196 INFO:tasks.workunit.client.0.smithi150.stdout:checking for an ANSI C-conforming const... yes 2024-09-17T04:51:46.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.234+0000 7fa72655f640 1 -- 172.21.15.150:0/833661851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa720158660 msgr2=0x7fa7201600d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:46.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.234+0000 7fa72655f640 1 --2- 172.21.15.150:0/833661851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa720158660 0x7fa7201600d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa718009bf0 tx=0x7fa71801cae0 comp rx=0 tx=0).stop 2024-09-17T04:51:46.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.234+0000 7fa72655f640 1 -- 172.21.15.150:0/833661851 shutdown_connections 2024-09-17T04:51:46.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.234+0000 7fa72655f640 1 --2- 172.21.15.150:0/833661851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa720158660 0x7fa7201600d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:46.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.234+0000 7fa72655f640 1 -- 172.21.15.150:0/833661851 >> 172.21.15.150:0/833661851 conn(0x7fa7200c5ae0 msgr2=0x7fa720073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:46.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.234+0000 7fa72655f640 1 -- 172.21.15.150:0/833661851 shutdown_connections 2024-09-17T04:51:46.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.234+0000 7fa72655f640 1 -- 172.21.15.150:0/833661851 wait complete. 2024-09-17T04:51:46.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa72655f640 1 Processor -- start 2024-09-17T04:51:46.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa72655f640 1 -- start start 2024-09-17T04:51:46.237 INFO:tasks.workunit.client.1.smithi176.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-17T04:51:46.237 INFO:tasks.workunit.client.1.smithi176.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-17T04:51:46.237 INFO:tasks.workunit.client.1.smithi176.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-17T04:51:46.237 INFO:tasks.workunit.client.1.smithi176.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-17T04:51:46.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa72655f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7200aab50 0x7fa720145c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:46.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa72655f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa72014b150 0x7fa7201eff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:46.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa72655f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa720158660 0x7fa7200c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:46.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa72655f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa720074ab0 con 0x7fa720158660 2024-09-17T04:51:46.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa72655f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa720074c30 con 0x7fa72014b150 2024-09-17T04:51:46.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa72655f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa720158a70 con 0x7fa7200aab50 2024-09-17T04:51:46.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa724afc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa720158660 0x7fa7200c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:46.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa724afc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa720158660 0x7fa7200c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52704/0 (socket says 172.21.15.150:52704) 2024-09-17T04:51:46.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa724afc640 1 -- 172.21.15.150:0/3639735098 learned_addr learned my addr 172.21.15.150:0/3639735098 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:46.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa71f7fe640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa72014b150 0x7fa7201eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:46.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa71ffff640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7200aab50 0x7fa720145c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:46.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa724afc640 1 -- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7200aab50 msgr2=0x7fa720145c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:46.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa724afc640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7200aab50 0x7fa720145c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:46.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa724afc640 1 -- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa72014b150 msgr2=0x7fa7201eff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:46.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa724afc640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa72014b150 0x7fa7201eff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:46.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa724afc640 1 -- 172.21.15.150:0/3639735098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7201ef530 con 0x7fa720158660 2024-09-17T04:51:46.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa724afc640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa720158660 0x7fa7200c1a50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa71801a710 tx=0x7fa7180a5970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:46.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa71cff9640 1 -- 172.21.15.150:0/3639735098 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7180bb070 con 0x7fa720158660 2024-09-17T04:51:46.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa71cff9640 1 -- 172.21.15.150:0/3639735098 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa718004d40 con 0x7fa720158660 2024-09-17T04:51:46.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa72655f640 1 -- 172.21.15.150:0/3639735098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa720144e80 con 0x7fa720158660 2024-09-17T04:51:46.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa71cff9640 1 -- 172.21.15.150:0/3639735098 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7180b2040 con 0x7fa720158660 2024-09-17T04:51:46.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa72655f640 1 -- 172.21.15.150:0/3639735098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7200c8360 con 0x7fa720158660 2024-09-17T04:51:46.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.238+0000 7fa72655f640 1 -- 172.21.15.150:0/3639735098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7201ef700 con 0x7fa720158660 2024-09-17T04:51:46.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.242+0000 7fa71cff9640 1 -- 172.21.15.150:0/3639735098 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa718004320 con 0x7fa720158660 2024-09-17T04:51:46.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.242+0000 7fa71cff9640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa70c073200 0x7fa70c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:46.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.242+0000 7fa71cff9640 1 -- 172.21.15.150:0/3639735098 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7fa71812e6a0 con 0x7fa720158660 2024-09-17T04:51:46.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.242+0000 7fa71ffff640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa70c073200 0x7fa70c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:46.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.242+0000 7fa71ffff640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa70c073200 0x7fa70c075710 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa7100046c0 tx=0x7fa710007ea0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:46.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.246+0000 7fa71cff9640 1 -- 172.21.15.150:0/3639735098 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa718016790 con 0x7fa720158660 2024-09-17T04:51:46.283 INFO:tasks.workunit.client.0.smithi150.stdout:checking for system... yes 2024-09-17T04:51:46.360 INFO:tasks.workunit.client.0.smithi150.stdout:checking for gettimeofday... yes 2024-09-17T04:51:46.390 INFO:tasks.workunit.client.1.smithi176.stdout:rm -f stamp-h1 2024-09-17T04:51:46.395 INFO:tasks.workunit.client.1.smithi176.stdout:touch config.h.in 2024-09-17T04:51:46.402 INFO:tasks.workunit.client.1.smithi176.stdout:cd . && /bin/bash ./config.status config.h 2024-09-17T04:51:46.447 INFO:tasks.workunit.client.0.smithi150.stdout:checking for mkdir... yes 2024-09-17T04:51:46.466 INFO:tasks.workunit.client.1.smithi176.stdout:config.status: creating config.h 2024-09-17T04:51:46.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.490+0000 7fa72655f640 1 -- 172.21.15.150:0/3639735098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa720070420 con 0x7fa720158660 2024-09-17T04:51:46.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.490+0000 7fa71cff9640 1 -- 172.21.15.150:0/3639735098 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5709 (secure 0 0 0) 0x7fa7180f8ca0 con 0x7fa720158660 2024-09-17T04:51:46.490 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:46.490 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":25,"btime":"2024-09-17T04:51:45:122600+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:51:45.122582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5106,"qdb_cluster":[5106]},"id":2}]} 2024-09-17T04:51:46.490 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 25 2024-09-17T04:51:46.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.494+0000 7fa72655f640 1 -- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa70c073200 msgr2=0x7fa70c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:46.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.494+0000 7fa72655f640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa70c073200 0x7fa70c075710 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa7100046c0 tx=0x7fa710007ea0 comp rx=0 tx=0).stop 2024-09-17T04:51:46.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.494+0000 7fa72655f640 1 -- 172.21.15.150:0/3639735098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa720158660 msgr2=0x7fa7200c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:46.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.494+0000 7fa72655f640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa720158660 0x7fa7200c1a50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa71801a710 tx=0x7fa7180a5970 comp rx=0 tx=0).stop 2024-09-17T04:51:46.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.494+0000 7fa72655f640 1 -- 172.21.15.150:0/3639735098 shutdown_connections 2024-09-17T04:51:46.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.494+0000 7fa72655f640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa70c073200 0x7fa70c075710 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:46.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.494+0000 7fa72655f640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa72014b150 0x7fa7201eff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:46.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.494+0000 7fa72655f640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa720158660 0x7fa7200c1a50 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:46.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.494+0000 7fa72655f640 1 --2- 172.21.15.150:0/3639735098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa7200aab50 0x7fa720145c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:46.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.494+0000 7fa72655f640 1 -- 172.21.15.150:0/3639735098 >> 172.21.15.150:0/3639735098 conn(0x7fa7200c5ae0 msgr2=0x7fa720148400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:46.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.494+0000 7fa72655f640 1 -- 172.21.15.150:0/3639735098 shutdown_connections 2024-09-17T04:51:46.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:46.494+0000 7fa72655f640 1 -- 172.21.15.150:0/3639735098 wait complete. 2024-09-17T04:51:46.507 INFO:tasks.workunit.client.1.smithi176.stdout:make all-am 2024-09-17T04:51:46.517 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=4401 2024-09-17T04:51:46.518 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.a 2024-09-17T04:51:46.526 INFO:tasks.workunit.client.0.smithi150.stdout:checking for strerror... yes 2024-09-17T04:51:46.539 INFO:tasks.workunit.client.1.smithi176.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb' 2024-09-17T04:51:46.540 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-09-17T04:51:46.595 INFO:tasks.workunit.client.0.smithi150.stdout:checking for open64... yes 2024-09-17T04:51:46.672 INFO:tasks.workunit.client.0.smithi150.stdout:checking for stat64... yes 2024-09-17T04:51:46.754 INFO:tasks.workunit.client.0.smithi150.stdout:checking for fseeko64... yes 2024-09-17T04:51:46.778 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-09-17T04:51:46.791 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-09-17T04:51:46.835 INFO:tasks.workunit.client.0.smithi150.stdout:checking for lrand48_r... yes 2024-09-17T04:51:46.910 INFO:tasks.workunit.client.0.smithi150.stdout:checking for srand48_r... yes 2024-09-17T04:51:46.929 INFO:tasks.workunit.client.0.smithi150.stdout:configure: creating ./config.status 2024-09-17T04:51:47.015 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-09-17T04:51:47.023 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-09-17T04:51:47.048 INFO:tasks.workunit.client.0.smithi150.stdout: /bin/bash ./config.status 2024-09-17T04:51:47.117 INFO:tasks.workunit.client.0.smithi150.stdout:config.status: creating Makefile 2024-09-17T04:51:47.136 INFO:tasks.workunit.client.0.smithi150.stdout:config.status: creating config.h 2024-09-17T04:51:47.168 INFO:tasks.workunit.client.0.smithi150.stdout:config.status: executing depfiles commands 2024-09-17T04:51:47.186 INFO:tasks.workunit.client.0.smithi150.stdout:config.status: executing default commands 2024-09-17T04:51:47.246 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-09-17T04:51:47.255 INFO:tasks.workunit.client.0.smithi150.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-09-17T04:51:47.256 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-09-17T04:51:47.324 INFO:tasks.workunit.client.0.smithi150.stderr:autoheader: warning: autoconf input should be named 'configure.ac', not 'configure.in' 2024-09-17T04:51:47.399 INFO:tasks.workunit.client.0.smithi150.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-09-17T04:51:47.399 INFO:tasks.workunit.client.0.smithi150.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-09-17T04:51:47.399 INFO:tasks.workunit.client.0.smithi150.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-09-17T04:51:47.399 INFO:tasks.workunit.client.0.smithi150.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-09-17T04:51:47.488 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-09-17T04:51:47.500 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-09-17T04:51:47.544 INFO:tasks.workunit.client.0.smithi150.stdout:rm -f stamp-h1 2024-09-17T04:51:47.550 INFO:tasks.workunit.client.0.smithi150.stdout:touch config.h.in 2024-09-17T04:51:47.556 INFO:tasks.workunit.client.0.smithi150.stdout:cd . && /bin/bash ./config.status config.h 2024-09-17T04:51:47.630 INFO:tasks.workunit.client.0.smithi150.stdout:config.status: creating config.h 2024-09-17T04:51:47.681 INFO:tasks.workunit.client.0.smithi150.stdout:make all-am 2024-09-17T04:51:47.712 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-09-17T04:51:47.712 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-09-17T04:51:47.815 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-09-17T04:51:47.830 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-09-17T04:51:47.969 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-09-17T04:51:47.977 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-09-17T04:51:48.048 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-09-17T04:51:48.063 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-09-17T04:51:48.106 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-09-17T04:51:48.117 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-09-17T04:51:48.203 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-09-17T04:51:48.218 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-09-17T04:51:48.262 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-09-17T04:51:48.273 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-09-17T04:51:48.368 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-09-17T04:51:48.381 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-09-17T04:51:48.439 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-09-17T04:51:48.450 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-09-17T04:51:48.686 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-09-17T04:51:48.701 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-09-17T04:51:48.727 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-09-17T04:51:48.743 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-09-17T04:51:48.806 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-09-17T04:51:48.820 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-09-17T04:51:48.893 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-09-17T04:51:48.904 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-09-17T04:51:48.992 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-09-17T04:51:49.006 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-09-17T04:51:49.013 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-09-17T04:51:49.026 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-09-17T04:51:49.274 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-09-17T04:51:49.289 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-09-17T04:51:49.358 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-09-17T04:51:49.373 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-09-17T04:51:49.487 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-09-17T04:51:49.501 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-09-17T04:51:49.518 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-17T04:51:49.518 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:51:49.518 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:51:49.521 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:51:49.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:49.619 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-09-17T04:51:49.627 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T04:51:49.631 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-09-17T04:51:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.686+0000 7f8f609f8640 1 -- 172.21.15.150:0/1090766822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f5c157ca0 msgr2=0x7f8f5c1580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.686+0000 7f8f609f8640 1 --2- 172.21.15.150:0/1090766822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f5c157ca0 0x7f8f5c1580b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0c7870 tx=0x7f8f5401ca60 comp rx=0 tx=0).stop 2024-09-17T04:51:49.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.686+0000 7f8f609f8640 1 -- 172.21.15.150:0/1090766822 shutdown_connections 2024-09-17T04:51:49.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.686+0000 7f8f609f8640 1 --2- 172.21.15.150:0/1090766822 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f5c157ca0 0x7f8f5c1580b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:49.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.686+0000 7f8f609f8640 1 --2- 172.21.15.150:0/1090766822 >> v2:172.21.15.176:3300/0 conn(0x7f8f5c0ad7f0 0x7f8f5c0c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:49.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.686+0000 7f8f609f8640 1 -- 172.21.15.150:0/1090766822 >> 172.21.15.150:0/1090766822 conn(0x7f8f5c0c9de0 msgr2=0x7f8f5c0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:49.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.686+0000 7f8f609f8640 1 -- 172.21.15.150:0/1090766822 shutdown_connections 2024-09-17T04:51:49.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.686+0000 7f8f609f8640 1 -- 172.21.15.150:0/1090766822 wait complete. 2024-09-17T04:51:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.686+0000 7f8f609f8640 1 Processor -- start 2024-09-17T04:51:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.686+0000 7f8f609f8640 1 -- start start 2024-09-17T04:51:49.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.686+0000 7f8f609f8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f5c0ad7f0 0x7f8f5c1f0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:49.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.686+0000 7f8f609f8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f5c151e10 0x7f8f5c14c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:49.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f609f8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f5c157ca0 0x7f8f5c146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:49.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f609f8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c158770 con 0x7f8f5c157ca0 2024-09-17T04:51:49.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f609f8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c1585f0 con 0x7f8f5c151e10 2024-09-17T04:51:49.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f609f8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8f5c0c7ce0 con 0x7f8f5c0ad7f0 2024-09-17T04:51:49.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f5ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f5c157ca0 0x7f8f5c146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:49.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f5ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f5c157ca0 0x7f8f5c146a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55320/0 (socket says 172.21.15.150:55320) 2024-09-17T04:51:49.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f5ad9d640 1 -- 172.21.15.150:0/903895706 learned_addr learned my addr 172.21.15.150:0/903895706 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:49.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f59d9b640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f5c151e10 0x7f8f5c14c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:49.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f5a59c640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f5c0ad7f0 0x7f8f5c1f0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:49.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f5ad9d640 1 -- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f5c0ad7f0 msgr2=0x7f8f5c1f0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:49.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f5ad9d640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f5c0ad7f0 0x7f8f5c1f0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:49.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f5ad9d640 1 -- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f5c151e10 msgr2=0x7f8f5c14c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:49.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f5ad9d640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f5c151e10 0x7f8f5c14c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:49.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f5ad9d640 1 -- 172.21.15.150:0/903895706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5c1485c0 con 0x7f8f5c157ca0 2024-09-17T04:51:49.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f5ad9d640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f5c157ca0 0x7f8f5c146a90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8f5401a690 tx=0x7f8f54009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f4affd640 1 -- 172.21.15.150:0/903895706 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f540b2040 con 0x7f8f5c157ca0 2024-09-17T04:51:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f4affd640 1 -- 172.21.15.150:0/903895706 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f54004200 con 0x7f8f5c157ca0 2024-09-17T04:51:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f4affd640 1 -- 172.21.15.150:0/903895706 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f54004630 con 0x7f8f5c157ca0 2024-09-17T04:51:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f609f8640 1 -- 172.21.15.150:0/903895706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f5c147780 con 0x7f8f5c157ca0 2024-09-17T04:51:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f609f8640 1 -- 172.21.15.150:0/903895706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f5c155c60 con 0x7f8f5c157ca0 2024-09-17T04:51:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.690+0000 7f8f609f8640 1 -- 172.21.15.150:0/903895706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f5c1487c0 con 0x7f8f5c157ca0 2024-09-17T04:51:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.694+0000 7f8f4affd640 1 -- 172.21.15.150:0/903895706 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8f540ad440 con 0x7f8f5c157ca0 2024-09-17T04:51:49.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.694+0000 7f8f4affd640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8f3c073230 0x7f8f3c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:49.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.694+0000 7f8f4affd640 1 -- 172.21.15.150:0/903895706 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f8f5412f5f0 con 0x7f8f5c157ca0 2024-09-17T04:51:49.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.694+0000 7f8f5a59c640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8f3c073230 0x7f8f3c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:49.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.694+0000 7f8f5a59c640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8f3c073230 0x7f8f3c075740 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c005fd0 tx=0x7f8f4c005f40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:49.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.698+0000 7f8f4affd640 1 -- 172.21.15.150:0/903895706 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8f54016770 con 0x7f8f5c157ca0 2024-09-17T04:51:49.752 INFO:tasks.workunit.client.1.smithi176.stderr:ffsb_op.c: In function 'print_op_results': 2024-09-17T04:51:49.753 INFO:tasks.workunit.client.1.smithi176.stderr:ffsb_op.c:83:37: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-09-17T04:51:49.753 INFO:tasks.workunit.client.1.smithi176.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-09-17T04:51:49.753 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~ 2024-09-17T04:51:49.753 INFO:tasks.workunit.client.1.smithi176.stderr:ffsb_op.c:86:33: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-09-17T04:51:49.753 INFO:tasks.workunit.client.1.smithi176.stderr: 86 | sprintf(buf, "NA\0"); 2024-09-17T04:51:49.753 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~ 2024-09-17T04:51:49.761 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-09-17T04:51:49.770 INFO:tasks.workunit.client.1.smithi176.stderr:ffsb_op.c: In function 'print_results': 2024-09-17T04:51:49.771 INFO:tasks.workunit.client.1.smithi176.stderr:ffsb_op.c:83:33: warning: '/sec' directive writing 4 bytes into a region of size between 1 and 256 [-Wformat-overflow=] 2024-09-17T04:51:49.771 INFO:tasks.workunit.client.1.smithi176.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-09-17T04:51:49.771 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~ 2024-09-17T04:51:49.771 INFO:tasks.workunit.client.1.smithi176.stderr:In file included from /usr/include/stdio.h:894, 2024-09-17T04:51:49.771 INFO:tasks.workunit.client.1.smithi176.stderr: from ffsb_op.c:20: 2024-09-17T04:51:49.771 INFO:tasks.workunit.client.1.smithi176.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 5 and 260 bytes into a destination of size 256 2024-09-17T04:51:49.771 INFO:tasks.workunit.client.1.smithi176.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, 2024-09-17T04:51:49.771 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:49.771 INFO:tasks.workunit.client.1.smithi176.stderr: 39 | __glibc_objsize (__s), __fmt, 2024-09-17T04:51:49.771 INFO:tasks.workunit.client.1.smithi176.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:49.772 INFO:tasks.workunit.client.1.smithi176.stderr: 40 | __va_arg_pack ()); 2024-09-17T04:51:49.772 INFO:tasks.workunit.client.1.smithi176.stderr: | ~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:49.772 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-09-17T04:51:49.822 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-09-17T04:51:49.835 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-09-17T04:51:49.878 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-09-17T04:51:49.887 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-09-17T04:51:49.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.950+0000 7f8f609f8640 1 -- 172.21.15.150:0/903895706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f5c0c6230 con 0x7f8f5c157ca0 2024-09-17T04:51:49.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.950+0000 7f8f4affd640 1 -- 172.21.15.150:0/903895706 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5709 (secure 0 0 0) 0x7f8f540f9d50 con 0x7f8f5c157ca0 2024-09-17T04:51:49.951 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:49.951 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":25,"btime":"2024-09-17T04:51:45:122600+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:51:45.122582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5106,"qdb_cluster":[5106]},"id":2}]} 2024-09-17T04:51:49.951 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 25 2024-09-17T04:51:49.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.954+0000 7f8f609f8640 1 -- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8f3c073230 msgr2=0x7f8f3c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:49.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.954+0000 7f8f609f8640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8f3c073230 0x7f8f3c075740 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c005fd0 tx=0x7f8f4c005f40 comp rx=0 tx=0).stop 2024-09-17T04:51:49.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.954+0000 7f8f609f8640 1 -- 172.21.15.150:0/903895706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f5c157ca0 msgr2=0x7f8f5c146a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:49.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.954+0000 7f8f609f8640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f5c157ca0 0x7f8f5c146a90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8f5401a690 tx=0x7f8f54009260 comp rx=0 tx=0).stop 2024-09-17T04:51:49.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.954+0000 7f8f609f8640 1 -- 172.21.15.150:0/903895706 shutdown_connections 2024-09-17T04:51:49.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.954+0000 7f8f609f8640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8f3c073230 0x7f8f3c075740 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:49.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.954+0000 7f8f609f8640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8f5c151e10 0x7f8f5c14c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:49.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.954+0000 7f8f609f8640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f5c157ca0 0x7f8f5c146a90 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:49.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.954+0000 7f8f609f8640 1 --2- 172.21.15.150:0/903895706 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8f5c0ad7f0 0x7f8f5c1f0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:49.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.954+0000 7f8f609f8640 1 -- 172.21.15.150:0/903895706 >> 172.21.15.150:0/903895706 conn(0x7f8f5c0c9de0 msgr2=0x7f8f5c0aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:49.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.954+0000 7f8f609f8640 1 -- 172.21.15.150:0/903895706 shutdown_connections 2024-09-17T04:51:49.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:49.954+0000 7f8f609f8640 1 -- 172.21.15.150:0/903895706 wait complete. 2024-09-17T04:51:49.955 INFO:tasks.workunit.client.0.smithi150.stderr:ffsb_op.c: In function 'print_op_results': 2024-09-17T04:51:49.955 INFO:tasks.workunit.client.0.smithi150.stderr:ffsb_op.c:83:37: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-09-17T04:51:49.956 INFO:tasks.workunit.client.0.smithi150.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-09-17T04:51:49.956 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~ 2024-09-17T04:51:49.956 INFO:tasks.workunit.client.0.smithi150.stderr:ffsb_op.c:86:33: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-09-17T04:51:49.956 INFO:tasks.workunit.client.0.smithi150.stderr: 86 | sprintf(buf, "NA\0"); 2024-09-17T04:51:49.956 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~ 2024-09-17T04:51:49.974 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-17T04:51:49.985 INFO:tasks.workunit.client.0.smithi150.stderr:ffsb_op.c: In function 'print_results': 2024-09-17T04:51:49.985 INFO:tasks.workunit.client.0.smithi150.stderr:ffsb_op.c:83:33: warning: '/sec' directive writing 4 bytes into a region of size between 1 and 256 [-Wformat-overflow=] 2024-09-17T04:51:49.985 INFO:tasks.workunit.client.0.smithi150.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-09-17T04:51:49.985 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~ 2024-09-17T04:51:49.985 INFO:tasks.workunit.client.0.smithi150.stderr:In file included from /usr/include/stdio.h:894, 2024-09-17T04:51:49.985 INFO:tasks.workunit.client.0.smithi150.stderr: from ffsb_op.c:20: 2024-09-17T04:51:49.985 INFO:tasks.workunit.client.0.smithi150.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 5 and 260 bytes into a destination of size 256 2024-09-17T04:51:49.985 INFO:tasks.workunit.client.0.smithi150.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, 2024-09-17T04:51:49.985 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:49.985 INFO:tasks.workunit.client.0.smithi150.stderr: 39 | __glibc_objsize (__s), __fmt, 2024-09-17T04:51:49.985 INFO:tasks.workunit.client.0.smithi150.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:49.986 INFO:tasks.workunit.client.0.smithi150.stderr: 40 | __va_arg_pack ()); 2024-09-17T04:51:49.986 INFO:tasks.workunit.client.0.smithi150.stderr: | ~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.032 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-09-17T04:51:50.044 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-09-17T04:51:50.164 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-09-17T04:51:50.178 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-09-17T04:51:50.213 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-09-17T04:51:50.221 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-09-17T04:51:50.295 INFO:tasks.workunit.client.0.smithi150.stderr:In file included from parser.h:22, 2024-09-17T04:51:50.295 INFO:tasks.workunit.client.0.smithi150.stderr: from parser.c:25: 2024-09-17T04:51:50.295 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c: In function 'init_filesys': 2024-09-17T04:51:50.296 INFO:tasks.workunit.client.1.smithi176.stderr:In file included from parser.h:22, 2024-09-17T04:51:50.296 INFO:tasks.workunit.client.1.smithi176.stderr: from parser.c:25: 2024-09-17T04:51:50.296 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c: In function 'init_filesys': 2024-09-17T04:51:50.297 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.297 INFO:tasks.workunit.client.0.smithi150.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.297 INFO:tasks.workunit.client.0.smithi150.stderr: | ^ 2024-09-17T04:51:50.297 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.297 INFO:tasks.workunit.client.0.smithi150.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.298 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.298 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.298 INFO:tasks.workunit.client.0.smithi150.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.298 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.298 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.298 INFO:tasks.workunit.client.0.smithi150.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-17T04:51:50.298 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.298 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.298 INFO:tasks.workunit.client.0.smithi150.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-17T04:51:50.298 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.298 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.298 INFO:tasks.workunit.client.0.smithi150.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr: | ^ 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-17T04:51:50.299 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr: | ^ 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.300 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr: | ^ 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-17T04:51:50.301 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.302 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.302 INFO:tasks.workunit.client.0.smithi150.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-17T04:51:50.302 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.302 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c: In function 'init_tg_stats': 2024-09-17T04:51:50.302 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.302 INFO:tasks.workunit.client.0.smithi150.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.302 INFO:tasks.workunit.client.0.smithi150.stderr: | ^ 2024-09-17T04:51:50.302 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.302 INFO:tasks.workunit.client.0.smithi150.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.302 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.302 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr: | ^ 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.303 INFO:tasks.workunit.client.0.smithi150.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr: | ^ 2024-09-17T04:51:50.304 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.305 INFO:tasks.workunit.client.0.smithi150.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.305 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.305 INFO:tasks.workunit.client.0.smithi150.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.305 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.305 INFO:tasks.workunit.client.0.smithi150.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-17T04:51:50.305 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.305 INFO:tasks.workunit.client.0.smithi150.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-17T04:51:50.305 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.305 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr: | ^ 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-17T04:51:50.306 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.308 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.308 INFO:tasks.workunit.client.1.smithi176.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.308 INFO:tasks.workunit.client.1.smithi176.stderr: | ^ 2024-09-17T04:51:50.308 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.308 INFO:tasks.workunit.client.1.smithi176.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.308 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.308 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.308 INFO:tasks.workunit.client.1.smithi176.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.308 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.309 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.309 INFO:tasks.workunit.client.1.smithi176.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-17T04:51:50.309 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.309 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.309 INFO:tasks.workunit.client.1.smithi176.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-17T04:51:50.309 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.309 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.309 INFO:tasks.workunit.client.1.smithi176.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.309 INFO:tasks.workunit.client.1.smithi176.stderr: | ^ 2024-09-17T04:51:50.309 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.309 INFO:tasks.workunit.client.1.smithi176.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.309 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr: | ^ 2024-09-17T04:51:50.310 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.311 INFO:tasks.workunit.client.1.smithi176.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.312 INFO:tasks.workunit.client.1.smithi176.stderr: | ^ 2024-09-17T04:51:50.312 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.312 INFO:tasks.workunit.client.1.smithi176.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.312 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.312 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.312 INFO:tasks.workunit.client.1.smithi176.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.312 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.312 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.312 INFO:tasks.workunit.client.1.smithi176.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c: In function 'init_tg_stats': 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr: | ^ 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.313 INFO:tasks.workunit.client.1.smithi176.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.314 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.314 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.314 INFO:tasks.workunit.client.1.smithi176.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-17T04:51:50.314 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.314 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.314 INFO:tasks.workunit.client.1.smithi176.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-17T04:51:50.315 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.315 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.315 INFO:tasks.workunit.client.1.smithi176.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.315 INFO:tasks.workunit.client.1.smithi176.stderr: | ^ 2024-09-17T04:51:50.315 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.315 INFO:tasks.workunit.client.1.smithi176.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.315 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.315 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.315 INFO:tasks.workunit.client.1.smithi176.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.315 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.315 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr: 971 | list_for_each_entry(tmp_list, 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr: | ^ 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.316 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr: | ^ 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~ 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-09-17T04:51:50.317 INFO:tasks.workunit.client.1.smithi176.stderr: 36 | container_of(ptr, type, member) 2024-09-17T04:51:50.318 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~ 2024-09-17T04:51:50.318 INFO:tasks.workunit.client.1.smithi176.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-09-17T04:51:50.318 INFO:tasks.workunit.client.1.smithi176.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-09-17T04:51:50.318 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~ 2024-09-17T04:51:50.318 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-09-17T04:51:50.318 INFO:tasks.workunit.client.1.smithi176.stderr: 980 | list_for_each_entry(tmp_list, 2024-09-17T04:51:50.318 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.360 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c: In function 'get_optstr': 2024-09-17T04:51:50.360 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c:206:23: warning: 'strnlen' specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-09-17T04:51:50.360 INFO:tasks.workunit.client.1.smithi176.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-09-17T04:51:50.360 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.360 INFO:tasks.workunit.client.1.smithi176.stderr:parser.c:200:14: note: source object declared here 2024-09-17T04:51:50.360 INFO:tasks.workunit.client.1.smithi176.stderr: 200 | char temp[BUFSIZE]; 2024-09-17T04:51:50.360 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~ 2024-09-17T04:51:50.374 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c: In function 'get_optstr': 2024-09-17T04:51:50.374 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c:206:23: warning: 'strnlen' specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-09-17T04:51:50.374 INFO:tasks.workunit.client.0.smithi150.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-09-17T04:51:50.374 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.374 INFO:tasks.workunit.client.0.smithi150.stderr:parser.c:200:14: note: source object declared here 2024-09-17T04:51:50.374 INFO:tasks.workunit.client.0.smithi150.stderr: 200 | char temp[BUFSIZE]; 2024-09-17T04:51:50.374 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~ 2024-09-17T04:51:50.377 INFO:tasks.workunit.client.1.smithi176.stderr:In file included from /usr/include/string.h:535, 2024-09-17T04:51:50.377 INFO:tasks.workunit.client.1.smithi176.stderr: from parser.c:19: 2024-09-17T04:51:50.377 INFO:tasks.workunit.client.1.smithi176.stderr:In function 'memcpy', 2024-09-17T04:51:50.377 INFO:tasks.workunit.client.1.smithi176.stderr: inlined from 'size64_convert' at parser.c:119:2: 2024-09-17T04:51:50.377 INFO:tasks.workunit.client.1.smithi176.stderr:/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10: warning: '__builtin_memcpy' reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-09-17T04:51:50.377 INFO:tasks.workunit.client.1.smithi176.stderr: 29 | return __builtin___memcpy_chk (__dest, __src, __len, 2024-09-17T04:51:50.377 INFO:tasks.workunit.client.1.smithi176.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.377 INFO:tasks.workunit.client.1.smithi176.stderr: 30 | __glibc_objsize0 (__dest)); 2024-09-17T04:51:50.377 INFO:tasks.workunit.client.1.smithi176.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.394 INFO:tasks.workunit.client.0.smithi150.stderr:In file included from /usr/include/string.h:535, 2024-09-17T04:51:50.394 INFO:tasks.workunit.client.0.smithi150.stderr: from parser.c:19: 2024-09-17T04:51:50.394 INFO:tasks.workunit.client.0.smithi150.stderr:In function 'memcpy', 2024-09-17T04:51:50.394 INFO:tasks.workunit.client.0.smithi150.stderr: inlined from 'size64_convert' at parser.c:119:2: 2024-09-17T04:51:50.394 INFO:tasks.workunit.client.0.smithi150.stderr:/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10: warning: '__builtin_memcpy' reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-09-17T04:51:50.394 INFO:tasks.workunit.client.0.smithi150.stderr: 29 | return __builtin___memcpy_chk (__dest, __src, __len, 2024-09-17T04:51:50.394 INFO:tasks.workunit.client.0.smithi150.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.394 INFO:tasks.workunit.client.0.smithi150.stderr: 30 | __glibc_objsize0 (__dest)); 2024-09-17T04:51:50.394 INFO:tasks.workunit.client.0.smithi150.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-17T04:51:50.725 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-09-17T04:51:50.733 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-09-17T04:51:50.738 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-09-17T04:51:50.746 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-09-17T04:51:50.857 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-09-17T04:51:50.868 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-09-17T04:51:50.922 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-09-17T04:51:50.938 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-09-17T04:51:51.014 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-09-17T04:51:51.024 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-09-17T04:51:51.068 INFO:tasks.workunit.client.0.smithi150.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-09-17T04:51:51.080 INFO:tasks.workunit.client.0.smithi150.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-09-17T04:51:51.160 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-09-17T04:51:51.172 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-09-17T04:51:51.217 INFO:tasks.workunit.client.0.smithi150.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-09-17T04:51:51.217 INFO:tasks.workunit.client.0.smithi150.stderr:+ cd .. 2024-09-17T04:51:51.219 INFO:tasks.workunit.client.0.smithi150.stderr:+ mkdir tmp 2024-09-17T04:51:51.221 INFO:tasks.workunit.client.0.smithi150.stderr:+ cd tmp 2024-09-17T04:51:51.222 INFO:tasks.workunit.client.0.smithi150.stderr:+ for f in $mydir/*.ffsb 2024-09-17T04:51:51.222 INFO:tasks.workunit.client.0.smithi150.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-09-17T04:51:51.223 INFO:tasks.workunit.client.0.smithi150.stdout:FFSB version 6.0-RC2 started 2024-09-17T04:51:51.223 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout:benchmark time = 300 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout:ThreadGroup 0 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout:================ 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout: num_threads = 32 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout: read_random = off 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout: read_size = 0 (0B) 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout: read_blocksize = 0 (0B) 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout: read_skip = off 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout: read_skipsize = 0 (0B) 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout: write_random = on 2024-09-17T04:51:51.227 INFO:tasks.workunit.client.0.smithi150.stdout: write_size = 5242880 (5MB) 2024-09-17T04:51:51.228 INFO:tasks.workunit.client.0.smithi150.stdout: fsync_file = 0 2024-09-17T04:51:51.228 INFO:tasks.workunit.client.0.smithi150.stdout: write_blocksize = 4096 (4KB) 2024-09-17T04:51:51.228 INFO:tasks.workunit.client.0.smithi150.stdout: wait time = 0 2024-09-17T04:51:51.228 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T04:51:51.228 INFO:tasks.workunit.client.0.smithi150.stdout: op weights 2024-09-17T04:51:51.228 INFO:tasks.workunit.client.0.smithi150.stdout: read = 0 (0.00%) 2024-09-17T04:51:51.228 INFO:tasks.workunit.client.0.smithi150.stdout: readall = 0 (0.00%) 2024-09-17T04:51:51.228 INFO:tasks.workunit.client.0.smithi150.stdout: write = 1 (100.00%) 2024-09-17T04:51:51.228 INFO:tasks.workunit.client.0.smithi150.stdout: create = 0 (0.00%) 2024-09-17T04:51:51.228 INFO:tasks.workunit.client.0.smithi150.stdout: append = 0 (0.00%) 2024-09-17T04:51:51.229 INFO:tasks.workunit.client.0.smithi150.stdout: delete = 0 (0.00%) 2024-09-17T04:51:51.229 INFO:tasks.workunit.client.0.smithi150.stdout: metaop = 0 (0.00%) 2024-09-17T04:51:51.229 INFO:tasks.workunit.client.0.smithi150.stdout: createdir = 0 (0.00%) 2024-09-17T04:51:51.229 INFO:tasks.workunit.client.0.smithi150.stdout: stat = 0 (0.00%) 2024-09-17T04:51:51.229 INFO:tasks.workunit.client.0.smithi150.stdout: writeall = 0 (0.00%) 2024-09-17T04:51:51.229 INFO:tasks.workunit.client.0.smithi150.stdout: writeall_fsync = 0 (0.00%) 2024-09-17T04:51:51.229 INFO:tasks.workunit.client.0.smithi150.stdout: open_close = 0 (0.00%) 2024-09-17T04:51:51.229 INFO:tasks.workunit.client.0.smithi150.stdout: write_fsync = 0 (0.00%) 2024-09-17T04:51:51.230 INFO:tasks.workunit.client.0.smithi150.stdout: create_fsync = 0 (0.00%) 2024-09-17T04:51:51.230 INFO:tasks.workunit.client.0.smithi150.stdout: append_fsync = 0 (0.00%) 2024-09-17T04:51:51.230 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T04:51:51.230 INFO:tasks.workunit.client.0.smithi150.stdout:FileSystem . 2024-09-17T04:51:51.230 INFO:tasks.workunit.client.0.smithi150.stdout:========== 2024-09-17T04:51:51.230 INFO:tasks.workunit.client.0.smithi150.stdout: num_dirs = 0 2024-09-17T04:51:51.231 INFO:tasks.workunit.client.0.smithi150.stdout: starting files = 128 2024-09-17T04:51:51.231 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T04:51:51.231 INFO:tasks.workunit.client.0.smithi150.stdout: min file size = 104857600 (100MB) 2024-09-17T04:51:51.231 INFO:tasks.workunit.client.0.smithi150.stdout: max file size = 104857600 (100MB) 2024-09-17T04:51:51.231 INFO:tasks.workunit.client.0.smithi150.stdout: directio = off 2024-09-17T04:51:51.231 INFO:tasks.workunit.client.0.smithi150.stdout: alignedio = on 2024-09-17T04:51:51.232 INFO:tasks.workunit.client.0.smithi150.stdout: bufferedio = off 2024-09-17T04:51:51.232 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T04:51:51.232 INFO:tasks.workunit.client.0.smithi150.stdout: aging is off 2024-09-17T04:51:51.232 INFO:tasks.workunit.client.0.smithi150.stdout: current utilization = 0.00% 2024-09-17T04:51:51.232 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T04:51:51.232 INFO:tasks.workunit.client.0.smithi150.stdout:checking existing fs: . 2024-09-17T04:51:51.232 INFO:tasks.workunit.client.0.smithi150.stdout:recreating new fileset 2024-09-17T04:51:51.233 INFO:tasks.workunit.client.0.smithi150.stderr:opendir: No such file or directory 2024-09-17T04:51:51.247 INFO:tasks.workunit.client.1.smithi176.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-09-17T04:51:51.261 INFO:tasks.workunit.client.1.smithi176.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-09-17T04:51:51.418 INFO:tasks.workunit.client.1.smithi176.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb' 2024-09-17T04:51:51.419 INFO:tasks.workunit.client.1.smithi176.stderr:+ cd .. 2024-09-17T04:51:51.421 INFO:tasks.workunit.client.1.smithi176.stderr:+ mkdir tmp 2024-09-17T04:51:51.423 INFO:tasks.workunit.client.1.smithi176.stderr:+ cd tmp 2024-09-17T04:51:51.425 INFO:tasks.workunit.client.1.smithi176.stderr:+ for f in $mydir/*.ffsb 2024-09-17T04:51:51.425 INFO:tasks.workunit.client.1.smithi176.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/random_write.32.ffsb 2024-09-17T04:51:51.426 INFO:tasks.workunit.client.1.smithi176.stdout:FFSB version 6.0-RC2 started 2024-09-17T04:51:51.426 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T04:51:51.429 INFO:tasks.workunit.client.1.smithi176.stdout:benchmark time = 300 2024-09-17T04:51:51.429 INFO:tasks.workunit.client.1.smithi176.stdout:ThreadGroup 0 2024-09-17T04:51:51.430 INFO:tasks.workunit.client.1.smithi176.stdout:================ 2024-09-17T04:51:51.430 INFO:tasks.workunit.client.1.smithi176.stdout: num_threads = 32 2024-09-17T04:51:51.430 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T04:51:51.430 INFO:tasks.workunit.client.1.smithi176.stdout: read_random = off 2024-09-17T04:51:51.430 INFO:tasks.workunit.client.1.smithi176.stdout: read_size = 0 (0B) 2024-09-17T04:51:51.430 INFO:tasks.workunit.client.1.smithi176.stdout: read_blocksize = 0 (0B) 2024-09-17T04:51:51.430 INFO:tasks.workunit.client.1.smithi176.stdout: read_skip = off 2024-09-17T04:51:51.430 INFO:tasks.workunit.client.1.smithi176.stdout: read_skipsize = 0 (0B) 2024-09-17T04:51:51.430 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: write_random = on 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: write_size = 5242880 (5MB) 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: fsync_file = 0 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: write_blocksize = 4096 (4KB) 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: wait time = 0 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: op weights 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: read = 0 (0.00%) 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: readall = 0 (0.00%) 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: write = 1 (100.00%) 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: create = 0 (0.00%) 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: append = 0 (0.00%) 2024-09-17T04:51:51.431 INFO:tasks.workunit.client.1.smithi176.stdout: delete = 0 (0.00%) 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout: metaop = 0 (0.00%) 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout: createdir = 0 (0.00%) 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout: stat = 0 (0.00%) 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout: writeall = 0 (0.00%) 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout: writeall_fsync = 0 (0.00%) 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout: open_close = 0 (0.00%) 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout: write_fsync = 0 (0.00%) 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout: create_fsync = 0 (0.00%) 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout: append_fsync = 0 (0.00%) 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout:FileSystem . 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout:========== 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout: num_dirs = 0 2024-09-17T04:51:51.432 INFO:tasks.workunit.client.1.smithi176.stdout: starting files = 128 2024-09-17T04:51:51.433 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T04:51:51.433 INFO:tasks.workunit.client.1.smithi176.stdout: min file size = 104857600 (100MB) 2024-09-17T04:51:51.433 INFO:tasks.workunit.client.1.smithi176.stdout: max file size = 104857600 (100MB) 2024-09-17T04:51:51.433 INFO:tasks.workunit.client.1.smithi176.stdout: directio = off 2024-09-17T04:51:51.433 INFO:tasks.workunit.client.1.smithi176.stdout: alignedio = on 2024-09-17T04:51:51.433 INFO:tasks.workunit.client.1.smithi176.stdout: bufferedio = off 2024-09-17T04:51:51.433 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T04:51:51.433 INFO:tasks.workunit.client.1.smithi176.stdout: aging is off 2024-09-17T04:51:51.434 INFO:tasks.workunit.client.1.smithi176.stdout: current utilization = 0.00% 2024-09-17T04:51:51.434 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T04:51:51.434 INFO:tasks.workunit.client.1.smithi176.stdout:checking existing fs: . 2024-09-17T04:51:51.434 INFO:tasks.workunit.client.1.smithi176.stdout:recreating new fileset 2024-09-17T04:51:51.434 INFO:tasks.workunit.client.1.smithi176.stderr:opendir: No such file or directory 2024-09-17T04:51:51.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:51:52.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.142+0000 7f39810db640 1 -- 172.21.15.150:0/2471502116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f397c0c42e0 msgr2=0x7f397c0c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:52.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.142+0000 7f39810db640 1 --2- 172.21.15.150:0/2471502116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f397c0c42e0 0x7f397c0c6690 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f397c0ae070 tx=0x7f396c01db00 comp rx=0 tx=0).stop 2024-09-17T04:51:52.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.142+0000 7f39810db640 1 -- 172.21.15.150:0/2471502116 shutdown_connections 2024-09-17T04:51:52.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.142+0000 7f39810db640 1 --2- 172.21.15.150:0/2471502116 >> v2:172.21.15.176:3300/0 conn(0x7f397c152cf0 0x7f397c153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:52.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.142+0000 7f39810db640 1 --2- 172.21.15.150:0/2471502116 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f397c0c42e0 0x7f397c0c6690 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:52.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.142+0000 7f39810db640 1 -- 172.21.15.150:0/2471502116 >> 172.21.15.150:0/2471502116 conn(0x7f397c0c4a00 msgr2=0x7f397c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:52.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.142+0000 7f39810db640 1 -- 172.21.15.150:0/2471502116 shutdown_connections 2024-09-17T04:51:52.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.142+0000 7f39810db640 1 -- 172.21.15.150:0/2471502116 wait complete. 2024-09-17T04:51:52.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f39810db640 1 Processor -- start 2024-09-17T04:51:52.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f39810db640 1 -- start start 2024-09-17T04:51:52.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f39810db640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f397c0a7350 0x7f397c14e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f39810db640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f397c0c42e0 0x7f397c0c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f39810db640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f397c152cf0 0x7f397c149510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f39810db640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f397c0c6d50 con 0x7f397c0a7350 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f39810db640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f397c0c6bd0 con 0x7f397c152cf0 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f39810db640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f397c153640 con 0x7f397c0c42e0 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f397ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f397c0a7350 0x7f397c14e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f397ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f397c0a7350 0x7f397c14e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55324/0 (socket says 172.21.15.150:55324) 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f397ad9d640 1 -- 172.21.15.150:0/2860032807 learned_addr learned my addr 172.21.15.150:0/2860032807 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f397b59e640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f397c152cf0 0x7f397c149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f397a59c640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f397c0c42e0 0x7f397c0c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f397ad9d640 1 -- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f397c0c42e0 msgr2=0x7f397c0c1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f397ad9d640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f397c0c42e0 0x7f397c0c1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:52.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f397ad9d640 1 -- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f397c152cf0 msgr2=0x7f397c149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:52.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f397ad9d640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f397c152cf0 0x7f397c149510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:52.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f397ad9d640 1 -- 172.21.15.150:0/2860032807 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f397c0a9670 con 0x7f397c0a7350 2024-09-17T04:51:52.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f397ad9d640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f397c0a7350 0x7f397c14e490 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f396c00aa10 tx=0x7f396c005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:52.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.146+0000 7f39637fe640 1 -- 172.21.15.150:0/2860032807 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396c0be070 con 0x7f397c0a7350 2024-09-17T04:51:52.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.150+0000 7f39637fe640 1 -- 172.21.15.150:0/2860032807 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f396c0036f0 con 0x7f397c0a7350 2024-09-17T04:51:52.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.150+0000 7f39810db640 1 -- 172.21.15.150:0/2860032807 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f397c14a520 con 0x7f397c0a7350 2024-09-17T04:51:52.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.150+0000 7f39637fe640 1 -- 172.21.15.150:0/2860032807 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396c0b3040 con 0x7f397c0a7350 2024-09-17T04:51:52.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.150+0000 7f39810db640 1 -- 172.21.15.150:0/2860032807 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f397c14aae0 con 0x7f397c0a7350 2024-09-17T04:51:52.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.150+0000 7f39810db640 1 -- 172.21.15.150:0/2860032807 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f397c150900 con 0x7f397c0a7350 2024-09-17T04:51:52.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.150+0000 7f39637fe640 1 -- 172.21.15.150:0/2860032807 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f396c0c1070 con 0x7f397c0a7350 2024-09-17T04:51:52.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.150+0000 7f39637fe640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3968073200 0x7f3968075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:51:52.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.150+0000 7f39637fe640 1 -- 172.21.15.150:0/2860032807 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f396c12f8c0 con 0x7f397c0a7350 2024-09-17T04:51:52.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.150+0000 7f397a59c640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3968073200 0x7f3968075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:51:52.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.150+0000 7f397a59c640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3968073200 0x7f3968075710 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3970007fd0 tx=0x7f3970009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:51:52.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.154+0000 7f39637fe640 1 -- 172.21.15.150:0/2860032807 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f396c0c8070 con 0x7f397c0a7350 2024-09-17T04:51:52.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.406+0000 7f39810db640 1 -- 172.21.15.150:0/2860032807 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f397c15c8e0 con 0x7f397c0a7350 2024-09-17T04:51:52.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.406+0000 7f39637fe640 1 -- 172.21.15.150:0/2860032807 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6534 (secure 0 0 0) 0x7f396c017720 con 0x7f397c0a7350 2024-09-17T04:51:52.408 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:51:52.408 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"btime":"2024-09-17T04:51:50:057989+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:51:45.122582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5106,"qdb_cluster":[5106]},"id":2}]} 2024-09-17T04:51:52.408 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 26 2024-09-17T04:51:52.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.410+0000 7f39810db640 1 -- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3968073200 msgr2=0x7f3968075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:52.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.410+0000 7f39810db640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3968073200 0x7f3968075710 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3970007fd0 tx=0x7f3970009040 comp rx=0 tx=0).stop 2024-09-17T04:51:52.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.410+0000 7f39810db640 1 -- 172.21.15.150:0/2860032807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f397c0a7350 msgr2=0x7f397c14e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:51:52.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.410+0000 7f39810db640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f397c0a7350 0x7f397c14e490 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f396c00aa10 tx=0x7f396c005760 comp rx=0 tx=0).stop 2024-09-17T04:51:52.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.410+0000 7f39810db640 1 -- 172.21.15.150:0/2860032807 shutdown_connections 2024-09-17T04:51:52.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.410+0000 7f39810db640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3968073200 0x7f3968075710 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:52.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.410+0000 7f39810db640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f397c152cf0 0x7f397c149510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:52.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.410+0000 7f39810db640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f397c0c42e0 0x7f397c0c1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:52.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.410+0000 7f39810db640 1 --2- 172.21.15.150:0/2860032807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f397c0a7350 0x7f397c14e490 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:51:52.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.410+0000 7f39810db640 1 -- 172.21.15.150:0/2860032807 >> 172.21.15.150:0/2860032807 conn(0x7f397c0c4a00 msgr2=0x7f397c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:51:52.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.410+0000 7f39810db640 1 -- 172.21.15.150:0/2860032807 shutdown_connections 2024-09-17T04:51:52.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:51:52.410+0000 7f39810db640 1 -- 172.21.15.150:0/2860032807 wait complete. 2024-09-17T04:51:52.432 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-17T04:51:52.432 INFO:tasks.mds_thrash.fs.[b]:waiting for 51 secs before thrashing 2024-09-17T04:51:58.507 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:51:58.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:52:29.012 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:52:29.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:52:43.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:52:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.601+0000 7faeb0019640 1 -- 172.21.15.150:0/550065507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faea80c4180 msgr2=0x7faea81542c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.601+0000 7faeb0019640 1 --2- 172.21.15.150:0/550065507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faea80c4180 0x7faea81542c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7faea8070e50 tx=0x7faea401c7a0 comp rx=0 tx=0).stop 2024-09-17T04:52:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.601+0000 7faeb0019640 1 -- 172.21.15.150:0/550065507 shutdown_connections 2024-09-17T04:52:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.601+0000 7faeb0019640 1 --2- 172.21.15.150:0/550065507 >> v2:172.21.15.176:3300/0 conn(0x7faea80aa9b0 0x7faea80aadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.601+0000 7faeb0019640 1 --2- 172.21.15.150:0/550065507 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faea8150e90 0x7faea8156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.601+0000 7faeb0019640 1 --2- 172.21.15.150:0/550065507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faea80c4180 0x7faea81542c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.601+0000 7faeb0019640 1 -- 172.21.15.150:0/550065507 >> 172.21.15.150:0/550065507 conn(0x7faea80c5ae0 msgr2=0x7faea8073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:43.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.601+0000 7faeb0019640 1 -- 172.21.15.150:0/550065507 shutdown_connections 2024-09-17T04:52:43.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.601+0000 7faeb0019640 1 -- 172.21.15.150:0/550065507 wait complete. 2024-09-17T04:52:43.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeb0019640 1 Processor -- start 2024-09-17T04:52:43.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeb0019640 1 -- start start 2024-09-17T04:52:43.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeb0019640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faea80aa9b0 0x7faea815c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:43.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeb0019640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faea80c4180 0x7faea80a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:43.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeb0019640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faea8150e90 0x7faea8157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:43.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeb0019640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7faea8157420 con 0x7faea8150e90 2024-09-17T04:52:43.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeb0019640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7faea80adbc0 con 0x7faea80c4180 2024-09-17T04:52:43.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeb0019640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faea806e860 con 0x7faea80aa9b0 2024-09-17T04:52:43.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeae5b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faea8150e90 0x7faea8157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:43.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeae5b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faea8150e90 0x7faea8157e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52024/0 (socket says 172.21.15.150:52024) 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeae5b6640 1 -- 172.21.15.150:0/1765747206 learned_addr learned my addr 172.21.15.150:0/1765747206 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeaddb5640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faea80aa9b0 0x7faea815c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faead5b4640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faea80c4180 0x7faea80a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeae5b6640 1 -- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faea80aa9b0 msgr2=0x7faea815c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeae5b6640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faea80aa9b0 0x7faea815c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeae5b6640 1 -- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faea80c4180 msgr2=0x7faea80a9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeae5b6640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faea80c4180 0x7faea80a9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeae5b6640 1 -- 172.21.15.150:0/1765747206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea81f17e0 con 0x7faea8150e90 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeae5b6640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faea8150e90 0x7faea8157e70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7faea000b440 tx=0x7faea000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7fae9a7fc640 1 -- 172.21.15.150:0/1765747206 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea0015020 con 0x7faea8150e90 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeb0019640 1 -- 172.21.15.150:0/1765747206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea81f1ac0 con 0x7faea8150e90 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7fae9a7fc640 1 -- 172.21.15.150:0/1765747206 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faea0011940 con 0x7faea8150e90 2024-09-17T04:52:43.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7fae9a7fc640 1 -- 172.21.15.150:0/1765747206 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea0011d90 con 0x7faea8150e90 2024-09-17T04:52:43.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.605+0000 7faeb0019640 1 -- 172.21.15.150:0/1765747206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea81588d0 con 0x7faea8150e90 2024-09-17T04:52:43.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.609+0000 7faeb0019640 1 -- 172.21.15.150:0/1765747206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea8158b30 con 0x7faea8150e90 2024-09-17T04:52:43.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.609+0000 7fae9a7fc640 1 -- 172.21.15.150:0/1765747206 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7faea0018020 con 0x7faea8150e90 2024-09-17T04:52:43.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.609+0000 7fae9a7fc640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae8c073200 0x7fae8c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:43.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.609+0000 7fae9a7fc640 1 -- 172.21.15.150:0/1765747206 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7faea0094580 con 0x7faea8150e90 2024-09-17T04:52:43.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.609+0000 7faeaddb5640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae8c073200 0x7fae8c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:43.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.613+0000 7faeaddb5640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae8c073200 0x7fae8c075710 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faea40af000 tx=0x7faea401afe0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:43.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.617+0000 7fae9a7fc640 1 -- 172.21.15.150:0/1765747206 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7faea001f020 con 0x7faea8150e90 2024-09-17T04:52:43.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.833+0000 7faeb0019640 1 -- 172.21.15.150:0/1765747206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faea8071860 con 0x7faea8150e90 2024-09-17T04:52:43.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7fae9a7fc640 1 -- 172.21.15.150:0/1765747206 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6534 (secure 0 0 0) 0x7faea005f1d0 con 0x7faea8150e90 2024-09-17T04:52:43.837 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:52:43.837 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"btime":"2024-09-17T04:51:50:057989+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:51:45.122582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5106,"qdb_cluster":[5106]},"id":2}]} 2024-09-17T04:52:43.837 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 26 2024-09-17T04:52:43.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7faeb0019640 1 -- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae8c073200 msgr2=0x7fae8c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:43.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7faeb0019640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae8c073200 0x7fae8c075710 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faea40af000 tx=0x7faea401afe0 comp rx=0 tx=0).stop 2024-09-17T04:52:43.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7faeb0019640 1 -- 172.21.15.150:0/1765747206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faea8150e90 msgr2=0x7faea8157e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:43.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7faeb0019640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faea8150e90 0x7faea8157e70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7faea000b440 tx=0x7faea000b900 comp rx=0 tx=0).stop 2024-09-17T04:52:43.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7faeb0019640 1 -- 172.21.15.150:0/1765747206 shutdown_connections 2024-09-17T04:52:43.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7faeb0019640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae8c073200 0x7fae8c075710 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:43.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7faeb0019640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faea80c4180 0x7faea80a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:43.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7faeb0019640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faea8150e90 0x7faea8157e70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:43.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7faeb0019640 1 --2- 172.21.15.150:0/1765747206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faea80aa9b0 0x7faea815c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:43.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7faeb0019640 1 -- 172.21.15.150:0/1765747206 >> 172.21.15.150:0/1765747206 conn(0x7faea80c5ae0 msgr2=0x7faea8150300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:43.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7faeb0019640 1 -- 172.21.15.150:0/1765747206 shutdown_connections 2024-09-17T04:52:43.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:43.837+0000 7faeb0019640 1 -- 172.21.15.150:0/1765747206 wait complete. 2024-09-17T04:52:43.861 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-17T04:52:43.861 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:52:43.861 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:52:43.875 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:52:43.875 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:52:44.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.085+0000 7f954b866640 1 -- 172.21.15.150:0/465233214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9530004810 msgr2=0x7f9530004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:44.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.085+0000 7f954b866640 1 --2- 172.21.15.150:0/465233214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9530004810 0x7f9530004c20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f95340016d0 tx=0x7f953401c730 comp rx=0 tx=0).stop 2024-09-17T04:52:44.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.085+0000 7f954b866640 1 -- 172.21.15.150:0/465233214 shutdown_connections 2024-09-17T04:52:44.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.085+0000 7f954b866640 1 --2- 172.21.15.150:0/465233214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9530004810 0x7f9530004c20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:44.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.085+0000 7f954b866640 1 -- 172.21.15.150:0/465233214 >> 172.21.15.150:0/465233214 conn(0x7f95440aab60 msgr2=0x7f95440aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:44.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.085+0000 7f954b866640 1 -- 172.21.15.150:0/465233214 shutdown_connections 2024-09-17T04:52:44.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.085+0000 7f954b866640 1 -- 172.21.15.150:0/465233214 wait complete. 2024-09-17T04:52:44.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.085+0000 7f954b866640 1 Processor -- start 2024-09-17T04:52:44.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.085+0000 7f954b866640 1 -- start start 2024-09-17T04:52:44.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f954b866640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9530004810 0x7f954414c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:44.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f954b866640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95441df7c0 0x7f95441e6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:44.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f954b866640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9544154fe0 0x7f95441e6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:44.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f954b866640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f95441544c0 con 0x7f9530004810 2024-09-17T04:52:44.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f954b866640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9544072500 con 0x7f95441df7c0 2024-09-17T04:52:44.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f954b866640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f95440c75a0 con 0x7f9544154fe0 2024-09-17T04:52:44.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9549602640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9530004810 0x7f954414c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:44.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9549602640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9530004810 0x7f954414c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52034/0 (socket says 172.21.15.150:52034) 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9549602640 1 -- 172.21.15.150:0/2681438924 learned_addr learned my addr 172.21.15.150:0/2681438924 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9548e01640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95441df7c0 0x7f95441e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9549e03640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9544154fe0 0x7f95441e6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9549602640 1 -- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9544154fe0 msgr2=0x7f95441e6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9549602640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9544154fe0 0x7f95441e6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9549602640 1 -- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95441df7c0 msgr2=0x7f95441e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9549602640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95441df7c0 0x7f95441e6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9549602640 1 -- 172.21.15.150:0/2681438924 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95441ddce0 con 0x7f9530004810 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9549e03640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9544154fe0 0x7f95441e6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9549602640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9530004810 0x7f954414c180 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9540001820 tx=0x7f954000e770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9539ffb640 1 -- 172.21.15.150:0/2681438924 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9540012850 con 0x7f9530004810 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f954b866640 1 -- 172.21.15.150:0/2681438924 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95441ddfc0 con 0x7f9530004810 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9539ffb640 1 -- 172.21.15.150:0/2681438924 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9540012b20 con 0x7f9530004810 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f9539ffb640 1 -- 172.21.15.150:0/2681438924 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954001a400 con 0x7f9530004810 2024-09-17T04:52:44.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f954b866640 1 -- 172.21.15.150:0/2681438924 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95441de580 con 0x7f9530004810 2024-09-17T04:52:44.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.089+0000 7f954b866640 1 -- 172.21.15.150:0/2681438924 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95440c7ab0 con 0x7f9530004810 2024-09-17T04:52:44.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.093+0000 7f9539ffb640 1 -- 172.21.15.150:0/2681438924 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f954000d030 con 0x7f9530004810 2024-09-17T04:52:44.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.093+0000 7f9539ffb640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9528073230 0x7f9528075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:44.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.093+0000 7f9539ffb640 1 -- 172.21.15.150:0/2681438924 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f9540097e00 con 0x7f9530004810 2024-09-17T04:52:44.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.093+0000 7f9548e01640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9528073230 0x7f9528075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:44.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.097+0000 7f9548e01640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9528073230 0x7f9528075740 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f95340016a0 tx=0x7f95340a9040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:44.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.097+0000 7f9539ffb640 1 -- 172.21.15.150:0/2681438924 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9540028020 con 0x7f9530004810 2024-09-17T04:52:44.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.325+0000 7f954b866640 1 -- 172.21.15.150:0/2681438924 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f954406d140 con 0x7f9530004810 2024-09-17T04:52:44.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.325+0000 7f9539ffb640 1 -- 172.21.15.150:0/2681438924 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6534 (secure 0 0 0) 0x7f9540016030 con 0x7f9530004810 2024-09-17T04:52:44.327 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:52:44.327 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"btime":"2024-09-17T04:51:50:057989+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:51:45.122582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5106,"qdb_cluster":[5106]},"id":2}]} 2024-09-17T04:52:44.327 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 26 2024-09-17T04:52:44.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.329+0000 7f954b866640 1 -- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9528073230 msgr2=0x7f9528075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:44.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.329+0000 7f954b866640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9528073230 0x7f9528075740 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f95340016a0 tx=0x7f95340a9040 comp rx=0 tx=0).stop 2024-09-17T04:52:44.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.329+0000 7f954b866640 1 -- 172.21.15.150:0/2681438924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9530004810 msgr2=0x7f954414c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:44.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.329+0000 7f954b866640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9530004810 0x7f954414c180 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9540001820 tx=0x7f954000e770 comp rx=0 tx=0).stop 2024-09-17T04:52:44.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.329+0000 7f954b866640 1 -- 172.21.15.150:0/2681438924 shutdown_connections 2024-09-17T04:52:44.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.329+0000 7f954b866640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9528073230 0x7f9528075740 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:44.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.329+0000 7f954b866640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f95441df7c0 0x7f95441e6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:44.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.329+0000 7f954b866640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9544154fe0 0x7f95441e6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:44.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.329+0000 7f954b866640 1 --2- 172.21.15.150:0/2681438924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9530004810 0x7f954414c180 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:44.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.329+0000 7f954b866640 1 -- 172.21.15.150:0/2681438924 >> 172.21.15.150:0/2681438924 conn(0x7f95440aab60 msgr2=0x7f95440c82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:44.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.329+0000 7f954b866640 1 -- 172.21.15.150:0/2681438924 shutdown_connections 2024-09-17T04:52:44.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:44.329+0000 7f954b866640 1 -- 172.21.15.150:0/2681438924 wait complete. 2024-09-17T04:52:44.352 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:52:46.353 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:52:46.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.533+0000 7f5a95df4640 1 -- 172.21.15.150:0/50787224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a900c8230 msgr2=0x7f5a90157190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:46.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.533+0000 7f5a95df4640 1 --2- 172.21.15.150:0/50787224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a900c8230 0x7f5a90157190 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5a900c6cd0 tx=0x7f5a8401c800 comp rx=0 tx=0).stop 2024-09-17T04:52:46.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.533+0000 7f5a95df4640 1 -- 172.21.15.150:0/50787224 shutdown_connections 2024-09-17T04:52:46.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.533+0000 7f5a95df4640 1 --2- 172.21.15.150:0/50787224 >> v2:172.21.15.176:3300/0 conn(0x7f5a901576d0 0x7f5a900c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:46.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.533+0000 7f5a95df4640 1 --2- 172.21.15.150:0/50787224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a900c8230 0x7f5a90157190 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:46.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.533+0000 7f5a95df4640 1 --2- 172.21.15.150:0/50787224 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5a900c5390 0x7f5a90074960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:46.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.533+0000 7f5a95df4640 1 -- 172.21.15.150:0/50787224 >> 172.21.15.150:0/50787224 conn(0x7f5a9006e360 msgr2=0x7f5a9006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:46.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.533+0000 7f5a95df4640 1 -- 172.21.15.150:0/50787224 shutdown_connections 2024-09-17T04:52:46.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.533+0000 7f5a95df4640 1 -- 172.21.15.150:0/50787224 wait complete. 2024-09-17T04:52:46.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a95df4640 1 Processor -- start 2024-09-17T04:52:46.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a95df4640 1 -- start start 2024-09-17T04:52:46.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a95df4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5a900c5390 0x7f5a9014a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:46.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a95df4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a900c8230 0x7f5a90153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:46.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a95df4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5a901576d0 0x7f5a90151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:46.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a95df4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9006d1e0 con 0x7f5a900c8230 2024-09-17T04:52:46.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a95df4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9006d060 con 0x7f5a901576d0 2024-09-17T04:52:46.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a95df4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5a9006d360 con 0x7f5a900c5390 2024-09-17T04:52:46.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a8effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a900c8230 0x7f5a90153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:46.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a8effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a900c8230 0x7f5a90153dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52042/0 (socket says 172.21.15.150:52042) 2024-09-17T04:52:46.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a8effd640 1 -- 172.21.15.150:0/2047547119 learned_addr learned my addr 172.21.15.150:0/2047547119 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:52:46.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a8ffff640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5a901576d0 0x7f5a90151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:46.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a8f7fe640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5a900c5390 0x7f5a9014a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:46.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a8effd640 1 -- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5a900c5390 msgr2=0x7f5a9014a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:46.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a8effd640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5a900c5390 0x7f5a9014a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:46.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a8effd640 1 -- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5a901576d0 msgr2=0x7f5a90151e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:46.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a8effd640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5a901576d0 0x7f5a90151e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:46.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a8effd640 1 -- 172.21.15.150:0/2047547119 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a90150190 con 0x7f5a900c8230 2024-09-17T04:52:46.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a8ffff640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5a901576d0 0x7f5a90151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:52:46.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a8effd640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a900c8230 0x7f5a90153dc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5a8401aeb0 tx=0x7f5a840a5630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:46.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a77fff640 1 -- 172.21.15.150:0/2047547119 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a840ba020 con 0x7f5a900c8230 2024-09-17T04:52:46.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a77fff640 1 -- 172.21.15.150:0/2047547119 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a840043b0 con 0x7f5a900c8230 2024-09-17T04:52:46.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a95df4640 1 -- 172.21.15.150:0/2047547119 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a9014f860 con 0x7f5a900c8230 2024-09-17T04:52:46.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.537+0000 7f5a77fff640 1 -- 172.21.15.150:0/2047547119 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a840b2040 con 0x7f5a900c8230 2024-09-17T04:52:46.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.541+0000 7f5a95df4640 1 -- 172.21.15.150:0/2047547119 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a9014fe20 con 0x7f5a900c8230 2024-09-17T04:52:46.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.541+0000 7f5a95df4640 1 -- 172.21.15.150:0/2047547119 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a9014e910 con 0x7f5a900c8230 2024-09-17T04:52:46.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.541+0000 7f5a77fff640 1 -- 172.21.15.150:0/2047547119 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5a84005290 con 0x7f5a900c8230 2024-09-17T04:52:46.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.541+0000 7f5a77fff640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5a7c073230 0x7f5a7c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:46.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.541+0000 7f5a77fff640 1 -- 172.21.15.150:0/2047547119 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f5a8412e340 con 0x7f5a900c8230 2024-09-17T04:52:46.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.541+0000 7f5a8f7fe640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5a7c073230 0x7f5a7c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:46.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.541+0000 7f5a8f7fe640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5a7c073230 0x7f5a7c075740 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5a80005fd0 tx=0x7f5a80005d80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:46.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.545+0000 7f5a77fff640 1 -- 172.21.15.150:0/2047547119 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5a840164b0 con 0x7f5a900c8230 2024-09-17T04:52:46.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.857+0000 7f5a95df4640 1 -- 172.21.15.150:0/2047547119 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a90157190 con 0x7f5a900c8230 2024-09-17T04:52:46.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.857+0000 7f5a77fff640 1 -- 172.21.15.150:0/2047547119 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6534 (secure 0 0 0) 0x7f5a840f8f90 con 0x7f5a900c8230 2024-09-17T04:52:46.858 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:52:46.858 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"btime":"2024-09-17T04:51:50:057989+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:51:45.122582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5106,"qdb_cluster":[5106]},"id":2}]} 2024-09-17T04:52:46.858 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 26 2024-09-17T04:52:46.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.861+0000 7f5a95df4640 1 -- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5a7c073230 msgr2=0x7f5a7c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:46.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.861+0000 7f5a95df4640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5a7c073230 0x7f5a7c075740 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5a80005fd0 tx=0x7f5a80005d80 comp rx=0 tx=0).stop 2024-09-17T04:52:46.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.861+0000 7f5a95df4640 1 -- 172.21.15.150:0/2047547119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a900c8230 msgr2=0x7f5a90153dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:46.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.861+0000 7f5a95df4640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a900c8230 0x7f5a90153dc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5a8401aeb0 tx=0x7f5a840a5630 comp rx=0 tx=0).stop 2024-09-17T04:52:46.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.861+0000 7f5a95df4640 1 -- 172.21.15.150:0/2047547119 shutdown_connections 2024-09-17T04:52:46.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.861+0000 7f5a95df4640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5a7c073230 0x7f5a7c075740 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:46.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.861+0000 7f5a95df4640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5a901576d0 0x7f5a90151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:46.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.861+0000 7f5a95df4640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5a900c8230 0x7f5a90153dc0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:46.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.861+0000 7f5a95df4640 1 --2- 172.21.15.150:0/2047547119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5a900c5390 0x7f5a9014a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:46.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.861+0000 7f5a95df4640 1 -- 172.21.15.150:0/2047547119 >> 172.21.15.150:0/2047547119 conn(0x7f5a9006e360 msgr2=0x7f5a900ae230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:46.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.861+0000 7f5a95df4640 1 -- 172.21.15.150:0/2047547119 shutdown_connections 2024-09-17T04:52:46.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:46.861+0000 7f5a95df4640 1 -- 172.21.15.150:0/2047547119 wait complete. 2024-09-17T04:52:46.880 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:52:48.881 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:52:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.069+0000 7fd04682a640 1 -- 172.21.15.150:0/1224096102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0400aa9b0 msgr2=0x7fd0400aadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.069+0000 7fd04682a640 1 --2- 172.21.15.150:0/1224096102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0400aa9b0 0x7fd0400aadc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd038009bc0 tx=0x7fd03801cb10 comp rx=0 tx=0).stop 2024-09-17T04:52:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.069+0000 7fd04682a640 1 -- 172.21.15.150:0/1224096102 shutdown_connections 2024-09-17T04:52:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.069+0000 7fd04682a640 1 --2- 172.21.15.150:0/1224096102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd0400aa9b0 0x7fd0400aadc0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.069+0000 7fd04682a640 1 --2- 172.21.15.150:0/1224096102 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd04006e630 0x7fd04006fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.069+0000 7fd04682a640 1 -- 172.21.15.150:0/1224096102 >> 172.21.15.150:0/1224096102 conn(0x7fd0400c5ae0 msgr2=0x7fd040073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.069+0000 7fd04682a640 1 -- 172.21.15.150:0/1224096102 shutdown_connections 2024-09-17T04:52:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.069+0000 7fd04682a640 1 -- 172.21.15.150:0/1224096102 wait complete. 2024-09-17T04:52:49.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd04682a640 1 Processor -- start 2024-09-17T04:52:49.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd04682a640 1 -- start start 2024-09-17T04:52:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd04682a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd04006e630 0x7fd04014cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd04682a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0400aa9b0 0x7fd0400c7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd04682a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd040150a90 0x7fd0400c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd04682a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd040151020 con 0x7fd040150a90 2024-09-17T04:52:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd04682a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd040150ea0 con 0x7fd0400aa9b0 2024-09-17T04:52:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd04682a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd040070610 con 0x7fd04006e630 2024-09-17T04:52:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd044dc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd040150a90 0x7fd0400c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd044dc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd040150a90 0x7fd0400c1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52068/0 (socket says 172.21.15.150:52068) 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd044dc7640 1 -- 172.21.15.150:0/4280667280 learned_addr learned my addr 172.21.15.150:0/4280667280 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd03ffff640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd04006e630 0x7fd04014cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd037fff640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0400aa9b0 0x7fd0400c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd044dc7640 1 -- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd04006e630 msgr2=0x7fd04014cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd044dc7640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd04006e630 0x7fd04014cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd044dc7640 1 -- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0400aa9b0 msgr2=0x7fd0400c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd044dc7640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0400aa9b0 0x7fd0400c7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.073+0000 7fd044dc7640 1 -- 172.21.15.150:0/4280667280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0400c6210 con 0x7fd040150a90 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.077+0000 7fd044dc7640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd040150a90 0x7fd0400c1b40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd038009b10 tx=0x7fd038003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.077+0000 7fd03d7fa640 1 -- 172.21.15.150:0/4280667280 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0380bb070 con 0x7fd040150a90 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.077+0000 7fd03d7fa640 1 -- 172.21.15.150:0/4280667280 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd038004e00 con 0x7fd040150a90 2024-09-17T04:52:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.077+0000 7fd03d7fa640 1 -- 172.21.15.150:0/4280667280 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0380b2040 con 0x7fd040150a90 2024-09-17T04:52:49.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.077+0000 7fd04682a640 1 -- 172.21.15.150:0/4280667280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd04014afb0 con 0x7fd040150a90 2024-09-17T04:52:49.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.077+0000 7fd04682a640 1 -- 172.21.15.150:0/4280667280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd04014d840 con 0x7fd040150a90 2024-09-17T04:52:49.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.077+0000 7fd04682a640 1 -- 172.21.15.150:0/4280667280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0401ee800 con 0x7fd040150a90 2024-09-17T04:52:49.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.077+0000 7fd03d7fa640 1 -- 172.21.15.150:0/4280667280 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd038004220 con 0x7fd040150a90 2024-09-17T04:52:49.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.077+0000 7fd03d7fa640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd020073200 0x7fd020075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:49.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.077+0000 7fd03d7fa640 1 -- 172.21.15.150:0/4280667280 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7fd03812e600 con 0x7fd040150a90 2024-09-17T04:52:49.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.077+0000 7fd03ffff640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd020073200 0x7fd020075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:49.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.081+0000 7fd03ffff640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd020073200 0x7fd020075710 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd0300058d0 tx=0x7fd030005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:49.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.085+0000 7fd03d7fa640 1 -- 172.21.15.150:0/4280667280 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd0401ee800 con 0x7fd040150a90 2024-09-17T04:52:49.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.313+0000 7fd04682a640 1 -- 172.21.15.150:0/4280667280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd04006fee0 con 0x7fd040150a90 2024-09-17T04:52:49.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.313+0000 7fd03d7fa640 1 -- 172.21.15.150:0/4280667280 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6534 (secure 0 0 0) 0x7fd038016790 con 0x7fd040150a90 2024-09-17T04:52:49.315 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:52:49.315 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"btime":"2024-09-17T04:51:50:057989+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:51:45.122582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5106,"qdb_cluster":[5106]},"id":2}]} 2024-09-17T04:52:49.315 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 26 2024-09-17T04:52:49.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.317+0000 7fd04682a640 1 -- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd020073200 msgr2=0x7fd020075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:49.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.317+0000 7fd04682a640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd020073200 0x7fd020075710 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd0300058d0 tx=0x7fd030005860 comp rx=0 tx=0).stop 2024-09-17T04:52:49.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.317+0000 7fd04682a640 1 -- 172.21.15.150:0/4280667280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd040150a90 msgr2=0x7fd0400c1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:49.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.317+0000 7fd04682a640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd040150a90 0x7fd0400c1b40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd038009b10 tx=0x7fd038003f80 comp rx=0 tx=0).stop 2024-09-17T04:52:49.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.317+0000 7fd04682a640 1 -- 172.21.15.150:0/4280667280 shutdown_connections 2024-09-17T04:52:49.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.317+0000 7fd04682a640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd020073200 0x7fd020075710 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:49.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.317+0000 7fd04682a640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0400aa9b0 0x7fd0400c7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:49.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.317+0000 7fd04682a640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd040150a90 0x7fd0400c1b40 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:49.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.317+0000 7fd04682a640 1 --2- 172.21.15.150:0/4280667280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd04006e630 0x7fd04014cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:49.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.317+0000 7fd04682a640 1 -- 172.21.15.150:0/4280667280 >> 172.21.15.150:0/4280667280 conn(0x7fd0400c5ae0 msgr2=0x7fd040073c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:49.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.317+0000 7fd04682a640 1 -- 172.21.15.150:0/4280667280 shutdown_connections 2024-09-17T04:52:49.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:49.317+0000 7fd04682a640 1 -- 172.21.15.150:0/4280667280 wait complete. 2024-09-17T04:52:49.337 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:52:51.338 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:52:51.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.509+0000 7fabcfb68640 1 -- 172.21.15.150:0/1403885503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc8154150 msgr2=0x7fabc80a8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:51.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.509+0000 7fabcfb68640 1 --2- 172.21.15.150:0/1403885503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc8154150 0x7fabc80a8340 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009930 tx=0x7fabc401c850 comp rx=0 tx=0).stop 2024-09-17T04:52:51.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.509+0000 7fabcfb68640 1 -- 172.21.15.150:0/1403885503 shutdown_connections 2024-09-17T04:52:51.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.509+0000 7fabcfb68640 1 --2- 172.21.15.150:0/1403885503 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc806ce50 0x7fabc806d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:51.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.509+0000 7fabcfb68640 1 --2- 172.21.15.150:0/1403885503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc8154150 0x7fabc80a8340 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:51.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.509+0000 7fabcfb68640 1 -- 172.21.15.150:0/1403885503 >> 172.21.15.150:0/1403885503 conn(0x7fabc80aab60 msgr2=0x7fabc80aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:51.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.509+0000 7fabcfb68640 1 -- 172.21.15.150:0/1403885503 shutdown_connections 2024-09-17T04:52:51.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.509+0000 7fabcfb68640 1 -- 172.21.15.150:0/1403885503 wait complete. 2024-09-17T04:52:51.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.513+0000 7fabcfb68640 1 Processor -- start 2024-09-17T04:52:51.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.513+0000 7fabcfb68640 1 -- start start 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.513+0000 7fabcfb68640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc806ce50 0x7fabc8145c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.513+0000 7fabcfb68640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc80c7140 0x7fabc81f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.513+0000 7fabcfb68640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc8154150 0x7fabc814f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.513+0000 7fabcfb68640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fabc80c7de0 con 0x7fabc806ce50 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.513+0000 7fabcfb68640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fabc80c7c60 con 0x7fabc80c7140 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.513+0000 7fabcfb68640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fabc8163a40 con 0x7fabc8154150 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.513+0000 7fabcd904640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc806ce50 0x7fabc8145c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.513+0000 7fabcd904640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc806ce50 0x7fabc8145c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37532/0 (socket says 172.21.15.150:37532) 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.513+0000 7fabcd904640 1 -- 172.21.15.150:0/661512256 learned_addr learned my addr 172.21.15.150:0/661512256 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabce105640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc8154150 0x7fabc814f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabcd103640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc80c7140 0x7fabc81f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabcd904640 1 -- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc8154150 msgr2=0x7fabc814f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabcd904640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc8154150 0x7fabc814f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:51.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabcd904640 1 -- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc80c7140 msgr2=0x7fabc81f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:51.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabcd904640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc80c7140 0x7fabc81f34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:51.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabcd904640 1 -- 172.21.15.150:0/661512256 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc81f3a00 con 0x7fabc806ce50 2024-09-17T04:52:51.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabcd904640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc806ce50 0x7fabc8145c60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fabc401cd30 tx=0x7fabc40a65d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:51.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabbe7fc640 1 -- 172.21.15.150:0/661512256 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc40ac070 con 0x7fabc806ce50 2024-09-17T04:52:51.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabbe7fc640 1 -- 172.21.15.150:0/661512256 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabc40054d0 con 0x7fabc806ce50 2024-09-17T04:52:51.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabcfb68640 1 -- 172.21.15.150:0/661512256 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc80c61d0 con 0x7fabc806ce50 2024-09-17T04:52:51.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabbe7fc640 1 -- 172.21.15.150:0/661512256 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc4004290 con 0x7fabc806ce50 2024-09-17T04:52:51.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabcfb68640 1 -- 172.21.15.150:0/661512256 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc80a9fb0 con 0x7fabc806ce50 2024-09-17T04:52:51.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fababfff640 1 -- 172.21.15.150:0/661512256 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faba40052a0 con 0x7fabc806ce50 2024-09-17T04:52:51.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabbe7fc640 1 -- 172.21.15.150:0/661512256 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fabc40af070 con 0x7fabc806ce50 2024-09-17T04:52:51.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabbe7fc640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabac073200 0x7fabac075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:51.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabbe7fc640 1 -- 172.21.15.150:0/661512256 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7fabc412e800 con 0x7fabc806ce50 2024-09-17T04:52:51.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.517+0000 7fabcd103640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabac073200 0x7fabac075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:51.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.521+0000 7fabcd103640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabac073200 0x7fabac075710 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fabc81606b0 tx=0x7fabb8005db0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:51.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.525+0000 7fabbe7fc640 1 -- 172.21.15.150:0/661512256 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fabc40164d0 con 0x7fabc806ce50 2024-09-17T04:52:51.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.757+0000 7fababfff640 1 -- 172.21.15.150:0/661512256 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faba4005860 con 0x7fabc806ce50 2024-09-17T04:52:51.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.757+0000 7fabbe7fc640 1 -- 172.21.15.150:0/661512256 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6534 (secure 0 0 0) 0x7fabc40f93c0 con 0x7fabc806ce50 2024-09-17T04:52:51.759 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:52:51.759 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"btime":"2024-09-17T04:51:50:057989+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:51:45.122582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5106,"qdb_cluster":[5106]},"id":2}]} 2024-09-17T04:52:51.759 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 26 2024-09-17T04:52:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.761+0000 7fababfff640 1 -- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabac073200 msgr2=0x7fabac075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.761+0000 7fababfff640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabac073200 0x7fabac075710 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fabc81606b0 tx=0x7fabb8005db0 comp rx=0 tx=0).stop 2024-09-17T04:52:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.761+0000 7fababfff640 1 -- 172.21.15.150:0/661512256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc806ce50 msgr2=0x7fabc8145c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.761+0000 7fababfff640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc806ce50 0x7fabc8145c60 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fabc401cd30 tx=0x7fabc40a65d0 comp rx=0 tx=0).stop 2024-09-17T04:52:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.761+0000 7fababfff640 1 -- 172.21.15.150:0/661512256 shutdown_connections 2024-09-17T04:52:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.761+0000 7fababfff640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabac073200 0x7fabac075710 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.761+0000 7fababfff640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc80c7140 0x7fabc81f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:51.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.761+0000 7fababfff640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc8154150 0x7fabc814f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:51.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.761+0000 7fababfff640 1 --2- 172.21.15.150:0/661512256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc806ce50 0x7fabc8145c60 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:51.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.761+0000 7fababfff640 1 -- 172.21.15.150:0/661512256 >> 172.21.15.150:0/661512256 conn(0x7fabc80aab60 msgr2=0x7fabc80c7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:51.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.761+0000 7fababfff640 1 -- 172.21.15.150:0/661512256 shutdown_connections 2024-09-17T04:52:51.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:51.761+0000 7fababfff640 1 -- 172.21.15.150:0/661512256 wait complete. 2024-09-17T04:52:51.785 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:52:53.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:52:53.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.961+0000 7f107cbd7640 1 -- 172.21.15.150:0/2144094061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10780c51b0 msgr2=0x7f1078151e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:53.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.961+0000 7f107cbd7640 1 --2- 172.21.15.150:0/2144094061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10780c51b0 0x7f1078151e40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1078152380 tx=0x7f106801c740 comp rx=0 tx=0).stop 2024-09-17T04:52:53.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.961+0000 7f107cbd7640 1 -- 172.21.15.150:0/2144094061 shutdown_connections 2024-09-17T04:52:53.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.961+0000 7f107cbd7640 1 --2- 172.21.15.150:0/2144094061 >> v2:172.21.15.176:3300/0 conn(0x7f10780c88e0 0x7f10780c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:53.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.961+0000 7f107cbd7640 1 --2- 172.21.15.150:0/2144094061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10780c51b0 0x7f1078151e40 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:53.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.961+0000 7f107cbd7640 1 -- 172.21.15.150:0/2144094061 >> 172.21.15.150:0/2144094061 conn(0x7f10780aa930 msgr2=0x7f10780aad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:53.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.961+0000 7f107cbd7640 1 -- 172.21.15.150:0/2144094061 shutdown_connections 2024-09-17T04:52:53.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.961+0000 7f107cbd7640 1 -- 172.21.15.150:0/2144094061 wait complete. 2024-09-17T04:52:53.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.961+0000 7f107cbd7640 1 Processor -- start 2024-09-17T04:52:53.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.961+0000 7f107cbd7640 1 -- start start 2024-09-17T04:52:53.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f107cbd7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1078074a20 0x7f10780c2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:53.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f107cbd7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10780c51b0 0x7f10780717a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:53.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f107cbd7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10780c88e0 0x7f107814c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:53.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f107cbd7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f10781553a0 con 0x7f10780c88e0 2024-09-17T04:52:53.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f107cbd7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f10780c9230 con 0x7f10780c51b0 2024-09-17T04:52:53.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f107cbd7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1078070250 con 0x7f1078074a20 2024-09-17T04:52:53.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1077fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10780c88e0 0x7f107814c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:53.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1077fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10780c88e0 0x7f107814c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37544/0 (socket says 172.21.15.150:37544) 2024-09-17T04:52:53.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1077fff640 1 -- 172.21.15.150:0/2576739250 learned_addr learned my addr 172.21.15.150:0/2576739250 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:52:53.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f10777fe640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1078074a20 0x7f10780c2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:53.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1077fff640 1 -- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1078074a20 msgr2=0x7f10780c2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:53.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1077fff640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1078074a20 0x7f10780c2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:53.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1077fff640 1 -- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10780c51b0 msgr2=0x7f10780717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:53.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1077fff640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10780c51b0 0x7f10780717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:53.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1077fff640 1 -- 172.21.15.150:0/2576739250 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10780ad860 con 0x7f10780c88e0 2024-09-17T04:52:53.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1077fff640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10780c88e0 0x7f107814c670 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f107000e940 tx=0x7f107000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:53.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1074ff9640 1 -- 172.21.15.150:0/2576739250 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070004d30 con 0x7f10780c88e0 2024-09-17T04:52:53.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1074ff9640 1 -- 172.21.15.150:0/2576739250 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f107000cb20 con 0x7f10780c88e0 2024-09-17T04:52:53.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1074ff9640 1 -- 172.21.15.150:0/2576739250 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070005020 con 0x7f10780c88e0 2024-09-17T04:52:53.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f107cbd7640 1 -- 172.21.15.150:0/2576739250 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10781efcd0 con 0x7f10780c88e0 2024-09-17T04:52:53.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f107cbd7640 1 -- 172.21.15.150:0/2576739250 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10780721b0 con 0x7f10780c88e0 2024-09-17T04:52:53.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f107cbd7640 1 -- 172.21.15.150:0/2576739250 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1078151e40 con 0x7f10780c88e0 2024-09-17T04:52:53.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1074ff9640 1 -- 172.21.15.150:0/2576739250 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f10700051c0 con 0x7f10780c88e0 2024-09-17T04:52:53.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.965+0000 7f1074ff9640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1058073200 0x7f1058075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:53.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.969+0000 7f1074ff9640 1 -- 172.21.15.150:0/2576739250 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6838+0+0 (secure 0 0 0) 0x7f1070094e00 con 0x7f10780c88e0 2024-09-17T04:52:53.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.969+0000 7f10777fe640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1058073200 0x7f1058075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:53.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.969+0000 7f10777fe640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1058073200 0x7f1058075710 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f10680af000 tx=0x7f106801af80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:53.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:53.973+0000 7f1074ff9640 1 -- 172.21.15.150:0/2576739250 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1070016030 con 0x7f10780c88e0 2024-09-17T04:52:54.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.245+0000 7f107cbd7640 1 -- 172.21.15.150:0/2576739250 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1078074800 con 0x7f10780c88e0 2024-09-17T04:52:54.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.245+0000 7f1074ff9640 1 -- 172.21.15.150:0/2576739250 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6534 (secure 0 0 0) 0x7f107005fad0 con 0x7f10780c88e0 2024-09-17T04:52:54.246 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:52:54.246 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"btime":"2024-09-17T04:51:50:057989+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4975,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:51:45.122582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5106":{"gid":5106,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/961213876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":961213876},{"type":"v1","addr":"172.21.15.150:6833","nonce":961213876}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5106,"qdb_cluster":[5106]},"id":2}]} 2024-09-17T04:52:54.246 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 26 2024-09-17T04:52:54.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.249+0000 7f107cbd7640 1 -- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1058073200 msgr2=0x7f1058075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:54.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.249+0000 7f107cbd7640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1058073200 0x7f1058075710 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f10680af000 tx=0x7f106801af80 comp rx=0 tx=0).stop 2024-09-17T04:52:54.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.249+0000 7f107cbd7640 1 -- 172.21.15.150:0/2576739250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10780c88e0 msgr2=0x7f107814c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:54.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.249+0000 7f107cbd7640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10780c88e0 0x7f107814c670 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f107000e940 tx=0x7f107000ee00 comp rx=0 tx=0).stop 2024-09-17T04:52:54.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.249+0000 7f107cbd7640 1 -- 172.21.15.150:0/2576739250 shutdown_connections 2024-09-17T04:52:54.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.249+0000 7f107cbd7640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1058073200 0x7f1058075710 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:54.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.249+0000 7f107cbd7640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10780c51b0 0x7f10780717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:54.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.249+0000 7f107cbd7640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10780c88e0 0x7f107814c670 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:54.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.249+0000 7f107cbd7640 1 --2- 172.21.15.150:0/2576739250 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1078074a20 0x7f10780c2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:54.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.249+0000 7f107cbd7640 1 -- 172.21.15.150:0/2576739250 >> 172.21.15.150:0/2576739250 conn(0x7f10780aa930 msgr2=0x7f107806ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:54.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.249+0000 7f107cbd7640 1 -- 172.21.15.150:0/2576739250 shutdown_connections 2024-09-17T04:52:54.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:54.249+0000 7f107cbd7640 1 -- 172.21.15.150:0/2576739250 wait complete. 2024-09-17T04:52:54.274 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:52:56.276 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:52:56.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 -- 172.21.15.150:0/801033836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f730c005360 msgr2=0x7f730c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:56.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 --2- 172.21.15.150:0/801033836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f730c005360 0x7f730c00b9d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f73240719d0 tx=0x7f731c01caf0 comp rx=0 tx=0).stop 2024-09-17T04:52:56.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f7323fff640 1 -- 172.21.15.150:0/801033836 reap_dead start 2024-09-17T04:52:56.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 -- 172.21.15.150:0/801033836 shutdown_connections 2024-09-17T04:52:56.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 -- 172.21.15.150:0/801033836 >> 172.21.15.150:0/801033836 conn(0x7f732406e360 msgr2=0x7f732406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:56.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 -- 172.21.15.150:0/801033836 shutdown_connections 2024-09-17T04:52:56.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 -- 172.21.15.150:0/801033836 wait complete. 2024-09-17T04:52:56.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 Processor -- start 2024-09-17T04:52:56.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 -- start start 2024-09-17T04:52:56.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73240c2b50 0x7f7324156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f732414f910 0x7f7324151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7324151fb0 0x7f73241574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7324071120 con 0x7f732414f910 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7324070fa0 con 0x7f7324151fb0 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f732a09a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7324072db0 con 0x7f73240c2b50 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f7322ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f732414f910 0x7f7324151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f7322ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f732414f910 0x7f7324151a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37558/0 (socket says 172.21.15.150:37558) 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f7322ffd640 1 -- 172.21.15.150:0/2891856011 learned_addr learned my addr 172.21.15.150:0/2891856011 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.425+0000 7f73237fe640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73240c2b50 0x7f7324156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f7323fff640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7324151fb0 0x7f73241574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f7322ffd640 1 -- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73240c2b50 msgr2=0x7f7324156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f7322ffd640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73240c2b50 0x7f7324156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:56.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f7322ffd640 1 -- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7324151fb0 msgr2=0x7f73241574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f7322ffd640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7324151fb0 0x7f73241574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f7322ffd640 1 -- 172.21.15.150:0/2891856011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732406fc70 con 0x7f732414f910 2024-09-17T04:52:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f7322ffd640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f732414f910 0x7f7324151a70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7324074510 tx=0x7f731800da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f730bfff640 1 -- 172.21.15.150:0/2891856011 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7318017070 con 0x7f732414f910 2024-09-17T04:52:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f730bfff640 1 -- 172.21.15.150:0/2891856011 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f731800f890 con 0x7f732414f910 2024-09-17T04:52:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f732a09a640 1 -- 172.21.15.150:0/2891856011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f732406d0a0 con 0x7f732414f910 2024-09-17T04:52:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f730bfff640 1 -- 172.21.15.150:0/2891856011 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731800fce0 con 0x7f732414f910 2024-09-17T04:52:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f732a09a640 1 -- 172.21.15.150:0/2891856011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73241ed100 con 0x7f732414f910 2024-09-17T04:52:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f732a09a640 1 -- 172.21.15.150:0/2891856011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73240c3d10 con 0x7f732414f910 2024-09-17T04:52:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f730bfff640 1 -- 172.21.15.150:0/2891856011 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f731801a070 con 0x7f732414f910 2024-09-17T04:52:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f730bfff640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7310073230 0x7f7310075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f730bfff640 1 -- 172.21.15.150:0/2891856011 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f73180943a0 con 0x7f732414f910 2024-09-17T04:52:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.429+0000 7f73237fe640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7310073230 0x7f7310075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:56.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.433+0000 7f73237fe640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7310073230 0x7f7310075740 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f732406eac0 tx=0x7f7314006040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:56.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.437+0000 7f730bfff640 1 -- 172.21.15.150:0/2891856011 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f731805e9d0 con 0x7f732414f910 2024-09-17T04:52:56.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.661+0000 7f732a09a640 1 -- 172.21.15.150:0/2891856011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73241df4b0 con 0x7f732414f910 2024-09-17T04:52:56.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.661+0000 7f730bfff640 1 -- 172.21.15.150:0/2891856011 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5702 (secure 0 0 0) 0x7f7318063400 con 0x7f732414f910 2024-09-17T04:52:56.664 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:52:56.664 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":27,"btime":"2024-09-17T04:52:55:839028+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:55.839014+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:52:56.664 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 27 2024-09-17T04:52:56.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.665+0000 7f732a09a640 1 -- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7310073230 msgr2=0x7f7310075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:56.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.665+0000 7f732a09a640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7310073230 0x7f7310075740 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f732406eac0 tx=0x7f7314006040 comp rx=0 tx=0).stop 2024-09-17T04:52:56.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.665+0000 7f732a09a640 1 -- 172.21.15.150:0/2891856011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f732414f910 msgr2=0x7f7324151a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:56.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.665+0000 7f732a09a640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f732414f910 0x7f7324151a70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7324074510 tx=0x7f731800da20 comp rx=0 tx=0).stop 2024-09-17T04:52:56.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.665+0000 7f732a09a640 1 -- 172.21.15.150:0/2891856011 shutdown_connections 2024-09-17T04:52:56.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.665+0000 7f732a09a640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7310073230 0x7f7310075740 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:56.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.665+0000 7f732a09a640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7324151fb0 0x7f73241574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:56.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.665+0000 7f732a09a640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f732414f910 0x7f7324151a70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:56.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.665+0000 7f732a09a640 1 --2- 172.21.15.150:0/2891856011 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73240c2b50 0x7f7324156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:56.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.665+0000 7f732a09a640 1 -- 172.21.15.150:0/2891856011 >> 172.21.15.150:0/2891856011 conn(0x7f732406e360 msgr2=0x7f73240c68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:56.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.665+0000 7f732a09a640 1 -- 172.21.15.150:0/2891856011 shutdown_connections 2024-09-17T04:52:56.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.665+0000 7f732a09a640 1 -- 172.21.15.150:0/2891856011 wait complete. 2024-09-17T04:52:56.688 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-17T04:52:56.688 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T04:52:56.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:52:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 -- 172.21.15.150:0/4233843382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f580c35b0 msgr2=0x7f2f580c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 --2- 172.21.15.150:0/4233843382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f580c35b0 0x7f2f580c8a80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2f580cd080 tx=0x7f2f4001c890 comp rx=0 tx=0).stop 2024-09-17T04:52:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 -- 172.21.15.150:0/4233843382 shutdown_connections 2024-09-17T04:52:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 --2- 172.21.15.150:0/4233843382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f580c8fc0 0x7f2f580c20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 --2- 172.21.15.150:0/4233843382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f580c35b0 0x7f2f580c8a80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 --2- 172.21.15.150:0/4233843382 >> v2:172.21.15.176:3300/0 conn(0x7f2f581568a0 0x7f2f580c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 -- 172.21.15.150:0/4233843382 >> 172.21.15.150:0/4233843382 conn(0x7f2f58074b80 msgr2=0x7f2f5806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 -- 172.21.15.150:0/4233843382 shutdown_connections 2024-09-17T04:52:56.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 -- 172.21.15.150:0/4233843382 wait complete. 2024-09-17T04:52:56.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 Processor -- start 2024-09-17T04:52:56.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 -- start start 2024-09-17T04:52:56.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f580c35b0 0x7f2f5814bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:56.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f580c8fc0 0x7f2f58150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:56.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f581568a0 0x7f2f580cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:56.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2f580c3b40 con 0x7f2f580c8fc0 2024-09-17T04:52:56.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2f580c39c0 con 0x7f2f581568a0 2024-09-17T04:52:56.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f5d904640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2f580c69f0 con 0x7f2f580c35b0 2024-09-17T04:52:56.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f4ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f580c8fc0 0x7f2f58150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:56.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f4ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f580c8fc0 0x7f2f58150ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37576/0 (socket says 172.21.15.150:37576) 2024-09-17T04:52:56.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.833+0000 7f2f4ffff640 1 -- 172.21.15.150:0/956358895 learned_addr learned my addr 172.21.15.150:0/956358895 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:52:56.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f56ffd640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f580c35b0 0x7f2f5814bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:56.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f577fe640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f581568a0 0x7f2f580cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:56.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f4ffff640 1 -- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f580c35b0 msgr2=0x7f2f5814bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:56.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f4ffff640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f580c35b0 0x7f2f5814bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:56.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f4ffff640 1 -- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f581568a0 msgr2=0x7f2f580cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:56.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f4ffff640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f581568a0 0x7f2f580cb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:56.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f4ffff640 1 -- 172.21.15.150:0/956358895 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f58147880 con 0x7f2f580c8fc0 2024-09-17T04:52:56.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f56ffd640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f580c35b0 0x7f2f5814bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:52:56.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f577fe640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f581568a0 0x7f2f580cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:52:56.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f4ffff640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f580c8fc0 0x7f2f58150ed0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2f4001af40 tx=0x7f2f400a5780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:56.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f54ff9640 1 -- 172.21.15.150:0/956358895 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f400b2040 con 0x7f2f580c8fc0 2024-09-17T04:52:56.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f54ff9640 1 -- 172.21.15.150:0/956358895 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f400044b0 con 0x7f2f580c8fc0 2024-09-17T04:52:56.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f5d904640 1 -- 172.21.15.150:0/956358895 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f5814c910 con 0x7f2f580c8fc0 2024-09-17T04:52:56.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f54ff9640 1 -- 172.21.15.150:0/956358895 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40003860 con 0x7f2f580c8fc0 2024-09-17T04:52:56.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f5d904640 1 -- 172.21.15.150:0/956358895 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f58151890 con 0x7f2f580c8fc0 2024-09-17T04:52:56.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f5d904640 1 -- 172.21.15.150:0/956358895 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f580c20a0 con 0x7f2f580c8fc0 2024-09-17T04:52:56.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f54ff9640 1 -- 172.21.15.150:0/956358895 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2f400bf060 con 0x7f2f580c8fc0 2024-09-17T04:52:56.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f54ff9640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2f38073230 0x7f2f38075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:56.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f54ff9640 1 -- 172.21.15.150:0/956358895 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f2f4012e2b0 con 0x7f2f580c8fc0 2024-09-17T04:52:56.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.837+0000 7f2f56ffd640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2f38073230 0x7f2f38075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:56.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.841+0000 7f2f56ffd640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2f38073230 0x7f2f38075740 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2f480058d0 tx=0x7f2f48005ca0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:56.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:56.845+0000 7f2f54ff9640 1 -- 172.21.15.150:0/956358895 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2f400164b0 con 0x7f2f580c8fc0 2024-09-17T04:52:57.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.069+0000 7f2f5d904640 1 -- 172.21.15.150:0/956358895 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f58157790 con 0x7f2f580c8fc0 2024-09-17T04:52:57.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.069+0000 7f2f54ff9640 1 -- 172.21.15.150:0/956358895 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5706 (secure 0 0 0) 0x7f2f400f8f30 con 0x7f2f580c8fc0 2024-09-17T04:52:57.070 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:52:57.070 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"btime":"2024-09-17T04:52:56:848068+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:56.204986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:52:57.070 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 28 2024-09-17T04:52:57.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.073+0000 7f2f5d904640 1 -- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2f38073230 msgr2=0x7f2f38075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:57.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.073+0000 7f2f5d904640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2f38073230 0x7f2f38075740 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2f480058d0 tx=0x7f2f48005ca0 comp rx=0 tx=0).stop 2024-09-17T04:52:57.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.073+0000 7f2f5d904640 1 -- 172.21.15.150:0/956358895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f580c8fc0 msgr2=0x7f2f58150ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:57.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.073+0000 7f2f5d904640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f580c8fc0 0x7f2f58150ed0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2f4001af40 tx=0x7f2f400a5780 comp rx=0 tx=0).stop 2024-09-17T04:52:57.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.073+0000 7f2f5d904640 1 -- 172.21.15.150:0/956358895 shutdown_connections 2024-09-17T04:52:57.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.073+0000 7f2f5d904640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2f38073230 0x7f2f38075740 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:57.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.073+0000 7f2f5d904640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f581568a0 0x7f2f580cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:57.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.073+0000 7f2f5d904640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f580c8fc0 0x7f2f58150ed0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:57.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.073+0000 7f2f5d904640 1 --2- 172.21.15.150:0/956358895 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f580c35b0 0x7f2f5814bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:57.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.073+0000 7f2f5d904640 1 -- 172.21.15.150:0/956358895 >> 172.21.15.150:0/956358895 conn(0x7f2f58074b80 msgr2=0x7f2f580c81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:57.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.073+0000 7f2f5d904640 1 -- 172.21.15.150:0/956358895 shutdown_connections 2024-09-17T04:52:57.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:57.073+0000 7f2f5d904640 1 -- 172.21.15.150:0/956358895 wait complete. 2024-09-17T04:52:57.096 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T04:52:56:848068+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 28, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4358": { "addr": "172.21.15.176:6837/1045115232", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 1045115232, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 1045115232, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4358, "incarnation": 16, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:51:28.833974+0000", "qdb_cluster": [ 4358 ], "qdb_leader": 4358, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4358 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 28, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4975": { "addr": "172.21.15.150:6835/151500360", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 151500360, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 151500360, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4975, "incarnation": 27, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:52:56.204986+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4975 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:52:59.092 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:52:59.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:52:59.097 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:52:59.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.305+0000 7f170150d640 1 -- 172.21.15.150:0/3538191070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc156900 msgr2=0x7f16fc0c3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:59.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.305+0000 7f170150d640 1 --2- 172.21.15.150:0/3538191070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc156900 0x7f16fc0c3670 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009840 tx=0x7f16ec01c760 comp rx=0 tx=0).stop 2024-09-17T04:52:59.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.305+0000 7f170150d640 1 -- 172.21.15.150:0/3538191070 shutdown_connections 2024-09-17T04:52:59.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.305+0000 7f170150d640 1 --2- 172.21.15.150:0/3538191070 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16fc0c90d0 0x7f16fc0c2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:59.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.305+0000 7f170150d640 1 --2- 172.21.15.150:0/3538191070 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc156900 0x7f16fc0c3670 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:59.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.305+0000 7f170150d640 1 -- 172.21.15.150:0/3538191070 >> 172.21.15.150:0/3538191070 conn(0x7f16fc06f760 msgr2=0x7f16fc074580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:59.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.305+0000 7f170150d640 1 -- 172.21.15.150:0/3538191070 shutdown_connections 2024-09-17T04:52:59.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.305+0000 7f170150d640 1 -- 172.21.15.150:0/3538191070 wait complete. 2024-09-17T04:52:59.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.305+0000 7f170150d640 1 Processor -- start 2024-09-17T04:52:59.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.305+0000 7f170150d640 1 -- start start 2024-09-17T04:52:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f170150d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16fc0c8cc0 0x7f16fc0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f170150d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16fc0c90d0 0x7f16fc153a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f170150d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc156900 0x7f16fc147bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f170150d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f16fc150e00 con 0x7f16fc156900 2024-09-17T04:52:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f170150d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f16fc150c80 con 0x7f16fc0c90d0 2024-09-17T04:52:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f170150d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f16fc0c3fe0 con 0x7f16fc0c8cc0 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16fb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc156900 0x7f16fc147bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16fb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc156900 0x7f16fc147bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36746/0 (socket says 172.21.15.150:36746) 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16fb7fe640 1 -- 172.21.15.150:0/3033359472 learned_addr learned my addr 172.21.15.150:0/3033359472 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16faffd640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16fc0c8cc0 0x7f16fc0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16fa7fc640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16fc0c90d0 0x7f16fc153a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16fb7fe640 1 -- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16fc0c8cc0 msgr2=0x7f16fc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16fb7fe640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16fc0c8cc0 0x7f16fc0cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16fb7fe640 1 -- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16fc0c90d0 msgr2=0x7f16fc153a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16fb7fe640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16fc0c90d0 0x7f16fc153a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16fb7fe640 1 -- 172.21.15.150:0/3033359472 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16fc153270 con 0x7f16fc156900 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16fb7fe640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc156900 0x7f16fc147bf0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f16f400e9a0 tx=0x7f16f400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16e37fe640 1 -- 172.21.15.150:0/3033359472 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f4004d30 con 0x7f16fc156900 2024-09-17T04:52:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16e37fe640 1 -- 172.21.15.150:0/3033359472 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16f400cb20 con 0x7f16fc156900 2024-09-17T04:52:59.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f170150d640 1 -- 172.21.15.150:0/3033359472 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16fc148a40 con 0x7f16fc156900 2024-09-17T04:52:59.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f16e37fe640 1 -- 172.21.15.150:0/3033359472 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f4005020 con 0x7f16fc156900 2024-09-17T04:52:59.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f170150d640 1 -- 172.21.15.150:0/3033359472 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16fc151e50 con 0x7f16fc156900 2024-09-17T04:52:59.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.309+0000 7f170150d640 1 -- 172.21.15.150:0/3033359472 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16fc0c2080 con 0x7f16fc156900 2024-09-17T04:52:59.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.313+0000 7f16e37fe640 1 -- 172.21.15.150:0/3033359472 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f16f40051c0 con 0x7f16fc156900 2024-09-17T04:52:59.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.313+0000 7f16e37fe640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f16e8073200 0x7f16e8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:52:59.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.313+0000 7f16e37fe640 1 -- 172.21.15.150:0/3033359472 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f16f40950c0 con 0x7f16fc156900 2024-09-17T04:52:59.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.313+0000 7f16faffd640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f16e8073200 0x7f16e8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:52:59.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.313+0000 7f16faffd640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f16e8073200 0x7f16e8075710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0af000 tx=0x7f16ec01afa0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:52:59.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.317+0000 7f16e37fe640 1 -- 172.21.15.150:0/3033359472 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f16f4016030 con 0x7f16fc156900 2024-09-17T04:52:59.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.537+0000 7f170150d640 1 -- 172.21.15.150:0/3033359472 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16fc0c63f0 con 0x7f16fc156900 2024-09-17T04:52:59.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.537+0000 7f16e37fe640 1 -- 172.21.15.150:0/3033359472 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5710 (secure 0 0 0) 0x7f16f405fc20 con 0x7f16fc156900 2024-09-17T04:52:59.538 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:52:59.539 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":30,"btime":"2024-09-17T04:52:58:862052+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:52:59.539 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 30 2024-09-17T04:52:59.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.541+0000 7f170150d640 1 -- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f16e8073200 msgr2=0x7f16e8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:59.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.541+0000 7f170150d640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f16e8073200 0x7f16e8075710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0af000 tx=0x7f16ec01afa0 comp rx=0 tx=0).stop 2024-09-17T04:52:59.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.541+0000 7f170150d640 1 -- 172.21.15.150:0/3033359472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc156900 msgr2=0x7f16fc147bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:52:59.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.541+0000 7f170150d640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc156900 0x7f16fc147bf0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f16f400e9a0 tx=0x7f16f400ee60 comp rx=0 tx=0).stop 2024-09-17T04:52:59.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.541+0000 7f170150d640 1 -- 172.21.15.150:0/3033359472 shutdown_connections 2024-09-17T04:52:59.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.541+0000 7f170150d640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f16e8073200 0x7f16e8075710 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:59.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.541+0000 7f170150d640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16fc0c90d0 0x7f16fc153a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:59.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.541+0000 7f170150d640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16fc156900 0x7f16fc147bf0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:59.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.541+0000 7f170150d640 1 --2- 172.21.15.150:0/3033359472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16fc0c8cc0 0x7f16fc0cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:52:59.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.541+0000 7f170150d640 1 -- 172.21.15.150:0/3033359472 >> 172.21.15.150:0/3033359472 conn(0x7f16fc06f760 msgr2=0x7f16fc154770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:52:59.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.541+0000 7f170150d640 1 -- 172.21.15.150:0/3033359472 shutdown_connections 2024-09-17T04:52:59.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:52:59.541+0000 7f170150d640 1 -- 172.21.15.150:0/3033359472 wait complete. 2024-09-17T04:52:59.561 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5106 2024-09-17T04:52:59.561 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.c 2024-09-17T04:53:00.072 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T04:53:00.073+0000 7f86f7979640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:53:02.562 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-17T04:53:02.563 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:53:02.563 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:53:02.566 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:53:02.566 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:02.679 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T04:53:02.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.733+0000 7f845b6a6640 1 -- 172.21.15.150:0/194647611 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8440004810 msgr2=0x7f8440004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:02.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.733+0000 7f845b6a6640 1 --2- 172.21.15.150:0/194647611 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8440004810 0x7f8440004c20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f844400a9b0 tx=0x7f844401c8d0 comp rx=0 tx=0).stop 2024-09-17T04:53:02.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.733+0000 7f8459c43640 1 -- 172.21.15.150:0/194647611 reap_dead start 2024-09-17T04:53:02.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.733+0000 7f845b6a6640 1 -- 172.21.15.150:0/194647611 shutdown_connections 2024-09-17T04:53:02.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.733+0000 7f845b6a6640 1 -- 172.21.15.150:0/194647611 >> 172.21.15.150:0/194647611 conn(0x7f84540aab60 msgr2=0x7f84540aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:02.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.733+0000 7f845b6a6640 1 -- 172.21.15.150:0/194647611 shutdown_connections 2024-09-17T04:53:02.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.733+0000 7f845b6a6640 1 -- 172.21.15.150:0/194647611 wait complete. 2024-09-17T04:53:02.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f845b6a6640 1 Processor -- start 2024-09-17T04:53:02.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f845b6a6640 1 -- start start 2024-09-17T04:53:02.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f845b6a6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8440004810 0x7f845414a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:02.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f845b6a6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84541489a0 0x7f845414cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f845b6a6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8454146dd0 0x7f84541471e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f845b6a6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f845414a480 con 0x7f8454146dd0 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f845b6a6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8454072500 con 0x7f84541489a0 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f845b6a6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f84541544c0 con 0x7f8440004810 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8459c43640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8454146dd0 0x7f84541471e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8459c43640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8454146dd0 0x7f84541471e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36772/0 (socket says 172.21.15.150:36772) 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8459c43640 1 -- 172.21.15.150:0/623879565 learned_addr learned my addr 172.21.15.150:0/623879565 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8458c41640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84541489a0 0x7f845414cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8459442640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8440004810 0x7f845414a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8459c43640 1 -- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8440004810 msgr2=0x7f845414a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8459c43640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8440004810 0x7f845414a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8459c43640 1 -- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84541489a0 msgr2=0x7f845414cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:02.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8459c43640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84541489a0 0x7f845414cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:02.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8459c43640 1 -- 172.21.15.150:0/623879565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84540aea50 con 0x7f8454146dd0 2024-09-17T04:53:02.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8459c43640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8454146dd0 0x7f84541471e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f84540744d0 tx=0x7f844c00fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:02.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8449ffb640 1 -- 172.21.15.150:0/623879565 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844c005220 con 0x7f8454146dd0 2024-09-17T04:53:02.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f845b6a6640 1 -- 172.21.15.150:0/623879565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84540c7bf0 con 0x7f8454146dd0 2024-09-17T04:53:02.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8449ffb640 1 -- 172.21.15.150:0/623879565 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f844c0054d0 con 0x7f8454146dd0 2024-09-17T04:53:02.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.737+0000 7f8449ffb640 1 -- 172.21.15.150:0/623879565 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844c004a10 con 0x7f8454146dd0 2024-09-17T04:53:02.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.741+0000 7f845b6a6640 1 -- 172.21.15.150:0/623879565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f845414d610 con 0x7f8454146dd0 2024-09-17T04:53:02.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.741+0000 7f845b6a6640 1 -- 172.21.15.150:0/623879565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84540aeec0 con 0x7f8454146dd0 2024-09-17T04:53:02.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.741+0000 7f8449ffb640 1 -- 172.21.15.150:0/623879565 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f844c019030 con 0x7f8454146dd0 2024-09-17T04:53:02.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.741+0000 7f8449ffb640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8438073140 0x7f8438075650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:02.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.741+0000 7f8449ffb640 1 -- 172.21.15.150:0/623879565 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f844c094c90 con 0x7f8454146dd0 2024-09-17T04:53:02.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.741+0000 7f8459442640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8438073140 0x7f8438075650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:02.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.745+0000 7f8459442640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8438073140 0x7f8438075650 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8450001340 tx=0x7f8450001920 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:02.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.749+0000 7f8449ffb640 1 -- 172.21.15.150:0/623879565 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f844c05f1b0 con 0x7f8454146dd0 2024-09-17T04:53:02.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.981+0000 7f845b6a6640 1 -- 172.21.15.150:0/623879565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84540c8330 con 0x7f8454146dd0 2024-09-17T04:53:02.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.981+0000 7f8449ffb640 1 -- 172.21.15.150:0/623879565 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6538 (secure 0 0 0) 0x7f844c063be0 con 0x7f8454146dd0 2024-09-17T04:53:02.983 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:02.984 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":31,"btime":"2024-09-17T04:53:02:753261+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:02.984 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 31 2024-09-17T04:53:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.985+0000 7f845b6a6640 1 -- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8438073140 msgr2=0x7f8438075650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.985+0000 7f845b6a6640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8438073140 0x7f8438075650 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8450001340 tx=0x7f8450001920 comp rx=0 tx=0).stop 2024-09-17T04:53:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.985+0000 7f845b6a6640 1 -- 172.21.15.150:0/623879565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8454146dd0 msgr2=0x7f84541471e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.985+0000 7f845b6a6640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8454146dd0 0x7f84541471e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f84540744d0 tx=0x7f844c00fa20 comp rx=0 tx=0).stop 2024-09-17T04:53:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.985+0000 7f845b6a6640 1 -- 172.21.15.150:0/623879565 shutdown_connections 2024-09-17T04:53:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.985+0000 7f845b6a6640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8438073140 0x7f8438075650 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.985+0000 7f845b6a6640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84541489a0 0x7f845414cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.985+0000 7f845b6a6640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8454146dd0 0x7f84541471e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.985+0000 7f845b6a6640 1 --2- 172.21.15.150:0/623879565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8440004810 0x7f845414a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.985+0000 7f845b6a6640 1 -- 172.21.15.150:0/623879565 >> 172.21.15.150:0/623879565 conn(0x7f84540aab60 msgr2=0x7f84540c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.985+0000 7f845b6a6640 1 -- 172.21.15.150:0/623879565 shutdown_connections 2024-09-17T04:53:02.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:02.985+0000 7f845b6a6640 1 -- 172.21.15.150:0/623879565 wait complete. 2024-09-17T04:53:03.008 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-17T04:53:03.008 INFO:tasks.mds_thrash.fs.[b]:waiting for 97 secs before thrashing 2024-09-17T04:53:29.152 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:53:29.156 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:53:33.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:33.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.293+0000 7fc0b3d75640 1 -- 172.21.15.150:0/2093479553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc098004d50 msgr2=0x7fc098005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:33.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.293+0000 7fc0b3d75640 1 --2- 172.21.15.150:0/2093479553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc098004d50 0x7fc098005160 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009eb0 tx=0x7fc0a401d5d0 comp rx=0 tx=0).stop 2024-09-17T04:53:33.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.293+0000 7fc0b2312640 1 -- 172.21.15.150:0/2093479553 reap_dead start 2024-09-17T04:53:33.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.293+0000 7fc0b3d75640 1 -- 172.21.15.150:0/2093479553 shutdown_connections 2024-09-17T04:53:33.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.297+0000 7fc0b3d75640 1 -- 172.21.15.150:0/2093479553 >> 172.21.15.150:0/2093479553 conn(0x7fc0ac0c5ae0 msgr2=0x7fc0ac073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:33.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.297+0000 7fc0b3d75640 1 -- 172.21.15.150:0/2093479553 shutdown_connections 2024-09-17T04:53:33.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.297+0000 7fc0b3d75640 1 -- 172.21.15.150:0/2093479553 wait complete. 2024-09-17T04:53:33.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.297+0000 7fc0b3d75640 1 Processor -- start 2024-09-17T04:53:33.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.297+0000 7fc0b3d75640 1 -- start start 2024-09-17T04:53:33.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.297+0000 7fc0b3d75640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0ac14a0f0 0x7fc0ac151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.297+0000 7fc0b3d75640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0ac1566d0 0x7fc0ac154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.297+0000 7fc0b3d75640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0ac0c7940 0x7fc0ac149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.297+0000 7fc0b3d75640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ac151240 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.297+0000 7fc0b3d75640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ac157620 con 0x7fc0ac1566d0 2024-09-17T04:53:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.297+0000 7fc0b3d75640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc0ac06e7a0 con 0x7fc0ac0c7940 2024-09-17T04:53:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b1b11640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0ac14a0f0 0x7fc0ac151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b1b11640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0ac14a0f0 0x7fc0ac151b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43352/0 (socket says 172.21.15.150:43352) 2024-09-17T04:53:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b1b11640 1 -- 172.21.15.150:0/3478165904 learned_addr learned my addr 172.21.15.150:0/3478165904 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:33.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b1310640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0ac1566d0 0x7fc0ac154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:33.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b2312640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0ac0c7940 0x7fc0ac149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:33.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b1b11640 1 -- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0ac0c7940 msgr2=0x7fc0ac149550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:33.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b1b11640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0ac0c7940 0x7fc0ac149550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:33.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b1b11640 1 -- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0ac1566d0 msgr2=0x7fc0ac154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:33.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b1b11640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0ac1566d0 0x7fc0ac154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:33.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b1b11640 1 -- 172.21.15.150:0/3478165904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ac0ad2b0 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b1b11640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0ac14a0f0 0x7fc0ac151b50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80119d0 tx=0x7fc0a8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:33.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0a27fc640 1 -- 172.21.15.150:0/3478165904 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a801f070 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0a27fc640 1 -- 172.21.15.150:0/3478165904 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0a800fde0 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b3d75640 1 -- 172.21.15.150:0/3478165904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0ac1ed580 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0a27fc640 1 -- 172.21.15.150:0/3478165904 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8012690 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b3d75640 1 -- 172.21.15.150:0/3478165904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0ac154f50 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0b3d75640 1 -- 172.21.15.150:0/3478165904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0ac0c3070 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0a27fc640 1 -- 172.21.15.150:0/3478165904 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc0a8022070 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.301+0000 7fc0a27fc640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc090073200 0x7fc090075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:33.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.305+0000 7fc0a27fc640 1 -- 172.21.15.150:0/3478165904 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7fc0a8095780 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.305+0000 7fc0b1310640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc090073200 0x7fc090075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:33.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.305+0000 7fc0b1310640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc090073200 0x7fc090075710 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac14a500 tx=0x7fc09c001830 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.305+0000 7fc0a27fc640 1 -- 172.21.15.150:0/3478165904 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc0a805fe90 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.553+0000 7fc0b3d75640 1 -- 172.21.15.150:0/3478165904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0ac1ed750 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.553+0000 7fc0a27fc640 1 -- 172.21.15.150:0/3478165904 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6538 (secure 0 0 0) 0x7fc0a80648c0 con 0x7fc0ac14a0f0 2024-09-17T04:53:33.553 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:33.553 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":31,"btime":"2024-09-17T04:53:02:753261+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:33.554 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 31 2024-09-17T04:53:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.557+0000 7fc0b3d75640 1 -- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc090073200 msgr2=0x7fc090075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.557+0000 7fc0b3d75640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc090073200 0x7fc090075710 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac14a500 tx=0x7fc09c001830 comp rx=0 tx=0).stop 2024-09-17T04:53:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.557+0000 7fc0b3d75640 1 -- 172.21.15.150:0/3478165904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0ac14a0f0 msgr2=0x7fc0ac151b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.557+0000 7fc0b3d75640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0ac14a0f0 0x7fc0ac151b50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80119d0 tx=0x7fc0a8011e90 comp rx=0 tx=0).stop 2024-09-17T04:53:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.557+0000 7fc0b3d75640 1 -- 172.21.15.150:0/3478165904 shutdown_connections 2024-09-17T04:53:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.557+0000 7fc0b3d75640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc090073200 0x7fc090075710 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.557+0000 7fc0b3d75640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0ac1566d0 0x7fc0ac154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.557+0000 7fc0b3d75640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0ac0c7940 0x7fc0ac149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.557+0000 7fc0b3d75640 1 --2- 172.21.15.150:0/3478165904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0ac14a0f0 0x7fc0ac151b50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.557+0000 7fc0b3d75640 1 -- 172.21.15.150:0/3478165904 >> 172.21.15.150:0/3478165904 conn(0x7fc0ac0c5ae0 msgr2=0x7fc0ac0aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.557+0000 7fc0b3d75640 1 -- 172.21.15.150:0/3478165904 shutdown_connections 2024-09-17T04:53:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.557+0000 7fc0b3d75640 1 -- 172.21.15.150:0/3478165904 wait complete. 2024-09-17T04:53:33.575 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-17T04:53:33.575 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:53:33.575 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:53:33.597 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:53:33.598 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:33.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.757+0000 7f7b33564640 1 -- 172.21.15.150:0/2081916126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b18003a60 msgr2=0x7f7b18003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:33.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.757+0000 7f7b33564640 1 --2- 172.21.15.150:0/2081916126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b18003a60 0x7f7b18003ef0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009b40 tx=0x7f7b2801d250 comp rx=0 tx=0).stop 2024-09-17T04:53:33.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.757+0000 7f7b33564640 1 -- 172.21.15.150:0/2081916126 shutdown_connections 2024-09-17T04:53:33.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.757+0000 7f7b33564640 1 --2- 172.21.15.150:0/2081916126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b18003a60 0x7f7b18003ef0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:33.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.757+0000 7f7b33564640 1 -- 172.21.15.150:0/2081916126 >> 172.21.15.150:0/2081916126 conn(0x7f7b2c074750 msgr2=0x7f7b2c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:33.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.757+0000 7f7b33564640 1 -- 172.21.15.150:0/2081916126 shutdown_connections 2024-09-17T04:53:33.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.757+0000 7f7b33564640 1 -- 172.21.15.150:0/2081916126 wait complete. 2024-09-17T04:53:33.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b33564640 1 Processor -- start 2024-09-17T04:53:33.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b33564640 1 -- start start 2024-09-17T04:53:33.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b33564640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b18003a60 0x7f7b2c1e21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:33.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b33564640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b2c14db10 0x7f7b2c14b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:33.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b33564640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b2c0c6c90 0x7f7b2c1498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:33.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b33564640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c153f50 con 0x7f7b2c14db10 2024-09-17T04:53:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b33564640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c153dd0 con 0x7f7b18003a60 2024-09-17T04:53:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b33564640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7b2c1540d0 con 0x7f7b2c0c6c90 2024-09-17T04:53:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b30aff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b2c14db10 0x7f7b2c14b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b30aff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b2c14db10 0x7f7b2c14b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43370/0 (socket says 172.21.15.150:43370) 2024-09-17T04:53:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b30aff640 1 -- 172.21.15.150:0/2951853501 learned_addr learned my addr 172.21.15.150:0/2951853501 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b31300640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b18003a60 0x7f7b2c1e21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b31b01640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b2c0c6c90 0x7f7b2c1498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b30aff640 1 -- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b2c0c6c90 msgr2=0x7f7b2c1498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b30aff640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b2c0c6c90 0x7f7b2c1498f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b30aff640 1 -- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b18003a60 msgr2=0x7f7b2c1e21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:33.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b30aff640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b18003a60 0x7f7b2c1e21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:33.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b30aff640 1 -- 172.21.15.150:0/2951853501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b2c14a5a0 con 0x7f7b2c14db10 2024-09-17T04:53:33.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b30aff640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b2c14db10 0x7f7b2c14b980 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00d780 tx=0x7f7b1c00dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:33.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b21ffb640 1 -- 172.21.15.150:0/2951853501 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c018070 con 0x7f7b2c14db10 2024-09-17T04:53:33.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b21ffb640 1 -- 172.21.15.150:0/2951853501 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b1c0050d0 con 0x7f7b2c14db10 2024-09-17T04:53:33.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b33564640 1 -- 172.21.15.150:0/2951853501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b2c0cbb30 con 0x7f7b2c14db10 2024-09-17T04:53:33.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b21ffb640 1 -- 172.21.15.150:0/2951853501 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c00b6c0 con 0x7f7b2c14db10 2024-09-17T04:53:33.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.761+0000 7f7b33564640 1 -- 172.21.15.150:0/2951853501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b2c0cadb0 con 0x7f7b2c14db10 2024-09-17T04:53:33.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.765+0000 7f7b33564640 1 -- 172.21.15.150:0/2951853501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b2c0cbd70 con 0x7f7b2c14db10 2024-09-17T04:53:33.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.765+0000 7f7b21ffb640 1 -- 172.21.15.150:0/2951853501 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7b1c01b070 con 0x7f7b2c14db10 2024-09-17T04:53:33.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.765+0000 7f7b21ffb640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7b10073200 0x7f7b10075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:33.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.765+0000 7f7b21ffb640 1 -- 172.21.15.150:0/2951853501 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f7b1c093c00 con 0x7f7b2c14db10 2024-09-17T04:53:33.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.765+0000 7f7b31300640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7b10073200 0x7f7b10075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:33.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.765+0000 7f7b31300640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7b10073200 0x7f7b10075710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7b280ad000 tx=0x7f7b28019ff0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:33.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:33.773+0000 7f7b21ffb640 1 -- 172.21.15.150:0/2951853501 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7b2c0cbd70 con 0x7f7b2c14db10 2024-09-17T04:53:34.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.017+0000 7f7b33564640 1 -- 172.21.15.150:0/2951853501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b2c1f3e20 con 0x7f7b2c14db10 2024-09-17T04:53:34.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.017+0000 7f7b21ffb640 1 -- 172.21.15.150:0/2951853501 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6538 (secure 0 0 0) 0x7f7b1c063e20 con 0x7f7b2c14db10 2024-09-17T04:53:34.017 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:34.018 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":31,"btime":"2024-09-17T04:53:02:753261+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:34.018 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 31 2024-09-17T04:53:34.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.021+0000 7f7b33564640 1 -- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7b10073200 msgr2=0x7f7b10075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:34.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.021+0000 7f7b33564640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7b10073200 0x7f7b10075710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7b280ad000 tx=0x7f7b28019ff0 comp rx=0 tx=0).stop 2024-09-17T04:53:34.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.021+0000 7f7b33564640 1 -- 172.21.15.150:0/2951853501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b2c14db10 msgr2=0x7f7b2c14b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:34.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.021+0000 7f7b33564640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b2c14db10 0x7f7b2c14b980 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00d780 tx=0x7f7b1c00dc40 comp rx=0 tx=0).stop 2024-09-17T04:53:34.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.021+0000 7f7b33564640 1 -- 172.21.15.150:0/2951853501 shutdown_connections 2024-09-17T04:53:34.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.021+0000 7f7b33564640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7b10073200 0x7f7b10075710 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:34.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.021+0000 7f7b33564640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b2c0c6c90 0x7f7b2c1498f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:34.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.021+0000 7f7b33564640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b2c14db10 0x7f7b2c14b980 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:34.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.021+0000 7f7b33564640 1 --2- 172.21.15.150:0/2951853501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b18003a60 0x7f7b2c1e21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:34.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.021+0000 7f7b33564640 1 -- 172.21.15.150:0/2951853501 >> 172.21.15.150:0/2951853501 conn(0x7f7b2c074750 msgr2=0x7f7b2c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:34.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.021+0000 7f7b33564640 1 -- 172.21.15.150:0/2951853501 shutdown_connections 2024-09-17T04:53:34.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:34.021+0000 7f7b33564640 1 -- 172.21.15.150:0/2951853501 wait complete. 2024-09-17T04:53:34.039 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:53:36.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.209+0000 7fa41ab9a640 1 -- 172.21.15.150:0/2752742906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa400004090 msgr2=0x7fa400004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.209+0000 7fa41ab9a640 1 --2- 172.21.15.150:0/2752742906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa400004090 0x7fa400004810 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc00c2e0 tx=0x7fa3fc01f740 comp rx=0 tx=0).stop 2024-09-17T04:53:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.209+0000 7fa419137640 1 -- 172.21.15.150:0/2752742906 reap_dead start 2024-09-17T04:53:36.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.209+0000 7fa41ab9a640 1 -- 172.21.15.150:0/2752742906 shutdown_connections 2024-09-17T04:53:36.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.209+0000 7fa41ab9a640 1 -- 172.21.15.150:0/2752742906 >> 172.21.15.150:0/2752742906 conn(0x7fa4140c4a00 msgr2=0x7fa4140c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:36.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.209+0000 7fa41ab9a640 1 -- 172.21.15.150:0/2752742906 shutdown_connections 2024-09-17T04:53:36.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.209+0000 7fa41ab9a640 1 -- 172.21.15.150:0/2752742906 wait complete. 2024-09-17T04:53:36.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa41ab9a640 1 Processor -- start 2024-09-17T04:53:36.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa41ab9a640 1 -- start start 2024-09-17T04:53:36.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa41ab9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa400004090 0x7fa4140ad4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:36.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa41ab9a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa41415d600 0x7fa41415da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:36.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa41ab9a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4140abfe0 0x7fa4140ac3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa41ab9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa414070da0 con 0x7fa400004090 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa41ab9a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa414070ac0 con 0x7fa4140abfe0 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa41ab9a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa414153640 con 0x7fa41415d600 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa418936640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa400004090 0x7fa4140ad4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa418936640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa400004090 0x7fa4140ad4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43388/0 (socket says 172.21.15.150:43388) 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa418936640 1 -- 172.21.15.150:0/1813013110 learned_addr learned my addr 172.21.15.150:0/1813013110 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa413fff640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa41415d600 0x7fa41415da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa419137640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4140abfe0 0x7fa4140ac3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa418936640 1 -- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa41415d600 msgr2=0x7fa41415da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa418936640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa41415d600 0x7fa41415da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa418936640 1 -- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4140abfe0 msgr2=0x7fa4140ac3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa418936640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4140abfe0 0x7fa4140ac3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:36.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa418936640 1 -- 172.21.15.150:0/1813013110 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4141f04d0 con 0x7fa400004090 2024-09-17T04:53:36.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa419137640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4140abfe0 0x7fa4140ac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:53:36.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa418936640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa400004090 0x7fa4140ad4e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa404011d00 tx=0x7fa40400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:36.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa4117fa640 1 -- 172.21.15.150:0/1813013110 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40401b670 con 0x7fa400004090 2024-09-17T04:53:36.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa4117fa640 1 -- 172.21.15.150:0/1813013110 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa404016070 con 0x7fa400004090 2024-09-17T04:53:36.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa4117fa640 1 -- 172.21.15.150:0/1813013110 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40401ba90 con 0x7fa400004090 2024-09-17T04:53:36.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa41ab9a640 1 -- 172.21.15.150:0/1813013110 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4141f0770 con 0x7fa400004090 2024-09-17T04:53:36.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa41ab9a640 1 -- 172.21.15.150:0/1813013110 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4140ab4a0 con 0x7fa400004090 2024-09-17T04:53:36.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.213+0000 7fa41ab9a640 1 -- 172.21.15.150:0/1813013110 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa414153100 con 0x7fa400004090 2024-09-17T04:53:36.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.217+0000 7fa4117fa640 1 -- 172.21.15.150:0/1813013110 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa40401bc30 con 0x7fa400004090 2024-09-17T04:53:36.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.217+0000 7fa4117fa640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa3f4073200 0x7fa3f4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:36.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.217+0000 7fa4117fa640 1 -- 172.21.15.150:0/1813013110 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7fa404095990 con 0x7fa400004090 2024-09-17T04:53:36.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.217+0000 7fa413fff640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa3f4073200 0x7fa3f4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:36.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.217+0000 7fa413fff640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa3f4073200 0x7fa3f4075710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0af000 tx=0x7fa3fc01c580 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:36.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.221+0000 7fa4117fa640 1 -- 172.21.15.150:0/1813013110 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa404060590 con 0x7fa400004090 2024-09-17T04:53:36.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.429+0000 7fa41ab9a640 1 -- 172.21.15.150:0/1813013110 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4140c2760 con 0x7fa400004090 2024-09-17T04:53:36.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.429+0000 7fa4117fa640 1 -- 172.21.15.150:0/1813013110 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6538 (secure 0 0 0) 0x7fa404093e20 con 0x7fa400004090 2024-09-17T04:53:36.432 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:36.432 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":31,"btime":"2024-09-17T04:53:02:753261+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:36.432 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 31 2024-09-17T04:53:36.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.433+0000 7fa41ab9a640 1 -- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa3f4073200 msgr2=0x7fa3f4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:36.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.433+0000 7fa41ab9a640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa3f4073200 0x7fa3f4075710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc0af000 tx=0x7fa3fc01c580 comp rx=0 tx=0).stop 2024-09-17T04:53:36.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.433+0000 7fa41ab9a640 1 -- 172.21.15.150:0/1813013110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa400004090 msgr2=0x7fa4140ad4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:36.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.433+0000 7fa41ab9a640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa400004090 0x7fa4140ad4e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa404011d00 tx=0x7fa40400f730 comp rx=0 tx=0).stop 2024-09-17T04:53:36.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.433+0000 7fa41ab9a640 1 -- 172.21.15.150:0/1813013110 shutdown_connections 2024-09-17T04:53:36.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.433+0000 7fa41ab9a640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa3f4073200 0x7fa3f4075710 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:36.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.433+0000 7fa41ab9a640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4140abfe0 0x7fa4140ac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:36.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.433+0000 7fa41ab9a640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa41415d600 0x7fa41415da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:36.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.433+0000 7fa41ab9a640 1 --2- 172.21.15.150:0/1813013110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa400004090 0x7fa4140ad4e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:36.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.433+0000 7fa41ab9a640 1 -- 172.21.15.150:0/1813013110 >> 172.21.15.150:0/1813013110 conn(0x7fa4140c4a00 msgr2=0x7fa4140c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:36.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.433+0000 7fa41ab9a640 1 -- 172.21.15.150:0/1813013110 shutdown_connections 2024-09-17T04:53:36.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:36.433+0000 7fa41ab9a640 1 -- 172.21.15.150:0/1813013110 wait complete. 2024-09-17T04:53:36.453 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:53:38.454 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:38.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.625+0000 7ff23e1e8640 1 -- 172.21.15.150:0/4161878157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224003a10 msgr2=0x7ff224003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:38.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.625+0000 7ff23e1e8640 1 --2- 172.21.15.150:0/4161878157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224003a10 0x7ff224003e80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff228009b30 tx=0x7ff22801d270 comp rx=0 tx=0).stop 2024-09-17T04:53:38.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.625+0000 7ff237fff640 1 -- 172.21.15.150:0/4161878157 reap_dead start 2024-09-17T04:53:38.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.625+0000 7ff23e1e8640 1 -- 172.21.15.150:0/4161878157 shutdown_connections 2024-09-17T04:53:38.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.625+0000 7ff23e1e8640 1 -- 172.21.15.150:0/4161878157 >> 172.21.15.150:0/4161878157 conn(0x7ff2380c4a00 msgr2=0x7ff2380c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:38.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.625+0000 7ff23e1e8640 1 -- 172.21.15.150:0/4161878157 shutdown_connections 2024-09-17T04:53:38.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.625+0000 7ff23e1e8640 1 -- 172.21.15.150:0/4161878157 wait complete. 2024-09-17T04:53:38.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.625+0000 7ff23e1e8640 1 Processor -- start 2024-09-17T04:53:38.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.625+0000 7ff23e1e8640 1 -- start start 2024-09-17T04:53:38.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff23e1e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224003a10 0x7ff2381f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:38.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff23e1e8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2381f01e0 0x7ff2381593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:38.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff23e1e8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff238142c70 0x7ff238143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:38.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff23e1e8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff238074f70 con 0x7ff224003a10 2024-09-17T04:53:38.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff23e1e8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff2380adef0 con 0x7ff2381f01e0 2024-09-17T04:53:38.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff23e1e8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff2380750f0 con 0x7ff238142c70 2024-09-17T04:53:38.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff2377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224003a10 0x7ff2381f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:38.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff2377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224003a10 0x7ff2381f41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43410/0 (socket says 172.21.15.150:43410) 2024-09-17T04:53:38.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff2377fe640 1 -- 172.21.15.150:0/955195452 learned_addr learned my addr 172.21.15.150:0/955195452 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:38.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff237fff640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff238142c70 0x7ff238143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:38.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff236ffd640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2381f01e0 0x7ff2381593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:38.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff2377fe640 1 -- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff238142c70 msgr2=0x7ff238143080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:38.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff2377fe640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff238142c70 0x7ff238143080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:38.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff2377fe640 1 -- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2381f01e0 msgr2=0x7ff2381593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:38.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff2377fe640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2381f01e0 0x7ff2381593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:38.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff2377fe640 1 -- 172.21.15.150:0/955195452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2380c7340 con 0x7ff224003a10 2024-09-17T04:53:38.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff236ffd640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2381f01e0 0x7ff2381593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:53:38.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff2377fe640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224003a10 0x7ff2381f41a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff2280af040 tx=0x7ff2280acd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:38.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff22ffff640 1 -- 172.21.15.150:0/955195452 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2280b1040 con 0x7ff224003a10 2024-09-17T04:53:38.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff22ffff640 1 -- 172.21.15.150:0/955195452 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff2280ac400 con 0x7ff224003a10 2024-09-17T04:53:38.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff22ffff640 1 -- 172.21.15.150:0/955195452 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228009730 con 0x7ff224003a10 2024-09-17T04:53:38.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff23e1e8640 1 -- 172.21.15.150:0/955195452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2381560b0 con 0x7ff224003a10 2024-09-17T04:53:38.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff23e1e8640 1 -- 172.21.15.150:0/955195452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2381f39a0 con 0x7ff224003a10 2024-09-17T04:53:38.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff23e1e8640 1 -- 172.21.15.150:0/955195452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2380ad990 con 0x7ff224003a10 2024-09-17T04:53:38.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff22ffff640 1 -- 172.21.15.150:0/955195452 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff2280c0020 con 0x7ff224003a10 2024-09-17T04:53:38.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff22ffff640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff218073200 0x7ff218075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:38.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff22ffff640 1 -- 172.21.15.150:0/955195452 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7ff228130430 con 0x7ff224003a10 2024-09-17T04:53:38.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.629+0000 7ff236ffd640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff218073200 0x7ff218075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:38.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.633+0000 7ff236ffd640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff218073200 0x7ff218075710 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff2380c7180 tx=0x7ff2200091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:38.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.633+0000 7ff22ffff640 1 -- 172.21.15.150:0/955195452 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff228016f80 con 0x7ff224003a10 2024-09-17T04:53:38.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff23e1e8640 1 -- 172.21.15.150:0/955195452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2380c5190 con 0x7ff224003a10 2024-09-17T04:53:38.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff22ffff640 1 -- 172.21.15.150:0/955195452 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6538 (secure 0 0 0) 0x7ff2280fb070 con 0x7ff224003a10 2024-09-17T04:53:38.845 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:38.846 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":31,"btime":"2024-09-17T04:53:02:753261+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:38.846 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 31 2024-09-17T04:53:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff23e1e8640 1 -- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff218073200 msgr2=0x7ff218075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff23e1e8640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff218073200 0x7ff218075710 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff2380c7180 tx=0x7ff2200091e0 comp rx=0 tx=0).stop 2024-09-17T04:53:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff23e1e8640 1 -- 172.21.15.150:0/955195452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224003a10 msgr2=0x7ff2381f41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff23e1e8640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224003a10 0x7ff2381f41a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff2280af040 tx=0x7ff2280acd20 comp rx=0 tx=0).stop 2024-09-17T04:53:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff23e1e8640 1 -- 172.21.15.150:0/955195452 shutdown_connections 2024-09-17T04:53:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff23e1e8640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff218073200 0x7ff218075710 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff23e1e8640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2381f01e0 0x7ff2381593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff23e1e8640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff238142c70 0x7ff238143080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff23e1e8640 1 --2- 172.21.15.150:0/955195452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff224003a10 0x7ff2381f41a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff23e1e8640 1 -- 172.21.15.150:0/955195452 >> 172.21.15.150:0/955195452 conn(0x7ff2380c4a00 msgr2=0x7ff238152910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.845+0000 7ff23e1e8640 1 -- 172.21.15.150:0/955195452 shutdown_connections 2024-09-17T04:53:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:38.849+0000 7ff23e1e8640 1 -- 172.21.15.150:0/955195452 wait complete. 2024-09-17T04:53:38.868 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:53:40.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:41.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.036+0000 7f719bfff640 1 -- 172.21.15.150:0/4129466316 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71940a5df0 con 0x7f71a40c35b0 2024-09-17T04:53:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.036+0000 7f71ab82d640 1 -- 172.21.15.150:0/4129466316 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40c35b0 msgr2=0x7f71a40c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.036+0000 7f71ab82d640 1 --2- 172.21.15.150:0/4129466316 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40c35b0 0x7f71a40c8a80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7194009840 tx=0x7f719401c7f0 comp rx=0 tx=0).stop 2024-09-17T04:53:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.036+0000 7f71ab82d640 1 -- 172.21.15.150:0/4129466316 shutdown_connections 2024-09-17T04:53:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.036+0000 7f71ab82d640 1 --2- 172.21.15.150:0/4129466316 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40c35b0 0x7f71a40c8a80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.036+0000 7f71ab82d640 1 --2- 172.21.15.150:0/4129466316 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71a41568a0 0x7f71a40c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.036+0000 7f71ab82d640 1 -- 172.21.15.150:0/4129466316 >> 172.21.15.150:0/4129466316 conn(0x7f71a4074b80 msgr2=0x7f71a406f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.036+0000 7f71ab82d640 1 -- 172.21.15.150:0/4129466316 shutdown_connections 2024-09-17T04:53:41.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.036+0000 7f71ab82d640 1 -- 172.21.15.150:0/4129466316 wait complete. 2024-09-17T04:53:41.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71ab82d640 1 Processor -- start 2024-09-17T04:53:41.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71ab82d640 1 -- start start 2024-09-17T04:53:41.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71ab82d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a40c35b0 0x7f71a41510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:41.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71ab82d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40c8fc0 0x7f71a41531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:41.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71ab82d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71a41568a0 0x7f71a41f3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:41.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71ab82d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f71a40c6b70 con 0x7f71a40c35b0 2024-09-17T04:53:41.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71ab82d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f71a40c69f0 con 0x7f71a40c8fc0 2024-09-17T04:53:41.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71ab82d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f71a40c27b0 con 0x7f71a41568a0 2024-09-17T04:53:41.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71a95c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a40c35b0 0x7f71a41510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:41.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71a95c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a40c35b0 0x7f71a41510a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37268/0 (socket says 172.21.15.150:37268) 2024-09-17T04:53:41.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71a95c9640 1 -- 172.21.15.150:0/1650815692 learned_addr learned my addr 172.21.15.150:0/1650815692 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:41.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71a8dc8640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40c8fc0 0x7f71a41531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:41.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71a9dca640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71a41568a0 0x7f71a41f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:41.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71a95c9640 1 -- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71a41568a0 msgr2=0x7f71a41f3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:41.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71a95c9640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71a41568a0 0x7f71a41f3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:41.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71a95c9640 1 -- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40c8fc0 msgr2=0x7f71a41531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:41.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71a95c9640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40c8fc0 0x7f71a41531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:41.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71a95c9640 1 -- 172.21.15.150:0/1650815692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a41f2c90 con 0x7f71a40c35b0 2024-09-17T04:53:41.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71a95c9640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a40c35b0 0x7f71a41510a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f71a000e9a0 tx=0x7f71a000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:41.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f7199ffb640 1 -- 172.21.15.150:0/1650815692 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a000cd80 con 0x7f71a40c35b0 2024-09-17T04:53:41.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f7199ffb640 1 -- 172.21.15.150:0/1650815692 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71a0004020 con 0x7f71a40c35b0 2024-09-17T04:53:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71ab82d640 1 -- 172.21.15.150:0/1650815692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a40cc470 con 0x7f71a40c35b0 2024-09-17T04:53:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f7199ffb640 1 -- 172.21.15.150:0/1650815692 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a0004450 con 0x7f71a40c35b0 2024-09-17T04:53:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.040+0000 7f71ab82d640 1 -- 172.21.15.150:0/1650815692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a40cca30 con 0x7f71a40c35b0 2024-09-17T04:53:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.044+0000 7f71ab82d640 1 -- 172.21.15.150:0/1650815692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71a41f2e60 con 0x7f71a40c35b0 2024-09-17T04:53:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.044+0000 7f7199ffb640 1 -- 172.21.15.150:0/1650815692 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f71a0003600 con 0x7f71a40c35b0 2024-09-17T04:53:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.044+0000 7f7199ffb640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7188073200 0x7f7188075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.044+0000 7f7199ffb640 1 -- 172.21.15.150:0/1650815692 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f71a00950b0 con 0x7f71a40c35b0 2024-09-17T04:53:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.044+0000 7f71a8dc8640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7188073200 0x7f7188075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.044+0000 7f71a8dc8640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7188073200 0x7f7188075710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f71940af000 tx=0x7f719401b030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:41.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.048+0000 7f7199ffb640 1 -- 172.21.15.150:0/1650815692 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f71a005fc20 con 0x7f71a40c35b0 2024-09-17T04:53:41.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.264+0000 7f71ab82d640 1 -- 172.21.15.150:0/1650815692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71a41550b0 con 0x7f71a40c35b0 2024-09-17T04:53:41.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.264+0000 7f7199ffb640 1 -- 172.21.15.150:0/1650815692 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6538 (secure 0 0 0) 0x7f71a005f520 con 0x7f71a40c35b0 2024-09-17T04:53:41.267 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:41.267 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":31,"btime":"2024-09-17T04:53:02:753261+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:41.267 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 31 2024-09-17T04:53:41.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.268+0000 7f71ab82d640 1 -- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7188073200 msgr2=0x7f7188075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.268+0000 7f71ab82d640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7188073200 0x7f7188075710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f71940af000 tx=0x7f719401b030 comp rx=0 tx=0).stop 2024-09-17T04:53:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.268+0000 7f71ab82d640 1 -- 172.21.15.150:0/1650815692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a40c35b0 msgr2=0x7f71a41510a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.268+0000 7f71ab82d640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a40c35b0 0x7f71a41510a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f71a000e9a0 tx=0x7f71a000ee60 comp rx=0 tx=0).stop 2024-09-17T04:53:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.268+0000 7f71ab82d640 1 -- 172.21.15.150:0/1650815692 shutdown_connections 2024-09-17T04:53:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.268+0000 7f71ab82d640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7188073200 0x7f7188075710 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.268+0000 7f71ab82d640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40c8fc0 0x7f71a41531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.268+0000 7f71ab82d640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71a41568a0 0x7f71a41f3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.268+0000 7f71ab82d640 1 --2- 172.21.15.150:0/1650815692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a40c35b0 0x7f71a41510a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.268+0000 7f71ab82d640 1 -- 172.21.15.150:0/1650815692 >> 172.21.15.150:0/1650815692 conn(0x7f71a4074b80 msgr2=0x7f71a414a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.268+0000 7f71ab82d640 1 -- 172.21.15.150:0/1650815692 shutdown_connections 2024-09-17T04:53:41.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:41.268+0000 7f71ab82d640 1 -- 172.21.15.150:0/1650815692 wait complete. 2024-09-17T04:53:41.288 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:53:43.289 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:43.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.452+0000 7fbb3e875640 1 -- 172.21.15.150:0/1291981681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb20004da0 msgr2=0x7fbb20005270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:43.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.452+0000 7fbb3e875640 1 --2- 172.21.15.150:0/1291981681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb20004da0 0x7fbb20005270 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009de0 tx=0x7fbb3001d130 comp rx=0 tx=0).stop 2024-09-17T04:53:43.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3ce12640 1 -- 172.21.15.150:0/1291981681 reap_dead start 2024-09-17T04:53:43.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3e875640 1 -- 172.21.15.150:0/1291981681 shutdown_connections 2024-09-17T04:53:43.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3e875640 1 -- 172.21.15.150:0/1291981681 >> 172.21.15.150:0/1291981681 conn(0x7fbb380aaa80 msgr2=0x7fbb380aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:43.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3e875640 1 -- 172.21.15.150:0/1291981681 shutdown_connections 2024-09-17T04:53:43.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3e875640 1 -- 172.21.15.150:0/1291981681 wait complete. 2024-09-17T04:53:43.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3e875640 1 Processor -- start 2024-09-17T04:53:43.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3e875640 1 -- start start 2024-09-17T04:53:43.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3e875640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb380acf80 0x7fbb38145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:43.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3e875640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb381ec840 0x7fbb380abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:43.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3e875640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb38143000 0x7fbb38143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:43.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3e875640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbb380c5370 con 0x7fbb38143000 2024-09-17T04:53:43.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3e875640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbb380718d0 con 0x7fbb381ec840 2024-09-17T04:53:43.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3e875640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbb380c54f0 con 0x7fbb380acf80 2024-09-17T04:53:43.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3ce12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb38143000 0x7fbb38143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:43.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3ce12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb38143000 0x7fbb38143410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37278/0 (socket says 172.21.15.150:37278) 2024-09-17T04:53:43.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3ce12640 1 -- 172.21.15.150:0/1105436875 learned_addr learned my addr 172.21.15.150:0/1105436875 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:43.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb377fe640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb381ec840 0x7fbb380abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:43.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb37fff640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb380acf80 0x7fbb38145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:43.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3ce12640 1 -- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb380acf80 msgr2=0x7fbb38145590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:43.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3ce12640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb380acf80 0x7fbb38145590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:43.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3ce12640 1 -- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb381ec840 msgr2=0x7fbb380abc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:43.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3ce12640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb381ec840 0x7fbb380abc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:43.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3ce12640 1 -- 172.21.15.150:0/1105436875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb38148f60 con 0x7fbb38143000 2024-09-17T04:53:43.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb3ce12640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb38143000 0x7fbb38143410 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbb300af040 tx=0x7fbb300accd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:43.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb34ff9640 1 -- 172.21.15.150:0/1105436875 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb300099e0 con 0x7fbb38143000 2024-09-17T04:53:43.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.456+0000 7fbb34ff9640 1 -- 172.21.15.150:0/1105436875 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb300ac400 con 0x7fbb38143000 2024-09-17T04:53:43.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.460+0000 7fbb3e875640 1 -- 172.21.15.150:0/1105436875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb38145ad0 con 0x7fbb38143000 2024-09-17T04:53:43.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.460+0000 7fbb34ff9640 1 -- 172.21.15.150:0/1105436875 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb30004cb0 con 0x7fbb38143000 2024-09-17T04:53:43.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.460+0000 7fbb3e875640 1 -- 172.21.15.150:0/1105436875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb38155fc0 con 0x7fbb38143000 2024-09-17T04:53:43.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.460+0000 7fbb3e875640 1 -- 172.21.15.150:0/1105436875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb38155dd0 con 0x7fbb38143000 2024-09-17T04:53:43.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.460+0000 7fbb34ff9640 1 -- 172.21.15.150:0/1105436875 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbb30005030 con 0x7fbb38143000 2024-09-17T04:53:43.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.460+0000 7fbb34ff9640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb24073200 0x7fbb24075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:43.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.460+0000 7fbb34ff9640 1 -- 172.21.15.150:0/1105436875 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7fbb30131b40 con 0x7fbb38143000 2024-09-17T04:53:43.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.460+0000 7fbb37fff640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb24073200 0x7fbb24075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:43.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.460+0000 7fbb37fff640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb24073200 0x7fbb24075710 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbb28001870 tx=0x7fbb28001cc0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:43.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.464+0000 7fbb34ff9640 1 -- 172.21.15.150:0/1105436875 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbb38148f60 con 0x7fbb38143000 2024-09-17T04:53:43.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.660+0000 7fbb3e875640 1 -- 172.21.15.150:0/1105436875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb380c6c50 con 0x7fbb38143000 2024-09-17T04:53:43.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb34ff9640 1 -- 172.21.15.150:0/1105436875 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6538 (secure 0 0 0) 0x7fbb30016e40 con 0x7fbb38143000 2024-09-17T04:53:43.664 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:43.664 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":31,"btime":"2024-09-17T04:53:02:753261+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:43.664 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 31 2024-09-17T04:53:43.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb3e875640 1 -- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb24073200 msgr2=0x7fbb24075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:43.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb3e875640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb24073200 0x7fbb24075710 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbb28001870 tx=0x7fbb28001cc0 comp rx=0 tx=0).stop 2024-09-17T04:53:43.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb3e875640 1 -- 172.21.15.150:0/1105436875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb38143000 msgr2=0x7fbb38143410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:43.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb3e875640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb38143000 0x7fbb38143410 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbb300af040 tx=0x7fbb300accd0 comp rx=0 tx=0).stop 2024-09-17T04:53:43.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb3e875640 1 -- 172.21.15.150:0/1105436875 shutdown_connections 2024-09-17T04:53:43.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb3e875640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb24073200 0x7fbb24075710 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:43.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb3e875640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb381ec840 0x7fbb380abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:43.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb3e875640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb38143000 0x7fbb38143410 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:43.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb3e875640 1 --2- 172.21.15.150:0/1105436875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb380acf80 0x7fbb38145590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:43.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb3e875640 1 -- 172.21.15.150:0/1105436875 >> 172.21.15.150:0/1105436875 conn(0x7fbb380aaa80 msgr2=0x7fbb38152180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:43.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb3e875640 1 -- 172.21.15.150:0/1105436875 shutdown_connections 2024-09-17T04:53:43.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:43.664+0000 7fbb3e875640 1 -- 172.21.15.150:0/1105436875 wait complete. 2024-09-17T04:53:43.684 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:53:45.685 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:45.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.852+0000 7f1a9981f640 1 -- 172.21.15.150:0/3689482729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a80004810 msgr2=0x7f1a80004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:45.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.852+0000 7f1a9981f640 1 --2- 172.21.15.150:0/3689482729 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a80004810 0x7f1a80004c20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1a84006fa0 tx=0x7f1a8401cf00 comp rx=0 tx=0).stop 2024-09-17T04:53:45.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.852+0000 7f1a937fe640 1 -- 172.21.15.150:0/3689482729 reap_dead start 2024-09-17T04:53:45.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.852+0000 7f1a9981f640 1 -- 172.21.15.150:0/3689482729 shutdown_connections 2024-09-17T04:53:45.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.852+0000 7f1a9981f640 1 -- 172.21.15.150:0/3689482729 >> 172.21.15.150:0/3689482729 conn(0x7f1a940c4a00 msgr2=0x7f1a940c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:45.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.852+0000 7f1a9981f640 1 -- 172.21.15.150:0/3689482729 shutdown_connections 2024-09-17T04:53:45.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.852+0000 7f1a9981f640 1 -- 172.21.15.150:0/3689482729 wait complete. 2024-09-17T04:53:45.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a9981f640 1 Processor -- start 2024-09-17T04:53:45.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a9981f640 1 -- start start 2024-09-17T04:53:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a9981f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a80004810 0x7f1a94155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a9981f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a9415c310 0x7f1a9414ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a9981f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a9415c720 0x7f1a94149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a9981f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1a94071a20 con 0x7f1a9415c310 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a9981f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1a940adef0 con 0x7f1a80004810 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a9981f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1a940ace00 con 0x7f1a9415c720 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a927fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a9415c310 0x7f1a9414ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a927fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a9415c310 0x7f1a9414ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37288/0 (socket says 172.21.15.150:37288) 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a927fc640 1 -- 172.21.15.150:0/288219983 learned_addr learned my addr 172.21.15.150:0/288219983 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a92ffd640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a80004810 0x7f1a94155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a937fe640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a9415c720 0x7f1a94149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a927fc640 1 -- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a9415c720 msgr2=0x7f1a94149e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a927fc640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a9415c720 0x7f1a94149e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a927fc640 1 -- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a80004810 msgr2=0x7f1a94155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a927fc640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a80004810 0x7f1a94155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a927fc640 1 -- 172.21.15.150:0/288219983 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a941ec660 con 0x7f1a9415c310 2024-09-17T04:53:45.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a92ffd640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a80004810 0x7f1a94155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:53:45.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a927fc640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a9415c310 0x7f1a9414ebb0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1a84006fa0 tx=0x7f1a84001c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:45.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a8bfff640 1 -- 172.21.15.150:0/288219983 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84001580 con 0x7f1a9415c310 2024-09-17T04:53:45.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a8bfff640 1 -- 172.21.15.150:0/288219983 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a84006280 con 0x7f1a9415c310 2024-09-17T04:53:45.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a9981f640 1 -- 172.21.15.150:0/288219983 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a9414ffe0 con 0x7f1a9415c310 2024-09-17T04:53:45.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a8bfff640 1 -- 172.21.15.150:0/288219983 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a840b0630 con 0x7f1a9415c310 2024-09-17T04:53:45.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a9981f640 1 -- 172.21.15.150:0/288219983 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a941505a0 con 0x7f1a9415c310 2024-09-17T04:53:45.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.856+0000 7f1a9981f640 1 -- 172.21.15.150:0/288219983 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a940a7300 con 0x7f1a9415c310 2024-09-17T04:53:45.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.860+0000 7f1a8bfff640 1 -- 172.21.15.150:0/288219983 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1a840b07d0 con 0x7f1a9415c310 2024-09-17T04:53:45.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.860+0000 7f1a8bfff640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1a74073200 0x7f1a74075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:45.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.860+0000 7f1a8bfff640 1 -- 172.21.15.150:0/288219983 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f1a84130950 con 0x7f1a9415c310 2024-09-17T04:53:45.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.860+0000 7f1a92ffd640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1a74073200 0x7f1a74075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:45.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.860+0000 7f1a92ffd640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1a74073200 0x7f1a74075710 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c005fd0 tx=0x7f1a7c00b530 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:45.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:45.864+0000 7f1a8bfff640 1 -- 172.21.15.150:0/288219983 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1a84016cb0 con 0x7f1a9415c310 2024-09-17T04:53:46.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.064+0000 7f1a9981f640 1 -- 172.21.15.150:0/288219983 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a940c60f0 con 0x7f1a9415c310 2024-09-17T04:53:46.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.064+0000 7f1a8bfff640 1 -- 172.21.15.150:0/288219983 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6538 (secure 0 0 0) 0x7f1a840fb6b0 con 0x7f1a9415c310 2024-09-17T04:53:46.067 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:46.067 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":31,"btime":"2024-09-17T04:53:02:753261+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:46.067 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 31 2024-09-17T04:53:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.068+0000 7f1a9981f640 1 -- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1a74073200 msgr2=0x7f1a74075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.068+0000 7f1a9981f640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1a74073200 0x7f1a74075710 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c005fd0 tx=0x7f1a7c00b530 comp rx=0 tx=0).stop 2024-09-17T04:53:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.068+0000 7f1a9981f640 1 -- 172.21.15.150:0/288219983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a9415c310 msgr2=0x7f1a9414ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.068+0000 7f1a9981f640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a9415c310 0x7f1a9414ebb0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1a84006fa0 tx=0x7f1a84001c40 comp rx=0 tx=0).stop 2024-09-17T04:53:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.068+0000 7f1a9981f640 1 -- 172.21.15.150:0/288219983 shutdown_connections 2024-09-17T04:53:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.068+0000 7f1a9981f640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1a74073200 0x7f1a74075710 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.068+0000 7f1a9981f640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a9415c720 0x7f1a94149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:46.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.068+0000 7f1a9981f640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a9415c310 0x7f1a9414ebb0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:46.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.068+0000 7f1a9981f640 1 --2- 172.21.15.150:0/288219983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a80004810 0x7f1a94155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:46.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.068+0000 7f1a9981f640 1 -- 172.21.15.150:0/288219983 >> 172.21.15.150:0/288219983 conn(0x7f1a940c4a00 msgr2=0x7f1a9414d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:46.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.068+0000 7f1a9981f640 1 -- 172.21.15.150:0/288219983 shutdown_connections 2024-09-17T04:53:46.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:46.068+0000 7f1a9981f640 1 -- 172.21.15.150:0/288219983 wait complete. 2024-09-17T04:53:46.088 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:53:48.090 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:48.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.256+0000 7f982702d640 1 -- 172.21.15.150:0/91158327 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98200717a0 msgr2=0x7f9820074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:48.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.256+0000 7f982702d640 1 --2- 172.21.15.150:0/91158327 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98200717a0 0x7f9820074450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9818009c80 tx=0x7f981801cd10 comp rx=0 tx=0).stop 2024-09-17T04:53:48.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.256+0000 7f982702d640 1 -- 172.21.15.150:0/91158327 shutdown_connections 2024-09-17T04:53:48.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.256+0000 7f982702d640 1 --2- 172.21.15.150:0/91158327 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98200717a0 0x7f9820074450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:48.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.256+0000 7f982702d640 1 --2- 172.21.15.150:0/91158327 >> v2:172.21.15.176:3300/0 conn(0x7f9820070e50 0x7f9820071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:48.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.256+0000 7f982702d640 1 -- 172.21.15.150:0/91158327 >> 172.21.15.150:0/91158327 conn(0x7f98200c4a00 msgr2=0x7f98200c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:48.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.256+0000 7f982702d640 1 -- 172.21.15.150:0/91158327 shutdown_connections 2024-09-17T04:53:48.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.256+0000 7f982702d640 1 -- 172.21.15.150:0/91158327 wait complete. 2024-09-17T04:53:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f982702d640 1 Processor -- start 2024-09-17T04:53:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f982702d640 1 -- start start 2024-09-17T04:53:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f982702d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9820070e50 0x7f98201452b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f982702d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98200717a0 0x7f982014f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f982702d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9820073440 0x7f98200c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f982702d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f98200ad190 con 0x7f9820073440 2024-09-17T04:53:48.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f982702d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f98200ad010 con 0x7f9820070e50 2024-09-17T04:53:48.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f982702d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f98200ad310 con 0x7f98200717a0 2024-09-17T04:53:48.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f98255ca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9820073440 0x7f98200c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:48.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f98255ca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9820073440 0x7f98200c64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37290/0 (socket says 172.21.15.150:37290) 2024-09-17T04:53:48.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f98255ca640 1 -- 172.21.15.150:0/1143287130 learned_addr learned my addr 172.21.15.150:0/1143287130 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:48.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f9824dc9640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9820070e50 0x7f98201452b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:48.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f981ffff640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98200717a0 0x7f982014f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:48.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f98255ca640 1 -- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98200717a0 msgr2=0x7f982014f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:48.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f98255ca640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98200717a0 0x7f982014f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:48.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f98255ca640 1 -- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9820070e50 msgr2=0x7f98201452b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:48.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f98255ca640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9820070e50 0x7f98201452b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:48.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f98255ca640 1 -- 172.21.15.150:0/1143287130 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982014e290 con 0x7f9820073440 2024-09-17T04:53:48.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f98255ca640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9820073440 0x7f98200c64c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9818009bd0 tx=0x7f9818005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:48.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f981d7fa640 1 -- 172.21.15.150:0/1143287130 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98180bb070 con 0x7f9820073440 2024-09-17T04:53:48.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f981d7fa640 1 -- 172.21.15.150:0/1143287130 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9818004b40 con 0x7f9820073440 2024-09-17T04:53:48.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f982702d640 1 -- 172.21.15.150:0/1143287130 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f982014e570 con 0x7f9820073440 2024-09-17T04:53:48.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f981d7fa640 1 -- 172.21.15.150:0/1143287130 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98180b2040 con 0x7f9820073440 2024-09-17T04:53:48.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.260+0000 7f982702d640 1 -- 172.21.15.150:0/1143287130 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9820145fd0 con 0x7f9820073440 2024-09-17T04:53:48.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.264+0000 7f982702d640 1 -- 172.21.15.150:0/1143287130 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98200ad810 con 0x7f9820073440 2024-09-17T04:53:48.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.264+0000 7f981d7fa640 1 -- 172.21.15.150:0/1143287130 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f98180be070 con 0x7f9820073440 2024-09-17T04:53:48.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.264+0000 7f981d7fa640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9804073200 0x7f9804075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:48.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.264+0000 7f981d7fa640 1 -- 172.21.15.150:0/1143287130 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7f981812e2e0 con 0x7f9820073440 2024-09-17T04:53:48.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.264+0000 7f9824dc9640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9804073200 0x7f9804075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:48.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.264+0000 7f9824dc9640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9804073200 0x7f9804075710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9814007fd0 tx=0x7f9814009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:48.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.268+0000 7f981d7fa640 1 -- 172.21.15.150:0/1143287130 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9818016990 con 0x7f9820073440 2024-09-17T04:53:48.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.480+0000 7f982702d640 1 -- 172.21.15.150:0/1143287130 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98200c60f0 con 0x7f9820073440 2024-09-17T04:53:48.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.480+0000 7f981d7fa640 1 -- 172.21.15.150:0/1143287130 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6538 (secure 0 0 0) 0x7f98180f8910 con 0x7f9820073440 2024-09-17T04:53:48.483 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:48.483 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":31,"btime":"2024-09-17T04:53:02:753261+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5259,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:51:28.833974+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4358":{"gid":4358,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":33,"addr":"172.21.15.176:6837/1045115232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1045115232},{"type":"v1","addr":"172.21.15.176:6837","nonce":1045115232}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4358,"qdb_cluster":[4358]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:48.483 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 31 2024-09-17T04:53:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.484+0000 7f982702d640 1 -- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9804073200 msgr2=0x7f9804075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.484+0000 7f982702d640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9804073200 0x7f9804075710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9814007fd0 tx=0x7f9814009040 comp rx=0 tx=0).stop 2024-09-17T04:53:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.484+0000 7f982702d640 1 -- 172.21.15.150:0/1143287130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9820073440 msgr2=0x7f98200c64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.484+0000 7f982702d640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9820073440 0x7f98200c64c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9818009bd0 tx=0x7f9818005b00 comp rx=0 tx=0).stop 2024-09-17T04:53:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.484+0000 7f982702d640 1 -- 172.21.15.150:0/1143287130 shutdown_connections 2024-09-17T04:53:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.484+0000 7f982702d640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9804073200 0x7f9804075710 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.484+0000 7f982702d640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9820073440 0x7f98200c64c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.484+0000 7f982702d640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f98200717a0 0x7f982014f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.484+0000 7f982702d640 1 --2- 172.21.15.150:0/1143287130 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9820070e50 0x7f98201452b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.484+0000 7f982702d640 1 -- 172.21.15.150:0/1143287130 >> 172.21.15.150:0/1143287130 conn(0x7f98200c4a00 msgr2=0x7f98200a7300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.484+0000 7f982702d640 1 -- 172.21.15.150:0/1143287130 shutdown_connections 2024-09-17T04:53:48.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:48.484+0000 7f982702d640 1 -- 172.21.15.150:0/1143287130 wait complete. 2024-09-17T04:53:48.505 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:53:50.506 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:50.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.656+0000 7fda4bd1a640 1 -- 172.21.15.150:0/2159068101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda440708a0 msgr2=0x7fda44150dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.656+0000 7fda4bd1a640 1 --2- 172.21.15.150:0/2159068101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda440708a0 0x7fda44150dc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fda440ab840 tx=0x7fda3c01c800 comp rx=0 tx=0).stop 2024-09-17T04:53:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.656+0000 7fda4bd1a640 1 -- 172.21.15.150:0/2159068101 shutdown_connections 2024-09-17T04:53:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.656+0000 7fda4bd1a640 1 --2- 172.21.15.150:0/2159068101 >> v2:172.21.15.176:3300/0 conn(0x7fda440a9e80 0x7fda44074990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.656+0000 7fda4bd1a640 1 --2- 172.21.15.150:0/2159068101 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda440708a0 0x7fda44150dc0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.656+0000 7fda4bd1a640 1 -- 172.21.15.150:0/2159068101 >> 172.21.15.150:0/2159068101 conn(0x7fda440c4d50 msgr2=0x7fda440c5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.656+0000 7fda4bd1a640 1 -- 172.21.15.150:0/2159068101 shutdown_connections 2024-09-17T04:53:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.656+0000 7fda4bd1a640 1 -- 172.21.15.150:0/2159068101 wait complete. 2024-09-17T04:53:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda4bd1a640 1 Processor -- start 2024-09-17T04:53:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda4bd1a640 1 -- start start 2024-09-17T04:53:50.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda4bd1a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda440708a0 0x7fda441532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:50.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda4bd1a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda440a9e80 0x7fda44156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:50.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda4bd1a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda440aa3d0 0x7fda44149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:50.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda4bd1a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fda44160c50 con 0x7fda440708a0 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda4bd1a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fda44160ad0 con 0x7fda440a9e80 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda4bd1a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fda44160dd0 con 0x7fda440aa3d0 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda49ab6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda440708a0 0x7fda441532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda49ab6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda440708a0 0x7fda441532e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35818/0 (socket says 172.21.15.150:35818) 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda49ab6640 1 -- 172.21.15.150:0/1161939678 learned_addr learned my addr 172.21.15.150:0/1161939678 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda492b5640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda440a9e80 0x7fda44156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda4a2b7640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda440aa3d0 0x7fda44149ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda49ab6640 1 -- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda440aa3d0 msgr2=0x7fda44149ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda49ab6640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda440aa3d0 0x7fda44149ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda49ab6640 1 -- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda440a9e80 msgr2=0x7fda44156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda49ab6640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda440a9e80 0x7fda44156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda49ab6640 1 -- 172.21.15.150:0/1161939678 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4415de30 con 0x7fda440708a0 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda49ab6640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda440708a0 0x7fda441532e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0b0040 tx=0x7fda3c0a56c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda3a7fc640 1 -- 172.21.15.150:0/1161939678 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c0b2040 con 0x7fda440708a0 2024-09-17T04:53:50.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda3a7fc640 1 -- 172.21.15.150:0/1161939678 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda3c0043b0 con 0x7fda440708a0 2024-09-17T04:53:50.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda4bd1a640 1 -- 172.21.15.150:0/1161939678 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda44157800 con 0x7fda440708a0 2024-09-17T04:53:50.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda3a7fc640 1 -- 172.21.15.150:0/1161939678 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c005290 con 0x7fda440708a0 2024-09-17T04:53:50.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda4bd1a640 1 -- 172.21.15.150:0/1161939678 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda441f0160 con 0x7fda440708a0 2024-09-17T04:53:50.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.660+0000 7fda4bd1a640 1 -- 172.21.15.150:0/1161939678 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda44155b60 con 0x7fda440708a0 2024-09-17T04:53:50.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.664+0000 7fda3a7fc640 1 -- 172.21.15.150:0/1161939678 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fda3c0bf060 con 0x7fda440708a0 2024-09-17T04:53:50.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.664+0000 7fda3a7fc640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fda28073200 0x7fda28075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:50.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.664+0000 7fda3a7fc640 1 -- 172.21.15.150:0/1161939678 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6924+0+0 (secure 0 0 0) 0x7fda3c12e190 con 0x7fda440708a0 2024-09-17T04:53:50.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.664+0000 7fda492b5640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fda28073200 0x7fda28075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:50.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.664+0000 7fda492b5640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fda28073200 0x7fda28075710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fda440c5dc0 tx=0x7fda3400b2e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:50.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.668+0000 7fda3a7fc640 1 -- 172.21.15.150:0/1161939678 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fda3c0164b0 con 0x7fda440708a0 2024-09-17T04:53:50.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.868+0000 7fda4bd1a640 1 -- 172.21.15.150:0/1161939678 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda44150dc0 con 0x7fda440708a0 2024-09-17T04:53:50.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.868+0000 7fda3a7fc640 1 -- 172.21.15.150:0/1161939678 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5702 (secure 0 0 0) 0x7fda3c0f8d90 con 0x7fda440708a0 2024-09-17T04:53:50.870 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:50.870 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":32,"btime":"2024-09-17T04:53:50:852542+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:50.852532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:50.871 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 32 2024-09-17T04:53:50.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.872+0000 7fda4bd1a640 1 -- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fda28073200 msgr2=0x7fda28075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:50.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.872+0000 7fda4bd1a640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fda28073200 0x7fda28075710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fda440c5dc0 tx=0x7fda3400b2e0 comp rx=0 tx=0).stop 2024-09-17T04:53:50.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.872+0000 7fda4bd1a640 1 -- 172.21.15.150:0/1161939678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda440708a0 msgr2=0x7fda441532e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:50.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.872+0000 7fda4bd1a640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda440708a0 0x7fda441532e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0b0040 tx=0x7fda3c0a56c0 comp rx=0 tx=0).stop 2024-09-17T04:53:50.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.872+0000 7fda4bd1a640 1 -- 172.21.15.150:0/1161939678 shutdown_connections 2024-09-17T04:53:50.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.872+0000 7fda4bd1a640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fda28073200 0x7fda28075710 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:50.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.872+0000 7fda4bd1a640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda440a9e80 0x7fda44156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:50.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.872+0000 7fda4bd1a640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fda440aa3d0 0x7fda44149ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:50.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.872+0000 7fda4bd1a640 1 --2- 172.21.15.150:0/1161939678 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda440708a0 0x7fda441532e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:50.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.872+0000 7fda4bd1a640 1 -- 172.21.15.150:0/1161939678 >> 172.21.15.150:0/1161939678 conn(0x7fda440c4d50 msgr2=0x7fda44150000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:50.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.872+0000 7fda4bd1a640 1 -- 172.21.15.150:0/1161939678 shutdown_connections 2024-09-17T04:53:50.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:50.872+0000 7fda4bd1a640 1 -- 172.21.15.150:0/1161939678 wait complete. 2024-09-17T04:53:50.905 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-17T04:53:50.905 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T04:53:50.905 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.044+0000 7fca36236640 1 -- 172.21.15.150:0/692947204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca300c3900 msgr2=0x7fca30074920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.044+0000 7fca36236640 1 --2- 172.21.15.150:0/692947204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca300c3900 0x7fca30074920 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fca18009900 tx=0x7fca1801c8a0 comp rx=0 tx=0).stop 2024-09-17T04:53:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.044+0000 7fca36236640 1 -- 172.21.15.150:0/692947204 shutdown_connections 2024-09-17T04:53:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.044+0000 7fca36236640 1 --2- 172.21.15.150:0/692947204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca300c3900 0x7fca30074920 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.044+0000 7fca36236640 1 --2- 172.21.15.150:0/692947204 >> v2:172.21.15.176:3300/0 conn(0x7fca300c4250 0x7fca30153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.044+0000 7fca36236640 1 -- 172.21.15.150:0/692947204 >> 172.21.15.150:0/692947204 conn(0x7fca3006e360 msgr2=0x7fca3006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.044+0000 7fca36236640 1 -- 172.21.15.150:0/692947204 shutdown_connections 2024-09-17T04:53:51.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.044+0000 7fca36236640 1 -- 172.21.15.150:0/692947204 wait complete. 2024-09-17T04:53:51.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca36236640 1 Processor -- start 2024-09-17T04:53:51.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca36236640 1 -- start start 2024-09-17T04:53:51.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca36236640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca30074e60 0x7fca301557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:51.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca36236640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca300c3900 0x7fca301562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca36236640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fca300c4250 0x7fca300c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca36236640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fca30072f30 con 0x7fca30074e60 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca36236640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fca30072db0 con 0x7fca300c3900 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca36236640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fca30160f90 con 0x7fca300c4250 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca30074e60 0x7fca301557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca30074e60 0x7fca301557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35844/0 (socket says 172.21.15.150:35844) 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2f7fe640 1 -- 172.21.15.150:0/1640877033 learned_addr learned my addr 172.21.15.150:0/1640877033 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca26ffd640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca300c3900 0x7fca301562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2ffff640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fca300c4250 0x7fca300c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2f7fe640 1 -- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fca300c4250 msgr2=0x7fca300c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2f7fe640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fca300c4250 0x7fca300c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2f7fe640 1 -- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca300c3900 msgr2=0x7fca301562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2f7fe640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca300c3900 0x7fca301562b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2f7fe640 1 -- 172.21.15.150:0/1640877033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca300c4e90 con 0x7fca30074e60 2024-09-17T04:53:51.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2f7fe640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca30074e60 0x7fca301557d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fca2000e7f0 tx=0x7fca2000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:51.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2cff9640 1 -- 172.21.15.150:0/1640877033 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2000eea0 con 0x7fca30074e60 2024-09-17T04:53:51.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2cff9640 1 -- 172.21.15.150:0/1640877033 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca2000cc80 con 0x7fca30074e60 2024-09-17T04:53:51.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca36236640 1 -- 172.21.15.150:0/1640877033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca30153300 con 0x7fca30074e60 2024-09-17T04:53:51.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca2cff9640 1 -- 172.21.15.150:0/1640877033 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca20005280 con 0x7fca30074e60 2024-09-17T04:53:51.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.048+0000 7fca36236640 1 -- 172.21.15.150:0/1640877033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca30154da0 con 0x7fca30074e60 2024-09-17T04:53:51.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.052+0000 7fca36236640 1 -- 172.21.15.150:0/1640877033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca30074920 con 0x7fca30074e60 2024-09-17T04:53:51.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.052+0000 7fca2cff9640 1 -- 172.21.15.150:0/1640877033 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fca200054f0 con 0x7fca30074e60 2024-09-17T04:53:51.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.052+0000 7fca2cff9640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fca10073200 0x7fca10075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:51.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.052+0000 7fca2cff9640 1 -- 172.21.15.150:0/1640877033 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fca20095400 con 0x7fca30074e60 2024-09-17T04:53:51.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.052+0000 7fca26ffd640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fca10073200 0x7fca10075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:51.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.052+0000 7fca26ffd640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fca10073200 0x7fca10075710 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fca180af000 tx=0x7fca1801b0e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:51.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.056+0000 7fca2cff9640 1 -- 172.21.15.150:0/1640877033 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fca20060020 con 0x7fca30074e60 2024-09-17T04:53:51.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.252+0000 7fca36236640 1 -- 172.21.15.150:0/1640877033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca3014a260 con 0x7fca30074e60 2024-09-17T04:53:51.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.252+0000 7fca2cff9640 1 -- 172.21.15.150:0/1640877033 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5702 (secure 0 0 0) 0x7fca2005f920 con 0x7fca30074e60 2024-09-17T04:53:51.256 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:51.256 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":32,"btime":"2024-09-17T04:53:50:852542+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:50.852532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:51.256 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 32 2024-09-17T04:53:51.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.256+0000 7fca36236640 1 -- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fca10073200 msgr2=0x7fca10075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:51.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.256+0000 7fca36236640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fca10073200 0x7fca10075710 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fca180af000 tx=0x7fca1801b0e0 comp rx=0 tx=0).stop 2024-09-17T04:53:51.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.256+0000 7fca36236640 1 -- 172.21.15.150:0/1640877033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca30074e60 msgr2=0x7fca301557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:51.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.256+0000 7fca36236640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca30074e60 0x7fca301557d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fca2000e7f0 tx=0x7fca2000ecb0 comp rx=0 tx=0).stop 2024-09-17T04:53:51.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.256+0000 7fca36236640 1 -- 172.21.15.150:0/1640877033 shutdown_connections 2024-09-17T04:53:51.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.256+0000 7fca36236640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fca10073200 0x7fca10075710 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:51.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.256+0000 7fca36236640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca300c3900 0x7fca301562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:51.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.256+0000 7fca36236640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fca300c4250 0x7fca300c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:51.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.256+0000 7fca36236640 1 --2- 172.21.15.150:0/1640877033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca30074e60 0x7fca301557d0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:51.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.256+0000 7fca36236640 1 -- 172.21.15.150:0/1640877033 >> 172.21.15.150:0/1640877033 conn(0x7fca3006e360 msgr2=0x7fca3015e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:51.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.256+0000 7fca36236640 1 -- 172.21.15.150:0/1640877033 shutdown_connections 2024-09-17T04:53:51.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:51.256+0000 7fca36236640 1 -- 172.21.15.150:0/1640877033 wait complete. 2024-09-17T04:53:51.277 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T04:53:50:852542+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5259": { "addr": "172.21.15.150:6833/3532647866", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 3532647866, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 3532647866, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5259, "incarnation": 32, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:53:50.852532+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5259 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4975": { "addr": "172.21.15.150:6835/151500360", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 151500360, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 151500360, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4975, "incarnation": 27, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:52:58.862037+0000", "qdb_cluster": [ 4975 ], "qdb_leader": 4975, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4975 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:53:53.278 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:53.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.448+0000 7f0c96e77640 1 -- 172.21.15.150:0/73521025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c78004e70 msgr2=0x7f0c780052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:53.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.448+0000 7f0c96e77640 1 --2- 172.21.15.150:0/73521025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c78004e70 0x7f0c780052a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800c630 tx=0x7f0c8801d2c0 comp rx=0 tx=0).stop 2024-09-17T04:53:53.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.448+0000 7f0c95414640 1 -- 172.21.15.150:0/73521025 reap_dead start 2024-09-17T04:53:53.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.448+0000 7f0c96e77640 1 -- 172.21.15.150:0/73521025 shutdown_connections 2024-09-17T04:53:53.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.448+0000 7f0c96e77640 1 -- 172.21.15.150:0/73521025 >> 172.21.15.150:0/73521025 conn(0x7f0c900c9de0 msgr2=0x7f0c900ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:53.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.448+0000 7f0c96e77640 1 -- 172.21.15.150:0/73521025 shutdown_connections 2024-09-17T04:53:53.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.448+0000 7f0c96e77640 1 -- 172.21.15.150:0/73521025 wait complete. 2024-09-17T04:53:53.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c96e77640 1 Processor -- start 2024-09-17T04:53:53.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c96e77640 1 -- start start 2024-09-17T04:53:53.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c96e77640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c78004e70 0x7f0c90156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:53.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c96e77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c90155df0 0x7f0c900c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:53.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c96e77640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c901576e0 0x7f0c90146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:53.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c96e77640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0c900c7ce0 con 0x7f0c90155df0 2024-09-17T04:53:53.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c96e77640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0c900c5a40 con 0x7f0c78004e70 2024-09-17T04:53:53.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c96e77640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0c900c7e60 con 0x7f0c901576e0 2024-09-17T04:53:53.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c8ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c90155df0 0x7f0c900c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c8ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c90155df0 0x7f0c900c32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35862/0 (socket says 172.21.15.150:35862) 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c8ffff640 1 -- 172.21.15.150:0/2635350866 learned_addr learned my addr 172.21.15.150:0/2635350866 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c94c13640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c78004e70 0x7f0c90156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c95414640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c901576e0 0x7f0c90146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c8ffff640 1 -- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c901576e0 msgr2=0x7f0c90146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c8ffff640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c901576e0 0x7f0c90146740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c8ffff640 1 -- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c78004e70 msgr2=0x7f0c90156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c8ffff640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c78004e70 0x7f0c90156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c8ffff640 1 -- 172.21.15.150:0/2635350866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c90149630 con 0x7f0c90155df0 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c94c13640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c78004e70 0x7f0c90156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c8ffff640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c90155df0 0x7f0c900c32e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0c84004f10 tx=0x7f0c84001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c8d7fa640 1 -- 172.21.15.150:0/2635350866 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84015020 con 0x7f0c90155df0 2024-09-17T04:53:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c8d7fa640 1 -- 172.21.15.150:0/2635350866 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c84011960 con 0x7f0c90155df0 2024-09-17T04:53:53.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c8d7fa640 1 -- 172.21.15.150:0/2635350866 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84011d90 con 0x7f0c90155df0 2024-09-17T04:53:53.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c96e77640 1 -- 172.21.15.150:0/2635350866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c901490c0 con 0x7f0c90155df0 2024-09-17T04:53:53.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.452+0000 7f0c96e77640 1 -- 172.21.15.150:0/2635350866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c901f05f0 con 0x7f0c90155df0 2024-09-17T04:53:53.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.456+0000 7f0c96e77640 1 -- 172.21.15.150:0/2635350866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c90156500 con 0x7f0c90155df0 2024-09-17T04:53:53.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.456+0000 7f0c8d7fa640 1 -- 172.21.15.150:0/2635350866 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0c84018020 con 0x7f0c90155df0 2024-09-17T04:53:53.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.456+0000 7f0c8d7fa640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0c7c073230 0x7f0c7c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:53.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.456+0000 7f0c8d7fa640 1 -- 172.21.15.150:0/2635350866 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7f0c84095250 con 0x7f0c90155df0 2024-09-17T04:53:53.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.456+0000 7f0c94c13640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0c7c073230 0x7f0c7c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:53.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.456+0000 7f0c94c13640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0c7c073230 0x7f0c7c075740 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0c80005fd0 tx=0x7f0c80005f40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:53.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.460+0000 7f0c8d7fa640 1 -- 172.21.15.150:0/2635350866 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0c8401f020 con 0x7f0c90155df0 2024-09-17T04:53:53.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 -- 172.21.15.150:0/2635350866 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c900c5330 con 0x7f0c90155df0 2024-09-17T04:53:53.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c8d7fa640 1 -- 172.21.15.150:0/2635350866 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5703 (secure 0 0 0) 0x7f0c8405fde0 con 0x7f0c90155df0 2024-09-17T04:53:53.665 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:53.665 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":34,"btime":"2024-09-17T04:53:52:916304+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:51.922102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:53.665 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 34 2024-09-17T04:53:53.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 -- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0c7c073230 msgr2=0x7f0c7c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:53.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0c7c073230 0x7f0c7c075740 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0c80005fd0 tx=0x7f0c80005f40 comp rx=0 tx=0).stop 2024-09-17T04:53:53.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 -- 172.21.15.150:0/2635350866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c90155df0 msgr2=0x7f0c900c32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:53.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c90155df0 0x7f0c900c32e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0c84004f10 tx=0x7f0c84001640 comp rx=0 tx=0).stop 2024-09-17T04:53:53.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 -- 172.21.15.150:0/2635350866 shutdown_connections 2024-09-17T04:53:53.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0c7c073230 0x7f0c7c075740 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:53.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0c901576e0 0x7f0c90146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:53.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0c90155df0 0x7f0c900c32e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:53.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 --2- 172.21.15.150:0/2635350866 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c78004e70 0x7f0c90156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:53.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 -- 172.21.15.150:0/2635350866 >> 172.21.15.150:0/2635350866 conn(0x7f0c900c9de0 msgr2=0x7f0c900c91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:53.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 -- 172.21.15.150:0/2635350866 shutdown_connections 2024-09-17T04:53:53.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:53.664+0000 7f0c96e77640 1 -- 172.21.15.150:0/2635350866 wait complete. 2024-09-17T04:53:53.687 INFO:tasks.mds_thrash.fs.[a]:no change 2024-09-17T04:53:55.688 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:55.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.848+0000 7fc64ce12640 1 -- 172.21.15.150:0/605619581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6480c35b0 msgr2=0x7fc6480c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:55.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.848+0000 7fc64ce12640 1 --2- 172.21.15.150:0/605619581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6480c35b0 0x7fc6480c8a80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc630009900 tx=0x7fc63001c8b0 comp rx=0 tx=0).stop 2024-09-17T04:53:55.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.848+0000 7fc64ce12640 1 -- 172.21.15.150:0/605619581 shutdown_connections 2024-09-17T04:53:55.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.848+0000 7fc64ce12640 1 --2- 172.21.15.150:0/605619581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6480c35b0 0x7fc6480c8a80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:55.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.848+0000 7fc64ce12640 1 --2- 172.21.15.150:0/605619581 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6481568a0 0x7fc6480c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:55.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.848+0000 7fc64ce12640 1 -- 172.21.15.150:0/605619581 >> 172.21.15.150:0/605619581 conn(0x7fc648074b80 msgr2=0x7fc64806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:55.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.848+0000 7fc64ce12640 1 -- 172.21.15.150:0/605619581 shutdown_connections 2024-09-17T04:53:55.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.848+0000 7fc64ce12640 1 -- 172.21.15.150:0/605619581 wait complete. 2024-09-17T04:53:55.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.848+0000 7fc64ce12640 1 Processor -- start 2024-09-17T04:53:55.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.848+0000 7fc64ce12640 1 -- start start 2024-09-17T04:53:55.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64ce12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6480c35b0 0x7fc6481f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:55.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64ce12640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6480c8fc0 0x7fc6480cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:55.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64ce12640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6481568a0 0x7fc648147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:55.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64ce12640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc6480c69f0 con 0x7fc6480c35b0 2024-09-17T04:53:55.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64ce12640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc648154590 con 0x7fc6480c8fc0 2024-09-17T04:53:55.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64ce12640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc6480c6b70 con 0x7fc6481568a0 2024-09-17T04:53:55.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6480c35b0 0x7fc6481f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:55.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6480c35b0 0x7fc6481f31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35866/0 (socket says 172.21.15.150:35866) 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64659c640 1 -- 172.21.15.150:0/1875871436 learned_addr learned my addr 172.21.15.150:0/1875871436 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc646d9d640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6481568a0 0x7fc648147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc63dd9b640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6480c8fc0 0x7fc6480cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64659c640 1 -- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6481568a0 msgr2=0x7fc648147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64659c640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6481568a0 0x7fc648147470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64659c640 1 -- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6480c8fc0 msgr2=0x7fc6480cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64659c640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6480c8fc0 0x7fc6480cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64659c640 1 -- 172.21.15.150:0/1875871436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6480cc880 con 0x7fc6480c35b0 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc63dd9b640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6480c8fc0 0x7fc6480cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc646d9d640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6481568a0 0x7fc648147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64659c640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6480c35b0 0x7fc6481f31a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc63800d720 tx=0x7fc63800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc63f7fe640 1 -- 172.21.15.150:0/1875871436 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc638017070 con 0x7fc6480c35b0 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64ce12640 1 -- 172.21.15.150:0/1875871436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6481513d0 con 0x7fc6480c35b0 2024-09-17T04:53:55.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc63f7fe640 1 -- 172.21.15.150:0/1875871436 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc638009d20 con 0x7fc6480c35b0 2024-09-17T04:53:55.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc63f7fe640 1 -- 172.21.15.150:0/1875871436 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc638005280 con 0x7fc6480c35b0 2024-09-17T04:53:55.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64ce12640 1 -- 172.21.15.150:0/1875871436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc648151990 con 0x7fc6480c35b0 2024-09-17T04:53:55.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.852+0000 7fc64ce12640 1 -- 172.21.15.150:0/1875871436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6480c8a80 con 0x7fc6480c35b0 2024-09-17T04:53:55.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.856+0000 7fc63f7fe640 1 -- 172.21.15.150:0/1875871436 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc638005420 con 0x7fc6480c35b0 2024-09-17T04:53:55.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.856+0000 7fc63f7fe640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc628073200 0x7fc628075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:55.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.856+0000 7fc63f7fe640 1 -- 172.21.15.150:0/1875871436 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fc638094bb0 con 0x7fc6480c35b0 2024-09-17T04:53:55.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.856+0000 7fc63dd9b640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc628073200 0x7fc628075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:55.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.856+0000 7fc63dd9b640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc628073200 0x7fc628075710 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc6300af000 tx=0x7fc63001b0f0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:55.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:55.860+0000 7fc63f7fe640 1 -- 172.21.15.150:0/1875871436 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc63805f820 con 0x7fc6480c35b0 2024-09-17T04:53:56.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.108+0000 7fc64ce12640 1 -- 172.21.15.150:0/1875871436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc64814a900 con 0x7fc6480c35b0 2024-09-17T04:53:56.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.108+0000 7fc63f7fe640 1 -- 172.21.15.150:0/1875871436 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5710 (secure 0 0 0) 0x7fc63805f120 con 0x7fc6480c35b0 2024-09-17T04:53:56.110 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:56.110 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":35,"btime":"2024-09-17T04:53:53:928953+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:56.110 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 35 2024-09-17T04:53:56.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.112+0000 7fc64ce12640 1 -- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc628073200 msgr2=0x7fc628075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:56.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.112+0000 7fc64ce12640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc628073200 0x7fc628075710 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc6300af000 tx=0x7fc63001b0f0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.112+0000 7fc64ce12640 1 -- 172.21.15.150:0/1875871436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6480c35b0 msgr2=0x7fc6481f31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:56.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.112+0000 7fc64ce12640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6480c35b0 0x7fc6481f31a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc63800d720 tx=0x7fc63800dbe0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.112+0000 7fc64ce12640 1 -- 172.21.15.150:0/1875871436 shutdown_connections 2024-09-17T04:53:56.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.112+0000 7fc64ce12640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc628073200 0x7fc628075710 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.112+0000 7fc64ce12640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6480c8fc0 0x7fc6480cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.112+0000 7fc64ce12640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6481568a0 0x7fc648147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.112+0000 7fc64ce12640 1 --2- 172.21.15.150:0/1875871436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6480c35b0 0x7fc6481f31a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.112+0000 7fc64ce12640 1 -- 172.21.15.150:0/1875871436 >> 172.21.15.150:0/1875871436 conn(0x7fc648074b80 msgr2=0x7fc64815a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:56.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.112+0000 7fc64ce12640 1 -- 172.21.15.150:0/1875871436 shutdown_connections 2024-09-17T04:53:56.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.112+0000 7fc64ce12640 1 -- 172.21.15.150:0/1875871436 wait complete. 2024-09-17T04:53:56.138 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=4358 2024-09-17T04:53:56.138 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.d 2024-09-17T04:53:56.139 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-17T04:53:56.139 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:53:56.139 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:53:56.142 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:53:56.142 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:56.260 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-09-17T04:53:56.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.296+0000 7fbf07afe640 1 -- 172.21.15.150:0/795911911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf000c5d20 msgr2=0x7fbf00152030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:56.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.296+0000 7fbf07afe640 1 --2- 172.21.15.150:0/795911911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf000c5d20 0x7fbf00152030 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbf000c2cb0 tx=0x7fbefc01c810 comp rx=0 tx=0).stop 2024-09-17T04:53:56.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.296+0000 7fbf07afe640 1 -- 172.21.15.150:0/795911911 shutdown_connections 2024-09-17T04:53:56.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.296+0000 7fbf07afe640 1 --2- 172.21.15.150:0/795911911 >> v2:172.21.15.176:3300/0 conn(0x7fbf001582b0 0x7fbf001586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.296+0000 7fbf07afe640 1 --2- 172.21.15.150:0/795911911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf000c5d20 0x7fbf00152030 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.296+0000 7fbf07afe640 1 -- 172.21.15.150:0/795911911 >> 172.21.15.150:0/795911911 conn(0x7fbf000c91e0 msgr2=0x7fbf000c51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:56.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.296+0000 7fbf07afe640 1 -- 172.21.15.150:0/795911911 shutdown_connections 2024-09-17T04:53:56.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.296+0000 7fbf07afe640 1 -- 172.21.15.150:0/795911911 wait complete. 2024-09-17T04:53:56.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf07afe640 1 Processor -- start 2024-09-17T04:53:56.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf07afe640 1 -- start start 2024-09-17T04:53:56.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf07afe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf00073d60 0x7fbf00142640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:56.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf07afe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf000c5d20 0x7fbf00146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:56.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf07afe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf001582b0 0x7fbf0013f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:56.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf07afe640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbf000c4eb0 con 0x7fbf001582b0 2024-09-17T04:53:56.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf07afe640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbf000c4d30 con 0x7fbf000c5d20 2024-09-17T04:53:56.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf07afe640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbf00074880 con 0x7fbf00073d60 2024-09-17T04:53:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf0609b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf001582b0 0x7fbf0013f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf0609b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf001582b0 0x7fbf0013f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35884/0 (socket says 172.21.15.150:35884) 2024-09-17T04:53:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf0609b640 1 -- 172.21.15.150:0/2688660626 learned_addr learned my addr 172.21.15.150:0/2688660626 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf05099640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf000c5d20 0x7fbf00146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf0589a640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf00073d60 0x7fbf00142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf0609b640 1 -- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf00073d60 msgr2=0x7fbf00142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf0609b640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf00073d60 0x7fbf00142640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf0609b640 1 -- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf000c5d20 msgr2=0x7fbf00146930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf0609b640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf000c5d20 0x7fbf00146930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.300+0000 7fbf0609b640 1 -- 172.21.15.150:0/2688660626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0013fe10 con 0x7fbf001582b0 2024-09-17T04:53:56.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.304+0000 7fbf0609b640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf001582b0 0x7fbf0013f180 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbef800c760 tx=0x7fbef800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:56.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.304+0000 7fbef67fc640 1 -- 172.21.15.150:0/2688660626 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef8003db0 con 0x7fbf001582b0 2024-09-17T04:53:56.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.304+0000 7fbef67fc640 1 -- 172.21.15.150:0/2688660626 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbef800ce10 con 0x7fbf001582b0 2024-09-17T04:53:56.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.304+0000 7fbef67fc640 1 -- 172.21.15.150:0/2688660626 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef80052d0 con 0x7fbf001582b0 2024-09-17T04:53:56.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.304+0000 7fbf07afe640 1 -- 172.21.15.150:0/2688660626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf00140b20 con 0x7fbf001582b0 2024-09-17T04:53:56.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.304+0000 7fbf07afe640 1 -- 172.21.15.150:0/2688660626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf001f0290 con 0x7fbf001582b0 2024-09-17T04:53:56.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.304+0000 7fbf07afe640 1 -- 172.21.15.150:0/2688660626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf001f0670 con 0x7fbf001582b0 2024-09-17T04:53:56.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.304+0000 7fbef67fc640 1 -- 172.21.15.150:0/2688660626 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbef8005470 con 0x7fbf001582b0 2024-09-17T04:53:56.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.304+0000 7fbef67fc640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbee4073200 0x7fbee4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:56.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.304+0000 7fbef67fc640 1 -- 172.21.15.150:0/2688660626 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fbef8094360 con 0x7fbf001582b0 2024-09-17T04:53:56.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.304+0000 7fbf0589a640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbee4073200 0x7fbee4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:56.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.308+0000 7fbf0589a640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbee4073200 0x7fbee4075710 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0af000 tx=0x7fbefc01b050 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:56.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.312+0000 7fbef67fc640 1 -- 172.21.15.150:0/2688660626 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbf001f0670 con 0x7fbf001582b0 2024-09-17T04:53:56.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.524+0000 7fbf07afe640 1 -- 172.21.15.150:0/2688660626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf00152030 con 0x7fbf001582b0 2024-09-17T04:53:56.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.524+0000 7fbef67fc640 1 -- 172.21.15.150:0/2688660626 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5710 (secure 0 0 0) 0x7fbef805e8a0 con 0x7fbf001582b0 2024-09-17T04:53:56.526 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:56.526 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":35,"btime":"2024-09-17T04:53:53:928953+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:56.526 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 35 2024-09-17T04:53:56.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.528+0000 7fbf07afe640 1 -- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbee4073200 msgr2=0x7fbee4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:56.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.528+0000 7fbf07afe640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbee4073200 0x7fbee4075710 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0af000 tx=0x7fbefc01b050 comp rx=0 tx=0).stop 2024-09-17T04:53:56.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.528+0000 7fbf07afe640 1 -- 172.21.15.150:0/2688660626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf001582b0 msgr2=0x7fbf0013f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:56.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.528+0000 7fbf07afe640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf001582b0 0x7fbf0013f180 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbef800c760 tx=0x7fbef800cc20 comp rx=0 tx=0).stop 2024-09-17T04:53:56.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.528+0000 7fbf07afe640 1 -- 172.21.15.150:0/2688660626 shutdown_connections 2024-09-17T04:53:56.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.528+0000 7fbf07afe640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbee4073200 0x7fbee4075710 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.528+0000 7fbf07afe640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf000c5d20 0x7fbf00146930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.528+0000 7fbf07afe640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf001582b0 0x7fbf0013f180 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.528+0000 7fbf07afe640 1 --2- 172.21.15.150:0/2688660626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf00073d60 0x7fbf00142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:56.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.528+0000 7fbf07afe640 1 -- 172.21.15.150:0/2688660626 >> 172.21.15.150:0/2688660626 conn(0x7fbf000c91e0 msgr2=0x7fbf0006df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:56.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.528+0000 7fbf07afe640 1 -- 172.21.15.150:0/2688660626 shutdown_connections 2024-09-17T04:53:56.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:56.528+0000 7fbf07afe640 1 -- 172.21.15.150:0/2688660626 wait complete. 2024-09-17T04:53:56.552 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-17T04:53:57.762 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T04:53:57.760+0000 7fac8ef38640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:53:58.553 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:53:58.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.752+0000 7f12f3922640 1 -- 172.21.15.150:0/3621818244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12ec0c3900 msgr2=0x7f12ec074920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.752+0000 7f12f3922640 1 --2- 172.21.15.150:0/3621818244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12ec0c3900 0x7f12ec074920 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009900 tx=0x7f12dc01c8e0 comp rx=0 tx=0).stop 2024-09-17T04:53:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.752+0000 7f12f3922640 1 -- 172.21.15.150:0/3621818244 shutdown_connections 2024-09-17T04:53:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.752+0000 7f12f3922640 1 --2- 172.21.15.150:0/3621818244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12ec0c3900 0x7f12ec074920 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.752+0000 7f12f3922640 1 --2- 172.21.15.150:0/3621818244 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12ec0c4250 0x7f12ec153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.752+0000 7f12f3922640 1 -- 172.21.15.150:0/3621818244 >> 172.21.15.150:0/3621818244 conn(0x7f12ec06e360 msgr2=0x7f12ec06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.752+0000 7f12f3922640 1 -- 172.21.15.150:0/3621818244 shutdown_connections 2024-09-17T04:53:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.752+0000 7f12f3922640 1 -- 172.21.15.150:0/3621818244 wait complete. 2024-09-17T04:53:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f3922640 1 Processor -- start 2024-09-17T04:53:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f3922640 1 -- start start 2024-09-17T04:53:58.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f3922640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12ec074e60 0x7f12ec14fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:58.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f3922640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12ec0c3900 0x7f12ec1f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f3922640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12ec0c4250 0x7f12ec149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f3922640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec072f30 con 0x7f12ec074e60 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f3922640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec072db0 con 0x7f12ec0c3900 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f3922640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f12ec160f90 con 0x7f12ec0c4250 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f16be640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12ec074e60 0x7f12ec14fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f16be640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12ec074e60 0x7f12ec14fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35906/0 (socket says 172.21.15.150:35906) 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f16be640 1 -- 172.21.15.150:0/4128866969 learned_addr learned my addr 172.21.15.150:0/4128866969 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f1ebf640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12ec0c4250 0x7f12ec149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f0ebd640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12ec0c3900 0x7f12ec1f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f16be640 1 -- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12ec0c4250 msgr2=0x7f12ec149120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f16be640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12ec0c4250 0x7f12ec149120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f16be640 1 -- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12ec0c3900 msgr2=0x7f12ec1f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:58.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f16be640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12ec0c3900 0x7f12ec1f33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f16be640 1 -- 172.21.15.150:0/4128866969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ec0a84d0 con 0x7f12ec074e60 2024-09-17T04:53:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.756+0000 7f12f16be640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12ec074e60 0x7f12ec14fb80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f12e80058d0 tx=0x7f12e800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.760+0000 7f12e1ffb640 1 -- 172.21.15.150:0/4128866969 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8017070 con 0x7f12ec074e60 2024-09-17T04:53:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.760+0000 7f12e1ffb640 1 -- 172.21.15.150:0/4128866969 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12e8009d20 con 0x7f12ec074e60 2024-09-17T04:53:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.760+0000 7f12f3922640 1 -- 172.21.15.150:0/4128866969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12ec155a70 con 0x7f12ec074e60 2024-09-17T04:53:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.760+0000 7f12e1ffb640 1 -- 172.21.15.150:0/4128866969 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e80106c0 con 0x7f12ec074e60 2024-09-17T04:53:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.760+0000 7f12f3922640 1 -- 172.21.15.150:0/4128866969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12ec0ae710 con 0x7f12ec074e60 2024-09-17T04:53:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.760+0000 7f12f3922640 1 -- 172.21.15.150:0/4128866969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12ec074920 con 0x7f12ec074e60 2024-09-17T04:53:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.760+0000 7f12e1ffb640 1 -- 172.21.15.150:0/4128866969 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f12e801a070 con 0x7f12ec074e60 2024-09-17T04:53:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.760+0000 7f12e1ffb640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f12d0073280 0x7f12d0075790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:53:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.760+0000 7f12e1ffb640 1 -- 172.21.15.150:0/4128866969 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7f12e8094820 con 0x7f12ec074e60 2024-09-17T04:53:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.760+0000 7f12f0ebd640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f12d0073280 0x7f12d0075790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:53:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.764+0000 7f12f0ebd640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f12d0073280 0x7f12d0075790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0af000 tx=0x7f12dc01b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:53:58.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:58.768+0000 7f12e1ffb640 1 -- 172.21.15.150:0/4128866969 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f12e805f490 con 0x7f12ec074e60 2024-09-17T04:53:59.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 -- 172.21.15.150:0/4128866969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12ec156f10 con 0x7f12ec074e60 2024-09-17T04:53:59.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12e1ffb640 1 -- 172.21.15.150:0/4128866969 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6538 (secure 0 0 0) 0x7f12e805ed90 con 0x7f12ec074e60 2024-09-17T04:53:59.012 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:53:59.012 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":36,"btime":"2024-09-17T04:53:56:949667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:53:59.013 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 36 2024-09-17T04:53:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 -- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f12d0073280 msgr2=0x7f12d0075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f12d0073280 0x7f12d0075790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f12dc0af000 tx=0x7f12dc01b120 comp rx=0 tx=0).stop 2024-09-17T04:53:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 -- 172.21.15.150:0/4128866969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12ec074e60 msgr2=0x7f12ec14fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:53:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12ec074e60 0x7f12ec14fb80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f12e80058d0 tx=0x7f12e800dde0 comp rx=0 tx=0).stop 2024-09-17T04:53:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 -- 172.21.15.150:0/4128866969 shutdown_connections 2024-09-17T04:53:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f12d0073280 0x7f12d0075790 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12ec0c3900 0x7f12ec1f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:59.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12ec0c4250 0x7f12ec149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:59.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 --2- 172.21.15.150:0/4128866969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12ec074e60 0x7f12ec14fb80 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:53:59.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 -- 172.21.15.150:0/4128866969 >> 172.21.15.150:0/4128866969 conn(0x7f12ec06e360 msgr2=0x7f12ec06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:53:59.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 -- 172.21.15.150:0/4128866969 shutdown_connections 2024-09-17T04:53:59.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:53:59.012+0000 7f12f3922640 1 -- 172.21.15.150:0/4128866969 wait complete. 2024-09-17T04:53:59.035 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-17T04:53:59.035 INFO:tasks.mds_thrash.fs.[a]:waiting for 54 secs before thrashing 2024-09-17T04:53:59.363 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:53:59.366 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:54:29.529 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:54:29.533 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:54:40.009 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:40.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.204+0000 7fa90afde640 1 -- 172.21.15.150:0/3795295556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8f4004810 msgr2=0x7fa8f4004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:40.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.204+0000 7fa90afde640 1 --2- 172.21.15.150:0/3795295556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8f4004810 0x7fa8f4004c20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0009910 tx=0x7fa8f001c700 comp rx=0 tx=0).stop 2024-09-17T04:54:40.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.204+0000 7fa90afde640 1 -- 172.21.15.150:0/3795295556 shutdown_connections 2024-09-17T04:54:40.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.204+0000 7fa90afde640 1 --2- 172.21.15.150:0/3795295556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8f4004810 0x7fa8f4004c20 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:40.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.204+0000 7fa90afde640 1 -- 172.21.15.150:0/3795295556 >> 172.21.15.150:0/3795295556 conn(0x7fa9040c5ae0 msgr2=0x7fa904073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:40.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.204+0000 7fa90afde640 1 -- 172.21.15.150:0/3795295556 shutdown_connections 2024-09-17T04:54:40.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa90afde640 1 -- 172.21.15.150:0/3795295556 wait complete. 2024-09-17T04:54:40.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa90afde640 1 Processor -- start 2024-09-17T04:54:40.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa90afde640 1 -- start start 2024-09-17T04:54:40.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa90afde640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8f4004810 0x7fa9040be740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:40.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa90afde640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9041553f0 0x7fa904149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:40.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa90afde640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa904155800 0x7fa9041eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:40.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa90afde640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa904070610 con 0x7fa904155800 2024-09-17T04:54:40.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa90afde640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa904070790 con 0x7fa8f4004810 2024-09-17T04:54:40.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa90afde640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa9040ab140 con 0x7fa9041553f0 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa90957b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa904155800 0x7fa9041eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa90957b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa904155800 0x7fa9041eeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42784/0 (socket says 172.21.15.150:42784) 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa90957b640 1 -- 172.21.15.150:0/2325033176 learned_addr learned my addr 172.21.15.150:0/2325033176 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.208+0000 7fa908d7a640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8f4004810 0x7fa9040be740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa903fff640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9041553f0 0x7fa904149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa90957b640 1 -- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9041553f0 msgr2=0x7fa904149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa90957b640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9041553f0 0x7fa904149f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa90957b640 1 -- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8f4004810 msgr2=0x7fa9040be740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa90957b640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8f4004810 0x7fa9040be740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa90957b640 1 -- 172.21.15.150:0/2325033176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9040bd6e0 con 0x7fa904155800 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa908d7a640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8f4004810 0x7fa9040be740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa90957b640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa904155800 0x7fa9041eeda0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa904075620 tx=0x7fa8fc00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa9017fa640 1 -- 172.21.15.150:0/2325033176 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc00cc20 con 0x7fa904155800 2024-09-17T04:54:40.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa9017fa640 1 -- 172.21.15.150:0/2325033176 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8fc017070 con 0x7fa904155800 2024-09-17T04:54:40.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa90afde640 1 -- 172.21.15.150:0/2325033176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9040abb80 con 0x7fa904155800 2024-09-17T04:54:40.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa9017fa640 1 -- 172.21.15.150:0/2325033176 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc0116c0 con 0x7fa904155800 2024-09-17T04:54:40.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa90afde640 1 -- 172.21.15.150:0/2325033176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9041ef9a0 con 0x7fa904155800 2024-09-17T04:54:40.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa90afde640 1 -- 172.21.15.150:0/2325033176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa904150d00 con 0x7fa904155800 2024-09-17T04:54:40.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa9017fa640 1 -- 172.21.15.150:0/2325033176 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa8fc011860 con 0x7fa904155800 2024-09-17T04:54:40.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa9017fa640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa8e8073200 0x7fa8e8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:40.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.212+0000 7fa9017fa640 1 -- 172.21.15.150:0/2325033176 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fa8fc095690 con 0x7fa904155800 2024-09-17T04:54:40.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.216+0000 7fa908d7a640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa8e8073200 0x7fa8e8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:40.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.216+0000 7fa908d7a640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa8e8073200 0x7fa8e8075710 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8004720 tx=0x7fa8f8006cb0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:40.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.216+0000 7fa9017fa640 1 -- 172.21.15.150:0/2325033176 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa8fc060300 con 0x7fa904155800 2024-09-17T04:54:40.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.488+0000 7fa90afde640 1 -- 172.21.15.150:0/2325033176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa90414a8e0 con 0x7fa904155800 2024-09-17T04:54:40.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.488+0000 7fa9017fa640 1 -- 172.21.15.150:0/2325033176 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6538 (secure 0 0 0) 0x7fa8fc05fc00 con 0x7fa904155800 2024-09-17T04:54:40.489 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:40.489 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":36,"btime":"2024-09-17T04:53:56:949667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:54:40.489 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 36 2024-09-17T04:54:40.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.488+0000 7fa90afde640 1 -- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa8e8073200 msgr2=0x7fa8e8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:40.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.488+0000 7fa90afde640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa8e8073200 0x7fa8e8075710 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8004720 tx=0x7fa8f8006cb0 comp rx=0 tx=0).stop 2024-09-17T04:54:40.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.488+0000 7fa90afde640 1 -- 172.21.15.150:0/2325033176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa904155800 msgr2=0x7fa9041eeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:40.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.488+0000 7fa90afde640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa904155800 0x7fa9041eeda0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa904075620 tx=0x7fa8fc00ecf0 comp rx=0 tx=0).stop 2024-09-17T04:54:40.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.488+0000 7fa90afde640 1 -- 172.21.15.150:0/2325033176 shutdown_connections 2024-09-17T04:54:40.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.488+0000 7fa90afde640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa8e8073200 0x7fa8e8075710 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:40.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.488+0000 7fa90afde640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa904155800 0x7fa9041eeda0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:40.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.488+0000 7fa90afde640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9041553f0 0x7fa904149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:40.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.488+0000 7fa90afde640 1 --2- 172.21.15.150:0/2325033176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8f4004810 0x7fa9040be740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:40.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.488+0000 7fa90afde640 1 -- 172.21.15.150:0/2325033176 >> 172.21.15.150:0/2325033176 conn(0x7fa9040c5ae0 msgr2=0x7fa9040aa530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:40.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.492+0000 7fa90afde640 1 -- 172.21.15.150:0/2325033176 shutdown_connections 2024-09-17T04:54:40.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.492+0000 7fa90afde640 1 -- 172.21.15.150:0/2325033176 wait complete. 2024-09-17T04:54:40.510 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-17T04:54:40.510 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:54:40.511 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:54:40.523 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:54:40.523 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:40.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.744+0000 7fb23ecf1640 1 -- 172.21.15.150:0/1540498901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380aa9b0 msgr2=0x7fb2380aadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:40.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.744+0000 7fb23ecf1640 1 --2- 172.21.15.150:0/1540498901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380aa9b0 0x7fb2380aadc0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb230009bc0 tx=0x7fb23001ca80 comp rx=0 tx=0).stop 2024-09-17T04:54:40.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.744+0000 7fb23ecf1640 1 -- 172.21.15.150:0/1540498901 shutdown_connections 2024-09-17T04:54:40.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.744+0000 7fb23ecf1640 1 --2- 172.21.15.150:0/1540498901 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380aa9b0 0x7fb2380aadc0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:40.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.744+0000 7fb23ecf1640 1 -- 172.21.15.150:0/1540498901 >> 172.21.15.150:0/1540498901 conn(0x7fb2380c5ae0 msgr2=0x7fb238073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:40.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.744+0000 7fb23ecf1640 1 -- 172.21.15.150:0/1540498901 shutdown_connections 2024-09-17T04:54:40.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.744+0000 7fb23ecf1640 1 -- 172.21.15.150:0/1540498901 wait complete. 2024-09-17T04:54:40.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb23ecf1640 1 Processor -- start 2024-09-17T04:54:40.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb23ecf1640 1 -- start start 2024-09-17T04:54:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb23ecf1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2380aa9b0 0x7fb2380ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb23ecf1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb238154200 0x7fb2381e7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb23ecf1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb238153390 0x7fb238144250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb23ecf1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb23806fca0 con 0x7fb238154200 2024-09-17T04:54:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb23ecf1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb238070820 con 0x7fb238153390 2024-09-17T04:54:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb23ecf1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb23806fe20 con 0x7fb2380aa9b0 2024-09-17T04:54:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb237fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb238154200 0x7fb2381e7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb237fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb238154200 0x7fb2381e7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42812/0 (socket says 172.21.15.150:42812) 2024-09-17T04:54:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb237fff640 1 -- 172.21.15.150:0/4123144875 learned_addr learned my addr 172.21.15.150:0/4123144875 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb23d28e640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb238153390 0x7fb238144250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb23ca8d640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2380aa9b0 0x7fb2380ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb237fff640 1 -- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2380aa9b0 msgr2=0x7fb2380ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb237fff640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2380aa9b0 0x7fb2380ab5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb237fff640 1 -- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb238153390 msgr2=0x7fb238144250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb237fff640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb238153390 0x7fb238144250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb237fff640 1 -- 172.21.15.150:0/4123144875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2381ed250 con 0x7fb238154200 2024-09-17T04:54:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb237fff640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb238154200 0x7fb2381e7110 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb22400c3c0 tx=0x7fb22400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb2357fa640 1 -- 172.21.15.150:0/4123144875 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224011790 con 0x7fb238154200 2024-09-17T04:54:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb2357fa640 1 -- 172.21.15.150:0/4123144875 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb224017070 con 0x7fb238154200 2024-09-17T04:54:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb23ecf1640 1 -- 172.21.15.150:0/4123144875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2381ed530 con 0x7fb238154200 2024-09-17T04:54:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb2357fa640 1 -- 172.21.15.150:0/4123144875 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224011cc0 con 0x7fb238154200 2024-09-17T04:54:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.748+0000 7fb23ecf1640 1 -- 172.21.15.150:0/4123144875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2381edaf0 con 0x7fb238154200 2024-09-17T04:54:40.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.752+0000 7fb23ecf1640 1 -- 172.21.15.150:0/4123144875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2380ad8f0 con 0x7fb238154200 2024-09-17T04:54:40.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.752+0000 7fb2357fa640 1 -- 172.21.15.150:0/4123144875 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb22401c460 con 0x7fb238154200 2024-09-17T04:54:40.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.752+0000 7fb2357fa640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb21c073200 0x7fb21c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:40.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.752+0000 7fb2357fa640 1 -- 172.21.15.150:0/4123144875 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fb224094390 con 0x7fb238154200 2024-09-17T04:54:40.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.752+0000 7fb23ca8d640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb21c073200 0x7fb21c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:40.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.752+0000 7fb23ca8d640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb21c073200 0x7fb21c075710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb22c000c10 tx=0x7fb22c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:40.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:40.756+0000 7fb2357fa640 1 -- 172.21.15.150:0/4123144875 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb22405efb0 con 0x7fb238154200 2024-09-17T04:54:41.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.020+0000 7fb23ecf1640 1 -- 172.21.15.150:0/4123144875 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2380adb30 con 0x7fb238154200 2024-09-17T04:54:41.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.020+0000 7fb2357fa640 1 -- 172.21.15.150:0/4123144875 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6538 (secure 0 0 0) 0x7fb22405e8b0 con 0x7fb238154200 2024-09-17T04:54:41.023 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:41.023 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":36,"btime":"2024-09-17T04:53:56:949667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:54:41.024 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 36 2024-09-17T04:54:41.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.024+0000 7fb23ecf1640 1 -- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb21c073200 msgr2=0x7fb21c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:41.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.024+0000 7fb23ecf1640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb21c073200 0x7fb21c075710 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb22c000c10 tx=0x7fb22c009040 comp rx=0 tx=0).stop 2024-09-17T04:54:41.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.024+0000 7fb23ecf1640 1 -- 172.21.15.150:0/4123144875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb238154200 msgr2=0x7fb2381e7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:41.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.024+0000 7fb23ecf1640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb238154200 0x7fb2381e7110 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb22400c3c0 tx=0x7fb22400c880 comp rx=0 tx=0).stop 2024-09-17T04:54:41.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.024+0000 7fb23ecf1640 1 -- 172.21.15.150:0/4123144875 shutdown_connections 2024-09-17T04:54:41.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.024+0000 7fb23ecf1640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb21c073200 0x7fb21c075710 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:41.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.024+0000 7fb23ecf1640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb238153390 0x7fb238144250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:41.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.024+0000 7fb23ecf1640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb238154200 0x7fb2381e7110 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:41.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.024+0000 7fb23ecf1640 1 --2- 172.21.15.150:0/4123144875 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2380aa9b0 0x7fb2380ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:41.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.024+0000 7fb23ecf1640 1 -- 172.21.15.150:0/4123144875 >> 172.21.15.150:0/4123144875 conn(0x7fb2380c5ae0 msgr2=0x7fb23806d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:41.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.024+0000 7fb23ecf1640 1 -- 172.21.15.150:0/4123144875 shutdown_connections 2024-09-17T04:54:41.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:41.024+0000 7fb23ecf1640 1 -- 172.21.15.150:0/4123144875 wait complete. 2024-09-17T04:54:41.052 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:54:43.053 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.244+0000 7eff47757640 1 -- 172.21.15.150:0/4288448597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff400c3900 msgr2=0x7eff40074920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.244+0000 7eff47757640 1 --2- 172.21.15.150:0/4288448597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff400c3900 0x7eff40074920 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7eff30009900 tx=0x7eff3001c8e0 comp rx=0 tx=0).stop 2024-09-17T04:54:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.244+0000 7eff47757640 1 -- 172.21.15.150:0/4288448597 shutdown_connections 2024-09-17T04:54:43.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.244+0000 7eff47757640 1 --2- 172.21.15.150:0/4288448597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff400c3900 0x7eff40074920 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:43.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.244+0000 7eff47757640 1 --2- 172.21.15.150:0/4288448597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff400c4250 0x7eff40153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:43.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.244+0000 7eff47757640 1 -- 172.21.15.150:0/4288448597 >> 172.21.15.150:0/4288448597 conn(0x7eff4006e360 msgr2=0x7eff4006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:43.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.244+0000 7eff47757640 1 -- 172.21.15.150:0/4288448597 shutdown_connections 2024-09-17T04:54:43.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.244+0000 7eff47757640 1 -- 172.21.15.150:0/4288448597 wait complete. 2024-09-17T04:54:43.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.244+0000 7eff47757640 1 Processor -- start 2024-09-17T04:54:43.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.244+0000 7eff47757640 1 -- start start 2024-09-17T04:54:43.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff47757640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff40074e60 0x7eff4014dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff47757640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff400c3900 0x7eff4014fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff47757640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff400c4250 0x7eff40147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff47757640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7eff40072f30 con 0x7eff400c4250 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff47757640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7eff40072db0 con 0x7eff400c3900 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff47757640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7eff40160f90 con 0x7eff40074e60 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff45cf4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff400c4250 0x7eff40147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff45cf4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff400c4250 0x7eff40147ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42822/0 (socket says 172.21.15.150:42822) 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff45cf4640 1 -- 172.21.15.150:0/1687405342 learned_addr learned my addr 172.21.15.150:0/1687405342 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff44cf2640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff400c3900 0x7eff4014fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff454f3640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff40074e60 0x7eff4014dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff45cf4640 1 -- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff40074e60 msgr2=0x7eff4014dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff45cf4640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff40074e60 0x7eff4014dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff45cf4640 1 -- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff400c3900 msgr2=0x7eff4014fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:43.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff45cf4640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff400c3900 0x7eff4014fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:43.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff45cf4640 1 -- 172.21.15.150:0/1687405342 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff401490b0 con 0x7eff400c4250 2024-09-17T04:54:43.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff45cf4640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff400c4250 0x7eff40147ec0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7eff3800c760 tx=0x7eff3800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:43.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff35ffb640 1 -- 172.21.15.150:0/1687405342 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff38003b80 con 0x7eff400c4250 2024-09-17T04:54:43.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff35ffb640 1 -- 172.21.15.150:0/1687405342 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff38003d20 con 0x7eff400c4250 2024-09-17T04:54:43.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff35ffb640 1 -- 172.21.15.150:0/1687405342 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3800aab0 con 0x7eff400c4250 2024-09-17T04:54:43.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff47757640 1 -- 172.21.15.150:0/1687405342 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff40148b60 con 0x7eff400c4250 2024-09-17T04:54:43.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff47757640 1 -- 172.21.15.150:0/1687405342 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff400ac740 con 0x7eff400c4250 2024-09-17T04:54:43.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff47757640 1 -- 172.21.15.150:0/1687405342 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff40074920 con 0x7eff400c4250 2024-09-17T04:54:43.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff35ffb640 1 -- 172.21.15.150:0/1687405342 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7eff3800ac50 con 0x7eff400c4250 2024-09-17T04:54:43.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff35ffb640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff24073230 0x7eff24075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:43.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.248+0000 7eff35ffb640 1 -- 172.21.15.150:0/1687405342 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7eff38094f10 con 0x7eff400c4250 2024-09-17T04:54:43.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.252+0000 7eff454f3640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff24073230 0x7eff24075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:43.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.252+0000 7eff454f3640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff24073230 0x7eff24075740 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7eff3c009ad0 tx=0x7eff3c0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:43.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.256+0000 7eff35ffb640 1 -- 172.21.15.150:0/1687405342 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7eff3805faa0 con 0x7eff400c4250 2024-09-17T04:54:43.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.576+0000 7eff47757640 1 -- 172.21.15.150:0/1687405342 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff40156f10 con 0x7eff400c4250 2024-09-17T04:54:43.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.576+0000 7eff35ffb640 1 -- 172.21.15.150:0/1687405342 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6538 (secure 0 0 0) 0x7eff3805f3a0 con 0x7eff400c4250 2024-09-17T04:54:43.579 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:43.579 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":36,"btime":"2024-09-17T04:53:56:949667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:54:43.579 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 36 2024-09-17T04:54:43.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.580+0000 7eff47757640 1 -- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff24073230 msgr2=0x7eff24075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:43.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.580+0000 7eff47757640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff24073230 0x7eff24075740 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7eff3c009ad0 tx=0x7eff3c0091e0 comp rx=0 tx=0).stop 2024-09-17T04:54:43.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.580+0000 7eff47757640 1 -- 172.21.15.150:0/1687405342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff400c4250 msgr2=0x7eff40147ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:43.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.580+0000 7eff47757640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff400c4250 0x7eff40147ec0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7eff3800c760 tx=0x7eff3800cc20 comp rx=0 tx=0).stop 2024-09-17T04:54:43.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.580+0000 7eff47757640 1 -- 172.21.15.150:0/1687405342 shutdown_connections 2024-09-17T04:54:43.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.580+0000 7eff47757640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff24073230 0x7eff24075740 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:43.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.580+0000 7eff47757640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff400c3900 0x7eff4014fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:43.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.580+0000 7eff47757640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff400c4250 0x7eff40147ec0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:43.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.580+0000 7eff47757640 1 --2- 172.21.15.150:0/1687405342 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff40074e60 0x7eff4014dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:43.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.580+0000 7eff47757640 1 -- 172.21.15.150:0/1687405342 >> 172.21.15.150:0/1687405342 conn(0x7eff4006e360 msgr2=0x7eff4006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:43.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.580+0000 7eff47757640 1 -- 172.21.15.150:0/1687405342 shutdown_connections 2024-09-17T04:54:43.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:43.580+0000 7eff47757640 1 -- 172.21.15.150:0/1687405342 wait complete. 2024-09-17T04:54:43.603 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:54:45.604 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:45.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.784+0000 7fbf7df3b640 1 -- 172.21.15.150:0/3904926330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78158690 msgr2=0x7fbf78151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:45.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.784+0000 7fbf7df3b640 1 --2- 172.21.15.150:0/3904926330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78158690 0x7fbf78151ba0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbf780c8040 tx=0x7fbf6801c820 comp rx=0 tx=0).stop 2024-09-17T04:54:45.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.784+0000 7fbf7df3b640 1 -- 172.21.15.150:0/3904926330 shutdown_connections 2024-09-17T04:54:45.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.784+0000 7fbf7df3b640 1 --2- 172.21.15.150:0/3904926330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78158690 0x7fbf78151ba0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:45.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.784+0000 7fbf7df3b640 1 -- 172.21.15.150:0/3904926330 >> 172.21.15.150:0/3904926330 conn(0x7fbf7806d670 msgr2=0x7fbf780ab830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:45.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.784+0000 7fbf7df3b640 1 -- 172.21.15.150:0/3904926330 shutdown_connections 2024-09-17T04:54:45.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.784+0000 7fbf7df3b640 1 -- 172.21.15.150:0/3904926330 wait complete. 2024-09-17T04:54:45.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7df3b640 1 Processor -- start 2024-09-17T04:54:45.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7df3b640 1 -- start start 2024-09-17T04:54:45.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7df3b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf780a91c0 0x7fbf780aa050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:45.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7df3b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf781520e0 0x7fbf78156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:45.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7df3b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78158690 0x7fbf780bc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:45.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7df3b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbf78162a40 con 0x7fbf78158690 2024-09-17T04:54:45.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7df3b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbf780ac370 con 0x7fbf780a91c0 2024-09-17T04:54:45.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7df3b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbf78162bc0 con 0x7fbf781520e0 2024-09-17T04:54:45.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7d73a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78158690 0x7fbf780bc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:45.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7d73a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78158690 0x7fbf780bc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42826/0 (socket says 172.21.15.150:42826) 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7d73a640 1 -- 172.21.15.150:0/864941981 learned_addr learned my addr 172.21.15.150:0/864941981 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7cf39640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf780a91c0 0x7fbf780aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf6ffff640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf781520e0 0x7fbf78156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7d73a640 1 -- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf781520e0 msgr2=0x7fbf78156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7d73a640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf781520e0 0x7fbf78156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7d73a640 1 -- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf780a91c0 msgr2=0x7fbf780aa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7d73a640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf780a91c0 0x7fbf780aa050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7d73a640 1 -- 172.21.15.150:0/864941981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf780ab630 con 0x7fbf78158690 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7d73a640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78158690 0x7fbf780bc0a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbf7000b9a0 tx=0x7fbf7000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf6d7fa640 1 -- 172.21.15.150:0/864941981 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf70019070 con 0x7fbf78158690 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf6d7fa640 1 -- 172.21.15.150:0/864941981 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf7000cc90 con 0x7fbf78158690 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf6d7fa640 1 -- 172.21.15.150:0/864941981 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf700146a0 con 0x7fbf78158690 2024-09-17T04:54:45.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.788+0000 7fbf7df3b640 1 -- 172.21.15.150:0/864941981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf781eeb00 con 0x7fbf78158690 2024-09-17T04:54:45.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.792+0000 7fbf7df3b640 1 -- 172.21.15.150:0/864941981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf780aa590 con 0x7fbf78158690 2024-09-17T04:54:45.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.792+0000 7fbf7df3b640 1 -- 172.21.15.150:0/864941981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf781eed40 con 0x7fbf78158690 2024-09-17T04:54:45.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.792+0000 7fbf6d7fa640 1 -- 172.21.15.150:0/864941981 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbf7001c070 con 0x7fbf78158690 2024-09-17T04:54:45.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.792+0000 7fbf6d7fa640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf64073230 0x7fbf64075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:45.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.792+0000 7fbf6d7fa640 1 -- 172.21.15.150:0/864941981 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fbf700943e0 con 0x7fbf78158690 2024-09-17T04:54:45.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.792+0000 7fbf7cf39640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf64073230 0x7fbf64075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:45.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.792+0000 7fbf7cf39640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf64073230 0x7fbf64075740 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbf740015f0 tx=0x7fbf74009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:45.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:45.796+0000 7fbf6d7fa640 1 -- 172.21.15.150:0/864941981 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbf781eed40 con 0x7fbf78158690 2024-09-17T04:54:46.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.096+0000 7fbf7df3b640 1 -- 172.21.15.150:0/864941981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf78151ba0 con 0x7fbf78158690 2024-09-17T04:54:46.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.096+0000 7fbf6d7fa640 1 -- 172.21.15.150:0/864941981 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6538 (secure 0 0 0) 0x7fbf7005eee0 con 0x7fbf78158690 2024-09-17T04:54:46.098 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:46.098 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":36,"btime":"2024-09-17T04:53:56:949667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:54:46.099 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 36 2024-09-17T04:54:46.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.100+0000 7fbf7df3b640 1 -- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf64073230 msgr2=0x7fbf64075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:46.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.100+0000 7fbf7df3b640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf64073230 0x7fbf64075740 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fbf740015f0 tx=0x7fbf74009040 comp rx=0 tx=0).stop 2024-09-17T04:54:46.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.100+0000 7fbf7df3b640 1 -- 172.21.15.150:0/864941981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78158690 msgr2=0x7fbf780bc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:46.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.100+0000 7fbf7df3b640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78158690 0x7fbf780bc0a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbf7000b9a0 tx=0x7fbf7000be60 comp rx=0 tx=0).stop 2024-09-17T04:54:46.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.100+0000 7fbf7df3b640 1 -- 172.21.15.150:0/864941981 shutdown_connections 2024-09-17T04:54:46.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.100+0000 7fbf7df3b640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf64073230 0x7fbf64075740 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:46.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.100+0000 7fbf7df3b640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf78158690 0x7fbf780bc0a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:46.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.100+0000 7fbf7df3b640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf781520e0 0x7fbf78156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:46.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.100+0000 7fbf7df3b640 1 --2- 172.21.15.150:0/864941981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf780a91c0 0x7fbf780aa050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:46.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.100+0000 7fbf7df3b640 1 -- 172.21.15.150:0/864941981 >> 172.21.15.150:0/864941981 conn(0x7fbf7806d670 msgr2=0x7fbf78151140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:46.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.100+0000 7fbf7df3b640 1 -- 172.21.15.150:0/864941981 shutdown_connections 2024-09-17T04:54:46.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:46.100+0000 7fbf7df3b640 1 -- 172.21.15.150:0/864941981 wait complete. 2024-09-17T04:54:46.122 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:54:48.123 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.303+0000 7f17c45f5640 1 -- 172.21.15.150:0/194262385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17bc157c00 msgr2=0x7f17bc158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:48.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.303+0000 7f17c45f5640 1 --2- 172.21.15.150:0/194262385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17bc157c00 0x7f17bc158010 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009bc0 tx=0x7f17b401cba0 comp rx=0 tx=0).stop 2024-09-17T04:54:48.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.303+0000 7f17c45f5640 1 -- 172.21.15.150:0/194262385 shutdown_connections 2024-09-17T04:54:48.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.303+0000 7f17c45f5640 1 --2- 172.21.15.150:0/194262385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17bc157c00 0x7f17bc158010 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:48.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.303+0000 7f17c45f5640 1 --2- 172.21.15.150:0/194262385 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17bc071670 0x7f17bc0c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:48.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.303+0000 7f17c45f5640 1 -- 172.21.15.150:0/194262385 >> 172.21.15.150:0/194262385 conn(0x7f17bc0ca460 msgr2=0x7f17bc0c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:48.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.303+0000 7f17c45f5640 1 -- 172.21.15.150:0/194262385 shutdown_connections 2024-09-17T04:54:48.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.303+0000 7f17c45f5640 1 -- 172.21.15.150:0/194262385 wait complete. 2024-09-17T04:54:48.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.303+0000 7f17c45f5640 1 Processor -- start 2024-09-17T04:54:48.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.303+0000 7f17c45f5640 1 -- start start 2024-09-17T04:54:48.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c45f5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17bc071670 0x7f17bc14d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:48.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c45f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17bc14e490 0x7f17bc1f06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:48.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c45f5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17bc157c00 0x7f17bc13f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:48.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c45f5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc14ea20 con 0x7f17bc14e490 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c45f5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc14e8a0 con 0x7f17bc157c00 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c45f5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f17bc072180 con 0x7f17bc071670 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c1b90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17bc14e490 0x7f17bc1f06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c1b90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17bc14e490 0x7f17bc1f06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42840/0 (socket says 172.21.15.150:42840) 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c1b90640 1 -- 172.21.15.150:0/3166826832 learned_addr learned my addr 172.21.15.150:0/3166826832 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c2391640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17bc071670 0x7f17bc14d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c2b92640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17bc157c00 0x7f17bc13f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c1b90640 1 -- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17bc071670 msgr2=0x7f17bc14d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c1b90640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17bc071670 0x7f17bc14d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c1b90640 1 -- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17bc157c00 msgr2=0x7f17bc13f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c1b90640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17bc157c00 0x7f17bc13f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c1b90640 1 -- 172.21.15.150:0/3166826832 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17bc140780 con 0x7f17bc14e490 2024-09-17T04:54:48.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c1b90640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17bc14e490 0x7f17bc1f06d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f17b000c3a0 tx=0x7f17b000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:48.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17aeffd640 1 -- 172.21.15.150:0/3166826832 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b0017070 con 0x7f17bc14e490 2024-09-17T04:54:48.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17aeffd640 1 -- 172.21.15.150:0/3166826832 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17b0003600 con 0x7f17bc14e490 2024-09-17T04:54:48.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c45f5640 1 -- 172.21.15.150:0/3166826832 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17bc147760 con 0x7f17bc14e490 2024-09-17T04:54:48.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17aeffd640 1 -- 172.21.15.150:0/3166826832 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b0003a70 con 0x7f17bc14e490 2024-09-17T04:54:48.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c45f5640 1 -- 172.21.15.150:0/3166826832 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17bc14d970 con 0x7f17bc14e490 2024-09-17T04:54:48.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17c45f5640 1 -- 172.21.15.150:0/3166826832 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17bc0c5680 con 0x7f17bc14e490 2024-09-17T04:54:48.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.307+0000 7f17aeffd640 1 -- 172.21.15.150:0/3166826832 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f17b001a070 con 0x7f17bc14e490 2024-09-17T04:54:48.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.311+0000 7f17aeffd640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f17a0073250 0x7f17a0075760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:48.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.311+0000 7f17aeffd640 1 -- 172.21.15.150:0/3166826832 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7f17b0093800 con 0x7f17bc14e490 2024-09-17T04:54:48.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.311+0000 7f17c2391640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f17a0073250 0x7f17a0075760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:48.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.311+0000 7f17c2391640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f17a0073250 0x7f17a0075760 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f17b800bb50 tx=0x7f17b800b2e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:48.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.315+0000 7f17aeffd640 1 -- 172.21.15.150:0/3166826832 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f17b005deb0 con 0x7f17bc14e490 2024-09-17T04:54:48.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.587+0000 7f17c45f5640 1 -- 172.21.15.150:0/3166826832 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17bc071c80 con 0x7f17bc14e490 2024-09-17T04:54:48.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17aeffd640 1 -- 172.21.15.150:0/3166826832 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6538 (secure 0 0 0) 0x7f17b00628e0 con 0x7f17bc14e490 2024-09-17T04:54:48.591 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:48.592 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":36,"btime":"2024-09-17T04:53:56:949667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:54:48.592 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 36 2024-09-17T04:54:48.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17c45f5640 1 -- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f17a0073250 msgr2=0x7f17a0075760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:48.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17c45f5640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f17a0073250 0x7f17a0075760 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f17b800bb50 tx=0x7f17b800b2e0 comp rx=0 tx=0).stop 2024-09-17T04:54:48.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17c45f5640 1 -- 172.21.15.150:0/3166826832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17bc14e490 msgr2=0x7f17bc1f06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:48.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17c45f5640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17bc14e490 0x7f17bc1f06d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f17b000c3a0 tx=0x7f17b000c860 comp rx=0 tx=0).stop 2024-09-17T04:54:48.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17c45f5640 1 -- 172.21.15.150:0/3166826832 shutdown_connections 2024-09-17T04:54:48.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17c45f5640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f17a0073250 0x7f17a0075760 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:48.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17c45f5640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17bc157c00 0x7f17bc13f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:48.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17c45f5640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17bc14e490 0x7f17bc1f06d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:48.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17c45f5640 1 --2- 172.21.15.150:0/3166826832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17bc071670 0x7f17bc14d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:48.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17c45f5640 1 -- 172.21.15.150:0/3166826832 >> 172.21.15.150:0/3166826832 conn(0x7f17bc0ca460 msgr2=0x7f17bc1499b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:48.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17c45f5640 1 -- 172.21.15.150:0/3166826832 shutdown_connections 2024-09-17T04:54:48.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:48.591+0000 7f17c45f5640 1 -- 172.21.15.150:0/3166826832 wait complete. 2024-09-17T04:54:48.614 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:54:50.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:50.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.795+0000 7f2d00cb4640 1 -- 172.21.15.150:0/1146502584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cfc0c5df0 msgr2=0x7f2cfc157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:50.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.795+0000 7f2d00cb4640 1 --2- 172.21.15.150:0/1146502584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cfc0c5df0 0x7f2cfc157330 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4009840 tx=0x7f2cf401c820 comp rx=0 tx=0).stop 2024-09-17T04:54:50.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.795+0000 7f2d00cb4640 1 -- 172.21.15.150:0/1146502584 shutdown_connections 2024-09-17T04:54:50.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.795+0000 7f2d00cb4640 1 --2- 172.21.15.150:0/1146502584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cfc0c6340 0x7f2cfc0c1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:50.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.795+0000 7f2d00cb4640 1 --2- 172.21.15.150:0/1146502584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cfc0c5df0 0x7f2cfc157330 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:50.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.795+0000 7f2d00cb4640 1 -- 172.21.15.150:0/1146502584 >> 172.21.15.150:0/1146502584 conn(0x7f2cfc074750 msgr2=0x7f2cfc074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:50.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.795+0000 7f2d00cb4640 1 -- 172.21.15.150:0/1146502584 shutdown_connections 2024-09-17T04:54:50.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.795+0000 7f2d00cb4640 1 -- 172.21.15.150:0/1146502584 wait complete. 2024-09-17T04:54:50.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2d00cb4640 1 Processor -- start 2024-09-17T04:54:50.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2d00cb4640 1 -- start start 2024-09-17T04:54:50.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2d00cb4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cfc0c4b20 0x7f2cfc0ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:50.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2d00cb4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cfc0c5df0 0x7f2cfc1524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:50.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2d00cb4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cfc0c6340 0x7f2cfc1f05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:50.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2d00cb4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2cfc154450 con 0x7f2cfc0c4b20 2024-09-17T04:54:50.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2d00cb4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2cfc1542d0 con 0x7f2cfc0c6340 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2d00cb4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2cfc163980 con 0x7f2cfc0c5df0 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cfa59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cfc0c4b20 0x7f2cfc0ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cfa59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cfc0c4b20 0x7f2cfc0ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55310/0 (socket says 172.21.15.150:55310) 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cfa59c640 1 -- 172.21.15.150:0/3918523994 learned_addr learned my addr 172.21.15.150:0/3918523994 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cf9d9b640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cfc0c5df0 0x7f2cfc1524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cfad9d640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cfc0c6340 0x7f2cfc1f05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cfa59c640 1 -- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cfc0c5df0 msgr2=0x7f2cfc1524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cfa59c640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cfc0c5df0 0x7f2cfc1524a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cfa59c640 1 -- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cfc0c6340 msgr2=0x7f2cfc1f05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cfa59c640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cfc0c6340 0x7f2cfc1f05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cfa59c640 1 -- 172.21.15.150:0/3918523994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cfc074c80 con 0x7f2cfc0c4b20 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cfad9d640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cfc0c6340 0x7f2cfc1f05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cf9d9b640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cfc0c5df0 0x7f2cfc1524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:54:50.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2cfa59c640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cfc0c4b20 0x7f2cfc0ca180 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2cf401cd00 tx=0x7f2cf4005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:50.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2ceaffd640 1 -- 172.21.15.150:0/3918523994 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf40b2040 con 0x7f2cfc0c4b20 2024-09-17T04:54:50.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2ceaffd640 1 -- 172.21.15.150:0/3918523994 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cf40bb070 con 0x7f2cfc0c4b20 2024-09-17T04:54:50.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2d00cb4640 1 -- 172.21.15.150:0/3918523994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cfc14ba30 con 0x7f2cfc0c4b20 2024-09-17T04:54:50.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2ceaffd640 1 -- 172.21.15.150:0/3918523994 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4003600 con 0x7f2cfc0c4b20 2024-09-17T04:54:50.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.799+0000 7f2d00cb4640 1 -- 172.21.15.150:0/3918523994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cfc158100 con 0x7f2cfc0c4b20 2024-09-17T04:54:50.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.803+0000 7f2d00cb4640 1 -- 172.21.15.150:0/3918523994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cfc157330 con 0x7f2cfc0c4b20 2024-09-17T04:54:50.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.803+0000 7f2ceaffd640 1 -- 172.21.15.150:0/3918523994 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2cf40037a0 con 0x7f2cfc0c4b20 2024-09-17T04:54:50.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.803+0000 7f2ceaffd640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cdc073230 0x7f2cdc075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:50.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.803+0000 7f2ceaffd640 1 -- 172.21.15.150:0/3918523994 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7f2cf412ed30 con 0x7f2cfc0c4b20 2024-09-17T04:54:50.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.803+0000 7f2cf9d9b640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cdc073230 0x7f2cdc075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:50.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.803+0000 7f2cf9d9b640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cdc073230 0x7f2cdc075740 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0005fd0 tx=0x7f2cf0005e00 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:50.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:50.807+0000 7f2ceaffd640 1 -- 172.21.15.150:0/3918523994 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2cf4016410 con 0x7f2cfc0c4b20 2024-09-17T04:54:51.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.035+0000 7f2d00cb4640 1 -- 172.21.15.150:0/3918523994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2cfc0b1b70 con 0x7f2cfc0c4b20 2024-09-17T04:54:51.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.035+0000 7f2ceaffd640 1 -- 172.21.15.150:0/3918523994 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6538 (secure 0 0 0) 0x7f2cf40f97b0 con 0x7f2cfc0c4b20 2024-09-17T04:54:51.038 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:51.039 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":36,"btime":"2024-09-17T04:53:56:949667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:54:51.039 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 36 2024-09-17T04:54:51.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.039+0000 7f2d00cb4640 1 -- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cdc073230 msgr2=0x7f2cdc075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:51.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.039+0000 7f2d00cb4640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cdc073230 0x7f2cdc075740 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0005fd0 tx=0x7f2cf0005e00 comp rx=0 tx=0).stop 2024-09-17T04:54:51.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.039+0000 7f2d00cb4640 1 -- 172.21.15.150:0/3918523994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cfc0c4b20 msgr2=0x7f2cfc0ca180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:51.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.039+0000 7f2d00cb4640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cfc0c4b20 0x7f2cfc0ca180 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2cf401cd00 tx=0x7f2cf4005db0 comp rx=0 tx=0).stop 2024-09-17T04:54:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.039+0000 7f2d00cb4640 1 -- 172.21.15.150:0/3918523994 shutdown_connections 2024-09-17T04:54:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.039+0000 7f2d00cb4640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cdc073230 0x7f2cdc075740 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.039+0000 7f2d00cb4640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cfc0c6340 0x7f2cfc1f05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.039+0000 7f2d00cb4640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cfc0c5df0 0x7f2cfc1524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.039+0000 7f2d00cb4640 1 --2- 172.21.15.150:0/3918523994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cfc0c4b20 0x7f2cfc0ca180 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.039+0000 7f2d00cb4640 1 -- 172.21.15.150:0/3918523994 >> 172.21.15.150:0/3918523994 conn(0x7f2cfc074750 msgr2=0x7f2cfc154ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.039+0000 7f2d00cb4640 1 -- 172.21.15.150:0/3918523994 shutdown_connections 2024-09-17T04:54:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:51.039+0000 7f2d00cb4640 1 -- 172.21.15.150:0/3918523994 wait complete. 2024-09-17T04:54:51.061 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:54:53.037 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:53.062 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:53.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 -- 172.21.15.150:0/2039864009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd014e490 msgr2=0x7fcfd00c7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 --2- 172.21.15.150:0/2039864009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd014e490 0x7fcfd00c7d60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0009900 tx=0x7fcfc001c8e0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 -- 172.21.15.150:0/2039864009 shutdown_connections 2024-09-17T04:54:53.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 --2- 172.21.15.150:0/2039864009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd014e490 0x7fcfd00c7d60 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 --2- 172.21.15.150:0/2039864009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfd0071670 0x7fcfd00c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 -- 172.21.15.150:0/2039864009 >> 172.21.15.150:0/2039864009 conn(0x7fcfd00ca460 msgr2=0x7fcfd00c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:53.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 -- 172.21.15.150:0/2039864009 shutdown_connections 2024-09-17T04:54:53.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 -- 172.21.15.150:0/2039864009 wait complete. 2024-09-17T04:54:53.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 Processor -- start 2024-09-17T04:54:53.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 -- start start 2024-09-17T04:54:53.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfd0071670 0x7fcfd0146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:53.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfd014e490 0x7fcfd00c2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:53.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd0157c00 0x7fcfd0145720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd00c6740 con 0x7fcfd0157c00 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd00c65c0 con 0x7fcfd014e490 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd7eff640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd014e8a0 con 0x7fcfd0071670 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd649c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd0157c00 0x7fcfd0145720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd649c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd0157c00 0x7fcfd0145720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55330/0 (socket says 172.21.15.150:55330) 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd649c640 1 -- 172.21.15.150:0/474304466 learned_addr learned my addr 172.21.15.150:0/474304466 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd549a640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfd014e490 0x7fcfd00c2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.219+0000 7fcfd5c9b640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfd0071670 0x7fcfd0146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfd649c640 1 -- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfd0071670 msgr2=0x7fcfd0146fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfd649c640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfd0071670 0x7fcfd0146fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfd649c640 1 -- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfd014e490 msgr2=0x7fcfd00c2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfd649c640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfd014e490 0x7fcfd00c2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfd649c640 1 -- 172.21.15.150:0/474304466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd00bf080 con 0x7fcfd0157c00 2024-09-17T04:54:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfd649c640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd0157c00 0x7fcfd0145720 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800c6b0 tx=0x7fcfc800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:53.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfc67fc640 1 -- 172.21.15.150:0/474304466 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8003b80 con 0x7fcfd0157c00 2024-09-17T04:54:53.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfd7eff640 1 -- 172.21.15.150:0/474304466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd01478e0 con 0x7fcfd0157c00 2024-09-17T04:54:53.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfd7eff640 1 -- 172.21.15.150:0/474304466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd0156ea0 con 0x7fcfd0157c00 2024-09-17T04:54:53.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfc67fc640 1 -- 172.21.15.150:0/474304466 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfc8003d20 con 0x7fcfd0157c00 2024-09-17T04:54:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfc67fc640 1 -- 172.21.15.150:0/474304466 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8005330 con 0x7fcfd0157c00 2024-09-17T04:54:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfd7eff640 1 -- 172.21.15.150:0/474304466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfb80052a0 con 0x7fcfd0157c00 2024-09-17T04:54:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.223+0000 7fcfc67fc640 1 -- 172.21.15.150:0/474304466 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fcfc80054d0 con 0x7fcfd0157c00 2024-09-17T04:54:53.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.227+0000 7fcfc67fc640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcfb4073230 0x7fcfb4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:53.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.227+0000 7fcfc67fc640 1 -- 172.21.15.150:0/474304466 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fcfc8094170 con 0x7fcfd0157c00 2024-09-17T04:54:53.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.227+0000 7fcfd5c9b640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcfb4073230 0x7fcfb4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:53.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.227+0000 7fcfd5c9b640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcfb4073230 0x7fcfb4075740 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc004630 tx=0x7fcfcc00b2e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:53.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.231+0000 7fcfc67fc640 1 -- 172.21.15.150:0/474304466 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcfc805e740 con 0x7fcfd0157c00 2024-09-17T04:54:53.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.259+0000 7f9af6256640 1 -- 172.21.15.150:0/2638407271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af00c7eb0 msgr2=0x7f9af01540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.259+0000 7f9af6256640 1 --2- 172.21.15.150:0/2638407271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af00c7eb0 0x7f9af01540b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0009900 tx=0x7f9ae001c8a0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.259+0000 7f9af6256640 1 -- 172.21.15.150:0/2638407271 shutdown_connections 2024-09-17T04:54:53.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.259+0000 7f9af6256640 1 --2- 172.21.15.150:0/2638407271 >> v2:172.21.15.176:3300/0 conn(0x7f9af014a180 0x7f9af00c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.259+0000 7f9af6256640 1 --2- 172.21.15.150:0/2638407271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af00c7eb0 0x7f9af01540b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.259+0000 7f9af6256640 1 -- 172.21.15.150:0/2638407271 >> 172.21.15.150:0/2638407271 conn(0x7f9af00aab60 msgr2=0x7f9af00aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:53.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.259+0000 7f9af6256640 1 -- 172.21.15.150:0/2638407271 shutdown_connections 2024-09-17T04:54:53.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.259+0000 7f9af6256640 1 -- 172.21.15.150:0/2638407271 wait complete. 2024-09-17T04:54:53.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9af6256640 1 Processor -- start 2024-09-17T04:54:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9af6256640 1 -- start start 2024-09-17T04:54:53.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9af6256640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9af00c7eb0 0x7f9af00ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:53.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9af6256640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af0149d70 0x7f9af014ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:53.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9af6256640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9af014a180 0x7f9af00ad6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:53.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9af6256640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9af00c7c60 con 0x7f9af0149d70 2024-09-17T04:54:53.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9af6256640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9af00c7ae0 con 0x7f9af00c7eb0 2024-09-17T04:54:53.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9af6256640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9af0156540 con 0x7f9af014a180 2024-09-17T04:54:53.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9ae7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af0149d70 0x7f9af014ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9ae7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af0149d70 0x7f9af014ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55350/0 (socket says 172.21.15.150:55350) 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9ae7fff640 1 -- 172.21.15.150:0/1758482848 learned_addr learned my addr 172.21.15.150:0/1758482848 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9aef7fe640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9af00c7eb0 0x7f9af00ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9ae7fff640 1 -- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9af014a180 msgr2=0x7f9af00ad6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9ae7fff640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9af014a180 0x7f9af00ad6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9ae7fff640 1 -- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9af00c7eb0 msgr2=0x7f9af00ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9ae7fff640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9af00c7eb0 0x7f9af00ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9ae7fff640 1 -- 172.21.15.150:0/1758482848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af014dc60 con 0x7f9af0149d70 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9aef7fe640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9af00c7eb0 0x7f9af00ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9ae7fff640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af0149d70 0x7f9af014ca60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800b440 tx=0x7f9ad800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9aecff9640 1 -- 172.21.15.150:0/1758482848 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8015020 con 0x7f9af0149d70 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9aecff9640 1 -- 172.21.15.150:0/1758482848 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ad8011980 con 0x7f9af0149d70 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9af6256640 1 -- 172.21.15.150:0/1758482848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9af01492d0 con 0x7f9af0149d70 2024-09-17T04:54:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9aecff9640 1 -- 172.21.15.150:0/1758482848 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8011dd0 con 0x7f9af0149d70 2024-09-17T04:54:53.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.263+0000 7f9af6256640 1 -- 172.21.15.150:0/1758482848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9af00aca20 con 0x7f9af0149d70 2024-09-17T04:54:53.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.267+0000 7f9af6256640 1 -- 172.21.15.150:0/1758482848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9af00c7190 con 0x7f9af0149d70 2024-09-17T04:54:53.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.267+0000 7f9aecff9640 1 -- 172.21.15.150:0/1758482848 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9ad8018020 con 0x7f9af0149d70 2024-09-17T04:54:53.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.267+0000 7f9aecff9640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ad0073200 0x7f9ad0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:53.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.267+0000 7f9aecff9640 1 -- 172.21.15.150:0/1758482848 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7f9ad8094640 con 0x7f9af0149d70 2024-09-17T04:54:53.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.267+0000 7f9aef7fe640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ad0073200 0x7f9ad0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:53.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.267+0000 7f9aef7fe640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ad0073200 0x7f9ad0075710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9ae00af000 tx=0x7f9ae001b0e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:53.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.271+0000 7f9aecff9640 1 -- 172.21.15.150:0/1758482848 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9ad801f030 con 0x7f9af0149d70 2024-09-17T04:54:53.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.503+0000 7f9af6256640 1 -- 172.21.15.150:0/1758482848 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9af006df30 con 0x7f9af0149d70 2024-09-17T04:54:53.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.503+0000 7f9aecff9640 1 -- 172.21.15.150:0/1758482848 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6538 (secure 0 0 0) 0x7f9ad800d040 con 0x7f9af0149d70 2024-09-17T04:54:53.505 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:53.505 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":36,"btime":"2024-09-17T04:53:56:949667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:54:53.505 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 36 2024-09-17T04:54:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.507+0000 7f9af6256640 1 -- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ad0073200 msgr2=0x7f9ad0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.507+0000 7f9af6256640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ad0073200 0x7f9ad0075710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9ae00af000 tx=0x7f9ae001b0e0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.507+0000 7f9af6256640 1 -- 172.21.15.150:0/1758482848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af0149d70 msgr2=0x7f9af014ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.507+0000 7f9af6256640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af0149d70 0x7f9af014ca60 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800b440 tx=0x7f9ad800b900 comp rx=0 tx=0).stop 2024-09-17T04:54:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.507+0000 7f9af6256640 1 -- 172.21.15.150:0/1758482848 shutdown_connections 2024-09-17T04:54:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.507+0000 7f9af6256640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ad0073200 0x7f9ad0075710 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.507+0000 7f9af6256640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9af014a180 0x7f9af00ad6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.507+0000 7f9af6256640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af0149d70 0x7f9af014ca60 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.507+0000 7f9af6256640 1 --2- 172.21.15.150:0/1758482848 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9af00c7eb0 0x7f9af00ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.507+0000 7f9af6256640 1 -- 172.21.15.150:0/1758482848 >> 172.21.15.150:0/1758482848 conn(0x7f9af00aab60 msgr2=0x7f9af0152e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:53.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.507+0000 7f9af6256640 1 -- 172.21.15.150:0/1758482848 shutdown_connections 2024-09-17T04:54:53.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.507+0000 7f9af6256640 1 -- 172.21.15.150:0/1758482848 wait complete. 2024-09-17T04:54:53.527 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:54:53.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.527+0000 7fcfd7eff640 1 -- 172.21.15.150:0/474304466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfb8005860 con 0x7fcfd0157c00 2024-09-17T04:54:53.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.527+0000 7fcfc67fc640 1 -- 172.21.15.150:0/474304466 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6538 (secure 0 0 0) 0x7fcfc8063170 con 0x7fcfd0157c00 2024-09-17T04:54:53.529 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:53.530 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":36,"btime":"2024-09-17T04:53:56:949667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:54:53.530 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 36 2024-09-17T04:54:53.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.531+0000 7fcfd7eff640 1 -- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcfb4073230 msgr2=0x7fcfb4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.531+0000 7fcfd7eff640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcfb4073230 0x7fcfb4075740 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc004630 tx=0x7fcfcc00b2e0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.531+0000 7fcfd7eff640 1 -- 172.21.15.150:0/474304466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd0157c00 msgr2=0x7fcfd0145720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.531+0000 7fcfd7eff640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd0157c00 0x7fcfd0145720 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcfc800c6b0 tx=0x7fcfc800cb70 comp rx=0 tx=0).stop 2024-09-17T04:54:53.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.531+0000 7fcfd7eff640 1 -- 172.21.15.150:0/474304466 shutdown_connections 2024-09-17T04:54:53.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.531+0000 7fcfd7eff640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcfb4073230 0x7fcfb4075740 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.531+0000 7fcfd7eff640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfd014e490 0x7fcfd00c2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.531+0000 7fcfd7eff640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd0157c00 0x7fcfd0145720 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.531+0000 7fcfd7eff640 1 --2- 172.21.15.150:0/474304466 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfd0071670 0x7fcfd0146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.531+0000 7fcfd7eff640 1 -- 172.21.15.150:0/474304466 >> 172.21.15.150:0/474304466 conn(0x7fcfd00ca460 msgr2=0x7fcfd00c7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:53.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.531+0000 7fcfd7eff640 1 -- 172.21.15.150:0/474304466 shutdown_connections 2024-09-17T04:54:53.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.531+0000 7fcfd7eff640 1 -- 172.21.15.150:0/474304466 wait complete. 2024-09-17T04:54:53.552 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-17T04:54:53.552 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:54:53.552 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:54:53.613 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:54:53.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:53.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.819+0000 7fc46670d640 1 -- 172.21.15.150:0/2731976235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc460074db0 msgr2=0x7fc460157460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.819+0000 7fc46670d640 1 --2- 172.21.15.150:0/2731976235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc460074db0 0x7fc460157460 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc460073490 tx=0x7fc45401c8c0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.819+0000 7fc46670d640 1 -- 172.21.15.150:0/2731976235 shutdown_connections 2024-09-17T04:54:53.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.819+0000 7fc46670d640 1 --2- 172.21.15.150:0/2731976235 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4601579a0 0x7fc4600c7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.819+0000 7fc46670d640 1 --2- 172.21.15.150:0/2731976235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc460074db0 0x7fc460157460 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.819+0000 7fc46670d640 1 --2- 172.21.15.150:0/2731976235 >> v2:172.21.15.176:3300/0 conn(0x7fc4600c8230 0x7fc460074870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.819+0000 7fc46670d640 1 -- 172.21.15.150:0/2731976235 >> 172.21.15.150:0/2731976235 conn(0x7fc46006e360 msgr2=0x7fc46006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:53.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.819+0000 7fc46670d640 1 -- 172.21.15.150:0/2731976235 shutdown_connections 2024-09-17T04:54:53.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.819+0000 7fc46670d640 1 -- 172.21.15.150:0/2731976235 wait complete. 2024-09-17T04:54:53.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.819+0000 7fc46670d640 1 Processor -- start 2024-09-17T04:54:53.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.819+0000 7fc46670d640 1 -- start start 2024-09-17T04:54:53.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc46670d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc460074db0 0x7fc4600c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:53.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc46670d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4600c8230 0x7fc4600a9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:53.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc46670d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4601579a0 0x7fc46014dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:53.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc46670d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc46006cfd0 con 0x7fc460074db0 2024-09-17T04:54:53.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc46670d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc46006ce50 con 0x7fc4601579a0 2024-09-17T04:54:53.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc46670d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc46006d150 con 0x7fc4600c8230 2024-09-17T04:54:53.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc460074db0 0x7fc4600c22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:53.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc460074db0 0x7fc4600c22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55380/0 (socket says 172.21.15.150:55380) 2024-09-17T04:54:53.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45ffff640 1 -- 172.21.15.150:0/20471631 learned_addr learned my addr 172.21.15.150:0/20471631 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:53.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc464caa640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4601579a0 0x7fc46014dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:53.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45f7fe640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4600c8230 0x7fc4600a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:53.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45ffff640 1 -- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4600c8230 msgr2=0x7fc4600a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45ffff640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4600c8230 0x7fc4600a9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45ffff640 1 -- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4601579a0 msgr2=0x7fc46014dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:53.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45ffff640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4601579a0 0x7fc46014dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:53.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45ffff640 1 -- 172.21.15.150:0/20471631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc46014f970 con 0x7fc460074db0 2024-09-17T04:54:53.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45f7fe640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4600c8230 0x7fc4600a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:54:53.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45ffff640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc460074db0 0x7fc4600c22b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc4500058d0 tx=0x7fc45000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:53.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45cff9640 1 -- 172.21.15.150:0/20471631 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc450017070 con 0x7fc460074db0 2024-09-17T04:54:53.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45cff9640 1 -- 172.21.15.150:0/20471631 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc450009cc0 con 0x7fc460074db0 2024-09-17T04:54:53.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc46670d640 1 -- 172.21.15.150:0/20471631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc46014eab0 con 0x7fc460074db0 2024-09-17T04:54:53.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45cff9640 1 -- 172.21.15.150:0/20471631 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4500106c0 con 0x7fc460074db0 2024-09-17T04:54:53.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc46670d640 1 -- 172.21.15.150:0/20471631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4601eb310 con 0x7fc460074db0 2024-09-17T04:54:53.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc46670d640 1 -- 172.21.15.150:0/20471631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc460074870 con 0x7fc460074db0 2024-09-17T04:54:53.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.823+0000 7fc45cff9640 1 -- 172.21.15.150:0/20471631 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc45001a070 con 0x7fc460074db0 2024-09-17T04:54:53.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.827+0000 7fc45cff9640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc44c073200 0x7fc44c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:53.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.827+0000 7fc45cff9640 1 -- 172.21.15.150:0/20471631 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7fc450094850 con 0x7fc460074db0 2024-09-17T04:54:53.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.827+0000 7fc45f7fe640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc44c073200 0x7fc44c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:53.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.827+0000 7fc45f7fe640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc44c073200 0x7fc44c075710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc4540af000 tx=0x7fc45401b100 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:53.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:53.831+0000 7fc45cff9640 1 -- 172.21.15.150:0/20471631 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc45005f3e0 con 0x7fc460074db0 2024-09-17T04:54:54.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.055+0000 7fc46670d640 1 -- 172.21.15.150:0/20471631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc46015ed70 con 0x7fc460074db0 2024-09-17T04:54:54.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.055+0000 7fc45cff9640 1 -- 172.21.15.150:0/20471631 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6538 (secure 0 0 0) 0x7fc45005ece0 con 0x7fc460074db0 2024-09-17T04:54:54.057 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:54.057 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":36,"btime":"2024-09-17T04:53:56:949667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5107,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:52:58.862037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4975},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4975":{"gid":4975,"name":"a","rank":0,"incarnation":27,"state":"up:active","state_seq":20,"addr":"172.21.15.150:6835/151500360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":151500360},{"type":"v1","addr":"172.21.15.150:6835","nonce":151500360}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4975,"qdb_cluster":[4975]},"id":2}]} 2024-09-17T04:54:54.058 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 36 2024-09-17T04:54:54.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.059+0000 7fc46670d640 1 -- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc44c073200 msgr2=0x7fc44c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:54.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.059+0000 7fc46670d640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc44c073200 0x7fc44c075710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc4540af000 tx=0x7fc45401b100 comp rx=0 tx=0).stop 2024-09-17T04:54:54.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.059+0000 7fc46670d640 1 -- 172.21.15.150:0/20471631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc460074db0 msgr2=0x7fc4600c22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:54.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.059+0000 7fc46670d640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc460074db0 0x7fc4600c22b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc4500058d0 tx=0x7fc45000dde0 comp rx=0 tx=0).stop 2024-09-17T04:54:54.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.059+0000 7fc46670d640 1 -- 172.21.15.150:0/20471631 shutdown_connections 2024-09-17T04:54:54.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.059+0000 7fc46670d640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc44c073200 0x7fc44c075710 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:54.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.059+0000 7fc46670d640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4601579a0 0x7fc46014dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:54.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.059+0000 7fc46670d640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4600c8230 0x7fc4600a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:54.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.059+0000 7fc46670d640 1 --2- 172.21.15.150:0/20471631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc460074db0 0x7fc4600c22b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:54.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.059+0000 7fc46670d640 1 -- 172.21.15.150:0/20471631 >> 172.21.15.150:0/20471631 conn(0x7fc46006e360 msgr2=0x7fc46006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:54.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.059+0000 7fc46670d640 1 -- 172.21.15.150:0/20471631 shutdown_connections 2024-09-17T04:54:54.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:54.059+0000 7fc46670d640 1 -- 172.21.15.150:0/20471631 wait complete. 2024-09-17T04:54:54.078 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:54:55.527 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:55.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.691+0000 7f35590fa640 1 -- 172.21.15.150:0/2189282533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35541568a0 msgr2=0x7f35540c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:55.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.691+0000 7f35590fa640 1 --2- 172.21.15.150:0/2189282533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35541568a0 0x7f35540c5810 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3544009900 tx=0x7f354401c8b0 comp rx=0 tx=0).stop 2024-09-17T04:54:55.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.691+0000 7f35590fa640 1 -- 172.21.15.150:0/2189282533 shutdown_connections 2024-09-17T04:54:55.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.691+0000 7f35590fa640 1 --2- 172.21.15.150:0/2189282533 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35540c20a0 0x7f35540c24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:55.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.691+0000 7f35590fa640 1 --2- 172.21.15.150:0/2189282533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35541568a0 0x7f35540c5810 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:55.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.691+0000 7f35590fa640 1 -- 172.21.15.150:0/2189282533 >> 172.21.15.150:0/2189282533 conn(0x7f3554074b80 msgr2=0x7f355406f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:55.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.691+0000 7f35590fa640 1 -- 172.21.15.150:0/2189282533 shutdown_connections 2024-09-17T04:54:55.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.691+0000 7f35590fa640 1 -- 172.21.15.150:0/2189282533 wait complete. 2024-09-17T04:54:55.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f35590fa640 1 Processor -- start 2024-09-17T04:54:55.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f35590fa640 1 -- start start 2024-09-17T04:54:55.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f35590fa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35540c20a0 0x7f3554147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:55.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f35590fa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35540c35b0 0x7f35540cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f35590fa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35541568a0 0x7f35541f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f35590fa640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f35540c3b40 con 0x7f35540c20a0 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f35590fa640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f35540c39c0 con 0x7f35541568a0 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f35590fa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f35540c69f0 con 0x7f35540c35b0 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f3552d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35540c20a0 0x7f3554147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f3552d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35540c20a0 0x7f3554147870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55406/0 (socket says 172.21.15.150:55406) 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f3552d9d640 1 -- 172.21.15.150:0/2983592739 learned_addr learned my addr 172.21.15.150:0/2983592739 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f355359e640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35541568a0 0x7f35541f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f355259c640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35540c35b0 0x7f35540cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f3552d9d640 1 -- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35540c35b0 msgr2=0x7f35540cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f3552d9d640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35540c35b0 0x7f35540cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f3552d9d640 1 -- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35541568a0 msgr2=0x7f35541f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:55.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f3552d9d640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35541568a0 0x7f35541f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:55.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f3552d9d640 1 -- 172.21.15.150:0/2983592739 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35541511b0 con 0x7f35540c20a0 2024-09-17T04:54:55.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f355359e640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35541568a0 0x7f35541f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:54:55.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f3552d9d640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35540c20a0 0x7f3554147870 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f35440b0040 tx=0x7f3544005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:55.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f353b7fe640 1 -- 172.21.15.150:0/2983592739 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35440b2040 con 0x7f35540c20a0 2024-09-17T04:54:55.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f353b7fe640 1 -- 172.21.15.150:0/2983592739 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3544005320 con 0x7f35540c20a0 2024-09-17T04:54:55.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f353b7fe640 1 -- 172.21.15.150:0/2983592739 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35440046d0 con 0x7f35540c20a0 2024-09-17T04:54:55.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f35590fa640 1 -- 172.21.15.150:0/2983592739 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f355414a8a0 con 0x7f35540c20a0 2024-09-17T04:54:55.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.695+0000 7f35590fa640 1 -- 172.21.15.150:0/2983592739 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35541f3d10 con 0x7f35540c20a0 2024-09-17T04:54:55.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.699+0000 7f35590fa640 1 -- 172.21.15.150:0/2983592739 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35540c24b0 con 0x7f35540c20a0 2024-09-17T04:54:55.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.699+0000 7f353b7fe640 1 -- 172.21.15.150:0/2983592739 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f35440be060 con 0x7f35540c20a0 2024-09-17T04:54:55.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.699+0000 7f353b7fe640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3540073200 0x7f3540075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:55.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.699+0000 7f353b7fe640 1 -- 172.21.15.150:0/2983592739 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 7010+0+0 (secure 0 0 0) 0x7f354412e760 con 0x7f35540c20a0 2024-09-17T04:54:55.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.699+0000 7f355259c640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3540073200 0x7f3540075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:55.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.699+0000 7f355259c640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3540073200 0x7f3540075710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f355414a470 tx=0x7f3548009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:55.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.703+0000 7f353b7fe640 1 -- 172.21.15.150:0/2983592739 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f35440164d0 con 0x7f35540c20a0 2024-09-17T04:54:55.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.919+0000 7f35590fa640 1 -- 172.21.15.150:0/2983592739 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35541552d0 con 0x7f35540c20a0 2024-09-17T04:54:55.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.919+0000 7f353b7fe640 1 -- 172.21.15.150:0/2983592739 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5705 (secure 0 0 0) 0x7f35440f93d0 con 0x7f35540c20a0 2024-09-17T04:54:55.921 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:55.922 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"btime":"2024-09-17T04:54:55:867317+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:55.867307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:54:55.922 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-09-17T04:54:55.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.919+0000 7f35590fa640 1 -- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3540073200 msgr2=0x7f3540075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:55.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.919+0000 7f35590fa640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3540073200 0x7f3540075710 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f355414a470 tx=0x7f3548009040 comp rx=0 tx=0).stop 2024-09-17T04:54:55.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.919+0000 7f35590fa640 1 -- 172.21.15.150:0/2983592739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35540c20a0 msgr2=0x7f3554147870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:55.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.919+0000 7f35590fa640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35540c20a0 0x7f3554147870 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f35440b0040 tx=0x7f3544005760 comp rx=0 tx=0).stop 2024-09-17T04:54:55.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.923+0000 7f35590fa640 1 -- 172.21.15.150:0/2983592739 shutdown_connections 2024-09-17T04:54:55.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.923+0000 7f35590fa640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3540073200 0x7f3540075710 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:55.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.923+0000 7f35590fa640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f35541568a0 0x7f35541f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:55.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.923+0000 7f35590fa640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f35540c35b0 0x7f35540cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:55.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.923+0000 7f35590fa640 1 --2- 172.21.15.150:0/2983592739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35540c20a0 0x7f3554147870 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:55.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.923+0000 7f35590fa640 1 -- 172.21.15.150:0/2983592739 >> 172.21.15.150:0/2983592739 conn(0x7f3554074b80 msgr2=0x7f355415a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:55.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.923+0000 7f35590fa640 1 -- 172.21.15.150:0/2983592739 shutdown_connections 2024-09-17T04:54:55.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:55.923+0000 7f35590fa640 1 -- 172.21.15.150:0/2983592739 wait complete. 2024-09-17T04:54:55.942 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-17T04:54:55.942 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T04:54:55.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:56.079 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:56.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.087+0000 7f450fa7a640 1 -- 172.21.15.150:0/1503265103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45081506d0 msgr2=0x7f45080c2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:56.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 --2- 172.21.15.150:0/1503265103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45081506d0 0x7f45080c2240 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f44f8009840 tx=0x7f44f801c820 comp rx=0 tx=0).stop 2024-09-17T04:54:56.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 -- 172.21.15.150:0/1503265103 shutdown_connections 2024-09-17T04:54:56.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 --2- 172.21.15.150:0/1503265103 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45081506d0 0x7f45080c2240 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 --2- 172.21.15.150:0/1503265103 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4508156820 0x7f45080c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 -- 172.21.15.150:0/1503265103 >> 172.21.15.150:0/1503265103 conn(0x7f4508074b80 msgr2=0x7f450806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:56.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 -- 172.21.15.150:0/1503265103 shutdown_connections 2024-09-17T04:54:56.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 -- 172.21.15.150:0/1503265103 wait complete. 2024-09-17T04:54:56.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 Processor -- start 2024-09-17T04:54:56.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 -- start start 2024-09-17T04:54:56.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45080c2780 0x7f450814df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:56.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45081506d0 0x7f45080ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:56.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4508156820 0x7f45080cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:56.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4508150c60 con 0x7f45081506d0 2024-09-17T04:54:56.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4508150ae0 con 0x7f45080c2780 2024-09-17T04:54:56.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450fa7a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f450806fd80 con 0x7f4508156820 2024-09-17T04:54:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450d015640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45081506d0 0x7f45080ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450d015640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45081506d0 0x7f45080ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55424/0 (socket says 172.21.15.150:55424) 2024-09-17T04:54:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450d015640 1 -- 172.21.15.150:0/2641509788 learned_addr learned my addr 172.21.15.150:0/2641509788 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450d816640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45080c2780 0x7f450814df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450e017640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4508156820 0x7f45080cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450d015640 1 -- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4508156820 msgr2=0x7f45080cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450d015640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4508156820 0x7f45080cace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450d015640 1 -- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45080c2780 msgr2=0x7f450814df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450d015640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45080c2780 0x7f450814df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450d015640 1 -- 172.21.15.150:0/2641509788 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45080cb8d0 con 0x7f45081506d0 2024-09-17T04:54:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450e017640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4508156820 0x7f45080cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:54:56.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.091+0000 7f450d015640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45081506d0 0x7f45080ca7a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f44f8009790 tx=0x7f44f80a64f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:56.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.095+0000 7f44fe7fc640 1 -- 172.21.15.150:0/2641509788 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8004240 con 0x7f45081506d0 2024-09-17T04:54:56.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.095+0000 7f44fe7fc640 1 -- 172.21.15.150:0/2641509788 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44f80bf070 con 0x7f45081506d0 2024-09-17T04:54:56.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.095+0000 7f450fa7a640 1 -- 172.21.15.150:0/2641509788 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45081585d0 con 0x7f45081506d0 2024-09-17T04:54:56.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.095+0000 7f44fe7fc640 1 -- 172.21.15.150:0/2641509788 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8003820 con 0x7f45081506d0 2024-09-17T04:54:56.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.095+0000 7f450fa7a640 1 -- 172.21.15.150:0/2641509788 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4508157dd0 con 0x7f45081506d0 2024-09-17T04:54:56.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.095+0000 7f450fa7a640 1 -- 172.21.15.150:0/2641509788 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45080c2240 con 0x7f45081506d0 2024-09-17T04:54:56.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.095+0000 7f44fe7fc640 1 -- 172.21.15.150:0/2641509788 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f44f80039c0 con 0x7f45081506d0 2024-09-17T04:54:56.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.095+0000 7f44fe7fc640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f44ec073260 0x7f44ec075770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:56.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.095+0000 7f44fe7fc640 1 -- 172.21.15.150:0/2641509788 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f44f812fa90 con 0x7f45081506d0 2024-09-17T04:54:56.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.095+0000 7f450d816640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f44ec073260 0x7f44ec075770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:56.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.095+0000 7f450d816640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f44ec073260 0x7f44ec075770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4504009bc0 tx=0x7f45040091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:56.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.099+0000 7f44fe7fc640 1 -- 172.21.15.150:0/2641509788 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f44f8016410 con 0x7f45081506d0 2024-09-17T04:54:56.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.239+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3555519177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7e40c6cb0 msgr2=0x7fc7e40ae340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:56.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.239+0000 7fc7e8e04640 1 --2- 172.21.15.150:0/3555519177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7e40c6cb0 0x7fc7e40ae340 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009840 tx=0x7fc7d401c790 comp rx=0 tx=0).stop 2024-09-17T04:54:56.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.239+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3555519177 shutdown_connections 2024-09-17T04:54:56.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.239+0000 7fc7e8e04640 1 --2- 172.21.15.150:0/3555519177 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7e40c6cb0 0x7fc7e40ae340 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.239+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3555519177 >> 172.21.15.150:0/3555519177 conn(0x7fc7e40aab60 msgr2=0x7fc7e40aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:56.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.239+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3555519177 shutdown_connections 2024-09-17T04:54:56.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.239+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3555519177 wait complete. 2024-09-17T04:54:56.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.239+0000 7fc7e8e04640 1 Processor -- start 2024-09-17T04:54:56.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.239+0000 7fc7e8e04640 1 -- start start 2024-09-17T04:54:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e8e04640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7e40a85b0 0x7fc7e4071e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e8e04640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7e40c6cb0 0x7fc7e40a9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e8e04640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7e4151980 0x7fc7e4145200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e8e04640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e40c3460 con 0x7fc7e40a85b0 2024-09-17T04:54:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e8e04640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e40c32e0 con 0x7fc7e40c6cb0 2024-09-17T04:54:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e8e04640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc7e41628a0 con 0x7fc7e4151980 2024-09-17T04:54:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7e40a85b0 0x7fc7e4071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7e40a85b0 0x7fc7e4071e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55446/0 (socket says 172.21.15.150:55446) 2024-09-17T04:54:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e259c640 1 -- 172.21.15.150:0/3809791558 learned_addr learned my addr 172.21.15.150:0/3809791558 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e259c640 1 -- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7e4151980 msgr2=0x7fc7e4145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e259c640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7e4151980 0x7fc7e4145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e259c640 1 -- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7e40c6cb0 msgr2=0x7fc7e40a9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:56.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e1d9b640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7e40c6cb0 0x7fc7e40a9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:56.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e259c640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7e40c6cb0 0x7fc7e40a9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e259c640 1 -- 172.21.15.150:0/3809791558 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e40705c0 con 0x7fc7e40a85b0 2024-09-17T04:54:56.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e259c640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7e40a85b0 0x7fc7e4071e20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009790 tx=0x7fc7d40a65c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:56.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7daffd640 1 -- 172.21.15.150:0/3809791558 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4004030 con 0x7fc7e40a85b0 2024-09-17T04:54:56.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7daffd640 1 -- 172.21.15.150:0/3809791558 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7d40041d0 con 0x7fc7e40a85b0 2024-09-17T04:54:56.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3809791558 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7e41f04b0 con 0x7fc7e40a85b0 2024-09-17T04:54:56.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7daffd640 1 -- 172.21.15.150:0/3809791558 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4003600 con 0x7fc7e40a85b0 2024-09-17T04:54:56.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3809791558 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7e4072360 con 0x7fc7e40a85b0 2024-09-17T04:54:56.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3809791558 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7e40ae340 con 0x7fc7e40a85b0 2024-09-17T04:54:56.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7daffd640 1 -- 172.21.15.150:0/3809791558 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc7d40b8030 con 0x7fc7e40a85b0 2024-09-17T04:54:56.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.243+0000 7fc7daffd640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7c4073200 0x7fc7c4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:56.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.247+0000 7fc7daffd640 1 -- 172.21.15.150:0/3809791558 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7fc7d412e200 con 0x7fc7e40a85b0 2024-09-17T04:54:56.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.247+0000 7fc7e1d9b640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7c4073200 0x7fc7c4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:56.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.247+0000 7fc7e1d9b640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7c4073200 0x7fc7c4075710 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc001230 tx=0x7fc7cc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:56.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.251+0000 7fc7daffd640 1 -- 172.21.15.150:0/3809791558 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc7d40bf070 con 0x7fc7e40a85b0 2024-09-17T04:54:56.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.331+0000 7f450fa7a640 1 -- 172.21.15.150:0/2641509788 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4508076ad0 con 0x7f45081506d0 2024-09-17T04:54:56.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.331+0000 7f44fe7fc640 1 -- 172.21.15.150:0/2641509788 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5705 (secure 0 0 0) 0x7f44f80fa5d0 con 0x7f45081506d0 2024-09-17T04:54:56.334 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:56.334 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"btime":"2024-09-17T04:54:55:867317+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:55.867307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:54:56.334 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-09-17T04:54:56.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.335+0000 7f450fa7a640 1 -- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f44ec073260 msgr2=0x7f44ec075770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:56.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.335+0000 7f450fa7a640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f44ec073260 0x7f44ec075770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4504009bc0 tx=0x7f45040091e0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.335+0000 7f450fa7a640 1 -- 172.21.15.150:0/2641509788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45081506d0 msgr2=0x7f45080ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:56.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.335+0000 7f450fa7a640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45081506d0 0x7f45080ca7a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f44f8009790 tx=0x7f44f80a64f0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.335+0000 7f450fa7a640 1 -- 172.21.15.150:0/2641509788 shutdown_connections 2024-09-17T04:54:56.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.335+0000 7f450fa7a640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f44ec073260 0x7f44ec075770 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.335+0000 7f450fa7a640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4508156820 0x7f45080cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.335+0000 7f450fa7a640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45081506d0 0x7f45080ca7a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.335+0000 7f450fa7a640 1 --2- 172.21.15.150:0/2641509788 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45080c2780 0x7f450814df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.335+0000 7f450fa7a640 1 -- 172.21.15.150:0/2641509788 >> 172.21.15.150:0/2641509788 conn(0x7f4508074b80 msgr2=0x7f450806f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:56.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.335+0000 7f450fa7a640 1 -- 172.21.15.150:0/2641509788 shutdown_connections 2024-09-17T04:54:56.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.335+0000 7f450fa7a640 1 -- 172.21.15.150:0/2641509788 wait complete. 2024-09-17T04:54:56.355 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T04:54:55:867317+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 35, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5259": { "addr": "172.21.15.150:6833/3532647866", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 3532647866, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 3532647866, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5259, "incarnation": 32, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:53:53.928936+0000", "qdb_cluster": [ 5259 ], "qdb_leader": 5259, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5259 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5107": { "addr": "172.21.15.176:6837/2374314256", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 2374314256, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 2374314256, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5107, "incarnation": 37, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:54:55.867307+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5107 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:54:56.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.443+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3809791558 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7e40c4ed0 con 0x7fc7e40a85b0 2024-09-17T04:54:56.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7daffd640 1 -- 172.21.15.150:0/3809791558 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5705 (secure 0 0 0) 0x7fc7d40ac030 con 0x7fc7e40a85b0 2024-09-17T04:54:56.447 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:56.447 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"btime":"2024-09-17T04:54:55:867317+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:55.867307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:54:56.448 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-09-17T04:54:56.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7c4073200 msgr2=0x7fc7c4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:56.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7e8e04640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7c4073200 0x7fc7c4075710 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc001230 tx=0x7fc7cc009040 comp rx=0 tx=0).stop 2024-09-17T04:54:56.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3809791558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7e40a85b0 msgr2=0x7fc7e4071e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:56.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7e8e04640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7e40a85b0 0x7fc7e4071e20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009790 tx=0x7fc7d40a65c0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3809791558 shutdown_connections 2024-09-17T04:54:56.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7e8e04640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7c4073200 0x7fc7c4075710 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7e8e04640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc7e40c6cb0 0x7fc7e40a9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7e8e04640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc7e4151980 0x7fc7e4145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7e8e04640 1 --2- 172.21.15.150:0/3809791558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7e40a85b0 0x7fc7e4071e20 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:56.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3809791558 >> 172.21.15.150:0/3809791558 conn(0x7fc7e40aab60 msgr2=0x7fc7e4151d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:56.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3809791558 shutdown_connections 2024-09-17T04:54:56.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:56.447+0000 7fc7e8e04640 1 -- 172.21.15.150:0/3809791558 wait complete. 2024-09-17T04:54:56.468 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:54:58.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:58.469 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:54:58.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.519+0000 7f423dd07640 1 -- 172.21.15.150:0/3614270573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f423806fca0 msgr2=0x7f423806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:58.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.519+0000 7f423dd07640 1 --2- 172.21.15.150:0/3614270573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f423806fca0 0x7f423806e090 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4220009840 tx=0x7f422001c790 comp rx=0 tx=0).stop 2024-09-17T04:54:58.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.519+0000 7f423dd07640 1 -- 172.21.15.150:0/3614270573 shutdown_connections 2024-09-17T04:54:58.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.519+0000 7f423dd07640 1 --2- 172.21.15.150:0/3614270573 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f423806e5d0 0x7f423815dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.519+0000 7f423dd07640 1 --2- 172.21.15.150:0/3614270573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f423806fca0 0x7f423806e090 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.519+0000 7f423dd07640 1 -- 172.21.15.150:0/3614270573 >> 172.21.15.150:0/3614270573 conn(0x7f42380c5ae0 msgr2=0x7f4238073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:58.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.519+0000 7f423dd07640 1 -- 172.21.15.150:0/3614270573 shutdown_connections 2024-09-17T04:54:58.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.519+0000 7f423dd07640 1 -- 172.21.15.150:0/3614270573 wait complete. 2024-09-17T04:54:58.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f423dd07640 1 Processor -- start 2024-09-17T04:54:58.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f423dd07640 1 -- start start 2024-09-17T04:54:58.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f423dd07640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f423806e5d0 0x7f42380a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:58.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f423dd07640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f423806fca0 0x7f4238151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:58.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f423dd07640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42380745c0 0x7f4238148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f423dd07640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4238070610 con 0x7f423806fca0 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f423dd07640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4238070790 con 0x7f423806e5d0 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f423dd07640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f42380ab140 con 0x7f42380745c0 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f4236ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f423806fca0 0x7f4238151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f4236ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f423806fca0 0x7f4238151e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55468/0 (socket says 172.21.15.150:55468) 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f4236ffd640 1 -- 172.21.15.150:0/1646797381 learned_addr learned my addr 172.21.15.150:0/1646797381 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f4236ffd640 1 -- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42380745c0 msgr2=0x7f4238148f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f42377fe640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f423806e5d0 0x7f42380a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f4237fff640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42380745c0 0x7f4238148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f4236ffd640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42380745c0 0x7f4238148f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f4236ffd640 1 -- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f423806e5d0 msgr2=0x7f42380a9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f4236ffd640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f423806e5d0 0x7f42380a9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f4236ffd640 1 -- 172.21.15.150:0/1646797381 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4238151650 con 0x7f423806fca0 2024-09-17T04:54:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f42377fe640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f423806e5d0 0x7f42380a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:54:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f4236ffd640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f423806fca0 0x7f4238151e30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f422001ae40 tx=0x7f42200a5670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f422ffff640 1 -- 172.21.15.150:0/1646797381 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42200bc070 con 0x7f423806fca0 2024-09-17T04:54:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f422ffff640 1 -- 172.21.15.150:0/1646797381 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4220004430 con 0x7f423806fca0 2024-09-17T04:54:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f423dd07640 1 -- 172.21.15.150:0/1646797381 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4238152f30 con 0x7f423806fca0 2024-09-17T04:54:58.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f422ffff640 1 -- 172.21.15.150:0/1646797381 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42200b2040 con 0x7f423806fca0 2024-09-17T04:54:58.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.523+0000 7f423dd07640 1 -- 172.21.15.150:0/1646797381 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4238151910 con 0x7f423806fca0 2024-09-17T04:54:58.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.527+0000 7f423dd07640 1 -- 172.21.15.150:0/1646797381 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f423806e090 con 0x7f423806fca0 2024-09-17T04:54:58.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.527+0000 7f422ffff640 1 -- 172.21.15.150:0/1646797381 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4220003860 con 0x7f423806fca0 2024-09-17T04:54:58.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.527+0000 7f422ffff640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4218073230 0x7f4218075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:58.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.527+0000 7f422ffff640 1 -- 172.21.15.150:0/1646797381 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f422012e1c0 con 0x7f423806fca0 2024-09-17T04:54:58.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.527+0000 7f42377fe640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4218073230 0x7f4218075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:58.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.527+0000 7f42377fe640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4218073230 0x7f4218075740 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4228001630 tx=0x7f4228009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:58.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.531+0000 7f422ffff640 1 -- 172.21.15.150:0/1646797381 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f42200f2db0 con 0x7f423806fca0 2024-09-17T04:54:58.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.631+0000 7fba40fd1640 1 -- 172.21.15.150:0/2572863675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba28004810 msgr2=0x7fba28004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:58.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.631+0000 7fba40fd1640 1 --2- 172.21.15.150:0/2572863675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba28004810 0x7fba28004c20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fba24009b60 tx=0x7fba2401d270 comp rx=0 tx=0).stop 2024-09-17T04:54:58.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.631+0000 7fba40fd1640 1 -- 172.21.15.150:0/2572863675 shutdown_connections 2024-09-17T04:54:58.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.631+0000 7fba40fd1640 1 --2- 172.21.15.150:0/2572863675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba28004810 0x7fba28004c20 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.631+0000 7fba40fd1640 1 -- 172.21.15.150:0/2572863675 >> 172.21.15.150:0/2572863675 conn(0x7fba3c0aaa80 msgr2=0x7fba3c0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:58.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.631+0000 7fba40fd1640 1 -- 172.21.15.150:0/2572863675 shutdown_connections 2024-09-17T04:54:58.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.631+0000 7fba40fd1640 1 -- 172.21.15.150:0/2572863675 wait complete. 2024-09-17T04:54:58.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba40fd1640 1 Processor -- start 2024-09-17T04:54:58.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba40fd1640 1 -- start start 2024-09-17T04:54:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba40fd1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba28004810 0x7fba3c14ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba40fd1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba3c157970 0x7fba3c157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba40fd1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c1edc70 0x7fba3c1ee080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba40fd1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c075370 con 0x7fba3c1edc70 2024-09-17T04:54:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba40fd1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c0c8230 con 0x7fba3c157970 2024-09-17T04:54:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba40fd1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fba3c0c75f0 con 0x7fba28004810 2024-09-17T04:54:58.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba3ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c1edc70 0x7fba3c1ee080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:58.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba3ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c1edc70 0x7fba3c1ee080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55494/0 (socket says 172.21.15.150:55494) 2024-09-17T04:54:58.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba3ad9d640 1 -- 172.21.15.150:0/1652637884 learned_addr learned my addr 172.21.15.150:0/1652637884 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:54:58.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba33fff640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba3c157970 0x7fba3c157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:58.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba3a59c640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba28004810 0x7fba3c14ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:58.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba3ad9d640 1 -- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba28004810 msgr2=0x7fba3c14ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:58.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba3ad9d640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba28004810 0x7fba3c14ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba3ad9d640 1 -- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba3c157970 msgr2=0x7fba3c157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:58.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba3ad9d640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba3c157970 0x7fba3c157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba3ad9d640 1 -- 172.21.15.150:0/1652637884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba3c14c6e0 con 0x7fba3c1edc70 2024-09-17T04:54:58.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba3ad9d640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c1edc70 0x7fba3c1ee080 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0c5410 tx=0x7fba3400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:58.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba32ffd640 1 -- 172.21.15.150:0/1652637884 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba34018070 con 0x7fba3c1edc70 2024-09-17T04:54:58.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba32ffd640 1 -- 172.21.15.150:0/1652637884 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba34003980 con 0x7fba3c1edc70 2024-09-17T04:54:58.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba40fd1640 1 -- 172.21.15.150:0/1652637884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba3c1ee5c0 con 0x7fba3c1edc70 2024-09-17T04:54:58.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba32ffd640 1 -- 172.21.15.150:0/1652637884 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba34003db0 con 0x7fba3c1edc70 2024-09-17T04:54:58.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba40fd1640 1 -- 172.21.15.150:0/1652637884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba3c1ee890 con 0x7fba3c1edc70 2024-09-17T04:54:58.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.635+0000 7fba40fd1640 1 -- 172.21.15.150:0/1652637884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba3c074420 con 0x7fba3c1edc70 2024-09-17T04:54:58.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.639+0000 7fba32ffd640 1 -- 172.21.15.150:0/1652637884 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fba3401b070 con 0x7fba3c1edc70 2024-09-17T04:54:58.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.639+0000 7fba32ffd640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba1c073200 0x7fba1c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:54:58.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.639+0000 7fba32ffd640 1 -- 172.21.15.150:0/1652637884 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7fba340940a0 con 0x7fba3c1edc70 2024-09-17T04:54:58.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.639+0000 7fba3a59c640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba1c073200 0x7fba1c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:54:58.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.639+0000 7fba3a59c640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba1c073200 0x7fba1c075710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fba2c001230 tx=0x7fba2c00c040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:54:58.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.643+0000 7fba32ffd640 1 -- 172.21.15.150:0/1652637884 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fba34022020 con 0x7fba3c1edc70 2024-09-17T04:54:58.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.755+0000 7f423dd07640 1 -- 172.21.15.150:0/1646797381 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4238151080 con 0x7f423806fca0 2024-09-17T04:54:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.755+0000 7f422ffff640 1 -- 172.21.15.150:0/1646797381 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+5706 (secure 0 0 0) 0x7f4220016410 con 0x7f423806fca0 2024-09-17T04:54:58.758 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:58.758 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":39,"btime":"2024-09-17T04:54:57:872694+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:56.878107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:54:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 39 2024-09-17T04:54:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.759+0000 7f423dd07640 1 -- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4218073230 msgr2=0x7f4218075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.759+0000 7f423dd07640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4218073230 0x7f4218075740 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4228001630 tx=0x7f4228009040 comp rx=0 tx=0).stop 2024-09-17T04:54:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.759+0000 7f423dd07640 1 -- 172.21.15.150:0/1646797381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f423806fca0 msgr2=0x7f4238151e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.759+0000 7f423dd07640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f423806fca0 0x7f4238151e30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f422001ae40 tx=0x7f42200a5670 comp rx=0 tx=0).stop 2024-09-17T04:54:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.759+0000 7f423dd07640 1 -- 172.21.15.150:0/1646797381 shutdown_connections 2024-09-17T04:54:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.759+0000 7f423dd07640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4218073230 0x7f4218075740 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.759+0000 7f423dd07640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42380745c0 0x7f4238148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.759+0000 7f423dd07640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f423806fca0 0x7f4238151e30 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.759+0000 7f423dd07640 1 --2- 172.21.15.150:0/1646797381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f423806e5d0 0x7f42380a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.759+0000 7f423dd07640 1 -- 172.21.15.150:0/1646797381 >> 172.21.15.150:0/1646797381 conn(0x7f42380c5ae0 msgr2=0x7f423814bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.759+0000 7f423dd07640 1 -- 172.21.15.150:0/1646797381 shutdown_connections 2024-09-17T04:54:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.759+0000 7f423dd07640 1 -- 172.21.15.150:0/1646797381 wait complete. 2024-09-17T04:54:58.780 INFO:tasks.mds_thrash.fs.[b]:no change 2024-09-17T04:54:58.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.847+0000 7fba40fd1640 1 -- 172.21.15.150:0/1652637884 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba3c0c5eb0 con 0x7fba3c1edc70 2024-09-17T04:54:58.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.847+0000 7fba32ffd640 1 -- 172.21.15.150:0/1652637884 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+5706 (secure 0 0 0) 0x7fba3405ecd0 con 0x7fba3c1edc70 2024-09-17T04:54:58.850 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:54:58.850 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":39,"btime":"2024-09-17T04:54:57:872694+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:56.878107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:54:58.851 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 39 2024-09-17T04:54:58.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.851+0000 7fba40fd1640 1 -- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba1c073200 msgr2=0x7fba1c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:58.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.851+0000 7fba40fd1640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba1c073200 0x7fba1c075710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fba2c001230 tx=0x7fba2c00c040 comp rx=0 tx=0).stop 2024-09-17T04:54:58.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.851+0000 7fba40fd1640 1 -- 172.21.15.150:0/1652637884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c1edc70 msgr2=0x7fba3c1ee080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:54:58.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.851+0000 7fba40fd1640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c1edc70 0x7fba3c1ee080 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0c5410 tx=0x7fba3400dc10 comp rx=0 tx=0).stop 2024-09-17T04:54:58.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.851+0000 7fba40fd1640 1 -- 172.21.15.150:0/1652637884 shutdown_connections 2024-09-17T04:54:58.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.851+0000 7fba40fd1640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fba1c073200 0x7fba1c075710 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.851+0000 7fba40fd1640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba3c157970 0x7fba3c157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.851+0000 7fba40fd1640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba3c1edc70 0x7fba3c1ee080 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.851+0000 7fba40fd1640 1 --2- 172.21.15.150:0/1652637884 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fba28004810 0x7fba3c14ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:54:58.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.851+0000 7fba40fd1640 1 -- 172.21.15.150:0/1652637884 >> 172.21.15.150:0/1652637884 conn(0x7fba3c0aaa80 msgr2=0x7fba3c06d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:54:58.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.851+0000 7fba40fd1640 1 -- 172.21.15.150:0/1652637884 shutdown_connections 2024-09-17T04:54:58.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:54:58.851+0000 7fba40fd1640 1 -- 172.21.15.150:0/1652637884 wait complete. 2024-09-17T04:54:58.869 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:54:59.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:54:59.638 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:55:00.781 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:00.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:00.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.939+0000 7f0999cf3640 1 -- 172.21.15.150:0/2854965697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f099406df60 msgr2=0x7f099406e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:00.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.939+0000 7f0999cf3640 1 --2- 172.21.15.150:0/2854965697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f099406df60 0x7f099406e370 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f099406e970 tx=0x7f097c01c7a0 comp rx=0 tx=0).stop 2024-09-17T04:55:00.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.939+0000 7f0999cf3640 1 -- 172.21.15.150:0/2854965697 shutdown_connections 2024-09-17T04:55:00.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.939+0000 7f0999cf3640 1 --2- 172.21.15.150:0/2854965697 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09941536e0 0x7f0994153af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:00.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.939+0000 7f0999cf3640 1 --2- 172.21.15.150:0/2854965697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f099406df60 0x7f099406e370 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:00.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.939+0000 7f0999cf3640 1 --2- 172.21.15.150:0/2854965697 >> v2:172.21.15.176:3300/0 conn(0x7f09940ab9d0 0x7f099414da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:00.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.939+0000 7f0999cf3640 1 -- 172.21.15.150:0/2854965697 >> 172.21.15.150:0/2854965697 conn(0x7f0994070b50 msgr2=0x7f0994070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:00.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.939+0000 7f0999cf3640 1 -- 172.21.15.150:0/2854965697 shutdown_connections 2024-09-17T04:55:00.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.939+0000 7f0999cf3640 1 -- 172.21.15.150:0/2854965697 wait complete. 2024-09-17T04:55:00.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0999cf3640 1 Processor -- start 2024-09-17T04:55:00.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0999cf3640 1 -- start start 2024-09-17T04:55:00.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0999cf3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f099406df60 0x7f09941509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:00.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0999cf3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09940ab9d0 0x7f09941500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:00.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0999cf3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09941536e0 0x7f09940acef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:00.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0999cf3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0994157510 con 0x7f09940ab9d0 2024-09-17T04:55:00.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0999cf3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0994069e00 con 0x7f09941536e0 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0999cf3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0994157690 con 0x7f099406df60 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f098affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09940ab9d0 0x7f09941500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f098affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09940ab9d0 0x7f09941500f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52638/0 (socket says 172.21.15.150:52638) 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f098affd640 1 -- 172.21.15.150:0/2202875986 learned_addr learned my addr 172.21.15.150:0/2202875986 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f09937fe640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f099406df60 0x7f09941509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0993fff640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09941536e0 0x7f09940acef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f098affd640 1 -- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f099406df60 msgr2=0x7f09941509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f098affd640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f099406df60 0x7f09941509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f098affd640 1 -- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09941536e0 msgr2=0x7f09940acef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f098affd640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09941536e0 0x7f09940acef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f098affd640 1 -- 172.21.15.150:0/2202875986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0994150ff0 con 0x7f09940ab9d0 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0993fff640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09941536e0 0x7f09940acef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:55:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f098affd640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09940ab9d0 0x7f09941500f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f097c01cc80 tx=0x7f097c0a64b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:00.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0990ff9640 1 -- 172.21.15.150:0/2202875986 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c0ac070 con 0x7f09940ab9d0 2024-09-17T04:55:00.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0990ff9640 1 -- 172.21.15.150:0/2202875986 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f097c004320 con 0x7f09940ab9d0 2024-09-17T04:55:00.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0999cf3640 1 -- 172.21.15.150:0/2202875986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09941470c0 con 0x7f09940ab9d0 2024-09-17T04:55:00.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0990ff9640 1 -- 172.21.15.150:0/2202875986 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c005320 con 0x7f09940ab9d0 2024-09-17T04:55:00.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.943+0000 7f0999cf3640 1 -- 172.21.15.150:0/2202875986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0994147470 con 0x7f09940ab9d0 2024-09-17T04:55:00.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.947+0000 7f0999cf3640 1 -- 172.21.15.150:0/2202875986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f099414da10 con 0x7f09940ab9d0 2024-09-17T04:55:00.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.947+0000 7f0990ff9640 1 -- 172.21.15.150:0/2202875986 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f097c0af070 con 0x7f09940ab9d0 2024-09-17T04:55:00.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.947+0000 7f0990ff9640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0974073200 0x7f0974075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:00.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.947+0000 7f0990ff9640 1 -- 172.21.15.150:0/2202875986 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f097c12e750 con 0x7f09940ab9d0 2024-09-17T04:55:00.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.947+0000 7f09937fe640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0974073200 0x7f0974075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:00.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.947+0000 7f09937fe640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0974073200 0x7f0974075710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f098400bad0 tx=0x7f098400b260 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:00.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:00.951+0000 7f0990ff9640 1 -- 172.21.15.150:0/2202875986 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f097c0164b0 con 0x7f09940ab9d0 2024-09-17T04:55:01.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.035+0000 7f04103f9640 1 -- 172.21.15.150:0/212975311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04080c3900 msgr2=0x7f0408074920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.035+0000 7f04103f9640 1 --2- 172.21.15.150:0/212975311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04080c3900 0x7f0408074920 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009840 tx=0x7f03f801c7e0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.035+0000 7f04103f9640 1 -- 172.21.15.150:0/212975311 shutdown_connections 2024-09-17T04:55:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.035+0000 7f04103f9640 1 --2- 172.21.15.150:0/212975311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04080c3900 0x7f0408074920 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.035+0000 7f04103f9640 1 --2- 172.21.15.150:0/212975311 >> v2:172.21.15.176:3300/0 conn(0x7f04080c4250 0x7f0408153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.035+0000 7f04103f9640 1 -- 172.21.15.150:0/212975311 >> 172.21.15.150:0/212975311 conn(0x7f040806e360 msgr2=0x7f040806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.035+0000 7f04103f9640 1 -- 172.21.15.150:0/212975311 shutdown_connections 2024-09-17T04:55:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.035+0000 7f04103f9640 1 -- 172.21.15.150:0/212975311 wait complete. 2024-09-17T04:55:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f04103f9640 1 Processor -- start 2024-09-17T04:55:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f04103f9640 1 -- start start 2024-09-17T04:55:01.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f04103f9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0408074e60 0x7f04080c2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:01.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f04103f9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04080c3900 0x7f04081550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:01.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f04103f9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04080c4250 0x7f04081f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:01.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f04103f9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0408072f30 con 0x7f04080c4250 2024-09-17T04:55:01.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f04103f9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0408072db0 con 0x7f0408074e60 2024-09-17T04:55:01.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f04103f9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0408160f90 con 0x7f04080c3900 2024-09-17T04:55:01.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f040e996640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04080c4250 0x7f04081f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:01.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f040e996640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04080c4250 0x7f04081f3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52658/0 (socket says 172.21.15.150:52658) 2024-09-17T04:55:01.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f040e996640 1 -- 172.21.15.150:0/3193067946 learned_addr learned my addr 172.21.15.150:0/3193067946 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:01.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f040d994640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04080c3900 0x7f04081550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:01.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f040e195640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0408074e60 0x7f04080c2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:01.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f040e996640 1 -- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04080c3900 msgr2=0x7f04081550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:01.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f040e996640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04080c3900 0x7f04081550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f040e996640 1 -- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0408074e60 msgr2=0x7f04080c2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:01.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f040e996640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0408074e60 0x7f04080c2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f040e996640 1 -- 172.21.15.150:0/3193067946 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04080a8c40 con 0x7f04080c4250 2024-09-17T04:55:01.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f040e996640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04080c4250 0x7f04081f3160 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f040400c3c0 tx=0x7f040400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:01.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f03feffd640 1 -- 172.21.15.150:0/3193067946 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0404011790 con 0x7f04080c4250 2024-09-17T04:55:01.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f03feffd640 1 -- 172.21.15.150:0/3193067946 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0404017070 con 0x7f04080c4250 2024-09-17T04:55:01.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f04103f9640 1 -- 172.21.15.150:0/3193067946 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04080a8960 con 0x7f04080c4250 2024-09-17T04:55:01.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f03feffd640 1 -- 172.21.15.150:0/3193067946 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0404011ca0 con 0x7f04080c4250 2024-09-17T04:55:01.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f04103f9640 1 -- 172.21.15.150:0/3193067946 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f040814b090 con 0x7f04080c4250 2024-09-17T04:55:01.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.039+0000 7f04103f9640 1 -- 172.21.15.150:0/3193067946 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0408074920 con 0x7f04080c4250 2024-09-17T04:55:01.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.043+0000 7f03feffd640 1 -- 172.21.15.150:0/3193067946 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f040401c460 con 0x7f04080c4250 2024-09-17T04:55:01.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.043+0000 7f03feffd640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03ec073230 0x7f03ec075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:01.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.043+0000 7f03feffd640 1 -- 172.21.15.150:0/3193067946 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f0404094420 con 0x7f04080c4250 2024-09-17T04:55:01.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.047+0000 7f040e195640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03ec073230 0x7f03ec075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:01.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.047+0000 7f040e195640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03ec073230 0x7f03ec075740 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0400005c50 tx=0x7f0400005be0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:01.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.047+0000 7f03feffd640 1 -- 172.21.15.150:0/3193067946 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f040405eed0 con 0x7f04080c4250 2024-09-17T04:55:01.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.171+0000 7f0999cf3640 1 -- 172.21.15.150:0/2202875986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0994075060 con 0x7f09940ab9d0 2024-09-17T04:55:01.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0990ff9640 1 -- 172.21.15.150:0/2202875986 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+5713 (secure 0 0 0) 0x7f097c0f9320 con 0x7f09940ab9d0 2024-09-17T04:55:01.175 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:01.175 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":40,"btime":"2024-09-17T04:54:58:881339+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:01.176 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 40 2024-09-17T04:55:01.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0999cf3640 1 -- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0974073200 msgr2=0x7f0974075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:01.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0999cf3640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0974073200 0x7f0974075710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f098400bad0 tx=0x7f098400b260 comp rx=0 tx=0).stop 2024-09-17T04:55:01.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0999cf3640 1 -- 172.21.15.150:0/2202875986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09940ab9d0 msgr2=0x7f09941500f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:01.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0999cf3640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09940ab9d0 0x7f09941500f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f097c01cc80 tx=0x7f097c0a64b0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0999cf3640 1 -- 172.21.15.150:0/2202875986 shutdown_connections 2024-09-17T04:55:01.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0999cf3640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0974073200 0x7f0974075710 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0999cf3640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09941536e0 0x7f09940acef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0999cf3640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09940ab9d0 0x7f09941500f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0999cf3640 1 --2- 172.21.15.150:0/2202875986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f099406df60 0x7f09941509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0999cf3640 1 -- 172.21.15.150:0/2202875986 >> 172.21.15.150:0/2202875986 conn(0x7f0994070b50 msgr2=0x7f09940aa390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:01.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0999cf3640 1 -- 172.21.15.150:0/2202875986 shutdown_connections 2024-09-17T04:55:01.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.175+0000 7f0999cf3640 1 -- 172.21.15.150:0/2202875986 wait complete. 2024-09-17T04:55:01.198 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=4975 2024-09-17T04:55:01.198 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.a 2024-09-17T04:55:01.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.259+0000 7f04103f9640 1 -- 172.21.15.150:0/3193067946 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04080a8370 con 0x7f04080c4250 2024-09-17T04:55:01.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.259+0000 7f03feffd640 1 -- 172.21.15.150:0/3193067946 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+5713 (secure 0 0 0) 0x7f040405e7d0 con 0x7f04080c4250 2024-09-17T04:55:01.262 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:01.262 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":40,"btime":"2024-09-17T04:54:58:881339+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:01.262 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 40 2024-09-17T04:55:01.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.263+0000 7f04103f9640 1 -- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03ec073230 msgr2=0x7f03ec075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:01.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.263+0000 7f04103f9640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03ec073230 0x7f03ec075740 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0400005c50 tx=0x7f0400005be0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.263+0000 7f04103f9640 1 -- 172.21.15.150:0/3193067946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04080c4250 msgr2=0x7f04081f3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:01.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.263+0000 7f04103f9640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04080c4250 0x7f04081f3160 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f040400c3c0 tx=0x7f040400c880 comp rx=0 tx=0).stop 2024-09-17T04:55:01.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.263+0000 7f04103f9640 1 -- 172.21.15.150:0/3193067946 shutdown_connections 2024-09-17T04:55:01.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.263+0000 7f04103f9640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03ec073230 0x7f03ec075740 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.263+0000 7f04103f9640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04080c4250 0x7f04081f3160 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.263+0000 7f04103f9640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04080c3900 0x7f04081550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.263+0000 7f04103f9640 1 --2- 172.21.15.150:0/3193067946 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0408074e60 0x7f04080c2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:01.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.263+0000 7f04103f9640 1 -- 172.21.15.150:0/3193067946 >> 172.21.15.150:0/3193067946 conn(0x7f040806e360 msgr2=0x7f04081501c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:01.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.263+0000 7f04103f9640 1 -- 172.21.15.150:0/3193067946 shutdown_connections 2024-09-17T04:55:01.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:01.263+0000 7f04103f9640 1 -- 172.21.15.150:0/3193067946 wait complete. 2024-09-17T04:55:01.289 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:55:01.965 INFO:tasks.ceph.mds.d.smithi176.stderr:2024-09-17T04:55:01.963+0000 7f0a378c2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:55:02.199 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-17T04:55:02.199 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:55:02.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:55:02.202 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:55:02.202 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:02.305 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T04:55:02.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 -- 172.21.15.150:0/624959079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d0003a40 msgr2=0x7f19d0003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 --2- 172.21.15.150:0/624959079 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d0003a40 0x7f19d0003eb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0016f0 tx=0x7f19dc01c7b0 comp rx=0 tx=0).stop 2024-09-17T04:55:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e7fff640 1 -- 172.21.15.150:0/624959079 reap_dead start 2024-09-17T04:55:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 -- 172.21.15.150:0/624959079 shutdown_connections 2024-09-17T04:55:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 -- 172.21.15.150:0/624959079 >> 172.21.15.150:0/624959079 conn(0x7f19e8074b80 msgr2=0x7f19e806f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 -- 172.21.15.150:0/624959079 shutdown_connections 2024-09-17T04:55:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 -- 172.21.15.150:0/624959079 wait complete. 2024-09-17T04:55:02.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 Processor -- start 2024-09-17T04:55:02.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 -- start start 2024-09-17T04:55:02.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d0003a40 0x7f19e81593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:02.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e81ef9e0 0x7f19e81efdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19e8144800 0x7f19e8144c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f19e806fd80 con 0x7f19d0003a40 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f19e81544e0 con 0x7f19e8144800 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19ee01c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f19e806ff00 con 0x7f19e81ef9e0 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d0003a40 0x7f19e81593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d0003a40 0x7f19e81593c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52710/0 (socket says 172.21.15.150:52710) 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e77fe640 1 -- 172.21.15.150:0/3630220189 learned_addr learned my addr 172.21.15.150:0/3630220189 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e7fff640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19e8144800 0x7f19e8144c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e6ffd640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e81ef9e0 0x7f19e81efdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e77fe640 1 -- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e81ef9e0 msgr2=0x7f19e81efdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e77fe640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e81ef9e0 0x7f19e81efdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e77fe640 1 -- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19e8144800 msgr2=0x7f19e8144c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e77fe640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19e8144800 0x7f19e8144c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:02.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e77fe640 1 -- 172.21.15.150:0/3630220189 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e80c6480 con 0x7f19d0003a40 2024-09-17T04:55:02.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e6ffd640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e81ef9e0 0x7f19e81efdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:55:02.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.355+0000 7f19e77fe640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d0003a40 0x7f19e81593c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f19dc000c10 tx=0x7f19dc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:02.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.359+0000 7f19cffff640 1 -- 172.21.15.150:0/3630220189 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc0bb070 con 0x7f19d0003a40 2024-09-17T04:55:02.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.359+0000 7f19cffff640 1 -- 172.21.15.150:0/3630220189 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f19dc006500 con 0x7f19d0003a40 2024-09-17T04:55:02.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.359+0000 7f19ee01c640 1 -- 172.21.15.150:0/3630220189 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e80ca120 con 0x7f19d0003a40 2024-09-17T04:55:02.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.359+0000 7f19cffff640 1 -- 172.21.15.150:0/3630220189 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc0ae070 con 0x7f19d0003a40 2024-09-17T04:55:02.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.359+0000 7f19ee01c640 1 -- 172.21.15.150:0/3630220189 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e80cc610 con 0x7f19d0003a40 2024-09-17T04:55:02.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.359+0000 7f19ee01c640 1 -- 172.21.15.150:0/3630220189 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19e80c8a80 con 0x7f19d0003a40 2024-09-17T04:55:02.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.359+0000 7f19cffff640 1 -- 172.21.15.150:0/3630220189 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f19dc0be070 con 0x7f19d0003a40 2024-09-17T04:55:02.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.359+0000 7f19cffff640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f19d4073230 0x7f19d4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:02.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.359+0000 7f19cffff640 1 -- 172.21.15.150:0/3630220189 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f19dc12e080 con 0x7f19d0003a40 2024-09-17T04:55:02.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.359+0000 7f19e6ffd640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f19d4073230 0x7f19d4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:02.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.359+0000 7f19e6ffd640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f19d4073230 0x7f19d4075740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f19d8000fc0 tx=0x7f19d80067b0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:02.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.363+0000 7f19cffff640 1 -- 172.21.15.150:0/3630220189 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f19dc016510 con 0x7f19d0003a40 2024-09-17T04:55:02.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.559+0000 7f19ee01c640 1 -- 172.21.15.150:0/3630220189 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19e80c66d0 con 0x7f19d0003a40 2024-09-17T04:55:02.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.559+0000 7f19cffff640 1 -- 172.21.15.150:0/3630220189 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+5713 (secure 0 0 0) 0x7f19dc0f8570 con 0x7f19d0003a40 2024-09-17T04:55:02.562 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:02.562 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":40,"btime":"2024-09-17T04:54:58:881339+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:02.563 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 40 2024-09-17T04:55:02.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.563+0000 7f19ee01c640 1 -- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f19d4073230 msgr2=0x7f19d4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:02.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.563+0000 7f19ee01c640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f19d4073230 0x7f19d4075740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f19d8000fc0 tx=0x7f19d80067b0 comp rx=0 tx=0).stop 2024-09-17T04:55:02.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.563+0000 7f19ee01c640 1 -- 172.21.15.150:0/3630220189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d0003a40 msgr2=0x7f19e81593c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:02.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.563+0000 7f19ee01c640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d0003a40 0x7f19e81593c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f19dc000c10 tx=0x7f19dc003c10 comp rx=0 tx=0).stop 2024-09-17T04:55:02.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.563+0000 7f19ee01c640 1 -- 172.21.15.150:0/3630220189 shutdown_connections 2024-09-17T04:55:02.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.563+0000 7f19ee01c640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f19d4073230 0x7f19d4075740 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:02.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.563+0000 7f19ee01c640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19e8144800 0x7f19e8144c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:02.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.563+0000 7f19ee01c640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f19e81ef9e0 0x7f19e81efdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:02.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.563+0000 7f19ee01c640 1 --2- 172.21.15.150:0/3630220189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19d0003a40 0x7f19e81593c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:02.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.563+0000 7f19ee01c640 1 -- 172.21.15.150:0/3630220189 >> 172.21.15.150:0/3630220189 conn(0x7f19e8074b80 msgr2=0x7f19e80c8030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:02.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.563+0000 7f19ee01c640 1 -- 172.21.15.150:0/3630220189 shutdown_connections 2024-09-17T04:55:02.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:02.563+0000 7f19ee01c640 1 -- 172.21.15.150:0/3630220189 wait complete. 2024-09-17T04:55:02.582 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-17T04:55:03.290 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:03.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.443+0000 7f8ed1944640 1 -- 172.21.15.150:0/1470098465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ecc0c6a00 msgr2=0x7f8ecc156830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:03.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.443+0000 7f8ed1944640 1 --2- 172.21.15.150:0/1470098465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ecc0c6a00 0x7f8ecc156830 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc08e060 tx=0x7f8ebc01c8c0 comp rx=0 tx=0).stop 2024-09-17T04:55:03.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.443+0000 7f8ed1944640 1 -- 172.21.15.150:0/1470098465 shutdown_connections 2024-09-17T04:55:03.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.443+0000 7f8ed1944640 1 --2- 172.21.15.150:0/1470098465 >> v2:172.21.15.176:3300/0 conn(0x7f8ecc150910 0x7f8ecc0c2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:03.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.443+0000 7f8ed1944640 1 --2- 172.21.15.150:0/1470098465 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ecc0c6a00 0x7f8ecc156830 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:03.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.443+0000 7f8ed1944640 1 -- 172.21.15.150:0/1470098465 >> 172.21.15.150:0/1470098465 conn(0x7f8ecc074b80 msgr2=0x7f8ecc06f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:03.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.443+0000 7f8ed1944640 1 -- 172.21.15.150:0/1470098465 shutdown_connections 2024-09-17T04:55:03.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.443+0000 7f8ed1944640 1 -- 172.21.15.150:0/1470098465 wait complete. 2024-09-17T04:55:03.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ed1944640 1 Processor -- start 2024-09-17T04:55:03.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ed1944640 1 -- start start 2024-09-17T04:55:03.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ed1944640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ecc0c2690 0x7f8ecc148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ed1944640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0c6a00 0x7f8ecc151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ed1944640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ecc150910 0x7f8ecc14d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ed1944640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8ecc0c9320 con 0x7f8ecc0c2690 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ed1944640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8ecc0c91a0 con 0x7f8ecc150910 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ed1944640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8ecc06fd80 con 0x7f8ecc0c6a00 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ecaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ecc0c2690 0x7f8ecc148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ecaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ecc0c2690 0x7f8ecc148400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52728/0 (socket says 172.21.15.150:52728) 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ecaffd640 1 -- 172.21.15.150:0/3803444475 learned_addr learned my addr 172.21.15.150:0/3803444475 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ecb7fe640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ecc150910 0x7f8ecc14d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8eca7fc640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0c6a00 0x7f8ecc151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ecaffd640 1 -- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0c6a00 msgr2=0x7f8ecc151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ecaffd640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0c6a00 0x7f8ecc151d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:03.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ecaffd640 1 -- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ecc150910 msgr2=0x7f8ecc14d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:03.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ecaffd640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ecc150910 0x7f8ecc14d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:03.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ecaffd640 1 -- 172.21.15.150:0/3803444475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ecc150f50 con 0x7f8ecc0c2690 2024-09-17T04:55:03.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ecaffd640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ecc0c2690 0x7f8ecc148400 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0b0040 tx=0x7f8ebc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:03.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8eb3fff640 1 -- 172.21.15.150:0/3803444475 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc0bc020 con 0x7f8ecc0c2690 2024-09-17T04:55:03.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8eb3fff640 1 -- 172.21.15.150:0/3803444475 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ebc003600 con 0x7f8ecc0c2690 2024-09-17T04:55:03.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ed1944640 1 -- 172.21.15.150:0/3803444475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ecc1f3f20 con 0x7f8ecc0c2690 2024-09-17T04:55:03.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8eb3fff640 1 -- 172.21.15.150:0/3803444475 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc0b2040 con 0x7f8ecc0c2690 2024-09-17T04:55:03.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.447+0000 7f8ed1944640 1 -- 172.21.15.150:0/3803444475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ecc148940 con 0x7f8ecc0c2690 2024-09-17T04:55:03.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.451+0000 7f8ed1944640 1 -- 172.21.15.150:0/3803444475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ecc0c2150 con 0x7f8ecc0c2690 2024-09-17T04:55:03.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.451+0000 7f8eb3fff640 1 -- 172.21.15.150:0/3803444475 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8ebc0bf070 con 0x7f8ecc0c2690 2024-09-17T04:55:03.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.451+0000 7f8eb3fff640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8eb8073200 0x7f8eb8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:03.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.451+0000 7f8eb3fff640 1 -- 172.21.15.150:0/3803444475 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f8ebc12e530 con 0x7f8ecc0c2690 2024-09-17T04:55:03.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.451+0000 7f8eca7fc640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8eb8073200 0x7f8eb8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:03.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.451+0000 7f8eca7fc640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8eb8073200 0x7f8eb8075710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc0c7700 tx=0x7f8ec0005d90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:03.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.455+0000 7f8eb3fff640 1 -- 172.21.15.150:0/3803444475 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8ebc0164b0 con 0x7f8ecc0c2690 2024-09-17T04:55:03.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.643+0000 7f8ed1944640 1 -- 172.21.15.150:0/3803444475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ecc0c66e0 con 0x7f8ecc0c2690 2024-09-17T04:55:03.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8eb3fff640 1 -- 172.21.15.150:0/3803444475 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6541 (secure 0 0 0) 0x7f8ebc0f9070 con 0x7f8ecc0c2690 2024-09-17T04:55:03.648 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:03.648 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":41,"btime":"2024-09-17T04:55:03:185493+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:03.648 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 41 2024-09-17T04:55:03.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8ed1944640 1 -- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8eb8073200 msgr2=0x7f8eb8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:03.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8ed1944640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8eb8073200 0x7f8eb8075710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc0c7700 tx=0x7f8ec0005d90 comp rx=0 tx=0).stop 2024-09-17T04:55:03.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8ed1944640 1 -- 172.21.15.150:0/3803444475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ecc0c2690 msgr2=0x7f8ecc148400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:03.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8ed1944640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ecc0c2690 0x7f8ecc148400 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc0b0040 tx=0x7f8ebc004040 comp rx=0 tx=0).stop 2024-09-17T04:55:03.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8ed1944640 1 -- 172.21.15.150:0/3803444475 shutdown_connections 2024-09-17T04:55:03.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8ed1944640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8eb8073200 0x7f8eb8075710 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:03.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8ed1944640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ecc150910 0x7f8ecc14d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:03.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8ed1944640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ecc0c6a00 0x7f8ecc151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:03.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8ed1944640 1 --2- 172.21.15.150:0/3803444475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ecc0c2690 0x7f8ecc148400 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:03.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8ed1944640 1 -- 172.21.15.150:0/3803444475 >> 172.21.15.150:0/3803444475 conn(0x7f8ecc074b80 msgr2=0x7f8ecc14c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:03.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8ed1944640 1 -- 172.21.15.150:0/3803444475 shutdown_connections 2024-09-17T04:55:03.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:03.647+0000 7f8ed1944640 1 -- 172.21.15.150:0/3803444475 wait complete. 2024-09-17T04:55:03.667 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:55:04.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 -- 172.21.15.150:0/1349741016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc071680 msgr2=0x7f31dc13c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 --2- 172.21.15.150:0/1349741016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc071680 0x7f31dc13c8f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f31dc155650 tx=0x7f31d801c7f0 comp rx=0 tx=0).stop 2024-09-17T04:55:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 -- 172.21.15.150:0/1349741016 shutdown_connections 2024-09-17T04:55:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 --2- 172.21.15.150:0/1349741016 >> v2:172.21.15.176:3300/0 conn(0x7f31dc13ce30 0x7f31dc0c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:04.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 --2- 172.21.15.150:0/1349741016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc071680 0x7f31dc13c8f0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:04.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 -- 172.21.15.150:0/1349741016 >> 172.21.15.150:0/1349741016 conn(0x7f31dc0abaa0 msgr2=0x7f31dc0abeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:04.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 -- 172.21.15.150:0/1349741016 shutdown_connections 2024-09-17T04:55:04.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 -- 172.21.15.150:0/1349741016 wait complete. 2024-09-17T04:55:04.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 Processor -- start 2024-09-17T04:55:04.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 -- start start 2024-09-17T04:55:04.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31dc071680 0x7f31dc138ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:04.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31dc13ce30 0x7f31dc154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:04.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc13db50 0x7f31dc156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:04.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc161110 con 0x7f31dc13db50 2024-09-17T04:55:04.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc160f90 con 0x7f31dc13ce30 2024-09-17T04:55:04.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e3e8d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f31dc161290 con 0x7f31dc071680 2024-09-17T04:55:04.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.751+0000 7f31e242a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc13db50 0x7f31dc156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e242a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc13db50 0x7f31dc156a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52752/0 (socket says 172.21.15.150:52752) 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e242a640 1 -- 172.21.15.150:0/3261869131 learned_addr learned my addr 172.21.15.150:0/3261869131 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e1428640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31dc13ce30 0x7f31dc154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e1c29640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31dc071680 0x7f31dc138ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e242a640 1 -- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31dc071680 msgr2=0x7f31dc138ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e242a640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31dc071680 0x7f31dc138ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e242a640 1 -- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31dc13ce30 msgr2=0x7f31dc154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e242a640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31dc13ce30 0x7f31dc154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e242a640 1 -- 172.21.15.150:0/3261869131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc1f15c0 con 0x7f31dc13db50 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e242a640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc13db50 0x7f31dc156a40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f31d400ca90 tx=0x7f31d400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31d27fc640 1 -- 172.21.15.150:0/3261869131 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d4007df0 con 0x7f31dc13db50 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31d27fc640 1 -- 172.21.15.150:0/3261869131 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31d4004b30 con 0x7f31dc13db50 2024-09-17T04:55:04.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e3e8d640 1 -- 172.21.15.150:0/3261869131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31dc1f1860 con 0x7f31dc13db50 2024-09-17T04:55:04.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31d27fc640 1 -- 172.21.15.150:0/3261869131 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d4005020 con 0x7f31dc13db50 2024-09-17T04:55:04.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e3e8d640 1 -- 172.21.15.150:0/3261869131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31dc0bfa10 con 0x7f31dc13db50 2024-09-17T04:55:04.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e3e8d640 1 -- 172.21.15.150:0/3261869131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31dc0c3890 con 0x7f31dc13db50 2024-09-17T04:55:04.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31d27fc640 1 -- 172.21.15.150:0/3261869131 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f31d40051c0 con 0x7f31dc13db50 2024-09-17T04:55:04.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31d27fc640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31c8073200 0x7f31c8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:04.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31d27fc640 1 -- 172.21.15.150:0/3261869131 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f31d4094ea0 con 0x7f31dc13db50 2024-09-17T04:55:04.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.755+0000 7f31e1c29640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31c8073200 0x7f31c8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:04.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.759+0000 7f31e1c29640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31c8073200 0x7f31c8075710 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00b810 tx=0x7f31cc008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:04.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.759+0000 7f31d27fc640 1 -- 172.21.15.150:0/3261869131 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f31d4014030 con 0x7f31dc13db50 2024-09-17T04:55:04.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.963+0000 7f31e3e8d640 1 -- 172.21.15.150:0/3261869131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31dc13c8f0 con 0x7f31dc13db50 2024-09-17T04:55:04.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.963+0000 7f31d27fc640 1 -- 172.21.15.150:0/3261869131 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6541 (secure 0 0 0) 0x7f31d405f9e0 con 0x7f31dc13db50 2024-09-17T04:55:04.966 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:04.966 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":41,"btime":"2024-09-17T04:55:03:185493+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:04.966 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 41 2024-09-17T04:55:04.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.967+0000 7f31e3e8d640 1 -- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31c8073200 msgr2=0x7f31c8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:04.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.967+0000 7f31e3e8d640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31c8073200 0x7f31c8075710 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00b810 tx=0x7f31cc008040 comp rx=0 tx=0).stop 2024-09-17T04:55:04.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.967+0000 7f31e3e8d640 1 -- 172.21.15.150:0/3261869131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc13db50 msgr2=0x7f31dc156a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:04.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.967+0000 7f31e3e8d640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc13db50 0x7f31dc156a40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f31d400ca90 tx=0x7f31d400cf50 comp rx=0 tx=0).stop 2024-09-17T04:55:04.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.967+0000 7f31e3e8d640 1 -- 172.21.15.150:0/3261869131 shutdown_connections 2024-09-17T04:55:04.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.967+0000 7f31e3e8d640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31c8073200 0x7f31c8075710 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:04.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.967+0000 7f31e3e8d640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31dc13ce30 0x7f31dc154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:04.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.967+0000 7f31e3e8d640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31dc13db50 0x7f31dc156a40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:04.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.967+0000 7f31e3e8d640 1 --2- 172.21.15.150:0/3261869131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f31dc071680 0x7f31dc138ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:04.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.967+0000 7f31e3e8d640 1 -- 172.21.15.150:0/3261869131 >> 172.21.15.150:0/3261869131 conn(0x7f31dc0abaa0 msgr2=0x7f31dc0aeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:04.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.967+0000 7f31e3e8d640 1 -- 172.21.15.150:0/3261869131 shutdown_connections 2024-09-17T04:55:04.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:04.967+0000 7f31e3e8d640 1 -- 172.21.15.150:0/3261869131 wait complete. 2024-09-17T04:55:04.986 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-17T04:55:04.986 INFO:tasks.mds_thrash.fs.[b]:waiting for 50 secs before thrashing 2024-09-17T04:55:05.668 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:05.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.827+0000 7f510485f640 1 -- 172.21.15.150:0/1022426640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50ec004810 msgr2=0x7f50ec004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:05.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.827+0000 7f510485f640 1 --2- 172.21.15.150:0/1022426640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50ec004810 0x7f50ec004c20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f50e800c260 tx=0x7f50e801cf40 comp rx=0 tx=0).stop 2024-09-17T04:55:05.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.827+0000 7f510485f640 1 -- 172.21.15.150:0/1022426640 shutdown_connections 2024-09-17T04:55:05.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.827+0000 7f510485f640 1 --2- 172.21.15.150:0/1022426640 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50ec004810 0x7f50ec004c20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:05.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.827+0000 7f510485f640 1 -- 172.21.15.150:0/1022426640 >> 172.21.15.150:0/1022426640 conn(0x7f5100074b80 msgr2=0x7f510006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:05.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.827+0000 7f510485f640 1 -- 172.21.15.150:0/1022426640 shutdown_connections 2024-09-17T04:55:05.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.827+0000 7f510485f640 1 -- 172.21.15.150:0/1022426640 wait complete. 2024-09-17T04:55:05.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f510485f640 1 Processor -- start 2024-09-17T04:55:05.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f510485f640 1 -- start start 2024-09-17T04:55:05.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f510485f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50ec004810 0x7f51001468c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:05.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f510485f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f510014a040 0x7f510014a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:05.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f510485f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5100159850 0x7f5100159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:05.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f510485f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f51000c92a0 con 0x7f5100159850 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f510485f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f51000c2a30 con 0x7f50ec004810 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f510485f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f510006fd80 con 0x7f510014a040 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50fed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5100159850 0x7f5100159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50fed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5100159850 0x7f5100159c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52770/0 (socket says 172.21.15.150:52770) 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50fed9d640 1 -- 172.21.15.150:0/4112680429 learned_addr learned my addr 172.21.15.150:0/4112680429 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50fdd9b640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f510014a040 0x7f510014a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50fe59c640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50ec004810 0x7f51001468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50fed9d640 1 -- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f510014a040 msgr2=0x7f510014a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50fed9d640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f510014a040 0x7f510014a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50fed9d640 1 -- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50ec004810 msgr2=0x7f51001468c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50fed9d640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50ec004810 0x7f51001468c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50fed9d640 1 -- 172.21.15.150:0/4112680429 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51001f0b20 con 0x7f5100159850 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50fed9d640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5100159850 0x7f5100159c60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f51000c3c60 tx=0x7f50f000dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:05.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50f6ffd640 1 -- 172.21.15.150:0/4112680429 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f0003b90 con 0x7f5100159850 2024-09-17T04:55:05.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50f6ffd640 1 -- 172.21.15.150:0/4112680429 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f50f0003e40 con 0x7f5100159850 2024-09-17T04:55:05.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f510485f640 1 -- 172.21.15.150:0/4112680429 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51001f0880 con 0x7f5100159850 2024-09-17T04:55:05.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f50f6ffd640 1 -- 172.21.15.150:0/4112680429 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f000ba90 con 0x7f5100159850 2024-09-17T04:55:05.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.831+0000 7f510485f640 1 -- 172.21.15.150:0/4112680429 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5100158d70 con 0x7f5100159850 2024-09-17T04:55:05.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.835+0000 7f510485f640 1 -- 172.21.15.150:0/4112680429 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5100147200 con 0x7f5100159850 2024-09-17T04:55:05.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.835+0000 7f50f6ffd640 1 -- 172.21.15.150:0/4112680429 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f50f000bc30 con 0x7f5100159850 2024-09-17T04:55:05.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.835+0000 7f50f6ffd640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f50e0073200 0x7f50e0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:05.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.835+0000 7f50f6ffd640 1 -- 172.21.15.150:0/4112680429 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f50f00957d0 con 0x7f5100159850 2024-09-17T04:55:05.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.835+0000 7f50fe59c640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f50e0073200 0x7f50e0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:05.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.835+0000 7f50fe59c640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f50e0073200 0x7f50e0075710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f50f80058d0 tx=0x7f50f8005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:05.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:05.839+0000 7f50f6ffd640 1 -- 172.21.15.150:0/4112680429 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5100147200 con 0x7f5100159850 2024-09-17T04:55:06.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.031+0000 7f510485f640 1 -- 172.21.15.150:0/4112680429 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51000c4f80 con 0x7f5100159850 2024-09-17T04:55:06.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.031+0000 7f50f6ffd640 1 -- 172.21.15.150:0/4112680429 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6541 (secure 0 0 0) 0x7f50f0060380 con 0x7f5100159850 2024-09-17T04:55:06.033 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:06.033 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":41,"btime":"2024-09-17T04:55:03:185493+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:06.034 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 41 2024-09-17T04:55:06.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.035+0000 7f510485f640 1 -- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f50e0073200 msgr2=0x7f50e0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:06.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.035+0000 7f510485f640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f50e0073200 0x7f50e0075710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f50f80058d0 tx=0x7f50f8005860 comp rx=0 tx=0).stop 2024-09-17T04:55:06.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.035+0000 7f510485f640 1 -- 172.21.15.150:0/4112680429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5100159850 msgr2=0x7f5100159c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:06.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.035+0000 7f510485f640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5100159850 0x7f5100159c60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f51000c3c60 tx=0x7f50f000dd80 comp rx=0 tx=0).stop 2024-09-17T04:55:06.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.035+0000 7f510485f640 1 -- 172.21.15.150:0/4112680429 shutdown_connections 2024-09-17T04:55:06.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.035+0000 7f510485f640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f50e0073200 0x7f50e0075710 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:06.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.035+0000 7f510485f640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5100159850 0x7f5100159c60 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:06.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.035+0000 7f510485f640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f510014a040 0x7f510014a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:06.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.035+0000 7f510485f640 1 --2- 172.21.15.150:0/4112680429 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f50ec004810 0x7f51001468c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:06.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.035+0000 7f510485f640 1 -- 172.21.15.150:0/4112680429 >> 172.21.15.150:0/4112680429 conn(0x7f5100074b80 msgr2=0x7f51000c8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:06.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.035+0000 7f510485f640 1 -- 172.21.15.150:0/4112680429 shutdown_connections 2024-09-17T04:55:06.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:06.035+0000 7f510485f640 1 -- 172.21.15.150:0/4112680429 wait complete. 2024-09-17T04:55:06.054 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:55:08.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:08.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.219+0000 7fab4766f640 1 -- 172.21.15.150:0/1410909676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c003ab0 msgr2=0x7fab2c003f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:08.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.219+0000 7fab4766f640 1 --2- 172.21.15.150:0/1410909676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c003ab0 0x7fab2c003f20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009b60 tx=0x7fab3c01d290 comp rx=0 tx=0).stop 2024-09-17T04:55:08.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.219+0000 7fab4766f640 1 -- 172.21.15.150:0/1410909676 shutdown_connections 2024-09-17T04:55:08.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.219+0000 7fab4766f640 1 --2- 172.21.15.150:0/1410909676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab2c003ab0 0x7fab2c003f20 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:08.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.219+0000 7fab4766f640 1 -- 172.21.15.150:0/1410909676 >> 172.21.15.150:0/1410909676 conn(0x7fab400abb00 msgr2=0x7fab400abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:08.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.219+0000 7fab4766f640 1 -- 172.21.15.150:0/1410909676 shutdown_connections 2024-09-17T04:55:08.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.219+0000 7fab4766f640 1 -- 172.21.15.150:0/1410909676 wait complete. 2024-09-17T04:55:08.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab4766f640 1 Processor -- start 2024-09-17T04:55:08.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab4766f640 1 -- start start 2024-09-17T04:55:08.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab4766f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab2c003ab0 0x7fab400c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:08.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab4766f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab400ad360 0x7fab401f03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:08.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab4766f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab401f0920 0x7fab4014e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab4766f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fab400c4d40 con 0x7fab401f0920 2024-09-17T04:55:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab4766f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fab400c4bc0 con 0x7fab400ad360 2024-09-17T04:55:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab4766f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fab40152080 con 0x7fab2c003ab0 2024-09-17T04:55:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab45c0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab401f0920 0x7fab4014e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab45c0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab401f0920 0x7fab4014e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52788/0 (socket says 172.21.15.150:52788) 2024-09-17T04:55:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab45c0c640 1 -- 172.21.15.150:0/2217991301 learned_addr learned my addr 172.21.15.150:0/2217991301 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab4540b640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab2c003ab0 0x7fab400c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab44c0a640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab400ad360 0x7fab401f03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab45c0c640 1 -- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab2c003ab0 msgr2=0x7fab400c5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab45c0c640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab2c003ab0 0x7fab400c5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab45c0c640 1 -- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab400ad360 msgr2=0x7fab401f03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:08.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab45c0c640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab400ad360 0x7fab401f03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:08.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab45c0c640 1 -- 172.21.15.150:0/2217991301 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab400ae710 con 0x7fab401f0920 2024-09-17T04:55:08.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab45c0c640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab401f0920 0x7fab4014e5b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fab400c9bb0 tx=0x7fab380019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:08.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab35ffb640 1 -- 172.21.15.150:0/2217991301 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38011870 con 0x7fab401f0920 2024-09-17T04:55:08.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab35ffb640 1 -- 172.21.15.150:0/2217991301 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab38017070 con 0x7fab401f0920 2024-09-17T04:55:08.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab4766f640 1 -- 172.21.15.150:0/2217991301 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab401538f0 con 0x7fab401f0920 2024-09-17T04:55:08.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab35ffb640 1 -- 172.21.15.150:0/2217991301 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38011da0 con 0x7fab401f0920 2024-09-17T04:55:08.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.223+0000 7fab4766f640 1 -- 172.21.15.150:0/2217991301 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab400af010 con 0x7fab401f0920 2024-09-17T04:55:08.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.227+0000 7fab4766f640 1 -- 172.21.15.150:0/2217991301 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab401584d0 con 0x7fab401f0920 2024-09-17T04:55:08.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.227+0000 7fab35ffb640 1 -- 172.21.15.150:0/2217991301 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fab3801c460 con 0x7fab401f0920 2024-09-17T04:55:08.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.227+0000 7fab35ffb640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fab24073230 0x7fab24075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:08.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.227+0000 7fab35ffb640 1 -- 172.21.15.150:0/2217991301 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7fab38094f00 con 0x7fab401f0920 2024-09-17T04:55:08.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.227+0000 7fab4540b640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fab24073230 0x7fab24075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:08.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.227+0000 7fab4540b640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fab24073230 0x7fab24075740 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0ae000 tx=0x7fab3c01a030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:08.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.231+0000 7fab35ffb640 1 -- 172.21.15.150:0/2217991301 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fab3805fa50 con 0x7fab401f0920 2024-09-17T04:55:08.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.439+0000 7fab4766f640 1 -- 172.21.15.150:0/2217991301 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab400c40f0 con 0x7fab401f0920 2024-09-17T04:55:08.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab35ffb640 1 -- 172.21.15.150:0/2217991301 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6541 (secure 0 0 0) 0x7fab3805f350 con 0x7fab401f0920 2024-09-17T04:55:08.444 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:08.444 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":41,"btime":"2024-09-17T04:55:03:185493+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:08.444 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 41 2024-09-17T04:55:08.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab4766f640 1 -- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fab24073230 msgr2=0x7fab24075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:08.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab4766f640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fab24073230 0x7fab24075740 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0ae000 tx=0x7fab3c01a030 comp rx=0 tx=0).stop 2024-09-17T04:55:08.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab4766f640 1 -- 172.21.15.150:0/2217991301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab401f0920 msgr2=0x7fab4014e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:08.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab4766f640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab401f0920 0x7fab4014e5b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fab400c9bb0 tx=0x7fab380019c0 comp rx=0 tx=0).stop 2024-09-17T04:55:08.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab4766f640 1 -- 172.21.15.150:0/2217991301 shutdown_connections 2024-09-17T04:55:08.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab4766f640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fab24073230 0x7fab24075740 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:08.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab4766f640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab400ad360 0x7fab401f03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:08.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab4766f640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab401f0920 0x7fab4014e5b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:08.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab4766f640 1 --2- 172.21.15.150:0/2217991301 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fab2c003ab0 0x7fab400c5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:08.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab4766f640 1 -- 172.21.15.150:0/2217991301 >> 172.21.15.150:0/2217991301 conn(0x7fab400abb00 msgr2=0x7fab401e3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:08.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab4766f640 1 -- 172.21.15.150:0/2217991301 shutdown_connections 2024-09-17T04:55:08.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:08.443+0000 7fab4766f640 1 -- 172.21.15.150:0/2217991301 wait complete. 2024-09-17T04:55:08.465 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:55:10.466 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:10.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.635+0000 7f6c49113640 1 -- 172.21.15.150:0/385882389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c441540b0 msgr2=0x7f6c440c8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:10.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.635+0000 7f6c49113640 1 --2- 172.21.15.150:0/385882389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c441540b0 0x7f6c440c8330 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6c38009840 tx=0x7f6c3801c7e0 comp rx=0 tx=0).stop 2024-09-17T04:55:10.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.635+0000 7f6c49113640 1 -- 172.21.15.150:0/385882389 shutdown_connections 2024-09-17T04:55:10.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.635+0000 7f6c49113640 1 --2- 172.21.15.150:0/385882389 >> v2:172.21.15.176:3300/0 conn(0x7f6c440c8870 0x7f6c440c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:10.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.635+0000 7f6c49113640 1 --2- 172.21.15.150:0/385882389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c441540b0 0x7f6c440c8330 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:10.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.635+0000 7f6c49113640 1 -- 172.21.15.150:0/385882389 >> 172.21.15.150:0/385882389 conn(0x7f6c440aab60 msgr2=0x7f6c440aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:10.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.635+0000 7f6c49113640 1 -- 172.21.15.150:0/385882389 shutdown_connections 2024-09-17T04:55:10.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.635+0000 7f6c49113640 1 -- 172.21.15.150:0/385882389 wait complete. 2024-09-17T04:55:10.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.635+0000 7f6c49113640 1 Processor -- start 2024-09-17T04:55:10.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.635+0000 7f6c49113640 1 -- start start 2024-09-17T04:55:10.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c49113640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c440c7eb0 0x7f6c440ae440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:10.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c49113640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c440c8870 0x7f6c441f0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:10.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c49113640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c441540b0 0x7f6c44147b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:10.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c49113640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6c440c7ae0 con 0x7f6c440c7eb0 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c49113640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6c441544c0 con 0x7f6c441540b0 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c49113640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6c440c7c60 con 0x7f6c440c8870 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c42d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c440c7eb0 0x7f6c440ae440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c42d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c440c7eb0 0x7f6c440ae440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35842/0 (socket says 172.21.15.150:35842) 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c42d9d640 1 -- 172.21.15.150:0/2344424085 learned_addr learned my addr 172.21.15.150:0/2344424085 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c4359e640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c441540b0 0x7f6c44147b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c4259c640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c440c8870 0x7f6c441f0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c42d9d640 1 -- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c440c8870 msgr2=0x7f6c441f0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c42d9d640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c440c8870 0x7f6c441f0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c42d9d640 1 -- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c441540b0 msgr2=0x7f6c44147b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c42d9d640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c441540b0 0x7f6c44147b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c42d9d640 1 -- 172.21.15.150:0/2344424085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c44150db0 con 0x7f6c440c7eb0 2024-09-17T04:55:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c4259c640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c440c8870 0x7f6c441f0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:55:10.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c42d9d640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c440c7eb0 0x7f6c440ae440 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00c780 tx=0x7f6c3c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:10.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c2b7fe640 1 -- 172.21.15.150:0/2344424085 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c017070 con 0x7f6c440c7eb0 2024-09-17T04:55:10.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c2b7fe640 1 -- 172.21.15.150:0/2344424085 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c3c005070 con 0x7f6c440c7eb0 2024-09-17T04:55:10.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c49113640 1 -- 172.21.15.150:0/2344424085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c441ee290 con 0x7f6c440c7eb0 2024-09-17T04:55:10.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c2b7fe640 1 -- 172.21.15.150:0/2344424085 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c005480 con 0x7f6c440c7eb0 2024-09-17T04:55:10.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c49113640 1 -- 172.21.15.150:0/2344424085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c440aedf0 con 0x7f6c440c7eb0 2024-09-17T04:55:10.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c49113640 1 -- 172.21.15.150:0/2344424085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c441567e0 con 0x7f6c440c7eb0 2024-09-17T04:55:10.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c2b7fe640 1 -- 172.21.15.150:0/2344424085 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6c3c01a070 con 0x7f6c440c7eb0 2024-09-17T04:55:10.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c2b7fe640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6c30073200 0x7f6c30075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:10.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c2b7fe640 1 -- 172.21.15.150:0/2344424085 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7096+0+0 (secure 0 0 0) 0x7f6c3c093dc0 con 0x7f6c440c7eb0 2024-09-17T04:55:10.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.639+0000 7f6c4259c640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6c30073200 0x7f6c30075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:10.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.643+0000 7f6c4259c640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6c30073200 0x7f6c30075710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6c380af000 tx=0x7f6c3801b020 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:10.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.647+0000 7f6c2b7fe640 1 -- 172.21.15.150:0/2344424085 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6c3c05e990 con 0x7f6c440c7eb0 2024-09-17T04:55:10.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.855+0000 7f6c49113640 1 -- 172.21.15.150:0/2344424085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c441ede80 con 0x7f6c440c7eb0 2024-09-17T04:55:10.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.855+0000 7f6c2b7fe640 1 -- 172.21.15.150:0/2344424085 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6541 (secure 0 0 0) 0x7f6c3c05e290 con 0x7f6c440c7eb0 2024-09-17T04:55:10.859 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:10.859 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":41,"btime":"2024-09-17T04:55:03:185493+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:53:53.928936+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5259},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5259":{"gid":5259,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3532647866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3532647866},{"type":"v1","addr":"172.21.15.150:6833","nonce":3532647866}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5259,"qdb_cluster":[5259]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:10.859 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 41 2024-09-17T04:55:10.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.859+0000 7f6c49113640 1 -- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6c30073200 msgr2=0x7f6c30075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:10.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.859+0000 7f6c49113640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6c30073200 0x7f6c30075710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6c380af000 tx=0x7f6c3801b020 comp rx=0 tx=0).stop 2024-09-17T04:55:10.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.859+0000 7f6c49113640 1 -- 172.21.15.150:0/2344424085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c440c7eb0 msgr2=0x7f6c440ae440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:10.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.859+0000 7f6c49113640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c440c7eb0 0x7f6c440ae440 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00c780 tx=0x7f6c3c00cc40 comp rx=0 tx=0).stop 2024-09-17T04:55:10.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.859+0000 7f6c49113640 1 -- 172.21.15.150:0/2344424085 shutdown_connections 2024-09-17T04:55:10.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.859+0000 7f6c49113640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6c30073200 0x7f6c30075710 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:10.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.859+0000 7f6c49113640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c441540b0 0x7f6c44147b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:10.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.859+0000 7f6c49113640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c440c8870 0x7f6c441f0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:10.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.859+0000 7f6c49113640 1 --2- 172.21.15.150:0/2344424085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c440c7eb0 0x7f6c440ae440 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:10.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.859+0000 7f6c49113640 1 -- 172.21.15.150:0/2344424085 >> 172.21.15.150:0/2344424085 conn(0x7f6c440aab60 msgr2=0x7f6c440a9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:10.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.859+0000 7f6c49113640 1 -- 172.21.15.150:0/2344424085 shutdown_connections 2024-09-17T04:55:10.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:10.859+0000 7f6c49113640 1 -- 172.21.15.150:0/2344424085 wait complete. 2024-09-17T04:55:10.887 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:55:12.889 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:13.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.059+0000 7f3113641640 1 -- 172.21.15.150:0/1132962274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f310c0c3900 msgr2=0x7f310c074920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:13.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.059+0000 7f3113641640 1 --2- 172.21.15.150:0/1132962274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f310c0c3900 0x7f310c074920 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009900 tx=0x7f30fc01c8a0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.059+0000 7f3113641640 1 -- 172.21.15.150:0/1132962274 shutdown_connections 2024-09-17T04:55:13.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.059+0000 7f3113641640 1 --2- 172.21.15.150:0/1132962274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f310c0c3900 0x7f310c074920 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.059+0000 7f3113641640 1 --2- 172.21.15.150:0/1132962274 >> v2:172.21.15.176:3300/0 conn(0x7f310c0c4250 0x7f310c153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.059+0000 7f3113641640 1 -- 172.21.15.150:0/1132962274 >> 172.21.15.150:0/1132962274 conn(0x7f310c06e360 msgr2=0x7f310c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:13.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.059+0000 7f3113641640 1 -- 172.21.15.150:0/1132962274 shutdown_connections 2024-09-17T04:55:13.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.059+0000 7f3113641640 1 -- 172.21.15.150:0/1132962274 wait complete. 2024-09-17T04:55:13.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3113641640 1 Processor -- start 2024-09-17T04:55:13.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3113641640 1 -- start start 2024-09-17T04:55:13.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3113641640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f310c074e60 0x7f310c152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:13.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3113641640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f310c0c3900 0x7f310c1f2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:13.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3113641640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f310c0c4250 0x7f310c14cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:13.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3113641640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f310c072f30 con 0x7f310c074e60 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3113641640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f310c072db0 con 0x7f310c0c4250 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3113641640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f310c160f90 con 0x7f310c0c3900 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f31113dd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f310c074e60 0x7f310c152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f31113dd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f310c074e60 0x7f310c152790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35866/0 (socket says 172.21.15.150:35866) 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f31113dd640 1 -- 172.21.15.150:0/3846895398 learned_addr learned my addr 172.21.15.150:0/3846895398 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3111bde640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f310c0c4250 0x7f310c14cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3110bdc640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f310c0c3900 0x7f310c1f2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f31113dd640 1 -- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f310c0c3900 msgr2=0x7f310c1f2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f31113dd640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f310c0c3900 0x7f310c1f2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f31113dd640 1 -- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f310c0c4250 msgr2=0x7f310c14cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f31113dd640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f310c0c4250 0x7f310c14cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f31113dd640 1 -- 172.21.15.150:0/3846895398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310c156b10 con 0x7f310c074e60 2024-09-17T04:55:13.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f31113dd640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f310c074e60 0x7f310c152790 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f310800d950 tx=0x7f310800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:13.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3101ffb640 1 -- 172.21.15.150:0/3846895398 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108017070 con 0x7f310c074e60 2024-09-17T04:55:13.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3101ffb640 1 -- 172.21.15.150:0/3846895398 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3108009d20 con 0x7f310c074e60 2024-09-17T04:55:13.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3113641640 1 -- 172.21.15.150:0/3846895398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f310c156610 con 0x7f310c074e60 2024-09-17T04:55:13.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3101ffb640 1 -- 172.21.15.150:0/3846895398 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31080106c0 con 0x7f310c074e60 2024-09-17T04:55:13.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.063+0000 7f3113641640 1 -- 172.21.15.150:0/3846895398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f310c1f07c0 con 0x7f310c074e60 2024-09-17T04:55:13.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.067+0000 7f3113641640 1 -- 172.21.15.150:0/3846895398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f310c074920 con 0x7f310c074e60 2024-09-17T04:55:13.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.067+0000 7f3101ffb640 1 -- 172.21.15.150:0/3846895398 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f310801a070 con 0x7f310c074e60 2024-09-17T04:55:13.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.067+0000 7f3101ffb640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f30f0073230 0x7f30f0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:13.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.067+0000 7f3101ffb640 1 -- 172.21.15.150:0/3846895398 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f3108094dc0 con 0x7f310c074e60 2024-09-17T04:55:13.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.067+0000 7f3110bdc640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f30f0073230 0x7f30f0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:13.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.067+0000 7f3110bdc640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f30f0073230 0x7f30f0075740 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f30fc0af000 tx=0x7f30fc01b0e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.071+0000 7f3101ffb640 1 -- 172.21.15.150:0/3846895398 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f310805f820 con 0x7f310c074e60 2024-09-17T04:55:13.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.279+0000 7f3113641640 1 -- 172.21.15.150:0/3846895398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f310c153f60 con 0x7f310c074e60 2024-09-17T04:55:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3101ffb640 1 -- 172.21.15.150:0/3846895398 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5705 (secure 0 0 0) 0x7f310805f120 con 0x7f310c074e60 2024-09-17T04:55:13.283 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:13.283 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":44,"btime":"2024-09-17T04:55:12:925815+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:11.931211+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:rejoin","state_seq":5,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:13.284 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 44 2024-09-17T04:55:13.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3113641640 1 -- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f30f0073230 msgr2=0x7f30f0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:13.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3113641640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f30f0073230 0x7f30f0075740 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f30fc0af000 tx=0x7f30fc01b0e0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3113641640 1 -- 172.21.15.150:0/3846895398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f310c074e60 msgr2=0x7f310c152790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3113641640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f310c074e60 0x7f310c152790 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f310800d950 tx=0x7f310800de10 comp rx=0 tx=0).stop 2024-09-17T04:55:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3113641640 1 -- 172.21.15.150:0/3846895398 shutdown_connections 2024-09-17T04:55:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3113641640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f30f0073230 0x7f30f0075740 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3113641640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f310c0c4250 0x7f310c14cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3113641640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f310c0c3900 0x7f310c1f2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3113641640 1 --2- 172.21.15.150:0/3846895398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f310c074e60 0x7f310c152790 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3113641640 1 -- 172.21.15.150:0/3846895398 >> 172.21.15.150:0/3846895398 conn(0x7f310c06e360 msgr2=0x7f310c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3113641640 1 -- 172.21.15.150:0/3846895398 shutdown_connections 2024-09-17T04:55:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.283+0000 7f3113641640 1 -- 172.21.15.150:0/3846895398 wait complete. 2024-09-17T04:55:13.307 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-17T04:55:13.307 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T04:55:13.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:13.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.459+0000 7f256ded1640 1 -- 172.21.15.150:0/1088870436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25680717a0 msgr2=0x7f2568074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:13.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.459+0000 7f256ded1640 1 --2- 172.21.15.150:0/1088870436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25680717a0 0x7f2568074450 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2560009c80 tx=0x7f256001cd10 comp rx=0 tx=0).stop 2024-09-17T04:55:13.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.459+0000 7f256ded1640 1 -- 172.21.15.150:0/1088870436 shutdown_connections 2024-09-17T04:55:13.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.459+0000 7f256ded1640 1 --2- 172.21.15.150:0/1088870436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25680717a0 0x7f2568074450 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.459+0000 7f256ded1640 1 --2- 172.21.15.150:0/1088870436 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2568070e50 0x7f2568071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.459+0000 7f256ded1640 1 -- 172.21.15.150:0/1088870436 >> 172.21.15.150:0/1088870436 conn(0x7f25680c4a00 msgr2=0x7f25680c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:13.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.459+0000 7f256ded1640 1 -- 172.21.15.150:0/1088870436 shutdown_connections 2024-09-17T04:55:13.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.459+0000 7f256ded1640 1 -- 172.21.15.150:0/1088870436 wait complete. 2024-09-17T04:55:13.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.459+0000 7f256ded1640 1 Processor -- start 2024-09-17T04:55:13.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.459+0000 7f256ded1640 1 -- start start 2024-09-17T04:55:13.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f256ded1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2568070e50 0x7f25680c6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:13.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f256ded1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25680717a0 0x7f2568146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:13.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f256ded1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2568073440 0x7f2568146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:13.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f256ded1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f25680ad210 con 0x7f2568070e50 2024-09-17T04:55:13.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f256ded1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f25680ad090 con 0x7f2568073440 2024-09-17T04:55:13.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f256ded1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f25680ad390 con 0x7f25680717a0 2024-09-17T04:55:13.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f25677fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2568070e50 0x7f25680c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:13.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f25677fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2568070e50 0x7f25680c6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35890/0 (socket says 172.21.15.150:35890) 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f25677fe640 1 -- 172.21.15.150:0/1813334549 learned_addr learned my addr 172.21.15.150:0/1813334549 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f2567fff640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2568073440 0x7f2568146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f255ffff640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25680717a0 0x7f2568146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f25677fe640 1 -- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25680717a0 msgr2=0x7f2568146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f25677fe640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25680717a0 0x7f2568146740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f25677fe640 1 -- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2568073440 msgr2=0x7f2568146c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f25677fe640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2568073440 0x7f2568146c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f25677fe640 1 -- 172.21.15.150:0/1813334549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256814e1d0 con 0x7f2568070e50 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f25677fe640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2568070e50 0x7f25680c6690 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f255800b3e0 tx=0x7f255800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f2564ff9640 1 -- 172.21.15.150:0/1813334549 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558015020 con 0x7f2568070e50 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f2564ff9640 1 -- 172.21.15.150:0/1813334549 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2558011920 con 0x7f2568070e50 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f256ded1640 1 -- 172.21.15.150:0/1813334549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2568150e80 con 0x7f2568070e50 2024-09-17T04:55:13.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f2564ff9640 1 -- 172.21.15.150:0/1813334549 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558011db0 con 0x7f2568070e50 2024-09-17T04:55:13.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f256ded1640 1 -- 172.21.15.150:0/1813334549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25680c6bd0 con 0x7f2568070e50 2024-09-17T04:55:13.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f256ded1640 1 -- 172.21.15.150:0/1813334549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25680ad8a0 con 0x7f2568070e50 2024-09-17T04:55:13.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.463+0000 7f2564ff9640 1 -- 172.21.15.150:0/1813334549 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2558018020 con 0x7f2568070e50 2024-09-17T04:55:13.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.467+0000 7f2564ff9640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2548073200 0x7f2548075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:13.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.467+0000 7f2564ff9640 1 -- 172.21.15.150:0/1813334549 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f25580947c0 con 0x7f2568070e50 2024-09-17T04:55:13.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.467+0000 7f255ffff640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2548073200 0x7f2548075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:13.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.467+0000 7f255ffff640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2548073200 0x7f2548075710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f25500058d0 tx=0x7f2550005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:13.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.471+0000 7f2564ff9640 1 -- 172.21.15.150:0/1813334549 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f255801f030 con 0x7f2568070e50 2024-09-17T04:55:13.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.671+0000 7f256ded1640 1 -- 172.21.15.150:0/1813334549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25680c60f0 con 0x7f2568070e50 2024-09-17T04:55:13.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.671+0000 7f2564ff9640 1 -- 172.21.15.150:0/1813334549 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5705 (secure 0 0 0) 0x7f255805f390 con 0x7f2568070e50 2024-09-17T04:55:13.673 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:13.673 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":44,"btime":"2024-09-17T04:55:12:925815+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:11.931211+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:rejoin","state_seq":5,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:13.673 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 44 2024-09-17T04:55:13.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.671+0000 7f256ded1640 1 -- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2548073200 msgr2=0x7f2548075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:13.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.671+0000 7f256ded1640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2548073200 0x7f2548075710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f25500058d0 tx=0x7f2550005860 comp rx=0 tx=0).stop 2024-09-17T04:55:13.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.671+0000 7f256ded1640 1 -- 172.21.15.150:0/1813334549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2568070e50 msgr2=0x7f25680c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:13.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.671+0000 7f256ded1640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2568070e50 0x7f25680c6690 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f255800b3e0 tx=0x7f255800b8a0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.675+0000 7f256ded1640 1 -- 172.21.15.150:0/1813334549 shutdown_connections 2024-09-17T04:55:13.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.675+0000 7f256ded1640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2548073200 0x7f2548075710 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.675+0000 7f256ded1640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2568073440 0x7f2568146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.675+0000 7f256ded1640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f25680717a0 0x7f2568146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.675+0000 7f256ded1640 1 --2- 172.21.15.150:0/1813334549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2568070e50 0x7f25680c6690 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:13.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.675+0000 7f256ded1640 1 -- 172.21.15.150:0/1813334549 >> 172.21.15.150:0/1813334549 conn(0x7f25680c4a00 msgr2=0x7f25680c35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:13.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.675+0000 7f256ded1640 1 -- 172.21.15.150:0/1813334549 shutdown_connections 2024-09-17T04:55:13.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:13.675+0000 7f256ded1640 1 -- 172.21.15.150:0/1813334549 wait complete. 2024-09-17T04:55:13.695 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T04:55:12:925815+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 44, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 44, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5466": { "addr": "172.21.15.150:6833/4286123257", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 4286123257, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 4286123257, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5466, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:55:11.931211+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5466 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 40, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5107": { "addr": "172.21.15.176:6837/2374314256", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 2374314256, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 2374314256, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5107, "incarnation": 37, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:54:58.881332+0000", "qdb_cluster": [ 5107 ], "qdb_leader": 5107, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5107 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:55:15.696 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:15.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.867+0000 7f0499dd9640 1 -- 172.21.15.150:0/226715309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04940c5df0 msgr2=0x7f0494157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:15.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.867+0000 7f0499dd9640 1 --2- 172.21.15.150:0/226715309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04940c5df0 0x7f0494157330 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f04940c3790 tx=0x7f048401c650 comp rx=0 tx=0).stop 2024-09-17T04:55:15.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.867+0000 7f0499dd9640 1 -- 172.21.15.150:0/226715309 shutdown_connections 2024-09-17T04:55:15.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.867+0000 7f0499dd9640 1 --2- 172.21.15.150:0/226715309 >> v2:172.21.15.176:3300/0 conn(0x7f04940c6340 0x7f04940c1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:15.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.867+0000 7f0499dd9640 1 --2- 172.21.15.150:0/226715309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04940c5df0 0x7f0494157330 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:15.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.867+0000 7f0499dd9640 1 -- 172.21.15.150:0/226715309 >> 172.21.15.150:0/226715309 conn(0x7f0494074750 msgr2=0x7f0494074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:15.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.871+0000 7f0499dd9640 1 -- 172.21.15.150:0/226715309 shutdown_connections 2024-09-17T04:55:15.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.871+0000 7f0499dd9640 1 -- 172.21.15.150:0/226715309 wait complete. 2024-09-17T04:55:15.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.871+0000 7f0499dd9640 1 Processor -- start 2024-09-17T04:55:15.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.871+0000 7f0499dd9640 1 -- start start 2024-09-17T04:55:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0499dd9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04940c4b20 0x7f0494157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0499dd9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04940c5df0 0x7f0494159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0499dd9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04940c6340 0x7f049414ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0499dd9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0494163980 con 0x7f04940c6340 2024-09-17T04:55:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0499dd9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f04940c1950 con 0x7f04940c4b20 2024-09-17T04:55:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0499dd9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0494163b00 con 0x7f04940c5df0 2024-09-17T04:55:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0493fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04940c6340 0x7f049414ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:15.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0493fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04940c6340 0x7f049414ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35902/0 (socket says 172.21.15.150:35902) 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0493fff640 1 -- 172.21.15.150:0/650781225 learned_addr learned my addr 172.21.15.150:0/650781225 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f04937fe640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04940c4b20 0x7f0494157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f048affd640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04940c5df0 0x7f0494159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0493fff640 1 -- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04940c5df0 msgr2=0x7f0494159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0493fff640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04940c5df0 0x7f0494159b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0493fff640 1 -- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04940c4b20 msgr2=0x7f0494157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0493fff640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04940c4b20 0x7f0494157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0493fff640 1 -- 172.21.15.150:0/650781225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049414b590 con 0x7f04940c6340 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0493fff640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04940c6340 0x7f049414ba80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f048c00ba10 tx=0x7f048c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0490ff9640 1 -- 172.21.15.150:0/650781225 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c019070 con 0x7f04940c6340 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0499dd9640 1 -- 172.21.15.150:0/650781225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04941f0a60 con 0x7f04940c6340 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0490ff9640 1 -- 172.21.15.150:0/650781225 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f048c00cc30 con 0x7f04940c6340 2024-09-17T04:55:15.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0490ff9640 1 -- 172.21.15.150:0/650781225 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c0146a0 con 0x7f04940c6340 2024-09-17T04:55:15.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0499dd9640 1 -- 172.21.15.150:0/650781225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f049414c940 con 0x7f04940c6340 2024-09-17T04:55:15.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0499dd9640 1 -- 172.21.15.150:0/650781225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0494157330 con 0x7f04940c6340 2024-09-17T04:55:15.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0490ff9640 1 -- 172.21.15.150:0/650781225 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f048c01c070 con 0x7f04940c6340 2024-09-17T04:55:15.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0490ff9640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0474073200 0x7f0474075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:15.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.875+0000 7f0490ff9640 1 -- 172.21.15.150:0/650781225 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f048c094440 con 0x7f04940c6340 2024-09-17T04:55:15.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.879+0000 7f04937fe640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0474073200 0x7f0474075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:15.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.879+0000 7f04937fe640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0474073200 0x7f0474075710 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f04840af000 tx=0x7f048401ae90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:15.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:15.883+0000 7f0490ff9640 1 -- 172.21.15.150:0/650781225 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f048c00ce20 con 0x7f04940c6340 2024-09-17T04:55:16.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.147+0000 7f0499dd9640 1 -- 172.21.15.150:0/650781225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04940b1b70 con 0x7f04940c6340 2024-09-17T04:55:16.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.147+0000 7f0490ff9640 1 -- 172.21.15.150:0/650781225 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5712 (secure 0 0 0) 0x7f048c05eeb0 con 0x7f04940c6340 2024-09-17T04:55:16.148 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:16.148 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":45,"btime":"2024-09-17T04:55:13:937031+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:13.937018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5466,"qdb_cluster":[5466]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:16.148 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 45 2024-09-17T04:55:16.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.151+0000 7f0499dd9640 1 -- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0474073200 msgr2=0x7f0474075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:16.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.151+0000 7f0499dd9640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0474073200 0x7f0474075710 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f04840af000 tx=0x7f048401ae90 comp rx=0 tx=0).stop 2024-09-17T04:55:16.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.151+0000 7f0499dd9640 1 -- 172.21.15.150:0/650781225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04940c6340 msgr2=0x7f049414ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:16.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.151+0000 7f0499dd9640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04940c6340 0x7f049414ba80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f048c00ba10 tx=0x7f048c00bed0 comp rx=0 tx=0).stop 2024-09-17T04:55:16.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.151+0000 7f0499dd9640 1 -- 172.21.15.150:0/650781225 shutdown_connections 2024-09-17T04:55:16.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.151+0000 7f0499dd9640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0474073200 0x7f0474075710 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:16.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.151+0000 7f0499dd9640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04940c6340 0x7f049414ba80 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:16.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.151+0000 7f0499dd9640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04940c5df0 0x7f0494159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:16.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.151+0000 7f0499dd9640 1 --2- 172.21.15.150:0/650781225 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04940c4b20 0x7f0494157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:16.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.151+0000 7f0499dd9640 1 -- 172.21.15.150:0/650781225 >> 172.21.15.150:0/650781225 conn(0x7f0494074750 msgr2=0x7f04941516c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:16.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.151+0000 7f0499dd9640 1 -- 172.21.15.150:0/650781225 shutdown_connections 2024-09-17T04:55:16.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:16.151+0000 7f0499dd9640 1 -- 172.21.15.150:0/650781225 wait complete. 2024-09-17T04:55:16.174 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=5259 2024-09-17T04:55:16.174 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.c 2024-09-17T04:55:18.175 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-17T04:55:18.176 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:55:18.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:55:18.178 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:55:18.179 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:18.199 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T04:55:18.199+0000 7fc449d00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:55:18.290 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T04:55:18.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.363+0000 7f87e110e640 1 -- 172.21.15.150:0/1090171308 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87dc0c7190 msgr2=0x7f87dc0c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:18.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.363+0000 7f87e110e640 1 --2- 172.21.15.150:0/1090171308 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87dc0c7190 0x7f87dc0c75a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009870 tx=0x7f87d401c860 comp rx=0 tx=0).stop 2024-09-17T04:55:18.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.363+0000 7f87e110e640 1 -- 172.21.15.150:0/1090171308 shutdown_connections 2024-09-17T04:55:18.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.363+0000 7f87e110e640 1 --2- 172.21.15.150:0/1090171308 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87dc0c7190 0x7f87dc0c75a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:18.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.363+0000 7f87e110e640 1 -- 172.21.15.150:0/1090171308 >> 172.21.15.150:0/1090171308 conn(0x7f87dc0aab60 msgr2=0x7f87dc0aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:18.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.363+0000 7f87e110e640 1 -- 172.21.15.150:0/1090171308 shutdown_connections 2024-09-17T04:55:18.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.363+0000 7f87e110e640 1 -- 172.21.15.150:0/1090171308 wait complete. 2024-09-17T04:55:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87e110e640 1 Processor -- start 2024-09-17T04:55:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87e110e640 1 -- start start 2024-09-17T04:55:18.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87e110e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0c7190 0x7f87dc14c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:18.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87e110e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87dc0c7eb0 0x7f87dc14ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:18.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87e110e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87dc1540b0 0x7f87dc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:18.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87e110e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc0c8bc0 con 0x7f87dc0c7190 2024-09-17T04:55:18.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87e110e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc0c8a40 con 0x7f87dc1540b0 2024-09-17T04:55:18.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87e110e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f87dc0c7ae0 con 0x7f87dc0c7eb0 2024-09-17T04:55:18.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87dad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0c7190 0x7f87dc14c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87dad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0c7190 0x7f87dc14c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35910/0 (socket says 172.21.15.150:35910) 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87dad9d640 1 -- 172.21.15.150:0/204098945 learned_addr learned my addr 172.21.15.150:0/204098945 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87d3fff640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87dc0c7eb0 0x7f87dc14ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87db59e640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87dc1540b0 0x7f87dc06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87dad9d640 1 -- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87dc0c7eb0 msgr2=0x7f87dc14ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87dad9d640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87dc0c7eb0 0x7f87dc14ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87dad9d640 1 -- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87dc1540b0 msgr2=0x7f87dc06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87dad9d640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87dc1540b0 0x7f87dc06d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87dad9d640 1 -- 172.21.15.150:0/204098945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87dc1ed6b0 con 0x7f87dc0c7190 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87dad9d640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0c7190 0x7f87dc14c9c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00cb50 tx=0x7f87cc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87d37fe640 1 -- 172.21.15.150:0/204098945 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc016020 con 0x7f87dc0c7190 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87e110e640 1 -- 172.21.15.150:0/204098945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87dc1ed990 con 0x7f87dc0c7190 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87d37fe640 1 -- 172.21.15.150:0/204098945 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87cc00ab00 con 0x7f87dc0c7190 2024-09-17T04:55:18.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87d37fe640 1 -- 172.21.15.150:0/204098945 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc012420 con 0x7f87dc0c7190 2024-09-17T04:55:18.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.367+0000 7f87e110e640 1 -- 172.21.15.150:0/204098945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87dc1edf50 con 0x7f87dc0c7190 2024-09-17T04:55:18.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.371+0000 7f87e110e640 1 -- 172.21.15.150:0/204098945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87dc0c75a0 con 0x7f87dc0c7190 2024-09-17T04:55:18.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.371+0000 7f87d37fe640 1 -- 172.21.15.150:0/204098945 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f87cc019020 con 0x7f87dc0c7190 2024-09-17T04:55:18.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.371+0000 7f87d37fe640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f87bc073200 0x7f87bc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:18.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.371+0000 7f87d37fe640 1 -- 172.21.15.150:0/204098945 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f87cc094eb0 con 0x7f87dc0c7190 2024-09-17T04:55:18.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.371+0000 7f87d3fff640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f87bc073200 0x7f87bc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:18.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.371+0000 7f87d3fff640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f87bc073200 0x7f87bc075710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f87c4000c10 tx=0x7f87c4009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:18.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.375+0000 7f87d37fe640 1 -- 172.21.15.150:0/204098945 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f87cc0c3db0 con 0x7f87dc0c7190 2024-09-17T04:55:18.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.615+0000 7f87e110e640 1 -- 172.21.15.150:0/204098945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87dc157440 con 0x7f87dc0c7190 2024-09-17T04:55:18.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.615+0000 7f87d37fe640 1 -- 172.21.15.150:0/204098945 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5712 (secure 0 0 0) 0x7f87cc020020 con 0x7f87dc0c7190 2024-09-17T04:55:18.617 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:18.618 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":45,"btime":"2024-09-17T04:55:13:937031+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:13.937018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5466,"qdb_cluster":[5466]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:18.618 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 45 2024-09-17T04:55:18.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.619+0000 7f87e110e640 1 -- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f87bc073200 msgr2=0x7f87bc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:18.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.619+0000 7f87e110e640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f87bc073200 0x7f87bc075710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f87c4000c10 tx=0x7f87c4009040 comp rx=0 tx=0).stop 2024-09-17T04:55:18.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.619+0000 7f87e110e640 1 -- 172.21.15.150:0/204098945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0c7190 msgr2=0x7f87dc14c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:18.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.619+0000 7f87e110e640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0c7190 0x7f87dc14c9c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00cb50 tx=0x7f87cc00cc40 comp rx=0 tx=0).stop 2024-09-17T04:55:18.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.619+0000 7f87e110e640 1 -- 172.21.15.150:0/204098945 shutdown_connections 2024-09-17T04:55:18.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.619+0000 7f87e110e640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f87bc073200 0x7f87bc075710 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:18.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.619+0000 7f87e110e640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87dc1540b0 0x7f87dc06d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:18.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.619+0000 7f87e110e640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87dc0c7eb0 0x7f87dc14ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:18.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.619+0000 7f87e110e640 1 --2- 172.21.15.150:0/204098945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0c7190 0x7f87dc14c9c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:18.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.619+0000 7f87e110e640 1 -- 172.21.15.150:0/204098945 >> 172.21.15.150:0/204098945 conn(0x7f87dc0aab60 msgr2=0x7f87dc068f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:18.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.619+0000 7f87e110e640 1 -- 172.21.15.150:0/204098945 shutdown_connections 2024-09-17T04:55:18.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:18.619+0000 7f87e110e640 1 -- 172.21.15.150:0/204098945 wait complete. 2024-09-17T04:55:18.646 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-17T04:55:20.647 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:20.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.831+0000 7f980944d640 1 -- 172.21.15.150:0/65522712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97f0003a60 msgr2=0x7f97f0003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:20.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.831+0000 7f980944d640 1 --2- 172.21.15.150:0/65522712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97f0003a60 0x7f97f0003ed0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f97f400c2e0 tx=0x7f97f401cf20 comp rx=0 tx=0).stop 2024-09-17T04:55:20.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.831+0000 7f980944d640 1 -- 172.21.15.150:0/65522712 shutdown_connections 2024-09-17T04:55:20.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.831+0000 7f980944d640 1 --2- 172.21.15.150:0/65522712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97f0003a60 0x7f97f0003ed0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:20.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.831+0000 7f980944d640 1 -- 172.21.15.150:0/65522712 >> 172.21.15.150:0/65522712 conn(0x7f98040abaa0 msgr2=0x7f98040abeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:20.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.831+0000 7f980944d640 1 -- 172.21.15.150:0/65522712 shutdown_connections 2024-09-17T04:55:20.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.831+0000 7f980944d640 1 -- 172.21.15.150:0/65522712 wait complete. 2024-09-17T04:55:20.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.831+0000 7f980944d640 1 Processor -- start 2024-09-17T04:55:20.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.831+0000 7f980944d640 1 -- start start 2024-09-17T04:55:20.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f980944d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f0003a60 0x7f98040c0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:20.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f980944d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98040aefa0 0x7f98041f12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:20.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f980944d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98040c22d0 0x7f98041386f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:20.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f980944d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f980413d3c0 con 0x7f98040aefa0 2024-09-17T04:55:20.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f980944d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f980413d240 con 0x7f98040c22d0 2024-09-17T04:55:20.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f980944d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f98040c8670 con 0x7f97f0003a60 2024-09-17T04:55:20.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fa7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98040aefa0 0x7f98041f12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:20.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fa7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98040aefa0 0x7f98041f12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57480/0 (socket says 172.21.15.150:57480) 2024-09-17T04:55:20.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fa7fc640 1 -- 172.21.15.150:0/717300566 learned_addr learned my addr 172.21.15.150:0/717300566 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:20.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f9802ffd640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f0003a60 0x7f98040c0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:20.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f98037fe640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98040c22d0 0x7f98041386f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:20.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fa7fc640 1 -- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f0003a60 msgr2=0x7f98040c0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:20.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fa7fc640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f0003a60 0x7f98040c0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:20.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fa7fc640 1 -- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98040c22d0 msgr2=0x7f98041386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:20.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fa7fc640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98040c22d0 0x7f98041386f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:20.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fa7fc640 1 -- 172.21.15.150:0/717300566 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98041f1dd0 con 0x7f98040aefa0 2024-09-17T04:55:20.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fa7fc640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98040aefa0 0x7f98041f12f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f97ec011780 tx=0x7f97ec011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:20.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fbfff640 1 -- 172.21.15.150:0/717300566 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec01f070 con 0x7f98040aefa0 2024-09-17T04:55:20.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fbfff640 1 -- 172.21.15.150:0/717300566 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97ec011e90 con 0x7f98040aefa0 2024-09-17T04:55:20.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f980944d640 1 -- 172.21.15.150:0/717300566 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98040c34d0 con 0x7f98040aefa0 2024-09-17T04:55:20.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fbfff640 1 -- 172.21.15.150:0/717300566 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec0126b0 con 0x7f98040aefa0 2024-09-17T04:55:20.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f980944d640 1 -- 172.21.15.150:0/717300566 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98040bef60 con 0x7f98040aefa0 2024-09-17T04:55:20.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f980944d640 1 -- 172.21.15.150:0/717300566 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9804154640 con 0x7f98040aefa0 2024-09-17T04:55:20.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.835+0000 7f97fbfff640 1 -- 172.21.15.150:0/717300566 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f97ec022070 con 0x7f98040aefa0 2024-09-17T04:55:20.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.839+0000 7f97fbfff640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f97e4073230 0x7f97e4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:20.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.839+0000 7f97fbfff640 1 -- 172.21.15.150:0/717300566 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f97ec094ff0 con 0x7f98040aefa0 2024-09-17T04:55:20.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.839+0000 7f9802ffd640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f97e4073230 0x7f97e4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:20.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.839+0000 7f9802ffd640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f97e4073230 0x7f97e4075740 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f97f40ab000 tx=0x7f97f4019d60 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:20.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:20.843+0000 7f97fbfff640 1 -- 172.21.15.150:0/717300566 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f97ec05f520 con 0x7f98040aefa0 2024-09-17T04:55:21.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.107+0000 7f980944d640 1 -- 172.21.15.150:0/717300566 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98040abf50 con 0x7f98040aefa0 2024-09-17T04:55:21.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.107+0000 7f97fbfff640 1 -- 172.21.15.150:0/717300566 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6540 (secure 0 0 0) 0x7f97ec063f50 con 0x7f98040aefa0 2024-09-17T04:55:21.111 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:21.111 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":46,"btime":"2024-09-17T04:55:18:968814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:13.937018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5466,"qdb_cluster":[5466]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:21.111 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 46 2024-09-17T04:55:21.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.111+0000 7f980944d640 1 -- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f97e4073230 msgr2=0x7f97e4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:21.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.111+0000 7f980944d640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f97e4073230 0x7f97e4075740 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f97f40ab000 tx=0x7f97f4019d60 comp rx=0 tx=0).stop 2024-09-17T04:55:21.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.111+0000 7f980944d640 1 -- 172.21.15.150:0/717300566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98040aefa0 msgr2=0x7f98041f12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:21.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.111+0000 7f980944d640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98040aefa0 0x7f98041f12f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f97ec011780 tx=0x7f97ec011c40 comp rx=0 tx=0).stop 2024-09-17T04:55:21.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.111+0000 7f980944d640 1 -- 172.21.15.150:0/717300566 shutdown_connections 2024-09-17T04:55:21.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.111+0000 7f980944d640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f97e4073230 0x7f97e4075740 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:21.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.111+0000 7f980944d640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f98040c22d0 0x7f98041386f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:21.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.111+0000 7f980944d640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98040aefa0 0x7f98041f12f0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:21.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.111+0000 7f980944d640 1 --2- 172.21.15.150:0/717300566 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97f0003a60 0x7f98040c0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:21.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.111+0000 7f980944d640 1 -- 172.21.15.150:0/717300566 >> 172.21.15.150:0/717300566 conn(0x7f98040abaa0 msgr2=0x7f98040ab1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:21.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.111+0000 7f980944d640 1 -- 172.21.15.150:0/717300566 shutdown_connections 2024-09-17T04:55:21.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:21.111+0000 7f980944d640 1 -- 172.21.15.150:0/717300566 wait complete. 2024-09-17T04:55:21.135 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-17T04:55:21.135 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before thrashing 2024-09-17T04:55:22.136 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:22.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.303+0000 7f9eea08c640 1 -- 172.21.15.150:0/4036677216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee4157c00 msgr2=0x7f9ee4158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:22.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.303+0000 7f9eea08c640 1 --2- 172.21.15.150:0/4036677216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee4157c00 0x7f9ee4158010 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9edc009bc0 tx=0x7f9edc01cba0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.303+0000 7f9eea08c640 1 -- 172.21.15.150:0/4036677216 shutdown_connections 2024-09-17T04:55:22.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.303+0000 7f9eea08c640 1 --2- 172.21.15.150:0/4036677216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee4157c00 0x7f9ee4158010 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.303+0000 7f9eea08c640 1 -- 172.21.15.150:0/4036677216 >> 172.21.15.150:0/4036677216 conn(0x7f9ee40ca460 msgr2=0x7f9ee40c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:22.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.303+0000 7f9eea08c640 1 -- 172.21.15.150:0/4036677216 shutdown_connections 2024-09-17T04:55:22.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.303+0000 7f9eea08c640 1 -- 172.21.15.150:0/4036677216 wait complete. 2024-09-17T04:55:22.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.303+0000 7f9eea08c640 1 Processor -- start 2024-09-17T04:55:22.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.303+0000 7f9eea08c640 1 -- start start 2024-09-17T04:55:22.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9eea08c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ee4071670 0x7f9ee4154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:22.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9eea08c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ee40c5680 0x7f9ee4152e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:22.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9eea08c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee4157c00 0x7f9ee41405d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:22.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9eea08c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee4072300 con 0x7f9ee4157c00 2024-09-17T04:55:22.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9eea08c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee4072180 con 0x7f9ee40c5680 2024-09-17T04:55:22.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9eea08c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee41627c0 con 0x7f9ee4071670 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee4157c00 0x7f9ee41405d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee4157c00 0x7f9ee41405d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57498/0 (socket says 172.21.15.150:57498) 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ee3fff640 1 -- 172.21.15.150:0/2418525664 learned_addr learned my addr 172.21.15.150:0/2418525664 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ee37fe640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ee4071670 0x7f9ee4154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ee2ffd640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ee40c5680 0x7f9ee4152e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ee3fff640 1 -- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ee4071670 msgr2=0x7f9ee4154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ee3fff640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ee4071670 0x7f9ee4154dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ee3fff640 1 -- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ee40c5680 msgr2=0x7f9ee4152e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ee3fff640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ee40c5680 0x7f9ee4152e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ee3fff640 1 -- 172.21.15.150:0/2418525664 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee40c41a0 con 0x7f9ee4157c00 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ee37fe640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ee4071670 0x7f9ee4154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ee3fff640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee4157c00 0x7f9ee41405d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9edc01c610 tx=0x7f9edc005a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ecbfff640 1 -- 172.21.15.150:0/2418525664 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc0b2040 con 0x7f9ee4157c00 2024-09-17T04:55:22.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ecbfff640 1 -- 172.21.15.150:0/2418525664 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9edc0bb070 con 0x7f9ee4157c00 2024-09-17T04:55:22.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9eea08c640 1 -- 172.21.15.150:0/2418525664 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee41edad0 con 0x7f9ee4157c00 2024-09-17T04:55:22.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9ecbfff640 1 -- 172.21.15.150:0/2418525664 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc004590 con 0x7f9ee4157c00 2024-09-17T04:55:22.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9eea08c640 1 -- 172.21.15.150:0/2418525664 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee4152900 con 0x7f9ee4157c00 2024-09-17T04:55:22.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.307+0000 7f9eea08c640 1 -- 172.21.15.150:0/2418525664 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee41edd10 con 0x7f9ee4157c00 2024-09-17T04:55:22.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.311+0000 7f9ecbfff640 1 -- 172.21.15.150:0/2418525664 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9edc0c0460 con 0x7f9ee4157c00 2024-09-17T04:55:22.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.311+0000 7f9ecbfff640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ed0073200 0x7f9ed0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:22.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.311+0000 7f9ecbfff640 1 -- 172.21.15.150:0/2418525664 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f9edc12fb40 con 0x7f9ee4157c00 2024-09-17T04:55:22.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.311+0000 7f9ee37fe640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ed0073200 0x7f9ed0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:22.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.311+0000 7f9ee37fe640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ed0073200 0x7f9ed0075710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4001230 tx=0x7f9ed4009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:22.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.315+0000 7f9ecbfff640 1 -- 172.21.15.150:0/2418525664 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9ee41edd10 con 0x7f9ee4157c00 2024-09-17T04:55:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.571+0000 7f9eea08c640 1 -- 172.21.15.150:0/2418525664 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ee4158010 con 0x7f9ee4157c00 2024-09-17T04:55:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.571+0000 7f9ecbfff640 1 -- 172.21.15.150:0/2418525664 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6540 (secure 0 0 0) 0x7f9edc016790 con 0x7f9ee4157c00 2024-09-17T04:55:22.574 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:22.574 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":46,"btime":"2024-09-17T04:55:18:968814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:13.937018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5466,"qdb_cluster":[5466]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 46 2024-09-17T04:55:22.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.575+0000 7f9eea08c640 1 -- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ed0073200 msgr2=0x7f9ed0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:22.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.575+0000 7f9eea08c640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ed0073200 0x7f9ed0075710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4001230 tx=0x7f9ed4009040 comp rx=0 tx=0).stop 2024-09-17T04:55:22.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.575+0000 7f9eea08c640 1 -- 172.21.15.150:0/2418525664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee4157c00 msgr2=0x7f9ee41405d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:22.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.575+0000 7f9eea08c640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee4157c00 0x7f9ee41405d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9edc01c610 tx=0x7f9edc005a60 comp rx=0 tx=0).stop 2024-09-17T04:55:22.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.575+0000 7f9eea08c640 1 -- 172.21.15.150:0/2418525664 shutdown_connections 2024-09-17T04:55:22.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.575+0000 7f9eea08c640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ed0073200 0x7f9ed0075710 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.575+0000 7f9eea08c640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ee40c5680 0x7f9ee4152e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.575+0000 7f9eea08c640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ee4157c00 0x7f9ee41405d0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.575+0000 7f9eea08c640 1 --2- 172.21.15.150:0/2418525664 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ee4071670 0x7f9ee4154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.575+0000 7f9eea08c640 1 -- 172.21.15.150:0/2418525664 >> 172.21.15.150:0/2418525664 conn(0x7f9ee40ca460 msgr2=0x7f9ee40c6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:22.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.575+0000 7f9eea08c640 1 -- 172.21.15.150:0/2418525664 shutdown_connections 2024-09-17T04:55:22.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.575+0000 7f9eea08c640 1 -- 172.21.15.150:0/2418525664 wait complete. 2024-09-17T04:55:22.594 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-17T04:55:22.595 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:55:22.595 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:55:22.608 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:55:22.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:22.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.799+0000 7f5973cf2640 1 -- 172.21.15.150:0/305375461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f596c074e60 msgr2=0x7f596c15df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:22.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.799+0000 7f5973cf2640 1 --2- 172.21.15.150:0/305375461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f596c074e60 0x7f596c15df70 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5964009c80 tx=0x7f596401cda0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.799+0000 7f5973cf2640 1 -- 172.21.15.150:0/305375461 shutdown_connections 2024-09-17T04:55:22.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.799+0000 7f5973cf2640 1 --2- 172.21.15.150:0/305375461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f596c074e60 0x7f596c15df70 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.799+0000 7f5973cf2640 1 --2- 172.21.15.150:0/305375461 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c0c3900 0x7f596c074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.799+0000 7f5973cf2640 1 -- 172.21.15.150:0/305375461 >> 172.21.15.150:0/305375461 conn(0x7f596c06e360 msgr2=0x7f596c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:22.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.799+0000 7f5973cf2640 1 -- 172.21.15.150:0/305375461 shutdown_connections 2024-09-17T04:55:22.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.799+0000 7f5973cf2640 1 -- 172.21.15.150:0/305375461 wait complete. 2024-09-17T04:55:22.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f5973cf2640 1 Processor -- start 2024-09-17T04:55:22.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f5973cf2640 1 -- start start 2024-09-17T04:55:22.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f5973cf2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f596c074e60 0x7f596c0acd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:22.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f5973cf2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f596c0c3900 0x7f596c14ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:22.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f5973cf2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c0c4250 0x7f596c0adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:22.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f5973cf2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f596c160e50 con 0x7f596c0c3900 2024-09-17T04:55:22.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f5973cf2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f596c160cd0 con 0x7f596c074e60 2024-09-17T04:55:22.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f5973cf2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f596c160fd0 con 0x7f596c0c4250 2024-09-17T04:55:22.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f597128d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f596c0c3900 0x7f596c14ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:22.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f597128d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f596c0c3900 0x7f596c14ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57512/0 (socket says 172.21.15.150:57512) 2024-09-17T04:55:22.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f597128d640 1 -- 172.21.15.150:0/189303187 learned_addr learned my addr 172.21.15.150:0/189303187 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:22.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f5971a8e640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f596c074e60 0x7f596c0acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:22.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f597228f640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c0c4250 0x7f596c0adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:22.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f597128d640 1 -- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c0c4250 msgr2=0x7f596c0adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:22.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f597128d640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c0c4250 0x7f596c0adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f597128d640 1 -- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f596c074e60 msgr2=0x7f596c0acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:22.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f597128d640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f596c074e60 0x7f596c0acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:22.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f597128d640 1 -- 172.21.15.150:0/189303187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f596c1504a0 con 0x7f596c0c3900 2024-09-17T04:55:22.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f597228f640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c0c4250 0x7f596c0adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:55:22.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f597128d640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f596c0c3900 0x7f596c14ff60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f595c00d6b0 tx=0x7f595c00db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:22.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f59627fc640 1 -- 172.21.15.150:0/189303187 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c017070 con 0x7f596c0c3900 2024-09-17T04:55:22.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f59627fc640 1 -- 172.21.15.150:0/189303187 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f595c009b30 con 0x7f596c0c3900 2024-09-17T04:55:22.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f5973cf2640 1 -- 172.21.15.150:0/189303187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f596c0ae880 con 0x7f596c0c3900 2024-09-17T04:55:22.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f59627fc640 1 -- 172.21.15.150:0/189303187 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c010460 con 0x7f596c0c3900 2024-09-17T04:55:22.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f5973cf2640 1 -- 172.21.15.150:0/189303187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f596c0ac840 con 0x7f596c0c3900 2024-09-17T04:55:22.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.803+0000 7f5973cf2640 1 -- 172.21.15.150:0/189303187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f596c074920 con 0x7f596c0c3900 2024-09-17T04:55:22.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.807+0000 7f59627fc640 1 -- 172.21.15.150:0/189303187 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f595c01a070 con 0x7f596c0c3900 2024-09-17T04:55:22.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.807+0000 7f59627fc640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5950073230 0x7f5950075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:22.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.807+0000 7f59627fc640 1 -- 172.21.15.150:0/189303187 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f595c094850 con 0x7f596c0c3900 2024-09-17T04:55:22.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.807+0000 7f5971a8e640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5950073230 0x7f5950075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:22.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.807+0000 7f5971a8e640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5950073230 0x7f5950075740 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f596800b810 tx=0x7f5968008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:22.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:22.811+0000 7f59627fc640 1 -- 172.21.15.150:0/189303187 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f595c05f3d0 con 0x7f596c0c3900 2024-09-17T04:55:23.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.071+0000 7f5973cf2640 1 -- 172.21.15.150:0/189303187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f596c0ae010 con 0x7f596c0c3900 2024-09-17T04:55:23.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.071+0000 7f59627fc640 1 -- 172.21.15.150:0/189303187 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6540 (secure 0 0 0) 0x7f595c05ecd0 con 0x7f596c0c3900 2024-09-17T04:55:23.073 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:23.074 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":46,"btime":"2024-09-17T04:55:18:968814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:13.937018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5466,"qdb_cluster":[5466]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:23.074 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 46 2024-09-17T04:55:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.075+0000 7f5973cf2640 1 -- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5950073230 msgr2=0x7f5950075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.075+0000 7f5973cf2640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5950073230 0x7f5950075740 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f596800b810 tx=0x7f5968008040 comp rx=0 tx=0).stop 2024-09-17T04:55:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.075+0000 7f5973cf2640 1 -- 172.21.15.150:0/189303187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f596c0c3900 msgr2=0x7f596c14ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.075+0000 7f5973cf2640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f596c0c3900 0x7f596c14ff60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f595c00d6b0 tx=0x7f595c00db70 comp rx=0 tx=0).stop 2024-09-17T04:55:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.075+0000 7f5973cf2640 1 -- 172.21.15.150:0/189303187 shutdown_connections 2024-09-17T04:55:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.075+0000 7f5973cf2640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5950073230 0x7f5950075740 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.075+0000 7f5973cf2640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f596c0c4250 0x7f596c0adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.075+0000 7f5973cf2640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f596c0c3900 0x7f596c14ff60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.075+0000 7f5973cf2640 1 --2- 172.21.15.150:0/189303187 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f596c074e60 0x7f596c0acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.075+0000 7f5973cf2640 1 -- 172.21.15.150:0/189303187 >> 172.21.15.150:0/189303187 conn(0x7f596c06e360 msgr2=0x7f596c14af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.075+0000 7f5973cf2640 1 -- 172.21.15.150:0/189303187 shutdown_connections 2024-09-17T04:55:23.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:23.075+0000 7f5973cf2640 1 -- 172.21.15.150:0/189303187 wait complete. 2024-09-17T04:55:23.095 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:55:25.097 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:25.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.267+0000 7f9341517640 1 -- 172.21.15.150:0/428900016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f933c1557b0 msgr2=0x7f933c0c4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:25.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.267+0000 7f9341517640 1 --2- 172.21.15.150:0/428900016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f933c1557b0 0x7f933c0c4990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9330009840 tx=0x7f933001c790 comp rx=0 tx=0).stop 2024-09-17T04:55:25.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.267+0000 7f9341517640 1 -- 172.21.15.150:0/428900016 shutdown_connections 2024-09-17T04:55:25.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.267+0000 7f9341517640 1 --2- 172.21.15.150:0/428900016 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f933c1557b0 0x7f933c0c4990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:25.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.267+0000 7f9341517640 1 -- 172.21.15.150:0/428900016 >> 172.21.15.150:0/428900016 conn(0x7f933c0aaa80 msgr2=0x7f933c0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:25.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.267+0000 7f9341517640 1 -- 172.21.15.150:0/428900016 shutdown_connections 2024-09-17T04:55:25.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.267+0000 7f9341517640 1 -- 172.21.15.150:0/428900016 wait complete. 2024-09-17T04:55:25.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.267+0000 7f9341517640 1 Processor -- start 2024-09-17T04:55:25.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.267+0000 7f9341517640 1 -- start start 2024-09-17T04:55:25.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f9341517640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f933c0c7580 0x7f933c14b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:25.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f9341517640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f933c0c7e20 0x7f933c14f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:25.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f9341517640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f933c1557b0 0x7f933c1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:25.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f9341517640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f933c0723a0 con 0x7f933c0c7580 2024-09-17T04:55:25.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f9341517640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f933c072220 con 0x7f933c1557b0 2024-09-17T04:55:25.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f9341517640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f933c1628d0 con 0x7f933c0c7e20 2024-09-17T04:55:25.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f933affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f933c0c7580 0x7f933c14b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:25.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f933affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f933c0c7580 0x7f933c14b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57520/0 (socket says 172.21.15.150:57520) 2024-09-17T04:55:25.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f933affd640 1 -- 172.21.15.150:0/2149863695 learned_addr learned my addr 172.21.15.150:0/2149863695 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:25.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f933b7fe640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f933c1557b0 0x7f933c1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f933a7fc640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f933c0c7e20 0x7f933c14f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f933affd640 1 -- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f933c0c7e20 msgr2=0x7f933c14f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f933affd640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f933c0c7e20 0x7f933c14f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f933affd640 1 -- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f933c1557b0 msgr2=0x7f933c1579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f933affd640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f933c1557b0 0x7f933c1579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f933affd640 1 -- 172.21.15.150:0/2149863695 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933c14e260 con 0x7f933c0c7580 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f933affd640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f933c0c7580 0x7f933c14b680 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9330098170 tx=0x7f93300a65c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f93237fe640 1 -- 172.21.15.150:0/2149863695 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93300be070 con 0x7f933c0c7580 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f93237fe640 1 -- 172.21.15.150:0/2149863695 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93300a6630 con 0x7f933c0c7580 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f9341517640 1 -- 172.21.15.150:0/2149863695 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f933c14e810 con 0x7f933c0c7580 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f93237fe640 1 -- 172.21.15.150:0/2149863695 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9330005260 con 0x7f933c0c7580 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f9341517640 1 -- 172.21.15.150:0/2149863695 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f933c155d80 con 0x7f933c0c7580 2024-09-17T04:55:25.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f9341517640 1 -- 172.21.15.150:0/2149863695 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f933c155fe0 con 0x7f933c0c7580 2024-09-17T04:55:25.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.271+0000 7f93237fe640 1 -- 172.21.15.150:0/2149863695 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9330005400 con 0x7f933c0c7580 2024-09-17T04:55:25.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.275+0000 7f93237fe640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9328073200 0x7f9328075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:25.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.275+0000 7f93237fe640 1 -- 172.21.15.150:0/2149863695 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f933012e450 con 0x7f933c0c7580 2024-09-17T04:55:25.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.275+0000 7f933a7fc640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9328073200 0x7f9328075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:25.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.275+0000 7f933a7fc640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9328073200 0x7f9328075710 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f933c0c9b70 tx=0x7f932c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:25.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.279+0000 7f93237fe640 1 -- 172.21.15.150:0/2149863695 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f93300ac030 con 0x7f933c0c7580 2024-09-17T04:55:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.483+0000 7f9341517640 1 -- 172.21.15.150:0/2149863695 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f933c0c4990 con 0x7f933c0c7580 2024-09-17T04:55:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f93237fe640 1 -- 172.21.15.150:0/2149863695 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6540 (secure 0 0 0) 0x7f9330016410 con 0x7f933c0c7580 2024-09-17T04:55:25.488 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:25.488 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":46,"btime":"2024-09-17T04:55:18:968814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:13.937018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5466,"qdb_cluster":[5466]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:25.488 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 46 2024-09-17T04:55:25.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f9341517640 1 -- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9328073200 msgr2=0x7f9328075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:25.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f9341517640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9328073200 0x7f9328075710 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f933c0c9b70 tx=0x7f932c009040 comp rx=0 tx=0).stop 2024-09-17T04:55:25.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f9341517640 1 -- 172.21.15.150:0/2149863695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f933c0c7580 msgr2=0x7f933c14b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:25.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f9341517640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f933c0c7580 0x7f933c14b680 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9330098170 tx=0x7f93300a65c0 comp rx=0 tx=0).stop 2024-09-17T04:55:25.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f9341517640 1 -- 172.21.15.150:0/2149863695 shutdown_connections 2024-09-17T04:55:25.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f9341517640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9328073200 0x7f9328075710 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:25.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f9341517640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f933c1557b0 0x7f933c1579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:25.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f9341517640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f933c0c7e20 0x7f933c14f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:25.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f9341517640 1 --2- 172.21.15.150:0/2149863695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f933c0c7580 0x7f933c14b680 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:25.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f9341517640 1 -- 172.21.15.150:0/2149863695 >> 172.21.15.150:0/2149863695 conn(0x7f933c0aaa80 msgr2=0x7f933c0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:25.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f9341517640 1 -- 172.21.15.150:0/2149863695 shutdown_connections 2024-09-17T04:55:25.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:25.487+0000 7f9341517640 1 -- 172.21.15.150:0/2149863695 wait complete. 2024-09-17T04:55:25.509 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:55:27.510 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:27.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.675+0000 7f4f3066b640 1 -- 172.21.15.150:0/350371202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f280c5df0 msgr2=0x7f4f28157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:27.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.675+0000 7f4f3066b640 1 --2- 172.21.15.150:0/350371202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f280c5df0 0x7f4f28157330 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009900 tx=0x7f4f2401c8e0 comp rx=0 tx=0).stop 2024-09-17T04:55:27.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.675+0000 7f4f3066b640 1 -- 172.21.15.150:0/350371202 shutdown_connections 2024-09-17T04:55:27.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.675+0000 7f4f3066b640 1 --2- 172.21.15.150:0/350371202 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f280c6340 0x7f4f280c1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:27.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.675+0000 7f4f3066b640 1 --2- 172.21.15.150:0/350371202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f280c5df0 0x7f4f28157330 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:27.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.675+0000 7f4f3066b640 1 -- 172.21.15.150:0/350371202 >> 172.21.15.150:0/350371202 conn(0x7f4f28074750 msgr2=0x7f4f28074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:27.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.675+0000 7f4f3066b640 1 -- 172.21.15.150:0/350371202 shutdown_connections 2024-09-17T04:55:27.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.675+0000 7f4f3066b640 1 -- 172.21.15.150:0/350371202 wait complete. 2024-09-17T04:55:27.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f3066b640 1 Processor -- start 2024-09-17T04:55:27.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f3066b640 1 -- start start 2024-09-17T04:55:27.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f3066b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f280c4b20 0x7f4f28159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:27.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f3066b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f280c5df0 0x7f4f281f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:27.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f3066b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f280c6340 0x7f4f280cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:27.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f3066b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4f28163980 con 0x7f4f280c6340 2024-09-17T04:55:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f3066b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4f280c1950 con 0x7f4f280c4b20 2024-09-17T04:55:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f3066b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4f28163b00 con 0x7f4f280c5df0 2024-09-17T04:55:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f2ec08640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f280c6340 0x7f4f280cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f2ec08640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f280c6340 0x7f4f280cc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57538/0 (socket says 172.21.15.150:57538) 2024-09-17T04:55:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f2ec08640 1 -- 172.21.15.150:0/2999168280 learned_addr learned my addr 172.21.15.150:0/2999168280 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f2e407640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f280c4b20 0x7f4f28159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f2dc06640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f280c5df0 0x7f4f281f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f2ec08640 1 -- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f280c5df0 msgr2=0x7f4f281f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f2ec08640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f280c5df0 0x7f4f281f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f2ec08640 1 -- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f280c4b20 msgr2=0x7f4f28159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:27.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f2ec08640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f280c4b20 0x7f4f28159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:27.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f2ec08640 1 -- 172.21.15.150:0/2999168280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f28158030 con 0x7f4f280c6340 2024-09-17T04:55:27.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f2dc06640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f280c5df0 0x7f4f281f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:55:27.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f2ec08640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f280c6340 0x7f4f280cc4f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000c760 tx=0x7f4f2000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:27.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f1effd640 1 -- 172.21.15.150:0/2999168280 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f20003b80 con 0x7f4f280c6340 2024-09-17T04:55:27.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f1effd640 1 -- 172.21.15.150:0/2999168280 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f20003d20 con 0x7f4f280c6340 2024-09-17T04:55:27.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f1effd640 1 -- 172.21.15.150:0/2999168280 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2000aa50 con 0x7f4f280c6340 2024-09-17T04:55:27.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f3066b640 1 -- 172.21.15.150:0/2999168280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f280c9d80 con 0x7f4f280c6340 2024-09-17T04:55:27.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.679+0000 7f4f3066b640 1 -- 172.21.15.150:0/2999168280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f28157d50 con 0x7f4f280c6340 2024-09-17T04:55:27.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.683+0000 7f4f3066b640 1 -- 172.21.15.150:0/2999168280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f28157330 con 0x7f4f280c6340 2024-09-17T04:55:27.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.683+0000 7f4f1effd640 1 -- 172.21.15.150:0/2999168280 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4f2000abf0 con 0x7f4f280c6340 2024-09-17T04:55:27.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.683+0000 7f4f1effd640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f0c073200 0x7f4f0c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:27.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.683+0000 7f4f1effd640 1 -- 172.21.15.150:0/2999168280 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f4f200945c0 con 0x7f4f280c6340 2024-09-17T04:55:27.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.683+0000 7f4f2e407640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f0c073200 0x7f4f0c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:27.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.683+0000 7f4f2e407640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f0c073200 0x7f4f0c075710 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4f240af000 tx=0x7f4f2401b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:27.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.687+0000 7f4f1effd640 1 -- 172.21.15.150:0/2999168280 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4f280b20f0 con 0x7f4f280c6340 2024-09-17T04:55:27.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.887+0000 7f4f3066b640 1 -- 172.21.15.150:0/2999168280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f280b1b70 con 0x7f4f280c6340 2024-09-17T04:55:27.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.887+0000 7f4f1effd640 1 -- 172.21.15.150:0/2999168280 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6540 (secure 0 0 0) 0x7f4f2005f010 con 0x7f4f280c6340 2024-09-17T04:55:27.891 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:27.891 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":46,"btime":"2024-09-17T04:55:18:968814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:13.937018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5466,"qdb_cluster":[5466]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:27.892 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 46 2024-09-17T04:55:27.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.891+0000 7f4f3066b640 1 -- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f0c073200 msgr2=0x7f4f0c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:27.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.891+0000 7f4f3066b640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f0c073200 0x7f4f0c075710 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4f240af000 tx=0x7f4f2401b120 comp rx=0 tx=0).stop 2024-09-17T04:55:27.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.891+0000 7f4f3066b640 1 -- 172.21.15.150:0/2999168280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f280c6340 msgr2=0x7f4f280cc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:27.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.891+0000 7f4f3066b640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f280c6340 0x7f4f280cc4f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000c760 tx=0x7f4f2000cc20 comp rx=0 tx=0).stop 2024-09-17T04:55:27.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.891+0000 7f4f3066b640 1 -- 172.21.15.150:0/2999168280 shutdown_connections 2024-09-17T04:55:27.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.891+0000 7f4f3066b640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f0c073200 0x7f4f0c075710 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:27.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.891+0000 7f4f3066b640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f280c6340 0x7f4f280cc4f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:27.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.891+0000 7f4f3066b640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f280c5df0 0x7f4f281f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:27.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.891+0000 7f4f3066b640 1 --2- 172.21.15.150:0/2999168280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f280c4b20 0x7f4f28159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:27.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.891+0000 7f4f3066b640 1 -- 172.21.15.150:0/2999168280 >> 172.21.15.150:0/2999168280 conn(0x7f4f28074750 msgr2=0x7f4f28154c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:27.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.891+0000 7f4f3066b640 1 -- 172.21.15.150:0/2999168280 shutdown_connections 2024-09-17T04:55:27.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:27.891+0000 7f4f3066b640 1 -- 172.21.15.150:0/2999168280 wait complete. 2024-09-17T04:55:27.911 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:55:29.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:29.970 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:55:29.972 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:55:30.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.075+0000 7f143db1d640 1 -- 172.21.15.150:0/4086127838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14381568a0 msgr2=0x7f14380c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:30.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.075+0000 7f143db1d640 1 --2- 172.21.15.150:0/4086127838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14381568a0 0x7f14380c5810 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1428009900 tx=0x7f142801c8b0 comp rx=0 tx=0).stop 2024-09-17T04:55:30.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.075+0000 7f143db1d640 1 -- 172.21.15.150:0/4086127838 shutdown_connections 2024-09-17T04:55:30.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.075+0000 7f143db1d640 1 --2- 172.21.15.150:0/4086127838 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14380c20a0 0x7f14380c24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:30.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.075+0000 7f143db1d640 1 --2- 172.21.15.150:0/4086127838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14381568a0 0x7f14380c5810 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:30.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.075+0000 7f143db1d640 1 -- 172.21.15.150:0/4086127838 >> 172.21.15.150:0/4086127838 conn(0x7f1438074b80 msgr2=0x7f143806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:30.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.075+0000 7f143db1d640 1 -- 172.21.15.150:0/4086127838 shutdown_connections 2024-09-17T04:55:30.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.075+0000 7f143db1d640 1 -- 172.21.15.150:0/4086127838 wait complete. 2024-09-17T04:55:30.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f143db1d640 1 Processor -- start 2024-09-17T04:55:30.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f143db1d640 1 -- start start 2024-09-17T04:55:30.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f143db1d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14380c20a0 0x7f1438147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:30.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f143db1d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14380c35b0 0x7f1438152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:30.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f143db1d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14381568a0 0x7f14381e8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:30.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f143db1d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f14380c39c0 con 0x7f14380c20a0 2024-09-17T04:55:30.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f143db1d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1438154560 con 0x7f14381568a0 2024-09-17T04:55:30.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f143db1d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f14380c3b40 con 0x7f14380c35b0 2024-09-17T04:55:30.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f14377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14380c20a0 0x7f1438147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:30.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f14377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14380c20a0 0x7f1438147870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49494/0 (socket says 172.21.15.150:49494) 2024-09-17T04:55:30.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f14377fe640 1 -- 172.21.15.150:0/2637120023 learned_addr learned my addr 172.21.15.150:0/2637120023 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:30.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f1437fff640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14381568a0 0x7f14381e8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:30.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f1436ffd640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14380c35b0 0x7f1438152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:30.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f14377fe640 1 -- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14380c35b0 msgr2=0x7f1438152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:30.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f14377fe640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14380c35b0 0x7f1438152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:30.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f14377fe640 1 -- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14381568a0 msgr2=0x7f14381e8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:30.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f14377fe640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14381568a0 0x7f14381e8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:30.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f14377fe640 1 -- 172.21.15.150:0/2637120023 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14381553d0 con 0x7f14380c20a0 2024-09-17T04:55:30.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f1436ffd640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14380c35b0 0x7f1438152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:55:30.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f14377fe640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14380c20a0 0x7f1438147870 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1428009850 tx=0x7f1428005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:30.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f141ffff640 1 -- 172.21.15.150:0/2637120023 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14280b2040 con 0x7f14380c20a0 2024-09-17T04:55:30.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f141ffff640 1 -- 172.21.15.150:0/2637120023 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1428005020 con 0x7f14380c20a0 2024-09-17T04:55:30.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f143db1d640 1 -- 172.21.15.150:0/2637120023 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14381f06d0 con 0x7f14380c20a0 2024-09-17T04:55:30.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f141ffff640 1 -- 172.21.15.150:0/2637120023 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1428005430 con 0x7f14380c20a0 2024-09-17T04:55:30.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f143db1d640 1 -- 172.21.15.150:0/2637120023 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1438152410 con 0x7f14380c20a0 2024-09-17T04:55:30.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.079+0000 7f143db1d640 1 -- 172.21.15.150:0/2637120023 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14380c24b0 con 0x7f14380c20a0 2024-09-17T04:55:30.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.083+0000 7f141ffff640 1 -- 172.21.15.150:0/2637120023 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f14280bf030 con 0x7f14380c20a0 2024-09-17T04:55:30.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.083+0000 7f141ffff640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1424073200 0x7f1424075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:30.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.083+0000 7f141ffff640 1 -- 172.21.15.150:0/2637120023 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f142812f620 con 0x7f14380c20a0 2024-09-17T04:55:30.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.083+0000 7f1436ffd640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1424073200 0x7f1424075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:30.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.083+0000 7f1436ffd640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1424073200 0x7f1424075710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f14381556f0 tx=0x7f142c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:30.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.087+0000 7f141ffff640 1 -- 172.21.15.150:0/2637120023 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f14280c6060 con 0x7f14380c20a0 2024-09-17T04:55:30.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.291+0000 7f143db1d640 1 -- 172.21.15.150:0/2637120023 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1438157520 con 0x7f14380c20a0 2024-09-17T04:55:30.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f141ffff640 1 -- 172.21.15.150:0/2637120023 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6540 (secure 0 0 0) 0x7f14280164d0 con 0x7f14380c20a0 2024-09-17T04:55:30.296 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:30.296 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":46,"btime":"2024-09-17T04:55:18:968814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:13.937018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5466,"qdb_cluster":[5466]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:30.296 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 46 2024-09-17T04:55:30.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f143db1d640 1 -- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1424073200 msgr2=0x7f1424075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:30.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f143db1d640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1424073200 0x7f1424075710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f14381556f0 tx=0x7f142c009040 comp rx=0 tx=0).stop 2024-09-17T04:55:30.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f143db1d640 1 -- 172.21.15.150:0/2637120023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14380c20a0 msgr2=0x7f1438147870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:30.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f143db1d640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14380c20a0 0x7f1438147870 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1428009850 tx=0x7f1428005e00 comp rx=0 tx=0).stop 2024-09-17T04:55:30.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f143db1d640 1 -- 172.21.15.150:0/2637120023 shutdown_connections 2024-09-17T04:55:30.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f143db1d640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1424073200 0x7f1424075710 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:30.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f143db1d640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14381568a0 0x7f14381e8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:30.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f143db1d640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14380c35b0 0x7f1438152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:30.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f143db1d640 1 --2- 172.21.15.150:0/2637120023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14380c20a0 0x7f1438147870 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:30.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f143db1d640 1 -- 172.21.15.150:0/2637120023 >> 172.21.15.150:0/2637120023 conn(0x7f1438074b80 msgr2=0x7f143815a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:30.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f143db1d640 1 -- 172.21.15.150:0/2637120023 shutdown_connections 2024-09-17T04:55:30.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:30.295+0000 7f143db1d640 1 -- 172.21.15.150:0/2637120023 wait complete. 2024-09-17T04:55:30.317 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:55:32.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:32.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.483+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2499224433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff488004810 msgr2=0x7ff488004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:32.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.483+0000 7ff4a1d6b640 1 --2- 172.21.15.150:0/2499224433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff488004810 0x7ff488004c20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff484009b60 tx=0x7ff48401d2b0 comp rx=0 tx=0).stop 2024-09-17T04:55:32.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.483+0000 7ff49bfff640 1 -- 172.21.15.150:0/2499224433 reap_dead start 2024-09-17T04:55:32.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.483+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2499224433 shutdown_connections 2024-09-17T04:55:32.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.483+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2499224433 >> 172.21.15.150:0/2499224433 conn(0x7ff49c0c4a00 msgr2=0x7ff49c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:32.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.483+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2499224433 shutdown_connections 2024-09-17T04:55:32.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.483+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2499224433 wait complete. 2024-09-17T04:55:32.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff4a1d6b640 1 Processor -- start 2024-09-17T04:55:32.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff4a1d6b640 1 -- start start 2024-09-17T04:55:32.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff4a1d6b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff488004810 0x7ff49c156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:32.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff4a1d6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff49c14df40 0x7ff49c14aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:32.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff4a1d6b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff49c14b000 0x7ff49c154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:32.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff4a1d6b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff49c074f70 con 0x7ff49c14df40 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff4a1d6b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff49c0adef0 con 0x7ff488004810 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff4a1d6b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff49c0750f0 con 0x7ff49c14b000 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff493fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff49c14df40 0x7ff49c14aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff493fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff49c14df40 0x7ff49c14aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49522/0 (socket says 172.21.15.150:49522) 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff493fff640 1 -- 172.21.15.150:0/2960544603 learned_addr learned my addr 172.21.15.150:0/2960544603 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff49b7fe640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff488004810 0x7ff49c156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff49bfff640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff49c14b000 0x7ff49c154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff493fff640 1 -- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff49c14b000 msgr2=0x7ff49c154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff493fff640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff49c14b000 0x7ff49c154760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff493fff640 1 -- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff488004810 msgr2=0x7ff49c156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff493fff640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff488004810 0x7ff49c156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff493fff640 1 -- 172.21.15.150:0/2960544603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff49c14b570 con 0x7ff49c14df40 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff49b7fe640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff488004810 0x7ff49c156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:55:32.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff493fff640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff49c14df40 0x7ff49c14aac0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff4840af040 tx=0x7ff484009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:32.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff498ff9640 1 -- 172.21.15.150:0/2960544603 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4840b1050 con 0x7ff49c14df40 2024-09-17T04:55:32.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff498ff9640 1 -- 172.21.15.150:0/2960544603 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4840b2040 con 0x7ff49c14df40 2024-09-17T04:55:32.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff498ff9640 1 -- 172.21.15.150:0/2960544603 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4840043c0 con 0x7ff49c14df40 2024-09-17T04:55:32.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2960544603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff49c1552a0 con 0x7ff49c14df40 2024-09-17T04:55:32.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.487+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2960544603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff49c1e06f0 con 0x7ff49c14df40 2024-09-17T04:55:32.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.491+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2960544603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff49c0c60f0 con 0x7ff49c14df40 2024-09-17T04:55:32.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.491+0000 7ff498ff9640 1 -- 172.21.15.150:0/2960544603 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff4840046d0 con 0x7ff49c14df40 2024-09-17T04:55:32.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.491+0000 7ff498ff9640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff47c073230 0x7ff47c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:32.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.491+0000 7ff498ff9640 1 -- 172.21.15.150:0/2960544603 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7ff484131190 con 0x7ff49c14df40 2024-09-17T04:55:32.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.491+0000 7ff49b7fe640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff47c073230 0x7ff47c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:32.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.491+0000 7ff49b7fe640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff47c073230 0x7ff47c075740 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff49c1513e0 tx=0x7ff48c0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:32.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.495+0000 7ff498ff9640 1 -- 172.21.15.150:0/2960544603 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff484016fc0 con 0x7ff49c14df40 2024-09-17T04:55:32.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.707+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2960544603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff49c0a9830 con 0x7ff49c14df40 2024-09-17T04:55:32.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.707+0000 7ff498ff9640 1 -- 172.21.15.150:0/2960544603 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6540 (secure 0 0 0) 0x7ff4840fbbf0 con 0x7ff49c14df40 2024-09-17T04:55:32.709 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:32.710 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":46,"btime":"2024-09-17T04:55:18:968814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:13.937018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5466,"qdb_cluster":[5466]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:32.710 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 46 2024-09-17T04:55:32.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.711+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff47c073230 msgr2=0x7ff47c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:32.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.711+0000 7ff4a1d6b640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff47c073230 0x7ff47c075740 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff49c1513e0 tx=0x7ff48c0091e0 comp rx=0 tx=0).stop 2024-09-17T04:55:32.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.711+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2960544603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff49c14df40 msgr2=0x7ff49c14aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:32.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.711+0000 7ff4a1d6b640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff49c14df40 0x7ff49c14aac0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff4840af040 tx=0x7ff484009730 comp rx=0 tx=0).stop 2024-09-17T04:55:32.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.711+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2960544603 shutdown_connections 2024-09-17T04:55:32.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.711+0000 7ff4a1d6b640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff47c073230 0x7ff47c075740 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:32.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.711+0000 7ff4a1d6b640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff49c14b000 0x7ff49c154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:32.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.711+0000 7ff4a1d6b640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff49c14df40 0x7ff49c14aac0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:32.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.711+0000 7ff4a1d6b640 1 --2- 172.21.15.150:0/2960544603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff488004810 0x7ff49c156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:32.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.711+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2960544603 >> 172.21.15.150:0/2960544603 conn(0x7ff49c0c4a00 msgr2=0x7ff49c0c3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:32.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.711+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2960544603 shutdown_connections 2024-09-17T04:55:32.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:32.711+0000 7ff4a1d6b640 1 -- 172.21.15.150:0/2960544603 wait complete. 2024-09-17T04:55:32.731 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:55:34.733 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.895+0000 7f89a1339640 1 -- 172.21.15.150:0/1440458798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8984005160 msgr2=0x7f898400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.895+0000 7f89a1339640 1 --2- 172.21.15.150:0/1440458798 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8984005160 0x7f898400b9d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f898c0016f0 tx=0x7f898c01c7b0 comp rx=0 tx=0).stop 2024-09-17T04:55:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.895+0000 7f899b7fe640 1 -- 172.21.15.150:0/1440458798 reap_dead start 2024-09-17T04:55:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.895+0000 7f89a1339640 1 -- 172.21.15.150:0/1440458798 shutdown_connections 2024-09-17T04:55:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.895+0000 7f89a1339640 1 -- 172.21.15.150:0/1440458798 >> 172.21.15.150:0/1440458798 conn(0x7f899c070b50 msgr2=0x7f899c070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.895+0000 7f89a1339640 1 -- 172.21.15.150:0/1440458798 shutdown_connections 2024-09-17T04:55:34.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.895+0000 7f89a1339640 1 -- 172.21.15.150:0/1440458798 wait complete. 2024-09-17T04:55:34.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89a1339640 1 Processor -- start 2024-09-17T04:55:34.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89a1339640 1 -- start start 2024-09-17T04:55:34.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89a1339640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f899c143ec0 0x7f899c140100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:34.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89a1339640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f899c1442d0 0x7f899c140640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:34.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89a1339640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f899c157e70 0x7f899c158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:34.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89a1339640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f899c069cd0 con 0x7f899c1442d0 2024-09-17T04:55:34.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89a1339640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f899c0c27d0 con 0x7f899c143ec0 2024-09-17T04:55:34.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89a1339640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f899c069e50 con 0x7f899c157e70 2024-09-17T04:55:34.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f899a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f899c1442d0 0x7f899c140640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:34.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f899a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f899c1442d0 0x7f899c140640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49530/0 (socket says 172.21.15.150:49530) 2024-09-17T04:55:34.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f899a7fc640 1 -- 172.21.15.150:0/1757097311 learned_addr learned my addr 172.21.15.150:0/1757097311 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:34.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f899b7fe640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f899c157e70 0x7f899c158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:34.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f899affd640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f899c143ec0 0x7f899c140100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:34.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f899a7fc640 1 -- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f899c157e70 msgr2=0x7f899c158280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:34.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f899a7fc640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f899c157e70 0x7f899c158280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:34.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f899a7fc640 1 -- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f899c143ec0 msgr2=0x7f899c140100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:34.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f899a7fc640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f899c143ec0 0x7f899c140100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:34.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f899a7fc640 1 -- 172.21.15.150:0/1757097311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f899c140f00 con 0x7f899c1442d0 2024-09-17T04:55:34.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f899affd640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f899c143ec0 0x7f899c140100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:55:34.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f899a7fc640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f899c1442d0 0x7f899c140640 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f89900015b0 tx=0x7f899000fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:34.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89837fe640 1 -- 172.21.15.150:0/1757097311 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899001a070 con 0x7f899c1442d0 2024-09-17T04:55:34.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89837fe640 1 -- 172.21.15.150:0/1757097311 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f899000d640 con 0x7f899c1442d0 2024-09-17T04:55:34.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89a1339640 1 -- 172.21.15.150:0/1757097311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f899c158f70 con 0x7f899c1442d0 2024-09-17T04:55:34.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89837fe640 1 -- 172.21.15.150:0/1757097311 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899000da90 con 0x7f899c1442d0 2024-09-17T04:55:34.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.899+0000 7f89a1339640 1 -- 172.21.15.150:0/1757097311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f899c144b90 con 0x7f899c1442d0 2024-09-17T04:55:34.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.903+0000 7f89a1339640 1 -- 172.21.15.150:0/1757097311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f899c0aa330 con 0x7f899c1442d0 2024-09-17T04:55:34.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.903+0000 7f89837fe640 1 -- 172.21.15.150:0/1757097311 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f899001d070 con 0x7f899c1442d0 2024-09-17T04:55:34.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.903+0000 7f89837fe640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8988073200 0x7f8988075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:34.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.903+0000 7f89837fe640 1 -- 172.21.15.150:0/1757097311 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f8990096d60 con 0x7f899c1442d0 2024-09-17T04:55:34.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.903+0000 7f899affd640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8988073200 0x7f8988075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:34.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.903+0000 7f899affd640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8988073200 0x7f8988075710 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f899c0c84c0 tx=0x7f89940091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:34.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:34.907+0000 7f89837fe640 1 -- 172.21.15.150:0/1757097311 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f89900618e0 con 0x7f899c1442d0 2024-09-17T04:55:35.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.107+0000 7f89a1339640 1 -- 172.21.15.150:0/1757097311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f899c0ae800 con 0x7f899c1442d0 2024-09-17T04:55:35.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89837fe640 1 -- 172.21.15.150:0/1757097311 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6540 (secure 0 0 0) 0x7f89900611e0 con 0x7f899c1442d0 2024-09-17T04:55:35.111 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:35.111 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":46,"btime":"2024-09-17T04:55:18:968814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:13.937018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5466,"qdb_cluster":[5466]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:35.111 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 46 2024-09-17T04:55:35.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89a1339640 1 -- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8988073200 msgr2=0x7f8988075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:35.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89a1339640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8988073200 0x7f8988075710 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f899c0c84c0 tx=0x7f89940091e0 comp rx=0 tx=0).stop 2024-09-17T04:55:35.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89a1339640 1 -- 172.21.15.150:0/1757097311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f899c1442d0 msgr2=0x7f899c140640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:35.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89a1339640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f899c1442d0 0x7f899c140640 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f89900015b0 tx=0x7f899000fc20 comp rx=0 tx=0).stop 2024-09-17T04:55:35.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89a1339640 1 -- 172.21.15.150:0/1757097311 shutdown_connections 2024-09-17T04:55:35.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89a1339640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8988073200 0x7f8988075710 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:35.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89a1339640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f899c157e70 0x7f899c158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:35.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89a1339640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f899c1442d0 0x7f899c140640 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:35.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89a1339640 1 --2- 172.21.15.150:0/1757097311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f899c143ec0 0x7f899c140100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:35.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89a1339640 1 -- 172.21.15.150:0/1757097311 >> 172.21.15.150:0/1757097311 conn(0x7f899c070b50 msgr2=0x7f899c14d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:35.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89a1339640 1 -- 172.21.15.150:0/1757097311 shutdown_connections 2024-09-17T04:55:35.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:35.111+0000 7f89a1339640 1 -- 172.21.15.150:0/1757097311 wait complete. 2024-09-17T04:55:35.131 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:55:37.132 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.299+0000 7f5990c0e640 1 -- 172.21.15.150:0/4257293181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1505a0 msgr2=0x7f598c152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.299+0000 7f5990c0e640 1 --2- 172.21.15.150:0/4257293181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1505a0 0x7f598c152cf0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f598c0c7180 tx=0x7f597c01c8a0 comp rx=0 tx=0).stop 2024-09-17T04:55:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.299+0000 7f5990c0e640 1 -- 172.21.15.150:0/4257293181 shutdown_connections 2024-09-17T04:55:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.299+0000 7f5990c0e640 1 --2- 172.21.15.150:0/4257293181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1505a0 0x7f598c152cf0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.299+0000 7f5990c0e640 1 --2- 172.21.15.150:0/4257293181 >> v2:172.21.15.176:3300/0 conn(0x7f598c0a7350 0x7f598c150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.299+0000 7f5990c0e640 1 --2- 172.21.15.150:0/4257293181 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f598c0c42e0 0x7f598c0c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.299+0000 7f5990c0e640 1 -- 172.21.15.150:0/4257293181 >> 172.21.15.150:0/4257293181 conn(0x7f598c0c4a00 msgr2=0x7f598c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.299+0000 7f5990c0e640 1 -- 172.21.15.150:0/4257293181 shutdown_connections 2024-09-17T04:55:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.299+0000 7f5990c0e640 1 -- 172.21.15.150:0/4257293181 wait complete. 2024-09-17T04:55:37.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.299+0000 7f5990c0e640 1 Processor -- start 2024-09-17T04:55:37.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.299+0000 7f5990c0e640 1 -- start start 2024-09-17T04:55:37.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5990c0e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f598c0a7350 0x7f598c142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:37.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5990c0e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f598c0c42e0 0x7f598c0c1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:37.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5990c0e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1505a0 0x7f598c1f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:37.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5990c0e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f598c0c6bd0 con 0x7f598c1505a0 2024-09-17T04:55:37.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5990c0e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f598c070b80 con 0x7f598c0a7350 2024-09-17T04:55:37.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5990c0e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f598c0c6d50 con 0x7f598c0c42e0 2024-09-17T04:55:37.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f598ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1505a0 0x7f598c1f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:37.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f598ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1505a0 0x7f598c1f3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49550/0 (socket says 172.21.15.150:49550) 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f598ad9d640 1 -- 172.21.15.150:0/1565154790 learned_addr learned my addr 172.21.15.150:0/1565154790 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f598a59c640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f598c0a7350 0x7f598c142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5989d9b640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f598c0c42e0 0x7f598c0c1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f598ad9d640 1 -- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f598c0c42e0 msgr2=0x7f598c0c1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f598ad9d640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f598c0c42e0 0x7f598c0c1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f598ad9d640 1 -- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f598c0a7350 msgr2=0x7f598c142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f598ad9d640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f598c0a7350 0x7f598c142a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f598ad9d640 1 -- 172.21.15.150:0/1565154790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f598c143220 con 0x7f598c1505a0 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f598ad9d640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1505a0 0x7f598c1f3300 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f597c0b0040 tx=0x7f597c0a5800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5982ffd640 1 -- 172.21.15.150:0/1565154790 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597c0b2040 con 0x7f598c1505a0 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5982ffd640 1 -- 172.21.15.150:0/1565154790 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f597c005070 con 0x7f598c1505a0 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5990c0e640 1 -- 172.21.15.150:0/1565154790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f598c1434e0 con 0x7f598c1505a0 2024-09-17T04:55:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5982ffd640 1 -- 172.21.15.150:0/1565154790 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597c0054c0 con 0x7f598c1505a0 2024-09-17T04:55:37.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5990c0e640 1 -- 172.21.15.150:0/1565154790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f598c0748b0 con 0x7f598c1505a0 2024-09-17T04:55:37.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.303+0000 7f5990c0e640 1 -- 172.21.15.150:0/1565154790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f598c0c6690 con 0x7f598c1505a0 2024-09-17T04:55:37.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.307+0000 7f5982ffd640 1 -- 172.21.15.150:0/1565154790 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f597c0be060 con 0x7f598c1505a0 2024-09-17T04:55:37.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.307+0000 7f5982ffd640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f596c073230 0x7f596c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:37.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.307+0000 7f5982ffd640 1 -- 172.21.15.150:0/1565154790 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f597c12e0a0 con 0x7f598c1505a0 2024-09-17T04:55:37.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.307+0000 7f598a59c640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f596c073230 0x7f596c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:37.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.307+0000 7f598a59c640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f596c073230 0x7f596c075740 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5984009d30 tx=0x7f5984009710 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:37.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.311+0000 7f5982ffd640 1 -- 172.21.15.150:0/1565154790 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f597c016490 con 0x7f598c1505a0 2024-09-17T04:55:37.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.519+0000 7f5990c0e640 1 -- 172.21.15.150:0/1565154790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f598c0a7130 con 0x7f598c1505a0 2024-09-17T04:55:37.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.519+0000 7f5982ffd640 1 -- 172.21.15.150:0/1565154790 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6540 (secure 0 0 0) 0x7f597c0f8c70 con 0x7f598c1505a0 2024-09-17T04:55:37.521 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:37.521 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":46,"btime":"2024-09-17T04:55:18:968814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:13.937018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5466,"qdb_cluster":[5466]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:37.521 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 46 2024-09-17T04:55:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.523+0000 7f5990c0e640 1 -- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f596c073230 msgr2=0x7f596c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.523+0000 7f5990c0e640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f596c073230 0x7f596c075740 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5984009d30 tx=0x7f5984009710 comp rx=0 tx=0).stop 2024-09-17T04:55:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.523+0000 7f5990c0e640 1 -- 172.21.15.150:0/1565154790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1505a0 msgr2=0x7f598c1f3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.523+0000 7f5990c0e640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1505a0 0x7f598c1f3300 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f597c0b0040 tx=0x7f597c0a5800 comp rx=0 tx=0).stop 2024-09-17T04:55:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.523+0000 7f5990c0e640 1 -- 172.21.15.150:0/1565154790 shutdown_connections 2024-09-17T04:55:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.523+0000 7f5990c0e640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f596c073230 0x7f596c075740 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.523+0000 7f5990c0e640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f598c1505a0 0x7f598c1f3300 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.523+0000 7f5990c0e640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f598c0c42e0 0x7f598c0c1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.523+0000 7f5990c0e640 1 --2- 172.21.15.150:0/1565154790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f598c0a7350 0x7f598c142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.523+0000 7f5990c0e640 1 -- 172.21.15.150:0/1565154790 >> 172.21.15.150:0/1565154790 conn(0x7f598c0c4a00 msgr2=0x7f598c0c2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.523+0000 7f5990c0e640 1 -- 172.21.15.150:0/1565154790 shutdown_connections 2024-09-17T04:55:37.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:37.523+0000 7f5990c0e640 1 -- 172.21.15.150:0/1565154790 wait complete. 2024-09-17T04:55:37.544 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:55:39.545 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:39.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.711+0000 7f113e90f640 1 -- 172.21.15.150:0/232213979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11380c35b0 msgr2=0x7f11380c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:39.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.711+0000 7f113e90f640 1 --2- 172.21.15.150:0/232213979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11380c35b0 0x7f11380c8a80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f112c009840 tx=0x7f112c01c7f0 comp rx=0 tx=0).stop 2024-09-17T04:55:39.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.711+0000 7f113e90f640 1 -- 172.21.15.150:0/232213979 shutdown_connections 2024-09-17T04:55:39.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.711+0000 7f113e90f640 1 --2- 172.21.15.150:0/232213979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11380c8fc0 0x7f11380c20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:39.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.711+0000 7f113e90f640 1 --2- 172.21.15.150:0/232213979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11380c35b0 0x7f11380c8a80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:39.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.711+0000 7f113e90f640 1 -- 172.21.15.150:0/232213979 >> 172.21.15.150:0/232213979 conn(0x7f1138074b80 msgr2=0x7f113806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:39.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.711+0000 7f113e90f640 1 -- 172.21.15.150:0/232213979 shutdown_connections 2024-09-17T04:55:39.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.711+0000 7f113e90f640 1 -- 172.21.15.150:0/232213979 wait complete. 2024-09-17T04:55:39.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f113e90f640 1 Processor -- start 2024-09-17T04:55:39.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f113e90f640 1 -- start start 2024-09-17T04:55:39.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f113e90f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11380c35b0 0x7f11381f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:39.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f113e90f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11380c8fc0 0x7f1138143430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:39.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f113e90f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11381568a0 0x7f1138159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:39.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f113e90f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f11380c6b70 con 0x7f11380c35b0 2024-09-17T04:55:39.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f113e90f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f11380c69f0 con 0x7f11380c8fc0 2024-09-17T04:55:39.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f113e90f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f11380b0fd0 con 0x7f11381568a0 2024-09-17T04:55:39.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f1137fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11380c35b0 0x7f11381f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:39.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f1137fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11380c35b0 0x7f11381f02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34622/0 (socket says 172.21.15.150:34622) 2024-09-17T04:55:39.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f1137fff640 1 -- 172.21.15.150:0/1904503921 learned_addr learned my addr 172.21.15.150:0/1904503921 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f113ceac640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11381568a0 0x7f1138159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f11377fe640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11380c8fc0 0x7f1138143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f1137fff640 1 -- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11381568a0 msgr2=0x7f1138159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f1137fff640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11381568a0 0x7f1138159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f1137fff640 1 -- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11380c8fc0 msgr2=0x7f1138143430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f1137fff640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11380c8fc0 0x7f1138143430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f1137fff640 1 -- 172.21.15.150:0/1904503921 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11381572a0 con 0x7f11380c35b0 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f1137fff640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11380c35b0 0x7f11381f02d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f112800c400 tx=0x7f112800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f1134ff9640 1 -- 172.21.15.150:0/1904503921 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128017070 con 0x7f11380c35b0 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f1134ff9640 1 -- 172.21.15.150:0/1904503921 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1128003600 con 0x7f11380c35b0 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f113e90f640 1 -- 172.21.15.150:0/1904503921 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11381f46a0 con 0x7f11380c35b0 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f1134ff9640 1 -- 172.21.15.150:0/1904503921 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128003a50 con 0x7f11380c35b0 2024-09-17T04:55:39.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f113e90f640 1 -- 172.21.15.150:0/1904503921 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11381f38e0 con 0x7f11380c35b0 2024-09-17T04:55:39.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.715+0000 7f113e90f640 1 -- 172.21.15.150:0/1904503921 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11380b14e0 con 0x7f11380c35b0 2024-09-17T04:55:39.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.719+0000 7f1134ff9640 1 -- 172.21.15.150:0/1904503921 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f112801a070 con 0x7f11380c35b0 2024-09-17T04:55:39.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.719+0000 7f1134ff9640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1124073200 0x7f1124075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:39.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.719+0000 7f1134ff9640 1 -- 172.21.15.150:0/1904503921 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7182+0+0 (secure 0 0 0) 0x7f1128093a30 con 0x7f11380c35b0 2024-09-17T04:55:39.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.719+0000 7f11377fe640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1124073200 0x7f1124075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:39.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.719+0000 7f11377fe640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1124073200 0x7f1124075710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f112c0af000 tx=0x7f112c01b030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:39.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.723+0000 7f1134ff9640 1 -- 172.21.15.150:0/1904503921 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f112805e5b0 con 0x7f11380c35b0 2024-09-17T04:55:39.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.931+0000 7f113e90f640 1 -- 172.21.15.150:0/1904503921 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11380c23c0 con 0x7f11380c35b0 2024-09-17T04:55:39.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f1134ff9640 1 -- 172.21.15.150:0/1904503921 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6540 (secure 0 0 0) 0x7f112805deb0 con 0x7f11380c35b0 2024-09-17T04:55:39.935 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:39.935 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":46,"btime":"2024-09-17T04:55:18:968814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5254,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:13.937018+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5466":{"gid":5466,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/4286123257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4286123257},{"type":"v1","addr":"172.21.15.150:6833","nonce":4286123257}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5466,"qdb_cluster":[5466]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:39.935 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 46 2024-09-17T04:55:39.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f113e90f640 1 -- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1124073200 msgr2=0x7f1124075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:39.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f113e90f640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1124073200 0x7f1124075710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f112c0af000 tx=0x7f112c01b030 comp rx=0 tx=0).stop 2024-09-17T04:55:39.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f113e90f640 1 -- 172.21.15.150:0/1904503921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11380c35b0 msgr2=0x7f11381f02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:39.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f113e90f640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11380c35b0 0x7f11381f02d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f112800c400 tx=0x7f112800c8c0 comp rx=0 tx=0).stop 2024-09-17T04:55:39.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f113e90f640 1 -- 172.21.15.150:0/1904503921 shutdown_connections 2024-09-17T04:55:39.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f113e90f640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1124073200 0x7f1124075710 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:39.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f113e90f640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11380c8fc0 0x7f1138143430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:39.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f113e90f640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11381568a0 0x7f1138159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:39.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f113e90f640 1 --2- 172.21.15.150:0/1904503921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11380c35b0 0x7f11381f02d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:39.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f113e90f640 1 -- 172.21.15.150:0/1904503921 >> 172.21.15.150:0/1904503921 conn(0x7f1138074b80 msgr2=0x7f1138161580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:39.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f113e90f640 1 -- 172.21.15.150:0/1904503921 shutdown_connections 2024-09-17T04:55:39.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:39.935+0000 7f113e90f640 1 -- 172.21.15.150:0/1904503921 wait complete. 2024-09-17T04:55:39.955 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:55:41.956 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:42.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.119+0000 7f23c7aee640 1 -- 172.21.15.150:0/307685656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c00c3db0 msgr2=0x7f23c00ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:42.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.119+0000 7f23c7aee640 1 --2- 172.21.15.150:0/307685656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c00c3db0 0x7f23c00ad2a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f23b0009900 tx=0x7f23b001c8e0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.119+0000 7f23c7aee640 1 -- 172.21.15.150:0/307685656 shutdown_connections 2024-09-17T04:55:42.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.119+0000 7f23c7aee640 1 --2- 172.21.15.150:0/307685656 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23c00ad7e0 0x7f23c00701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.119+0000 7f23c7aee640 1 --2- 172.21.15.150:0/307685656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c00c3db0 0x7f23c00ad2a0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.119+0000 7f23c7aee640 1 -- 172.21.15.150:0/307685656 >> 172.21.15.150:0/307685656 conn(0x7f23c00c4a00 msgr2=0x7f23c00c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:42.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.119+0000 7f23c7aee640 1 -- 172.21.15.150:0/307685656 shutdown_connections 2024-09-17T04:55:42.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.119+0000 7f23c7aee640 1 -- 172.21.15.150:0/307685656 wait complete. 2024-09-17T04:55:42.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c7aee640 1 Processor -- start 2024-09-17T04:55:42.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c7aee640 1 -- start start 2024-09-17T04:55:42.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c7aee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c00ad7e0 0x7f23c01560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:42.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c7aee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23c00c3db0 0x7f23c01f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:42.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c7aee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23c00c41c0 0x7f23c014b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c7aee640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f23c00a75f0 con 0x7f23c00ad7e0 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c7aee640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f23c00add70 con 0x7f23c00c41c0 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c7aee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f23c00a7770 con 0x7f23c00c3db0 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c588a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c00ad7e0 0x7f23c01560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c588a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c00ad7e0 0x7f23c01560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34646/0 (socket says 172.21.15.150:34646) 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c588a640 1 -- 172.21.15.150:0/1557912458 learned_addr learned my addr 172.21.15.150:0/1557912458 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c5089640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23c00c3db0 0x7f23c01f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c608b640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23c00c41c0 0x7f23c014b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c588a640 1 -- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23c00c3db0 msgr2=0x7f23c01f0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c588a640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23c00c3db0 0x7f23c01f0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c588a640 1 -- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23c00c41c0 msgr2=0x7f23c014b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c588a640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23c00c41c0 0x7f23c014b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c588a640 1 -- 172.21.15.150:0/1557912458 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c0145ca0 con 0x7f23c00ad7e0 2024-09-17T04:55:42.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c588a640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c00ad7e0 0x7f23c01560b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f23b00b0040 tx=0x7f23b00a5620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:42.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23b67fc640 1 -- 172.21.15.150:0/1557912458 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b00bb070 con 0x7f23c00ad7e0 2024-09-17T04:55:42.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c7aee640 1 -- 172.21.15.150:0/1557912458 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23c01462f0 con 0x7f23c00ad7e0 2024-09-17T04:55:42.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23b67fc640 1 -- 172.21.15.150:0/1557912458 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f23b0004470 con 0x7f23c00ad7e0 2024-09-17T04:55:42.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23b67fc640 1 -- 172.21.15.150:0/1557912458 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b00b2040 con 0x7f23c00ad7e0 2024-09-17T04:55:42.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c7aee640 1 -- 172.21.15.150:0/1557912458 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23c0152320 con 0x7f23c00ad7e0 2024-09-17T04:55:42.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.123+0000 7f23c7aee640 1 -- 172.21.15.150:0/1557912458 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23c00ad2a0 con 0x7f23c00ad7e0 2024-09-17T04:55:42.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.127+0000 7f23b67fc640 1 -- 172.21.15.150:0/1557912458 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f23b00052d0 con 0x7f23c00ad7e0 2024-09-17T04:55:42.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.127+0000 7f23b67fc640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f23a4073230 0x7f23a4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:42.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.127+0000 7f23b67fc640 1 -- 172.21.15.150:0/1557912458 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f23b012e300 con 0x7f23c00ad7e0 2024-09-17T04:55:42.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.127+0000 7f23c5089640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f23a4073230 0x7f23a4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:42.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.127+0000 7f23c5089640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f23a4073230 0x7f23a4075740 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f23c01587b0 tx=0x7f23bc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:42.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.131+0000 7f23b67fc640 1 -- 172.21.15.150:0/1557912458 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f23b00164d0 con 0x7f23c00ad7e0 2024-09-17T04:55:42.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.339+0000 7f23c7aee640 1 -- 172.21.15.150:0/1557912458 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23c00aacc0 con 0x7f23c00ad7e0 2024-09-17T04:55:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.339+0000 7f23b67fc640 1 -- 172.21.15.150:0/1557912458 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+5708 (secure 0 0 0) 0x7f23b00f8c70 con 0x7f23c00ad7e0 2024-09-17T04:55:42.341 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:42.341 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":48,"btime":"2024-09-17T04:55:41:882347+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:41.169113+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:42.341 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 48 2024-09-17T04:55:42.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.339+0000 7f23c7aee640 1 -- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f23a4073230 msgr2=0x7f23a4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:42.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.339+0000 7f23c7aee640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f23a4073230 0x7f23a4075740 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f23c01587b0 tx=0x7f23bc009040 comp rx=0 tx=0).stop 2024-09-17T04:55:42.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.339+0000 7f23c7aee640 1 -- 172.21.15.150:0/1557912458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c00ad7e0 msgr2=0x7f23c01560b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:42.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.339+0000 7f23c7aee640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c00ad7e0 0x7f23c01560b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f23b00b0040 tx=0x7f23b00a5620 comp rx=0 tx=0).stop 2024-09-17T04:55:42.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.343+0000 7f23c7aee640 1 -- 172.21.15.150:0/1557912458 shutdown_connections 2024-09-17T04:55:42.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.343+0000 7f23c7aee640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f23a4073230 0x7f23a4075740 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.343+0000 7f23c7aee640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f23c00c41c0 0x7f23c014b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.343+0000 7f23c7aee640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23c00c3db0 0x7f23c01f0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.343+0000 7f23c7aee640 1 --2- 172.21.15.150:0/1557912458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23c00ad7e0 0x7f23c01560b0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.343+0000 7f23c7aee640 1 -- 172.21.15.150:0/1557912458 >> 172.21.15.150:0/1557912458 conn(0x7f23c00c4a00 msgr2=0x7f23c015e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:42.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.343+0000 7f23c7aee640 1 -- 172.21.15.150:0/1557912458 shutdown_connections 2024-09-17T04:55:42.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.343+0000 7f23c7aee640 1 -- 172.21.15.150:0/1557912458 wait complete. 2024-09-17T04:55:42.364 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-17T04:55:42.364 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T04:55:42.364 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:42.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 -- 172.21.15.150:0/2415057020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34151d00 msgr2=0x7f7d340c3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:42.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 --2- 172.21.15.150:0/2415057020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34151d00 0x7f7d340c3430 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7d340adcf0 tx=0x7f7d2401cd80 comp rx=0 tx=0).stop 2024-09-17T04:55:42.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 -- 172.21.15.150:0/2415057020 shutdown_connections 2024-09-17T04:55:42.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 --2- 172.21.15.150:0/2415057020 >> v2:172.21.15.176:3300/0 conn(0x7f7d340c3970 0x7f7d340c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 --2- 172.21.15.150:0/2415057020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34151d00 0x7f7d340c3430 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 -- 172.21.15.150:0/2415057020 >> 172.21.15.150:0/2415057020 conn(0x7f7d340c4a00 msgr2=0x7f7d340c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:42.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 -- 172.21.15.150:0/2415057020 shutdown_connections 2024-09-17T04:55:42.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 -- 172.21.15.150:0/2415057020 wait complete. 2024-09-17T04:55:42.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 Processor -- start 2024-09-17T04:55:42.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 -- start start 2024-09-17T04:55:42.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d340c3970 0x7f7d340a7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:42.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d34151d00 0x7f7d3414c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:42.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34157c30 0x7f7d34148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:42.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7d341619b0 con 0x7f7d34157c30 2024-09-17T04:55:42.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7d34161830 con 0x7f7d340c3970 2024-09-17T04:55:42.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d38f33640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7d34161b30 con 0x7f7d34151d00 2024-09-17T04:55:42.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d32d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34157c30 0x7f7d34148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:42.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d32d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34157c30 0x7f7d34148ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34658/0 (socket says 172.21.15.150:34658) 2024-09-17T04:55:42.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.515+0000 7f7d32d9d640 1 -- 172.21.15.150:0/440805724 learned_addr learned my addr 172.21.15.150:0/440805724 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:42.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d3259c640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d340c3970 0x7f7d340a7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:42.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d31d9b640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d34151d00 0x7f7d3414c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:42.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d32d9d640 1 -- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d34151d00 msgr2=0x7f7d3414c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d32d9d640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d34151d00 0x7f7d3414c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d32d9d640 1 -- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d340c3970 msgr2=0x7f7d340a7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d32d9d640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d340c3970 0x7f7d340a7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d32d9d640 1 -- 172.21.15.150:0/440805724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d340a8510 con 0x7f7d34157c30 2024-09-17T04:55:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d3259c640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d340c3970 0x7f7d340a7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:55:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d32d9d640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34157c30 0x7f7d34148ef0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00c3c0 tx=0x7f7d2c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d22ffd640 1 -- 172.21.15.150:0/440805724 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c017070 con 0x7f7d34157c30 2024-09-17T04:55:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d38f33640 1 -- 172.21.15.150:0/440805724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d340a9690 con 0x7f7d34157c30 2024-09-17T04:55:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d22ffd640 1 -- 172.21.15.150:0/440805724 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d2c0053c0 con 0x7f7d34157c30 2024-09-17T04:55:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d22ffd640 1 -- 172.21.15.150:0/440805724 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c003860 con 0x7f7d34157c30 2024-09-17T04:55:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d38f33640 1 -- 172.21.15.150:0/440805724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3414a3a0 con 0x7f7d34157c30 2024-09-17T04:55:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d38f33640 1 -- 172.21.15.150:0/440805724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d340c3d80 con 0x7f7d34157c30 2024-09-17T04:55:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d22ffd640 1 -- 172.21.15.150:0/440805724 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7d2c01a070 con 0x7f7d34157c30 2024-09-17T04:55:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d22ffd640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d14073230 0x7f7d14075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d22ffd640 1 -- 172.21.15.150:0/440805724 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f7d2c093bd0 con 0x7f7d34157c30 2024-09-17T04:55:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.519+0000 7f7d3259c640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d14073230 0x7f7d14075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:42.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.523+0000 7f7d3259c640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d14073230 0x7f7d14075740 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7d240af000 tx=0x7f7d2401b5c0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:42.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.523+0000 7f7d22ffd640 1 -- 172.21.15.150:0/440805724 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7d2c05e5e0 con 0x7f7d34157c30 2024-09-17T04:55:42.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.731+0000 7f7d38f33640 1 -- 172.21.15.150:0/440805724 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d340a7730 con 0x7f7d34157c30 2024-09-17T04:55:42.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.731+0000 7f7d22ffd640 1 -- 172.21.15.150:0/440805724 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+5708 (secure 0 0 0) 0x7f7d2c05dee0 con 0x7f7d34157c30 2024-09-17T04:55:42.734 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:42.734 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":48,"btime":"2024-09-17T04:55:41:882347+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:41.169113+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:42.734 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 48 2024-09-17T04:55:42.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.735+0000 7f7d38f33640 1 -- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d14073230 msgr2=0x7f7d14075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:42.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.735+0000 7f7d38f33640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d14073230 0x7f7d14075740 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7d240af000 tx=0x7f7d2401b5c0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.735+0000 7f7d38f33640 1 -- 172.21.15.150:0/440805724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34157c30 msgr2=0x7f7d34148ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:42.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.735+0000 7f7d38f33640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34157c30 0x7f7d34148ef0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00c3c0 tx=0x7f7d2c00c880 comp rx=0 tx=0).stop 2024-09-17T04:55:42.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.735+0000 7f7d38f33640 1 -- 172.21.15.150:0/440805724 shutdown_connections 2024-09-17T04:55:42.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.735+0000 7f7d38f33640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d14073230 0x7f7d14075740 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.735+0000 7f7d38f33640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d34157c30 0x7f7d34148ef0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.735+0000 7f7d38f33640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d34151d00 0x7f7d3414c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.735+0000 7f7d38f33640 1 --2- 172.21.15.150:0/440805724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d340c3970 0x7f7d340a7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:42.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.735+0000 7f7d38f33640 1 -- 172.21.15.150:0/440805724 >> 172.21.15.150:0/440805724 conn(0x7f7d340c4a00 msgr2=0x7f7d3415e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:42.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.735+0000 7f7d38f33640 1 -- 172.21.15.150:0/440805724 shutdown_connections 2024-09-17T04:55:42.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:42.735+0000 7f7d38f33640 1 -- 172.21.15.150:0/440805724 wait complete. 2024-09-17T04:55:42.759 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T04:55:41:882347+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5254": { "addr": "172.21.15.150:6835/3079958649", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 3079958649, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 3079958649, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5254, "incarnation": 47, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:55:41.169113+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5254 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 40, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5107": { "addr": "172.21.15.176:6837/2374314256", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 2374314256, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 2374314256, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5107, "incarnation": 37, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:54:58.881332+0000", "qdb_cluster": [ 5107 ], "qdb_leader": 5107, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5107 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:55:43.982 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T04:55:43.979+0000 7facac1b1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:55:44.760 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:44.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.935+0000 7fa5f5086640 1 -- 172.21.15.150:0/1918693841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5f00c35b0 msgr2=0x7fa5f00c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:44.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.935+0000 7fa5f5086640 1 --2- 172.21.15.150:0/1918693841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5f00c35b0 0x7fa5f00c8a80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8009840 tx=0x7fa5d801c7f0 comp rx=0 tx=0).stop 2024-09-17T04:55:44.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.935+0000 7fa5f5086640 1 -- 172.21.15.150:0/1918693841 shutdown_connections 2024-09-17T04:55:44.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.935+0000 7fa5f5086640 1 --2- 172.21.15.150:0/1918693841 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5f00c8fc0 0x7fa5f00c20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:44.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.935+0000 7fa5f5086640 1 --2- 172.21.15.150:0/1918693841 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5f00c35b0 0x7fa5f00c8a80 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:44.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.935+0000 7fa5f5086640 1 -- 172.21.15.150:0/1918693841 >> 172.21.15.150:0/1918693841 conn(0x7fa5f0074b80 msgr2=0x7fa5f006f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:44.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.935+0000 7fa5f5086640 1 -- 172.21.15.150:0/1918693841 shutdown_connections 2024-09-17T04:55:44.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.935+0000 7fa5f5086640 1 -- 172.21.15.150:0/1918693841 wait complete. 2024-09-17T04:55:44.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.935+0000 7fa5f5086640 1 Processor -- start 2024-09-17T04:55:44.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5f5086640 1 -- start start 2024-09-17T04:55:44.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5f5086640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5f00c35b0 0x7fa5f0149350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:44.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5f5086640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5f00c8fc0 0x7fa5f0144240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:44.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5f5086640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5f01568a0 0x7fa5f014b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:44.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5f5086640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f00c6b70 con 0x7fa5f01568a0 2024-09-17T04:55:44.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5f5086640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f00c69f0 con 0x7fa5f00c8fc0 2024-09-17T04:55:44.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5f5086640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa5f00b0fd0 con 0x7fa5f00c35b0 2024-09-17T04:55:44.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5ef59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5f01568a0 0x7fa5f014b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:44.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5ef59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5f01568a0 0x7fa5f014b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34668/0 (socket says 172.21.15.150:34668) 2024-09-17T04:55:44.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5ef59e640 1 -- 172.21.15.150:0/3555405912 learned_addr learned my addr 172.21.15.150:0/3555405912 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:44.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5e659c640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5f00c8fc0 0x7fa5f0144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:44.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5eed9d640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5f00c35b0 0x7fa5f0149350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:44.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5ef59e640 1 -- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5f00c35b0 msgr2=0x7fa5f0149350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:44.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5ef59e640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5f00c35b0 0x7fa5f0149350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:44.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5ef59e640 1 -- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5f00c8fc0 msgr2=0x7fa5f0144240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:44.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5ef59e640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5f00c8fc0 0x7fa5f0144240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:44.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5ef59e640 1 -- 172.21.15.150:0/3555405912 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f0149890 con 0x7fa5f01568a0 2024-09-17T04:55:44.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5ef59e640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5f01568a0 0x7fa5f014b170 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa5e800cad0 tx=0x7fa5e800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:44.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5e7fff640 1 -- 172.21.15.150:0/3555405912 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e800adc0 con 0x7fa5f01568a0 2024-09-17T04:55:44.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5e7fff640 1 -- 172.21.15.150:0/3555405912 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5e8003a70 con 0x7fa5f01568a0 2024-09-17T04:55:44.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5e7fff640 1 -- 172.21.15.150:0/3555405912 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8004030 con 0x7fa5f01568a0 2024-09-17T04:55:44.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5f5086640 1 -- 172.21.15.150:0/3555405912 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f0144780 con 0x7fa5f01568a0 2024-09-17T04:55:44.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.939+0000 7fa5f5086640 1 -- 172.21.15.150:0/3555405912 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f0148b50 con 0x7fa5f01568a0 2024-09-17T04:55:44.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.943+0000 7fa5f5086640 1 -- 172.21.15.150:0/3555405912 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5f00b14a0 con 0x7fa5f01568a0 2024-09-17T04:55:44.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.943+0000 7fa5e7fff640 1 -- 172.21.15.150:0/3555405912 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa5e80041d0 con 0x7fa5f01568a0 2024-09-17T04:55:44.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.943+0000 7fa5e7fff640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa5d0073230 0x7fa5d0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:44.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.943+0000 7fa5e7fff640 1 -- 172.21.15.150:0/3555405912 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7fa5e8094f00 con 0x7fa5f01568a0 2024-09-17T04:55:44.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.943+0000 7fa5eed9d640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa5d0073230 0x7fa5d0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:44.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.943+0000 7fa5eed9d640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa5d0073230 0x7fa5d0075740 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa5e00015f0 tx=0x7fa5e0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:44.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:44.951+0000 7fa5e7fff640 1 -- 172.21.15.150:0/3555405912 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa5e8014070 con 0x7fa5f01568a0 2024-09-17T04:55:45.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.219+0000 7fa5f5086640 1 -- 172.21.15.150:0/3555405912 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5f0150c90 con 0x7fa5f01568a0 2024-09-17T04:55:45.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.219+0000 7fa5e7fff640 1 -- 172.21.15.150:0/3555405912 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5712 (secure 0 0 0) 0x7fa5e805fa20 con 0x7fa5f01568a0 2024-09-17T04:55:45.223 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:45.223 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":50,"btime":"2024-09-17T04:55:43:893606+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:45.223 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 50 2024-09-17T04:55:45.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.223+0000 7fa5f5086640 1 -- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa5d0073230 msgr2=0x7fa5d0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:45.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.223+0000 7fa5f5086640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa5d0073230 0x7fa5d0075740 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa5e00015f0 tx=0x7fa5e0009040 comp rx=0 tx=0).stop 2024-09-17T04:55:45.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.223+0000 7fa5f5086640 1 -- 172.21.15.150:0/3555405912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5f01568a0 msgr2=0x7fa5f014b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:45.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.223+0000 7fa5f5086640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5f01568a0 0x7fa5f014b170 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa5e800cad0 tx=0x7fa5e800cf90 comp rx=0 tx=0).stop 2024-09-17T04:55:45.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.223+0000 7fa5f5086640 1 -- 172.21.15.150:0/3555405912 shutdown_connections 2024-09-17T04:55:45.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.223+0000 7fa5f5086640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa5d0073230 0x7fa5d0075740 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:45.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.223+0000 7fa5f5086640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5f00c8fc0 0x7fa5f0144240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:45.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.223+0000 7fa5f5086640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5f01568a0 0x7fa5f014b170 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:45.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.223+0000 7fa5f5086640 1 --2- 172.21.15.150:0/3555405912 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5f00c35b0 0x7fa5f0149350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:45.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.223+0000 7fa5f5086640 1 -- 172.21.15.150:0/3555405912 >> 172.21.15.150:0/3555405912 conn(0x7fa5f0074b80 msgr2=0x7fa5f00c7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:45.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.223+0000 7fa5f5086640 1 -- 172.21.15.150:0/3555405912 shutdown_connections 2024-09-17T04:55:45.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:45.223+0000 7fa5f5086640 1 -- 172.21.15.150:0/3555405912 wait complete. 2024-09-17T04:55:45.251 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=5466 2024-09-17T04:55:45.251 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before reviving mds.a 2024-09-17T04:55:53.252 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-17T04:55:53.253 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:55:53.253 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:55:53.255 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:55:53.256 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:53.372 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T04:55:53.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 -- 172.21.15.150:0/2976176549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f389c153620 msgr2=0x7f389c0aa9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:53.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 --2- 172.21.15.150:0/2976176549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f389c153620 0x7f389c0aa9b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f389c072ea0 tx=0x7f389401ccf0 comp rx=0 tx=0).stop 2024-09-17T04:55:53.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 -- 172.21.15.150:0/2976176549 shutdown_connections 2024-09-17T04:55:53.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 --2- 172.21.15.150:0/2976176549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f389c153620 0x7f389c0aa9b0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:53.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 --2- 172.21.15.150:0/2976176549 >> v2:172.21.15.176:3300/0 conn(0x7f389c0705a0 0x7f389c06ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:53.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 -- 172.21.15.150:0/2976176549 >> 172.21.15.150:0/2976176549 conn(0x7f389c0c5ae0 msgr2=0x7f389c073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:53.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 -- 172.21.15.150:0/2976176549 shutdown_connections 2024-09-17T04:55:53.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 -- 172.21.15.150:0/2976176549 wait complete. 2024-09-17T04:55:53.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 Processor -- start 2024-09-17T04:55:53.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 -- start start 2024-09-17T04:55:53.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f389c06e660 0x7f389c1f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:53.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f389c0705a0 0x7f389c0a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:53.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f389c153620 0x7f389c1ede30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:53.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f389c160b50 con 0x7f389c06e660 2024-09-17T04:55:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f389c1609d0 con 0x7f389c0705a0 2024-09-17T04:55:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a25de640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f389c160cd0 con 0x7f389c153620 2024-09-17T04:55:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f389bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f389c06e660 0x7f389c1f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f389bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f389c06e660 0x7f389c1f1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36208/0 (socket says 172.21.15.150:36208) 2024-09-17T04:55:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f389bfff640 1 -- 172.21.15.150:0/4042352917 learned_addr learned my addr 172.21.15.150:0/4042352917 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f38a0b7b640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f389c153620 0x7f389c1ede30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.427+0000 7f389b7fe640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f389c0705a0 0x7f389c0a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f389bfff640 1 -- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f389c153620 msgr2=0x7f389c1ede30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f389bfff640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f389c153620 0x7f389c1ede30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f389bfff640 1 -- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f389c0705a0 msgr2=0x7f389c0a9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f389bfff640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f389c0705a0 0x7f389c0a9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:53.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f389bfff640 1 -- 172.21.15.150:0/4042352917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389c15a130 con 0x7f389c06e660 2024-09-17T04:55:53.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f389b7fe640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f389c0705a0 0x7f389c0a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:55:53.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f389bfff640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f389c06e660 0x7f389c1f1c30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f388c00c3f0 tx=0x7f388c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:53.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f3898ff9640 1 -- 172.21.15.150:0/4042352917 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388c017070 con 0x7f389c06e660 2024-09-17T04:55:53.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f3898ff9640 1 -- 172.21.15.150:0/4042352917 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f388c0053c0 con 0x7f389c06e660 2024-09-17T04:55:53.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f38a25de640 1 -- 172.21.15.150:0/4042352917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f389c1f2170 con 0x7f389c06e660 2024-09-17T04:55:53.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f3898ff9640 1 -- 172.21.15.150:0/4042352917 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388c003860 con 0x7f389c06e660 2024-09-17T04:55:53.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f38a25de640 1 -- 172.21.15.150:0/4042352917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f389c15a3a0 con 0x7f389c06e660 2024-09-17T04:55:53.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f38a25de640 1 -- 172.21.15.150:0/4042352917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f389c06ff80 con 0x7f389c06e660 2024-09-17T04:55:53.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.431+0000 7f3898ff9640 1 -- 172.21.15.150:0/4042352917 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f388c01a070 con 0x7f389c06e660 2024-09-17T04:55:53.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.435+0000 7f3898ff9640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3888073230 0x7f3888075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:53.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.435+0000 7f3898ff9640 1 -- 172.21.15.150:0/4042352917 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f388c093f30 con 0x7f389c06e660 2024-09-17T04:55:53.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.435+0000 7f389b7fe640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3888073230 0x7f3888075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:53.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.435+0000 7f389b7fe640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3888073230 0x7f3888075740 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3890005fd0 tx=0x7f3890005de0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.439+0000 7f3898ff9640 1 -- 172.21.15.150:0/4042352917 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f388c021020 con 0x7f389c06e660 2024-09-17T04:55:53.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.687+0000 7f38a25de640 1 -- 172.21.15.150:0/4042352917 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f389c0ac270 con 0x7f389c06e660 2024-09-17T04:55:53.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.687+0000 7f3898ff9640 1 -- 172.21.15.150:0/4042352917 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5712 (secure 0 0 0) 0x7f388c05ea10 con 0x7f389c06e660 2024-09-17T04:55:53.689 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:53.689 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":50,"btime":"2024-09-17T04:55:43:893606+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:53.690 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 50 2024-09-17T04:55:53.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.691+0000 7f38a25de640 1 -- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3888073230 msgr2=0x7f3888075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:53.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.691+0000 7f38a25de640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3888073230 0x7f3888075740 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3890005fd0 tx=0x7f3890005de0 comp rx=0 tx=0).stop 2024-09-17T04:55:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.691+0000 7f38a25de640 1 -- 172.21.15.150:0/4042352917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f389c06e660 msgr2=0x7f389c1f1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.691+0000 7f38a25de640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f389c06e660 0x7f389c1f1c30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f388c00c3f0 tx=0x7f388c00c8b0 comp rx=0 tx=0).stop 2024-09-17T04:55:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.691+0000 7f38a25de640 1 -- 172.21.15.150:0/4042352917 shutdown_connections 2024-09-17T04:55:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.691+0000 7f38a25de640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3888073230 0x7f3888075740 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.691+0000 7f38a25de640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f389c0705a0 0x7f389c0a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.691+0000 7f38a25de640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f389c153620 0x7f389c1ede30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.691+0000 7f38a25de640 1 --2- 172.21.15.150:0/4042352917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f389c06e660 0x7f389c1f1c30 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.691+0000 7f38a25de640 1 -- 172.21.15.150:0/4042352917 >> 172.21.15.150:0/4042352917 conn(0x7f389c0c5ae0 msgr2=0x7f389c068f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.691+0000 7f38a25de640 1 -- 172.21.15.150:0/4042352917 shutdown_connections 2024-09-17T04:55:53.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:53.691+0000 7f38a25de640 1 -- 172.21.15.150:0/4042352917 wait complete. 2024-09-17T04:55:53.713 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-17T04:55:54.987 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:55.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.147+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/1610345458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1c80732b0 msgr2=0x7fe1c80c83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.147+0000 7fe1cdf3b640 1 --2- 172.21.15.150:0/1610345458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1c80732b0 0x7fe1c80c83e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0009bf0 tx=0x7fe1c001cb10 comp rx=0 tx=0).stop 2024-09-17T04:55:55.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.147+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/1610345458 shutdown_connections 2024-09-17T04:55:55.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.147+0000 7fe1cdf3b640 1 --2- 172.21.15.150:0/1610345458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1c80732b0 0x7fe1c80c83e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.147+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/1610345458 >> 172.21.15.150:0/1610345458 conn(0x7fe1c80abb00 msgr2=0x7fe1c80abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:55.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.151+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/1610345458 shutdown_connections 2024-09-17T04:55:55.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.151+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/1610345458 wait complete. 2024-09-17T04:55:55.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.151+0000 7fe1cdf3b640 1 Processor -- start 2024-09-17T04:55:55.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.151+0000 7fe1cdf3b640 1 -- start start 2024-09-17T04:55:55.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1cdf3b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1c8072ea0 0x7fe1c81f3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:55.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1cdf3b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1c80732b0 0x7fe1c81f08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1cdf3b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1c80c97a0 0x7fe1c80c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1cdf3b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c80c8920 con 0x7fe1c80c97a0 2024-09-17T04:55:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1cdf3b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe1c80c4df0 con 0x7fe1c80732b0 2024-09-17T04:55:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1cdf3b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe1c80c8aa0 con 0x7fe1c8072ea0 2024-09-17T04:55:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1c80c97a0 0x7fe1c80c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1c80c97a0 0x7fe1c80c3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36226/0 (socket says 172.21.15.150:36226) 2024-09-17T04:55:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c7fff640 1 -- 172.21.15.150:0/747670258 learned_addr learned my addr 172.21.15.150:0/747670258 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1bffff640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1c80732b0 0x7fe1c81f08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c77fe640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1c8072ea0 0x7fe1c81f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c7fff640 1 -- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1c8072ea0 msgr2=0x7fe1c81f3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c7fff640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1c8072ea0 0x7fe1c81f3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c7fff640 1 -- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1c80732b0 msgr2=0x7fe1c81f08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c7fff640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1c80732b0 0x7fe1c81f08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c7fff640 1 -- 172.21.15.150:0/747670258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1c81f0e10 con 0x7fe1c80c97a0 2024-09-17T04:55:55.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c77fe640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1c8072ea0 0x7fe1c81f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:55:55.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c7fff640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1c80c97a0 0x7fe1c80c3350 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0009bf0 tx=0x7fe1c0005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:55.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c4ff9640 1 -- 172.21.15.150:0/747670258 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00ba020 con 0x7fe1c80c97a0 2024-09-17T04:55:55.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c4ff9640 1 -- 172.21.15.150:0/747670258 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1c00043f0 con 0x7fe1c80c97a0 2024-09-17T04:55:55.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/747670258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1c8148a30 con 0x7fe1c80c97a0 2024-09-17T04:55:55.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c4ff9640 1 -- 172.21.15.150:0/747670258 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00b2040 con 0x7fe1c80c97a0 2024-09-17T04:55:55.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/747670258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1c806f060 con 0x7fe1c80c97a0 2024-09-17T04:55:55.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/747670258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1c81efec0 con 0x7fe1c80c97a0 2024-09-17T04:55:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c4ff9640 1 -- 172.21.15.150:0/747670258 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe1c00bd020 con 0x7fe1c80c97a0 2024-09-17T04:55:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.155+0000 7fe1c4ff9640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe1a8073200 0x7fe1a8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.159+0000 7fe1c4ff9640 1 -- 172.21.15.150:0/747670258 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7fe1c012f110 con 0x7fe1c80c97a0 2024-09-17T04:55:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.159+0000 7fe1c77fe640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe1a8073200 0x7fe1a8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.159+0000 7fe1c77fe640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe1a8073200 0x7fe1a8075710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8000f70 tx=0x7fe1b8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:55.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.163+0000 7fe1c4ff9640 1 -- 172.21.15.150:0/747670258 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe1c00c4030 con 0x7fe1c80c97a0 2024-09-17T04:55:55.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/747670258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1c8151c70 con 0x7fe1c80c97a0 2024-09-17T04:55:55.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1c4ff9640 1 -- 172.21.15.150:0/747670258 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6537 (secure 0 0 0) 0x7fe1c0016790 con 0x7fe1c80c97a0 2024-09-17T04:55:55.440 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:55.440 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":51,"btime":"2024-09-17T04:55:54:232004+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:55.440 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 51 2024-09-17T04:55:55.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe1a8073200 msgr2=0x7fe1a8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe1a8073200 0x7fe1a8075710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8000f70 tx=0x7fe1b8009040 comp rx=0 tx=0).stop 2024-09-17T04:55:55.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/747670258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1c80c97a0 msgr2=0x7fe1c80c3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1c80c97a0 0x7fe1c80c3350 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0009bf0 tx=0x7fe1c0005ae0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/747670258 shutdown_connections 2024-09-17T04:55:55.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe1a8073200 0x7fe1a8075710 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1c80732b0 0x7fe1c81f08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1c80c97a0 0x7fe1c80c3350 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 --2- 172.21.15.150:0/747670258 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1c8072ea0 0x7fe1c81f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/747670258 >> 172.21.15.150:0/747670258 conn(0x7fe1c80abb00 msgr2=0x7fe1c80a9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:55.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/747670258 shutdown_connections 2024-09-17T04:55:55.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.439+0000 7fe1cdf3b640 1 -- 172.21.15.150:0/747670258 wait complete. 2024-09-17T04:55:55.463 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-17T04:55:55.463 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:55:55.463 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:55:55.484 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:55:55.484 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:55.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.643+0000 7f94e1c20640 1 -- 172.21.15.150:0/876144278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94dc1535b0 msgr2=0x7f94dc157440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.643+0000 7f94e1c20640 1 --2- 172.21.15.150:0/876144278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94dc1535b0 0x7f94dc157440 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f94cc009840 tx=0x7f94cc01c790 comp rx=0 tx=0).stop 2024-09-17T04:55:55.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.643+0000 7f94e1c20640 1 -- 172.21.15.150:0/876144278 shutdown_connections 2024-09-17T04:55:55.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.643+0000 7f94e1c20640 1 --2- 172.21.15.150:0/876144278 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94dc0c5b70 0x7f94dc149b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.643+0000 7f94e1c20640 1 --2- 172.21.15.150:0/876144278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94dc1535b0 0x7f94dc157440 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.643+0000 7f94e1c20640 1 -- 172.21.15.150:0/876144278 >> 172.21.15.150:0/876144278 conn(0x7f94dc0aaa80 msgr2=0x7f94dc0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:55.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.643+0000 7f94e1c20640 1 -- 172.21.15.150:0/876144278 shutdown_connections 2024-09-17T04:55:55.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.643+0000 7f94e1c20640 1 -- 172.21.15.150:0/876144278 wait complete. 2024-09-17T04:55:55.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94e1c20640 1 Processor -- start 2024-09-17T04:55:55.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94e1c20640 1 -- start start 2024-09-17T04:55:55.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94e1c20640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94dc0c5760 0x7f94dc0c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:55.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94e1c20640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94dc0c5b70 0x7f94dc1f10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:55.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94e1c20640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94dc1535b0 0x7f94dc145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:55.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94e1c20640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc0c5370 con 0x7f94dc0c5b70 2024-09-17T04:55:55.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94e1c20640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc0718d0 con 0x7f94dc0c5760 2024-09-17T04:55:55.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94e1c20640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f94dc0c54f0 con 0x7f94dc1535b0 2024-09-17T04:55:55.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94daffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94dc0c5b70 0x7f94dc1f10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:55.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94daffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94dc0c5b70 0x7f94dc1f10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36246/0 (socket says 172.21.15.150:36246) 2024-09-17T04:55:55.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94daffd640 1 -- 172.21.15.150:0/2005519584 learned_addr learned my addr 172.21.15.150:0/2005519584 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:55.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94dbfff640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94dc1535b0 0x7f94dc145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:55.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94db7fe640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94dc0c5760 0x7f94dc0c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:55.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94daffd640 1 -- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94dc1535b0 msgr2=0x7f94dc145020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94daffd640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94dc1535b0 0x7f94dc145020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94daffd640 1 -- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94dc0c5760 msgr2=0x7f94dc0c22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94daffd640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94dc0c5760 0x7f94dc0c22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94daffd640 1 -- 172.21.15.150:0/2005519584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94dc1f0850 con 0x7f94dc0c5b70 2024-09-17T04:55:55.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94db7fe640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94dc0c5760 0x7f94dc0c22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:55:55.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.647+0000 7f94dbfff640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94dc1535b0 0x7f94dc145020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:55:55.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.651+0000 7f94daffd640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94dc0c5b70 0x7f94dc1f10b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f94d000c430 tx=0x7f94d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:55.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.651+0000 7f94c3fff640 1 -- 172.21.15.150:0/2005519584 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d0011790 con 0x7f94dc0c5b70 2024-09-17T04:55:55.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.651+0000 7f94c3fff640 1 -- 172.21.15.150:0/2005519584 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94d0017070 con 0x7f94dc0c5b70 2024-09-17T04:55:55.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.651+0000 7f94e1c20640 1 -- 172.21.15.150:0/2005519584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94dc0abca0 con 0x7f94dc0c5b70 2024-09-17T04:55:55.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.651+0000 7f94c3fff640 1 -- 172.21.15.150:0/2005519584 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d0011ce0 con 0x7f94dc0c5b70 2024-09-17T04:55:55.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.651+0000 7f94e1c20640 1 -- 172.21.15.150:0/2005519584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94dc1ed770 con 0x7f94dc0c5b70 2024-09-17T04:55:55.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.651+0000 7f94e1c20640 1 -- 172.21.15.150:0/2005519584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94bc0052a0 con 0x7f94dc0c5b70 2024-09-17T04:55:55.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.651+0000 7f94c3fff640 1 -- 172.21.15.150:0/2005519584 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f94d001c460 con 0x7f94dc0c5b70 2024-09-17T04:55:55.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.651+0000 7f94c3fff640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f94c8073200 0x7f94c8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:55.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.651+0000 7f94c3fff640 1 -- 172.21.15.150:0/2005519584 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f94d0094480 con 0x7f94dc0c5b70 2024-09-17T04:55:55.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.651+0000 7f94db7fe640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f94c8073200 0x7f94c8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:55.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.655+0000 7f94db7fe640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f94c8073200 0x7f94c8075710 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0ae000 tx=0x7f94cc01afd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:55.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.655+0000 7f94c3fff640 1 -- 172.21.15.150:0/2005519584 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f94d005ef40 con 0x7f94dc0c5b70 2024-09-17T04:55:55.714 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:55.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.871+0000 7fe22cb34640 1 -- 172.21.15.150:0/1820526404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2280c8870 msgr2=0x7fe2280c7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.871+0000 7fe22cb34640 1 --2- 172.21.15.150:0/1820526404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2280c8870 0x7fe2280c7190 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe2280a8a60 tx=0x7fe22001c8e0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.871+0000 7fe22cb34640 1 -- 172.21.15.150:0/1820526404 shutdown_connections 2024-09-17T04:55:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.871+0000 7fe22cb34640 1 --2- 172.21.15.150:0/1820526404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2280c8870 0x7fe2280c7190 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.871+0000 7fe22cb34640 1 --2- 172.21.15.150:0/1820526404 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2281540b0 0x7fe2280c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.871+0000 7fe22cb34640 1 -- 172.21.15.150:0/1820526404 >> 172.21.15.150:0/1820526404 conn(0x7fe2280aab60 msgr2=0x7fe2280aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.871+0000 7fe22cb34640 1 -- 172.21.15.150:0/1820526404 shutdown_connections 2024-09-17T04:55:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.871+0000 7fe22cb34640 1 -- 172.21.15.150:0/1820526404 wait complete. 2024-09-17T04:55:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22cb34640 1 Processor -- start 2024-09-17T04:55:55.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22cb34640 1 -- start start 2024-09-17T04:55:55.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22cb34640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2280c7eb0 0x7fe22814e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:55.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22cb34640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2280c8870 0x7fe2280be390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:55.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22cb34640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2281540b0 0x7fe2281f13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:55.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22cb34640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe2280c7ae0 con 0x7fe2280c7eb0 2024-09-17T04:55:55.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22cb34640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe2280c7c60 con 0x7fe2280c8870 2024-09-17T04:55:55.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22cb34640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe228156540 con 0x7fe2281540b0 2024-09-17T04:55:55.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2280c7eb0 0x7fe22814e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:55.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2280c7eb0 0x7fe22814e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36266/0 (socket says 172.21.15.150:36266) 2024-09-17T04:55:55.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22659c640 1 -- 172.21.15.150:0/3848654184 learned_addr learned my addr 172.21.15.150:0/3848654184 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:55.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe226d9d640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2281540b0 0x7fe2281f13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:55.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe225d9b640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2280c8870 0x7fe2280be390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:55.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22659c640 1 -- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2281540b0 msgr2=0x7fe2281f13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22659c640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2281540b0 0x7fe2281f13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22659c640 1 -- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2280c8870 msgr2=0x7fe2280be390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22659c640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2280c8870 0x7fe2280be390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22659c640 1 -- 172.21.15.150:0/3848654184 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2281f1ca0 con 0x7fe2280c7eb0 2024-09-17T04:55:55.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22659c640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2280c7eb0 0x7fe22814e0d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe21800ba80 tx=0x7fe21800bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:55.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe216ffd640 1 -- 172.21.15.150:0/3848654184 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218019070 con 0x7fe2280c7eb0 2024-09-17T04:55:55.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe216ffd640 1 -- 172.21.15.150:0/3848654184 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe21800cd80 con 0x7fe2280c7eb0 2024-09-17T04:55:55.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22cb34640 1 -- 172.21.15.150:0/3848654184 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe228148bd0 con 0x7fe2280c7eb0 2024-09-17T04:55:55.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe22cb34640 1 -- 172.21.15.150:0/3848654184 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe22814e6a0 con 0x7fe2280c7eb0 2024-09-17T04:55:55.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.875+0000 7fe216ffd640 1 -- 172.21.15.150:0/3848654184 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2180146a0 con 0x7fe2280c7eb0 2024-09-17T04:55:55.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.879+0000 7fe22cb34640 1 -- 172.21.15.150:0/3848654184 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe20c0052a0 con 0x7fe2280c7eb0 2024-09-17T04:55:55.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.879+0000 7fe216ffd640 1 -- 172.21.15.150:0/3848654184 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe21801c070 con 0x7fe2280c7eb0 2024-09-17T04:55:55.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.879+0000 7fe216ffd640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe208077610 0x7fe208079b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:55.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.879+0000 7fe216ffd640 1 -- 172.21.15.150:0/3848654184 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7fe2180948d0 con 0x7fe2280c7eb0 2024-09-17T04:55:55.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.879+0000 7fe225d9b640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe208077610 0x7fe208079b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:55.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.879+0000 7fe225d9b640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe208077610 0x7fe208079b20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0058d0 tx=0x7fe21c005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:55.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.883+0000 7fe216ffd640 1 -- 172.21.15.150:0/3848654184 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe218023020 con 0x7fe2280c7eb0 2024-09-17T04:55:55.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.931+0000 7f94e1c20640 1 -- 172.21.15.150:0/2005519584 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94bc005860 con 0x7f94dc0c5b70 2024-09-17T04:55:55.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.931+0000 7f94c3fff640 1 -- 172.21.15.150:0/2005519584 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6537 (secure 0 0 0) 0x7f94d005e840 con 0x7f94dc0c5b70 2024-09-17T04:55:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:55.933 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":51,"btime":"2024-09-17T04:55:54:232004+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:55.933 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 51 2024-09-17T04:55:55.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.935+0000 7f94e1c20640 1 -- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f94c8073200 msgr2=0x7f94c8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.935+0000 7f94e1c20640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f94c8073200 0x7f94c8075710 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0ae000 tx=0x7f94cc01afd0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.935+0000 7f94e1c20640 1 -- 172.21.15.150:0/2005519584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94dc0c5b70 msgr2=0x7f94dc1f10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:55.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.935+0000 7f94e1c20640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94dc0c5b70 0x7f94dc1f10b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f94d000c430 tx=0x7f94d000c8f0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.935+0000 7f94e1c20640 1 -- 172.21.15.150:0/2005519584 shutdown_connections 2024-09-17T04:55:55.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.935+0000 7f94e1c20640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f94c8073200 0x7f94c8075710 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.935+0000 7f94e1c20640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94dc1535b0 0x7f94dc145020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.935+0000 7f94e1c20640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94dc0c5b70 0x7f94dc1f10b0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.935+0000 7f94e1c20640 1 --2- 172.21.15.150:0/2005519584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94dc0c5760 0x7f94dc0c22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:55.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.935+0000 7f94e1c20640 1 -- 172.21.15.150:0/2005519584 >> 172.21.15.150:0/2005519584 conn(0x7f94dc0aaa80 msgr2=0x7f94dc0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:55.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.935+0000 7f94e1c20640 1 -- 172.21.15.150:0/2005519584 shutdown_connections 2024-09-17T04:55:55.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:55.935+0000 7f94e1c20640 1 -- 172.21.15.150:0/2005519584 wait complete. 2024-09-17T04:55:55.961 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:55:56.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe22cb34640 1 -- 172.21.15.150:0/3848654184 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe20c005590 con 0x7fe2280c7eb0 2024-09-17T04:55:56.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe216ffd640 1 -- 172.21.15.150:0/3848654184 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6537 (secure 0 0 0) 0x7fe21805f440 con 0x7fe2280c7eb0 2024-09-17T04:55:56.140 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:56.140 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":51,"btime":"2024-09-17T04:55:54:232004+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:56.141 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 51 2024-09-17T04:55:56.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe22cb34640 1 -- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe208077610 msgr2=0x7fe208079b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:56.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe22cb34640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe208077610 0x7fe208079b20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0058d0 tx=0x7fe21c005860 comp rx=0 tx=0).stop 2024-09-17T04:55:56.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe22cb34640 1 -- 172.21.15.150:0/3848654184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2280c7eb0 msgr2=0x7fe22814e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:56.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe22cb34640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2280c7eb0 0x7fe22814e0d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe21800ba80 tx=0x7fe21800bf40 comp rx=0 tx=0).stop 2024-09-17T04:55:56.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe22cb34640 1 -- 172.21.15.150:0/3848654184 shutdown_connections 2024-09-17T04:55:56.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe22cb34640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe208077610 0x7fe208079b20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:56.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe22cb34640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2280c8870 0x7fe2280be390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:56.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe22cb34640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2281540b0 0x7fe2281f13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:56.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe22cb34640 1 --2- 172.21.15.150:0/3848654184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2280c7eb0 0x7fe22814e0d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:56.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe22cb34640 1 -- 172.21.15.150:0/3848654184 >> 172.21.15.150:0/3848654184 conn(0x7fe2280aab60 msgr2=0x7fe228152e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:56.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.139+0000 7fe22cb34640 1 -- 172.21.15.150:0/3848654184 shutdown_connections 2024-09-17T04:55:56.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:56.143+0000 7fe22cb34640 1 -- 172.21.15.150:0/3848654184 wait complete. 2024-09-17T04:55:56.161 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-17T04:55:56.161 INFO:tasks.mds_thrash.fs.[a]:waiting for 109 secs before thrashing 2024-09-17T04:55:57.963 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:55:58.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.135+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/4111485105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c4250 msgr2=0x7f8cb4153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:58.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.135+0000 7f8cb9dfd640 1 --2- 172.21.15.150:0/4111485105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c4250 0x7f8cb4153ee0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009900 tx=0x7f8ca401c8a0 comp rx=0 tx=0).stop 2024-09-17T04:55:58.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.135+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/4111485105 shutdown_connections 2024-09-17T04:55:58.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.135+0000 7f8cb9dfd640 1 --2- 172.21.15.150:0/4111485105 >> v2:172.21.15.176:3300/0 conn(0x7f8cb40c3d10 0x7f8cb415df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:58.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.135+0000 7f8cb9dfd640 1 --2- 172.21.15.150:0/4111485105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c4250 0x7f8cb4153ee0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:58.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.135+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/4111485105 >> 172.21.15.150:0/4111485105 conn(0x7f8cb406e360 msgr2=0x7f8cb406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:58.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.135+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/4111485105 shutdown_connections 2024-09-17T04:55:58.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.135+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/4111485105 wait complete. 2024-09-17T04:55:58.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.135+0000 7f8cb9dfd640 1 Processor -- start 2024-09-17T04:55:58.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.135+0000 7f8cb9dfd640 1 -- start start 2024-09-17T04:55:58.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb9dfd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c3900 0x7f8cb40acf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:58.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb9dfd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb40c3d10 0x7f8cb40c2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb9dfd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb40c4250 0x7f8cb414ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb9dfd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4071120 con 0x7f8cb40c3900 2024-09-17T04:55:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb9dfd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb4070fa0 con 0x7f8cb40c3d10 2024-09-17T04:55:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb9dfd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8cb4072db0 con 0x7f8cb40c4250 2024-09-17T04:55:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c3900 0x7f8cb40acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c3900 0x7f8cb40acf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36276/0 (socket says 172.21.15.150:36276) 2024-09-17T04:55:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb37fe640 1 -- 172.21.15.150:0/851136145 learned_addr learned my addr 172.21.15.150:0/851136145 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:55:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cabfff640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb40c3d10 0x7f8cb40c2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb3fff640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb40c4250 0x7f8cb414ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb37fe640 1 -- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb40c4250 msgr2=0x7f8cb414ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb37fe640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb40c4250 0x7f8cb414ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb37fe640 1 -- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb40c3d10 msgr2=0x7f8cb40c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb37fe640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb40c3d10 0x7f8cb40c2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb37fe640 1 -- 172.21.15.150:0/851136145 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb41edb60 con 0x7f8cb40c3900 2024-09-17T04:55:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cabfff640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb40c3d10 0x7f8cb40c2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:55:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb37fe640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c3900 0x7f8cb40acf00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8ca401af50 tx=0x7f8ca40a55a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb0ff9640 1 -- 172.21.15.150:0/851136145 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca40b2040 con 0x7f8cb40c3900 2024-09-17T04:55:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb0ff9640 1 -- 172.21.15.150:0/851136145 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ca40043b0 con 0x7f8cb40c3900 2024-09-17T04:55:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/851136145 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb40be250 con 0x7f8cb40c3900 2024-09-17T04:55:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb0ff9640 1 -- 172.21.15.150:0/851136145 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4003860 con 0x7f8cb40c3900 2024-09-17T04:55:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/851136145 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb40ad6e0 con 0x7f8cb40c3900 2024-09-17T04:55:58.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/851136145 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cb41edd30 con 0x7f8cb40c3900 2024-09-17T04:55:58.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb0ff9640 1 -- 172.21.15.150:0/851136145 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8ca40bd060 con 0x7f8cb40c3900 2024-09-17T04:55:58.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb0ff9640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c94073230 0x7f8c94075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:55:58.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.139+0000 7f8cb0ff9640 1 -- 172.21.15.150:0/851136145 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f8ca412dd70 con 0x7f8cb40c3900 2024-09-17T04:55:58.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.143+0000 7f8cabfff640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c94073230 0x7f8c94075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:55:58.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.143+0000 7f8cabfff640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c94073230 0x7f8c94075740 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8cb40c3350 tx=0x7f8c9c008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:55:58.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.147+0000 7f8cb0ff9640 1 -- 172.21.15.150:0/851136145 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8ca4016490 con 0x7f8cb40c3900 2024-09-17T04:55:58.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/851136145 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cb4153ee0 con 0x7f8cb40c3900 2024-09-17T04:55:58.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb0ff9640 1 -- 172.21.15.150:0/851136145 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6537 (secure 0 0 0) 0x7f8ca40f8810 con 0x7f8cb40c3900 2024-09-17T04:55:58.392 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:55:58.392 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":51,"btime":"2024-09-17T04:55:54:232004+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:55:58.392 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 51 2024-09-17T04:55:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c94073230 msgr2=0x7f8c94075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c94073230 0x7f8c94075740 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8cb40c3350 tx=0x7f8c9c008040 comp rx=0 tx=0).stop 2024-09-17T04:55:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/851136145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c3900 msgr2=0x7f8cb40acf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:55:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c3900 0x7f8cb40acf00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8ca401af50 tx=0x7f8ca40a55a0 comp rx=0 tx=0).stop 2024-09-17T04:55:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/851136145 shutdown_connections 2024-09-17T04:55:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c94073230 0x7f8c94075740 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb40c3d10 0x7f8cb40c2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb40c4250 0x7f8cb414ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 --2- 172.21.15.150:0/851136145 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40c3900 0x7f8cb40acf00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:55:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/851136145 >> 172.21.15.150:0/851136145 conn(0x7f8cb406e360 msgr2=0x7f8cb4072340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:55:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/851136145 shutdown_connections 2024-09-17T04:55:58.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:55:58.391+0000 7f8cb9dfd640 1 -- 172.21.15.150:0/851136145 wait complete. 2024-09-17T04:55:58.416 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:56:00.068 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:56:00.071 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:56:00.417 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:56:00.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.603+0000 7f77c42fc640 1 -- 172.21.15.150:0/3135005557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc0c4250 msgr2=0x7f77bc153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:00.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.603+0000 7f77c42fc640 1 --2- 172.21.15.150:0/3135005557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc0c4250 0x7f77bc153ee0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f77bc071d50 tx=0x7f77ac01c8c0 comp rx=0 tx=0).stop 2024-09-17T04:56:00.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.603+0000 7f77c42fc640 1 -- 172.21.15.150:0/3135005557 shutdown_connections 2024-09-17T04:56:00.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.603+0000 7f77c42fc640 1 --2- 172.21.15.150:0/3135005557 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77bc074e60 0x7f77bc156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:00.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.603+0000 7f77c42fc640 1 --2- 172.21.15.150:0/3135005557 >> v2:172.21.15.176:3300/0 conn(0x7f77bc0c3900 0x7f77bc074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:00.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.603+0000 7f77c42fc640 1 --2- 172.21.15.150:0/3135005557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc0c4250 0x7f77bc153ee0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:00.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.603+0000 7f77c42fc640 1 -- 172.21.15.150:0/3135005557 >> 172.21.15.150:0/3135005557 conn(0x7f77bc06e360 msgr2=0x7f77bc06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:00.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.603+0000 7f77c42fc640 1 -- 172.21.15.150:0/3135005557 shutdown_connections 2024-09-17T04:56:00.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.603+0000 7f77c42fc640 1 -- 172.21.15.150:0/3135005557 wait complete. 2024-09-17T04:56:00.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c42fc640 1 Processor -- start 2024-09-17T04:56:00.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c42fc640 1 -- start start 2024-09-17T04:56:00.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c42fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc074e60 0x7f77bc15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:00.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c42fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc0c3900 0x7f77bc14d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:00.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c42fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77bc0c4250 0x7f77bc158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:00.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c42fc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f77bc072f30 con 0x7f77bc0c3900 2024-09-17T04:56:00.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c42fc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f77bc072db0 con 0x7f77bc074e60 2024-09-17T04:56:00.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c42fc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f77bc15ff80 con 0x7f77bc0c4250 2024-09-17T04:56:00.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c1897640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc0c3900 0x7f77bc14d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:00.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c1897640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc0c3900 0x7f77bc14d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33184/0 (socket says 172.21.15.150:33184) 2024-09-17T04:56:00.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c1897640 1 -- 172.21.15.150:0/3106491164 learned_addr learned my addr 172.21.15.150:0/3106491164 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:56:00.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c2098640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc074e60 0x7f77bc15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:00.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c2899640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77bc0c4250 0x7f77bc158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:00.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c1897640 1 -- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77bc0c4250 msgr2=0x7f77bc158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:00.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c1897640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77bc0c4250 0x7f77bc158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:00.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c1897640 1 -- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc074e60 msgr2=0x7f77bc15bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:00.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c1897640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc074e60 0x7f77bc15bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:00.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c1897640 1 -- 172.21.15.150:0/3106491164 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77bc150930 con 0x7f77bc0c3900 2024-09-17T04:56:00.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c1897640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc0c3900 0x7f77bc14d170 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f77b800c9a0 tx=0x7f77b800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:00.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77b2ffd640 1 -- 172.21.15.150:0/3106491164 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8014070 con 0x7f77bc0c3900 2024-09-17T04:56:00.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77b2ffd640 1 -- 172.21.15.150:0/3106491164 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77b80077a0 con 0x7f77bc0c3900 2024-09-17T04:56:00.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c42fc640 1 -- 172.21.15.150:0/3106491164 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77bc15c2e0 con 0x7f77bc0c3900 2024-09-17T04:56:00.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77b2ffd640 1 -- 172.21.15.150:0/3106491164 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8007bd0 con 0x7f77bc0c3900 2024-09-17T04:56:00.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.607+0000 7f77c42fc640 1 -- 172.21.15.150:0/3106491164 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77bc1ee4b0 con 0x7f77bc0c3900 2024-09-17T04:56:00.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.611+0000 7f77c42fc640 1 -- 172.21.15.150:0/3106491164 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77bc074920 con 0x7f77bc0c3900 2024-09-17T04:56:00.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.611+0000 7f77b2ffd640 1 -- 172.21.15.150:0/3106491164 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f77b8017070 con 0x7f77bc0c3900 2024-09-17T04:56:00.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.611+0000 7f77b2ffd640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f77a0073230 0x7f77a0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:00.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.611+0000 7f77b2ffd640 1 -- 172.21.15.150:0/3106491164 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f77b8094a30 con 0x7f77bc0c3900 2024-09-17T04:56:00.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.611+0000 7f77c2098640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f77a0073230 0x7f77a0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:00.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.611+0000 7f77c2098640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f77a0073230 0x7f77a0075740 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f77ac0af000 tx=0x7f77ac01b100 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:00.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.619+0000 7f77b2ffd640 1 -- 172.21.15.150:0/3106491164 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f77b805f550 con 0x7f77bc0c3900 2024-09-17T04:56:00.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.899+0000 7f77c42fc640 1 -- 172.21.15.150:0/3106491164 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77bc0008e0 con 0x7f77bc0c3900 2024-09-17T04:56:00.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.899+0000 7f77b2ffd640 1 -- 172.21.15.150:0/3106491164 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6537 (secure 0 0 0) 0x7f77b805ee50 con 0x7f77bc0c3900 2024-09-17T04:56:00.902 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:56:00.902 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":51,"btime":"2024-09-17T04:55:54:232004+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:56:00.902 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 51 2024-09-17T04:56:00.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.903+0000 7f77c42fc640 1 -- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f77a0073230 msgr2=0x7f77a0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:00.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.903+0000 7f77c42fc640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f77a0073230 0x7f77a0075740 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f77ac0af000 tx=0x7f77ac01b100 comp rx=0 tx=0).stop 2024-09-17T04:56:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.903+0000 7f77c42fc640 1 -- 172.21.15.150:0/3106491164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc0c3900 msgr2=0x7f77bc14d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.903+0000 7f77c42fc640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc0c3900 0x7f77bc14d170 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f77b800c9a0 tx=0x7f77b800ce60 comp rx=0 tx=0).stop 2024-09-17T04:56:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.903+0000 7f77c42fc640 1 -- 172.21.15.150:0/3106491164 shutdown_connections 2024-09-17T04:56:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.903+0000 7f77c42fc640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f77a0073230 0x7f77a0075740 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.903+0000 7f77c42fc640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77bc0c4250 0x7f77bc158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.903+0000 7f77c42fc640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc0c3900 0x7f77bc14d170 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.903+0000 7f77c42fc640 1 --2- 172.21.15.150:0/3106491164 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc074e60 0x7f77bc15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.903+0000 7f77c42fc640 1 -- 172.21.15.150:0/3106491164 >> 172.21.15.150:0/3106491164 conn(0x7f77bc06e360 msgr2=0x7f77bc0a8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.903+0000 7f77c42fc640 1 -- 172.21.15.150:0/3106491164 shutdown_connections 2024-09-17T04:56:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:00.903+0000 7f77c42fc640 1 -- 172.21.15.150:0/3106491164 wait complete. 2024-09-17T04:56:00.929 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:56:02.930 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:56:03.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.099+0000 7f43dc19b640 1 -- 172.21.15.150:0/426516815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43d413dbe0 msgr2=0x7f43d4158710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:03.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.099+0000 7f43dc19b640 1 --2- 172.21.15.150:0/426516815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43d413dbe0 0x7f43d4158710 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f43d4158c50 tx=0x7f43d001c750 comp rx=0 tx=0).stop 2024-09-17T04:56:03.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.099+0000 7f43dc19b640 1 -- 172.21.15.150:0/426516815 shutdown_connections 2024-09-17T04:56:03.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.099+0000 7f43dc19b640 1 --2- 172.21.15.150:0/426516815 >> v2:172.21.15.176:3300/0 conn(0x7f43d40c6c20 0x7f43d40c47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:03.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.099+0000 7f43dc19b640 1 --2- 172.21.15.150:0/426516815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43d413dbe0 0x7f43d4158710 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:03.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.099+0000 7f43dc19b640 1 -- 172.21.15.150:0/426516815 >> 172.21.15.150:0/426516815 conn(0x7f43d40abb30 msgr2=0x7f43d40abf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:03.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.099+0000 7f43dc19b640 1 -- 172.21.15.150:0/426516815 shutdown_connections 2024-09-17T04:56:03.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.099+0000 7f43dc19b640 1 -- 172.21.15.150:0/426516815 wait complete. 2024-09-17T04:56:03.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43dc19b640 1 Processor -- start 2024-09-17T04:56:03.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43dc19b640 1 -- start start 2024-09-17T04:56:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43dc19b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f43d40c6c20 0x7f43d4156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43dc19b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d413dbe0 0x7f43d40aa350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43dc19b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43d4154710 0x7f43d414f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43dc19b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f43d4071ae0 con 0x7f43d4154710 2024-09-17T04:56:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43dc19b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f43d4071960 con 0x7f43d413dbe0 2024-09-17T04:56:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43dc19b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f43d41618b0 con 0x7f43d40c6c20 2024-09-17T04:56:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43da738640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43d4154710 0x7f43d414f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43da738640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43d4154710 0x7f43d414f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33194/0 (socket says 172.21.15.150:33194) 2024-09-17T04:56:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43da738640 1 -- 172.21.15.150:0/1854612325 learned_addr learned my addr 172.21.15.150:0/1854612325 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:56:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43d9f37640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f43d40c6c20 0x7f43d4156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43d9736640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d413dbe0 0x7f43d40aa350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43da738640 1 -- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f43d40c6c20 msgr2=0x7f43d4156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43da738640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f43d40c6c20 0x7f43d4156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43da738640 1 -- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d413dbe0 msgr2=0x7f43d40aa350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43da738640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d413dbe0 0x7f43d40aa350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43da738640 1 -- 172.21.15.150:0/1854612325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d4153b10 con 0x7f43d4154710 2024-09-17T04:56:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43da738640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43d4154710 0x7f43d414f7e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00e940 tx=0x7f43cc00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43ca7fc640 1 -- 172.21.15.150:0/1854612325 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc004d30 con 0x7f43d4154710 2024-09-17T04:56:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43ca7fc640 1 -- 172.21.15.150:0/1854612325 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43cc00cb20 con 0x7f43d4154710 2024-09-17T04:56:03.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43dc19b640 1 -- 172.21.15.150:0/1854612325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d4151980 con 0x7f43d4154710 2024-09-17T04:56:03.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43ca7fc640 1 -- 172.21.15.150:0/1854612325 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc005020 con 0x7f43d4154710 2024-09-17T04:56:03.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43dc19b640 1 -- 172.21.15.150:0/1854612325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d4151c20 con 0x7f43d4154710 2024-09-17T04:56:03.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.103+0000 7f43dc19b640 1 -- 172.21.15.150:0/1854612325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43d40c47e0 con 0x7f43d4154710 2024-09-17T04:56:03.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.107+0000 7f43ca7fc640 1 -- 172.21.15.150:0/1854612325 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f43cc0051c0 con 0x7f43d4154710 2024-09-17T04:56:03.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.107+0000 7f43ca7fc640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f43b8073230 0x7f43b8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:03.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.107+0000 7f43ca7fc640 1 -- 172.21.15.150:0/1854612325 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f43cc095000 con 0x7f43d4154710 2024-09-17T04:56:03.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.107+0000 7f43d9f37640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f43b8073230 0x7f43b8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:03.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.107+0000 7f43d9f37640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f43b8073230 0x7f43b8075740 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f43d00ae000 tx=0x7f43d001af90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.111+0000 7f43ca7fc640 1 -- 172.21.15.150:0/1854612325 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f43cc016030 con 0x7f43d4154710 2024-09-17T04:56:03.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.370+0000 7f43dc19b640 1 -- 172.21.15.150:0/1854612325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43d40af310 con 0x7f43d4154710 2024-09-17T04:56:03.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.370+0000 7f43ca7fc640 1 -- 172.21.15.150:0/1854612325 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6537 (secure 0 0 0) 0x7f43cc05fb20 con 0x7f43d4154710 2024-09-17T04:56:03.372 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:56:03.372 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":51,"btime":"2024-09-17T04:55:54:232004+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:56:03.372 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 51 2024-09-17T04:56:03.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.370+0000 7f43dc19b640 1 -- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f43b8073230 msgr2=0x7f43b8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:03.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.370+0000 7f43dc19b640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f43b8073230 0x7f43b8075740 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f43d00ae000 tx=0x7f43d001af90 comp rx=0 tx=0).stop 2024-09-17T04:56:03.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.370+0000 7f43dc19b640 1 -- 172.21.15.150:0/1854612325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43d4154710 msgr2=0x7f43d414f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:03.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.370+0000 7f43dc19b640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43d4154710 0x7f43d414f7e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00e940 tx=0x7f43cc00ee00 comp rx=0 tx=0).stop 2024-09-17T04:56:03.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.370+0000 7f43dc19b640 1 -- 172.21.15.150:0/1854612325 shutdown_connections 2024-09-17T04:56:03.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.370+0000 7f43dc19b640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f43b8073230 0x7f43b8075740 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:03.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.370+0000 7f43dc19b640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43d413dbe0 0x7f43d40aa350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:03.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.370+0000 7f43dc19b640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43d4154710 0x7f43d414f7e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:03.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.370+0000 7f43dc19b640 1 --2- 172.21.15.150:0/1854612325 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f43d40c6c20 0x7f43d4156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:03.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.370+0000 7f43dc19b640 1 -- 172.21.15.150:0/1854612325 >> 172.21.15.150:0/1854612325 conn(0x7f43d40abb30 msgr2=0x7f43d4072ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:03.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.374+0000 7f43dc19b640 1 -- 172.21.15.150:0/1854612325 shutdown_connections 2024-09-17T04:56:03.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:03.374+0000 7f43dc19b640 1 -- 172.21.15.150:0/1854612325 wait complete. 2024-09-17T04:56:03.396 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:56:05.439 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:56:05.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.602+0000 7ff2cbf38640 1 -- 172.21.15.150:0/133342690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2c40c3900 msgr2=0x7ff2c4074920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:05.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.602+0000 7ff2cbf38640 1 --2- 172.21.15.150:0/133342690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2c40c3900 0x7ff2c4074920 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009870 tx=0x7ff2b401c820 comp rx=0 tx=0).stop 2024-09-17T04:56:05.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.602+0000 7ff2cbf38640 1 -- 172.21.15.150:0/133342690 shutdown_connections 2024-09-17T04:56:05.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.602+0000 7ff2cbf38640 1 --2- 172.21.15.150:0/133342690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2c40c3900 0x7ff2c4074920 secure :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009870 tx=0x7ff2b401c820 comp rx=0 tx=0).stop 2024-09-17T04:56:05.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.602+0000 7ff2cbf38640 1 -- 172.21.15.150:0/133342690 >> 172.21.15.150:0/133342690 conn(0x7ff2c406e360 msgr2=0x7ff2c406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:05.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.602+0000 7ff2cbf38640 1 -- 172.21.15.150:0/133342690 shutdown_connections 2024-09-17T04:56:05.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.602+0000 7ff2cbf38640 1 -- 172.21.15.150:0/133342690 wait complete. 2024-09-17T04:56:05.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.606+0000 7ff2cbf38640 1 Processor -- start 2024-09-17T04:56:05.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.606+0000 7ff2cbf38640 1 -- start start 2024-09-17T04:56:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2cbf38640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2c4074e60 0x7ff2c4144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2cbf38640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2c40c4250 0x7ff2c40adc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2cbf38640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2c40be8a0 0x7ff2c40ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2cbf38640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c4161eb0 con 0x7ff2c4074e60 2024-09-17T04:56:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2cbf38640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c4072fc0 con 0x7ff2c40be8a0 2024-09-17T04:56:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2cbf38640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff2c4162030 con 0x7ff2c40c4250 2024-09-17T04:56:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2c9cd4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2c4074e60 0x7ff2c4144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2c9cd4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2c4074e60 0x7ff2c4144e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33206/0 (socket says 172.21.15.150:33206) 2024-09-17T04:56:05.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2c9cd4640 1 -- 172.21.15.150:0/1405384602 learned_addr learned my addr 172.21.15.150:0/1405384602 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:56:05.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2ca4d5640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2c40be8a0 0x7ff2c40ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:05.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2c94d3640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2c40c4250 0x7ff2c40adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:05.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2c9cd4640 1 -- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2c40c4250 msgr2=0x7ff2c40adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:05.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2c9cd4640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2c40c4250 0x7ff2c40adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2c9cd4640 1 -- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2c40be8a0 msgr2=0x7ff2c40ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2c9cd4640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2c40be8a0 0x7ff2c40ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2c9cd4640 1 -- 172.21.15.150:0/1405384602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c40ad1e0 con 0x7ff2c4074e60 2024-09-17T04:56:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2c9cd4640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2c4074e60 0x7ff2c4144e60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff2c000c420 tx=0x7ff2c000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2ba7fc640 1 -- 172.21.15.150:0/1405384602 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0017070 con 0x7ff2c4074e60 2024-09-17T04:56:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2ba7fc640 1 -- 172.21.15.150:0/1405384602 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff2c00053c0 con 0x7ff2c4074e60 2024-09-17T04:56:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2cbf38640 1 -- 172.21.15.150:0/1405384602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2c41ed330 con 0x7ff2c4074e60 2024-09-17T04:56:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2ba7fc640 1 -- 172.21.15.150:0/1405384602 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0003860 con 0x7ff2c4074e60 2024-09-17T04:56:05.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2cbf38640 1 -- 172.21.15.150:0/1405384602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2c40ad760 con 0x7ff2c4074e60 2024-09-17T04:56:05.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2cbf38640 1 -- 172.21.15.150:0/1405384602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2c4074920 con 0x7ff2c4074e60 2024-09-17T04:56:05.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2ba7fc640 1 -- 172.21.15.150:0/1405384602 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff2c001a070 con 0x7ff2c4074e60 2024-09-17T04:56:05.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.610+0000 7ff2ba7fc640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff2a8073200 0x7ff2a8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:05.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.614+0000 7ff2ba7fc640 1 -- 172.21.15.150:0/1405384602 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7ff2c0093b30 con 0x7ff2c4074e60 2024-09-17T04:56:05.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.614+0000 7ff2c94d3640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff2a8073200 0x7ff2a8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:05.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.614+0000 7ff2c94d3640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff2a8073200 0x7ff2a8075710 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff2b40af000 tx=0x7ff2b401b060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:05.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.618+0000 7ff2ba7fc640 1 -- 172.21.15.150:0/1405384602 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff2c005e5c0 con 0x7ff2c4074e60 2024-09-17T04:56:05.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.874+0000 7ff2cbf38640 1 -- 172.21.15.150:0/1405384602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2c4152110 con 0x7ff2c4074e60 2024-09-17T04:56:05.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.874+0000 7ff2ba7fc640 1 -- 172.21.15.150:0/1405384602 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6537 (secure 0 0 0) 0x7ff2c005dec0 con 0x7ff2c4074e60 2024-09-17T04:56:05.877 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:56:05.877 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":51,"btime":"2024-09-17T04:55:54:232004+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:56:05.877 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 51 2024-09-17T04:56:05.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.878+0000 7ff2cbf38640 1 -- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff2a8073200 msgr2=0x7ff2a8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:05.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.878+0000 7ff2cbf38640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff2a8073200 0x7ff2a8075710 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff2b40af000 tx=0x7ff2b401b060 comp rx=0 tx=0).stop 2024-09-17T04:56:05.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.878+0000 7ff2cbf38640 1 -- 172.21.15.150:0/1405384602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2c4074e60 msgr2=0x7ff2c4144e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:05.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.878+0000 7ff2cbf38640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2c4074e60 0x7ff2c4144e60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff2c000c420 tx=0x7ff2c000c8e0 comp rx=0 tx=0).stop 2024-09-17T04:56:05.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.878+0000 7ff2cbf38640 1 -- 172.21.15.150:0/1405384602 shutdown_connections 2024-09-17T04:56:05.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.878+0000 7ff2cbf38640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff2a8073200 0x7ff2a8075710 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:05.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.878+0000 7ff2cbf38640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2c40be8a0 0x7ff2c40ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:05.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.878+0000 7ff2cbf38640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2c40c4250 0x7ff2c40adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:05.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.878+0000 7ff2cbf38640 1 --2- 172.21.15.150:0/1405384602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2c4074e60 0x7ff2c4144e60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:05.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.878+0000 7ff2cbf38640 1 -- 172.21.15.150:0/1405384602 >> 172.21.15.150:0/1405384602 conn(0x7ff2c406e360 msgr2=0x7ff2c4152de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:05.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.878+0000 7ff2cbf38640 1 -- 172.21.15.150:0/1405384602 shutdown_connections 2024-09-17T04:56:05.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:05.878+0000 7ff2cbf38640 1 -- 172.21.15.150:0/1405384602 wait complete. 2024-09-17T04:56:05.902 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:56:07.903 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:56:08.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.050+0000 7f18415d8640 1 -- 172.21.15.150:0/3217329807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1824003a60 msgr2=0x7f1824003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:08.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.050+0000 7f18415d8640 1 --2- 172.21.15.150:0/3217329807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1824003a60 0x7f1824003ed0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f182c0016f0 tx=0x7f182c01c790 comp rx=0 tx=0).stop 2024-09-17T04:56:08.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.050+0000 7f18415d8640 1 -- 172.21.15.150:0/3217329807 shutdown_connections 2024-09-17T04:56:08.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.050+0000 7f18415d8640 1 --2- 172.21.15.150:0/3217329807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1824003a60 0x7f1824003ed0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:08.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.050+0000 7f18415d8640 1 -- 172.21.15.150:0/3217329807 >> 172.21.15.150:0/3217329807 conn(0x7f183c0aab60 msgr2=0x7f183c0aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:08.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.050+0000 7f18415d8640 1 -- 172.21.15.150:0/3217329807 shutdown_connections 2024-09-17T04:56:08.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.050+0000 7f18415d8640 1 -- 172.21.15.150:0/3217329807 wait complete. 2024-09-17T04:56:08.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f18415d8640 1 Processor -- start 2024-09-17T04:56:08.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f18415d8640 1 -- start start 2024-09-17T04:56:08.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f18415d8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1824003a60 0x7f183c1e1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:08.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f18415d8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f183c1f3090 0x7f183c1e32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:08.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f18415d8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f183c1e3810 0x7f183c148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:08.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f18415d8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f183c0c7190 con 0x7f183c1e3810 2024-09-17T04:56:08.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f18415d8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f183c0c6cd0 con 0x7f183c1f3090 2024-09-17T04:56:08.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f18415d8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f183c0c7310 con 0x7f1824003a60 2024-09-17T04:56:08.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f183b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f183c1e3810 0x7f183c148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:08.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f183b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f183c1e3810 0x7f183c148130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33236/0 (socket says 172.21.15.150:33236) 2024-09-17T04:56:08.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f183b7fe640 1 -- 172.21.15.150:0/3967356993 learned_addr learned my addr 172.21.15.150:0/3967356993 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:56:08.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f183affd640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1824003a60 0x7f183c1e1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:08.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f183a7fc640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f183c1f3090 0x7f183c1e32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:08.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f183b7fe640 1 -- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1824003a60 msgr2=0x7f183c1e1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:08.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f183b7fe640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1824003a60 0x7f183c1e1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:08.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f183b7fe640 1 -- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f183c1f3090 msgr2=0x7f183c1e32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:08.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f183b7fe640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f183c1f3090 0x7f183c1e32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:08.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f183b7fe640 1 -- 172.21.15.150:0/3967356993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183c148d60 con 0x7f183c1e3810 2024-09-17T04:56:08.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.054+0000 7f183affd640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1824003a60 0x7f183c1e1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:56:08.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.058+0000 7f183b7fe640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f183c1e3810 0x7f183c148130 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f183c0abbb0 tx=0x7f183400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:08.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.058+0000 7f18237fe640 1 -- 172.21.15.150:0/3967356993 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1834018070 con 0x7f183c1e3810 2024-09-17T04:56:08.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.058+0000 7f18237fe640 1 -- 172.21.15.150:0/3967356993 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f183400de60 con 0x7f183c1e3810 2024-09-17T04:56:08.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.058+0000 7f18237fe640 1 -- 172.21.15.150:0/3967356993 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1834003dd0 con 0x7f183c1e3810 2024-09-17T04:56:08.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.058+0000 7f18415d8640 1 -- 172.21.15.150:0/3967356993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f183c1e23c0 con 0x7f183c1e3810 2024-09-17T04:56:08.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.058+0000 7f18415d8640 1 -- 172.21.15.150:0/3967356993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f183c1e3c50 con 0x7f183c1e3810 2024-09-17T04:56:08.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.058+0000 7f18415d8640 1 -- 172.21.15.150:0/3967356993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f183c1e3eb0 con 0x7f183c1e3810 2024-09-17T04:56:08.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.058+0000 7f18237fe640 1 -- 172.21.15.150:0/3967356993 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f183401b070 con 0x7f183c1e3810 2024-09-17T04:56:08.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.058+0000 7f18237fe640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1828073200 0x7f1828075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:08.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.058+0000 7f18237fe640 1 -- 172.21.15.150:0/3967356993 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f1834093e30 con 0x7f183c1e3810 2024-09-17T04:56:08.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.058+0000 7f183affd640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1828073200 0x7f1828075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:08.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.062+0000 7f183affd640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1828073200 0x7f1828075710 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f182c0016c0 tx=0x7f182c019580 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:08.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.066+0000 7f18237fe640 1 -- 172.21.15.150:0/3967356993 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f183405e950 con 0x7f183c1e3810 2024-09-17T04:56:08.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.270+0000 7f18415d8640 1 -- 172.21.15.150:0/3967356993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f183c0c7780 con 0x7f183c1e3810 2024-09-17T04:56:08.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.270+0000 7f18237fe640 1 -- 172.21.15.150:0/3967356993 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6537 (secure 0 0 0) 0x7f183405e250 con 0x7f183c1e3810 2024-09-17T04:56:08.274 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:56:08.275 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":51,"btime":"2024-09-17T04:55:54:232004+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:56:08.275 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 51 2024-09-17T04:56:08.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.274+0000 7f18415d8640 1 -- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1828073200 msgr2=0x7f1828075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:08.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.274+0000 7f18415d8640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1828073200 0x7f1828075710 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f182c0016c0 tx=0x7f182c019580 comp rx=0 tx=0).stop 2024-09-17T04:56:08.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.274+0000 7f18415d8640 1 -- 172.21.15.150:0/3967356993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f183c1e3810 msgr2=0x7f183c148130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:08.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.274+0000 7f18415d8640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f183c1e3810 0x7f183c148130 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f183c0abbb0 tx=0x7f183400dc10 comp rx=0 tx=0).stop 2024-09-17T04:56:08.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.274+0000 7f18415d8640 1 -- 172.21.15.150:0/3967356993 shutdown_connections 2024-09-17T04:56:08.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.274+0000 7f18415d8640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1828073200 0x7f1828075710 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:08.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.274+0000 7f18415d8640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f183c1f3090 0x7f183c1e32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:08.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.274+0000 7f18415d8640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f183c1e3810 0x7f183c148130 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:08.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.274+0000 7f18415d8640 1 --2- 172.21.15.150:0/3967356993 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1824003a60 0x7f183c1e1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:08.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.274+0000 7f18415d8640 1 -- 172.21.15.150:0/3967356993 >> 172.21.15.150:0/3967356993 conn(0x7f183c0aab60 msgr2=0x7f183c0c21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:08.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.274+0000 7f18415d8640 1 -- 172.21.15.150:0/3967356993 shutdown_connections 2024-09-17T04:56:08.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:08.274+0000 7f18415d8640 1 -- 172.21.15.150:0/3967356993 wait complete. 2024-09-17T04:56:08.296 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:56:10.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:56:10.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.474+0000 7f8629af7640 1 -- 172.21.15.150:0/768040913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860c004d50 msgr2=0x7f860c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:10.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.474+0000 7f8629af7640 1 --2- 172.21.15.150:0/768040913 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860c004d50 0x7f860c005160 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f861c009eb0 tx=0x7f861c01d5d0 comp rx=0 tx=0).stop 2024-09-17T04:56:10.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.474+0000 7f8623fff640 1 -- 172.21.15.150:0/768040913 reap_dead start 2024-09-17T04:56:10.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.474+0000 7f8629af7640 1 -- 172.21.15.150:0/768040913 shutdown_connections 2024-09-17T04:56:10.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.474+0000 7f8629af7640 1 -- 172.21.15.150:0/768040913 >> 172.21.15.150:0/768040913 conn(0x7f862406e360 msgr2=0x7f862406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:10.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.474+0000 7f8629af7640 1 -- 172.21.15.150:0/768040913 shutdown_connections 2024-09-17T04:56:10.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.474+0000 7f8629af7640 1 -- 172.21.15.150:0/768040913 wait complete. 2024-09-17T04:56:10.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8629af7640 1 Processor -- start 2024-09-17T04:56:10.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8629af7640 1 -- start start 2024-09-17T04:56:10.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8629af7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f860c004d50 0x7f862414ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:10.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8629af7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86240ae350 0x7f862406ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:10.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8629af7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f862406d390 0x7f86241df5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:10.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8629af7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8624071120 con 0x7f862406d390 2024-09-17T04:56:10.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8629af7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8624070fa0 con 0x7f860c004d50 2024-09-17T04:56:10.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8629af7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8624072db0 con 0x7f86240ae350 2024-09-17T04:56:10.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8623fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f862406d390 0x7f86241df5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:10.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8623fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f862406d390 0x7f86241df5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43158/0 (socket says 172.21.15.150:43158) 2024-09-17T04:56:10.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8623fff640 1 -- 172.21.15.150:0/2179957560 learned_addr learned my addr 172.21.15.150:0/2179957560 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:56:10.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f86237fe640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f860c004d50 0x7f862414ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:10.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8622ffd640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86240ae350 0x7f862406ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:10.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8623fff640 1 -- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86240ae350 msgr2=0x7f862406ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:10.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8623fff640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86240ae350 0x7f862406ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:10.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8623fff640 1 -- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f860c004d50 msgr2=0x7f862414ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8623fff640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f860c004d50 0x7f862414ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8623fff640 1 -- 172.21.15.150:0/2179957560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86241ea0a0 con 0x7f862406d390 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.478+0000 7f8622ffd640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86240ae350 0x7f862406ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.482+0000 7f8623fff640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f862406d390 0x7f86241df5b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f861c01d040 tx=0x7f861c009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.482+0000 7f860bfff640 1 -- 172.21.15.150:0/2179957560 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861c009200 con 0x7f862406d390 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.482+0000 7f860bfff640 1 -- 172.21.15.150:0/2179957560 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f861c0a9510 con 0x7f862406d390 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.482+0000 7f8629af7640 1 -- 172.21.15.150:0/2179957560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f862414dbb0 con 0x7f862406d390 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.482+0000 7f860bfff640 1 -- 172.21.15.150:0/2179957560 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861c0a9cd0 con 0x7f862406d390 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.482+0000 7f8629af7640 1 -- 172.21.15.150:0/2179957560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86241e0720 con 0x7f862406d390 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.482+0000 7f8629af7640 1 -- 172.21.15.150:0/2179957560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86241ea270 con 0x7f862406d390 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.482+0000 7f860bfff640 1 -- 172.21.15.150:0/2179957560 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f861c0b9030 con 0x7f862406d390 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.482+0000 7f860bfff640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8610073200 0x7f8610075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:10.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.482+0000 7f860bfff640 1 -- 172.21.15.150:0/2179957560 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7268+0+0 (secure 0 0 0) 0x7f861c12fa20 con 0x7f862406d390 2024-09-17T04:56:10.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.482+0000 7f86237fe640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8610073200 0x7f8610075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:10.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.486+0000 7f86237fe640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8610073200 0x7f8610075710 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8614004ab0 tx=0x7f8614006040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:10.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.490+0000 7f860bfff640 1 -- 172.21.15.150:0/2179957560 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f861c0ad030 con 0x7f862406d390 2024-09-17T04:56:10.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 -- 172.21.15.150:0/2179957560 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f862414d6c0 con 0x7f862406d390 2024-09-17T04:56:10.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f860bfff640 1 -- 172.21.15.150:0/2179957560 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6537 (secure 0 0 0) 0x7f862414d6c0 con 0x7f862406d390 2024-09-17T04:56:10.748 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:56:10.748 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":51,"btime":"2024-09-17T04:55:54:232004+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:54:58.881332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5107":{"gid":5107,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/2374314256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2374314256},{"type":"v1","addr":"172.21.15.176:6837","nonce":2374314256}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5107,"qdb_cluster":[5107]},"id":2}]} 2024-09-17T04:56:10.748 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 51 2024-09-17T04:56:10.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 -- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8610073200 msgr2=0x7f8610075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:10.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8610073200 0x7f8610075710 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8614004ab0 tx=0x7f8614006040 comp rx=0 tx=0).stop 2024-09-17T04:56:10.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 -- 172.21.15.150:0/2179957560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f862406d390 msgr2=0x7f86241df5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:10.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f862406d390 0x7f86241df5b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f861c01d040 tx=0x7f861c009480 comp rx=0 tx=0).stop 2024-09-17T04:56:10.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 -- 172.21.15.150:0/2179957560 shutdown_connections 2024-09-17T04:56:10.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8610073200 0x7f8610075710 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:10.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f862406d390 0x7f86241df5b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:10.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86240ae350 0x7f862406ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:10.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 --2- 172.21.15.150:0/2179957560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f860c004d50 0x7f862414ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:10.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 -- 172.21.15.150:0/2179957560 >> 172.21.15.150:0/2179957560 conn(0x7f862406e360 msgr2=0x7f8624153d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:10.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 -- 172.21.15.150:0/2179957560 shutdown_connections 2024-09-17T04:56:10.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:10.746+0000 7f8629af7640 1 -- 172.21.15.150:0/2179957560 wait complete. 2024-09-17T04:56:10.771 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:56:12.771 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:56:12.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.954+0000 7f57868da640 1 -- 172.21.15.150:0/2831968403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57800c4250 msgr2=0x7f5780153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:12.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.954+0000 7f57868da640 1 --2- 172.21.15.150:0/2831968403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57800c4250 0x7f5780153ee0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5768009840 tx=0x7f576801c7e0 comp rx=0 tx=0).stop 2024-09-17T04:56:12.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.954+0000 7f57868da640 1 -- 172.21.15.150:0/2831968403 shutdown_connections 2024-09-17T04:56:12.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.954+0000 7f57868da640 1 --2- 172.21.15.150:0/2831968403 >> v2:172.21.15.176:3300/0 conn(0x7f57800c3d10 0x7f578015df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:12.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.954+0000 7f57868da640 1 --2- 172.21.15.150:0/2831968403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57800c4250 0x7f5780153ee0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:12.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.954+0000 7f57868da640 1 -- 172.21.15.150:0/2831968403 >> 172.21.15.150:0/2831968403 conn(0x7f578006e360 msgr2=0x7f578006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:12.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.954+0000 7f57868da640 1 -- 172.21.15.150:0/2831968403 shutdown_connections 2024-09-17T04:56:12.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.954+0000 7f57868da640 1 -- 172.21.15.150:0/2831968403 wait complete. 2024-09-17T04:56:12.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.954+0000 7f57868da640 1 Processor -- start 2024-09-17T04:56:12.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.954+0000 7f57868da640 1 -- start start 2024-09-17T04:56:12.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f57868da640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57800c3900 0x7f5780152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:12.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f57868da640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57800c3d10 0x7f57801ef6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:12.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f57868da640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57800c4250 0x7f5780149ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:12.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f57868da640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5780071120 con 0x7f57800c4250 2024-09-17T04:56:12.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f57868da640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5780070fa0 con 0x7f57800c3900 2024-09-17T04:56:12.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f57868da640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5780072db0 con 0x7f57800c3d10 2024-09-17T04:56:12.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f5784e77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57800c4250 0x7f5780149ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:12.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f5784e77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57800c4250 0x7f5780149ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43180/0 (socket says 172.21.15.150:43180) 2024-09-17T04:56:12.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f5784e77640 1 -- 172.21.15.150:0/608403938 learned_addr learned my addr 172.21.15.150:0/608403938 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:56:12.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f577ffff640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57800c3900 0x7f5780152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:12.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f5777fff640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57800c3d10 0x7f57801ef6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:12.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f5784e77640 1 -- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57800c3d10 msgr2=0x7f57801ef6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:12.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f5784e77640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57800c3d10 0x7f57801ef6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:12.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f5784e77640 1 -- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57800c3900 msgr2=0x7f5780152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:12.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f5784e77640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57800c3900 0x7f5780152710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:12.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f5784e77640 1 -- 172.21.15.150:0/608403938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57800abc80 con 0x7f57800c4250 2024-09-17T04:56:12.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f5784e77640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57800c4250 0x7f5780149ab0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f577800e970 tx=0x7f577800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:12.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f577d7fa640 1 -- 172.21.15.150:0/608403938 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5778004d30 con 0x7f57800c4250 2024-09-17T04:56:12.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f577d7fa640 1 -- 172.21.15.150:0/608403938 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f577800cb20 con 0x7f57800c4250 2024-09-17T04:56:12.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f57868da640 1 -- 172.21.15.150:0/608403938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57800abf60 con 0x7f57800c4250 2024-09-17T04:56:12.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f577d7fa640 1 -- 172.21.15.150:0/608403938 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5778005020 con 0x7f57800c4250 2024-09-17T04:56:12.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f57868da640 1 -- 172.21.15.150:0/608403938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57801f0130 con 0x7f57800c4250 2024-09-17T04:56:12.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f57868da640 1 -- 172.21.15.150:0/608403938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5780153ee0 con 0x7f57800c4250 2024-09-17T04:56:12.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f577d7fa640 1 -- 172.21.15.150:0/608403938 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f57780051c0 con 0x7f57800c4250 2024-09-17T04:56:12.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f577d7fa640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5760073230 0x7f5760075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:12.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.958+0000 7f577d7fa640 1 -- 172.21.15.150:0/608403938 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7f5778095400 con 0x7f57800c4250 2024-09-17T04:56:12.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.962+0000 7f577ffff640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5760073230 0x7f5760075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:12.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.962+0000 7f577ffff640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5760073230 0x7f5760075740 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f57680af000 tx=0x7f576801b020 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:12.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:12.966+0000 7f577d7fa640 1 -- 172.21.15.150:0/608403938 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5778016030 con 0x7f57800c4250 2024-09-17T04:56:13.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.198+0000 7f57868da640 1 -- 172.21.15.150:0/608403938 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f578006f9e0 con 0x7f57800c4250 2024-09-17T04:56:13.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f577d7fa640 1 -- 172.21.15.150:0/608403938 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5701 (secure 0 0 0) 0x7f577805ffb0 con 0x7f57800c4250 2024-09-17T04:56:13.203 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:56:13.203 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":54,"btime":"2024-09-17T04:56:12:904048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:56:11.910146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"a","rank":0,"incarnation":52,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:56:13.203 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 54 2024-09-17T04:56:13.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f57868da640 1 -- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5760073230 msgr2=0x7f5760075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:13.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f57868da640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5760073230 0x7f5760075740 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f57680af000 tx=0x7f576801b020 comp rx=0 tx=0).stop 2024-09-17T04:56:13.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f57868da640 1 -- 172.21.15.150:0/608403938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57800c4250 msgr2=0x7f5780149ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:13.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f57868da640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57800c4250 0x7f5780149ab0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f577800e970 tx=0x7f577800ee30 comp rx=0 tx=0).stop 2024-09-17T04:56:13.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f57868da640 1 -- 172.21.15.150:0/608403938 shutdown_connections 2024-09-17T04:56:13.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f57868da640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5760073230 0x7f5760075740 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:13.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f57868da640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f57800c4250 0x7f5780149ab0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:13.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f57868da640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57800c3d10 0x7f57801ef6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:13.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f57868da640 1 --2- 172.21.15.150:0/608403938 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57800c3900 0x7f5780152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:13.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f57868da640 1 -- 172.21.15.150:0/608403938 >> 172.21.15.150:0/608403938 conn(0x7f578006e360 msgr2=0x7f578014f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:13.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f57868da640 1 -- 172.21.15.150:0/608403938 shutdown_connections 2024-09-17T04:56:13.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.202+0000 7f57868da640 1 -- 172.21.15.150:0/608403938 wait complete. 2024-09-17T04:56:13.225 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-17T04:56:13.225 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T04:56:13.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:56:13.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.390+0000 7f938b79a640 1 -- 172.21.15.150:0/2551697098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9370004810 msgr2=0x7f9370004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:13.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.390+0000 7f938b79a640 1 --2- 172.21.15.150:0/2551697098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9370004810 0x7f9370004c20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f937400bb10 tx=0x7f937401cde0 comp rx=0 tx=0).stop 2024-09-17T04:56:13.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.390+0000 7f9389d37640 1 -- 172.21.15.150:0/2551697098 reap_dead start 2024-09-17T04:56:13.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.390+0000 7f938b79a640 1 -- 172.21.15.150:0/2551697098 shutdown_connections 2024-09-17T04:56:13.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.390+0000 7f938b79a640 1 -- 172.21.15.150:0/2551697098 >> 172.21.15.150:0/2551697098 conn(0x7f9384074b80 msgr2=0x7f938406f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:13.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.390+0000 7f938b79a640 1 -- 172.21.15.150:0/2551697098 shutdown_connections 2024-09-17T04:56:13.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.390+0000 7f938b79a640 1 -- 172.21.15.150:0/2551697098 wait complete. 2024-09-17T04:56:13.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f938b79a640 1 Processor -- start 2024-09-17T04:56:13.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f938b79a640 1 -- start start 2024-09-17T04:56:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f938b79a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9370004810 0x7f93840abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f938b79a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93840ac090 0x7f938414b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f938b79a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f938414bc20 0x7f93840aced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f938b79a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f93840c39c0 con 0x7f93840ac090 2024-09-17T04:56:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f938b79a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9384154450 con 0x7f9370004810 2024-09-17T04:56:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f938b79a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f93840c3b40 con 0x7f938414bc20 2024-09-17T04:56:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9388d35640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93840ac090 0x7f938414b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9388d35640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93840ac090 0x7f938414b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43190/0 (socket says 172.21.15.150:43190) 2024-09-17T04:56:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9388d35640 1 -- 172.21.15.150:0/615118477 learned_addr learned my addr 172.21.15.150:0/615118477 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:56:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9389536640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9370004810 0x7f93840abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9389d37640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f938414bc20 0x7f93840aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9388d35640 1 -- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f938414bc20 msgr2=0x7f93840aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9388d35640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f938414bc20 0x7f93840aced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9388d35640 1 -- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9370004810 msgr2=0x7f93840abb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9388d35640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9370004810 0x7f93840abb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9388d35640 1 -- 172.21.15.150:0/615118477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f938414c030 con 0x7f93840ac090 2024-09-17T04:56:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9388d35640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93840ac090 0x7f938414b6e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f93740b1040 tx=0x7f9374005630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9379ffb640 1 -- 172.21.15.150:0/615118477 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93740bd020 con 0x7f93840ac090 2024-09-17T04:56:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f938b79a640 1 -- 172.21.15.150:0/615118477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93840ac5b0 con 0x7f93840ac090 2024-09-17T04:56:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9379ffb640 1 -- 172.21.15.150:0/615118477 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9374004020 con 0x7f93840ac090 2024-09-17T04:56:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f9379ffb640 1 -- 172.21.15.150:0/615118477 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93740ab040 con 0x7f93840ac090 2024-09-17T04:56:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.394+0000 7f938b79a640 1 -- 172.21.15.150:0/615118477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93841534e0 con 0x7f93840ac090 2024-09-17T04:56:13.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.398+0000 7f938b79a640 1 -- 172.21.15.150:0/615118477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93840c8a80 con 0x7f93840ac090 2024-09-17T04:56:13.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.398+0000 7f9379ffb640 1 -- 172.21.15.150:0/615118477 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f93740c0070 con 0x7f93840ac090 2024-09-17T04:56:13.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.398+0000 7f9379ffb640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9368073260 0x7f9368075770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:13.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.398+0000 7f9379ffb640 1 -- 172.21.15.150:0/615118477 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7f937412e910 con 0x7f93840ac090 2024-09-17T04:56:13.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.398+0000 7f9389536640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9368073260 0x7f9368075770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:13.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.398+0000 7f9389536640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9368073260 0x7f9368075770 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f938000dbc0 tx=0x7f938000d390 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:13.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.402+0000 7f9379ffb640 1 -- 172.21.15.150:0/615118477 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f93740f3db0 con 0x7f93840ac090 2024-09-17T04:56:13.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.682+0000 7f938b79a640 1 -- 172.21.15.150:0/615118477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93840ab770 con 0x7f93840ac090 2024-09-17T04:56:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.682+0000 7f9379ffb640 1 -- 172.21.15.150:0/615118477 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5701 (secure 0 0 0) 0x7f9374016af0 con 0x7f93840ac090 2024-09-17T04:56:13.685 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:56:13.686 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":54,"btime":"2024-09-17T04:56:12:904048+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:56:11.910146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"a","rank":0,"incarnation":52,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:56:13.686 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 54 2024-09-17T04:56:13.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.686+0000 7f938b79a640 1 -- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9368073260 msgr2=0x7f9368075770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:13.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.686+0000 7f938b79a640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9368073260 0x7f9368075770 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f938000dbc0 tx=0x7f938000d390 comp rx=0 tx=0).stop 2024-09-17T04:56:13.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.686+0000 7f938b79a640 1 -- 172.21.15.150:0/615118477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93840ac090 msgr2=0x7f938414b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:13.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.686+0000 7f938b79a640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93840ac090 0x7f938414b6e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f93740b1040 tx=0x7f9374005630 comp rx=0 tx=0).stop 2024-09-17T04:56:13.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.686+0000 7f938b79a640 1 -- 172.21.15.150:0/615118477 shutdown_connections 2024-09-17T04:56:13.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.686+0000 7f938b79a640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9368073260 0x7f9368075770 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:13.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.686+0000 7f938b79a640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f938414bc20 0x7f93840aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:13.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.686+0000 7f938b79a640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93840ac090 0x7f938414b6e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:13.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.686+0000 7f938b79a640 1 --2- 172.21.15.150:0/615118477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9370004810 0x7f93840abb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:13.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.686+0000 7f938b79a640 1 -- 172.21.15.150:0/615118477 >> 172.21.15.150:0/615118477 conn(0x7f9384074b80 msgr2=0x7f93840c7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:13.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.686+0000 7f938b79a640 1 -- 172.21.15.150:0/615118477 shutdown_connections 2024-09-17T04:56:13.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:13.686+0000 7f938b79a640 1 -- 172.21.15.150:0/615118477 wait complete. 2024-09-17T04:56:13.711 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T04:56:12:904048+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 54, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5254": { "addr": "172.21.15.150:6835/3079958649", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 3079958649, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 3079958649, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5254, "incarnation": 47, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:55:43.893595+0000", "qdb_cluster": [ 5254 ], "qdb_leader": 5254, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5254 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 54, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5631": { "addr": "172.21.15.150:6833/284564893", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 284564893, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 284564893, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5631, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:56:11.910146+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5631 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:56:14.239 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T04:56:14.238+0000 7f20a1ed0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:56:15.713 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:56:15.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.886+0000 7f0e1a18b640 1 -- 172.21.15.150:0/248150927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e00004d50 msgr2=0x7f0e00005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:15.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.886+0000 7f0e1a18b640 1 --2- 172.21.15.150:0/248150927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e00004d50 0x7f0e00005160 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c009ee0 tx=0x7f0e0c01d610 comp rx=0 tx=0).stop 2024-09-17T04:56:15.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.886+0000 7f0e13fff640 1 -- 172.21.15.150:0/248150927 reap_dead start 2024-09-17T04:56:15.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.886+0000 7f0e1a18b640 1 -- 172.21.15.150:0/248150927 shutdown_connections 2024-09-17T04:56:15.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.886+0000 7f0e1a18b640 1 -- 172.21.15.150:0/248150927 >> 172.21.15.150:0/248150927 conn(0x7f0e140aaa80 msgr2=0x7f0e140aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:15.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.886+0000 7f0e1a18b640 1 -- 172.21.15.150:0/248150927 shutdown_connections 2024-09-17T04:56:15.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.886+0000 7f0e1a18b640 1 -- 172.21.15.150:0/248150927 wait complete. 2024-09-17T04:56:15.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e1a18b640 1 Processor -- start 2024-09-17T04:56:15.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e1a18b640 1 -- start start 2024-09-17T04:56:15.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e1a18b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e14148e70 0x7f0e14143620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:15.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e1a18b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e140adf80 0x7f0e140ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:15.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e1a18b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e141537a0 0x7f0e14153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:15.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e1a18b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0e14075350 con 0x7f0e14148e70 2024-09-17T04:56:15.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e1a18b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0e140c8230 con 0x7f0e141537a0 2024-09-17T04:56:15.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e1a18b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0e14071b10 con 0x7f0e140adf80 2024-09-17T04:56:15.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e137fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e14148e70 0x7f0e14143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:15.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e137fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e14148e70 0x7f0e14143620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43206/0 (socket says 172.21.15.150:43206) 2024-09-17T04:56:15.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e137fe640 1 -- 172.21.15.150:0/3420464599 learned_addr learned my addr 172.21.15.150:0/3420464599 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:56:15.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e0bfff640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e140adf80 0x7f0e140ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:15.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e13fff640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e141537a0 0x7f0e14153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:15.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e137fe640 1 -- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e140adf80 msgr2=0x7f0e140ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:15.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e137fe640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e140adf80 0x7f0e140ae390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:15.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e137fe640 1 -- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e141537a0 msgr2=0x7f0e14153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:15.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e137fe640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e141537a0 0x7f0e14153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:15.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e137fe640 1 -- 172.21.15.150:0/3420464599 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e140ad880 con 0x7f0e14148e70 2024-09-17T04:56:15.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e13fff640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e141537a0 0x7f0e14153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:56:15.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e137fe640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e14148e70 0x7f0e14143620 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0e04011a50 tx=0x7f0e04011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:15.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e10ff9640 1 -- 172.21.15.150:0/3420464599 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0401f070 con 0x7f0e14148e70 2024-09-17T04:56:15.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e10ff9640 1 -- 172.21.15.150:0/3420464599 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e0400fe20 con 0x7f0e14148e70 2024-09-17T04:56:15.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e1a18b640 1 -- 172.21.15.150:0/3420464599 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e140a9fd0 con 0x7f0e14148e70 2024-09-17T04:56:15.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e10ff9640 1 -- 172.21.15.150:0/3420464599 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e04012690 con 0x7f0e14148e70 2024-09-17T04:56:15.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.890+0000 7f0e1a18b640 1 -- 172.21.15.150:0/3420464599 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e141480c0 con 0x7f0e14148e70 2024-09-17T04:56:15.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.894+0000 7f0e1a18b640 1 -- 172.21.15.150:0/3420464599 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e14072000 con 0x7f0e14148e70 2024-09-17T04:56:15.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.894+0000 7f0e10ff9640 1 -- 172.21.15.150:0/3420464599 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0e04022070 con 0x7f0e14148e70 2024-09-17T04:56:15.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.894+0000 7f0e10ff9640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0df4073200 0x7f0df4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:15.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.894+0000 7f0e10ff9640 1 -- 172.21.15.150:0/3420464599 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7f0e04095320 con 0x7f0e14148e70 2024-09-17T04:56:15.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.894+0000 7f0e0bfff640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0df4073200 0x7f0df4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:15.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.894+0000 7f0e0bfff640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0df4073200 0x7f0df4075710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0e14149280 tx=0x7f0dfc0014c0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:15.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:15.898+0000 7f0e10ff9640 1 -- 172.21.15.150:0/3420464599 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0e0405f710 con 0x7f0e14148e70 2024-09-17T04:56:16.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.126+0000 7f0e1a18b640 1 -- 172.21.15.150:0/3420464599 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e14148940 con 0x7f0e14148e70 2024-09-17T04:56:16.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.126+0000 7f0e10ff9640 1 -- 172.21.15.150:0/3420464599 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5708 (secure 0 0 0) 0x7f0e04064140 con 0x7f0e14148e70 2024-09-17T04:56:16.129 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:56:16.129 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":55,"btime":"2024-09-17T04:56:13:913295+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:56:13.913286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631]},"id":2}]} 2024-09-17T04:56:16.130 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 55 2024-09-17T04:56:16.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.130+0000 7f0e1a18b640 1 -- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0df4073200 msgr2=0x7f0df4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:16.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.130+0000 7f0e1a18b640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0df4073200 0x7f0df4075710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0e14149280 tx=0x7f0dfc0014c0 comp rx=0 tx=0).stop 2024-09-17T04:56:16.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.130+0000 7f0e1a18b640 1 -- 172.21.15.150:0/3420464599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e14148e70 msgr2=0x7f0e14143620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:16.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.130+0000 7f0e1a18b640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e14148e70 0x7f0e14143620 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0e04011a50 tx=0x7f0e04011f10 comp rx=0 tx=0).stop 2024-09-17T04:56:16.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.130+0000 7f0e1a18b640 1 -- 172.21.15.150:0/3420464599 shutdown_connections 2024-09-17T04:56:16.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.130+0000 7f0e1a18b640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0df4073200 0x7f0df4075710 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:16.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.130+0000 7f0e1a18b640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e141537a0 0x7f0e14153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:16.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.130+0000 7f0e1a18b640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e140adf80 0x7f0e140ae390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:16.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.130+0000 7f0e1a18b640 1 --2- 172.21.15.150:0/3420464599 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e14148e70 0x7f0e14143620 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:16.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.130+0000 7f0e1a18b640 1 -- 172.21.15.150:0/3420464599 >> 172.21.15.150:0/3420464599 conn(0x7f0e140aaa80 msgr2=0x7f0e140a89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:16.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.130+0000 7f0e1a18b640 1 -- 172.21.15.150:0/3420464599 shutdown_connections 2024-09-17T04:56:16.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:16.130+0000 7f0e1a18b640 1 -- 172.21.15.150:0/3420464599 wait complete. 2024-09-17T04:56:16.155 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5107 2024-09-17T04:56:16.155 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.d 2024-09-17T04:56:24.156 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-17T04:56:24.157 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:56:24.157 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:56:24.160 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:56:24.161 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:56:24.278 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-09-17T04:56:24.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.334+0000 7fd6902d4640 1 -- 172.21.15.150:0/1539671433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6880c4950 msgr2=0x7fd6880aa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:24.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.334+0000 7fd6902d4640 1 --2- 172.21.15.150:0/1539671433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6880c4950 0x7fd6880aa1d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fd684009840 tx=0x7fd68401c790 comp rx=0 tx=0).stop 2024-09-17T04:56:24.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.334+0000 7fd6902d4640 1 -- 172.21.15.150:0/1539671433 shutdown_connections 2024-09-17T04:56:24.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.334+0000 7fd6902d4640 1 --2- 172.21.15.150:0/1539671433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6880c4950 0x7fd6880aa1d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:24.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.334+0000 7fd6902d4640 1 -- 172.21.15.150:0/1539671433 >> 172.21.15.150:0/1539671433 conn(0x7fd688070b50 msgr2=0x7fd688070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:24.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.334+0000 7fd6902d4640 1 -- 172.21.15.150:0/1539671433 shutdown_connections 2024-09-17T04:56:24.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.334+0000 7fd6902d4640 1 -- 172.21.15.150:0/1539671433 wait complete. 2024-09-17T04:56:24.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd6902d4640 1 Processor -- start 2024-09-17T04:56:24.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd6902d4640 1 -- start start 2024-09-17T04:56:24.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd6902d4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6880aa7b0 0x7fd68814b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:24.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd6902d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6880c4950 0x7fd6881efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd6902d4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd68814cec0 0x7fd6881f3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd6902d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd688160ed0 con 0x7fd6880c4950 2024-09-17T04:56:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd6902d4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd688153c70 con 0x7fd6880aa7b0 2024-09-17T04:56:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd6902d4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd688161050 con 0x7fd68814cec0 2024-09-17T04:56:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd68d86f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6880c4950 0x7fd6881efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd68d86f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6880c4950 0x7fd6881efc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46936/0 (socket says 172.21.15.150:46936) 2024-09-17T04:56:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd68d86f640 1 -- 172.21.15.150:0/468172072 learned_addr learned my addr 172.21.15.150:0/468172072 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:56:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd68e070640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6880aa7b0 0x7fd68814b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd68e871640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd68814cec0 0x7fd6881f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd68d86f640 1 -- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd68814cec0 msgr2=0x7fd6881f3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd68d86f640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd68814cec0 0x7fd6881f3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd68d86f640 1 -- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6880aa7b0 msgr2=0x7fd68814b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd68d86f640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6880aa7b0 0x7fd68814b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd68d86f640 1 -- 172.21.15.150:0/468172072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd688147a10 con 0x7fd6880c4950 2024-09-17T04:56:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd68d86f640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6880c4950 0x7fd6881efc00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd67800c450 tx=0x7fd67800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd67effd640 1 -- 172.21.15.150:0/468172072 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678016020 con 0x7fd6880c4950 2024-09-17T04:56:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd67effd640 1 -- 172.21.15.150:0/468172072 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd678012920 con 0x7fd6880c4950 2024-09-17T04:56:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd67effd640 1 -- 172.21.15.150:0/468172072 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678012d50 con 0x7fd6880c4950 2024-09-17T04:56:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd6902d4640 1 -- 172.21.15.150:0/468172072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6880c1ff0 con 0x7fd6880c4950 2024-09-17T04:56:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.338+0000 7fd6902d4640 1 -- 172.21.15.150:0/468172072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6880c25b0 con 0x7fd6880c4950 2024-09-17T04:56:24.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.342+0000 7fd6902d4640 1 -- 172.21.15.150:0/468172072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6880aa1d0 con 0x7fd6880c4950 2024-09-17T04:56:24.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.342+0000 7fd67effd640 1 -- 172.21.15.150:0/468172072 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd678019020 con 0x7fd6880c4950 2024-09-17T04:56:24.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.342+0000 7fd67effd640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd66c073230 0x7fd66c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:56:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.342+0000 7fd67effd640 1 -- 172.21.15.150:0/468172072 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7fd6780945b0 con 0x7fd6880c4950 2024-09-17T04:56:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.342+0000 7fd68e070640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd66c073230 0x7fd66c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:56:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.342+0000 7fd68e070640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd66c073230 0x7fd66c075740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd6840af000 tx=0x7fd6840b2040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:56:24.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.346+0000 7fd67effd640 1 -- 172.21.15.150:0/468172072 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd678020020 con 0x7fd6880c4950 2024-09-17T04:56:24.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.590+0000 7fd6902d4640 1 -- 172.21.15.150:0/468172072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd688071bf0 con 0x7fd6880c4950 2024-09-17T04:56:24.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.590+0000 7fd67effd640 1 -- 172.21.15.150:0/468172072 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6536 (secure 0 0 0) 0x7fd67805ea10 con 0x7fd6880c4950 2024-09-17T04:56:24.592 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:56:24.592 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":56,"btime":"2024-09-17T04:56:24:353754+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5404,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:56:13.913286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631]},"id":2}]} 2024-09-17T04:56:24.593 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 56 2024-09-17T04:56:24.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.590+0000 7fd6902d4640 1 -- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd66c073230 msgr2=0x7fd66c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:24.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.590+0000 7fd6902d4640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd66c073230 0x7fd66c075740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd6840af000 tx=0x7fd6840b2040 comp rx=0 tx=0).stop 2024-09-17T04:56:24.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.590+0000 7fd6902d4640 1 -- 172.21.15.150:0/468172072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6880c4950 msgr2=0x7fd6881efc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:56:24.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.590+0000 7fd6902d4640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6880c4950 0x7fd6881efc00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd67800c450 tx=0x7fd67800c910 comp rx=0 tx=0).stop 2024-09-17T04:56:24.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.590+0000 7fd6902d4640 1 -- 172.21.15.150:0/468172072 shutdown_connections 2024-09-17T04:56:24.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.594+0000 7fd6902d4640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd66c073230 0x7fd66c075740 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:24.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.594+0000 7fd6902d4640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd68814cec0 0x7fd6881f3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:24.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.594+0000 7fd6902d4640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6880c4950 0x7fd6881efc00 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:24.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.594+0000 7fd6902d4640 1 --2- 172.21.15.150:0/468172072 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd6880aa7b0 0x7fd68814b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:56:24.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.594+0000 7fd6902d4640 1 -- 172.21.15.150:0/468172072 >> 172.21.15.150:0/468172072 conn(0x7fd688070b50 msgr2=0x7fd68806ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:56:24.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.594+0000 7fd6902d4640 1 -- 172.21.15.150:0/468172072 shutdown_connections 2024-09-17T04:56:24.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:56:24.594+0000 7fd6902d4640 1 -- 172.21.15.150:0/468172072 wait complete. 2024-09-17T04:56:24.615 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-17T04:56:24.615 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2024-09-17T04:56:30.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:56:30.123 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:57:00.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:57:00.253 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:57:26.616 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:26.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.801+0000 7f908cf68640 1 -- 172.21.15.150:0/1519072404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9088072ea0 msgr2=0x7f90880732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:26.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.801+0000 7f908cf68640 1 --2- 172.21.15.150:0/1519072404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9088072ea0 0x7f90880732b0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9070009840 tx=0x7f907001c820 comp rx=0 tx=0).stop 2024-09-17T04:57:26.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.801+0000 7f908cf68640 1 -- 172.21.15.150:0/1519072404 shutdown_connections 2024-09-17T04:57:26.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.801+0000 7f908cf68640 1 --2- 172.21.15.150:0/1519072404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9088072ea0 0x7f90880732b0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:26.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.801+0000 7f908cf68640 1 --2- 172.21.15.150:0/1519072404 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f90880c97a0 0x7f9088151c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:26.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.801+0000 7f908cf68640 1 -- 172.21.15.150:0/1519072404 >> 172.21.15.150:0/1519072404 conn(0x7f90880abb00 msgr2=0x7f90880abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:26.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.801+0000 7f908cf68640 1 -- 172.21.15.150:0/1519072404 shutdown_connections 2024-09-17T04:57:26.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.801+0000 7f908cf68640 1 -- 172.21.15.150:0/1519072404 wait complete. 2024-09-17T04:57:26.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908cf68640 1 Processor -- start 2024-09-17T04:57:26.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908cf68640 1 -- start start 2024-09-17T04:57:26.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908cf68640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9088072ea0 0x7f908815d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:26.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908cf68640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90880c97a0 0x7f90881f0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:26.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908cf68640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9088157be0 0x7f90881f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:26.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908cf68640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9088161910 con 0x7f9088072ea0 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908cf68640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9088161790 con 0x7f90880c97a0 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908cf68640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9088161a90 con 0x7f9088157be0 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9088072ea0 0x7f908815d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9088072ea0 0x7f908815d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38436/0 (socket says 172.21.15.150:38436) 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908659c640 1 -- 172.21.15.150:0/1467867731 learned_addr learned my addr 172.21.15.150:0/1467867731 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f9086d9d640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9088157be0 0x7f90881f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f907ffff640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90880c97a0 0x7f90881f0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908659c640 1 -- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9088157be0 msgr2=0x7f90881f13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908659c640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9088157be0 0x7f90881f13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908659c640 1 -- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90880c97a0 msgr2=0x7f90881f0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908659c640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90880c97a0 0x7f90881f0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908659c640 1 -- 172.21.15.150:0/1467867731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f908814e950 con 0x7f9088072ea0 2024-09-17T04:57:26.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f907ffff640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90880c97a0 0x7f90881f0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:57:26.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908659c640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9088072ea0 0x7f908815d170 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f907800c970 tx=0x7f907800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:26.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f907effd640 1 -- 172.21.15.150:0/1467867731 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9078004220 con 0x7f9088072ea0 2024-09-17T04:57:26.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f907effd640 1 -- 172.21.15.150:0/1467867731 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90780079b0 con 0x7f9088072ea0 2024-09-17T04:57:26.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908cf68640 1 -- 172.21.15.150:0/1467867731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90881f19a0 con 0x7f9088072ea0 2024-09-17T04:57:26.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f907effd640 1 -- 172.21.15.150:0/1467867731 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90780044b0 con 0x7f9088072ea0 2024-09-17T04:57:26.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.805+0000 7f908cf68640 1 -- 172.21.15.150:0/1467867731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9088151f00 con 0x7f9088072ea0 2024-09-17T04:57:26.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.809+0000 7f908cf68640 1 -- 172.21.15.150:0/1467867731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90880732b0 con 0x7f9088072ea0 2024-09-17T04:57:26.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.809+0000 7f907effd640 1 -- 172.21.15.150:0/1467867731 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9078003600 con 0x7f9088072ea0 2024-09-17T04:57:26.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.809+0000 7f907effd640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9068073230 0x7f9068075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:26.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.809+0000 7f907effd640 1 -- 172.21.15.150:0/1467867731 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7f9078094940 con 0x7f9088072ea0 2024-09-17T04:57:26.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.809+0000 7f907ffff640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9068073230 0x7f9068075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:26.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.809+0000 7f907ffff640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9068073230 0x7f9068075740 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f90700af000 tx=0x7f907001b060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:26.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:26.813+0000 7f907effd640 1 -- 172.21.15.150:0/1467867731 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9078014030 con 0x7f9088072ea0 2024-09-17T04:57:27.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.065+0000 7f908cf68640 1 -- 172.21.15.150:0/1467867731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90880c40f0 con 0x7f9088072ea0 2024-09-17T04:57:27.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.065+0000 7f907effd640 1 -- 172.21.15.150:0/1467867731 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6536 (secure 0 0 0) 0x7f907805f380 con 0x7f9088072ea0 2024-09-17T04:57:27.065 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:27.065 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":56,"btime":"2024-09-17T04:56:24:353754+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5404,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:56:13.913286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631]},"id":2}]} 2024-09-17T04:57:27.065 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 56 2024-09-17T04:57:27.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.069+0000 7f908cf68640 1 -- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9068073230 msgr2=0x7f9068075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:27.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.069+0000 7f908cf68640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9068073230 0x7f9068075740 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f90700af000 tx=0x7f907001b060 comp rx=0 tx=0).stop 2024-09-17T04:57:27.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.069+0000 7f908cf68640 1 -- 172.21.15.150:0/1467867731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9088072ea0 msgr2=0x7f908815d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:27.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.069+0000 7f908cf68640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9088072ea0 0x7f908815d170 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f907800c970 tx=0x7f907800ce30 comp rx=0 tx=0).stop 2024-09-17T04:57:27.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.069+0000 7f908cf68640 1 -- 172.21.15.150:0/1467867731 shutdown_connections 2024-09-17T04:57:27.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.069+0000 7f908cf68640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9068073230 0x7f9068075740 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.069+0000 7f908cf68640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f90880c97a0 0x7f90881f0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.069+0000 7f908cf68640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9088157be0 0x7f90881f13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.069+0000 7f908cf68640 1 --2- 172.21.15.150:0/1467867731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9088072ea0 0x7f908815d170 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.069+0000 7f908cf68640 1 -- 172.21.15.150:0/1467867731 >> 172.21.15.150:0/1467867731 conn(0x7f90880abb00 msgr2=0x7f90880c79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:27.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.069+0000 7f908cf68640 1 -- 172.21.15.150:0/1467867731 shutdown_connections 2024-09-17T04:57:27.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.069+0000 7f908cf68640 1 -- 172.21.15.150:0/1467867731 wait complete. 2024-09-17T04:57:27.089 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-17T04:57:27.089 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:57:27.089 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:57:27.102 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:57:27.103 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:27.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 -- 172.21.15.150:0/566337120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d80074e60 msgr2=0x7f8d80156be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:27.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 --2- 172.21.15.150:0/566337120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d80074e60 0x7f8d80156be0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8d800c9e00 tx=0x7f8d7801cd80 comp rx=0 tx=0).stop 2024-09-17T04:57:27.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 -- 172.21.15.150:0/566337120 shutdown_connections 2024-09-17T04:57:27.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 --2- 172.21.15.150:0/566337120 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d80074e60 0x7f8d80156be0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 --2- 172.21.15.150:0/566337120 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d800c3900 0x7f8d80074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 --2- 172.21.15.150:0/566337120 >> v2:172.21.15.176:3300/0 conn(0x7f8d800c4250 0x7f8d80153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 -- 172.21.15.150:0/566337120 >> 172.21.15.150:0/566337120 conn(0x7f8d8006e360 msgr2=0x7f8d8006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:27.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 -- 172.21.15.150:0/566337120 shutdown_connections 2024-09-17T04:57:27.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 -- 172.21.15.150:0/566337120 wait complete. 2024-09-17T04:57:27.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 Processor -- start 2024-09-17T04:57:27.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 -- start start 2024-09-17T04:57:27.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d80074e60 0x7f8d8015a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d800c3900 0x7f8d801f10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d800c4250 0x7f8d800aa140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80072f30 con 0x7f8d80074e60 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8d80072db0 con 0x7f8d800c4250 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d87c7e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8d8015ff80 con 0x7f8d800c3900 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d85a1a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d80074e60 0x7f8d8015a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d85a1a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d80074e60 0x7f8d8015a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38450/0 (socket says 172.21.15.150:38450) 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d85a1a640 1 -- 172.21.15.150:0/1111100644 learned_addr learned my addr 172.21.15.150:0/1111100644 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d85219640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d800c3900 0x7f8d801f10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.313+0000 7f8d8621b640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d800c4250 0x7f8d800aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d85a1a640 1 -- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d800c3900 msgr2=0x7f8d801f10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d85a1a640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d800c3900 0x7f8d801f10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d85a1a640 1 -- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d800c4250 msgr2=0x7f8d800aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:27.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d85a1a640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d800c4250 0x7f8d800aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d85a1a640 1 -- 172.21.15.150:0/1111100644 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d80156c10 con 0x7f8d80074e60 2024-09-17T04:57:27.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d85219640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d800c3900 0x7f8d801f10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:57:27.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d85a1a640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d80074e60 0x7f8d8015a190 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0058d0 tx=0x7f8d7c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:27.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d767fc640 1 -- 172.21.15.150:0/1111100644 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c017070 con 0x7f8d80074e60 2024-09-17T04:57:27.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d767fc640 1 -- 172.21.15.150:0/1111100644 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d7c009d20 con 0x7f8d80074e60 2024-09-17T04:57:27.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d87c7e640 1 -- 172.21.15.150:0/1111100644 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d80156000 con 0x7f8d80074e60 2024-09-17T04:57:27.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d767fc640 1 -- 172.21.15.150:0/1111100644 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c0106c0 con 0x7f8d80074e60 2024-09-17T04:57:27.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d87c7e640 1 -- 172.21.15.150:0/1111100644 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8015a6d0 con 0x7f8d80074e60 2024-09-17T04:57:27.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d87c7e640 1 -- 172.21.15.150:0/1111100644 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d80074920 con 0x7f8d80074e60 2024-09-17T04:57:27.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d767fc640 1 -- 172.21.15.150:0/1111100644 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8d7c01a070 con 0x7f8d80074e60 2024-09-17T04:57:27.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d767fc640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8d64073230 0x7f8d64075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:27.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d767fc640 1 -- 172.21.15.150:0/1111100644 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7f8d7c094900 con 0x7f8d80074e60 2024-09-17T04:57:27.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.317+0000 7f8d85219640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8d64073230 0x7f8d64075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:27.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.321+0000 7f8d85219640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8d64073230 0x7f8d64075740 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8d70005fd0 tx=0x7f8d70005ea0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:27.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.325+0000 7f8d767fc640 1 -- 172.21.15.150:0/1111100644 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8d7c05ed00 con 0x7f8d80074e60 2024-09-17T04:57:27.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.553+0000 7f8d87c7e640 1 -- 172.21.15.150:0/1111100644 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d800a80d0 con 0x7f8d80074e60 2024-09-17T04:57:27.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.553+0000 7f8d767fc640 1 -- 172.21.15.150:0/1111100644 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6536 (secure 0 0 0) 0x7f8d7c063730 con 0x7f8d80074e60 2024-09-17T04:57:27.552 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:27.552 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":56,"btime":"2024-09-17T04:56:24:353754+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5404,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:56:13.913286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631]},"id":2}]} 2024-09-17T04:57:27.552 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 56 2024-09-17T04:57:27.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.557+0000 7f8d87c7e640 1 -- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8d64073230 msgr2=0x7f8d64075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:27.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.557+0000 7f8d87c7e640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8d64073230 0x7f8d64075740 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8d70005fd0 tx=0x7f8d70005ea0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.557+0000 7f8d87c7e640 1 -- 172.21.15.150:0/1111100644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d80074e60 msgr2=0x7f8d8015a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:27.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.557+0000 7f8d87c7e640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d80074e60 0x7f8d8015a190 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0058d0 tx=0x7f8d7c00dde0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.557+0000 7f8d87c7e640 1 -- 172.21.15.150:0/1111100644 shutdown_connections 2024-09-17T04:57:27.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.557+0000 7f8d87c7e640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8d64073230 0x7f8d64075740 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.557+0000 7f8d87c7e640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8d800c4250 0x7f8d800aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.557+0000 7f8d87c7e640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8d800c3900 0x7f8d801f10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.557+0000 7f8d87c7e640 1 --2- 172.21.15.150:0/1111100644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d80074e60 0x7f8d8015a190 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:27.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.557+0000 7f8d87c7e640 1 -- 172.21.15.150:0/1111100644 >> 172.21.15.150:0/1111100644 conn(0x7f8d8006e360 msgr2=0x7f8d801502a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:27.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.557+0000 7f8d87c7e640 1 -- 172.21.15.150:0/1111100644 shutdown_connections 2024-09-17T04:57:27.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:27.557+0000 7f8d87c7e640 1 -- 172.21.15.150:0/1111100644 wait complete. 2024-09-17T04:57:27.579 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:57:29.580 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:29.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.737+0000 7f93df92b640 1 -- 172.21.15.150:0/2139616415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93d80c8870 msgr2=0x7f93d80c7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:29.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.737+0000 7f93df92b640 1 --2- 172.21.15.150:0/2139616415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93d80c8870 0x7f93d80c7190 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009c80 tx=0x7f93d001cd60 comp rx=0 tx=0).stop 2024-09-17T04:57:29.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.737+0000 7f93df92b640 1 -- 172.21.15.150:0/2139616415 shutdown_connections 2024-09-17T04:57:29.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.737+0000 7f93df92b640 1 --2- 172.21.15.150:0/2139616415 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93d80c8870 0x7f93d80c7190 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:29.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.737+0000 7f93df92b640 1 --2- 172.21.15.150:0/2139616415 >> v2:172.21.15.176:3300/0 conn(0x7f93d81540b0 0x7f93d80c8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:29.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.737+0000 7f93df92b640 1 -- 172.21.15.150:0/2139616415 >> 172.21.15.150:0/2139616415 conn(0x7f93d80aab60 msgr2=0x7f93d80aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:29.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.737+0000 7f93df92b640 1 -- 172.21.15.150:0/2139616415 shutdown_connections 2024-09-17T04:57:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.737+0000 7f93df92b640 1 -- 172.21.15.150:0/2139616415 wait complete. 2024-09-17T04:57:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93df92b640 1 Processor -- start 2024-09-17T04:57:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93df92b640 1 -- start start 2024-09-17T04:57:29.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93df92b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d80c7eb0 0x7f93d814d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:29.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93df92b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93d80c8870 0x7f93d80aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:29.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93df92b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93d81540b0 0x7f93d8147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:29.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93df92b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f93d80c7ae0 con 0x7f93d80c8870 2024-09-17T04:57:29.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93df92b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f93d80c7c60 con 0x7f93d80c7eb0 2024-09-17T04:57:29.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93df92b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f93d8156540 con 0x7f93d81540b0 2024-09-17T04:57:29.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93dcec6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93d80c8870 0x7f93d80aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93dcec6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93d80c8870 0x7f93d80aeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49120/0 (socket says 172.21.15.150:49120) 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93dcec6640 1 -- 172.21.15.150:0/2305975899 learned_addr learned my addr 172.21.15.150:0/2305975899 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93dd6c7640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d80c7eb0 0x7f93d814d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93dcec6640 1 -- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93d81540b0 msgr2=0x7f93d8147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93dcec6640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93d81540b0 0x7f93d8147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93dcec6640 1 -- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d80c7eb0 msgr2=0x7f93d814d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93dcec6640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d80c7eb0 0x7f93d814d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93dcec6640 1 -- 172.21.15.150:0/2305975899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d80ae420 con 0x7f93d80c8870 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93dd6c7640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d80c7eb0 0x7f93d814d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.741+0000 7f93dcec6640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93d80c8870 0x7f93d80aeaa0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f93c800d720 tx=0x7f93c800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.745+0000 7f93cdffb640 1 -- 172.21.15.150:0/2305975899 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8017070 con 0x7f93d80c8870 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.745+0000 7f93df92b640 1 -- 172.21.15.150:0/2305975899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d814c8f0 con 0x7f93d80c8870 2024-09-17T04:57:29.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.745+0000 7f93df92b640 1 -- 172.21.15.150:0/2305975899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d80ace70 con 0x7f93d80c8870 2024-09-17T04:57:29.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.745+0000 7f93cdffb640 1 -- 172.21.15.150:0/2305975899 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93c8009cc0 con 0x7f93d80c8870 2024-09-17T04:57:29.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.745+0000 7f93cdffb640 1 -- 172.21.15.150:0/2305975899 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8005280 con 0x7f93d80c8870 2024-09-17T04:57:29.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.745+0000 7f93df92b640 1 -- 172.21.15.150:0/2305975899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b80052a0 con 0x7f93d80c8870 2024-09-17T04:57:29.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.745+0000 7f93cdffb640 1 -- 172.21.15.150:0/2305975899 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f93c8005420 con 0x7f93d80c8870 2024-09-17T04:57:29.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.745+0000 7f93cdffb640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f93bc073230 0x7f93bc075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:29.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.745+0000 7f93cdffb640 1 -- 172.21.15.150:0/2305975899 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7f93c80946f0 con 0x7f93d80c8870 2024-09-17T04:57:29.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.745+0000 7f93dd6c7640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f93bc073230 0x7f93bc075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:29.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.749+0000 7f93dd6c7640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f93bc073230 0x7f93bc075740 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f93d400a850 tx=0x7f93d4008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:29.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.753+0000 7f93cdffb640 1 -- 172.21.15.150:0/2305975899 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f93c805f140 con 0x7f93d80c8870 2024-09-17T04:57:29.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.977+0000 7f93df92b640 1 -- 172.21.15.150:0/2305975899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93b8005860 con 0x7f93d80c8870 2024-09-17T04:57:29.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.977+0000 7f93cdffb640 1 -- 172.21.15.150:0/2305975899 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6536 (secure 0 0 0) 0x7f93c805ea40 con 0x7f93d80c8870 2024-09-17T04:57:29.976 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:29.976 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":56,"btime":"2024-09-17T04:56:24:353754+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5404,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:56:13.913286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631]},"id":2}]} 2024-09-17T04:57:29.976 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 56 2024-09-17T04:57:29.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.981+0000 7f93df92b640 1 -- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f93bc073230 msgr2=0x7f93bc075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:29.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.981+0000 7f93df92b640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f93bc073230 0x7f93bc075740 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f93d400a850 tx=0x7f93d4008040 comp rx=0 tx=0).stop 2024-09-17T04:57:29.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.981+0000 7f93df92b640 1 -- 172.21.15.150:0/2305975899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93d80c8870 msgr2=0x7f93d80aeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:29.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.981+0000 7f93df92b640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93d80c8870 0x7f93d80aeaa0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f93c800d720 tx=0x7f93c800dbe0 comp rx=0 tx=0).stop 2024-09-17T04:57:29.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.981+0000 7f93df92b640 1 -- 172.21.15.150:0/2305975899 shutdown_connections 2024-09-17T04:57:29.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.981+0000 7f93df92b640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f93bc073230 0x7f93bc075740 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:29.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.981+0000 7f93df92b640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93d81540b0 0x7f93d8147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:29.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.981+0000 7f93df92b640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93d80c8870 0x7f93d80aeaa0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:29.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.981+0000 7f93df92b640 1 --2- 172.21.15.150:0/2305975899 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d80c7eb0 0x7f93d814d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:29.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.981+0000 7f93df92b640 1 -- 172.21.15.150:0/2305975899 >> 172.21.15.150:0/2305975899 conn(0x7f93d80aab60 msgr2=0x7f93d806df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:29.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.981+0000 7f93df92b640 1 -- 172.21.15.150:0/2305975899 shutdown_connections 2024-09-17T04:57:29.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:29.981+0000 7f93df92b640 1 -- 172.21.15.150:0/2305975899 wait complete. 2024-09-17T04:57:29.998 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:57:30.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:57:30.299 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:57:31.999 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:32.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.169+0000 7fc4ff63f640 1 -- 172.21.15.150:0/3061964534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4e4003a10 msgr2=0x7fc4e4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:32.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.169+0000 7fc4ff63f640 1 --2- 172.21.15.150:0/3061964534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4e4003a10 0x7fc4e4003e80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009b60 tx=0x7fc4f401d290 comp rx=0 tx=0).stop 2024-09-17T04:57:32.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.169+0000 7fc4fdbdc640 1 -- 172.21.15.150:0/3061964534 reap_dead start 2024-09-17T04:57:32.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.169+0000 7fc4ff63f640 1 -- 172.21.15.150:0/3061964534 shutdown_connections 2024-09-17T04:57:32.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.169+0000 7fc4ff63f640 1 -- 172.21.15.150:0/3061964534 >> 172.21.15.150:0/3061964534 conn(0x7fc4f806e360 msgr2=0x7fc4f806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:32.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.169+0000 7fc4ff63f640 1 -- 172.21.15.150:0/3061964534 shutdown_connections 2024-09-17T04:57:32.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.169+0000 7fc4ff63f640 1 -- 172.21.15.150:0/3061964534 wait complete. 2024-09-17T04:57:32.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4ff63f640 1 Processor -- start 2024-09-17T04:57:32.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4ff63f640 1 -- start start 2024-09-17T04:57:32.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4ff63f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4e4003a10 0x7fc4f814f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:32.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4ff63f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f80c0db0 0x7fc4f814f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:32.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4ff63f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4f81f1610 0x7fc4f81f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:32.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4ff63f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f8072f30 con 0x7fc4f80c0db0 2024-09-17T04:57:32.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4ff63f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f8072db0 con 0x7fc4e4003a10 2024-09-17T04:57:32.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4ff63f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc4f815ef80 con 0x7fc4f81f1610 2024-09-17T04:57:32.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4fcbda640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f80c0db0 0x7fc4f814f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:32.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4fcbda640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f80c0db0 0x7fc4f814f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49132/0 (socket says 172.21.15.150:49132) 2024-09-17T04:57:32.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4fcbda640 1 -- 172.21.15.150:0/1614100530 learned_addr learned my addr 172.21.15.150:0/1614100530 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:32.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4fd3db640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4e4003a10 0x7fc4f814f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:32.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4fdbdc640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4f81f1610 0x7fc4f81f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:32.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4fcbda640 1 -- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4f81f1610 msgr2=0x7fc4f81f1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:32.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4fcbda640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4f81f1610 0x7fc4f81f1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:32.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4fcbda640 1 -- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4e4003a10 msgr2=0x7fc4f814f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:32.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4fcbda640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4e4003a10 0x7fc4f814f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:32.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4fcbda640 1 -- 172.21.15.150:0/1614100530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f80a8490 con 0x7fc4f80c0db0 2024-09-17T04:57:32.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4fcbda640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f80c0db0 0x7fc4f814f6a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8011850 tx=0x7fc4e8011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:32.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4edffb640 1 -- 172.21.15.150:0/1614100530 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8004460 con 0x7fc4f80c0db0 2024-09-17T04:57:32.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4edffb640 1 -- 172.21.15.150:0/1614100530 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4e8016070 con 0x7fc4f80c0db0 2024-09-17T04:57:32.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4edffb640 1 -- 172.21.15.150:0/1614100530 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e800fba0 con 0x7fc4f80c0db0 2024-09-17T04:57:32.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4ff63f640 1 -- 172.21.15.150:0/1614100530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f8150470 con 0x7fc4f80c0db0 2024-09-17T04:57:32.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.173+0000 7fc4ff63f640 1 -- 172.21.15.150:0/1614100530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f80bde70 con 0x7fc4f80c0db0 2024-09-17T04:57:32.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.177+0000 7fc4ff63f640 1 -- 172.21.15.150:0/1614100530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4f8074920 con 0x7fc4f80c0db0 2024-09-17T04:57:32.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.177+0000 7fc4edffb640 1 -- 172.21.15.150:0/1614100530 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc4e8018030 con 0x7fc4f80c0db0 2024-09-17T04:57:32.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.177+0000 7fc4edffb640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc4dc073200 0x7fc4dc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:32.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.177+0000 7fc4edffb640 1 -- 172.21.15.150:0/1614100530 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7fc4e8094ec0 con 0x7fc4f80c0db0 2024-09-17T04:57:32.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.177+0000 7fc4fd3db640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc4dc073200 0x7fc4dc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:32.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.177+0000 7fc4fd3db640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc4dc073200 0x7fc4dc075710 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009b60 tx=0x7fc4f401a030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:32.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.181+0000 7fc4edffb640 1 -- 172.21.15.150:0/1614100530 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc4e805f900 con 0x7fc4f80c0db0 2024-09-17T04:57:32.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.429+0000 7fc4ff63f640 1 -- 172.21.15.150:0/1614100530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4f80c4e20 con 0x7fc4f80c0db0 2024-09-17T04:57:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.429+0000 7fc4edffb640 1 -- 172.21.15.150:0/1614100530 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6536 (secure 0 0 0) 0x7fc4e805f200 con 0x7fc4f80c0db0 2024-09-17T04:57:32.429 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:32.429 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":56,"btime":"2024-09-17T04:56:24:353754+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5404,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:56:13.913286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631]},"id":2}]} 2024-09-17T04:57:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 56 2024-09-17T04:57:32.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.433+0000 7fc4ff63f640 1 -- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc4dc073200 msgr2=0x7fc4dc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:32.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.433+0000 7fc4ff63f640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc4dc073200 0x7fc4dc075710 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009b60 tx=0x7fc4f401a030 comp rx=0 tx=0).stop 2024-09-17T04:57:32.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.433+0000 7fc4ff63f640 1 -- 172.21.15.150:0/1614100530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f80c0db0 msgr2=0x7fc4f814f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:32.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.433+0000 7fc4ff63f640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f80c0db0 0x7fc4f814f6a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8011850 tx=0x7fc4e8011d10 comp rx=0 tx=0).stop 2024-09-17T04:57:32.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.433+0000 7fc4ff63f640 1 -- 172.21.15.150:0/1614100530 shutdown_connections 2024-09-17T04:57:32.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.433+0000 7fc4ff63f640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc4dc073200 0x7fc4dc075710 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:32.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.433+0000 7fc4ff63f640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4f81f1610 0x7fc4f81f1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:32.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.433+0000 7fc4ff63f640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f80c0db0 0x7fc4f814f6a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:32.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.433+0000 7fc4ff63f640 1 --2- 172.21.15.150:0/1614100530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4e4003a10 0x7fc4f814f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:32.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.433+0000 7fc4ff63f640 1 -- 172.21.15.150:0/1614100530 >> 172.21.15.150:0/1614100530 conn(0x7fc4f806e360 msgr2=0x7fc4f81527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:32.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.433+0000 7fc4ff63f640 1 -- 172.21.15.150:0/1614100530 shutdown_connections 2024-09-17T04:57:32.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:32.433+0000 7fc4ff63f640 1 -- 172.21.15.150:0/1614100530 wait complete. 2024-09-17T04:57:32.451 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:57:34.452 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:34.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.625+0000 7f3bebaa2640 1 -- 172.21.15.150:0/1851033036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be40c8fc0 msgr2=0x7f3be40c20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:34.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.625+0000 7f3bebaa2640 1 --2- 172.21.15.150:0/1851033036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be40c8fc0 0x7f3be40c20a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3be40c5f20 tx=0x7f3bdc01cb50 comp rx=0 tx=0).stop 2024-09-17T04:57:34.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.625+0000 7f3bebaa2640 1 -- 172.21.15.150:0/1851033036 shutdown_connections 2024-09-17T04:57:34.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.625+0000 7f3bebaa2640 1 --2- 172.21.15.150:0/1851033036 >> v2:172.21.15.176:3300/0 conn(0x7f3be40c35b0 0x7f3be40c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:34.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.625+0000 7f3bebaa2640 1 --2- 172.21.15.150:0/1851033036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be40c8fc0 0x7f3be40c20a0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:34.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.625+0000 7f3bebaa2640 1 -- 172.21.15.150:0/1851033036 >> 172.21.15.150:0/1851033036 conn(0x7f3be4074b80 msgr2=0x7f3be406f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:34.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.625+0000 7f3bebaa2640 1 -- 172.21.15.150:0/1851033036 shutdown_connections 2024-09-17T04:57:34.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.625+0000 7f3bebaa2640 1 -- 172.21.15.150:0/1851033036 wait complete. 2024-09-17T04:57:34.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.625+0000 7f3bebaa2640 1 Processor -- start 2024-09-17T04:57:34.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.625+0000 7f3bebaa2640 1 -- start start 2024-09-17T04:57:34.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bebaa2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3be40c35b0 0x7f3be4159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:34.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bebaa2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3be40c8fc0 0x7f3be4155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:34.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bebaa2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be41568a0 0x7f3be41454c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:34.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bebaa2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3be40c6b70 con 0x7f3be41568a0 2024-09-17T04:57:34.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bebaa2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3be40c69f0 con 0x7f3be40c35b0 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bebaa2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3be40b0fd0 con 0x7f3be40c8fc0 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bea03f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be41568a0 0x7f3be41454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bea03f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be41568a0 0x7f3be41454c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49148/0 (socket says 172.21.15.150:49148) 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bea03f640 1 -- 172.21.15.150:0/57549572 learned_addr learned my addr 172.21.15.150:0/57549572 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3be903d640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3be40c8fc0 0x7f3be4155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3be983e640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3be40c35b0 0x7f3be4159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bea03f640 1 -- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3be40c8fc0 msgr2=0x7f3be4155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bea03f640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3be40c8fc0 0x7f3be4155390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bea03f640 1 -- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3be40c35b0 msgr2=0x7f3be4159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bea03f640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3be40c35b0 0x7f3be4159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bea03f640 1 -- 172.21.15.150:0/57549572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be4155dc0 con 0x7f3be41568a0 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3be983e640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3be40c35b0 0x7f3be4159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:57:34.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bea03f640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be41568a0 0x7f3be41454c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc098650 tx=0x7f3bdc003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:34.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bda7fc640 1 -- 172.21.15.150:0/57549572 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc0b2040 con 0x7f3be41568a0 2024-09-17T04:57:34.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bda7fc640 1 -- 172.21.15.150:0/57549572 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bdc0bb070 con 0x7f3be41568a0 2024-09-17T04:57:34.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bebaa2640 1 -- 172.21.15.150:0/57549572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be4158650 con 0x7f3be41568a0 2024-09-17T04:57:34.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bda7fc640 1 -- 172.21.15.150:0/57549572 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc004330 con 0x7f3be41568a0 2024-09-17T04:57:34.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bebaa2640 1 -- 172.21.15.150:0/57549572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be4159670 con 0x7f3be41568a0 2024-09-17T04:57:34.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.629+0000 7f3bebaa2640 1 -- 172.21.15.150:0/57549572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3be40b14a0 con 0x7f3be41568a0 2024-09-17T04:57:34.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.633+0000 7f3bda7fc640 1 -- 172.21.15.150:0/57549572 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3bdc0044d0 con 0x7f3be41568a0 2024-09-17T04:57:34.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.633+0000 7f3bda7fc640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3bc8073200 0x7f3bc8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:34.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.633+0000 7f3bda7fc640 1 -- 172.21.15.150:0/57549572 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7f3bdc12f1f0 con 0x7f3be41568a0 2024-09-17T04:57:34.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.633+0000 7f3be983e640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3bc8073200 0x7f3bc8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:34.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.633+0000 7f3be983e640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3bc8073200 0x7f3bc8075710 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3be0001230 tx=0x7f3be0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:34.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.637+0000 7f3bda7fc640 1 -- 172.21.15.150:0/57549572 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3bdc016770 con 0x7f3be41568a0 2024-09-17T04:57:34.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.909+0000 7f3bebaa2640 1 -- 172.21.15.150:0/57549572 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3be4157790 con 0x7f3be41568a0 2024-09-17T04:57:34.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.909+0000 7f3bda7fc640 1 -- 172.21.15.150:0/57549572 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6536 (secure 0 0 0) 0x7f3bdc0f9ba0 con 0x7f3be41568a0 2024-09-17T04:57:34.909 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:34.909 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":56,"btime":"2024-09-17T04:56:24:353754+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5404,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:56:13.913286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631]},"id":2}]} 2024-09-17T04:57:34.910 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 56 2024-09-17T04:57:34.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.913+0000 7f3bebaa2640 1 -- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3bc8073200 msgr2=0x7f3bc8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:34.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.913+0000 7f3bebaa2640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3bc8073200 0x7f3bc8075710 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3be0001230 tx=0x7f3be0009040 comp rx=0 tx=0).stop 2024-09-17T04:57:34.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.913+0000 7f3bebaa2640 1 -- 172.21.15.150:0/57549572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be41568a0 msgr2=0x7f3be41454c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:34.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.913+0000 7f3bebaa2640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be41568a0 0x7f3be41454c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc098650 tx=0x7f3bdc003f40 comp rx=0 tx=0).stop 2024-09-17T04:57:34.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.913+0000 7f3bebaa2640 1 -- 172.21.15.150:0/57549572 shutdown_connections 2024-09-17T04:57:34.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.913+0000 7f3bebaa2640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3bc8073200 0x7f3bc8075710 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:34.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.913+0000 7f3bebaa2640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be41568a0 0x7f3be41454c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:34.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.913+0000 7f3bebaa2640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3be40c8fc0 0x7f3be4155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:34.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.913+0000 7f3bebaa2640 1 --2- 172.21.15.150:0/57549572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3be40c35b0 0x7f3be4159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:34.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.913+0000 7f3bebaa2640 1 -- 172.21.15.150:0/57549572 >> 172.21.15.150:0/57549572 conn(0x7f3be4074b80 msgr2=0x7f3be41614f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:34.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.913+0000 7f3bebaa2640 1 -- 172.21.15.150:0/57549572 shutdown_connections 2024-09-17T04:57:34.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:34.913+0000 7f3bebaa2640 1 -- 172.21.15.150:0/57549572 wait complete. 2024-09-17T04:57:34.933 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:57:36.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:37.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.097+0000 7fbe848e1640 1 -- 172.21.15.150:0/2233829683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe800c7eb0 msgr2=0x7fbe801540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:37.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.097+0000 7fbe848e1640 1 --2- 172.21.15.150:0/2233829683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe800c7eb0 0x7fbe801540b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009840 tx=0x7fbe7001c7e0 comp rx=0 tx=0).stop 2024-09-17T04:57:37.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.097+0000 7fbe848e1640 1 -- 172.21.15.150:0/2233829683 shutdown_connections 2024-09-17T04:57:37.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.097+0000 7fbe848e1640 1 --2- 172.21.15.150:0/2233829683 >> v2:172.21.15.176:3300/0 conn(0x7fbe80149d70 0x7fbe8014a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:37.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.097+0000 7fbe848e1640 1 --2- 172.21.15.150:0/2233829683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe800c7eb0 0x7fbe801540b0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:37.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.097+0000 7fbe848e1640 1 -- 172.21.15.150:0/2233829683 >> 172.21.15.150:0/2233829683 conn(0x7fbe800aab60 msgr2=0x7fbe800aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:37.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.097+0000 7fbe848e1640 1 -- 172.21.15.150:0/2233829683 shutdown_connections 2024-09-17T04:57:37.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.097+0000 7fbe848e1640 1 -- 172.21.15.150:0/2233829683 wait complete. 2024-09-17T04:57:37.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe848e1640 1 Processor -- start 2024-09-17T04:57:37.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe848e1640 1 -- start start 2024-09-17T04:57:37.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe848e1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe800c7eb0 0x7fbe800bdf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:37.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe848e1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe800c8330 0x7fbe80148b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:37.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe848e1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe80149d70 0x7fbe8014bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:37.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe848e1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbe801568d0 con 0x7fbe800c8330 2024-09-17T04:57:37.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe848e1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbe80156750 con 0x7fbe80149d70 2024-09-17T04:57:37.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe848e1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbe8014c900 con 0x7fbe800c7eb0 2024-09-17T04:57:37.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe7dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe800c8330 0x7fbe80148b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:37.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe7dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe800c8330 0x7fbe80148b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49170/0 (socket says 172.21.15.150:49170) 2024-09-17T04:57:37.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe7dd9b640 1 -- 172.21.15.150:0/2282359985 learned_addr learned my addr 172.21.15.150:0/2282359985 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe7e59c640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe800c7eb0 0x7fbe800bdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe7ed9d640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe80149d70 0x7fbe8014bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe7dd9b640 1 -- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe800c7eb0 msgr2=0x7fbe800bdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe7dd9b640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe800c7eb0 0x7fbe800bdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe7dd9b640 1 -- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe80149d70 msgr2=0x7fbe8014bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe7dd9b640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe80149d70 0x7fbe8014bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe7dd9b640 1 -- 172.21.15.150:0/2282359985 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe80149ae0 con 0x7fbe800c8330 2024-09-17T04:57:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe7dd9b640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe800c8330 0x7fbe80148b30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbe740058d0 tx=0x7fbe7400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe6effd640 1 -- 172.21.15.150:0/2282359985 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe74017070 con 0x7fbe800c8330 2024-09-17T04:57:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe6effd640 1 -- 172.21.15.150:0/2282359985 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe74009d20 con 0x7fbe800c8330 2024-09-17T04:57:37.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe848e1640 1 -- 172.21.15.150:0/2282359985 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe80149500 con 0x7fbe800c8330 2024-09-17T04:57:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe6effd640 1 -- 172.21.15.150:0/2282359985 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe740106c0 con 0x7fbe800c8330 2024-09-17T04:57:37.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe848e1640 1 -- 172.21.15.150:0/2282359985 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe8014c640 con 0x7fbe800c8330 2024-09-17T04:57:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.101+0000 7fbe848e1640 1 -- 172.21.15.150:0/2282359985 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe801540b0 con 0x7fbe800c8330 2024-09-17T04:57:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.105+0000 7fbe6effd640 1 -- 172.21.15.150:0/2282359985 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbe7401a070 con 0x7fbe800c8330 2024-09-17T04:57:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.105+0000 7fbe6effd640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe60073200 0x7fbe60075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.105+0000 7fbe6effd640 1 -- 172.21.15.150:0/2282359985 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7fbe74094940 con 0x7fbe800c8330 2024-09-17T04:57:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.105+0000 7fbe7e59c640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe60073200 0x7fbe60075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:37.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.105+0000 7fbe7e59c640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe60073200 0x7fbe60075710 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbe700af000 tx=0x7fbe7001b020 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:37.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.113+0000 7fbe6effd640 1 -- 172.21.15.150:0/2282359985 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbe7405f390 con 0x7fbe800c8330 2024-09-17T04:57:37.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 -- 172.21.15.150:0/2282359985 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe800af140 con 0x7fbe800c8330 2024-09-17T04:57:37.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe6effd640 1 -- 172.21.15.150:0/2282359985 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6536 (secure 0 0 0) 0x7fbe7405ec90 con 0x7fbe800c8330 2024-09-17T04:57:37.408 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:37.408 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":56,"btime":"2024-09-17T04:56:24:353754+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5404,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:56:13.913286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631]},"id":2}]} 2024-09-17T04:57:37.408 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 56 2024-09-17T04:57:37.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 -- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe60073200 msgr2=0x7fbe60075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:37.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe60073200 0x7fbe60075710 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbe700af000 tx=0x7fbe7001b020 comp rx=0 tx=0).stop 2024-09-17T04:57:37.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 -- 172.21.15.150:0/2282359985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe800c8330 msgr2=0x7fbe80148b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:37.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe800c8330 0x7fbe80148b30 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbe740058d0 tx=0x7fbe7400dde0 comp rx=0 tx=0).stop 2024-09-17T04:57:37.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 -- 172.21.15.150:0/2282359985 shutdown_connections 2024-09-17T04:57:37.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe60073200 0x7fbe60075710 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:37.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe80149d70 0x7fbe8014bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:37.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe800c8330 0x7fbe80148b30 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:37.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 --2- 172.21.15.150:0/2282359985 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe800c7eb0 0x7fbe800bdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:37.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 -- 172.21.15.150:0/2282359985 >> 172.21.15.150:0/2282359985 conn(0x7fbe800aab60 msgr2=0x7fbe80153330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:37.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 -- 172.21.15.150:0/2282359985 shutdown_connections 2024-09-17T04:57:37.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:37.409+0000 7fbe848e1640 1 -- 172.21.15.150:0/2282359985 wait complete. 2024-09-17T04:57:37.432 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:57:39.434 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:39.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.601+0000 7f426c8ee640 1 -- 172.21.15.150:0/953192820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42680c8fc0 msgr2=0x7f42680c20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:39.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.601+0000 7f426c8ee640 1 --2- 172.21.15.150:0/953192820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42680c8fc0 0x7f42680c20a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4260009c80 tx=0x7f426001cd70 comp rx=0 tx=0).stop 2024-09-17T04:57:39.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.601+0000 7f426c8ee640 1 -- 172.21.15.150:0/953192820 shutdown_connections 2024-09-17T04:57:39.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.601+0000 7f426c8ee640 1 --2- 172.21.15.150:0/953192820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42680c8fc0 0x7f42680c20a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:39.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.601+0000 7f426c8ee640 1 --2- 172.21.15.150:0/953192820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42680c35b0 0x7f42680c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:39.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f426c8ee640 1 -- 172.21.15.150:0/953192820 >> 172.21.15.150:0/953192820 conn(0x7f4268074b80 msgr2=0x7f426806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:39.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f426c8ee640 1 -- 172.21.15.150:0/953192820 shutdown_connections 2024-09-17T04:57:39.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f426c8ee640 1 -- 172.21.15.150:0/953192820 wait complete. 2024-09-17T04:57:39.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f426c8ee640 1 Processor -- start 2024-09-17T04:57:39.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f426c8ee640 1 -- start start 2024-09-17T04:57:39.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f426c8ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42680c35b0 0x7f4268152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:39.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f426c8ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42680c8fc0 0x7f4268152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f426c8ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42681568a0 0x7f426813dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f426c8ee640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f42680c6b70 con 0x7f42680c8fc0 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f426c8ee640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f42680c69f0 con 0x7f42681568a0 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f426c8ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f42680b0fd0 con 0x7f42680c35b0 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f4265d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42680c8fc0 0x7f4268152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f4265d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42680c8fc0 0x7f4268152340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55874/0 (socket says 172.21.15.150:55874) 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f4265d9b640 1 -- 172.21.15.150:0/197709597 learned_addr learned my addr 172.21.15.150:0/197709597 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f426659c640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42680c35b0 0x7f4268152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.605+0000 7f4266d9d640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42681568a0 0x7f426813dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4265d9b640 1 -- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42680c35b0 msgr2=0x7f4268152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4265d9b640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42680c35b0 0x7f4268152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4265d9b640 1 -- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42681568a0 msgr2=0x7f426813dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:39.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4265d9b640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42681568a0 0x7f426813dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:39.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4265d9b640 1 -- 172.21.15.150:0/197709597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42681513d0 con 0x7f42680c8fc0 2024-09-17T04:57:39.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4266d9d640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42681568a0 0x7f426813dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:57:39.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4265d9b640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42680c8fc0 0x7f4268152340 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f425c00e7d0 tx=0x7f425c00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:39.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4256ffd640 1 -- 172.21.15.150:0/197709597 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c00ccc0 con 0x7f42680c8fc0 2024-09-17T04:57:39.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4256ffd640 1 -- 172.21.15.150:0/197709597 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f425c010430 con 0x7f42680c8fc0 2024-09-17T04:57:39.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f426c8ee640 1 -- 172.21.15.150:0/197709597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4268150ed0 con 0x7f42680c8fc0 2024-09-17T04:57:39.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4256ffd640 1 -- 172.21.15.150:0/197709597 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c010860 con 0x7f42680c8fc0 2024-09-17T04:57:39.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f426c8ee640 1 -- 172.21.15.150:0/197709597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f426815a100 con 0x7f42680c8fc0 2024-09-17T04:57:39.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f426c8ee640 1 -- 172.21.15.150:0/197709597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42680b14c0 con 0x7f42680c8fc0 2024-09-17T04:57:39.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4256ffd640 1 -- 172.21.15.150:0/197709597 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f425c010a00 con 0x7f42680c8fc0 2024-09-17T04:57:39.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4256ffd640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4248073200 0x7f4248075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:39.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f4256ffd640 1 -- 172.21.15.150:0/197709597 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7354+0+0 (secure 0 0 0) 0x7f425c095150 con 0x7f42680c8fc0 2024-09-17T04:57:39.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.609+0000 7f426659c640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4248073200 0x7f4248075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:39.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.613+0000 7f426659c640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4248073200 0x7f4248075710 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4258007fd0 tx=0x7f4258009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:39.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.617+0000 7f4256ffd640 1 -- 172.21.15.150:0/197709597 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f42681513d0 con 0x7f42680c8fc0 2024-09-17T04:57:39.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.845+0000 7f426c8ee640 1 -- 172.21.15.150:0/197709597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4268157790 con 0x7f42680c8fc0 2024-09-17T04:57:39.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.845+0000 7f4256ffd640 1 -- 172.21.15.150:0/197709597 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6536 (secure 0 0 0) 0x7f425c05f5d0 con 0x7f42680c8fc0 2024-09-17T04:57:39.845 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:39.845 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":56,"btime":"2024-09-17T04:56:24:353754+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5404,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:56:13.913286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5631":{"gid":5631,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/284564893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":284564893},{"type":"v1","addr":"172.21.15.150:6833","nonce":284564893}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631]},"id":2}]} 2024-09-17T04:57:39.845 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 56 2024-09-17T04:57:39.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.849+0000 7f426c8ee640 1 -- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4248073200 msgr2=0x7f4248075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:39.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.849+0000 7f426c8ee640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4248073200 0x7f4248075710 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4258007fd0 tx=0x7f4258009040 comp rx=0 tx=0).stop 2024-09-17T04:57:39.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.849+0000 7f426c8ee640 1 -- 172.21.15.150:0/197709597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42680c8fc0 msgr2=0x7f4268152340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:39.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.849+0000 7f426c8ee640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42680c8fc0 0x7f4268152340 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f425c00e7d0 tx=0x7f425c00ec90 comp rx=0 tx=0).stop 2024-09-17T04:57:39.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.849+0000 7f426c8ee640 1 -- 172.21.15.150:0/197709597 shutdown_connections 2024-09-17T04:57:39.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.849+0000 7f426c8ee640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4248073200 0x7f4248075710 secure :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4258007fd0 tx=0x7f4258009040 comp rx=0 tx=0).stop 2024-09-17T04:57:39.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.849+0000 7f426c8ee640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42681568a0 0x7f426813dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:39.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.849+0000 7f426c8ee640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42680c8fc0 0x7f4268152340 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:39.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.849+0000 7f426c8ee640 1 --2- 172.21.15.150:0/197709597 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42680c35b0 0x7f4268152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:39.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.849+0000 7f426c8ee640 1 -- 172.21.15.150:0/197709597 >> 172.21.15.150:0/197709597 conn(0x7f4268074b80 msgr2=0x7f4268161510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:39.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.849+0000 7f426c8ee640 1 -- 172.21.15.150:0/197709597 shutdown_connections 2024-09-17T04:57:39.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:39.849+0000 7f426c8ee640 1 -- 172.21.15.150:0/197709597 wait complete. 2024-09-17T04:57:39.866 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:57:41.867 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:42.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 -- 172.21.15.150:0/566669392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31f00051b0 msgr2=0x7f31f000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:42.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 --2- 172.21.15.150:0/566669392 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f31f00051b0 0x7f31f000b9d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3208075610 tx=0x7f320001d110 comp rx=0 tx=0).stop 2024-09-17T04:57:42.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f32077fe640 1 -- 172.21.15.150:0/566669392 reap_dead start 2024-09-17T04:57:42.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 -- 172.21.15.150:0/566669392 shutdown_connections 2024-09-17T04:57:42.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 -- 172.21.15.150:0/566669392 >> 172.21.15.150:0/566669392 conn(0x7f32080ca460 msgr2=0x7f32080c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:42.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 -- 172.21.15.150:0/566669392 shutdown_connections 2024-09-17T04:57:42.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 -- 172.21.15.150:0/566669392 wait complete. 2024-09-17T04:57:42.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 Processor -- start 2024-09-17T04:57:42.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 -- start start 2024-09-17T04:57:42.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32080c31b0 0x7f32081e0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:42.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32081e1460 0x7f3208146cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:42.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32081e1870 0x7f3208147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:42.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f32080c62d0 con 0x7f32081e1460 2024-09-17T04:57:42.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f32080c51f0 con 0x7f32081e1870 2024-09-17T04:57:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.057+0000 7f320d4e7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f32080c65c0 con 0x7f32080c31b0 2024-09-17T04:57:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f32067fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32081e1460 0x7f3208146cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f32067fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32081e1460 0x7f3208146cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55882/0 (socket says 172.21.15.150:55882) 2024-09-17T04:57:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f32067fc640 1 -- 172.21.15.150:0/447890019 learned_addr learned my addr 172.21.15.150:0/447890019 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f3206ffd640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32080c31b0 0x7f32081e0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f32077fe640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32081e1870 0x7f3208147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f32067fc640 1 -- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32080c31b0 msgr2=0x7f32081e0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f32067fc640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32080c31b0 0x7f32081e0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f32067fc640 1 -- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32081e1870 msgr2=0x7f3208147230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f32067fc640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32081e1870 0x7f3208147230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f32067fc640 1 -- 172.21.15.150:0/447890019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32081f08b0 con 0x7f32081e1460 2024-09-17T04:57:42.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f32077fe640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32081e1870 0x7f3208147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:57:42.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f32067fc640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32081e1460 0x7f3208146cf0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f31fc001060 tx=0x7f31fc00ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:42.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f31ef7fe640 1 -- 172.21.15.150:0/447890019 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc015070 con 0x7f32081e1460 2024-09-17T04:57:42.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f320d4e7640 1 -- 172.21.15.150:0/447890019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32081478e0 con 0x7f32081e1460 2024-09-17T04:57:42.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f31ef7fe640 1 -- 172.21.15.150:0/447890019 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31fc00ca00 con 0x7f32081e1460 2024-09-17T04:57:42.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f31ef7fe640 1 -- 172.21.15.150:0/447890019 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc00ce50 con 0x7f32081e1460 2024-09-17T04:57:42.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f320d4e7640 1 -- 172.21.15.150:0/447890019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32081552b0 con 0x7f32081e1460 2024-09-17T04:57:42.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f320d4e7640 1 -- 172.21.15.150:0/447890019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f320814ea10 con 0x7f32081e1460 2024-09-17T04:57:42.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f31ef7fe640 1 -- 172.21.15.150:0/447890019 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f31fc018070 con 0x7f32081e1460 2024-09-17T04:57:42.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f31ef7fe640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31f40733a0 0x7f31f40758b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:42.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.061+0000 7f31ef7fe640 1 -- 172.21.15.150:0/447890019 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f31fc094d20 con 0x7f32081e1460 2024-09-17T04:57:42.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.065+0000 7f3206ffd640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31f40733a0 0x7f31f40758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:42.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.065+0000 7f3206ffd640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31f40733a0 0x7f31f40758b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f31f8005d10 tx=0x7f31f8005c80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.069+0000 7f31ef7fe640 1 -- 172.21.15.150:0/447890019 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f31fc05f720 con 0x7f32081e1460 2024-09-17T04:57:42.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.337+0000 7f320d4e7640 1 -- 172.21.15.150:0/447890019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3208070ac0 con 0x7f32081e1460 2024-09-17T04:57:42.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.337+0000 7f31ef7fe640 1 -- 172.21.15.150:0/447890019 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5708 (secure 0 0 0) 0x7f31fc05f020 con 0x7f32081e1460 2024-09-17T04:57:42.337 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:42.337 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":58,"btime":"2024-09-17T04:57:41:915078+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:41.223912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:57:42.337 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 58 2024-09-17T04:57:42.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.341+0000 7f320d4e7640 1 -- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31f40733a0 msgr2=0x7f31f40758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.341+0000 7f320d4e7640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31f40733a0 0x7f31f40758b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f31f8005d10 tx=0x7f31f8005c80 comp rx=0 tx=0).stop 2024-09-17T04:57:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.341+0000 7f320d4e7640 1 -- 172.21.15.150:0/447890019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32081e1460 msgr2=0x7f3208146cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.341+0000 7f320d4e7640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32081e1460 0x7f3208146cf0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f31fc001060 tx=0x7f31fc00ea90 comp rx=0 tx=0).stop 2024-09-17T04:57:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.341+0000 7f320d4e7640 1 -- 172.21.15.150:0/447890019 shutdown_connections 2024-09-17T04:57:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.341+0000 7f320d4e7640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f31f40733a0 0x7f31f40758b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.341+0000 7f320d4e7640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f32081e1870 0x7f3208147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.341+0000 7f320d4e7640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32081e1460 0x7f3208146cf0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.341+0000 7f320d4e7640 1 --2- 172.21.15.150:0/447890019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f32080c31b0 0x7f32081e0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.341+0000 7f320d4e7640 1 -- 172.21.15.150:0/447890019 >> 172.21.15.150:0/447890019 conn(0x7f32080ca460 msgr2=0x7f3208071ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.341+0000 7f320d4e7640 1 -- 172.21.15.150:0/447890019 shutdown_connections 2024-09-17T04:57:42.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.341+0000 7f320d4e7640 1 -- 172.21.15.150:0/447890019 wait complete. 2024-09-17T04:57:42.359 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-17T04:57:42.359 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T04:57:42.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:42.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.517+0000 7f9b4db71640 1 -- 172.21.15.150:0/1163243701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b480c4250 msgr2=0x7f9b48153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:42.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.517+0000 7f9b4db71640 1 --2- 172.21.15.150:0/1163243701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b480c4250 0x7f9b48153ee0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009930 tx=0x7f9b3801c8e0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.517+0000 7f9b4db71640 1 -- 172.21.15.150:0/1163243701 shutdown_connections 2024-09-17T04:57:42.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.517+0000 7f9b4db71640 1 --2- 172.21.15.150:0/1163243701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b480c4250 0x7f9b48153ee0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.517+0000 7f9b4db71640 1 -- 172.21.15.150:0/1163243701 >> 172.21.15.150:0/1163243701 conn(0x7f9b4806e360 msgr2=0x7f9b4806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:42.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.517+0000 7f9b4db71640 1 -- 172.21.15.150:0/1163243701 shutdown_connections 2024-09-17T04:57:42.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.517+0000 7f9b4db71640 1 -- 172.21.15.150:0/1163243701 wait complete. 2024-09-17T04:57:42.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b4db71640 1 Processor -- start 2024-09-17T04:57:42.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b4db71640 1 -- start start 2024-09-17T04:57:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b4db71640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b480c3900 0x7f9b48148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b4db71640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b480c3d10 0x7f9b4814e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b4db71640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b480c4250 0x7f9b480ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b4db71640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9b48071120 con 0x7f9b480c3900 2024-09-17T04:57:42.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b4db71640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9b48070fa0 con 0x7f9b480c4250 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b4db71640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9b48072db0 con 0x7f9b480c3d10 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b477fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b480c3900 0x7f9b48148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b477fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b480c3900 0x7f9b48148990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55904/0 (socket says 172.21.15.150:55904) 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b477fe640 1 -- 172.21.15.150:0/4095194162 learned_addr learned my addr 172.21.15.150:0/4095194162 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b3ffff640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b480c3d10 0x7f9b4814e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b477fe640 1 -- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b480c3d10 msgr2=0x7f9b4814e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b477fe640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b480c3d10 0x7f9b4814e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b477fe640 1 -- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b480c4250 msgr2=0x7f9b480ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b477fe640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b480c4250 0x7f9b480ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b477fe640 1 -- 172.21.15.150:0/4095194162 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b480abf70 con 0x7f9b480c3900 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b477fe640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b480c3900 0x7f9b48148990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9b380b0040 tx=0x7f9b380a5710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b44ff9640 1 -- 172.21.15.150:0/4095194162 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b380b2040 con 0x7f9b480c3900 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b44ff9640 1 -- 172.21.15.150:0/4095194162 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b380043c0 con 0x7f9b480c3900 2024-09-17T04:57:42.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b4db71640 1 -- 172.21.15.150:0/4095194162 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b480c2fc0 con 0x7f9b480c3900 2024-09-17T04:57:42.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b44ff9640 1 -- 172.21.15.150:0/4095194162 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38003840 con 0x7f9b480c3900 2024-09-17T04:57:42.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.521+0000 7f9b4db71640 1 -- 172.21.15.150:0/4095194162 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b4814e030 con 0x7f9b480c3900 2024-09-17T04:57:42.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.525+0000 7f9b4db71640 1 -- 172.21.15.150:0/4095194162 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b48153ee0 con 0x7f9b480c3900 2024-09-17T04:57:42.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.525+0000 7f9b44ff9640 1 -- 172.21.15.150:0/4095194162 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9b380039e0 con 0x7f9b480c3900 2024-09-17T04:57:42.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.525+0000 7f9b44ff9640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b28073200 0x7f9b28075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:42.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.525+0000 7f9b44ff9640 1 -- 172.21.15.150:0/4095194162 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f9b3812ecd0 con 0x7f9b480c3900 2024-09-17T04:57:42.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.525+0000 7f9b3ffff640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b28073200 0x7f9b28075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:42.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.525+0000 7f9b3ffff640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b28073200 0x7f9b28075710 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9b480aebe0 tx=0x7f9b30008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:42.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.529+0000 7f9b44ff9640 1 -- 172.21.15.150:0/4095194162 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9b380164d0 con 0x7f9b480c3900 2024-09-17T04:57:42.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 -- 172.21.15.150:0/4095194162 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b480a97c0 con 0x7f9b480c3900 2024-09-17T04:57:42.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b44ff9640 1 -- 172.21.15.150:0/4095194162 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5708 (secure 0 0 0) 0x7f9b38100e20 con 0x7f9b480c3900 2024-09-17T04:57:42.755 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:42.755 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":58,"btime":"2024-09-17T04:57:41:915078+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:41.223912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:57:42.755 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 58 2024-09-17T04:57:42.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 -- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b28073200 msgr2=0x7f9b28075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:42.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b28073200 0x7f9b28075710 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9b480aebe0 tx=0x7f9b30008040 comp rx=0 tx=0).stop 2024-09-17T04:57:42.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 -- 172.21.15.150:0/4095194162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b480c3900 msgr2=0x7f9b48148990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:42.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b480c3900 0x7f9b48148990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9b380b0040 tx=0x7f9b380a5710 comp rx=0 tx=0).stop 2024-09-17T04:57:42.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 -- 172.21.15.150:0/4095194162 shutdown_connections 2024-09-17T04:57:42.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b28073200 0x7f9b28075710 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b480c4250 0x7f9b480ae2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b480c3d10 0x7f9b4814e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 --2- 172.21.15.150:0/4095194162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b480c3900 0x7f9b48148990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:42.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 -- 172.21.15.150:0/4095194162 >> 172.21.15.150:0/4095194162 conn(0x7f9b4806e360 msgr2=0x7f9b48068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:42.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 -- 172.21.15.150:0/4095194162 shutdown_connections 2024-09-17T04:57:42.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:42.757+0000 7f9b4db71640 1 -- 172.21.15.150:0/4095194162 wait complete. 2024-09-17T04:57:42.779 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T04:57:41:915078+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 58, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5254": { "addr": "172.21.15.150:6835/3079958649", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 3079958649, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 3079958649, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5254, "incarnation": 47, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:55:43.893595+0000", "qdb_cluster": [ 5254 ], "qdb_leader": 5254, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5254 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 58, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5404": { "addr": "172.21.15.176:6837/3543629780", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 3543629780, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 3543629780, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5404, "incarnation": 57, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:57:41.223912+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5404 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:57:44.360 INFO:tasks.ceph.mds.d.smithi176.stderr:2024-09-17T04:57:44.358+0000 7fd75b952640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:57:44.779 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:44.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.961+0000 7f2b0f54a640 1 -- 172.21.15.150:0/1901308484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b080700b0 msgr2=0x7f2b0815dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:44.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.961+0000 7f2b0f54a640 1 --2- 172.21.15.150:0/1901308484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b080700b0 0x7f2b0815dfa0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009bc0 tx=0x7f2b0001cb10 comp rx=0 tx=0).stop 2024-09-17T04:57:44.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.961+0000 7f2b0f54a640 1 -- 172.21.15.150:0/1901308484 shutdown_connections 2024-09-17T04:57:44.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.961+0000 7f2b0f54a640 1 --2- 172.21.15.150:0/1901308484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b080700b0 0x7f2b0815dfa0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:44.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.961+0000 7f2b0f54a640 1 --2- 172.21.15.150:0/1901308484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b080745c0 0x7f2b08150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:44.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.961+0000 7f2b0f54a640 1 -- 172.21.15.150:0/1901308484 >> 172.21.15.150:0/1901308484 conn(0x7f2b080c5ae0 msgr2=0x7f2b08073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:44.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.961+0000 7f2b0f54a640 1 -- 172.21.15.150:0/1901308484 shutdown_connections 2024-09-17T04:57:44.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.961+0000 7f2b0f54a640 1 -- 172.21.15.150:0/1901308484 wait complete. 2024-09-17T04:57:44.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0f54a640 1 Processor -- start 2024-09-17T04:57:44.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0f54a640 1 -- start start 2024-09-17T04:57:44.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0f54a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b0806fca0 0x7f2b081ef5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:44.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0f54a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b080700b0 0x7f2b080c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:44.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0f54a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b080745c0 0x7f2b080ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0f54a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2b080c31a0 con 0x7f2b0806fca0 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0f54a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2b080c3020 con 0x7f2b080700b0 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0f54a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2b08070610 con 0x7f2b080745c0 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0d2e6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b0806fca0 0x7f2b081ef5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0d2e6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b0806fca0 0x7f2b081ef5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55922/0 (socket says 172.21.15.150:55922) 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0d2e6640 1 -- 172.21.15.150:0/142798362 learned_addr learned my addr 172.21.15.150:0/142798362 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0cae5640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b080700b0 0x7f2b080c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0dae7640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b080745c0 0x7f2b080ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0d2e6640 1 -- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b080745c0 msgr2=0x7f2b080ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0d2e6640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b080745c0 0x7f2b080ad2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0d2e6640 1 -- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b080700b0 msgr2=0x7f2b080c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0d2e6640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b080700b0 0x7f2b080c7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:44.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0d2e6640 1 -- 172.21.15.150:0/142798362 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b080c8260 con 0x7f2b0806fca0 2024-09-17T04:57:44.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0d2e6640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b0806fca0 0x7f2b081ef5b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2b0400e7a0 tx=0x7f2b0400ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:44.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2afdffb640 1 -- 172.21.15.150:0/142798362 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0400ccc0 con 0x7f2b0806fca0 2024-09-17T04:57:44.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2afdffb640 1 -- 172.21.15.150:0/142798362 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b04010430 con 0x7f2b0806fca0 2024-09-17T04:57:44.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0f54a640 1 -- 172.21.15.150:0/142798362 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b081f2db0 con 0x7f2b0806fca0 2024-09-17T04:57:44.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2afdffb640 1 -- 172.21.15.150:0/142798362 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04010860 con 0x7f2b0806fca0 2024-09-17T04:57:44.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.965+0000 7f2b0f54a640 1 -- 172.21.15.150:0/142798362 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b081f3370 con 0x7f2b0806fca0 2024-09-17T04:57:44.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.969+0000 7f2b0f54a640 1 -- 172.21.15.150:0/142798362 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b081f2ba0 con 0x7f2b0806fca0 2024-09-17T04:57:44.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.969+0000 7f2afdffb640 1 -- 172.21.15.150:0/142798362 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2b04010a00 con 0x7f2b0806fca0 2024-09-17T04:57:44.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.969+0000 7f2afdffb640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2aec073230 0x7f2aec075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:44.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.969+0000 7f2afdffb640 1 -- 172.21.15.150:0/142798362 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f2b040952e0 con 0x7f2b0806fca0 2024-09-17T04:57:44.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.969+0000 7f2b0cae5640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2aec073230 0x7f2aec075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:44.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.969+0000 7f2b0cae5640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2aec073230 0x7f2aec075740 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2b080c4360 tx=0x7f2af8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:44.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:44.973+0000 7f2afdffb640 1 -- 172.21.15.150:0/142798362 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2b0405fc40 con 0x7f2b0806fca0 2024-09-17T04:57:45.162 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:45.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.205+0000 7f2b0f54a640 1 -- 172.21.15.150:0/142798362 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b080ab140 con 0x7f2b0806fca0 2024-09-17T04:57:45.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.205+0000 7f2afdffb640 1 -- 172.21.15.150:0/142798362 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5712 (secure 0 0 0) 0x7f2b0405f540 con 0x7f2b0806fca0 2024-09-17T04:57:45.205 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:45.205 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":60,"btime":"2024-09-17T04:57:43:934233+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:57:45.206 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 60 2024-09-17T04:57:45.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.209+0000 7f2b0f54a640 1 -- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2aec073230 msgr2=0x7f2aec075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:45.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.209+0000 7f2b0f54a640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2aec073230 0x7f2aec075740 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2b080c4360 tx=0x7f2af8009040 comp rx=0 tx=0).stop 2024-09-17T04:57:45.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.209+0000 7f2b0f54a640 1 -- 172.21.15.150:0/142798362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b0806fca0 msgr2=0x7f2b081ef5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:45.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.209+0000 7f2b0f54a640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b0806fca0 0x7f2b081ef5b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2b0400e7a0 tx=0x7f2b0400ec60 comp rx=0 tx=0).stop 2024-09-17T04:57:45.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.209+0000 7f2b0f54a640 1 -- 172.21.15.150:0/142798362 shutdown_connections 2024-09-17T04:57:45.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.209+0000 7f2b0f54a640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2aec073230 0x7f2aec075740 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.209+0000 7f2b0f54a640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b080700b0 0x7f2b080c7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.209+0000 7f2b0f54a640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b080745c0 0x7f2b080ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.209+0000 7f2b0f54a640 1 --2- 172.21.15.150:0/142798362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b0806fca0 0x7f2b081ef5b0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.209+0000 7f2b0f54a640 1 -- 172.21.15.150:0/142798362 >> 172.21.15.150:0/142798362 conn(0x7f2b080c5ae0 msgr2=0x7f2b080c3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:45.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.209+0000 7f2b0f54a640 1 -- 172.21.15.150:0/142798362 shutdown_connections 2024-09-17T04:57:45.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.209+0000 7f2b0f54a640 1 -- 172.21.15.150:0/142798362 wait complete. 2024-09-17T04:57:45.233 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=5631 2024-09-17T04:57:45.234 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before reviving mds.a 2024-09-17T04:57:45.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.325+0000 7f2deb249640 1 -- 172.21.15.150:0/2725906107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de40c3900 msgr2=0x7f2de4074920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:45.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.325+0000 7f2deb249640 1 --2- 172.21.15.150:0/2725906107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de40c3900 0x7f2de4074920 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009840 tx=0x7f2dd401c7e0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.325+0000 7f2deb249640 1 -- 172.21.15.150:0/2725906107 shutdown_connections 2024-09-17T04:57:45.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.325+0000 7f2deb249640 1 --2- 172.21.15.150:0/2725906107 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de40c3900 0x7f2de4074920 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.325+0000 7f2deb249640 1 --2- 172.21.15.150:0/2725906107 >> v2:172.21.15.176:3300/0 conn(0x7f2de40c4250 0x7f2de4153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.325+0000 7f2deb249640 1 -- 172.21.15.150:0/2725906107 >> 172.21.15.150:0/2725906107 conn(0x7f2de406e360 msgr2=0x7f2de406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:45.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.325+0000 7f2deb249640 1 -- 172.21.15.150:0/2725906107 shutdown_connections 2024-09-17T04:57:45.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.325+0000 7f2deb249640 1 -- 172.21.15.150:0/2725906107 wait complete. 2024-09-17T04:57:45.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.325+0000 7f2deb249640 1 Processor -- start 2024-09-17T04:57:45.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.325+0000 7f2deb249640 1 -- start start 2024-09-17T04:57:45.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2deb249640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de4074e60 0x7f2de4156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2deb249640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2de40c3900 0x7f2de414af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2deb249640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2de40c4250 0x7f2de414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2deb249640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2de4072f30 con 0x7f2de4074e60 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2deb249640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2de4072db0 con 0x7f2de40c4250 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2deb249640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2de4160f90 con 0x7f2de40c3900 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2de8fe5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de4074e60 0x7f2de4156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2de8fe5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de4074e60 0x7f2de4156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55936/0 (socket says 172.21.15.150:55936) 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2de8fe5640 1 -- 172.21.15.150:0/4121158325 learned_addr learned my addr 172.21.15.150:0/4121158325 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2ddbfff640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2de40c3900 0x7f2de414af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2de97e6640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2de40c4250 0x7f2de414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2de8fe5640 1 -- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2de40c3900 msgr2=0x7f2de414af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2de8fe5640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2de40c3900 0x7f2de414af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2de8fe5640 1 -- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2de40c4250 msgr2=0x7f2de414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2de8fe5640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2de40c4250 0x7f2de414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2de8fe5640 1 -- 172.21.15.150:0/4121158325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de414ba00 con 0x7f2de4074e60 2024-09-17T04:57:45.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2de8fe5640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de4074e60 0x7f2de4156260 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2de000d950 tx=0x7f2de000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:45.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2dd97fa640 1 -- 172.21.15.150:0/4121158325 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de0017070 con 0x7f2de4074e60 2024-09-17T04:57:45.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2dd97fa640 1 -- 172.21.15.150:0/4121158325 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2de0009d20 con 0x7f2de4074e60 2024-09-17T04:57:45.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2dd97fa640 1 -- 172.21.15.150:0/4121158325 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de00106c0 con 0x7f2de4074e60 2024-09-17T04:57:45.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2deb249640 1 -- 172.21.15.150:0/4121158325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de40ac6e0 con 0x7f2de4074e60 2024-09-17T04:57:45.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2deb249640 1 -- 172.21.15.150:0/4121158325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de40c26d0 con 0x7f2de4074e60 2024-09-17T04:57:45.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2deb249640 1 -- 172.21.15.150:0/4121158325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de4074920 con 0x7f2de4074e60 2024-09-17T04:57:45.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.329+0000 7f2dd97fa640 1 -- 172.21.15.150:0/4121158325 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2de001a070 con 0x7f2de4074e60 2024-09-17T04:57:45.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.333+0000 7f2dd97fa640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2dd0073230 0x7f2dd0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:45.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.333+0000 7f2dd97fa640 1 -- 172.21.15.150:0/4121158325 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f2de00949d0 con 0x7f2de4074e60 2024-09-17T04:57:45.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.333+0000 7f2ddbfff640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2dd0073230 0x7f2dd0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:45.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.333+0000 7f2ddbfff640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2dd0073230 0x7f2dd0075740 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40af000 tx=0x7f2dd401b020 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:45.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.337+0000 7f2dd97fa640 1 -- 172.21.15.150:0/4121158325 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2de005f3c0 con 0x7f2de4074e60 2024-09-17T04:57:45.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.605+0000 7f2deb249640 1 -- 172.21.15.150:0/4121158325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2de4153f80 con 0x7f2de4074e60 2024-09-17T04:57:45.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.605+0000 7f2dd97fa640 1 -- 172.21.15.150:0/4121158325 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5712 (secure 0 0 0) 0x7f2de005ecc0 con 0x7f2de4074e60 2024-09-17T04:57:45.603 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:45.604 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":60,"btime":"2024-09-17T04:57:43:934233+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:57:45.604 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 60 2024-09-17T04:57:45.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.609+0000 7f2deb249640 1 -- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2dd0073230 msgr2=0x7f2dd0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:45.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.609+0000 7f2deb249640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2dd0073230 0x7f2dd0075740 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40af000 tx=0x7f2dd401b020 comp rx=0 tx=0).stop 2024-09-17T04:57:45.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.609+0000 7f2deb249640 1 -- 172.21.15.150:0/4121158325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de4074e60 msgr2=0x7f2de4156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:45.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.609+0000 7f2deb249640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de4074e60 0x7f2de4156260 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2de000d950 tx=0x7f2de000de10 comp rx=0 tx=0).stop 2024-09-17T04:57:45.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.609+0000 7f2deb249640 1 -- 172.21.15.150:0/4121158325 shutdown_connections 2024-09-17T04:57:45.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.609+0000 7f2deb249640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2dd0073230 0x7f2dd0075740 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.609+0000 7f2deb249640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2de40c4250 0x7f2de414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.609+0000 7f2deb249640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2de40c3900 0x7f2de414af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.609+0000 7f2deb249640 1 --2- 172.21.15.150:0/4121158325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2de4074e60 0x7f2de4156260 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.609+0000 7f2deb249640 1 -- 172.21.15.150:0/4121158325 >> 172.21.15.150:0/4121158325 conn(0x7f2de406e360 msgr2=0x7f2de414f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:45.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.609+0000 7f2deb249640 1 -- 172.21.15.150:0/4121158325 shutdown_connections 2024-09-17T04:57:45.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.609+0000 7f2deb249640 1 -- 172.21.15.150:0/4121158325 wait complete. 2024-09-17T04:57:45.631 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-17T04:57:45.631 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:57:45.631 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:57:45.649 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:57:45.650 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:45.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 -- 172.21.15.150:0/3431904349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778070e50 msgr2=0x7f9778071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:45.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 --2- 172.21.15.150:0/3431904349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778070e50 0x7f9778071260 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9778072370 tx=0x7f976801c800 comp rx=0 tx=0).stop 2024-09-17T04:57:45.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 -- 172.21.15.150:0/3431904349 shutdown_connections 2024-09-17T04:57:45.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 --2- 172.21.15.150:0/3431904349 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97780717a0 0x7f9778074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 --2- 172.21.15.150:0/3431904349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778070e50 0x7f9778071260 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 --2- 172.21.15.150:0/3431904349 >> v2:172.21.15.176:3300/0 conn(0x7f9778073440 0x7f97780a7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 -- 172.21.15.150:0/3431904349 >> 172.21.15.150:0/3431904349 conn(0x7f97780c4a00 msgr2=0x7f97780c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:45.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 -- 172.21.15.150:0/3431904349 shutdown_connections 2024-09-17T04:57:45.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 -- 172.21.15.150:0/3431904349 wait complete. 2024-09-17T04:57:45.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 Processor -- start 2024-09-17T04:57:45.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 -- start start 2024-09-17T04:57:45.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778070e50 0x7f977814c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:45.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97780717a0 0x7f97781f0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:45.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9778073440 0x7f97781500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:45.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f97780ace00 con 0x7f9778070e50 2024-09-17T04:57:45.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f97780adef0 con 0x7f9778073440 2024-09-17T04:57:45.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977fa07640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f97780acf80 con 0x7f97780717a0 2024-09-17T04:57:45.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977d7a3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778070e50 0x7f977814c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:45.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977d7a3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778070e50 0x7f977814c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55958/0 (socket says 172.21.15.150:55958) 2024-09-17T04:57:45.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977d7a3640 1 -- 172.21.15.150:0/3988886710 learned_addr learned my addr 172.21.15.150:0/3988886710 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:45.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977cfa2640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97780717a0 0x7f97781f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:45.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977dfa4640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9778073440 0x7f97781500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:45.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977d7a3640 1 -- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97780717a0 msgr2=0x7f97781f0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:45.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977d7a3640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97780717a0 0x7f97781f0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977d7a3640 1 -- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9778073440 msgr2=0x7f97781500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:45.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977d7a3640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9778073440 0x7f97781500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:45.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.849+0000 7f977d7a3640 1 -- 172.21.15.150:0/3988886710 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97780ac7a0 con 0x7f9778070e50 2024-09-17T04:57:45.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.853+0000 7f977d7a3640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778070e50 0x7f977814c680 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f977400c9d0 tx=0x7f977400ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.853+0000 7f976dffb640 1 -- 172.21.15.150:0/3988886710 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774004220 con 0x7f9778070e50 2024-09-17T04:57:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.853+0000 7f976dffb640 1 -- 172.21.15.150:0/3988886710 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97740044b0 con 0x7f9778070e50 2024-09-17T04:57:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.853+0000 7f977fa07640 1 -- 172.21.15.150:0/3988886710 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97780aca80 con 0x7f9778070e50 2024-09-17T04:57:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.853+0000 7f976dffb640 1 -- 172.21.15.150:0/3988886710 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774007c10 con 0x7f9778070e50 2024-09-17T04:57:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.853+0000 7f977fa07640 1 -- 172.21.15.150:0/3988886710 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f977814ccf0 con 0x7f9778070e50 2024-09-17T04:57:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.853+0000 7f977fa07640 1 -- 172.21.15.150:0/3988886710 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9778074450 con 0x7f9778070e50 2024-09-17T04:57:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.853+0000 7f976dffb640 1 -- 172.21.15.150:0/3988886710 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9774003620 con 0x7f9778070e50 2024-09-17T04:57:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.853+0000 7f976dffb640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f975c073230 0x7f975c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.853+0000 7f976dffb640 1 -- 172.21.15.150:0/3988886710 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f9774094ba0 con 0x7f9778070e50 2024-09-17T04:57:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.853+0000 7f977cfa2640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f975c073230 0x7f975c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.853+0000 7f977cfa2640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f975c073230 0x7f975c075740 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f97680af000 tx=0x7f976801b040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:45.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:45.857+0000 7f976dffb640 1 -- 172.21.15.150:0/3988886710 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9774014030 con 0x7f9778070e50 2024-09-17T04:57:46.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.065+0000 7f977fa07640 1 -- 172.21.15.150:0/3988886710 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97780c60f0 con 0x7f9778070e50 2024-09-17T04:57:46.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.065+0000 7f976dffb640 1 -- 172.21.15.150:0/3988886710 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5712 (secure 0 0 0) 0x7f977405f620 con 0x7f9778070e50 2024-09-17T04:57:46.066 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:46.066 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":60,"btime":"2024-09-17T04:57:43:934233+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:57:46.066 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 60 2024-09-17T04:57:46.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.069+0000 7f977fa07640 1 -- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f975c073230 msgr2=0x7f975c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:46.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.069+0000 7f977fa07640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f975c073230 0x7f975c075740 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f97680af000 tx=0x7f976801b040 comp rx=0 tx=0).stop 2024-09-17T04:57:46.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.069+0000 7f977fa07640 1 -- 172.21.15.150:0/3988886710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778070e50 msgr2=0x7f977814c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:46.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.069+0000 7f977fa07640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778070e50 0x7f977814c680 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f977400c9d0 tx=0x7f977400ce90 comp rx=0 tx=0).stop 2024-09-17T04:57:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.069+0000 7f977fa07640 1 -- 172.21.15.150:0/3988886710 shutdown_connections 2024-09-17T04:57:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.069+0000 7f977fa07640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f975c073230 0x7f975c075740 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.069+0000 7f977fa07640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9778073440 0x7f97781500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.069+0000 7f977fa07640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97780717a0 0x7f97781f0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.069+0000 7f977fa07640 1 --2- 172.21.15.150:0/3988886710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9778070e50 0x7f977814c680 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.069+0000 7f977fa07640 1 -- 172.21.15.150:0/3988886710 >> 172.21.15.150:0/3988886710 conn(0x7f97780c4a00 msgr2=0x7f97780c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.069+0000 7f977fa07640 1 -- 172.21.15.150:0/3988886710 shutdown_connections 2024-09-17T04:57:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:46.069+0000 7f977fa07640 1 -- 172.21.15.150:0/3988886710 wait complete. 2024-09-17T04:57:46.088 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:57:48.089 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:48.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.257+0000 7f275ead4640 1 -- 172.21.15.150:0/1034061218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27580717a0 msgr2=0x7f2758074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:48.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.257+0000 7f275ead4640 1 --2- 172.21.15.150:0/1034061218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27580717a0 0x7f2758074450 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2750009ca0 tx=0x7f275001ef20 comp rx=0 tx=0).stop 2024-09-17T04:57:48.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.257+0000 7f275ead4640 1 -- 172.21.15.150:0/1034061218 shutdown_connections 2024-09-17T04:57:48.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.257+0000 7f275ead4640 1 --2- 172.21.15.150:0/1034061218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27580717a0 0x7f2758074450 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:48.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.257+0000 7f275ead4640 1 --2- 172.21.15.150:0/1034061218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2758070e50 0x7f2758071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:48.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.257+0000 7f275ead4640 1 -- 172.21.15.150:0/1034061218 >> 172.21.15.150:0/1034061218 conn(0x7f27580c4a00 msgr2=0x7f27580c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:48.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.257+0000 7f275ead4640 1 -- 172.21.15.150:0/1034061218 shutdown_connections 2024-09-17T04:57:48.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.257+0000 7f275ead4640 1 -- 172.21.15.150:0/1034061218 wait complete. 2024-09-17T04:57:48.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275ead4640 1 Processor -- start 2024-09-17T04:57:48.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275ead4640 1 -- start start 2024-09-17T04:57:48.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275ead4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2758070e50 0x7f2758154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:48.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275ead4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27580717a0 0x7f2758146480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275ead4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2758073440 0x7f275814c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275ead4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f27580ad210 con 0x7f2758070e50 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275ead4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f27580ad090 con 0x7f27580717a0 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275ead4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f27580ad390 con 0x7f2758073440 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275c870640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2758070e50 0x7f2758154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275c870640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2758070e50 0x7f2758154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55972/0 (socket says 172.21.15.150:55972) 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275c870640 1 -- 172.21.15.150:0/2927854291 learned_addr learned my addr 172.21.15.150:0/2927854291 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275d071640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2758073440 0x7f275814c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f2757fff640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27580717a0 0x7f2758146480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275c870640 1 -- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2758073440 msgr2=0x7f275814c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275c870640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2758073440 0x7f275814c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275c870640 1 -- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27580717a0 msgr2=0x7f2758146480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:48.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275c870640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27580717a0 0x7f2758146480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:48.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275c870640 1 -- 172.21.15.150:0/2927854291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2758155230 con 0x7f2758070e50 2024-09-17T04:57:48.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275c870640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2758070e50 0x7f2758154930 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f274800c3f0 tx=0x7f274800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:48.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f27557fa640 1 -- 172.21.15.150:0/2927854291 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2748017070 con 0x7f2758070e50 2024-09-17T04:57:48.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f27557fa640 1 -- 172.21.15.150:0/2927854291 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27480053c0 con 0x7f2758070e50 2024-09-17T04:57:48.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275ead4640 1 -- 172.21.15.150:0/2927854291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275814bfc0 con 0x7f2758070e50 2024-09-17T04:57:48.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f27557fa640 1 -- 172.21.15.150:0/2927854291 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2748003860 con 0x7f2758070e50 2024-09-17T04:57:48.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.261+0000 7f275ead4640 1 -- 172.21.15.150:0/2927854291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27581554d0 con 0x7f2758070e50 2024-09-17T04:57:48.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.265+0000 7f275ead4640 1 -- 172.21.15.150:0/2927854291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27580ad860 con 0x7f2758070e50 2024-09-17T04:57:48.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.265+0000 7f27557fa640 1 -- 172.21.15.150:0/2927854291 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f274801a070 con 0x7f2758070e50 2024-09-17T04:57:48.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.265+0000 7f27557fa640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2744073200 0x7f2744075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:48.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.265+0000 7f27557fa640 1 -- 172.21.15.150:0/2927854291 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f2748093d00 con 0x7f2758070e50 2024-09-17T04:57:48.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.265+0000 7f2757fff640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2744073200 0x7f2744075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:48.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.265+0000 7f2757fff640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2744073200 0x7f2744075710 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f274c0058d0 tx=0x7f274c005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:48.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.269+0000 7f27557fa640 1 -- 172.21.15.150:0/2927854291 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2748001090 con 0x7f2758070e50 2024-09-17T04:57:48.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.477+0000 7f275ead4640 1 -- 172.21.15.150:0/2927854291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2758147970 con 0x7f2758070e50 2024-09-17T04:57:48.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.477+0000 7f27557fa640 1 -- 172.21.15.150:0/2927854291 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5712 (secure 0 0 0) 0x7f274805e5e0 con 0x7f2758070e50 2024-09-17T04:57:48.477 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:48.477 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":60,"btime":"2024-09-17T04:57:43:934233+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:57:48.477 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 60 2024-09-17T04:57:48.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.481+0000 7f275ead4640 1 -- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2744073200 msgr2=0x7f2744075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:48.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.481+0000 7f275ead4640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2744073200 0x7f2744075710 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f274c0058d0 tx=0x7f274c005860 comp rx=0 tx=0).stop 2024-09-17T04:57:48.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.481+0000 7f275ead4640 1 -- 172.21.15.150:0/2927854291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2758070e50 msgr2=0x7f2758154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:48.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.481+0000 7f275ead4640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2758070e50 0x7f2758154930 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f274800c3f0 tx=0x7f274800c8b0 comp rx=0 tx=0).stop 2024-09-17T04:57:48.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.481+0000 7f275ead4640 1 -- 172.21.15.150:0/2927854291 shutdown_connections 2024-09-17T04:57:48.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.481+0000 7f275ead4640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2744073200 0x7f2744075710 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:48.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.481+0000 7f275ead4640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27580717a0 0x7f2758146480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:48.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.481+0000 7f275ead4640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2758073440 0x7f275814c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:48.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.481+0000 7f275ead4640 1 --2- 172.21.15.150:0/2927854291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2758070e50 0x7f2758154930 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:48.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.481+0000 7f275ead4640 1 -- 172.21.15.150:0/2927854291 >> 172.21.15.150:0/2927854291 conn(0x7f27580c4a00 msgr2=0x7f27580aab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:48.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.481+0000 7f275ead4640 1 -- 172.21.15.150:0/2927854291 shutdown_connections 2024-09-17T04:57:48.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:48.481+0000 7f275ead4640 1 -- 172.21.15.150:0/2927854291 wait complete. 2024-09-17T04:57:48.498 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:57:50.235 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-17T04:57:50.235 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:57:50.235 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:57:50.238 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:57:50.239 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:50.340 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T04:57:50.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.397+0000 7f71aba0c640 1 -- 172.21.15.150:0/657443887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a40732b0 msgr2=0x7f71a40c83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:50.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.397+0000 7f71aba0c640 1 --2- 172.21.15.150:0/657443887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a40732b0 0x7f71a40c83e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f719c009cb0 tx=0x7f719c01cd10 comp rx=0 tx=0).stop 2024-09-17T04:57:50.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.397+0000 7f71aba0c640 1 -- 172.21.15.150:0/657443887 shutdown_connections 2024-09-17T04:57:50.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.397+0000 7f71aba0c640 1 --2- 172.21.15.150:0/657443887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a40732b0 0x7f71a40c83e0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.397+0000 7f71aba0c640 1 -- 172.21.15.150:0/657443887 >> 172.21.15.150:0/657443887 conn(0x7f71a40abb00 msgr2=0x7f71a40abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:50.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.397+0000 7f71aba0c640 1 -- 172.21.15.150:0/657443887 shutdown_connections 2024-09-17T04:57:50.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.397+0000 7f71aba0c640 1 -- 172.21.15.150:0/657443887 wait complete. 2024-09-17T04:57:50.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71aba0c640 1 Processor -- start 2024-09-17T04:57:50.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71aba0c640 1 -- start start 2024-09-17T04:57:50.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71aba0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a4072ea0 0x7f71a41f0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:50.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71aba0c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40732b0 0x7f71a4153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:50.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71aba0c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71a40c97a0 0x7f71a414db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:50.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71aba0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f71a40c8aa0 con 0x7f71a4072ea0 2024-09-17T04:57:50.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71aba0c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f71a40c8920 con 0x7f71a40732b0 2024-09-17T04:57:50.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71aba0c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f71a41627a0 con 0x7f71a40c97a0 2024-09-17T04:57:50.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71a97a8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a4072ea0 0x7f71a41f0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:50.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71a97a8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a4072ea0 0x7f71a41f0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37908/0 (socket says 172.21.15.150:37908) 2024-09-17T04:57:50.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71a97a8640 1 -- 172.21.15.150:0/609377276 learned_addr learned my addr 172.21.15.150:0/609377276 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71a8fa7640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40732b0 0x7f71a4153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71a9fa9640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71a40c97a0 0x7f71a414db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71a97a8640 1 -- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71a40c97a0 msgr2=0x7f71a414db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71a97a8640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71a40c97a0 0x7f71a414db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71a97a8640 1 -- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40732b0 msgr2=0x7f71a4153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71a97a8640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40732b0 0x7f71a4153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71a97a8640 1 -- 172.21.15.150:0/609377276 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a414a360 con 0x7f71a4072ea0 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71a97a8640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a4072ea0 0x7f71a41f0080 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f71a000b410 tx=0x7f71a000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f7199ffb640 1 -- 172.21.15.150:0/609377276 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a0015030 con 0x7f71a4072ea0 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f7199ffb640 1 -- 172.21.15.150:0/609377276 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71a0004020 con 0x7f71a4072ea0 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71aba0c640 1 -- 172.21.15.150:0/609377276 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a41f3fc0 con 0x7f71a4072ea0 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f7199ffb640 1 -- 172.21.15.150:0/609377276 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a0004470 con 0x7f71a4072ea0 2024-09-17T04:57:50.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71aba0c640 1 -- 172.21.15.150:0/609377276 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a41f4260 con 0x7f71a4072ea0 2024-09-17T04:57:50.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.401+0000 7f71aba0c640 1 -- 172.21.15.150:0/609377276 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71a4151c70 con 0x7f71a4072ea0 2024-09-17T04:57:50.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.405+0000 7f7199ffb640 1 -- 172.21.15.150:0/609377276 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f71a0018070 con 0x7f71a4072ea0 2024-09-17T04:57:50.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.405+0000 7f7199ffb640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7188073230 0x7f7188075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:50.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.405+0000 7f7199ffb640 1 -- 172.21.15.150:0/609377276 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f71a00943c0 con 0x7f71a4072ea0 2024-09-17T04:57:50.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.405+0000 7f71a8fa7640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7188073230 0x7f7188075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:50.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.405+0000 7f71a8fa7640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7188073230 0x7f7188075740 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f71a406e0f0 tx=0x7f7194009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:50.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.409+0000 7f7199ffb640 1 -- 172.21.15.150:0/609377276 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f71a005ee40 con 0x7f71a4072ea0 2024-09-17T04:57:50.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:50.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.617+0000 7f71aba0c640 1 -- 172.21.15.150:0/609377276 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71a40ac130 con 0x7f71a4072ea0 2024-09-17T04:57:50.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.617+0000 7f7199ffb640 1 -- 172.21.15.150:0/609377276 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5712 (secure 0 0 0) 0x7f71a005e740 con 0x7f71a4072ea0 2024-09-17T04:57:50.617 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:50.617 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":60,"btime":"2024-09-17T04:57:43:934233+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:57:50.618 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 60 2024-09-17T04:57:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.621+0000 7f71aba0c640 1 -- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7188073230 msgr2=0x7f7188075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.621+0000 7f71aba0c640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7188073230 0x7f7188075740 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f71a406e0f0 tx=0x7f7194009040 comp rx=0 tx=0).stop 2024-09-17T04:57:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.621+0000 7f71aba0c640 1 -- 172.21.15.150:0/609377276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a4072ea0 msgr2=0x7f71a41f0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.621+0000 7f71aba0c640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a4072ea0 0x7f71a41f0080 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f71a000b410 tx=0x7f71a000b8d0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.621+0000 7f71aba0c640 1 -- 172.21.15.150:0/609377276 shutdown_connections 2024-09-17T04:57:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.621+0000 7f71aba0c640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7188073230 0x7f7188075740 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.621+0000 7f71aba0c640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71a40732b0 0x7f71a4153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.621+0000 7f71aba0c640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71a40c97a0 0x7f71a414db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.621+0000 7f71aba0c640 1 --2- 172.21.15.150:0/609377276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71a4072ea0 0x7f71a41f0080 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.621+0000 7f71aba0c640 1 -- 172.21.15.150:0/609377276 >> 172.21.15.150:0/609377276 conn(0x7f71a40abb00 msgr2=0x7f71a4147900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.621+0000 7f71aba0c640 1 -- 172.21.15.150:0/609377276 shutdown_connections 2024-09-17T04:57:50.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.621+0000 7f71aba0c640 1 -- 172.21.15.150:0/609377276 wait complete. 2024-09-17T04:57:50.640 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-17T04:57:50.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.657+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3970553233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9aa40c8fc0 msgr2=0x7f9aa40c20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:50.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.657+0000 7f9aabe1e640 1 --2- 172.21.15.150:0/3970553233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9aa40c8fc0 0x7f9aa40c20a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009c80 tx=0x7f9a9c01cd70 comp rx=0 tx=0).stop 2024-09-17T04:57:50.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.657+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3970553233 shutdown_connections 2024-09-17T04:57:50.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.657+0000 7f9aabe1e640 1 --2- 172.21.15.150:0/3970553233 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9aa40c8fc0 0x7f9aa40c20a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.657+0000 7f9aabe1e640 1 --2- 172.21.15.150:0/3970553233 >> v2:172.21.15.176:3300/0 conn(0x7f9aa40c35b0 0x7f9aa40c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.657+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3970553233 >> 172.21.15.150:0/3970553233 conn(0x7f9aa4074b80 msgr2=0x7f9aa406f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:50.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.657+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3970553233 shutdown_connections 2024-09-17T04:57:50.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.657+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3970553233 wait complete. 2024-09-17T04:57:50.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.657+0000 7f9aabe1e640 1 Processor -- start 2024-09-17T04:57:50.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.657+0000 7f9aabe1e640 1 -- start start 2024-09-17T04:57:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aabe1e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9aa40c35b0 0x7f9aa4144840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aabe1e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9aa40c8fc0 0x7f9aa414b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aabe1e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa41568a0 0x7f9aa414ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aabe1e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa40c6b70 con 0x7f9aa41568a0 2024-09-17T04:57:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aabe1e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa40c69f0 con 0x7f9aa40c35b0 2024-09-17T04:57:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aabe1e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa40b0fd0 con 0x7f9aa40c8fc0 2024-09-17T04:57:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aaa3bb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa41568a0 0x7f9aa414ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aaa3bb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa41568a0 0x7f9aa414ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37912/0 (socket says 172.21.15.150:37912) 2024-09-17T04:57:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aaa3bb640 1 -- 172.21.15.150:0/3536218099 learned_addr learned my addr 172.21.15.150:0/3536218099 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aa9bba640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9aa40c35b0 0x7f9aa4144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aa93b9640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9aa40c8fc0 0x7f9aa414b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aaa3bb640 1 -- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9aa40c8fc0 msgr2=0x7f9aa414b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aaa3bb640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9aa40c8fc0 0x7f9aa414b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aaa3bb640 1 -- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9aa40c35b0 msgr2=0x7f9aa4144840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:50.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aaa3bb640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9aa40c35b0 0x7f9aa4144840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aaa3bb640 1 -- 172.21.15.150:0/3536218099 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa4144d80 con 0x7f9aa41568a0 2024-09-17T04:57:50.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aaa3bb640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa41568a0 0x7f9aa414ec50 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c098870 tx=0x7f9a9c003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:50.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9a9a7fc640 1 -- 172.21.15.150:0/3536218099 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c0b2040 con 0x7f9aa41568a0 2024-09-17T04:57:50.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9a9a7fc640 1 -- 172.21.15.150:0/3536218099 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a9c0bb070 con 0x7f9aa41568a0 2024-09-17T04:57:50.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3536218099 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa4158c00 con 0x7f9aa41568a0 2024-09-17T04:57:50.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9a9a7fc640 1 -- 172.21.15.150:0/3536218099 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c004330 con 0x7f9aa41568a0 2024-09-17T04:57:50.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3536218099 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa414e730 con 0x7f9aa41568a0 2024-09-17T04:57:50.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3536218099 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aa40b1480 con 0x7f9aa41568a0 2024-09-17T04:57:50.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9a9a7fc640 1 -- 172.21.15.150:0/3536218099 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9a9c0044d0 con 0x7f9aa41568a0 2024-09-17T04:57:50.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9a9a7fc640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9a88073200 0x7f9a88075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:50.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.661+0000 7f9a9a7fc640 1 -- 172.21.15.150:0/3536218099 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f9a9c12efd0 con 0x7f9aa41568a0 2024-09-17T04:57:50.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.665+0000 7f9aa9bba640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9a88073200 0x7f9a88075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.665+0000 7f9aa9bba640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9a88073200 0x7f9a88075710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0001780 tx=0x7f9aa0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:50.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.669+0000 7f9a9a7fc640 1 -- 172.21.15.150:0/3536218099 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9a9c016990 con 0x7f9aa41568a0 2024-09-17T04:57:50.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.881+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3536218099 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9aa4157790 con 0x7f9aa41568a0 2024-09-17T04:57:50.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.881+0000 7f9a9a7fc640 1 -- 172.21.15.150:0/3536218099 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5712 (secure 0 0 0) 0x7f9a9c0f9380 con 0x7f9aa41568a0 2024-09-17T04:57:50.880 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:50.880 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":60,"btime":"2024-09-17T04:57:43:934233+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:57:50.881 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 60 2024-09-17T04:57:50.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.885+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9a88073200 msgr2=0x7f9a88075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:50.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.885+0000 7f9aabe1e640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9a88073200 0x7f9a88075710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0001780 tx=0x7f9aa0009040 comp rx=0 tx=0).stop 2024-09-17T04:57:50.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.885+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3536218099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa41568a0 msgr2=0x7f9aa414ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:50.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.885+0000 7f9aabe1e640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa41568a0 0x7f9aa414ec50 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c098870 tx=0x7f9a9c003f40 comp rx=0 tx=0).stop 2024-09-17T04:57:50.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.885+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3536218099 shutdown_connections 2024-09-17T04:57:50.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.885+0000 7f9aabe1e640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9a88073200 0x7f9a88075710 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.885+0000 7f9aabe1e640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9aa41568a0 0x7f9aa414ec50 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.885+0000 7f9aabe1e640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9aa40c8fc0 0x7f9aa414b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.885+0000 7f9aabe1e640 1 --2- 172.21.15.150:0/3536218099 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9aa40c35b0 0x7f9aa4144840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:50.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.885+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3536218099 >> 172.21.15.150:0/3536218099 conn(0x7f9aa4074b80 msgr2=0x7f9aa40c7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:50.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.885+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3536218099 shutdown_connections 2024-09-17T04:57:50.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:50.885+0000 7f9aabe1e640 1 -- 172.21.15.150:0/3536218099 wait complete. 2024-09-17T04:57:50.902 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:57:52.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:52.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.809+0000 7fa63463e640 1 -- 172.21.15.150:0/1426509519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa62c0c41c0 msgr2=0x7fa62c06fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:52.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.809+0000 7fa63463e640 1 --2- 172.21.15.150:0/1426509519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa62c0c41c0 0x7fa62c06fab0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa62c071360 tx=0x7fa62801c8a0 comp rx=0 tx=0).stop 2024-09-17T04:57:52.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.809+0000 7fa63463e640 1 -- 172.21.15.150:0/1426509519 shutdown_connections 2024-09-17T04:57:52.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.809+0000 7fa63463e640 1 --2- 172.21.15.150:0/1426509519 >> v2:172.21.15.176:3300/0 conn(0x7fa62c06fff0 0x7fa62c070400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:52.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.809+0000 7fa63463e640 1 --2- 172.21.15.150:0/1426509519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa62c0c41c0 0x7fa62c06fab0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:52.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.809+0000 7fa63463e640 1 -- 172.21.15.150:0/1426509519 >> 172.21.15.150:0/1426509519 conn(0x7fa62c0c4a00 msgr2=0x7fa62c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:52.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.809+0000 7fa63463e640 1 -- 172.21.15.150:0/1426509519 shutdown_connections 2024-09-17T04:57:52.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.809+0000 7fa63463e640 1 -- 172.21.15.150:0/1426509519 wait complete. 2024-09-17T04:57:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.809+0000 7fa63463e640 1 Processor -- start 2024-09-17T04:57:52.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.809+0000 7fa63463e640 1 -- start start 2024-09-17T04:57:52.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa63463e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa62c06fff0 0x7fa62c1584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:52.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa63463e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa62c0c3db0 0x7fa62c1573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:52.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa63463e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa62c0c41c0 0x7fa62c0ab280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa63463e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c0adb30 con 0x7fa62c0c41c0 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa63463e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c0ad9b0 con 0x7fa62c06fff0 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa63463e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa62c0a75f0 con 0x7fa62c0c3db0 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa632bdb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa62c0c41c0 0x7fa62c0ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa632bdb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa62c0c41c0 0x7fa62c0ab280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37922/0 (socket says 172.21.15.150:37922) 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa632bdb640 1 -- 172.21.15.150:0/1685532397 learned_addr learned my addr 172.21.15.150:0/1685532397 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa631bd9640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa62c0c3db0 0x7fa62c1573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa6323da640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa62c06fff0 0x7fa62c1584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa632bdb640 1 -- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa62c0c3db0 msgr2=0x7fa62c1573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa632bdb640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa62c0c3db0 0x7fa62c1573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa632bdb640 1 -- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa62c06fff0 msgr2=0x7fa62c1584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa632bdb640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa62c06fff0 0x7fa62c1584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa632bdb640 1 -- 172.21.15.150:0/1685532397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c155c40 con 0x7fa62c0c41c0 2024-09-17T04:57:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa631bd9640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa62c0c3db0 0x7fa62c1573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:57:52.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa6323da640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa62c06fff0 0x7fa62c1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:57:52.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa632bdb640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa62c0c41c0 0x7fa62c0ab280 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa62400cae0 tx=0x7fa62400a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:52.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa622ffd640 1 -- 172.21.15.150:0/1685532397 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62400ae30 con 0x7fa62c0c41c0 2024-09-17T04:57:52.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa622ffd640 1 -- 172.21.15.150:0/1685532397 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa62400f560 con 0x7fa62c0c41c0 2024-09-17T04:57:52.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa63463e640 1 -- 172.21.15.150:0/1685532397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c149950 con 0x7fa62c0c41c0 2024-09-17T04:57:52.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa622ffd640 1 -- 172.21.15.150:0/1685532397 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62400f9b0 con 0x7fa62c0c41c0 2024-09-17T04:57:52.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa63463e640 1 -- 172.21.15.150:0/1685532397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c157920 con 0x7fa62c0c41c0 2024-09-17T04:57:52.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa63463e640 1 -- 172.21.15.150:0/1685532397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa62c157b50 con 0x7fa62c0c41c0 2024-09-17T04:57:52.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa622ffd640 1 -- 172.21.15.150:0/1685532397 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa62400fb50 con 0x7fa62c0c41c0 2024-09-17T04:57:52.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa622ffd640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa610073200 0x7fa610075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:52.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa622ffd640 1 -- 172.21.15.150:0/1685532397 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7fa6240953f0 con 0x7fa62c0c41c0 2024-09-17T04:57:52.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.813+0000 7fa6323da640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa610073200 0x7fa610075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:52.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.817+0000 7fa6323da640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa610073200 0x7fa610075710 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0c2500 tx=0x7fa61c008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:52.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:52.817+0000 7fa622ffd640 1 -- 172.21.15.150:0/1685532397 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa62c157b50 con 0x7fa62c0c41c0 2024-09-17T04:57:52.903 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:53.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.021+0000 7fa63463e640 1 -- 172.21.15.150:0/1685532397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa62c070400 con 0x7fa62c0c41c0 2024-09-17T04:57:53.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.021+0000 7fa622ffd640 1 -- 172.21.15.150:0/1685532397 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6540 (secure 0 0 0) 0x7fa624014030 con 0x7fa62c0c41c0 2024-09-17T04:57:53.019 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:53.020 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":61,"btime":"2024-09-17T04:57:50:975022+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5841,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:57:53.020 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 61 2024-09-17T04:57:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.021+0000 7fa63463e640 1 -- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa610073200 msgr2=0x7fa610075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.021+0000 7fa63463e640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa610073200 0x7fa610075710 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0c2500 tx=0x7fa61c008040 comp rx=0 tx=0).stop 2024-09-17T04:57:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.021+0000 7fa63463e640 1 -- 172.21.15.150:0/1685532397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa62c0c41c0 msgr2=0x7fa62c0ab280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.021+0000 7fa63463e640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa62c0c41c0 0x7fa62c0ab280 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa62400cae0 tx=0x7fa62400a250 comp rx=0 tx=0).stop 2024-09-17T04:57:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.025+0000 7fa63463e640 1 -- 172.21.15.150:0/1685532397 shutdown_connections 2024-09-17T04:57:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.025+0000 7fa63463e640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa610073200 0x7fa610075710 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.025+0000 7fa63463e640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa62c0c41c0 0x7fa62c0ab280 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.025+0000 7fa63463e640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa62c0c3db0 0x7fa62c1573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.025+0000 7fa63463e640 1 --2- 172.21.15.150:0/1685532397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa62c06fff0 0x7fa62c1584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.025+0000 7fa63463e640 1 -- 172.21.15.150:0/1685532397 >> 172.21.15.150:0/1685532397 conn(0x7fa62c0c4a00 msgr2=0x7fa62c0a8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:53.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.025+0000 7fa63463e640 1 -- 172.21.15.150:0/1685532397 shutdown_connections 2024-09-17T04:57:53.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.025+0000 7fa63463e640 1 -- 172.21.15.150:0/1685532397 wait complete. 2024-09-17T04:57:53.041 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-17T04:57:53.041 INFO:tasks.mds_thrash.fs.[b]:waiting for 23 secs before thrashing 2024-09-17T04:57:53.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.057+0000 7f64212d2640 1 -- 172.21.15.150:0/2224567545 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f641c0c35b0 msgr2=0x7f641c0c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:53.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.057+0000 7f64212d2640 1 --2- 172.21.15.150:0/2224567545 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f641c0c35b0 0x7f641c0c8a80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6404009900 tx=0x7f640401c8b0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.057+0000 7f64212d2640 1 -- 172.21.15.150:0/2224567545 shutdown_connections 2024-09-17T04:57:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.057+0000 7f64212d2640 1 --2- 172.21.15.150:0/2224567545 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f641c0c35b0 0x7f641c0c8a80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.057+0000 7f64212d2640 1 --2- 172.21.15.150:0/2224567545 >> v2:172.21.15.176:3300/0 conn(0x7f641c1568a0 0x7f641c0c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.057+0000 7f64212d2640 1 -- 172.21.15.150:0/2224567545 >> 172.21.15.150:0/2224567545 conn(0x7f641c074b80 msgr2=0x7f641c06f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.057+0000 7f64212d2640 1 -- 172.21.15.150:0/2224567545 shutdown_connections 2024-09-17T04:57:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.057+0000 7f64212d2640 1 -- 172.21.15.150:0/2224567545 wait complete. 2024-09-17T04:57:53.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.057+0000 7f64212d2640 1 Processor -- start 2024-09-17T04:57:53.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.057+0000 7f64212d2640 1 -- start start 2024-09-17T04:57:53.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64212d2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f641c0c35b0 0x7f641c1f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:53.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64212d2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641c0c8fc0 0x7f641c152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:53.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64212d2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f641c1568a0 0x7f641c159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:53.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64212d2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f641c0c6b70 con 0x7f641c0c8fc0 2024-09-17T04:57:53.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64212d2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f641c0c69f0 con 0x7f641c0c35b0 2024-09-17T04:57:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64212d2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f641c0c27b0 con 0x7f641c1568a0 2024-09-17T04:57:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f641a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641c0c8fc0 0x7f641c152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f641a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641c0c8fc0 0x7f641c152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37936/0 (socket says 172.21.15.150:37936) 2024-09-17T04:57:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f641a7fc640 1 -- 172.21.15.150:0/2781209990 learned_addr learned my addr 172.21.15.150:0/2781209990 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f641b7fe640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f641c1568a0 0x7f641c159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f641affd640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f641c0c35b0 0x7f641c1f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f641a7fc640 1 -- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f641c1568a0 msgr2=0x7f641c159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f641a7fc640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f641c1568a0 0x7f641c159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f641a7fc640 1 -- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f641c0c35b0 msgr2=0x7f641c1f0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f641a7fc640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f641c0c35b0 0x7f641c1f0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f641a7fc640 1 -- 172.21.15.150:0/2781209990 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641c14d400 con 0x7f641c0c8fc0 2024-09-17T04:57:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f641a7fc640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641c0c8fc0 0x7f641c152a70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6404009850 tx=0x7f6404005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64137fe640 1 -- 172.21.15.150:0/2781209990 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64040b2040 con 0x7f641c0c8fc0 2024-09-17T04:57:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64212d2640 1 -- 172.21.15.150:0/2781209990 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f641c0c9f90 con 0x7f641c0c8fc0 2024-09-17T04:57:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64137fe640 1 -- 172.21.15.150:0/2781209990 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6404005320 con 0x7f641c0c8fc0 2024-09-17T04:57:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64137fe640 1 -- 172.21.15.150:0/2781209990 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64040046d0 con 0x7f641c0c8fc0 2024-09-17T04:57:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64212d2640 1 -- 172.21.15.150:0/2781209990 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f641c1efa60 con 0x7f641c0c8fc0 2024-09-17T04:57:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64212d2640 1 -- 172.21.15.150:0/2781209990 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f641c0c8a80 con 0x7f641c0c8fc0 2024-09-17T04:57:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64137fe640 1 -- 172.21.15.150:0/2781209990 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f64040bd060 con 0x7f641c0c8fc0 2024-09-17T04:57:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64137fe640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f63fc073200 0x7f63fc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f64137fe640 1 -- 172.21.15.150:0/2781209990 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f640412e690 con 0x7f641c0c8fc0 2024-09-17T04:57:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.061+0000 7f641affd640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f63fc073200 0x7f63fc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.065+0000 7f641affd640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f63fc073200 0x7f63fc075710 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f640c009eb0 tx=0x7f640c009500 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:53.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.069+0000 7f64137fe640 1 -- 172.21.15.150:0/2781209990 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f64040164d0 con 0x7f641c0c8fc0 2024-09-17T04:57:53.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.273+0000 7f64212d2640 1 -- 172.21.15.150:0/2781209990 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f641c0691c0 con 0x7f641c0c8fc0 2024-09-17T04:57:53.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.273+0000 7f64137fe640 1 -- 172.21.15.150:0/2781209990 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6540 (secure 0 0 0) 0x7f64040f9000 con 0x7f641c0c8fc0 2024-09-17T04:57:53.272 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:53.272 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":61,"btime":"2024-09-17T04:57:50:975022+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5841,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:57:53.273 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 61 2024-09-17T04:57:53.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.277+0000 7f64212d2640 1 -- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f63fc073200 msgr2=0x7f63fc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.277+0000 7f64212d2640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f63fc073200 0x7f63fc075710 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f640c009eb0 tx=0x7f640c009500 comp rx=0 tx=0).stop 2024-09-17T04:57:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.277+0000 7f64212d2640 1 -- 172.21.15.150:0/2781209990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641c0c8fc0 msgr2=0x7f641c152a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.277+0000 7f64212d2640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641c0c8fc0 0x7f641c152a70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6404009850 tx=0x7f6404005de0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.277+0000 7f64212d2640 1 -- 172.21.15.150:0/2781209990 shutdown_connections 2024-09-17T04:57:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.277+0000 7f64212d2640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f63fc073200 0x7f63fc075710 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.277+0000 7f64212d2640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f641c1568a0 0x7f641c159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.277+0000 7f64212d2640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f641c0c8fc0 0x7f641c152a70 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.277+0000 7f64212d2640 1 --2- 172.21.15.150:0/2781209990 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f641c0c35b0 0x7f641c1f0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.277+0000 7f64212d2640 1 -- 172.21.15.150:0/2781209990 >> 172.21.15.150:0/2781209990 conn(0x7f641c074b80 msgr2=0x7f641c154aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.277+0000 7f64212d2640 1 -- 172.21.15.150:0/2781209990 shutdown_connections 2024-09-17T04:57:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:53.277+0000 7f64212d2640 1 -- 172.21.15.150:0/2781209990 wait complete. 2024-09-17T04:57:53.294 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:57:55.294 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:55.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.457+0000 7f6f598b2640 1 -- 172.21.15.150:0/370780886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f540c3db0 msgr2=0x7f6f540ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:55.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.457+0000 7f6f598b2640 1 --2- 172.21.15.150:0/370780886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f540c3db0 0x7f6f540ad2a0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6f5406f7a0 tx=0x7f6f4401c7e0 comp rx=0 tx=0).stop 2024-09-17T04:57:55.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.457+0000 7f6f598b2640 1 -- 172.21.15.150:0/370780886 shutdown_connections 2024-09-17T04:57:55.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.457+0000 7f6f598b2640 1 --2- 172.21.15.150:0/370780886 >> v2:172.21.15.176:3300/0 conn(0x7f6f540c41c0 0x7f6f540ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:55.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.457+0000 7f6f598b2640 1 --2- 172.21.15.150:0/370780886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f540c3db0 0x7f6f540ad2a0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:55.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.457+0000 7f6f598b2640 1 -- 172.21.15.150:0/370780886 >> 172.21.15.150:0/370780886 conn(0x7f6f540c4a00 msgr2=0x7f6f540c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:55.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.457+0000 7f6f598b2640 1 -- 172.21.15.150:0/370780886 shutdown_connections 2024-09-17T04:57:55.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.457+0000 7f6f598b2640 1 -- 172.21.15.150:0/370780886 wait complete. 2024-09-17T04:57:55.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.457+0000 7f6f598b2640 1 Processor -- start 2024-09-17T04:57:55.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.457+0000 7f6f598b2640 1 -- start start 2024-09-17T04:57:55.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f598b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f5406fab0 0x7f6f541f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:55.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f598b2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f540c3db0 0x7f6f5415d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:55.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f598b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f540c41c0 0x7f6f54145ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:55.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f598b2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6f54070750 con 0x7f6f5406fab0 2024-09-17T04:57:55.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f598b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6f540705d0 con 0x7f6f540c41c0 2024-09-17T04:57:55.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f598b2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6f540a75f0 con 0x7f6f540c3db0 2024-09-17T04:57:55.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f52ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f5406fab0 0x7f6f541f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:55.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f52ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f5406fab0 0x7f6f541f29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37956/0 (socket says 172.21.15.150:37956) 2024-09-17T04:57:55.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f52ffd640 1 -- 172.21.15.150:0/14862084 learned_addr learned my addr 172.21.15.150:0/14862084 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:55.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f527fc640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f540c3db0 0x7f6f5415d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f537fe640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f540c41c0 0x7f6f54145ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f52ffd640 1 -- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f540c3db0 msgr2=0x7f6f5415d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f52ffd640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f540c3db0 0x7f6f5415d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f52ffd640 1 -- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f540c41c0 msgr2=0x7f6f54145ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f52ffd640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f540c41c0 0x7f6f54145ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f52ffd640 1 -- 172.21.15.150:0/14862084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f5414a3c0 con 0x7f6f5406fab0 2024-09-17T04:57:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f52ffd640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f5406fab0 0x7f6f541f29a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6f440982d0 tx=0x7f6f440a56c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f3bfff640 1 -- 172.21.15.150:0/14862084 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f440b2040 con 0x7f6f5406fab0 2024-09-17T04:57:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f3bfff640 1 -- 172.21.15.150:0/14862084 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f440bb070 con 0x7f6f5406fab0 2024-09-17T04:57:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f598b2640 1 -- 172.21.15.150:0/14862084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f54074360 con 0x7f6f5406fab0 2024-09-17T04:57:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f3bfff640 1 -- 172.21.15.150:0/14862084 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f44003600 con 0x7f6f5406fab0 2024-09-17T04:57:55.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f598b2640 1 -- 172.21.15.150:0/14862084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f5414a660 con 0x7f6f5406fab0 2024-09-17T04:57:55.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f598b2640 1 -- 172.21.15.150:0/14862084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f5406fec0 con 0x7f6f5406fab0 2024-09-17T04:57:55.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.461+0000 7f6f3bfff640 1 -- 172.21.15.150:0/14862084 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6f440037a0 con 0x7f6f5406fab0 2024-09-17T04:57:55.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.465+0000 7f6f3bfff640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6f40073200 0x7f6f40075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:55.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.465+0000 7f6f3bfff640 1 -- 172.21.15.150:0/14862084 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f6f4412f970 con 0x7f6f5406fab0 2024-09-17T04:57:55.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.465+0000 7f6f527fc640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6f40073200 0x7f6f40075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:55.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.465+0000 7f6f527fc640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6f40073200 0x7f6f40075710 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6f48005fd0 tx=0x7f6f48005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:55.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.469+0000 7f6f3bfff640 1 -- 172.21.15.150:0/14862084 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6f440163d0 con 0x7f6f5406fab0 2024-09-17T04:57:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.677+0000 7f6f598b2640 1 -- 172.21.15.150:0/14862084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f5406db00 con 0x7f6f5406fab0 2024-09-17T04:57:55.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.677+0000 7f6f3bfff640 1 -- 172.21.15.150:0/14862084 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6540 (secure 0 0 0) 0x7f6f440fa4f0 con 0x7f6f5406fab0 2024-09-17T04:57:55.677 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:55.677 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":61,"btime":"2024-09-17T04:57:50:975022+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5841,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:57:55.677 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 61 2024-09-17T04:57:55.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.681+0000 7f6f598b2640 1 -- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6f40073200 msgr2=0x7f6f40075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.681+0000 7f6f598b2640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6f40073200 0x7f6f40075710 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6f48005fd0 tx=0x7f6f48005860 comp rx=0 tx=0).stop 2024-09-17T04:57:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.681+0000 7f6f598b2640 1 -- 172.21.15.150:0/14862084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f5406fab0 msgr2=0x7f6f541f29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.681+0000 7f6f598b2640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f5406fab0 0x7f6f541f29a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6f440982d0 tx=0x7f6f440a56c0 comp rx=0 tx=0).stop 2024-09-17T04:57:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.681+0000 7f6f598b2640 1 -- 172.21.15.150:0/14862084 shutdown_connections 2024-09-17T04:57:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.681+0000 7f6f598b2640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6f40073200 0x7f6f40075710 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.681+0000 7f6f598b2640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6f540c41c0 0x7f6f54145ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.681+0000 7f6f598b2640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6f540c3db0 0x7f6f5415d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.681+0000 7f6f598b2640 1 --2- 172.21.15.150:0/14862084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6f5406fab0 0x7f6f541f29a0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.681+0000 7f6f598b2640 1 -- 172.21.15.150:0/14862084 >> 172.21.15.150:0/14862084 conn(0x7f6f540c4a00 msgr2=0x7f6f540c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.681+0000 7f6f598b2640 1 -- 172.21.15.150:0/14862084 shutdown_connections 2024-09-17T04:57:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:55.681+0000 7f6f598b2640 1 -- 172.21.15.150:0/14862084 wait complete. 2024-09-17T04:57:55.699 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:57:57.700 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:57:57.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.869+0000 7f6018228640 1 -- 172.21.15.150:0/1408786816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60100c3db0 msgr2=0x7f60100ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:57.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.869+0000 7f6018228640 1 --2- 172.21.15.150:0/1408786816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60100c3db0 0x7f60100ad2a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f601006f7a0 tx=0x7f600c01c820 comp rx=0 tx=0).stop 2024-09-17T04:57:57.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.869+0000 7f6018228640 1 -- 172.21.15.150:0/1408786816 shutdown_connections 2024-09-17T04:57:57.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.869+0000 7f6018228640 1 --2- 172.21.15.150:0/1408786816 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60100c41c0 0x7f60100ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:57.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.869+0000 7f6018228640 1 --2- 172.21.15.150:0/1408786816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60100c3db0 0x7f60100ad2a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:57.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.869+0000 7f6018228640 1 -- 172.21.15.150:0/1408786816 >> 172.21.15.150:0/1408786816 conn(0x7f60100c4a00 msgr2=0x7f60100c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:57.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.869+0000 7f6018228640 1 -- 172.21.15.150:0/1408786816 shutdown_connections 2024-09-17T04:57:57.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.869+0000 7f6018228640 1 -- 172.21.15.150:0/1408786816 wait complete. 2024-09-17T04:57:57.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6018228640 1 Processor -- start 2024-09-17T04:57:57.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6018228640 1 -- start start 2024-09-17T04:57:57.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6018228640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601006fab0 0x7f6010158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:57.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6018228640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60100c3db0 0x7f601014def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6018228640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60100c41c0 0x7f60100c1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6018228640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f60100ade50 con 0x7f601006fab0 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6018228640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f60100707e0 con 0x7f60100c3db0 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6018228640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f60100a75f0 con 0x7f60100c41c0 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6015fc4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601006fab0 0x7f6010158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6015fc4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601006fab0 0x7f6010158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37970/0 (socket says 172.21.15.150:37970) 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6015fc4640 1 -- 172.21.15.150:0/3165741074 learned_addr learned my addr 172.21.15.150:0/3165741074 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f60157c3640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60100c3db0 0x7f601014def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f60167c5640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60100c41c0 0x7f60100c1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6015fc4640 1 -- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60100c41c0 msgr2=0x7f60100c1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6015fc4640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60100c41c0 0x7f60100c1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6015fc4640 1 -- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60100c3db0 msgr2=0x7f601014def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6015fc4640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60100c3db0 0x7f601014def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:57.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6015fc4640 1 -- 172.21.15.150:0/3165741074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601014e430 con 0x7f601006fab0 2024-09-17T04:57:57.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6015fc4640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601006fab0 0x7f6010158350 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f600c0b0040 tx=0x7f600c0a5670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:57.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f60067fc640 1 -- 172.21.15.150:0/3165741074 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600c0bc070 con 0x7f601006fab0 2024-09-17T04:57:57.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6018228640 1 -- 172.21.15.150:0/3165741074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601014f030 con 0x7f601006fab0 2024-09-17T04:57:57.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f60067fc640 1 -- 172.21.15.150:0/3165741074 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f600c0043b0 con 0x7f601006fab0 2024-09-17T04:57:57.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f60067fc640 1 -- 172.21.15.150:0/3165741074 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600c0b2040 con 0x7f601006fab0 2024-09-17T04:57:57.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6018228640 1 -- 172.21.15.150:0/3165741074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601014e930 con 0x7f601006fab0 2024-09-17T04:57:57.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.873+0000 7f6018228640 1 -- 172.21.15.150:0/3165741074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f601006fec0 con 0x7f601006fab0 2024-09-17T04:57:57.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.877+0000 7f60067fc640 1 -- 172.21.15.150:0/3165741074 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f600c003820 con 0x7f601006fab0 2024-09-17T04:57:57.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.877+0000 7f60067fc640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5ff4073200 0x7f5ff4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:57:57.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.877+0000 7f60067fc640 1 -- 172.21.15.150:0/3165741074 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7f600c12e3d0 con 0x7f601006fab0 2024-09-17T04:57:57.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.877+0000 7f60157c3640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5ff4073200 0x7f5ff4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:57:57.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.877+0000 7f60157c3640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5ff4073200 0x7f5ff4075710 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6010152280 tx=0x7f60000091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:57:57.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:57.881+0000 7f60067fc640 1 -- 172.21.15.150:0/3165741074 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f600c016410 con 0x7f601006fab0 2024-09-17T04:57:58.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.093+0000 7f6018228640 1 -- 172.21.15.150:0/3165741074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60100c5190 con 0x7f601006fab0 2024-09-17T04:57:58.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.093+0000 7f60067fc640 1 -- 172.21.15.150:0/3165741074 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6540 (secure 0 0 0) 0x7f600c0f8670 con 0x7f601006fab0 2024-09-17T04:57:58.093 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:57:58.093 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":61,"btime":"2024-09-17T04:57:50:975022+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5841,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:57:58.093 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 61 2024-09-17T04:57:58.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.097+0000 7f6018228640 1 -- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5ff4073200 msgr2=0x7f5ff4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:58.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.097+0000 7f6018228640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5ff4073200 0x7f5ff4075710 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6010152280 tx=0x7f60000091e0 comp rx=0 tx=0).stop 2024-09-17T04:57:58.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.097+0000 7f6018228640 1 -- 172.21.15.150:0/3165741074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601006fab0 msgr2=0x7f6010158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:57:58.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.097+0000 7f6018228640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601006fab0 0x7f6010158350 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f600c0b0040 tx=0x7f600c0a5670 comp rx=0 tx=0).stop 2024-09-17T04:57:58.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.097+0000 7f6018228640 1 -- 172.21.15.150:0/3165741074 shutdown_connections 2024-09-17T04:57:58.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.097+0000 7f6018228640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5ff4073200 0x7f5ff4075710 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:58.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.097+0000 7f6018228640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60100c3db0 0x7f601014def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:58.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.097+0000 7f6018228640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60100c41c0 0x7f60100c1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:58.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.097+0000 7f6018228640 1 --2- 172.21.15.150:0/3165741074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601006fab0 0x7f6010158350 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:57:58.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.097+0000 7f6018228640 1 -- 172.21.15.150:0/3165741074 >> 172.21.15.150:0/3165741074 conn(0x7f60100c4a00 msgr2=0x7f601006d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:57:58.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.097+0000 7f6018228640 1 -- 172.21.15.150:0/3165741074 shutdown_connections 2024-09-17T04:57:58.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:57:58.097+0000 7f6018228640 1 -- 172.21.15.150:0/3165741074 wait complete. 2024-09-17T04:57:58.116 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:58:00.117 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:00.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.285+0000 7fb3d1efc640 1 -- 172.21.15.150:0/26973739 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3cc0717a0 msgr2=0x7fb3cc074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:00.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.285+0000 7fb3d1efc640 1 --2- 172.21.15.150:0/26973739 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3cc0717a0 0x7fb3cc074450 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009bc0 tx=0x7fb3c401cb10 comp rx=0 tx=0).stop 2024-09-17T04:58:00.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.285+0000 7fb3d1efc640 1 -- 172.21.15.150:0/26973739 shutdown_connections 2024-09-17T04:58:00.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.285+0000 7fb3d1efc640 1 --2- 172.21.15.150:0/26973739 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3cc0717a0 0x7fb3cc074450 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:00.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.285+0000 7fb3d1efc640 1 --2- 172.21.15.150:0/26973739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3cc070e50 0x7fb3cc071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:00.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.285+0000 7fb3d1efc640 1 -- 172.21.15.150:0/26973739 >> 172.21.15.150:0/26973739 conn(0x7fb3cc0c4a00 msgr2=0x7fb3cc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:00.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.285+0000 7fb3d1efc640 1 -- 172.21.15.150:0/26973739 shutdown_connections 2024-09-17T04:58:00.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.285+0000 7fb3d1efc640 1 -- 172.21.15.150:0/26973739 wait complete. 2024-09-17T04:58:00.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.285+0000 7fb3d1efc640 1 Processor -- start 2024-09-17T04:58:00.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.285+0000 7fb3d1efc640 1 -- start start 2024-09-17T04:58:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3d1efc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3cc070e50 0x7fb3cc1467b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3d1efc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3cc0717a0 0x7fb3cc14cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3d1efc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3cc073440 0x7fb3cc142e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3d1efc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc0ad210 con 0x7fb3cc073440 2024-09-17T04:58:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3d1efc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc0ad090 con 0x7fb3cc070e50 2024-09-17T04:58:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3d1efc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb3cc0ad390 con 0x7fb3cc0717a0 2024-09-17T04:58:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3cbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3cc073440 0x7fb3cc142e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3cbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3cc073440 0x7fb3cc142e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48874/0 (socket says 172.21.15.150:48874) 2024-09-17T04:58:00.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3cbfff640 1 -- 172.21.15.150:0/1320276708 learned_addr learned my addr 172.21.15.150:0/1320276708 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3cb7fe640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3cc070e50 0x7fb3cc1467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3caffd640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3cc0717a0 0x7fb3cc14cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3cbfff640 1 -- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3cc0717a0 msgr2=0x7fb3cc14cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3cbfff640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3cc0717a0 0x7fb3cc14cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3cbfff640 1 -- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3cc070e50 msgr2=0x7fb3cc1467b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3cbfff640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3cc070e50 0x7fb3cc1467b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3cbfff640 1 -- 172.21.15.150:0/1320276708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3cc146cf0 con 0x7fb3cc073440 2024-09-17T04:58:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3cbfff640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3cc073440 0x7fb3cc142e00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009b10 tx=0x7fb3c4009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3b3fff640 1 -- 172.21.15.150:0/1320276708 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c40b2040 con 0x7fb3cc073440 2024-09-17T04:58:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3d1efc640 1 -- 172.21.15.150:0/1320276708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3cc145110 con 0x7fb3cc073440 2024-09-17T04:58:00.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3b3fff640 1 -- 172.21.15.150:0/1320276708 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3c40bc070 con 0x7fb3cc073440 2024-09-17T04:58:00.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3b3fff640 1 -- 172.21.15.150:0/1320276708 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4004000 con 0x7fb3cc073440 2024-09-17T04:58:00.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3d1efc640 1 -- 172.21.15.150:0/1320276708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3cc14d2f0 con 0x7fb3cc073440 2024-09-17T04:58:00.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3d1efc640 1 -- 172.21.15.150:0/1320276708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3cc0a9380 con 0x7fb3cc073440 2024-09-17T04:58:00.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.289+0000 7fb3b3fff640 1 -- 172.21.15.150:0/1320276708 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb3c40041a0 con 0x7fb3cc073440 2024-09-17T04:58:00.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.293+0000 7fb3b3fff640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb3b8073200 0x7fb3b8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:00.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.293+0000 7fb3b3fff640 1 -- 172.21.15.150:0/1320276708 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7440+0+0 (secure 0 0 0) 0x7fb3c4131390 con 0x7fb3cc073440 2024-09-17T04:58:00.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.293+0000 7fb3cb7fe640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb3b8073200 0x7fb3b8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:00.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.293+0000 7fb3cb7fe640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb3b8073200 0x7fb3b8075710 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc007fd0 tx=0x7fb3bc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:00.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.297+0000 7fb3b3fff640 1 -- 172.21.15.150:0/1320276708 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb3c4016790 con 0x7fb3cc073440 2024-09-17T04:58:00.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.501+0000 7fb3d1efc640 1 -- 172.21.15.150:0/1320276708 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3cc0a7a20 con 0x7fb3cc073440 2024-09-17T04:58:00.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.501+0000 7fb3b3fff640 1 -- 172.21.15.150:0/1320276708 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6540 (secure 0 0 0) 0x7fb3c40fb6b0 con 0x7fb3cc073440 2024-09-17T04:58:00.505 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:00.505 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":61,"btime":"2024-09-17T04:57:50:975022+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5841,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:55:43.893595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5254":{"gid":5254,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/3079958649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3079958649},{"type":"v1","addr":"172.21.15.150:6835","nonce":3079958649}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5254,"qdb_cluster":[5254]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:00.506 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 61 2024-09-17T04:58:00.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.505+0000 7fb3d1efc640 1 -- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb3b8073200 msgr2=0x7fb3b8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:00.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.505+0000 7fb3d1efc640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb3b8073200 0x7fb3b8075710 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc007fd0 tx=0x7fb3bc009040 comp rx=0 tx=0).stop 2024-09-17T04:58:00.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.505+0000 7fb3d1efc640 1 -- 172.21.15.150:0/1320276708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3cc073440 msgr2=0x7fb3cc142e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:00.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.505+0000 7fb3d1efc640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3cc073440 0x7fb3cc142e00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009b10 tx=0x7fb3c4009200 comp rx=0 tx=0).stop 2024-09-17T04:58:00.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.505+0000 7fb3d1efc640 1 -- 172.21.15.150:0/1320276708 shutdown_connections 2024-09-17T04:58:00.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.505+0000 7fb3d1efc640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb3b8073200 0x7fb3b8075710 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:00.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.505+0000 7fb3d1efc640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3cc073440 0x7fb3cc142e00 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:00.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.505+0000 7fb3d1efc640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3cc0717a0 0x7fb3cc14cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:00.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.505+0000 7fb3d1efc640 1 --2- 172.21.15.150:0/1320276708 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3cc070e50 0x7fb3cc1467b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:00.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.505+0000 7fb3d1efc640 1 -- 172.21.15.150:0/1320276708 >> 172.21.15.150:0/1320276708 conn(0x7fb3cc0c4a00 msgr2=0x7fb3cc15e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:00.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.505+0000 7fb3d1efc640 1 -- 172.21.15.150:0/1320276708 shutdown_connections 2024-09-17T04:58:00.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:00.505+0000 7fb3d1efc640 1 -- 172.21.15.150:0/1320276708 wait complete. 2024-09-17T04:58:00.512 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:58:00.514 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:58:00.524 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T04:58:02.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:02.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.693+0000 7fabbdb0b640 1 -- 172.21.15.150:0/333281284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faba0004810 msgr2=0x7faba0004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:02.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.693+0000 7fabbdb0b640 1 --2- 172.21.15.150:0/333281284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faba0004810 0x7faba0004c20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fabac009e30 tx=0x7fabac01d170 comp rx=0 tx=0).stop 2024-09-17T04:58:02.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.693+0000 7fabbdb0b640 1 -- 172.21.15.150:0/333281284 shutdown_connections 2024-09-17T04:58:02.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.693+0000 7fabbdb0b640 1 --2- 172.21.15.150:0/333281284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faba0004810 0x7faba0004c20 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:02.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.693+0000 7fabbdb0b640 1 -- 172.21.15.150:0/333281284 >> 172.21.15.150:0/333281284 conn(0x7fabb80c4a00 msgr2=0x7fabb80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:02.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.693+0000 7fabbdb0b640 1 -- 172.21.15.150:0/333281284 shutdown_connections 2024-09-17T04:58:02.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.693+0000 7fabbdb0b640 1 -- 172.21.15.150:0/333281284 wait complete. 2024-09-17T04:58:02.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.693+0000 7fabbdb0b640 1 Processor -- start 2024-09-17T04:58:02.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.693+0000 7fabbdb0b640 1 -- start start 2024-09-17T04:58:02.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabbdb0b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faba0004810 0x7fabb80bef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:02.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabbdb0b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabb80bd0c0 0x7fabb80c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:02.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabbdb0b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabb80bd530 0x7fabb814b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:02.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabbdb0b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fabb8074e70 con 0x7faba0004810 2024-09-17T04:58:02.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabbdb0b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fabb80adef0 con 0x7fabb80bd530 2024-09-17T04:58:02.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabbdb0b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fabb8074ff0 con 0x7fabb80bd0c0 2024-09-17T04:58:02.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabb77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faba0004810 0x7fabb80bef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:02.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabb77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faba0004810 0x7fabb80bef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48894/0 (socket says 172.21.15.150:48894) 2024-09-17T04:58:02.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabb77fe640 1 -- 172.21.15.150:0/478435855 learned_addr learned my addr 172.21.15.150:0/478435855 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:02.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabb7fff640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabb80bd530 0x7fabb814b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:02.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabb6ffd640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabb80bd0c0 0x7fabb80c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:02.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabb77fe640 1 -- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabb80bd0c0 msgr2=0x7fabb80c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:02.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabb77fe640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabb80bd0c0 0x7fabb80c18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:02.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabb77fe640 1 -- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabb80bd530 msgr2=0x7fabb814b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:02.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabb77fe640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabb80bd530 0x7fabb814b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:02.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabb77fe640 1 -- 172.21.15.150:0/478435855 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb814b8c0 con 0x7faba0004810 2024-09-17T04:58:02.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabb7fff640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabb80bd530 0x7fabb814b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:58:02.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabb77fe640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faba0004810 0x7fabb80bef20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fabb0004e30 tx=0x7fabb0001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:02.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fab9ffff640 1 -- 172.21.15.150:0/478435855 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0015020 con 0x7faba0004810 2024-09-17T04:58:02.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fab9ffff640 1 -- 172.21.15.150:0/478435855 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabb0011960 con 0x7faba0004810 2024-09-17T04:58:02.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabbdb0b640 1 -- 172.21.15.150:0/478435855 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb8157820 con 0x7faba0004810 2024-09-17T04:58:02.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fab9ffff640 1 -- 172.21.15.150:0/478435855 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0011db0 con 0x7faba0004810 2024-09-17T04:58:02.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabbdb0b640 1 -- 172.21.15.150:0/478435855 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb80bdc40 con 0x7faba0004810 2024-09-17T04:58:02.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fabbdb0b640 1 -- 172.21.15.150:0/478435855 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabb80ace00 con 0x7faba0004810 2024-09-17T04:58:02.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fab9ffff640 1 -- 172.21.15.150:0/478435855 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fabb0018020 con 0x7faba0004810 2024-09-17T04:58:02.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fab9ffff640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faba4073230 0x7faba4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:02.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.697+0000 7fab9ffff640 1 -- 172.21.15.150:0/478435855 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7526+0+0 (secure 0 0 0) 0x7fabb00950f0 con 0x7faba0004810 2024-09-17T04:58:02.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.701+0000 7fabb6ffd640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faba4073230 0x7faba4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:02.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.701+0000 7fabb6ffd640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faba4073230 0x7faba4075740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fabac0ae000 tx=0x7fabac019f10 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:02.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.705+0000 7fab9ffff640 1 -- 172.21.15.150:0/478435855 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fabb001f030 con 0x7faba0004810 2024-09-17T04:58:02.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.913+0000 7fabbdb0b640 1 -- 172.21.15.150:0/478435855 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabb80c5300 con 0x7faba0004810 2024-09-17T04:58:02.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.913+0000 7fab9ffff640 1 -- 172.21.15.150:0/478435855 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5708 (secure 0 0 0) 0x7fabb005fac0 con 0x7faba0004810 2024-09-17T04:58:02.912 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:02.913 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":63,"btime":"2024-09-17T04:58:01:929969+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:01.237286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:02.913 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 63 2024-09-17T04:58:02.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.917+0000 7fabbdb0b640 1 -- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faba4073230 msgr2=0x7faba4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:02.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.917+0000 7fabbdb0b640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faba4073230 0x7faba4075740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fabac0ae000 tx=0x7fabac019f10 comp rx=0 tx=0).stop 2024-09-17T04:58:02.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.917+0000 7fabbdb0b640 1 -- 172.21.15.150:0/478435855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faba0004810 msgr2=0x7fabb80bef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:02.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.917+0000 7fabbdb0b640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faba0004810 0x7fabb80bef20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fabb0004e30 tx=0x7fabb0001640 comp rx=0 tx=0).stop 2024-09-17T04:58:02.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.917+0000 7fabbdb0b640 1 -- 172.21.15.150:0/478435855 shutdown_connections 2024-09-17T04:58:02.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.917+0000 7fabbdb0b640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faba4073230 0x7faba4075740 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:02.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.917+0000 7fabbdb0b640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabb80bd530 0x7fabb814b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:02.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.917+0000 7fabbdb0b640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabb80bd0c0 0x7fabb80c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:02.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.917+0000 7fabbdb0b640 1 --2- 172.21.15.150:0/478435855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faba0004810 0x7fabb80bef20 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:02.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.917+0000 7fabbdb0b640 1 -- 172.21.15.150:0/478435855 >> 172.21.15.150:0/478435855 conn(0x7fabb80c4a00 msgr2=0x7fabb80c6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:02.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.917+0000 7fabbdb0b640 1 -- 172.21.15.150:0/478435855 shutdown_connections 2024-09-17T04:58:02.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:02.917+0000 7fabbdb0b640 1 -- 172.21.15.150:0/478435855 wait complete. 2024-09-17T04:58:02.937 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-17T04:58:02.937 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T04:58:02.938 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:03.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.093+0000 7fa96fac6640 1 -- 172.21.15.150:0/1570694372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96813db50 msgr2=0x7fa968071680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:03.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.093+0000 7fa96fac6640 1 --2- 172.21.15.150:0/1570694372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96813db50 0x7fa968071680 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fa964009900 tx=0x7fa96401c780 comp rx=0 tx=0).stop 2024-09-17T04:58:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.093+0000 7fa96fac6640 1 -- 172.21.15.150:0/1570694372 shutdown_connections 2024-09-17T04:58:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.093+0000 7fa96fac6640 1 --2- 172.21.15.150:0/1570694372 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa96813ce30 0x7fa9680c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.093+0000 7fa96fac6640 1 --2- 172.21.15.150:0/1570694372 >> v2:172.21.15.176:3300/0 conn(0x7fa9680c9520 0x7fa96813c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.093+0000 7fa96fac6640 1 --2- 172.21.15.150:0/1570694372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96813db50 0x7fa968071680 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.093+0000 7fa96fac6640 1 -- 172.21.15.150:0/1570694372 >> 172.21.15.150:0/1570694372 conn(0x7fa9680abaa0 msgr2=0x7fa9680abeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.093+0000 7fa96fac6640 1 -- 172.21.15.150:0/1570694372 shutdown_connections 2024-09-17T04:58:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.093+0000 7fa96fac6640 1 -- 172.21.15.150:0/1570694372 wait complete. 2024-09-17T04:58:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.093+0000 7fa96fac6640 1 Processor -- start 2024-09-17T04:58:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.093+0000 7fa96fac6640 1 -- start start 2024-09-17T04:58:03.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96fac6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9680c9520 0x7fa9681559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:03.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96fac6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96813ce30 0x7fa96815c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:03.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96fac6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa96813db50 0x7fa96815d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:03.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96fac6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa968154640 con 0x7fa96813ce30 2024-09-17T04:58:03.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96fac6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa9681547c0 con 0x7fa96813db50 2024-09-17T04:58:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96fac6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa968154940 con 0x7fa9680c9520 2024-09-17T04:58:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96d061640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96813ce30 0x7fa96815c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96d061640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96813ce30 0x7fa96815c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48910/0 (socket says 172.21.15.150:48910) 2024-09-17T04:58:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96d061640 1 -- 172.21.15.150:0/1007024135 learned_addr learned my addr 172.21.15.150:0/1007024135 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96d862640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9680c9520 0x7fa9681559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96e063640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa96813db50 0x7fa96815d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96d061640 1 -- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9680c9520 msgr2=0x7fa9681559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96d061640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9680c9520 0x7fa9681559f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96d061640 1 -- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa96813db50 msgr2=0x7fa96815d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96d061640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa96813db50 0x7fa96815d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96d061640 1 -- 172.21.15.150:0/1007024135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa968139b90 con 0x7fa96813ce30 2024-09-17T04:58:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96d862640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9680c9520 0x7fa9681559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:58:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96d061640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96813ce30 0x7fa96815c130 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00cce0 tx=0x7fa95c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa95a7fc640 1 -- 172.21.15.150:0/1007024135 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c00f450 con 0x7fa96813ce30 2024-09-17T04:58:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa95a7fc640 1 -- 172.21.15.150:0/1007024135 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa95c00f6e0 con 0x7fa96813ce30 2024-09-17T04:58:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96fac6640 1 -- 172.21.15.150:0/1007024135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa96815dd00 con 0x7fa96813ce30 2024-09-17T04:58:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa95a7fc640 1 -- 172.21.15.150:0/1007024135 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c00fb10 con 0x7fa96813ce30 2024-09-17T04:58:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96fac6640 1 -- 172.21.15.150:0/1007024135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa968156340 con 0x7fa96813ce30 2024-09-17T04:58:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96fac6640 1 -- 172.21.15.150:0/1007024135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9680c7e80 con 0x7fa96813ce30 2024-09-17T04:58:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa95a7fc640 1 -- 172.21.15.150:0/1007024135 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa95c00fcb0 con 0x7fa96813ce30 2024-09-17T04:58:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa95a7fc640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa94c073200 0x7fa94c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa95a7fc640 1 -- 172.21.15.150:0/1007024135 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7526+0+0 (secure 0 0 0) 0x7fa95c0954a0 con 0x7fa96813ce30 2024-09-17T04:58:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.097+0000 7fa96d862640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa94c073200 0x7fa94c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.101+0000 7fa96d862640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa94c073200 0x7fa94c075710 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa9640af000 tx=0x7fa96401afc0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:03.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.105+0000 7fa95a7fc640 1 -- 172.21.15.150:0/1007024135 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa95c014030 con 0x7fa96813ce30 2024-09-17T04:58:03.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.309+0000 7fa96fac6640 1 -- 172.21.15.150:0/1007024135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9680c4a80 con 0x7fa96813ce30 2024-09-17T04:58:03.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.309+0000 7fa95a7fc640 1 -- 172.21.15.150:0/1007024135 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5705 (secure 0 0 0) 0x7fa95c05fe40 con 0x7fa96813ce30 2024-09-17T04:58:03.308 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:03.308 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":64,"btime":"2024-09-17T04:58:02:932849+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:01.939931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":5,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:03.309 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 64 2024-09-17T04:58:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.313+0000 7fa96fac6640 1 -- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa94c073200 msgr2=0x7fa94c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:03.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.313+0000 7fa96fac6640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa94c073200 0x7fa94c075710 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa9640af000 tx=0x7fa96401afc0 comp rx=0 tx=0).stop 2024-09-17T04:58:03.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.313+0000 7fa96fac6640 1 -- 172.21.15.150:0/1007024135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96813ce30 msgr2=0x7fa96815c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:03.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.313+0000 7fa96fac6640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96813ce30 0x7fa96815c130 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00cce0 tx=0x7fa95c00a620 comp rx=0 tx=0).stop 2024-09-17T04:58:03.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.313+0000 7fa96fac6640 1 -- 172.21.15.150:0/1007024135 shutdown_connections 2024-09-17T04:58:03.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.313+0000 7fa96fac6640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa94c073200 0x7fa94c075710 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:03.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.313+0000 7fa96fac6640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa96813db50 0x7fa96815d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:03.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.313+0000 7fa96fac6640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa96813ce30 0x7fa96815c130 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:03.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.313+0000 7fa96fac6640 1 --2- 172.21.15.150:0/1007024135 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9680c9520 0x7fa9681559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:03.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.313+0000 7fa96fac6640 1 -- 172.21.15.150:0/1007024135 >> 172.21.15.150:0/1007024135 conn(0x7fa9680abaa0 msgr2=0x7fa9681524b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:03.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.313+0000 7fa96fac6640 1 -- 172.21.15.150:0/1007024135 shutdown_connections 2024-09-17T04:58:03.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:03.313+0000 7fa96fac6640 1 -- 172.21.15.150:0/1007024135 wait complete. 2024-09-17T04:58:03.332 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T04:58:02:932849+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 64, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 64, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5841": { "addr": "172.21.15.150:6833/1275062894", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 1275062894, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 1275062894, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5841, "incarnation": 62, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:58:01.939931+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5841 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 60, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5404": { "addr": "172.21.15.176:6837/3543629780", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 3543629780, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 3543629780, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5404, "incarnation": 57, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:57:43.934217+0000", "qdb_cluster": [ 5404 ], "qdb_leader": 5404, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5404 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:58:05.333 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.513+0000 7f70090ad640 1 -- 172.21.15.150:0/233884568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff0004d50 msgr2=0x7f6ff00051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.513+0000 7f70090ad640 1 --2- 172.21.15.150:0/233884568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff0004d50 0x7f6ff00051a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc006fd0 tx=0x7f6ffc0209b0 comp rx=0 tx=0).stop 2024-09-17T04:58:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.513+0000 7f700359e640 1 -- 172.21.15.150:0/233884568 reap_dead start 2024-09-17T04:58:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.513+0000 7f70090ad640 1 -- 172.21.15.150:0/233884568 shutdown_connections 2024-09-17T04:58:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.513+0000 7f70090ad640 1 -- 172.21.15.150:0/233884568 >> 172.21.15.150:0/233884568 conn(0x7f70040c92d0 msgr2=0x7f70040c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.513+0000 7f70090ad640 1 -- 172.21.15.150:0/233884568 shutdown_connections 2024-09-17T04:58:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.513+0000 7f70090ad640 1 -- 172.21.15.150:0/233884568 wait complete. 2024-09-17T04:58:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.517+0000 7f70090ad640 1 Processor -- start 2024-09-17T04:58:05.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.517+0000 7f70090ad640 1 -- start start 2024-09-17T04:58:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.517+0000 7f70090ad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff0004d50 0x7f700414c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.517+0000 7f70090ad640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f700414a370 0x7f700414eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.517+0000 7f70090ad640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f700414a780 0x7f700414f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.517+0000 7f70090ad640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f70040c3020 con 0x7f6ff0004d50 2024-09-17T04:58:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.517+0000 7f70090ad640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f70040c4bc0 con 0x7f700414a370 2024-09-17T04:58:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.517+0000 7f70090ad640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f70040c31a0 con 0x7f700414a780 2024-09-17T04:58:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.517+0000 7f7002d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff0004d50 0x7f700414c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.517+0000 7f7002d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff0004d50 0x7f700414c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48924/0 (socket says 172.21.15.150:48924) 2024-09-17T04:58:05.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.517+0000 7f7002d9d640 1 -- 172.21.15.150:0/1678498238 learned_addr learned my addr 172.21.15.150:0/1678498238 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:05.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.517+0000 7f700259c640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f700414a370 0x7f700414eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:05.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f700359e640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f700414a780 0x7f700414f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:05.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f7002d9d640 1 -- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f700414a780 msgr2=0x7f700414f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:05.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f7002d9d640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f700414a780 0x7f700414f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:05.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f7002d9d640 1 -- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f700414a370 msgr2=0x7f700414eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:05.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f7002d9d640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f700414a370 0x7f700414eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:05.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f7002d9d640 1 -- 172.21.15.150:0/1678498238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7004147490 con 0x7f6ff0004d50 2024-09-17T04:58:05.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f7002d9d640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff0004d50 0x7f700414c910 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400dc90 tx=0x7f6ff400b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:05.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f6ffb7fe640 1 -- 172.21.15.150:0/1678498238 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff400bcd0 con 0x7f6ff0004d50 2024-09-17T04:58:05.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f6ffb7fe640 1 -- 172.21.15.150:0/1678498238 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ff4015020 con 0x7f6ff0004d50 2024-09-17T04:58:05.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f70090ad640 1 -- 172.21.15.150:0/1678498238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7004146fd0 con 0x7f6ff0004d50 2024-09-17T04:58:05.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f6ffb7fe640 1 -- 172.21.15.150:0/1678498238 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4003bc0 con 0x7f6ff0004d50 2024-09-17T04:58:05.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f70090ad640 1 -- 172.21.15.150:0/1678498238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7004149f60 con 0x7f6ff0004d50 2024-09-17T04:58:05.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f70090ad640 1 -- 172.21.15.150:0/1678498238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7004156840 con 0x7f6ff0004d50 2024-09-17T04:58:05.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f6ffb7fe640 1 -- 172.21.15.150:0/1678498238 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6ff40060d0 con 0x7f6ff0004d50 2024-09-17T04:58:05.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f6ffb7fe640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6fe4073230 0x7f6fe4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:05.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f6ffb7fe640 1 -- 172.21.15.150:0/1678498238 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7526+0+0 (secure 0 0 0) 0x7f6ff4095330 con 0x7f6ff0004d50 2024-09-17T04:58:05.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.521+0000 7f700259c640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6fe4073230 0x7f6fe4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:05.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.525+0000 7f700259c640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6fe4073230 0x7f6fe4075740 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f700414f760 tx=0x7f6fec0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:05.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.529+0000 7f6ffb7fe640 1 -- 172.21.15.150:0/1678498238 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6ff401f020 con 0x7f6ff0004d50 2024-09-17T04:58:05.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.757+0000 7f70090ad640 1 -- 172.21.15.150:0/1678498238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7004154dc0 con 0x7f6ff0004d50 2024-09-17T04:58:05.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.757+0000 7f6ffb7fe640 1 -- 172.21.15.150:0/1678498238 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+5712 (secure 0 0 0) 0x7f6ff405fd50 con 0x7f6ff0004d50 2024-09-17T04:58:05.755 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:05.755 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":65,"btime":"2024-09-17T04:58:03:946025+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:05.756 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 65 2024-09-17T04:58:05.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.757+0000 7f70090ad640 1 -- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6fe4073230 msgr2=0x7f6fe4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:05.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.761+0000 7f70090ad640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6fe4073230 0x7f6fe4075740 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f700414f760 tx=0x7f6fec0091e0 comp rx=0 tx=0).stop 2024-09-17T04:58:05.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.761+0000 7f70090ad640 1 -- 172.21.15.150:0/1678498238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff0004d50 msgr2=0x7f700414c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:05.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.761+0000 7f70090ad640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff0004d50 0x7f700414c910 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400dc90 tx=0x7f6ff400b620 comp rx=0 tx=0).stop 2024-09-17T04:58:05.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.761+0000 7f70090ad640 1 -- 172.21.15.150:0/1678498238 shutdown_connections 2024-09-17T04:58:05.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.761+0000 7f70090ad640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6fe4073230 0x7f6fe4075740 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:05.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.761+0000 7f70090ad640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f700414a370 0x7f700414eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:05.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.761+0000 7f70090ad640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f700414a780 0x7f700414f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:05.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.761+0000 7f70090ad640 1 --2- 172.21.15.150:0/1678498238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ff0004d50 0x7f700414c910 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:05.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.761+0000 7f70090ad640 1 -- 172.21.15.150:0/1678498238 >> 172.21.15.150:0/1678498238 conn(0x7f70040c92d0 msgr2=0x7f70040698c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:05.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.761+0000 7f70090ad640 1 -- 172.21.15.150:0/1678498238 shutdown_connections 2024-09-17T04:58:05.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:05.761+0000 7f70090ad640 1 -- 172.21.15.150:0/1678498238 wait complete. 2024-09-17T04:58:05.779 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=5254 2024-09-17T04:58:05.780 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before reviving mds.c 2024-09-17T04:58:05.985 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T04:58:05.985+0000 7f8da7233640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:58:14.781 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-17T04:58:14.781 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:58:14.781 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:58:14.785 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:58:14.785 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:14.901 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T04:58:14.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.965+0000 7f7e7d51d640 1 -- 172.21.15.150:0/1664133527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e60003fe0 msgr2=0x7f7e60004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:14.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.965+0000 7f7e7d51d640 1 --2- 172.21.15.150:0/1664133527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e60003fe0 0x7f7e60004810 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c006fd0 tx=0x7f7e6c01cf60 comp rx=0 tx=0).stop 2024-09-17T04:58:14.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.965+0000 7f7e777fe640 1 -- 172.21.15.150:0/1664133527 reap_dead start 2024-09-17T04:58:14.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.965+0000 7f7e7d51d640 1 -- 172.21.15.150:0/1664133527 shutdown_connections 2024-09-17T04:58:14.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.965+0000 7f7e7d51d640 1 -- 172.21.15.150:0/1664133527 >> 172.21.15.150:0/1664133527 conn(0x7f7e780c5ae0 msgr2=0x7f7e78073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:14.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.965+0000 7f7e7d51d640 1 -- 172.21.15.150:0/1664133527 shutdown_connections 2024-09-17T04:58:14.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.965+0000 7f7e7d51d640 1 -- 172.21.15.150:0/1664133527 wait complete. 2024-09-17T04:58:14.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.965+0000 7f7e7d51d640 1 Processor -- start 2024-09-17T04:58:14.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.965+0000 7f7e7d51d640 1 -- start start 2024-09-17T04:58:14.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e7d51d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e60003fe0 0x7f7e781522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:14.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e7d51d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e78151260 0x7f7e780c1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:14.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e7d51d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e78151670 0x7f7e781f2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:14.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e7d51d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7e780c4670 con 0x7f7e78151260 2024-09-17T04:58:14.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e7d51d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7e780c44f0 con 0x7f7e78151670 2024-09-17T04:58:14.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e7d51d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7e780c47f0 con 0x7f7e60003fe0 2024-09-17T04:58:14.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e767fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e78151260 0x7f7e780c1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:14.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e767fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e78151260 0x7f7e780c1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59132/0 (socket says 172.21.15.150:59132) 2024-09-17T04:58:14.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e767fc640 1 -- 172.21.15.150:0/2766686528 learned_addr learned my addr 172.21.15.150:0/2766686528 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:14.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e76ffd640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e60003fe0 0x7f7e781522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:14.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e777fe640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e78151670 0x7f7e781f2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e767fc640 1 -- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e60003fe0 msgr2=0x7f7e781522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e767fc640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e60003fe0 0x7f7e781522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e767fc640 1 -- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e78151670 msgr2=0x7f7e781f2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e767fc640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e78151670 0x7f7e781f2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e767fc640 1 -- 172.21.15.150:0/2766686528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e78143ff0 con 0x7f7e78151260 2024-09-17T04:58:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e767fc640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e78151260 0x7f7e780c1c50 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c01ca20 tx=0x7f7e6c006fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e5f7fe640 1 -- 172.21.15.150:0/2766686528 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c006090 con 0x7f7e78151260 2024-09-17T04:58:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e5f7fe640 1 -- 172.21.15.150:0/2766686528 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e6c006230 con 0x7f7e78151260 2024-09-17T04:58:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e7d51d640 1 -- 172.21.15.150:0/2766686528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e781442b0 con 0x7f7e78151260 2024-09-17T04:58:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e5f7fe640 1 -- 172.21.15.150:0/2766686528 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c003600 con 0x7f7e78151260 2024-09-17T04:58:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e7d51d640 1 -- 172.21.15.150:0/2766686528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7815e730 con 0x7f7e78151260 2024-09-17T04:58:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e7d51d640 1 -- 172.21.15.150:0/2766686528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e780c3fb0 con 0x7f7e78151260 2024-09-17T04:58:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e5f7fe640 1 -- 172.21.15.150:0/2766686528 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7e6c0037a0 con 0x7f7e78151260 2024-09-17T04:58:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e5f7fe640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7e64073230 0x7f7e64075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e5f7fe640 1 -- 172.21.15.150:0/2766686528 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7526+0+0 (secure 0 0 0) 0x7f7e6c12e7d0 con 0x7f7e78151260 2024-09-17T04:58:14.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.969+0000 7f7e76ffd640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7e64073230 0x7f7e64075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:14.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.973+0000 7f7e76ffd640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7e64073230 0x7f7e64075740 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009a10 tx=0x7f7e680091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:14.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:14.977+0000 7f7e5f7fe640 1 -- 172.21.15.150:0/2766686528 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7e78153cd0 con 0x7f7e78151260 2024-09-17T04:58:15.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.221+0000 7f7e7d51d640 1 -- 172.21.15.150:0/2766686528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e7806f110 con 0x7f7e78151260 2024-09-17T04:58:15.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.221+0000 7f7e5f7fe640 1 -- 172.21.15.150:0/2766686528 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+5712 (secure 0 0 0) 0x7f7e6c0f91f0 con 0x7f7e78151260 2024-09-17T04:58:15.220 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:15.220 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":65,"btime":"2024-09-17T04:58:03:946025+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:15.220 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 65 2024-09-17T04:58:15.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.225+0000 7f7e7d51d640 1 -- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7e64073230 msgr2=0x7f7e64075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:15.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.225+0000 7f7e7d51d640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7e64073230 0x7f7e64075740 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009a10 tx=0x7f7e680091e0 comp rx=0 tx=0).stop 2024-09-17T04:58:15.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.225+0000 7f7e7d51d640 1 -- 172.21.15.150:0/2766686528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e78151260 msgr2=0x7f7e780c1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:15.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.225+0000 7f7e7d51d640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e78151260 0x7f7e780c1c50 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c01ca20 tx=0x7f7e6c006fb0 comp rx=0 tx=0).stop 2024-09-17T04:58:15.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.225+0000 7f7e7d51d640 1 -- 172.21.15.150:0/2766686528 shutdown_connections 2024-09-17T04:58:15.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.225+0000 7f7e7d51d640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7e64073230 0x7f7e64075740 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:15.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.225+0000 7f7e7d51d640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e78151670 0x7f7e781f2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:15.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.225+0000 7f7e7d51d640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e78151260 0x7f7e780c1c50 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:15.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.225+0000 7f7e7d51d640 1 --2- 172.21.15.150:0/2766686528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e60003fe0 0x7f7e781522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:15.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.225+0000 7f7e7d51d640 1 -- 172.21.15.150:0/2766686528 >> 172.21.15.150:0/2766686528 conn(0x7f7e780c5ae0 msgr2=0x7f7e78073bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:15.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.225+0000 7f7e7d51d640 1 -- 172.21.15.150:0/2766686528 shutdown_connections 2024-09-17T04:58:15.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:15.225+0000 7f7e7d51d640 1 -- 172.21.15.150:0/2766686528 wait complete. 2024-09-17T04:58:15.245 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-17T04:58:16.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:16.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.241+0000 7f4cb670c640 1 -- 172.21.15.150:0/2208421579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cb00c38c0 msgr2=0x7f4cb0151be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:16.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.241+0000 7f4cb670c640 1 --2- 172.21.15.150:0/2208421579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cb00c38c0 0x7f4cb0151be0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009900 tx=0x7f4ca001c820 comp rx=0 tx=0).stop 2024-09-17T04:58:16.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.241+0000 7f4cb670c640 1 -- 172.21.15.150:0/2208421579 shutdown_connections 2024-09-17T04:58:16.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.241+0000 7f4cb670c640 1 --2- 172.21.15.150:0/2208421579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4cb00c3fa0 0x7f4cb01547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:16.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.241+0000 7f4cb670c640 1 --2- 172.21.15.150:0/2208421579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cb00c38c0 0x7f4cb0151be0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:16.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.241+0000 7f4cb670c640 1 -- 172.21.15.150:0/2208421579 >> 172.21.15.150:0/2208421579 conn(0x7f4cb00c5ae0 msgr2=0x7f4cb0073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:16.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.241+0000 7f4cb670c640 1 -- 172.21.15.150:0/2208421579 shutdown_connections 2024-09-17T04:58:16.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.241+0000 7f4cb670c640 1 -- 172.21.15.150:0/2208421579 wait complete. 2024-09-17T04:58:16.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.241+0000 7f4cb670c640 1 Processor -- start 2024-09-17T04:58:16.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb670c640 1 -- start start 2024-09-17T04:58:16.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb670c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cb00c38c0 0x7f4cb01ee2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:16.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb670c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4cb00c3fa0 0x7f4cb01f1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:16.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb670c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cb00c43b0 0x7f4cb0070920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:16.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb670c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb0074c30 con 0x7f4cb00c43b0 2024-09-17T04:58:16.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb670c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb0074ab0 con 0x7f4cb00c38c0 2024-09-17T04:58:16.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb670c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4cb0154d30 con 0x7f4cb00c3fa0 2024-09-17T04:58:16.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb4ca9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cb00c43b0 0x7f4cb0070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:16.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb4ca9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cb00c43b0 0x7f4cb0070920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59156/0 (socket says 172.21.15.150:59156) 2024-09-17T04:58:16.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb4ca9640 1 -- 172.21.15.150:0/400947670 learned_addr learned my addr 172.21.15.150:0/400947670 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:16.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4caffff640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cb00c38c0 0x7f4cb01ee2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4ca77fe640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4cb00c3fa0 0x7f4cb01f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb4ca9640 1 -- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4cb00c3fa0 msgr2=0x7f4cb01f1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb4ca9640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4cb00c3fa0 0x7f4cb01f1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb4ca9640 1 -- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cb00c38c0 msgr2=0x7f4cb01ee2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb4ca9640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cb00c38c0 0x7f4cb01ee2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb4ca9640 1 -- 172.21.15.150:0/400947670 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb0143410 con 0x7f4cb00c43b0 2024-09-17T04:58:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb4ca9640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cb00c43b0 0x7f4cb0070920 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800c740 tx=0x7f4ca800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cad7fa640 1 -- 172.21.15.150:0/400947670 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8003b80 con 0x7f4cb00c43b0 2024-09-17T04:58:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cad7fa640 1 -- 172.21.15.150:0/400947670 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ca800ce50 con 0x7f4cb00c43b0 2024-09-17T04:58:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cad7fa640 1 -- 172.21.15.150:0/400947670 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca800a850 con 0x7f4cb00c43b0 2024-09-17T04:58:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb670c640 1 -- 172.21.15.150:0/400947670 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb01487e0 con 0x7f4cb00c43b0 2024-09-17T04:58:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.245+0000 7f4cb670c640 1 -- 172.21.15.150:0/400947670 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb0148da0 con 0x7f4cb00c43b0 2024-09-17T04:58:16.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.249+0000 7f4cb670c640 1 -- 172.21.15.150:0/400947670 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cb0148fa0 con 0x7f4cb00c43b0 2024-09-17T04:58:16.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.249+0000 7f4cad7fa640 1 -- 172.21.15.150:0/400947670 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4ca800a9f0 con 0x7f4cb00c43b0 2024-09-17T04:58:16.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.249+0000 7f4cad7fa640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4c90073200 0x7f4c90075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:16.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.249+0000 7f4cad7fa640 1 -- 172.21.15.150:0/400947670 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7526+0+0 (secure 0 0 0) 0x7f4ca8094600 con 0x7f4cb00c43b0 2024-09-17T04:58:16.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.249+0000 7f4caffff640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4c90073200 0x7f4c90075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:16.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.249+0000 7f4caffff640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4c90073200 0x7f4c90075710 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00af000 tx=0x7f4ca001b060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:16.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.257+0000 7f4cad7fa640 1 -- 172.21.15.150:0/400947670 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4cb0148fa0 con 0x7f4cb00c43b0 2024-09-17T04:58:16.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.521+0000 7f4cb670c640 1 -- 172.21.15.150:0/400947670 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cb01436a0 con 0x7f4cb00c43b0 2024-09-17T04:58:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.521+0000 7f4cad7fa640 1 -- 172.21.15.150:0/400947670 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6540 (secure 0 0 0) 0x7f4ca805e8b0 con 0x7f4cb00c43b0 2024-09-17T04:58:16.520 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:16.521 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":66,"btime":"2024-09-17T04:58:15:273458+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:16.521 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 66 2024-09-17T04:58:16.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.525+0000 7f4cb670c640 1 -- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4c90073200 msgr2=0x7f4c90075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:16.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.525+0000 7f4cb670c640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4c90073200 0x7f4c90075710 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00af000 tx=0x7f4ca001b060 comp rx=0 tx=0).stop 2024-09-17T04:58:16.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.525+0000 7f4cb670c640 1 -- 172.21.15.150:0/400947670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cb00c43b0 msgr2=0x7f4cb0070920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:16.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.525+0000 7f4cb670c640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cb00c43b0 0x7f4cb0070920 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800c740 tx=0x7f4ca800cc00 comp rx=0 tx=0).stop 2024-09-17T04:58:16.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.525+0000 7f4cb670c640 1 -- 172.21.15.150:0/400947670 shutdown_connections 2024-09-17T04:58:16.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.525+0000 7f4cb670c640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4c90073200 0x7f4c90075710 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:16.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.525+0000 7f4cb670c640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cb00c43b0 0x7f4cb0070920 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:16.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.525+0000 7f4cb670c640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4cb00c3fa0 0x7f4cb01f1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:16.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.525+0000 7f4cb670c640 1 --2- 172.21.15.150:0/400947670 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4cb00c38c0 0x7f4cb01ee2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:16.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.525+0000 7f4cb670c640 1 -- 172.21.15.150:0/400947670 >> 172.21.15.150:0/400947670 conn(0x7f4cb00c5ae0 msgr2=0x7f4cb0073c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:16.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.525+0000 7f4cb670c640 1 -- 172.21.15.150:0/400947670 shutdown_connections 2024-09-17T04:58:16.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.525+0000 7f4cb670c640 1 -- 172.21.15.150:0/400947670 wait complete. 2024-09-17T04:58:16.545 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-17T04:58:16.546 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:58:16.546 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:58:16.565 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:58:16.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:16.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.733+0000 7fbcb51c9640 1 -- 172.21.15.150:0/236376510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb00ab9d0 msgr2=0x7fbcb014da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:16.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.733+0000 7fbcb51c9640 1 --2- 172.21.15.150:0/236376510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb00ab9d0 0x7fbcb014da10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009840 tx=0x7fbca401c700 comp rx=0 tx=0).stop 2024-09-17T04:58:16.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.733+0000 7fbcb51c9640 1 -- 172.21.15.150:0/236376510 shutdown_connections 2024-09-17T04:58:16.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.733+0000 7fbcb51c9640 1 --2- 172.21.15.150:0/236376510 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcb006df60 0x7fbcb006e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:16.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.733+0000 7fbcb51c9640 1 --2- 172.21.15.150:0/236376510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb00ab9d0 0x7fbcb014da10 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:16.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.733+0000 7fbcb51c9640 1 -- 172.21.15.150:0/236376510 >> 172.21.15.150:0/236376510 conn(0x7fbcb0070b50 msgr2=0x7fbcb0070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:16.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.733+0000 7fbcb51c9640 1 -- 172.21.15.150:0/236376510 shutdown_connections 2024-09-17T04:58:16.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.733+0000 7fbcb51c9640 1 -- 172.21.15.150:0/236376510 wait complete. 2024-09-17T04:58:16.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.733+0000 7fbcb51c9640 1 Processor -- start 2024-09-17T04:58:16.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.733+0000 7fbcb51c9640 1 -- start start 2024-09-17T04:58:16.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcb51c9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcb006df60 0x7fbcb01f1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:16.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcb51c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb00ab9d0 0x7fbcb015d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:16.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcb51c9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcb01536e0 0x7fbcb0146f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:16.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcb51c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0154200 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcb51c9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0069e00 con 0x7fbcb01536e0 2024-09-17T04:58:16.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcb51c9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb0154380 con 0x7fbcb006df60 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcae59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb00ab9d0 0x7fbcb015d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcae59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb00ab9d0 0x7fbcb015d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59164/0 (socket says 172.21.15.150:59164) 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcae59c640 1 -- 172.21.15.150:0/1920188876 learned_addr learned my addr 172.21.15.150:0/1920188876 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcaed9d640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcb006df60 0x7fbcb01f1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcaf59e640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcb01536e0 0x7fbcb0146f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcae59c640 1 -- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcb006df60 msgr2=0x7fbcb01f1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcae59c640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcb006df60 0x7fbcb01f1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcae59c640 1 -- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcb01536e0 msgr2=0x7fbcb0146f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcae59c640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcb01536e0 0x7fbcb0146f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcae59c640 1 -- 172.21.15.150:0/1920188876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb00ade30 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcaed9d640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcb006df60 0x7fbcb01f1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcae59c640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb00ab9d0 0x7fbcb015d110 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fbca000e7a0 tx=0x7fbca000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:16.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbc977fe640 1 -- 172.21.15.150:0/1920188876 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca000ccc0 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbc977fe640 1 -- 172.21.15.150:0/1920188876 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbca0010430 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcb51c9640 1 -- 172.21.15.150:0/1920188876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcb015dd20 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbc977fe640 1 -- 172.21.15.150:0/1920188876 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca0010860 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcb51c9640 1 -- 172.21.15.150:0/1920188876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcb00bd8e0 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbcb51c9640 1 -- 172.21.15.150:0/1920188876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcb00ad3f0 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.737+0000 7fbc977fe640 1 -- 172.21.15.150:0/1920188876 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbca0010a00 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.741+0000 7fbc977fe640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbc9c073200 0x7fbc9c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:16.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.741+0000 7fbc977fe640 1 -- 172.21.15.150:0/1920188876 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7526+0+0 (secure 0 0 0) 0x7fbca0095be0 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.741+0000 7fbcaed9d640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbc9c073200 0x7fbc9c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:16.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.741+0000 7fbcaed9d640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbc9c073200 0x7fbc9c075710 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbca40af000 tx=0x7fbca40b2040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:16.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:16.745+0000 7fbc977fe640 1 -- 172.21.15.150:0/1920188876 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbca00605c0 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 -- 172.21.15.150:0/1920188876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbcb00bd6d0 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbc977fe640 1 -- 172.21.15.150:0/1920188876 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6540 (secure 0 0 0) 0x7fbca005fec0 con 0x7fbcb00ab9d0 2024-09-17T04:58:16.998 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:16.998 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":66,"btime":"2024-09-17T04:58:15:273458+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:16.999 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 66 2024-09-17T04:58:17.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 -- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbc9c073200 msgr2=0x7fbc9c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:17.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbc9c073200 0x7fbc9c075710 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbca40af000 tx=0x7fbca40b2040 comp rx=0 tx=0).stop 2024-09-17T04:58:17.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 -- 172.21.15.150:0/1920188876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb00ab9d0 msgr2=0x7fbcb015d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:17.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb00ab9d0 0x7fbcb015d110 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fbca000e7a0 tx=0x7fbca000ec60 comp rx=0 tx=0).stop 2024-09-17T04:58:17.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 -- 172.21.15.150:0/1920188876 shutdown_connections 2024-09-17T04:58:17.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbc9c073200 0x7fbc9c075710 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcb01536e0 0x7fbcb0146f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcb00ab9d0 0x7fbcb015d110 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 --2- 172.21.15.150:0/1920188876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbcb006df60 0x7fbcb01f1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 -- 172.21.15.150:0/1920188876 >> 172.21.15.150:0/1920188876 conn(0x7fbcb0070b50 msgr2=0x7fbcb0074610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:17.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 -- 172.21.15.150:0/1920188876 shutdown_connections 2024-09-17T04:58:17.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.001+0000 7fbcb51c9640 1 -- 172.21.15.150:0/1920188876 wait complete. 2024-09-17T04:58:17.019 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:58:17.246 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:17.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.425+0000 7f6809a3a640 1 -- 172.21.15.150:0/3808364516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68040c6230 msgr2=0x7f68040c5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:17.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.425+0000 7f6809a3a640 1 --2- 172.21.15.150:0/3808364516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68040c6230 0x7f68040c5330 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f67f8009900 tx=0x7f67f801c7c0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.425+0000 7f6809a3a640 1 -- 172.21.15.150:0/3808364516 shutdown_connections 2024-09-17T04:58:17.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.425+0000 7f6809a3a640 1 --2- 172.21.15.150:0/3808364516 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6804157ca0 0x7f68041580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.425+0000 7f6809a3a640 1 --2- 172.21.15.150:0/3808364516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68040c6230 0x7f68040c5330 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.425+0000 7f6809a3a640 1 -- 172.21.15.150:0/3808364516 >> 172.21.15.150:0/3808364516 conn(0x7f68040c9de0 msgr2=0x7f68040ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:17.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.425+0000 7f6809a3a640 1 -- 172.21.15.150:0/3808364516 shutdown_connections 2024-09-17T04:58:17.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.425+0000 7f6809a3a640 1 -- 172.21.15.150:0/3808364516 wait complete. 2024-09-17T04:58:17.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.425+0000 7f6809a3a640 1 Processor -- start 2024-09-17T04:58:17.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.425+0000 7f6809a3a640 1 -- start start 2024-09-17T04:58:17.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f6809a3a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68040ad7f0 0x7f68041edbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:17.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f6809a3a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68040c6230 0x7f6804155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:17.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f6809a3a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6804157ca0 0x7f6804150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f6809a3a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f68040c59f0 con 0x7f6804157ca0 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f6809a3a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f68040c5870 con 0x7f68040ad7f0 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f6809a3a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f68040c7ce0 con 0x7f68040c6230 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f68037fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6804157ca0 0x7f6804150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f68037fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6804157ca0 0x7f6804150720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59178/0 (socket says 172.21.15.150:59178) 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f68037fe640 1 -- 172.21.15.150:0/2273383363 learned_addr learned my addr 172.21.15.150:0/2273383363 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f68027fc640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68040c6230 0x7f6804155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f6802ffd640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68040ad7f0 0x7f68041edbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f68037fe640 1 -- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68040c6230 msgr2=0x7f6804155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f68037fe640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68040c6230 0x7f6804155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f68037fe640 1 -- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68040ad7f0 msgr2=0x7f68041edbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f68037fe640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68040ad7f0 0x7f68041edbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f68037fe640 1 -- 172.21.15.150:0/2273383363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f680414d240 con 0x7f6804157ca0 2024-09-17T04:58:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f68027fc640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68040c6230 0x7f6804155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:58:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f6802ffd640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68040ad7f0 0x7f68041edbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:58:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f68037fe640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6804157ca0 0x7f6804150720 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00ca30 tx=0x7f67fc00cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f67ebfff640 1 -- 172.21.15.150:0/2273383363 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67fc007d90 con 0x7f6804157ca0 2024-09-17T04:58:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f67ebfff640 1 -- 172.21.15.150:0/2273383363 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67fc004c20 con 0x7f6804157ca0 2024-09-17T04:58:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f6809a3a640 1 -- 172.21.15.150:0/2273383363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f680414d520 con 0x7f6804157ca0 2024-09-17T04:58:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f67ebfff640 1 -- 172.21.15.150:0/2273383363 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67fc005280 con 0x7f6804157ca0 2024-09-17T04:58:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f6809a3a640 1 -- 172.21.15.150:0/2273383363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68041ee130 con 0x7f6804157ca0 2024-09-17T04:58:17.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f6809a3a640 1 -- 172.21.15.150:0/2273383363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68041f7e30 con 0x7f6804157ca0 2024-09-17T04:58:17.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f67ebfff640 1 -- 172.21.15.150:0/2273383363 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f67fc005420 con 0x7f6804157ca0 2024-09-17T04:58:17.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.429+0000 7f67ebfff640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f67f0073200 0x7f67f0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:17.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.433+0000 7f67ebfff640 1 -- 172.21.15.150:0/2273383363 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7526+0+0 (secure 0 0 0) 0x7f67fc0952c0 con 0x7f6804157ca0 2024-09-17T04:58:17.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.433+0000 7f6802ffd640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f67f0073200 0x7f67f0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:17.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.433+0000 7f6802ffd640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f67f0073200 0x7f67f0075710 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f67f4007bd0 tx=0x7f67f4009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:17.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.437+0000 7f67ebfff640 1 -- 172.21.15.150:0/2273383363 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f67fc014070 con 0x7f6804157ca0 2024-09-17T04:58:17.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.669+0000 7f6809a3a640 1 -- 172.21.15.150:0/2273383363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68040c5330 con 0x7f6804157ca0 2024-09-17T04:58:17.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.669+0000 7f67ebfff640 1 -- 172.21.15.150:0/2273383363 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6540 (secure 0 0 0) 0x7f67fc05fcc0 con 0x7f6804157ca0 2024-09-17T04:58:17.669 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:17.669 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":66,"btime":"2024-09-17T04:58:15:273458+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:17.669 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 66 2024-09-17T04:58:17.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.673+0000 7f6809a3a640 1 -- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f67f0073200 msgr2=0x7f67f0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:17.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.673+0000 7f6809a3a640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f67f0073200 0x7f67f0075710 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f67f4007bd0 tx=0x7f67f4009040 comp rx=0 tx=0).stop 2024-09-17T04:58:17.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.673+0000 7f6809a3a640 1 -- 172.21.15.150:0/2273383363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6804157ca0 msgr2=0x7f6804150720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:17.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.673+0000 7f6809a3a640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6804157ca0 0x7f6804150720 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00ca30 tx=0x7f67fc00cef0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.673+0000 7f6809a3a640 1 -- 172.21.15.150:0/2273383363 shutdown_connections 2024-09-17T04:58:17.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.673+0000 7f6809a3a640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f67f0073200 0x7f67f0075710 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.673+0000 7f6809a3a640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6804157ca0 0x7f6804150720 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.673+0000 7f6809a3a640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f68040c6230 0x7f6804155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.673+0000 7f6809a3a640 1 --2- 172.21.15.150:0/2273383363 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f68040ad7f0 0x7f68041edbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:17.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.673+0000 7f6809a3a640 1 -- 172.21.15.150:0/2273383363 >> 172.21.15.150:0/2273383363 conn(0x7f68040c9de0 msgr2=0x7f68040aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:17.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.673+0000 7f6809a3a640 1 -- 172.21.15.150:0/2273383363 shutdown_connections 2024-09-17T04:58:17.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:17.673+0000 7f6809a3a640 1 -- 172.21.15.150:0/2273383363 wait complete. 2024-09-17T04:58:17.694 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-17T04:58:17.694 INFO:tasks.mds_thrash.fs.[a]:waiting for 15 secs before thrashing 2024-09-17T04:58:19.021 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:19.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.185+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3273465263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd8003a90 msgr2=0x7f3dd8003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:19.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.185+0000 7f3df5a4a640 1 --2- 172.21.15.150:0/3273465263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd8003a90 0x7f3dd8003f00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3de40016f0 tx=0x7f3de401c770 comp rx=0 tx=0).stop 2024-09-17T04:58:19.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.185+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3273465263 shutdown_connections 2024-09-17T04:58:19.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.185+0000 7f3df5a4a640 1 --2- 172.21.15.150:0/3273465263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd8003a90 0x7f3dd8003f00 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:19.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.185+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3273465263 >> 172.21.15.150:0/3273465263 conn(0x7f3df006e360 msgr2=0x7f3df006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:19.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.185+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3273465263 shutdown_connections 2024-09-17T04:58:19.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.185+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3273465263 wait complete. 2024-09-17T04:58:19.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3df5a4a640 1 Processor -- start 2024-09-17T04:58:19.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3df5a4a640 1 -- start start 2024-09-17T04:58:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3df5a4a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dd8003a90 0x7f3df014def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3df5a4a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01e3070 0x7f3df01e3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3df5a4a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3df01efbe0 0x7f3df01efff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:19.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3df5a4a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3df0071120 con 0x7f3df01e3070 2024-09-17T04:58:19.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3df5a4a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3df0070fa0 con 0x7f3df01efbe0 2024-09-17T04:58:19.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3df5a4a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3df0072db0 con 0x7f3dd8003a90 2024-09-17T04:58:19.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3dee7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01e3070 0x7f3df01e3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:19.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3dee7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01e3070 0x7f3df01e3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59204/0 (socket says 172.21.15.150:59204) 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3dee7fc640 1 -- 172.21.15.150:0/3026415098 learned_addr learned my addr 172.21.15.150:0/3026415098 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3def7fe640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3df01efbe0 0x7f3df01efff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3deeffd640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dd8003a90 0x7f3df014def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3dee7fc640 1 -- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dd8003a90 msgr2=0x7f3df014def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3dee7fc640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dd8003a90 0x7f3df014def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3dee7fc640 1 -- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3df01efbe0 msgr2=0x7f3df01efff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3dee7fc640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3df01efbe0 0x7f3df01efff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3dee7fc640 1 -- 172.21.15.150:0/3026415098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df01f3e20 con 0x7f3df01e3070 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3def7fe640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3df01efbe0 0x7f3df01efff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3dee7fc640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01e3070 0x7f3df01e3480 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3de000d780 tx=0x7f3de000dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3dd7fff640 1 -- 172.21.15.150:0/3026415098 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de0018070 con 0x7f3df01e3070 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3dd7fff640 1 -- 172.21.15.150:0/3026415098 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3de00050d0 con 0x7f3df01e3070 2024-09-17T04:58:19.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3026415098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df01f3b40 con 0x7f3df01e3070 2024-09-17T04:58:19.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3dd7fff640 1 -- 172.21.15.150:0/3026415098 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de000b6c0 con 0x7f3df01e3070 2024-09-17T04:58:19.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.189+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3026415098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df00ad3f0 con 0x7f3df01e3070 2024-09-17T04:58:19.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.193+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3026415098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3df0156be0 con 0x7f3df01e3070 2024-09-17T04:58:19.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.193+0000 7f3dd7fff640 1 -- 172.21.15.150:0/3026415098 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3de001b070 con 0x7f3df01e3070 2024-09-17T04:58:19.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.193+0000 7f3dd7fff640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3ddc073200 0x7f3ddc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:19.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.193+0000 7f3dd7fff640 1 -- 172.21.15.150:0/3026415098 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7526+0+0 (secure 0 0 0) 0x7f3de0093f40 con 0x7f3df01e3070 2024-09-17T04:58:19.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.193+0000 7f3deeffd640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3ddc073200 0x7f3ddc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:19.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.193+0000 7f3deeffd640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3ddc073200 0x7f3ddc075710 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3de40016c0 tx=0x7f3de4019560 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:19.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.197+0000 7f3dd7fff640 1 -- 172.21.15.150:0/3026415098 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3de005e310 con 0x7f3df01e3070 2024-09-17T04:58:19.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.437+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3026415098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3df006eb60 con 0x7f3df01e3070 2024-09-17T04:58:19.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3dd7fff640 1 -- 172.21.15.150:0/3026415098 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6540 (secure 0 0 0) 0x7f3de0062d40 con 0x7f3df01e3070 2024-09-17T04:58:19.438 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:19.438 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":66,"btime":"2024-09-17T04:58:15:273458+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:19.438 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 66 2024-09-17T04:58:19.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3ddc073200 msgr2=0x7f3ddc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:19.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3df5a4a640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3ddc073200 0x7f3ddc075710 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3de40016c0 tx=0x7f3de4019560 comp rx=0 tx=0).stop 2024-09-17T04:58:19.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3026415098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01e3070 msgr2=0x7f3df01e3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:19.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3df5a4a640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01e3070 0x7f3df01e3480 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3de000d780 tx=0x7f3de000dc40 comp rx=0 tx=0).stop 2024-09-17T04:58:19.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3026415098 shutdown_connections 2024-09-17T04:58:19.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3df5a4a640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3ddc073200 0x7f3ddc075710 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:19.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3df5a4a640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3df01efbe0 0x7f3df01efff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:19.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3df5a4a640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df01e3070 0x7f3df01e3480 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:19.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3df5a4a640 1 --2- 172.21.15.150:0/3026415098 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dd8003a90 0x7f3df014def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:19.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3026415098 >> 172.21.15.150:0/3026415098 conn(0x7f3df006e360 msgr2=0x7f3df0152e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:19.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3026415098 shutdown_connections 2024-09-17T04:58:19.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:19.441+0000 7f3df5a4a640 1 -- 172.21.15.150:0/3026415098 wait complete. 2024-09-17T04:58:19.466 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:58:21.467 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:21.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.665+0000 7f2f1992e640 1 -- 172.21.15.150:0/710755658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f140c8230 msgr2=0x7f2f14074870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:21.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.665+0000 7f2f1992e640 1 --2- 172.21.15.150:0/710755658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f140c8230 0x7f2f14074870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009900 tx=0x7f2f0801c8e0 comp rx=0 tx=0).stop 2024-09-17T04:58:21.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.665+0000 7f2f1992e640 1 -- 172.21.15.150:0/710755658 shutdown_connections 2024-09-17T04:58:21.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.665+0000 7f2f1992e640 1 --2- 172.21.15.150:0/710755658 >> v2:172.21.15.176:3300/0 conn(0x7f2f14074db0 0x7f2f14157460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:21.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.669+0000 7f2f1992e640 1 --2- 172.21.15.150:0/710755658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f140c8230 0x7f2f14074870 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:21.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.669+0000 7f2f1992e640 1 -- 172.21.15.150:0/710755658 >> 172.21.15.150:0/710755658 conn(0x7f2f1406e360 msgr2=0x7f2f1406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:21.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.669+0000 7f2f1992e640 1 -- 172.21.15.150:0/710755658 shutdown_connections 2024-09-17T04:58:21.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.669+0000 7f2f1992e640 1 -- 172.21.15.150:0/710755658 wait complete. 2024-09-17T04:58:21.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.669+0000 7f2f1992e640 1 Processor -- start 2024-09-17T04:58:21.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.669+0000 7f2f1992e640 1 -- start start 2024-09-17T04:58:21.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f1992e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14074db0 0x7f2f14158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:21.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f1992e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f140c8230 0x7f2f14154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:21.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f1992e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f141579a0 0x7f2f1415ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:21.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f1992e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2f140c7de0 con 0x7f2f141579a0 2024-09-17T04:58:21.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f1992e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2f140c7c60 con 0x7f2f14074db0 2024-09-17T04:58:21.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f1992e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2f141617b0 con 0x7f2f140c8230 2024-09-17T04:58:21.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f137fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f141579a0 0x7f2f1415ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:21.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f137fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f141579a0 0x7f2f1415ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59338/0 (socket says 172.21.15.150:59338) 2024-09-17T04:58:21.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f137fe640 1 -- 172.21.15.150:0/2549076488 learned_addr learned my addr 172.21.15.150:0/2549076488 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:21.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f12ffd640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14074db0 0x7f2f14158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:21.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f127fc640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f140c8230 0x7f2f14154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:21.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f137fe640 1 -- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f140c8230 msgr2=0x7f2f14154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:21.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f137fe640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f140c8230 0x7f2f14154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:21.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f137fe640 1 -- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14074db0 msgr2=0x7f2f14158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:21.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f137fe640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14074db0 0x7f2f14158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:21.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f137fe640 1 -- 172.21.15.150:0/2549076488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f14159530 con 0x7f2f141579a0 2024-09-17T04:58:21.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f127fc640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f140c8230 0x7f2f14154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:58:21.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f137fe640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f141579a0 0x7f2f1415ce80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00c760 tx=0x7f2f0c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:21.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2efbfff640 1 -- 172.21.15.150:0/2549076488 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c003b80 con 0x7f2f141579a0 2024-09-17T04:58:21.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2efbfff640 1 -- 172.21.15.150:0/2549076488 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f0c003d20 con 0x7f2f141579a0 2024-09-17T04:58:21.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f1992e640 1 -- 172.21.15.150:0/2549076488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f140a8450 con 0x7f2f141579a0 2024-09-17T04:58:21.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2efbfff640 1 -- 172.21.15.150:0/2549076488 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c00aab0 con 0x7f2f141579a0 2024-09-17T04:58:21.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f1992e640 1 -- 172.21.15.150:0/2549076488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f141ee9e0 con 0x7f2f141579a0 2024-09-17T04:58:21.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2f1992e640 1 -- 172.21.15.150:0/2549076488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f140a8690 con 0x7f2f141579a0 2024-09-17T04:58:21.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.673+0000 7f2efbfff640 1 -- 172.21.15.150:0/2549076488 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2f0c00ac50 con 0x7f2f141579a0 2024-09-17T04:58:21.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.677+0000 7f2efbfff640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2f00073200 0x7f2f00075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:21.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.677+0000 7f2efbfff640 1 -- 172.21.15.150:0/2549076488 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7526+0+0 (secure 0 0 0) 0x7f2f0c0946f0 con 0x7f2f141579a0 2024-09-17T04:58:21.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.677+0000 7f2f12ffd640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2f00073200 0x7f2f00075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:21.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.677+0000 7f2f12ffd640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2f00073200 0x7f2f00075710 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2f080af000 tx=0x7f2f0801b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:21.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.681+0000 7f2efbfff640 1 -- 172.21.15.150:0/2549076488 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2f140a8690 con 0x7f2f141579a0 2024-09-17T04:58:21.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.901+0000 7f2f1992e640 1 -- 172.21.15.150:0/2549076488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f141ee510 con 0x7f2f141579a0 2024-09-17T04:58:21.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.901+0000 7f2efbfff640 1 -- 172.21.15.150:0/2549076488 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6540 (secure 0 0 0) 0x7f2f0c05f000 con 0x7f2f141579a0 2024-09-17T04:58:21.902 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:21.902 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":66,"btime":"2024-09-17T04:58:15:273458+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:21.902 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 66 2024-09-17T04:58:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.905+0000 7f2f1992e640 1 -- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2f00073200 msgr2=0x7f2f00075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.905+0000 7f2f1992e640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2f00073200 0x7f2f00075710 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2f080af000 tx=0x7f2f0801b120 comp rx=0 tx=0).stop 2024-09-17T04:58:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.905+0000 7f2f1992e640 1 -- 172.21.15.150:0/2549076488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f141579a0 msgr2=0x7f2f1415ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.905+0000 7f2f1992e640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f141579a0 0x7f2f1415ce80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c00c760 tx=0x7f2f0c00cc20 comp rx=0 tx=0).stop 2024-09-17T04:58:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.905+0000 7f2f1992e640 1 -- 172.21.15.150:0/2549076488 shutdown_connections 2024-09-17T04:58:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.905+0000 7f2f1992e640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2f00073200 0x7f2f00075710 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.905+0000 7f2f1992e640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2f141579a0 0x7f2f1415ce80 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.905+0000 7f2f1992e640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f140c8230 0x7f2f14154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.905+0000 7f2f1992e640 1 --2- 172.21.15.150:0/2549076488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f14074db0 0x7f2f14158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.905+0000 7f2f1992e640 1 -- 172.21.15.150:0/2549076488 >> 172.21.15.150:0/2549076488 conn(0x7f2f1406e360 msgr2=0x7f2f14068fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.905+0000 7f2f1992e640 1 -- 172.21.15.150:0/2549076488 shutdown_connections 2024-09-17T04:58:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:21.905+0000 7f2f1992e640 1 -- 172.21.15.150:0/2549076488 wait complete. 2024-09-17T04:58:21.926 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:58:23.927 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:24.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.117+0000 7f9c65b5a640 1 -- 172.21.15.150:0/912539038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c48004010 msgr2=0x7f9c48004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:24.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.117+0000 7f9c65b5a640 1 --2- 172.21.15.150:0/912539038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c48004010 0x7f9c48004810 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9c540016f0 tx=0x7f9c5401c7b0 comp rx=0 tx=0).stop 2024-09-17T04:58:24.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.117+0000 7f9c5ffff640 1 -- 172.21.15.150:0/912539038 reap_dead start 2024-09-17T04:58:24.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.117+0000 7f9c65b5a640 1 -- 172.21.15.150:0/912539038 shutdown_connections 2024-09-17T04:58:24.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.117+0000 7f9c65b5a640 1 -- 172.21.15.150:0/912539038 >> 172.21.15.150:0/912539038 conn(0x7f9c600c5ae0 msgr2=0x7f9c60073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:24.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.117+0000 7f9c65b5a640 1 -- 172.21.15.150:0/912539038 shutdown_connections 2024-09-17T04:58:24.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.117+0000 7f9c65b5a640 1 -- 172.21.15.150:0/912539038 wait complete. 2024-09-17T04:58:24.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.117+0000 7f9c65b5a640 1 Processor -- start 2024-09-17T04:58:24.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.117+0000 7f9c65b5a640 1 -- start start 2024-09-17T04:58:24.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c65b5a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c48004010 0x7f9c600c1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:24.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c65b5a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c601f31f0 0x7f9c601f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:24.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c65b5a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c60149020 0x7f9c60149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:24.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c65b5a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9c600c45a0 con 0x7f9c601f31f0 2024-09-17T04:58:24.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c65b5a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9c600c4420 con 0x7f9c48004010 2024-09-17T04:58:24.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c65b5a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9c600c4720 con 0x7f9c60149020 2024-09-17T04:58:24.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c5effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c601f31f0 0x7f9c601f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:24.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c5effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c601f31f0 0x7f9c601f3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59350/0 (socket says 172.21.15.150:59350) 2024-09-17T04:58:24.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c5effd640 1 -- 172.21.15.150:0/145879323 learned_addr learned my addr 172.21.15.150:0/145879323 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:24.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c5f7fe640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c48004010 0x7f9c600c1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:24.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c5ffff640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c60149020 0x7f9c60149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:24.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c5effd640 1 -- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c60149020 msgr2=0x7f9c60149430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:24.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c5effd640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c60149020 0x7f9c60149430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:24.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c5effd640 1 -- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c48004010 msgr2=0x7f9c600c1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:24.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c5effd640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c48004010 0x7f9c600c1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:24.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c5effd640 1 -- 172.21.15.150:0/145879323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c601594b0 con 0x7f9c601f31f0 2024-09-17T04:58:24.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c5ffff640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c60149020 0x7f9c60149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:58:24.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c5effd640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c601f31f0 0x7f9c601f3600 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f9c54000c10 tx=0x7f9c54003be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:24.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c47fff640 1 -- 172.21.15.150:0/145879323 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c540ba070 con 0x7f9c601f31f0 2024-09-17T04:58:24.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c47fff640 1 -- 172.21.15.150:0/145879323 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c54006500 con 0x7f9c601f31f0 2024-09-17T04:58:24.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c47fff640 1 -- 172.21.15.150:0/145879323 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c540ad070 con 0x7f9c601f31f0 2024-09-17T04:58:24.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c65b5a640 1 -- 172.21.15.150:0/145879323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c601f00c0 con 0x7f9c601f31f0 2024-09-17T04:58:24.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c65b5a640 1 -- 172.21.15.150:0/145879323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c60158920 con 0x7f9c601f31f0 2024-09-17T04:58:24.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.121+0000 7f9c65b5a640 1 -- 172.21.15.150:0/145879323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c600aaa40 con 0x7f9c601f31f0 2024-09-17T04:58:24.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.125+0000 7f9c47fff640 1 -- 172.21.15.150:0/145879323 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9c540a7450 con 0x7f9c601f31f0 2024-09-17T04:58:24.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.125+0000 7f9c47fff640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9c4c073230 0x7f9c4c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:24.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.125+0000 7f9c47fff640 1 -- 172.21.15.150:0/145879323 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7526+0+0 (secure 0 0 0) 0x7f9c5412e2a0 con 0x7f9c601f31f0 2024-09-17T04:58:24.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.125+0000 7f9c5f7fe640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9c4c073230 0x7f9c4c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:24.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.125+0000 7f9c5f7fe640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9c4c073230 0x7f9c4c075740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9c50001ec0 tx=0x7f9c500014c0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:24.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.129+0000 7f9c47fff640 1 -- 172.21.15.150:0/145879323 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9c54016510 con 0x7f9c601f31f0 2024-09-17T04:58:24.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.393+0000 7f9c65b5a640 1 -- 172.21.15.150:0/145879323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c601f02e0 con 0x7f9c601f31f0 2024-09-17T04:58:24.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.393+0000 7f9c47fff640 1 -- 172.21.15.150:0/145879323 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6540 (secure 0 0 0) 0x7f9c540f85e0 con 0x7f9c601f31f0 2024-09-17T04:58:24.393 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:24.393 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":66,"btime":"2024-09-17T04:58:15:273458+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:24.394 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 66 2024-09-17T04:58:24.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.397+0000 7f9c65b5a640 1 -- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9c4c073230 msgr2=0x7f9c4c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:24.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.397+0000 7f9c65b5a640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9c4c073230 0x7f9c4c075740 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9c50001ec0 tx=0x7f9c500014c0 comp rx=0 tx=0).stop 2024-09-17T04:58:24.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.397+0000 7f9c65b5a640 1 -- 172.21.15.150:0/145879323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c601f31f0 msgr2=0x7f9c601f3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:24.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.397+0000 7f9c65b5a640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c601f31f0 0x7f9c601f3600 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f9c54000c10 tx=0x7f9c54003be0 comp rx=0 tx=0).stop 2024-09-17T04:58:24.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.397+0000 7f9c65b5a640 1 -- 172.21.15.150:0/145879323 shutdown_connections 2024-09-17T04:58:24.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.397+0000 7f9c65b5a640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9c4c073230 0x7f9c4c075740 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:24.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.397+0000 7f9c65b5a640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9c60149020 0x7f9c60149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:24.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.397+0000 7f9c65b5a640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c601f31f0 0x7f9c601f3600 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:24.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.397+0000 7f9c65b5a640 1 --2- 172.21.15.150:0/145879323 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c48004010 0x7f9c600c1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:24.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.397+0000 7f9c65b5a640 1 -- 172.21.15.150:0/145879323 >> 172.21.15.150:0/145879323 conn(0x7f9c600c5ae0 msgr2=0x7f9c600c72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:24.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.397+0000 7f9c65b5a640 1 -- 172.21.15.150:0/145879323 shutdown_connections 2024-09-17T04:58:24.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:24.397+0000 7f9c65b5a640 1 -- 172.21.15.150:0/145879323 wait complete. 2024-09-17T04:58:24.417 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:58:26.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:26.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.601+0000 7f2450b00640 1 -- 172.21.15.150:0/3155646115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244c157ca0 msgr2=0x7f244c1580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:26.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.601+0000 7f2450b00640 1 --2- 172.21.15.150:0/3155646115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244c157ca0 0x7f244c1580b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2444009c80 tx=0x7f244401cc80 comp rx=0 tx=0).stop 2024-09-17T04:58:26.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.601+0000 7f2450b00640 1 -- 172.21.15.150:0/3155646115 shutdown_connections 2024-09-17T04:58:26.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.601+0000 7f2450b00640 1 --2- 172.21.15.150:0/3155646115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244c157ca0 0x7f244c1580b0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:26.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.601+0000 7f2450b00640 1 --2- 172.21.15.150:0/3155646115 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f244c0ad7f0 0x7f244c0c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:26.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.601+0000 7f2450b00640 1 -- 172.21.15.150:0/3155646115 >> 172.21.15.150:0/3155646115 conn(0x7f244c0c9de0 msgr2=0x7f244c0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:26.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.601+0000 7f2450b00640 1 -- 172.21.15.150:0/3155646115 shutdown_connections 2024-09-17T04:58:26.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.601+0000 7f2450b00640 1 -- 172.21.15.150:0/3155646115 wait complete. 2024-09-17T04:58:26.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.601+0000 7f2450b00640 1 Processor -- start 2024-09-17T04:58:26.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2450b00640 1 -- start start 2024-09-17T04:58:26.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2450b00640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f244c0ad7f0 0x7f244c14d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:26.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2450b00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244c151e10 0x7f244c14bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:26.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2450b00640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f244c157ca0 0x7f244c0c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:26.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2450b00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f244c158770 con 0x7f244c151e10 2024-09-17T04:58:26.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2450b00640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1585f0 con 0x7f244c0ad7f0 2024-09-17T04:58:26.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2450b00640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f244c0c7ce0 con 0x7f244c157ca0 2024-09-17T04:58:26.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2449d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244c151e10 0x7f244c14bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2449d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244c151e10 0x7f244c14bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59374/0 (socket says 172.21.15.150:59374) 2024-09-17T04:58:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2449d9b640 1 -- 172.21.15.150:0/2461921073 learned_addr learned my addr 172.21.15.150:0/2461921073 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f244a59c640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f244c0ad7f0 0x7f244c14d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f244ad9d640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f244c157ca0 0x7f244c0c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2449d9b640 1 -- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f244c157ca0 msgr2=0x7f244c0c32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2449d9b640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f244c157ca0 0x7f244c0c32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2449d9b640 1 -- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f244c0ad7f0 msgr2=0x7f244c14d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2449d9b640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f244c0ad7f0 0x7f244c14d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2449d9b640 1 -- 172.21.15.150:0/2461921073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244c158420 con 0x7f244c151e10 2024-09-17T04:58:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2449d9b640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244c151e10 0x7f244c14bf10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f244000b410 tx=0x7f244000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f243affd640 1 -- 172.21.15.150:0/2461921073 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2440015030 con 0x7f244c151e10 2024-09-17T04:58:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f243affd640 1 -- 172.21.15.150:0/2461921073 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2440004020 con 0x7f244c151e10 2024-09-17T04:58:26.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2450b00640 1 -- 172.21.15.150:0/2461921073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244c14ba30 con 0x7f244c151e10 2024-09-17T04:58:26.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f243affd640 1 -- 172.21.15.150:0/2461921073 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2440004470 con 0x7f244c151e10 2024-09-17T04:58:26.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.609+0000 7f2450b00640 1 -- 172.21.15.150:0/2461921073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244c06f760 con 0x7f244c151e10 2024-09-17T04:58:26.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.613+0000 7f2450b00640 1 -- 172.21.15.150:0/2461921073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f244c0c6230 con 0x7f244c151e10 2024-09-17T04:58:26.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.613+0000 7f243affd640 1 -- 172.21.15.150:0/2461921073 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2440018070 con 0x7f244c151e10 2024-09-17T04:58:26.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.613+0000 7f243affd640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f242c073230 0x7f242c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:26.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.613+0000 7f243affd640 1 -- 172.21.15.150:0/2461921073 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7526+0+0 (secure 0 0 0) 0x7f2440094490 con 0x7f244c151e10 2024-09-17T04:58:26.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.613+0000 7f244a59c640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f242c073230 0x7f242c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:26.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.613+0000 7f244a59c640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f242c073230 0x7f242c075740 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f243c00bb00 tx=0x7f243c005dd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:26.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.617+0000 7f243affd640 1 -- 172.21.15.150:0/2461921073 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f244005ee30 con 0x7f244c151e10 2024-09-17T04:58:26.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.849+0000 7f2450b00640 1 -- 172.21.15.150:0/2461921073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f244c0c8e80 con 0x7f244c151e10 2024-09-17T04:58:26.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.849+0000 7f243affd640 1 -- 172.21.15.150:0/2461921073 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6540 (secure 0 0 0) 0x7f244005e730 con 0x7f244c151e10 2024-09-17T04:58:26.849 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:26.850 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":66,"btime":"2024-09-17T04:58:15:273458+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:26.850 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 66 2024-09-17T04:58:26.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.853+0000 7f2450b00640 1 -- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f242c073230 msgr2=0x7f242c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:26.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.853+0000 7f2450b00640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f242c073230 0x7f242c075740 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f243c00bb00 tx=0x7f243c005dd0 comp rx=0 tx=0).stop 2024-09-17T04:58:26.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.853+0000 7f2450b00640 1 -- 172.21.15.150:0/2461921073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244c151e10 msgr2=0x7f244c14bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:26.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.853+0000 7f2450b00640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244c151e10 0x7f244c14bf10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f244000b410 tx=0x7f244000b8d0 comp rx=0 tx=0).stop 2024-09-17T04:58:26.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.853+0000 7f2450b00640 1 -- 172.21.15.150:0/2461921073 shutdown_connections 2024-09-17T04:58:26.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.853+0000 7f2450b00640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f242c073230 0x7f242c075740 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:26.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.853+0000 7f2450b00640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f244c157ca0 0x7f244c0c32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:26.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.853+0000 7f2450b00640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f244c151e10 0x7f244c14bf10 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:26.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.853+0000 7f2450b00640 1 --2- 172.21.15.150:0/2461921073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f244c0ad7f0 0x7f244c14d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:26.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.853+0000 7f2450b00640 1 -- 172.21.15.150:0/2461921073 >> 172.21.15.150:0/2461921073 conn(0x7f244c0c9de0 msgr2=0x7f244c0aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:26.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.853+0000 7f2450b00640 1 -- 172.21.15.150:0/2461921073 shutdown_connections 2024-09-17T04:58:26.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:26.853+0000 7f2450b00640 1 -- 172.21.15.150:0/2461921073 wait complete. 2024-09-17T04:58:26.874 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:58:28.875 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:29.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.049+0000 7f60f2b4e640 1 -- 172.21.15.150:0/4287572854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec06df60 msgr2=0x7f60ec06e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:29.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.049+0000 7f60f2b4e640 1 --2- 172.21.15.150:0/4287572854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec06df60 0x7f60ec06e370 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f60d8009900 tx=0x7f60d801c850 comp rx=0 tx=0).stop 2024-09-17T04:58:29.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.049+0000 7f60f2b4e640 1 -- 172.21.15.150:0/4287572854 shutdown_connections 2024-09-17T04:58:29.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.049+0000 7f60f2b4e640 1 --2- 172.21.15.150:0/4287572854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec06df60 0x7f60ec06e370 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:29.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.049+0000 7f60f2b4e640 1 --2- 172.21.15.150:0/4287572854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60ec0ac9a0 0x7f60ec150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:29.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.049+0000 7f60f2b4e640 1 -- 172.21.15.150:0/4287572854 >> 172.21.15.150:0/4287572854 conn(0x7f60ec0aab60 msgr2=0x7f60ec0aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:29.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.049+0000 7f60f2b4e640 1 -- 172.21.15.150:0/4287572854 shutdown_connections 2024-09-17T04:58:29.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.049+0000 7f60f2b4e640 1 -- 172.21.15.150:0/4287572854 wait complete. 2024-09-17T04:58:29.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f2b4e640 1 Processor -- start 2024-09-17T04:58:29.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f2b4e640 1 -- start start 2024-09-17T04:58:29.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f2b4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec06df60 0x7f60ec14f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:29.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f2b4e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60ec0ac9a0 0x7f60ec14d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:29.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f2b4e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60ec0ad380 0x7f60ec1ef910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f2b4e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec0c8860 con 0x7f60ec06df60 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f2b4e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec157a40 con 0x7f60ec0ac9a0 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f2b4e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f60ec0c89e0 con 0x7f60ec0ad380 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f08ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec06df60 0x7f60ec14f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f08ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec06df60 0x7f60ec14f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59390/0 (socket says 172.21.15.150:59390) 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f08ea640 1 -- 172.21.15.150:0/1581633706 learned_addr learned my addr 172.21.15.150:0/1581633706 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60ebfff640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60ec0ac9a0 0x7f60ec14d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f10eb640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60ec0ad380 0x7f60ec1ef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f08ea640 1 -- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60ec0ad380 msgr2=0x7f60ec1ef910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f08ea640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60ec0ad380 0x7f60ec1ef910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f08ea640 1 -- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60ec0ac9a0 msgr2=0x7f60ec14d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f08ea640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60ec0ac9a0 0x7f60ec14d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:29.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.053+0000 7f60f08ea640 1 -- 172.21.15.150:0/1581633706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ec1f07d0 con 0x7f60ec06df60 2024-09-17T04:58:29.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.057+0000 7f60f08ea640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec06df60 0x7f60ec14f0b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f60e000d750 tx=0x7f60e000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:29.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.057+0000 7f60e97fa640 1 -- 172.21.15.150:0/1581633706 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e0017070 con 0x7f60ec06df60 2024-09-17T04:58:29.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.057+0000 7f60e97fa640 1 -- 172.21.15.150:0/1581633706 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60e0009d20 con 0x7f60ec06df60 2024-09-17T04:58:29.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.057+0000 7f60f2b4e640 1 -- 172.21.15.150:0/1581633706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ec1f32e0 con 0x7f60ec06df60 2024-09-17T04:58:29.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.057+0000 7f60e97fa640 1 -- 172.21.15.150:0/1581633706 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e0005280 con 0x7f60ec06df60 2024-09-17T04:58:29.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.057+0000 7f60f2b4e640 1 -- 172.21.15.150:0/1581633706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ec071990 con 0x7f60ec06df60 2024-09-17T04:58:29.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.057+0000 7f60f2b4e640 1 -- 172.21.15.150:0/1581633706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60ec1f09a0 con 0x7f60ec06df60 2024-09-17T04:58:29.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.057+0000 7f60e97fa640 1 -- 172.21.15.150:0/1581633706 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f60e0005420 con 0x7f60ec06df60 2024-09-17T04:58:29.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.057+0000 7f60e97fa640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60d0073230 0x7f60d0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:29.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.057+0000 7f60e97fa640 1 -- 172.21.15.150:0/1581633706 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7526+0+0 (secure 0 0 0) 0x7f60e0094a20 con 0x7f60ec06df60 2024-09-17T04:58:29.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.057+0000 7f60ebfff640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60d0073230 0x7f60d0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:29.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.061+0000 7f60ebfff640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60d0073230 0x7f60d0075740 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f60d80af000 tx=0x7f60d801b090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:29.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.065+0000 7f60e97fa640 1 -- 172.21.15.150:0/1581633706 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f60e000de60 con 0x7f60ec06df60 2024-09-17T04:58:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.301+0000 7f60f2b4e640 1 -- 172.21.15.150:0/1581633706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60ec06e370 con 0x7f60ec06df60 2024-09-17T04:58:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.301+0000 7f60e97fa640 1 -- 172.21.15.150:0/1581633706 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6540 (secure 0 0 0) 0x7f60e005f320 con 0x7f60ec06df60 2024-09-17T04:58:29.299 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:29.299 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":66,"btime":"2024-09-17T04:58:15:273458+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:57:43.934217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5404":{"gid":5404,"name":"d","rank":0,"incarnation":57,"state":"up:active","state_seq":23,"addr":"172.21.15.176:6837/3543629780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3543629780},{"type":"v1","addr":"172.21.15.176:6837","nonce":3543629780}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5404,"qdb_cluster":[5404]},"id":2}]} 2024-09-17T04:58:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 66 2024-09-17T04:58:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.305+0000 7f60f2b4e640 1 -- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60d0073230 msgr2=0x7f60d0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.305+0000 7f60f2b4e640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60d0073230 0x7f60d0075740 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f60d80af000 tx=0x7f60d801b090 comp rx=0 tx=0).stop 2024-09-17T04:58:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.305+0000 7f60f2b4e640 1 -- 172.21.15.150:0/1581633706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec06df60 msgr2=0x7f60ec14f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.305+0000 7f60f2b4e640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec06df60 0x7f60ec14f0b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f60e000d750 tx=0x7f60e000dc10 comp rx=0 tx=0).stop 2024-09-17T04:58:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.305+0000 7f60f2b4e640 1 -- 172.21.15.150:0/1581633706 shutdown_connections 2024-09-17T04:58:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.305+0000 7f60f2b4e640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60d0073230 0x7f60d0075740 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.305+0000 7f60f2b4e640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60ec0ac9a0 0x7f60ec14d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.305+0000 7f60f2b4e640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60ec0ad380 0x7f60ec1ef910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.305+0000 7f60f2b4e640 1 --2- 172.21.15.150:0/1581633706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60ec06df60 0x7f60ec14f0b0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.305+0000 7f60f2b4e640 1 -- 172.21.15.150:0/1581633706 >> 172.21.15.150:0/1581633706 conn(0x7f60ec0aab60 msgr2=0x7f60ec1502e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.305+0000 7f60f2b4e640 1 -- 172.21.15.150:0/1581633706 shutdown_connections 2024-09-17T04:58:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:29.305+0000 7f60f2b4e640 1 -- 172.21.15.150:0/1581633706 wait complete. 2024-09-17T04:58:29.322 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:58:30.566 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:58:30.568 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:58:31.323 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:31.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.477+0000 7f1c66e00640 1 -- 172.21.15.150:0/1243003975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c600c3090 msgr2=0x7f1c60152030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:31.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.477+0000 7f1c66e00640 1 --2- 172.21.15.150:0/1243003975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c600c3090 0x7f1c60152030 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1c600c61b0 tx=0x7f1c5401c780 comp rx=0 tx=0).stop 2024-09-17T04:58:31.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 -- 172.21.15.150:0/1243003975 shutdown_connections 2024-09-17T04:58:31.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 --2- 172.21.15.150:0/1243003975 >> v2:172.21.15.176:3300/0 conn(0x7f1c600c3a30 0x7f1c600743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 --2- 172.21.15.150:0/1243003975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c600c3090 0x7f1c60152030 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 -- 172.21.15.150:0/1243003975 >> 172.21.15.150:0/1243003975 conn(0x7f1c600c5ae0 msgr2=0x7f1c60073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:31.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 -- 172.21.15.150:0/1243003975 shutdown_connections 2024-09-17T04:58:31.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 -- 172.21.15.150:0/1243003975 wait complete. 2024-09-17T04:58:31.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 Processor -- start 2024-09-17T04:58:31.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 -- start start 2024-09-17T04:58:31.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c600748e0 0x7f1c601478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:31.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c600c3090 0x7f1c600c7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:31.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c600c3a30 0x7f1c601f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:31.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1c60161bc0 con 0x7f1c600c3090 2024-09-17T04:58:31.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1c601588d0 con 0x7f1c600748e0 2024-09-17T04:58:31.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c66e00640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1c60161d40 con 0x7f1c600c3a30 2024-09-17T04:58:31.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c5ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c600c3090 0x7f1c600c7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c5ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c600c3090 0x7f1c600c7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47678/0 (socket says 172.21.15.150:47678) 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c5ffff640 1 -- 172.21.15.150:0/422437616 learned_addr learned my addr 172.21.15.150:0/422437616 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c64b9c640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c600748e0 0x7f1c601478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c6539d640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c600c3a30 0x7f1c601f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c5ffff640 1 -- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c600c3a30 msgr2=0x7f1c601f0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c5ffff640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c600c3a30 0x7f1c601f0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c5ffff640 1 -- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c600748e0 msgr2=0x7f1c601478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c5ffff640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c600748e0 0x7f1c601478b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.481+0000 7f1c5ffff640 1 -- 172.21.15.150:0/422437616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c60147df0 con 0x7f1c600c3090 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.485+0000 7f1c5ffff640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c600c3090 0x7f1c600c7180 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00c970 tx=0x7f1c4c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.485+0000 7f1c5d7fa640 1 -- 172.21.15.150:0/422437616 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c004220 con 0x7f1c600c3090 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.485+0000 7f1c5d7fa640 1 -- 172.21.15.150:0/422437616 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1c4c0079b0 con 0x7f1c600c3090 2024-09-17T04:58:31.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.485+0000 7f1c5d7fa640 1 -- 172.21.15.150:0/422437616 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c0044b0 con 0x7f1c600c3090 2024-09-17T04:58:31.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.485+0000 7f1c66e00640 1 -- 172.21.15.150:0/422437616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c600c7ae0 con 0x7f1c600c3090 2024-09-17T04:58:31.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.485+0000 7f1c66e00640 1 -- 172.21.15.150:0/422437616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c601f1900 con 0x7f1c600c3090 2024-09-17T04:58:31.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.485+0000 7f1c66e00640 1 -- 172.21.15.150:0/422437616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c60152030 con 0x7f1c600c3090 2024-09-17T04:58:31.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.485+0000 7f1c5d7fa640 1 -- 172.21.15.150:0/422437616 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1c4c003600 con 0x7f1c600c3090 2024-09-17T04:58:31.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.485+0000 7f1c5d7fa640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1c44073200 0x7f1c44075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:31.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.485+0000 7f1c5d7fa640 1 -- 172.21.15.150:0/422437616 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f1c4c094a40 con 0x7f1c600c3090 2024-09-17T04:58:31.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.485+0000 7f1c64b9c640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1c44073200 0x7f1c44075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:31.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.489+0000 7f1c64b9c640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1c44073200 0x7f1c44075710 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1c540ae000 tx=0x7f1c5401afc0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:31.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.493+0000 7f1c5d7fa640 1 -- 172.21.15.150:0/422437616 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1c60147df0 con 0x7f1c600c3090 2024-09-17T04:58:31.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.725+0000 7f1c66e00640 1 -- 172.21.15.150:0/422437616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c600aa880 con 0x7f1c600c3090 2024-09-17T04:58:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c5d7fa640 1 -- 172.21.15.150:0/422437616 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5704 (secure 0 0 0) 0x7f1c4c014030 con 0x7f1c600c3090 2024-09-17T04:58:31.726 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:31.726 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":67,"btime":"2024-09-17T04:58:30:930551+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:30.930546+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:58:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 67 2024-09-17T04:58:31.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c66e00640 1 -- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1c44073200 msgr2=0x7f1c44075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:31.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c66e00640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1c44073200 0x7f1c44075710 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1c540ae000 tx=0x7f1c5401afc0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c66e00640 1 -- 172.21.15.150:0/422437616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c600c3090 msgr2=0x7f1c600c7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:31.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c66e00640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c600c3090 0x7f1c600c7180 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c00c970 tx=0x7f1c4c00ce30 comp rx=0 tx=0).stop 2024-09-17T04:58:31.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c66e00640 1 -- 172.21.15.150:0/422437616 shutdown_connections 2024-09-17T04:58:31.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c66e00640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1c44073200 0x7f1c44075710 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c66e00640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1c600c3a30 0x7f1c601f0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c66e00640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1c600c3090 0x7f1c600c7180 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c66e00640 1 --2- 172.21.15.150:0/422437616 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c600748e0 0x7f1c601478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c66e00640 1 -- 172.21.15.150:0/422437616 >> 172.21.15.150:0/422437616 conn(0x7f1c600c5ae0 msgr2=0x7f1c60154df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:31.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c66e00640 1 -- 172.21.15.150:0/422437616 shutdown_connections 2024-09-17T04:58:31.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.729+0000 7f1c66e00640 1 -- 172.21.15.150:0/422437616 wait complete. 2024-09-17T04:58:31.748 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-17T04:58:31.748 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T04:58:31.748 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:31.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.897+0000 7fae5909c640 1 -- 172.21.15.150:0/603065421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae541535b0 msgr2=0x7fae54157440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:31.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.897+0000 7fae5909c640 1 --2- 172.21.15.150:0/603065421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae541535b0 0x7fae54157440 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fae44009900 tx=0x7fae4401c850 comp rx=0 tx=0).stop 2024-09-17T04:58:31.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.897+0000 7fae5909c640 1 -- 172.21.15.150:0/603065421 shutdown_connections 2024-09-17T04:58:31.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.897+0000 7fae5909c640 1 --2- 172.21.15.150:0/603065421 >> v2:172.21.15.176:3300/0 conn(0x7fae540c5760 0x7fae540c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.897+0000 7fae5909c640 1 --2- 172.21.15.150:0/603065421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae541535b0 0x7fae54157440 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.897+0000 7fae5909c640 1 -- 172.21.15.150:0/603065421 >> 172.21.15.150:0/603065421 conn(0x7fae540aaa80 msgr2=0x7fae540aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:31.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.897+0000 7fae5909c640 1 -- 172.21.15.150:0/603065421 shutdown_connections 2024-09-17T04:58:31.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.897+0000 7fae5909c640 1 -- 172.21.15.150:0/603065421 wait complete. 2024-09-17T04:58:31.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.897+0000 7fae5909c640 1 Processor -- start 2024-09-17T04:58:31.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.897+0000 7fae5909c640 1 -- start start 2024-09-17T04:58:31.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5909c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae540a7f10 0x7fae541441e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:31.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5909c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae540c5760 0x7fae541f0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:31.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5909c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae541535b0 0x7fae541420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:31.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5909c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fae540718d0 con 0x7fae541535b0 2024-09-17T04:58:31.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5909c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fae540a8730 con 0x7fae540a7f10 2024-09-17T04:58:31.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5909c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fae540c5370 con 0x7fae540c5760 2024-09-17T04:58:31.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae541535b0 0x7fae541420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:31.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae541535b0 0x7fae541420b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47690/0 (socket says 172.21.15.150:47690) 2024-09-17T04:58:31.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5359e640 1 -- 172.21.15.150:0/2064508639 learned_addr learned my addr 172.21.15.150:0/2064508639 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:31.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae4bfff640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae540c5760 0x7fae541f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:31.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae52d9d640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae540a7f10 0x7fae541441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:31.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5359e640 1 -- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae540c5760 msgr2=0x7fae541f0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:31.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5359e640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae540c5760 0x7fae541f0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5359e640 1 -- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae540a7f10 msgr2=0x7fae541441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:31.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5359e640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae540a7f10 0x7fae541441e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:31.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5359e640 1 -- 172.21.15.150:0/2064508639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae541483f0 con 0x7fae541535b0 2024-09-17T04:58:31.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae5359e640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae541535b0 0x7fae541420b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00c6e0 tx=0x7fae4c00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:31.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae4b7fe640 1 -- 172.21.15.150:0/2064508639 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4c003b80 con 0x7fae541535b0 2024-09-17T04:58:31.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae4b7fe640 1 -- 172.21.15.150:0/2064508639 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae4c003d20 con 0x7fae541535b0 2024-09-17T04:58:31.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.901+0000 7fae4b7fe640 1 -- 172.21.15.150:0/2064508639 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4c005330 con 0x7fae541535b0 2024-09-17T04:58:31.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.905+0000 7fae5909c640 1 -- 172.21.15.150:0/2064508639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae54143b60 con 0x7fae541535b0 2024-09-17T04:58:31.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.905+0000 7fae5909c640 1 -- 172.21.15.150:0/2064508639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae541433f0 con 0x7fae541535b0 2024-09-17T04:58:31.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.905+0000 7fae5909c640 1 -- 172.21.15.150:0/2064508639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae541539c0 con 0x7fae541535b0 2024-09-17T04:58:31.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.905+0000 7fae4b7fe640 1 -- 172.21.15.150:0/2064508639 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fae4c0054d0 con 0x7fae541535b0 2024-09-17T04:58:31.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.905+0000 7fae4b7fe640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae34073230 0x7fae34075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:31.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.905+0000 7fae4b7fe640 1 -- 172.21.15.150:0/2064508639 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7fae4c094270 con 0x7fae541535b0 2024-09-17T04:58:31.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.905+0000 7fae52d9d640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae34073230 0x7fae34075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:31.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.909+0000 7fae52d9d640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae34073230 0x7fae34075740 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fae440af000 tx=0x7fae4401b090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:31.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:31.913+0000 7fae4b7fe640 1 -- 172.21.15.150:0/2064508639 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fae4c020060 con 0x7fae541535b0 2024-09-17T04:58:32.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.197+0000 7fae5909c640 1 -- 172.21.15.150:0/2064508639 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae54149fa0 con 0x7fae541535b0 2024-09-17T04:58:32.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.197+0000 7fae4b7fe640 1 -- 172.21.15.150:0/2064508639 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5707 (secure 0 0 0) 0x7fae4c065020 con 0x7fae541535b0 2024-09-17T04:58:32.197 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:32.197 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":68,"btime":"2024-09-17T04:58:31:932660+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:31.302923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:58:32.198 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 68 2024-09-17T04:58:32.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.201+0000 7fae5909c640 1 -- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae34073230 msgr2=0x7fae34075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:32.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.201+0000 7fae5909c640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae34073230 0x7fae34075740 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fae440af000 tx=0x7fae4401b090 comp rx=0 tx=0).stop 2024-09-17T04:58:32.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.201+0000 7fae5909c640 1 -- 172.21.15.150:0/2064508639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae541535b0 msgr2=0x7fae541420b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:32.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.201+0000 7fae5909c640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae541535b0 0x7fae541420b0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00c6e0 tx=0x7fae4c00cba0 comp rx=0 tx=0).stop 2024-09-17T04:58:32.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.201+0000 7fae5909c640 1 -- 172.21.15.150:0/2064508639 shutdown_connections 2024-09-17T04:58:32.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.201+0000 7fae5909c640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae34073230 0x7fae34075740 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:32.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.201+0000 7fae5909c640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae541535b0 0x7fae541420b0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:32.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.201+0000 7fae5909c640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae540c5760 0x7fae541f0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:32.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.201+0000 7fae5909c640 1 --2- 172.21.15.150:0/2064508639 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae540a7f10 0x7fae541441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:32.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.201+0000 7fae5909c640 1 -- 172.21.15.150:0/2064508639 >> 172.21.15.150:0/2064508639 conn(0x7fae540aaa80 msgr2=0x7fae54074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:32.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.201+0000 7fae5909c640 1 -- 172.21.15.150:0/2064508639 shutdown_connections 2024-09-17T04:58:32.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.201+0000 7fae5909c640 1 -- 172.21.15.150:0/2064508639 wait complete. 2024-09-17T04:58:32.220 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T04:58:31:932660+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 68, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 65, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5841": { "addr": "172.21.15.150:6833/1275062894", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 1275062894, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 1275062894, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5841, "incarnation": 62, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:58:03.946007+0000", "qdb_cluster": [ 5841 ], "qdb_leader": 5841, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5841 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 68, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5901": { "addr": "172.21.15.150:6835/2081398022", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 2081398022, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2081398022, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5901, "incarnation": 67, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:58:31.302923+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5901 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:58:32.695 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:32.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.845+0000 7f15e8813640 1 -- 172.21.15.150:0/2345574854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e40c7eb0 msgr2=0x7f15e41540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:32.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.845+0000 7f15e8813640 1 --2- 172.21.15.150:0/2345574854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e40c7eb0 0x7f15e41540b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009870 tx=0x7f15d401c820 comp rx=0 tx=0).stop 2024-09-17T04:58:32.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.845+0000 7f15e8813640 1 -- 172.21.15.150:0/2345574854 shutdown_connections 2024-09-17T04:58:32.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.845+0000 7f15e8813640 1 --2- 172.21.15.150:0/2345574854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e40c7eb0 0x7f15e41540b0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:32.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.845+0000 7f15e8813640 1 -- 172.21.15.150:0/2345574854 >> 172.21.15.150:0/2345574854 conn(0x7f15e40aab60 msgr2=0x7f15e40aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:32.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.845+0000 7f15e8813640 1 -- 172.21.15.150:0/2345574854 shutdown_connections 2024-09-17T04:58:32.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.845+0000 7f15e8813640 1 -- 172.21.15.150:0/2345574854 wait complete. 2024-09-17T04:58:32.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.845+0000 7f15e8813640 1 Processor -- start 2024-09-17T04:58:32.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.845+0000 7f15e8813640 1 -- start start 2024-09-17T04:58:32.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e8813640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e40c7eb0 0x7f15e41f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:32.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e8813640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15e4149d70 0x7f15e414cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:32.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e8813640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e414a180 0x7f15e41500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:32.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e8813640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f15e4156540 con 0x7f15e40c7eb0 2024-09-17T04:58:32.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e8813640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41566c0 con 0x7f15e414a180 2024-09-17T04:58:32.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e8813640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f15e4156840 con 0x7f15e4149d70 2024-09-17T04:58:32.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e40c7eb0 0x7f15e41f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:32.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e40c7eb0 0x7f15e41f19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47702/0 (socket says 172.21.15.150:47702) 2024-09-17T04:58:32.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e259c640 1 -- 172.21.15.150:0/48151001 learned_addr learned my addr 172.21.15.150:0/48151001 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:32.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e2d9d640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e414a180 0x7f15e41500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:32.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e1d9b640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15e4149d70 0x7f15e414cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:32.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e259c640 1 -- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15e4149d70 msgr2=0x7f15e414cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:32.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e259c640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15e4149d70 0x7f15e414cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:32.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e259c640 1 -- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e414a180 msgr2=0x7f15e41500c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:32.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e259c640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e414a180 0x7f15e41500c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:32.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e259c640 1 -- 172.21.15.150:0/48151001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e4148eb0 con 0x7f15e40c7eb0 2024-09-17T04:58:32.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e259c640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e40c7eb0 0x7f15e41f19a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f15d4019430 tx=0x7f15d40a67b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:32.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15d2ffd640 1 -- 172.21.15.150:0/48151001 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d40041c0 con 0x7f15e40c7eb0 2024-09-17T04:58:32.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15d2ffd640 1 -- 172.21.15.150:0/48151001 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15d4004450 con 0x7f15e40c7eb0 2024-09-17T04:58:32.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e8813640 1 -- 172.21.15.150:0/48151001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e414d3e0 con 0x7f15e40c7eb0 2024-09-17T04:58:32.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15d2ffd640 1 -- 172.21.15.150:0/48151001 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4005020 con 0x7f15e40c7eb0 2024-09-17T04:58:32.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e8813640 1 -- 172.21.15.150:0/48151001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e4152a40 con 0x7f15e40c7eb0 2024-09-17T04:58:32.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15e8813640 1 -- 172.21.15.150:0/48151001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e40ae070 con 0x7f15e40c7eb0 2024-09-17T04:58:32.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15d2ffd640 1 -- 172.21.15.150:0/48151001 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f15d40c1030 con 0x7f15e40c7eb0 2024-09-17T04:58:32.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15d2ffd640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f15c4073200 0x7f15c4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:32.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.849+0000 7f15d2ffd640 1 -- 172.21.15.150:0/48151001 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f15d412f340 con 0x7f15e40c7eb0 2024-09-17T04:58:32.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.853+0000 7f15e1d9b640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f15c4073200 0x7f15c4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:32.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.853+0000 7f15e1d9b640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f15c4073200 0x7f15c4075710 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f15e40c2c90 tx=0x7f15d8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:32.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:32.857+0000 7f15d2ffd640 1 -- 172.21.15.150:0/48151001 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f15d40c8020 con 0x7f15e40c7eb0 2024-09-17T04:58:33.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.077+0000 7f15e8813640 1 -- 172.21.15.150:0/48151001 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15e41540b0 con 0x7f15e40c7eb0 2024-09-17T04:58:33.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.077+0000 7f15d2ffd640 1 -- 172.21.15.150:0/48151001 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5704 (secure 0 0 0) 0x7f15d4016410 con 0x7f15e40c7eb0 2024-09-17T04:58:33.077 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:33.077 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":69,"btime":"2024-09-17T04:58:32:943065+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:31.946626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:58:33.078 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 69 2024-09-17T04:58:33.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.081+0000 7f15e8813640 1 -- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f15c4073200 msgr2=0x7f15c4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:33.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.081+0000 7f15e8813640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f15c4073200 0x7f15c4075710 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f15e40c2c90 tx=0x7f15d8009040 comp rx=0 tx=0).stop 2024-09-17T04:58:33.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.081+0000 7f15e8813640 1 -- 172.21.15.150:0/48151001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e40c7eb0 msgr2=0x7f15e41f19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:33.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.081+0000 7f15e8813640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e40c7eb0 0x7f15e41f19a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f15d4019430 tx=0x7f15d40a67b0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.081+0000 7f15e8813640 1 -- 172.21.15.150:0/48151001 shutdown_connections 2024-09-17T04:58:33.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.081+0000 7f15e8813640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f15c4073200 0x7f15c4075710 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.081+0000 7f15e8813640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e414a180 0x7f15e41500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.081+0000 7f15e8813640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15e4149d70 0x7f15e414cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.081+0000 7f15e8813640 1 --2- 172.21.15.150:0/48151001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e40c7eb0 0x7f15e41f19a0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.081+0000 7f15e8813640 1 -- 172.21.15.150:0/48151001 >> 172.21.15.150:0/48151001 conn(0x7f15e40aab60 msgr2=0x7f15e4156b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:33.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.081+0000 7f15e8813640 1 -- 172.21.15.150:0/48151001 shutdown_connections 2024-09-17T04:58:33.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.081+0000 7f15e8813640 1 -- 172.21.15.150:0/48151001 wait complete. 2024-09-17T04:58:33.099 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-17T04:58:33.100 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:58:33.100 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:58:33.114 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:58:33.115 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:33.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.301+0000 7f0773f0d640 1 -- 172.21.15.150:0/4194117198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f076c074980 msgr2=0x7f076c0aaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.301+0000 7f0773f0d640 1 --2- 172.21.15.150:0/4194117198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f076c074980 0x7f076c0aaf80 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f076c0c2be0 tx=0x7f076401ccf0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.301+0000 7f0773f0d640 1 -- 172.21.15.150:0/4194117198 shutdown_connections 2024-09-17T04:58:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.301+0000 7f0773f0d640 1 --2- 172.21.15.150:0/4194117198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f076c074980 0x7f076c0aaf80 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.301+0000 7f0773f0d640 1 --2- 172.21.15.150:0/4194117198 >> v2:172.21.15.176:3300/0 conn(0x7f076c074570 0x7f076c0aaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.301+0000 7f0773f0d640 1 -- 172.21.15.150:0/4194117198 >> 172.21.15.150:0/4194117198 conn(0x7f076c0c5ae0 msgr2=0x7f076c073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:33.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.301+0000 7f0773f0d640 1 -- 172.21.15.150:0/4194117198 shutdown_connections 2024-09-17T04:58:33.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.301+0000 7f0773f0d640 1 -- 172.21.15.150:0/4194117198 wait complete. 2024-09-17T04:58:33.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.301+0000 7f0773f0d640 1 Processor -- start 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.301+0000 7f0773f0d640 1 -- start start 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0773f0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f076c06fca0 0x7f076c14c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0773f0d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f076c074570 0x7f076c1f0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0773f0d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f076c074980 0x7f076c1450c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0773f0d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f076c15f6a0 con 0x7f076c06fca0 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0773f0d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f076c0c4be0 con 0x7f076c074570 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0773f0d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f076c15f820 con 0x7f076c074980 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0771ca9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f076c06fca0 0x7f076c14c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0771ca9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f076c06fca0 0x7f076c14c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47722/0 (socket says 172.21.15.150:47722) 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0771ca9640 1 -- 172.21.15.150:0/1494860532 learned_addr learned my addr 172.21.15.150:0/1494860532 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f07714a8640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f076c074570 0x7f076c1f0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f07724aa640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f076c074980 0x7f076c1450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0771ca9640 1 -- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f076c074980 msgr2=0x7f076c1450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0771ca9640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f076c074980 0x7f076c1450c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0771ca9640 1 -- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f076c074570 msgr2=0x7f076c1f0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0771ca9640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f076c074570 0x7f076c1f0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0771ca9640 1 -- 172.21.15.150:0/1494860532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076c155b20 con 0x7f076c06fca0 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f07714a8640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f076c074570 0x7f076c1f0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0771ca9640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f076c06fca0 0x7f076c14c780 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f076800b410 tx=0x7f076800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f07627fc640 1 -- 172.21.15.150:0/1494860532 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768015030 con 0x7f076c06fca0 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f07627fc640 1 -- 172.21.15.150:0/1494860532 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0768004020 con 0x7f076c06fca0 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0773f0d640 1 -- 172.21.15.150:0/1494860532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f076c1542a0 con 0x7f076c06fca0 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f07627fc640 1 -- 172.21.15.150:0/1494860532 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07680044b0 con 0x7f076c06fca0 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0773f0d640 1 -- 172.21.15.150:0/1494860532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f076c1500a0 con 0x7f076c06fca0 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f0773f0d640 1 -- 172.21.15.150:0/1494860532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f076c1f1b10 con 0x7f076c06fca0 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.305+0000 7f07627fc640 1 -- 172.21.15.150:0/1494860532 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0768018070 con 0x7f076c06fca0 2024-09-17T04:58:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.309+0000 7f07627fc640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0750073230 0x7f0750075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:33.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.309+0000 7f07627fc640 1 -- 172.21.15.150:0/1494860532 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f0768094400 con 0x7f076c06fca0 2024-09-17T04:58:33.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.309+0000 7f07714a8640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0750073230 0x7f0750075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:33.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.309+0000 7f07714a8640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0750073230 0x7f0750075740 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f075c005fd0 tx=0x7f075c005de0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:33.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.313+0000 7f07627fc640 1 -- 172.21.15.150:0/1494860532 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f076805edd0 con 0x7f076c06fca0 2024-09-17T04:58:33.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 -- 172.21.15.150:0/1494860532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f076c153790 con 0x7f076c06fca0 2024-09-17T04:58:33.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f07627fc640 1 -- 172.21.15.150:0/1494860532 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5704 (secure 0 0 0) 0x7f076805e6d0 con 0x7f076c06fca0 2024-09-17T04:58:33.554 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:33.555 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":69,"btime":"2024-09-17T04:58:32:943065+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:31.946626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T04:58:33.555 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 69 2024-09-17T04:58:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 -- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0750073230 msgr2=0x7f0750075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0750073230 0x7f0750075740 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f075c005fd0 tx=0x7f075c005de0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 -- 172.21.15.150:0/1494860532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f076c06fca0 msgr2=0x7f076c14c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f076c06fca0 0x7f076c14c780 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f076800b410 tx=0x7f076800b8d0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 -- 172.21.15.150:0/1494860532 shutdown_connections 2024-09-17T04:58:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0750073230 0x7f0750075740 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f076c074570 0x7f076c1f0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f076c074980 0x7f076c1450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 --2- 172.21.15.150:0/1494860532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f076c06fca0 0x7f076c14c780 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:33.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 -- 172.21.15.150:0/1494860532 >> 172.21.15.150:0/1494860532 conn(0x7f076c0c5ae0 msgr2=0x7f076c148280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:33.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 -- 172.21.15.150:0/1494860532 shutdown_connections 2024-09-17T04:58:33.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:33.557+0000 7f0773f0d640 1 -- 172.21.15.150:0/1494860532 wait complete. 2024-09-17T04:58:33.576 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:58:34.221 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:34.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.389+0000 7f862c108640 1 -- 172.21.15.150:0/526599149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8624070e50 msgr2=0x7f8624071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:34.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.389+0000 7f862c108640 1 --2- 172.21.15.150:0/526599149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8624070e50 0x7f8624071260 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8620009900 tx=0x7f862001c850 comp rx=0 tx=0).stop 2024-09-17T04:58:34.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.389+0000 7f862c108640 1 -- 172.21.15.150:0/526599149 shutdown_connections 2024-09-17T04:58:34.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.389+0000 7f862c108640 1 --2- 172.21.15.150:0/526599149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8624070e50 0x7f8624071260 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:34.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.389+0000 7f862c108640 1 -- 172.21.15.150:0/526599149 >> 172.21.15.150:0/526599149 conn(0x7f86240c4a00 msgr2=0x7f86240c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:34.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.389+0000 7f862c108640 1 -- 172.21.15.150:0/526599149 shutdown_connections 2024-09-17T04:58:34.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.389+0000 7f862c108640 1 -- 172.21.15.150:0/526599149 wait complete. 2024-09-17T04:58:34.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.389+0000 7f862c108640 1 Processor -- start 2024-09-17T04:58:34.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.389+0000 7f862c108640 1 -- start start 2024-09-17T04:58:34.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862c108640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8624070e50 0x7f862414b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:34.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862c108640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86240717a0 0x7f8624154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:34.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862c108640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8624073440 0x7f8624156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:34.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862c108640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f86240ad820 con 0x7f8624073440 2024-09-17T04:58:34.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862c108640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8624075180 con 0x7f8624070e50 2024-09-17T04:58:34.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862c108640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f86240ad9a0 con 0x7f86240717a0 2024-09-17T04:58:34.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862a6a5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8624073440 0x7f8624156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:34.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862a6a5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8624073440 0x7f8624156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47740/0 (socket says 172.21.15.150:47740) 2024-09-17T04:58:34.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862a6a5640 1 -- 172.21.15.150:0/2991732139 learned_addr learned my addr 172.21.15.150:0/2991732139 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:34.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f8629ea4640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8624070e50 0x7f862414b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:34.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f86296a3640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86240717a0 0x7f8624154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:34.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862a6a5640 1 -- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86240717a0 msgr2=0x7f8624154990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:34.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862a6a5640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86240717a0 0x7f8624154990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:34.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862a6a5640 1 -- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8624070e50 msgr2=0x7f862414b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:34.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862a6a5640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8624070e50 0x7f862414b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:34.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862a6a5640 1 -- 172.21.15.150:0/2991732139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862414bac0 con 0x7f8624073440 2024-09-17T04:58:34.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862a6a5640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8624073440 0x7f8624156d00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f861c00c740 tx=0x7f861c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:34.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f861a7fc640 1 -- 172.21.15.150:0/2991732139 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861c003b80 con 0x7f8624073440 2024-09-17T04:58:34.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f861a7fc640 1 -- 172.21.15.150:0/2991732139 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f861c00ce50 con 0x7f8624073440 2024-09-17T04:58:34.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862c108640 1 -- 172.21.15.150:0/2991732139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86240c2200 con 0x7f8624073440 2024-09-17T04:58:34.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f861a7fc640 1 -- 172.21.15.150:0/2991732139 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861c00a850 con 0x7f8624073440 2024-09-17T04:58:34.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862c108640 1 -- 172.21.15.150:0/2991732139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f862414c020 con 0x7f8624073440 2024-09-17T04:58:34.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f862c108640 1 -- 172.21.15.150:0/2991732139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8624071260 con 0x7f8624073440 2024-09-17T04:58:34.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.393+0000 7f861a7fc640 1 -- 172.21.15.150:0/2991732139 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f861c00a9f0 con 0x7f8624073440 2024-09-17T04:58:34.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.397+0000 7f861a7fc640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8608073200 0x7f8608075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:34.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.397+0000 7f861a7fc640 1 -- 172.21.15.150:0/2991732139 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f861c094740 con 0x7f8624073440 2024-09-17T04:58:34.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.397+0000 7f8629ea4640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8608073200 0x7f8608075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:34.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.397+0000 7f8629ea4640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8608073200 0x7f8608075710 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f86241513e0 tx=0x7f8614008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:34.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.401+0000 7f861a7fc640 1 -- 172.21.15.150:0/2991732139 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f862414bac0 con 0x7f8624073440 2024-09-17T04:58:34.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.609+0000 7f862c108640 1 -- 172.21.15.150:0/2991732139 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86240c60f0 con 0x7f8624073440 2024-09-17T04:58:34.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.609+0000 7f861a7fc640 1 -- 172.21.15.150:0/2991732139 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5711 (secure 0 0 0) 0x7f861c05f080 con 0x7f8624073440 2024-09-17T04:58:34.609 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:34.610 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":70,"btime":"2024-09-17T04:58:33:945215+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:34.610 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 70 2024-09-17T04:58:34.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.613+0000 7f862c108640 1 -- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8608073200 msgr2=0x7f8608075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:34.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.613+0000 7f862c108640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8608073200 0x7f8608075710 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f86241513e0 tx=0x7f8614008040 comp rx=0 tx=0).stop 2024-09-17T04:58:34.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.613+0000 7f862c108640 1 -- 172.21.15.150:0/2991732139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8624073440 msgr2=0x7f8624156d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:34.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.613+0000 7f862c108640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8624073440 0x7f8624156d00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f861c00c740 tx=0x7f861c00cc00 comp rx=0 tx=0).stop 2024-09-17T04:58:34.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.613+0000 7f862c108640 1 -- 172.21.15.150:0/2991732139 shutdown_connections 2024-09-17T04:58:34.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.613+0000 7f862c108640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8608073200 0x7f8608075710 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:34.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.613+0000 7f862c108640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8624073440 0x7f8624156d00 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:34.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.613+0000 7f862c108640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86240717a0 0x7f8624154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:34.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.613+0000 7f862c108640 1 --2- 172.21.15.150:0/2991732139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8624070e50 0x7f862414b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:34.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.613+0000 7f862c108640 1 -- 172.21.15.150:0/2991732139 >> 172.21.15.150:0/2991732139 conn(0x7f86240c4a00 msgr2=0x7f86240a84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:34.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.613+0000 7f862c108640 1 -- 172.21.15.150:0/2991732139 shutdown_connections 2024-09-17T04:58:34.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:34.613+0000 7f862c108640 1 -- 172.21.15.150:0/2991732139 wait complete. 2024-09-17T04:58:34.632 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5404 2024-09-17T04:58:34.633 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.d 2024-09-17T04:58:35.286 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T04:58:35.289+0000 7f93345b7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:58:35.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:35.633 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-17T04:58:35.634 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:58:35.634 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:58:35.636 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:58:35.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:35.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.749+0000 7f6aeee24640 1 -- 172.21.15.150:0/3040458977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac80079b0 msgr2=0x7f6ac8009dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:35.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.749+0000 7f6aeee24640 1 --2- 172.21.15.150:0/3040458977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac80079b0 0x7f6ac8009dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0001830 tx=0x7f6ad001c800 comp rx=0 tx=0).stop 2024-09-17T04:58:35.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.749+0000 7f6aeee24640 1 -- 172.21.15.150:0/3040458977 shutdown_connections 2024-09-17T04:58:35.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.749+0000 7f6aeee24640 1 --2- 172.21.15.150:0/3040458977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac80079b0 0x7f6ac8009dd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:35.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.749+0000 7f6aeee24640 1 -- 172.21.15.150:0/3040458977 >> 172.21.15.150:0/3040458977 conn(0x7f6ae8070b50 msgr2=0x7f6ae8070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:35.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.749+0000 7f6aeee24640 1 -- 172.21.15.150:0/3040458977 shutdown_connections 2024-09-17T04:58:35.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.749+0000 7f6aeee24640 1 -- 172.21.15.150:0/3040458977 wait complete. 2024-09-17T04:58:35.750 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-09-17T04:58:35.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.753+0000 7f6aeee24640 1 Processor -- start 2024-09-17T04:58:35.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.753+0000 7f6aeee24640 1 -- start start 2024-09-17T04:58:35.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aeee24640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ac80079b0 0x7f6ae8149660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:35.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aeee24640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ae81490c0 0x7f6ae8158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:35.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aeee24640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ae81e5d60 0x7f6ae8144c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:35.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aeee24640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae80aa410 con 0x7f6ae81e5d60 2024-09-17T04:58:35.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aeee24640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae80aa290 con 0x7f6ae81490c0 2024-09-17T04:58:35.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aeee24640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6ae80a9e20 con 0x7f6ac80079b0 2024-09-17T04:58:35.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aed3c1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ae81e5d60 0x7f6ae8144c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aecbc0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ac80079b0 0x7f6ae8149660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aed3c1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ae81e5d60 0x7f6ae8144c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47744/0 (socket says 172.21.15.150:47744) 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aed3c1640 1 -- 172.21.15.150:0/1294473709 learned_addr learned my addr 172.21.15.150:0/1294473709 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6ae7fff640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ae81490c0 0x7f6ae8158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aed3c1640 1 -- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ac80079b0 msgr2=0x7f6ae8149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aed3c1640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ac80079b0 0x7f6ae8149660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aed3c1640 1 -- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ae81490c0 msgr2=0x7f6ae8158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aed3c1640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ae81490c0 0x7f6ae8158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aed3c1640 1 -- 172.21.15.150:0/1294473709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae8158a60 con 0x7f6ae81e5d60 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aed3c1640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ae81e5d60 0x7f6ae8144c00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8075450 tx=0x7f6ad800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6ae57fa640 1 -- 172.21.15.150:0/1294473709 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8003d90 con 0x7f6ae81e5d60 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6ae57fa640 1 -- 172.21.15.150:0/1294473709 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ad8004b30 con 0x7f6ae81e5d60 2024-09-17T04:58:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aeee24640 1 -- 172.21.15.150:0/1294473709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae8149ca0 con 0x7f6ae81e5d60 2024-09-17T04:58:35.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6ae57fa640 1 -- 172.21.15.150:0/1294473709 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8004030 con 0x7f6ae81e5d60 2024-09-17T04:58:35.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aeee24640 1 -- 172.21.15.150:0/1294473709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae81e6640 con 0x7f6ae81e5d60 2024-09-17T04:58:35.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.757+0000 7f6aeee24640 1 -- 172.21.15.150:0/1294473709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae8159f90 con 0x7f6ae81e5d60 2024-09-17T04:58:35.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.761+0000 7f6ae57fa640 1 -- 172.21.15.150:0/1294473709 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6ad80041d0 con 0x7f6ae81e5d60 2024-09-17T04:58:35.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.761+0000 7f6ae57fa640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6acc073230 0x7f6acc075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.761+0000 7f6ae57fa640 1 -- 172.21.15.150:0/1294473709 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f6ad8095bf0 con 0x7f6ae81e5d60 2024-09-17T04:58:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.761+0000 7f6aecbc0640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6acc073230 0x7f6acc075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.761+0000 7f6aecbc0640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6acc073230 0x7f6acc075740 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0006b80 tx=0x7f6ae0000fa0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:35.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.765+0000 7f6ae57fa640 1 -- 172.21.15.150:0/1294473709 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6ad8020020 con 0x7f6ae81e5d60 2024-09-17T04:58:35.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.797+0000 7f96437bc640 1 -- 172.21.15.150:0/123745800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f963c0c97a0 msgr2=0x7f963c151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:35.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.797+0000 7f96437bc640 1 --2- 172.21.15.150:0/123745800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f963c0c97a0 0x7f963c151c70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f9630009840 tx=0x7f963001c820 comp rx=0 tx=0).stop 2024-09-17T04:58:35.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.797+0000 7f96437bc640 1 -- 172.21.15.150:0/123745800 shutdown_connections 2024-09-17T04:58:35.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.797+0000 7f96437bc640 1 --2- 172.21.15.150:0/123745800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f963c0c97a0 0x7f963c151c70 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:35.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.797+0000 7f96437bc640 1 -- 172.21.15.150:0/123745800 >> 172.21.15.150:0/123745800 conn(0x7f963c0abb00 msgr2=0x7f963c0abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:35.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.797+0000 7f96437bc640 1 -- 172.21.15.150:0/123745800 shutdown_connections 2024-09-17T04:58:35.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.797+0000 7f96437bc640 1 -- 172.21.15.150:0/123745800 wait complete. 2024-09-17T04:58:35.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f96437bc640 1 Processor -- start 2024-09-17T04:58:35.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f96437bc640 1 -- start start 2024-09-17T04:58:35.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f96437bc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f963c072ea0 0x7f963c06fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:35.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f96437bc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f963c0732b0 0x7f963c15a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f96437bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f963c0c97a0 0x7f963c1547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f96437bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f963c0c4d60 con 0x7f963c0c97a0 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f96437bc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f963c0c4be0 con 0x7f963c0732b0 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f96437bc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f963c1627a0 con 0x7f963c072ea0 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f9641d59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f963c0c97a0 0x7f963c1547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f9641d59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f963c0c97a0 0x7f963c1547f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47762/0 (socket says 172.21.15.150:47762) 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f9641d59640 1 -- 172.21.15.150:0/1075476207 learned_addr learned my addr 172.21.15.150:0/1075476207 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f9641558640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f963c072ea0 0x7f963c06fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f9640d57640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f963c0732b0 0x7f963c15a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f9641d59640 1 -- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f963c072ea0 msgr2=0x7f963c06fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f9641d59640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f963c072ea0 0x7f963c06fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f9641d59640 1 -- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f963c0732b0 msgr2=0x7f963c15a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:35.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f9641d59640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f963c0732b0 0x7f963c15a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:35.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f9641d59640 1 -- 172.21.15.150:0/1075476207 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f963c159250 con 0x7f963c0c97a0 2024-09-17T04:58:35.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f9641d59640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f963c0c97a0 0x7f963c1547f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f963400c760 tx=0x7f963400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:35.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f962dffb640 1 -- 172.21.15.150:0/1075476207 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9634003b80 con 0x7f963c0c97a0 2024-09-17T04:58:35.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f962dffb640 1 -- 172.21.15.150:0/1075476207 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9634003d20 con 0x7f963c0c97a0 2024-09-17T04:58:35.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f962dffb640 1 -- 172.21.15.150:0/1075476207 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963400aab0 con 0x7f963c0c97a0 2024-09-17T04:58:35.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f96437bc640 1 -- 172.21.15.150:0/1075476207 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f963c0c3cf0 con 0x7f963c0c97a0 2024-09-17T04:58:35.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.801+0000 7f96437bc640 1 -- 172.21.15.150:0/1075476207 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f963c1f3dc0 con 0x7f963c0c97a0 2024-09-17T04:58:35.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.805+0000 7f96437bc640 1 -- 172.21.15.150:0/1075476207 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f963c151c70 con 0x7f963c0c97a0 2024-09-17T04:58:35.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.805+0000 7f962dffb640 1 -- 172.21.15.150:0/1075476207 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f963400ac50 con 0x7f963c0c97a0 2024-09-17T04:58:35.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.805+0000 7f962dffb640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9620073200 0x7f9620075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:35.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.805+0000 7f962dffb640 1 -- 172.21.15.150:0/1075476207 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f96340945f0 con 0x7f963c0c97a0 2024-09-17T04:58:35.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.805+0000 7f9641558640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9620073200 0x7f9620075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:35.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.805+0000 7f9641558640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9620073200 0x7f9620075710 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f96300af000 tx=0x7f963001b060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:35.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.809+0000 7f962dffb640 1 -- 172.21.15.150:0/1075476207 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f963405e8e0 con 0x7f963c0c97a0 2024-09-17T04:58:35.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.997+0000 7f6aeee24640 1 -- 172.21.15.150:0/1294473709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ae80c35c0 con 0x7f6ae81e5d60 2024-09-17T04:58:35.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:35.997+0000 7f6ae57fa640 1 -- 172.21.15.150:0/1294473709 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5711 (secure 0 0 0) 0x7f6ae8158a60 con 0x7f6ae81e5d60 2024-09-17T04:58:35.997 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:35.998 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":70,"btime":"2024-09-17T04:58:33:945215+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:35.998 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 70 2024-09-17T04:58:35.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.001+0000 7f6aeee24640 1 -- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6acc073230 msgr2=0x7f6acc075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:35.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.001+0000 7f6aeee24640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6acc073230 0x7f6acc075740 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0006b80 tx=0x7f6ae0000fa0 comp rx=0 tx=0).stop 2024-09-17T04:58:35.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.001+0000 7f6aeee24640 1 -- 172.21.15.150:0/1294473709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ae81e5d60 msgr2=0x7f6ae8144c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:35.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.001+0000 7f6aeee24640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ae81e5d60 0x7f6ae8144c00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8075450 tx=0x7f6ad800f790 comp rx=0 tx=0).stop 2024-09-17T04:58:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.001+0000 7f6aeee24640 1 -- 172.21.15.150:0/1294473709 shutdown_connections 2024-09-17T04:58:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.001+0000 7f6aeee24640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6acc073230 0x7f6acc075740 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.001+0000 7f6aeee24640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ae81490c0 0x7f6ae8158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.001+0000 7f6aeee24640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ae81e5d60 0x7f6ae8144c00 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.001+0000 7f6aeee24640 1 --2- 172.21.15.150:0/1294473709 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ac80079b0 0x7f6ae8149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.001+0000 7f6aeee24640 1 -- 172.21.15.150:0/1294473709 >> 172.21.15.150:0/1294473709 conn(0x7f6ae8070b50 msgr2=0x7f6ae814f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.001+0000 7f6aeee24640 1 -- 172.21.15.150:0/1294473709 shutdown_connections 2024-09-17T04:58:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.001+0000 7f6aeee24640 1 -- 172.21.15.150:0/1294473709 wait complete. 2024-09-17T04:58:36.020 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:58:36.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.037+0000 7f96437bc640 1 -- 172.21.15.150:0/1075476207 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f963c0c40f0 con 0x7f963c0c97a0 2024-09-17T04:58:36.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.037+0000 7f962dffb640 1 -- 172.21.15.150:0/1075476207 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5711 (secure 0 0 0) 0x7f9634063310 con 0x7f963c0c97a0 2024-09-17T04:58:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:36.036 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":70,"btime":"2024-09-17T04:58:33:945215+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:36.036 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 70 2024-09-17T04:58:36.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.041+0000 7f96437bc640 1 -- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9620073200 msgr2=0x7f9620075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:36.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.041+0000 7f96437bc640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9620073200 0x7f9620075710 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f96300af000 tx=0x7f963001b060 comp rx=0 tx=0).stop 2024-09-17T04:58:36.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.041+0000 7f96437bc640 1 -- 172.21.15.150:0/1075476207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f963c0c97a0 msgr2=0x7f963c1547f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.041+0000 7f96437bc640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f963c0c97a0 0x7f963c1547f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f963400c760 tx=0x7f963400cc20 comp rx=0 tx=0).stop 2024-09-17T04:58:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.041+0000 7f96437bc640 1 -- 172.21.15.150:0/1075476207 shutdown_connections 2024-09-17T04:58:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.041+0000 7f96437bc640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9620073200 0x7f9620075710 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.041+0000 7f96437bc640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f963c0732b0 0x7f963c15a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.041+0000 7f96437bc640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f963c0c97a0 0x7f963c1547f0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.041+0000 7f96437bc640 1 --2- 172.21.15.150:0/1075476207 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f963c072ea0 0x7f963c06fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.041+0000 7f96437bc640 1 -- 172.21.15.150:0/1075476207 >> 172.21.15.150:0/1075476207 conn(0x7f963c0abb00 msgr2=0x7f963c0c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.041+0000 7f96437bc640 1 -- 172.21.15.150:0/1075476207 shutdown_connections 2024-09-17T04:58:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:36.041+0000 7f96437bc640 1 -- 172.21.15.150:0/1075476207 wait complete. 2024-09-17T04:58:36.057 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-17T04:58:38.020 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:38.058 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:38.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.193+0000 7f4d4fb82640 1 -- 172.21.15.150:0/1586172724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d34003a40 msgr2=0x7f4d34003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:38.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.193+0000 7f4d4fb82640 1 --2- 172.21.15.150:0/1586172724 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d34003a40 0x7f4d34003eb0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4d44006fb0 tx=0x7f4d4401cf20 comp rx=0 tx=0).stop 2024-09-17T04:58:38.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.193+0000 7f4d4e11f640 1 -- 172.21.15.150:0/1586172724 reap_dead start 2024-09-17T04:58:38.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.193+0000 7f4d4fb82640 1 -- 172.21.15.150:0/1586172724 shutdown_connections 2024-09-17T04:58:38.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.193+0000 7f4d4fb82640 1 -- 172.21.15.150:0/1586172724 >> 172.21.15.150:0/1586172724 conn(0x7f4d480c4a00 msgr2=0x7f4d480c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:38.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.193+0000 7f4d4fb82640 1 -- 172.21.15.150:0/1586172724 shutdown_connections 2024-09-17T04:58:38.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.193+0000 7f4d4fb82640 1 -- 172.21.15.150:0/1586172724 wait complete. 2024-09-17T04:58:38.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4fb82640 1 Processor -- start 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4fb82640 1 -- start start 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4fb82640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d34003a40 0x7f4d480a88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4fb82640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d481413e0 0x7f4d481457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4fb82640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d480aab60 0x7f4d480aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4fb82640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4d480ad010 con 0x7f4d34003a40 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4fb82640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4d480adef0 con 0x7f4d481413e0 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4fb82640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4d480ad190 con 0x7f4d480aab60 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4d91e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d34003a40 0x7f4d480a88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4d91e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d34003a40 0x7f4d480a88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47780/0 (socket says 172.21.15.150:47780) 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4d91e640 1 -- 172.21.15.150:0/804330590 learned_addr learned my addr 172.21.15.150:0/804330590 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4e11f640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d480aab60 0x7f4d480aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4d11d640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d481413e0 0x7f4d481457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:38.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4d91e640 1 -- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d480aab60 msgr2=0x7f4d480aaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:38.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4d91e640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d480aab60 0x7f4d480aaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4d91e640 1 -- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d481413e0 msgr2=0x7f4d481457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:38.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4d91e640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d481413e0 0x7f4d481457a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4d91e640 1 -- 172.21.15.150:0/804330590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d480a9550 con 0x7f4d34003a40 2024-09-17T04:58:38.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4d91e640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d34003a40 0x7f4d480a88b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4d4401d400 tx=0x7f4d44006c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:38.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d3e7fc640 1 -- 172.21.15.150:0/804330590 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d440014c0 con 0x7f4d34003a40 2024-09-17T04:58:38.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d3e7fc640 1 -- 172.21.15.150:0/804330590 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d44001660 con 0x7f4d34003a40 2024-09-17T04:58:38.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4fb82640 1 -- 172.21.15.150:0/804330590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d48147680 con 0x7f4d34003a40 2024-09-17T04:58:38.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d3e7fc640 1 -- 172.21.15.150:0/804330590 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44006330 con 0x7f4d34003a40 2024-09-17T04:58:38.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.197+0000 7f4d4fb82640 1 -- 172.21.15.150:0/804330590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d480ac180 con 0x7f4d34003a40 2024-09-17T04:58:38.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.201+0000 7f4d4fb82640 1 -- 172.21.15.150:0/804330590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d480ad720 con 0x7f4d34003a40 2024-09-17T04:58:38.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.201+0000 7f4d3e7fc640 1 -- 172.21.15.150:0/804330590 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4d440a5c70 con 0x7f4d34003a40 2024-09-17T04:58:38.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.201+0000 7f4d3e7fc640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d2c073200 0x7f4d2c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:38.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.201+0000 7f4d3e7fc640 1 -- 172.21.15.150:0/804330590 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f4d4412e270 con 0x7f4d34003a40 2024-09-17T04:58:38.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.201+0000 7f4d4d11d640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d2c073200 0x7f4d2c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:38.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.201+0000 7f4d4d11d640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d2c073200 0x7f4d2c075710 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4d480acdd0 tx=0x7f4d38006b80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.205+0000 7f4d3e7fc640 1 -- 172.21.15.150:0/804330590 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4d44016cd0 con 0x7f4d34003a40 2024-09-17T04:58:38.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.221+0000 7fede87dc640 1 -- 172.21.15.150:0/2835811483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fede00a85b0 msgr2=0x7fede00c7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:38.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.221+0000 7fede87dc640 1 --2- 172.21.15.150:0/2835811483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fede00a85b0 0x7fede00c7270 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fede00c8d80 tx=0x7feddc01c830 comp rx=0 tx=0).stop 2024-09-17T04:58:38.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 -- 172.21.15.150:0/2835811483 shutdown_connections 2024-09-17T04:58:38.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 --2- 172.21.15.150:0/2835811483 >> v2:172.21.15.176:3300/0 conn(0x7fede00c77b0 0x7fede0151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 --2- 172.21.15.150:0/2835811483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fede00a85b0 0x7fede00c7270 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 --2- 172.21.15.150:0/2835811483 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fede00c6cb0 0x7fede00ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 -- 172.21.15.150:0/2835811483 >> 172.21.15.150:0/2835811483 conn(0x7fede00aab60 msgr2=0x7fede00aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:38.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 -- 172.21.15.150:0/2835811483 shutdown_connections 2024-09-17T04:58:38.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 -- 172.21.15.150:0/2835811483 wait complete. 2024-09-17T04:58:38.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 Processor -- start 2024-09-17T04:58:38.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 -- start start 2024-09-17T04:58:38.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fede00a85b0 0x7fede014d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:38.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fede00c6cb0 0x7fede0071b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:38.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fede00c77b0 0x7fede01ed980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:38.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fede01552f0 con 0x7fede00c6cb0 2024-09-17T04:58:38.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fede0155170 con 0x7fede00c77b0 2024-09-17T04:58:38.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede87dc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fede00c32e0 con 0x7fede00a85b0 2024-09-17T04:58:38.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede5d77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fede00c6cb0 0x7fede0071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:38.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede5d77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fede00c6cb0 0x7fede0071b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47788/0 (socket says 172.21.15.150:47788) 2024-09-17T04:58:38.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede5d77640 1 -- 172.21.15.150:0/540623460 learned_addr learned my addr 172.21.15.150:0/540623460 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:38.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede6578640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fede00a85b0 0x7fede014d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:38.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede6d79640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fede00c77b0 0x7fede01ed980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:38.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede5d77640 1 -- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fede00a85b0 msgr2=0x7fede014d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:38.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede5d77640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fede00a85b0 0x7fede014d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede5d77640 1 -- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fede00c77b0 msgr2=0x7fede01ed980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:38.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede5d77640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fede00c77b0 0x7fede01ed980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede5d77640 1 -- 172.21.15.150:0/540623460 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede01ed480 con 0x7fede00c6cb0 2024-09-17T04:58:38.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede6d79640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fede00c77b0 0x7fede01ed980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:58:38.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fede5d77640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fede00c6cb0 0x7fede0071b90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7feddc01aee0 tx=0x7feddc005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:38.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.225+0000 7fedd6ffd640 1 -- 172.21.15.150:0/540623460 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc0b2040 con 0x7fede00c6cb0 2024-09-17T04:58:38.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.229+0000 7fedd6ffd640 1 -- 172.21.15.150:0/540623460 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feddc004240 con 0x7fede00c6cb0 2024-09-17T04:58:38.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.229+0000 7fede87dc640 1 -- 172.21.15.150:0/540623460 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede0154410 con 0x7fede00c6cb0 2024-09-17T04:58:38.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.229+0000 7fedd6ffd640 1 -- 172.21.15.150:0/540623460 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc005030 con 0x7fede00c6cb0 2024-09-17T04:58:38.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.229+0000 7fede87dc640 1 -- 172.21.15.150:0/540623460 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede0147160 con 0x7fede00c6cb0 2024-09-17T04:58:38.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.229+0000 7fede87dc640 1 -- 172.21.15.150:0/540623460 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede0154180 con 0x7fede00c6cb0 2024-09-17T04:58:38.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.229+0000 7fedd6ffd640 1 -- 172.21.15.150:0/540623460 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7feddc0051d0 con 0x7fede00c6cb0 2024-09-17T04:58:38.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.229+0000 7fedd6ffd640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fedc4073200 0x7fedc4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:38.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.229+0000 7fedd6ffd640 1 -- 172.21.15.150:0/540623460 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7feddc12e530 con 0x7fede00c6cb0 2024-09-17T04:58:38.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.229+0000 7fede6578640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fedc4073200 0x7fedc4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:38.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.229+0000 7fede6578640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fedc4073200 0x7fedc4075710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fedd0005fd0 tx=0x7fedd0005d80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:38.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.241+0000 7fedd6ffd640 1 -- 172.21.15.150:0/540623460 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7feddc0164b0 con 0x7fede00c6cb0 2024-09-17T04:58:38.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.421+0000 7f4d4fb82640 1 -- 172.21.15.150:0/804330590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d480abc40 con 0x7f4d34003a40 2024-09-17T04:58:38.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.421+0000 7f4d3e7fc640 1 -- 172.21.15.150:0/804330590 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+6536 (secure 0 0 0) 0x7f4d440f8bc0 con 0x7f4d34003a40 2024-09-17T04:58:38.420 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:38.420 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":71,"btime":"2024-09-17T04:58:36:360546+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:38.421 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 71 2024-09-17T04:58:38.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.425+0000 7f4d4fb82640 1 -- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d2c073200 msgr2=0x7f4d2c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:38.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.425+0000 7f4d4fb82640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d2c073200 0x7f4d2c075710 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4d480acdd0 tx=0x7f4d38006b80 comp rx=0 tx=0).stop 2024-09-17T04:58:38.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.425+0000 7f4d4fb82640 1 -- 172.21.15.150:0/804330590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d34003a40 msgr2=0x7f4d480a88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:38.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.425+0000 7f4d4fb82640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d34003a40 0x7f4d480a88b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f4d4401d400 tx=0x7f4d44006c70 comp rx=0 tx=0).stop 2024-09-17T04:58:38.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.425+0000 7f4d4fb82640 1 -- 172.21.15.150:0/804330590 shutdown_connections 2024-09-17T04:58:38.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.425+0000 7f4d4fb82640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d2c073200 0x7f4d2c075710 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.425+0000 7f4d4fb82640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d481413e0 0x7f4d481457a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.425+0000 7f4d4fb82640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d480aab60 0x7f4d480aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.425+0000 7f4d4fb82640 1 --2- 172.21.15.150:0/804330590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d34003a40 0x7f4d480a88b0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.425+0000 7f4d4fb82640 1 -- 172.21.15.150:0/804330590 >> 172.21.15.150:0/804330590 conn(0x7f4d480c4a00 msgr2=0x7f4d4815d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:38.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.425+0000 7f4d4fb82640 1 -- 172.21.15.150:0/804330590 shutdown_connections 2024-09-17T04:58:38.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.425+0000 7f4d4fb82640 1 -- 172.21.15.150:0/804330590 wait complete. 2024-09-17T04:58:38.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 -- 172.21.15.150:0/540623460 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fede00ae340 con 0x7fede00c6cb0 2024-09-17T04:58:38.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fedd6ffd640 1 -- 172.21.15.150:0/540623460 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+6536 (secure 0 0 0) 0x7feddc0f8df0 con 0x7fede00c6cb0 2024-09-17T04:58:38.438 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:38.438 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":71,"btime":"2024-09-17T04:58:36:360546+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:38.439 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 71 2024-09-17T04:58:38.441 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:58:38.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 -- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fedc4073200 msgr2=0x7fedc4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:38.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fedc4073200 0x7fedc4075710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fedd0005fd0 tx=0x7fedd0005d80 comp rx=0 tx=0).stop 2024-09-17T04:58:38.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 -- 172.21.15.150:0/540623460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fede00c6cb0 msgr2=0x7fede0071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:38.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fede00c6cb0 0x7fede0071b90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7feddc01aee0 tx=0x7feddc005e70 comp rx=0 tx=0).stop 2024-09-17T04:58:38.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 -- 172.21.15.150:0/540623460 shutdown_connections 2024-09-17T04:58:38.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fedc4073200 0x7fedc4075710 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fede00c77b0 0x7fede01ed980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fede00c6cb0 0x7fede0071b90 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 --2- 172.21.15.150:0/540623460 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fede00a85b0 0x7fede014d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:38.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 -- 172.21.15.150:0/540623460 >> 172.21.15.150:0/540623460 conn(0x7fede00aab60 msgr2=0x7fede0068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:38.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 -- 172.21.15.150:0/540623460 shutdown_connections 2024-09-17T04:58:38.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:38.441+0000 7fede87dc640 1 -- 172.21.15.150:0/540623460 wait complete. 2024-09-17T04:58:38.458 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-17T04:58:38.458 INFO:tasks.mds_thrash.fs.[b]:waiting for 87 secs before thrashing 2024-09-17T04:58:40.442 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:40.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.605+0000 7f83d59fc640 1 -- 172.21.15.150:0/3895399637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83b8004000 msgr2=0x7f83b8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:40.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.605+0000 7f83d59fc640 1 --2- 172.21.15.150:0/3895399637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83b8004000 0x7f83b8004810 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f83c4009b60 tx=0x7f83c401d260 comp rx=0 tx=0).stop 2024-09-17T04:58:40.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.605+0000 7f83cf7fe640 1 -- 172.21.15.150:0/3895399637 reap_dead start 2024-09-17T04:58:40.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.605+0000 7f83d59fc640 1 -- 172.21.15.150:0/3895399637 shutdown_connections 2024-09-17T04:58:40.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.605+0000 7f83d59fc640 1 -- 172.21.15.150:0/3895399637 >> 172.21.15.150:0/3895399637 conn(0x7f83d006f760 msgr2=0x7f83d0074580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:40.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.605+0000 7f83d59fc640 1 -- 172.21.15.150:0/3895399637 shutdown_connections 2024-09-17T04:58:40.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.605+0000 7f83d59fc640 1 -- 172.21.15.150:0/3895399637 wait complete. 2024-09-17T04:58:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83d59fc640 1 Processor -- start 2024-09-17T04:58:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83d59fc640 1 -- start start 2024-09-17T04:58:40.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83d59fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83b8004000 0x7f83d0153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:40.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83d59fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83d00c9cb0 0x7f83d00cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:40.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83d59fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83d0149d50 0x7f83d01f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:40.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83d59fc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f83d00b0830 con 0x7f83d00c9cb0 2024-09-17T04:58:40.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83d59fc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f83d00c41f0 con 0x7f83b8004000 2024-09-17T04:58:40.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83d59fc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f83d00b09b0 con 0x7f83d0149d50 2024-09-17T04:58:40.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83ce7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83d00c9cb0 0x7f83d00cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:40.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83ce7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83d00c9cb0 0x7f83d00cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59290/0 (socket says 172.21.15.150:59290) 2024-09-17T04:58:40.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83ce7fc640 1 -- 172.21.15.150:0/4206316823 learned_addr learned my addr 172.21.15.150:0/4206316823 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:40.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83ceffd640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83b8004000 0x7f83d0153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:40.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83cf7fe640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83d0149d50 0x7f83d01f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:40.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83ce7fc640 1 -- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83d0149d50 msgr2=0x7f83d01f33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:40.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83ce7fc640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83d0149d50 0x7f83d01f33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:40.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83ce7fc640 1 -- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83b8004000 msgr2=0x7f83d0153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:40.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83ce7fc640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83b8004000 0x7f83d0153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:40.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83ce7fc640 1 -- 172.21.15.150:0/4206316823 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d01ed9c0 con 0x7f83d00c9cb0 2024-09-17T04:58:40.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83ce7fc640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83d00c9cb0 0x7f83d00cb1f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f83c401d090 tx=0x7f83c4005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:40.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83b7fff640 1 -- 172.21.15.150:0/4206316823 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c40ba020 con 0x7f83d00c9cb0 2024-09-17T04:58:40.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.609+0000 7f83b7fff640 1 -- 172.21.15.150:0/4206316823 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83c40ac7b0 con 0x7f83d00c9cb0 2024-09-17T04:58:40.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.613+0000 7f83d59fc640 1 -- 172.21.15.150:0/4206316823 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d00cb730 con 0x7f83d00c9cb0 2024-09-17T04:58:40.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.613+0000 7f83b7fff640 1 -- 172.21.15.150:0/4206316823 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c40b1040 con 0x7f83d00c9cb0 2024-09-17T04:58:40.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.613+0000 7f83d59fc640 1 -- 172.21.15.150:0/4206316823 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d0152fc0 con 0x7f83d00c9cb0 2024-09-17T04:58:40.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.613+0000 7f83d59fc640 1 -- 172.21.15.150:0/4206316823 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83d00c3670 con 0x7f83d00c9cb0 2024-09-17T04:58:40.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.613+0000 7f83b7fff640 1 -- 172.21.15.150:0/4206316823 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f83c40bd020 con 0x7f83d00c9cb0 2024-09-17T04:58:40.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.613+0000 7f83b7fff640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f83bc073200 0x7f83bc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:40.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.613+0000 7f83b7fff640 1 -- 172.21.15.150:0/4206316823 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f83c412f820 con 0x7f83d00c9cb0 2024-09-17T04:58:40.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.613+0000 7f83ceffd640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f83bc073200 0x7f83bc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:40.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.613+0000 7f83ceffd640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f83bc073200 0x7f83bc075710 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f83c0005950 tx=0x7f83c00091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:40.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.617+0000 7f83b7fff640 1 -- 172.21.15.150:0/4206316823 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f83c40c4030 con 0x7f83d00c9cb0 2024-09-17T04:58:40.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.829+0000 7f83d59fc640 1 -- 172.21.15.150:0/4206316823 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83d00c63f0 con 0x7f83d00c9cb0 2024-09-17T04:58:40.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.829+0000 7f83b7fff640 1 -- 172.21.15.150:0/4206316823 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+6536 (secure 0 0 0) 0x7f83c4016fa0 con 0x7f83d00c9cb0 2024-09-17T04:58:40.827 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:40.827 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":71,"btime":"2024-09-17T04:58:36:360546+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:40.827 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 71 2024-09-17T04:58:40.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.833+0000 7f83d59fc640 1 -- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f83bc073200 msgr2=0x7f83bc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:40.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.833+0000 7f83d59fc640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f83bc073200 0x7f83bc075710 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f83c0005950 tx=0x7f83c00091e0 comp rx=0 tx=0).stop 2024-09-17T04:58:40.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.833+0000 7f83d59fc640 1 -- 172.21.15.150:0/4206316823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83d00c9cb0 msgr2=0x7f83d00cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:40.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.833+0000 7f83d59fc640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83d00c9cb0 0x7f83d00cb1f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f83c401d090 tx=0x7f83c4005640 comp rx=0 tx=0).stop 2024-09-17T04:58:40.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.833+0000 7f83d59fc640 1 -- 172.21.15.150:0/4206316823 shutdown_connections 2024-09-17T04:58:40.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.833+0000 7f83d59fc640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f83bc073200 0x7f83bc075710 secure :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f83c0005950 tx=0x7f83c00091e0 comp rx=0 tx=0).stop 2024-09-17T04:58:40.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.833+0000 7f83d59fc640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83d0149d50 0x7f83d01f33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:40.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.833+0000 7f83d59fc640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83d00c9cb0 0x7f83d00cb1f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:40.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.833+0000 7f83d59fc640 1 --2- 172.21.15.150:0/4206316823 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83b8004000 0x7f83d0153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:40.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.833+0000 7f83d59fc640 1 -- 172.21.15.150:0/4206316823 >> 172.21.15.150:0/4206316823 conn(0x7f83d006f760 msgr2=0x7f83d0068f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:40.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.833+0000 7f83d59fc640 1 -- 172.21.15.150:0/4206316823 shutdown_connections 2024-09-17T04:58:40.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:40.833+0000 7f83d59fc640 1 -- 172.21.15.150:0/4206316823 wait complete. 2024-09-17T04:58:40.851 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:58:42.852 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:43.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.021+0000 7fe87dfab640 1 -- 172.21.15.150:0/1467468479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8781568a0 msgr2=0x7fe8780c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:43.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.021+0000 7fe87dfab640 1 --2- 172.21.15.150:0/1467468479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8781568a0 0x7fe8780c5810 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fe87808d880 tx=0x7fe86801c890 comp rx=0 tx=0).stop 2024-09-17T04:58:43.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.021+0000 7fe87dfab640 1 -- 172.21.15.150:0/1467468479 shutdown_connections 2024-09-17T04:58:43.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.021+0000 7fe87dfab640 1 --2- 172.21.15.150:0/1467468479 >> v2:172.21.15.176:3300/0 conn(0x7fe8780c8fc0 0x7fe8780c20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:43.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.021+0000 7fe87dfab640 1 --2- 172.21.15.150:0/1467468479 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8780c35b0 0x7fe8780c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:43.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.021+0000 7fe87dfab640 1 --2- 172.21.15.150:0/1467468479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8781568a0 0x7fe8780c5810 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:43.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.021+0000 7fe87dfab640 1 -- 172.21.15.150:0/1467468479 >> 172.21.15.150:0/1467468479 conn(0x7fe878074b80 msgr2=0x7fe87806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:43.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.021+0000 7fe87dfab640 1 -- 172.21.15.150:0/1467468479 shutdown_connections 2024-09-17T04:58:43.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.021+0000 7fe87dfab640 1 -- 172.21.15.150:0/1467468479 wait complete. 2024-09-17T04:58:43.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe87dfab640 1 Processor -- start 2024-09-17T04:58:43.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe87dfab640 1 -- start start 2024-09-17T04:58:43.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe87dfab640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8780c35b0 0x7fe8780c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:43.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe87dfab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8780c8fc0 0x7fe8781537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:43.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe87dfab640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8781568a0 0x7fe878143d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:43.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe87dfab640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe8780c69f0 con 0x7fe8780c8fc0 2024-09-17T04:58:43.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe87dfab640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe878154530 con 0x7fe8780c35b0 2024-09-17T04:58:43.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe87dfab640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe8780c6b70 con 0x7fe8781568a0 2024-09-17T04:58:43.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe876ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8780c8fc0 0x7fe8781537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:43.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe876ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8780c8fc0 0x7fe8781537f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59300/0 (socket says 172.21.15.150:59300) 2024-09-17T04:58:43.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe876ffd640 1 -- 172.21.15.150:0/2330884026 learned_addr learned my addr 172.21.15.150:0/2330884026 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:43.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe8777fe640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8780c35b0 0x7fe8780c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:43.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe877fff640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8781568a0 0x7fe878143d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:43.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe876ffd640 1 -- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8781568a0 msgr2=0x7fe878143d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:43.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe876ffd640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8781568a0 0x7fe878143d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:43.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe876ffd640 1 -- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8780c35b0 msgr2=0x7fe8780c7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:43.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe876ffd640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8780c35b0 0x7fe8780c7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:43.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe876ffd640 1 -- 172.21.15.150:0/2330884026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe878151da0 con 0x7fe8780c8fc0 2024-09-17T04:58:43.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe8777fe640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8780c35b0 0x7fe8780c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:58:43.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe876ffd640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8780c8fc0 0x7fe8781537f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00d630 tx=0x7fe86c00daf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:43.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe85ffff640 1 -- 172.21.15.150:0/2330884026 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c017070 con 0x7fe8780c8fc0 2024-09-17T04:58:43.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe85ffff640 1 -- 172.21.15.150:0/2330884026 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe86c005310 con 0x7fe8780c8fc0 2024-09-17T04:58:43.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe87dfab640 1 -- 172.21.15.150:0/2330884026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe878144be0 con 0x7fe8780c8fc0 2024-09-17T04:58:43.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe85ffff640 1 -- 172.21.15.150:0/2330884026 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0047b0 con 0x7fe8780c8fc0 2024-09-17T04:58:43.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe87dfab640 1 -- 172.21.15.150:0/2330884026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8780cb1b0 con 0x7fe8780c8fc0 2024-09-17T04:58:43.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.025+0000 7fe87dfab640 1 -- 172.21.15.150:0/2330884026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8780c20a0 con 0x7fe8780c8fc0 2024-09-17T04:58:43.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.029+0000 7fe85ffff640 1 -- 172.21.15.150:0/2330884026 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe86c004950 con 0x7fe8780c8fc0 2024-09-17T04:58:43.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.029+0000 7fe85ffff640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe864073230 0x7fe864075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:43.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.029+0000 7fe85ffff640 1 -- 172.21.15.150:0/2330884026 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7fe86c094a00 con 0x7fe8780c8fc0 2024-09-17T04:58:43.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.029+0000 7fe8777fe640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe864073230 0x7fe864075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:43.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.029+0000 7fe8777fe640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe864073230 0x7fe864075740 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe8680af000 tx=0x7fe86801b0d0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:43.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.033+0000 7fe85ffff640 1 -- 172.21.15.150:0/2330884026 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe86c05f340 con 0x7fe8780c8fc0 2024-09-17T04:58:43.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.241+0000 7fe87dfab640 1 -- 172.21.15.150:0/2330884026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe878155530 con 0x7fe8780c8fc0 2024-09-17T04:58:43.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe85ffff640 1 -- 172.21.15.150:0/2330884026 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+6536 (secure 0 0 0) 0x7fe86c05ec40 con 0x7fe8780c8fc0 2024-09-17T04:58:43.242 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:43.242 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":71,"btime":"2024-09-17T04:58:36:360546+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:43.242 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 71 2024-09-17T04:58:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe87dfab640 1 -- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe864073230 msgr2=0x7fe864075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe87dfab640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe864073230 0x7fe864075740 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe8680af000 tx=0x7fe86801b0d0 comp rx=0 tx=0).stop 2024-09-17T04:58:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe87dfab640 1 -- 172.21.15.150:0/2330884026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8780c8fc0 msgr2=0x7fe8781537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe87dfab640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8780c8fc0 0x7fe8781537f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00d630 tx=0x7fe86c00daf0 comp rx=0 tx=0).stop 2024-09-17T04:58:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe87dfab640 1 -- 172.21.15.150:0/2330884026 shutdown_connections 2024-09-17T04:58:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe87dfab640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe864073230 0x7fe864075740 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe87dfab640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8781568a0 0x7fe878143d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe87dfab640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8780c8fc0 0x7fe8781537f0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:43.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe87dfab640 1 --2- 172.21.15.150:0/2330884026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8780c35b0 0x7fe8780c7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:43.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe87dfab640 1 -- 172.21.15.150:0/2330884026 >> 172.21.15.150:0/2330884026 conn(0x7fe878074b80 msgr2=0x7fe878161580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:43.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe87dfab640 1 -- 172.21.15.150:0/2330884026 shutdown_connections 2024-09-17T04:58:43.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:43.245+0000 7fe87dfab640 1 -- 172.21.15.150:0/2330884026 wait complete. 2024-09-17T04:58:43.264 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:58:45.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:45.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.433+0000 7f615c501640 1 -- 172.21.15.150:0/2974702688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c004d50 msgr2=0x7f613c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:45.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.433+0000 7f615c501640 1 --2- 172.21.15.150:0/2974702688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f613c004d50 0x7f613c005160 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f614c00c290 tx=0x7f614c01cfa0 comp rx=0 tx=0).stop 2024-09-17T04:58:45.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.433+0000 7f615aa9e640 1 -- 172.21.15.150:0/2974702688 reap_dead start 2024-09-17T04:58:45.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.433+0000 7f615c501640 1 -- 172.21.15.150:0/2974702688 shutdown_connections 2024-09-17T04:58:45.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.433+0000 7f615c501640 1 -- 172.21.15.150:0/2974702688 >> 172.21.15.150:0/2974702688 conn(0x7f61540abb00 msgr2=0x7f61540abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:45.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.433+0000 7f615c501640 1 -- 172.21.15.150:0/2974702688 shutdown_connections 2024-09-17T04:58:45.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.433+0000 7f615c501640 1 -- 172.21.15.150:0/2974702688 wait complete. 2024-09-17T04:58:45.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f615c501640 1 Processor -- start 2024-09-17T04:58:45.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f615c501640 1 -- start start 2024-09-17T04:58:45.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f615c501640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61540ae470 0x7f61541eff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:45.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f615c501640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6154145ab0 0x7f6154156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:45.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f615c501640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61540c3180 0x7f61540c3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f615c501640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f61540c4be0 con 0x7f6154145ab0 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f615c501640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f61541521b0 con 0x7f61540ae470 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f615c501640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f61540c4d60 con 0x7f61540c3180 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f6159a9c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6154145ab0 0x7f6154156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f6159a9c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6154145ab0 0x7f6154156ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59318/0 (socket says 172.21.15.150:59318) 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f6159a9c640 1 -- 172.21.15.150:0/2100776129 learned_addr learned my addr 172.21.15.150:0/2100776129 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f615a29d640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61540ae470 0x7f61541eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f615aa9e640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61540c3180 0x7f61540c3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f6159a9c640 1 -- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61540c3180 msgr2=0x7f61540c3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f6159a9c640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61540c3180 0x7f61540c3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f6159a9c640 1 -- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61540ae470 msgr2=0x7f61541eff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f6159a9c640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61540ae470 0x7f61541eff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f6159a9c640 1 -- 172.21.15.150:0/2100776129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61541f0480 con 0x7f6154145ab0 2024-09-17T04:58:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f6159a9c640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6154145ab0 0x7f6154156ee0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6148010a40 tx=0x7f6148010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:45.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f6152ffd640 1 -- 172.21.15.150:0/2100776129 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614800ece0 con 0x7f6154145ab0 2024-09-17T04:58:45.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f6152ffd640 1 -- 172.21.15.150:0/2100776129 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6148012450 con 0x7f6154145ab0 2024-09-17T04:58:45.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f615c501640 1 -- 172.21.15.150:0/2100776129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61540acbb0 con 0x7f6154145ab0 2024-09-17T04:58:45.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f6152ffd640 1 -- 172.21.15.150:0/2100776129 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6148012880 con 0x7f6154145ab0 2024-09-17T04:58:45.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.437+0000 7f615c501640 1 -- 172.21.15.150:0/2100776129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61540adf50 con 0x7f6154145ab0 2024-09-17T04:58:45.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.441+0000 7f615c501640 1 -- 172.21.15.150:0/2100776129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61541587e0 con 0x7f6154145ab0 2024-09-17T04:58:45.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.441+0000 7f6152ffd640 1 -- 172.21.15.150:0/2100776129 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6148012a20 con 0x7f6154145ab0 2024-09-17T04:58:45.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.441+0000 7f6152ffd640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6140073200 0x7f6140075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:45.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.441+0000 7f6152ffd640 1 -- 172.21.15.150:0/2100776129 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f6148095720 con 0x7f6154145ab0 2024-09-17T04:58:45.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.441+0000 7f615a29d640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6140073200 0x7f6140075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:45.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.441+0000 7f615a29d640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6140073200 0x7f6140075710 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f61541f1040 tx=0x7f6144009500 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:45.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.445+0000 7f6152ffd640 1 -- 172.21.15.150:0/2100776129 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f61480600f0 con 0x7f6154145ab0 2024-09-17T04:58:45.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 -- 172.21.15.150:0/2100776129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61540acd80 con 0x7f6154145ab0 2024-09-17T04:58:45.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f6152ffd640 1 -- 172.21.15.150:0/2100776129 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+6536 (secure 0 0 0) 0x7f614805f9f0 con 0x7f6154145ab0 2024-09-17T04:58:45.655 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:45.655 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":71,"btime":"2024-09-17T04:58:36:360546+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:45.655 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 71 2024-09-17T04:58:45.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 -- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6140073200 msgr2=0x7f6140075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:45.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6140073200 0x7f6140075710 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f61541f1040 tx=0x7f6144009500 comp rx=0 tx=0).stop 2024-09-17T04:58:45.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 -- 172.21.15.150:0/2100776129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6154145ab0 msgr2=0x7f6154156ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:45.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6154145ab0 0x7f6154156ee0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6148010a40 tx=0x7f6148010f00 comp rx=0 tx=0).stop 2024-09-17T04:58:45.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 -- 172.21.15.150:0/2100776129 shutdown_connections 2024-09-17T04:58:45.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6140073200 0x7f6140075710 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:45.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f61540c3180 0x7f61540c3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:45.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6154145ab0 0x7f6154156ee0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:45.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 --2- 172.21.15.150:0/2100776129 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61540ae470 0x7f61541eff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:45.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 -- 172.21.15.150:0/2100776129 >> 172.21.15.150:0/2100776129 conn(0x7f61540abb00 msgr2=0x7f61540a9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:45.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 -- 172.21.15.150:0/2100776129 shutdown_connections 2024-09-17T04:58:45.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:45.657+0000 7f615c501640 1 -- 172.21.15.150:0/2100776129 wait complete. 2024-09-17T04:58:45.678 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:58:47.678 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.845+0000 7fe3efdbf640 1 -- 172.21.15.150:0/2622911086 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3e814cda0 msgr2=0x7fe3e814d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.845+0000 7fe3efdbf640 1 --2- 172.21.15.150:0/2622911086 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3e814cda0 0x7fe3e814d1b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009c80 tx=0x7fe3e001ccb0 comp rx=0 tx=0).stop 2024-09-17T04:58:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.845+0000 7fe3efdbf640 1 -- 172.21.15.150:0/2622911086 shutdown_connections 2024-09-17T04:58:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.845+0000 7fe3efdbf640 1 --2- 172.21.15.150:0/2622911086 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3e814cda0 0x7fe3e814d1b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.845+0000 7fe3efdbf640 1 --2- 172.21.15.150:0/2622911086 >> v2:172.21.15.176:3300/0 conn(0x7fe3e8156eb0 0x7fe3e81572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.845+0000 7fe3efdbf640 1 -- 172.21.15.150:0/2622911086 >> 172.21.15.150:0/2622911086 conn(0x7fe3e80c4a00 msgr2=0x7fe3e80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.845+0000 7fe3efdbf640 1 -- 172.21.15.150:0/2622911086 shutdown_connections 2024-09-17T04:58:47.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.845+0000 7fe3efdbf640 1 -- 172.21.15.150:0/2622911086 wait complete. 2024-09-17T04:58:47.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3efdbf640 1 Processor -- start 2024-09-17T04:58:47.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3efdbf640 1 -- start start 2024-09-17T04:58:47.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3efdbf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3e80c7570 0x7fe3e81f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:47.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3efdbf640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3e814cda0 0x7fe3e81f31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:47.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3efdbf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3e8156eb0 0x7fe3e8143580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:47.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3efdbf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e80c65c0 con 0x7fe3e8156eb0 2024-09-17T04:58:47.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3efdbf640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e80c6440 con 0x7fe3e814cda0 2024-09-17T04:58:47.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3efdbf640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe3e80c6740 con 0x7fe3e80c7570 2024-09-17T04:58:47.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3ee35c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3e8156eb0 0x7fe3e8143580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:47.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3ee35c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3e8156eb0 0x7fe3e8143580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59326/0 (socket says 172.21.15.150:59326) 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3ee35c640 1 -- 172.21.15.150:0/3299827542 learned_addr learned my addr 172.21.15.150:0/3299827542 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3ed35a640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3e814cda0 0x7fe3e81f31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3edb5b640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3e80c7570 0x7fe3e81f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3ee35c640 1 -- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3e80c7570 msgr2=0x7fe3e81f3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3ee35c640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3e80c7570 0x7fe3e81f3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3ee35c640 1 -- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3e814cda0 msgr2=0x7fe3e81f31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3ee35c640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3e814cda0 0x7fe3e81f31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3ee35c640 1 -- 172.21.15.150:0/3299827542 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e8146c10 con 0x7fe3e8156eb0 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3ee35c640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3e8156eb0 0x7fe3e8143580 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009bd0 tx=0x7fe3e0005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3de7fc640 1 -- 172.21.15.150:0/3299827542 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e00bb070 con 0x7fe3e8156eb0 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3de7fc640 1 -- 172.21.15.150:0/3299827542 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3e0004b40 con 0x7fe3e8156eb0 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3efdbf640 1 -- 172.21.15.150:0/3299827542 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e806f380 con 0x7fe3e8156eb0 2024-09-17T04:58:47.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3de7fc640 1 -- 172.21.15.150:0/3299827542 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e00b2040 con 0x7fe3e8156eb0 2024-09-17T04:58:47.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.849+0000 7fe3efdbf640 1 -- 172.21.15.150:0/3299827542 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e8161f40 con 0x7fe3e8156eb0 2024-09-17T04:58:47.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.853+0000 7fe3efdbf640 1 -- 172.21.15.150:0/3299827542 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e8071570 con 0x7fe3e8156eb0 2024-09-17T04:58:47.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.853+0000 7fe3de7fc640 1 -- 172.21.15.150:0/3299827542 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe3e00be070 con 0x7fe3e8156eb0 2024-09-17T04:58:47.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.853+0000 7fe3de7fc640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3cc073200 0x7fe3cc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:47.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.853+0000 7fe3de7fc640 1 -- 172.21.15.150:0/3299827542 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7fe3e012e590 con 0x7fe3e8156eb0 2024-09-17T04:58:47.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.853+0000 7fe3edb5b640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3cc073200 0x7fe3cc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:47.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.853+0000 7fe3edb5b640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3cc073200 0x7fe3cc075710 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4001650 tx=0x7fe3e4009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:47.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:47.857+0000 7fe3de7fc640 1 -- 172.21.15.150:0/3299827542 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe3e00a5e20 con 0x7fe3e8156eb0 2024-09-17T04:58:48.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.061+0000 7fe3efdbf640 1 -- 172.21.15.150:0/3299827542 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3e814d1b0 con 0x7fe3e8156eb0 2024-09-17T04:58:48.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.061+0000 7fe3de7fc640 1 -- 172.21.15.150:0/3299827542 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+6536 (secure 0 0 0) 0x7fe3e0016990 con 0x7fe3e8156eb0 2024-09-17T04:58:48.061 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:48.061 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":71,"btime":"2024-09-17T04:58:36:360546+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:48.061 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 71 2024-09-17T04:58:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.065+0000 7fe3efdbf640 1 -- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3cc073200 msgr2=0x7fe3cc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.065+0000 7fe3efdbf640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3cc073200 0x7fe3cc075710 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4001650 tx=0x7fe3e4009040 comp rx=0 tx=0).stop 2024-09-17T04:58:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.065+0000 7fe3efdbf640 1 -- 172.21.15.150:0/3299827542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3e8156eb0 msgr2=0x7fe3e8143580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.065+0000 7fe3efdbf640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3e8156eb0 0x7fe3e8143580 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009bd0 tx=0x7fe3e0005b00 comp rx=0 tx=0).stop 2024-09-17T04:58:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.065+0000 7fe3efdbf640 1 -- 172.21.15.150:0/3299827542 shutdown_connections 2024-09-17T04:58:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.065+0000 7fe3efdbf640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3cc073200 0x7fe3cc075710 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.065+0000 7fe3efdbf640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3e814cda0 0x7fe3e81f31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.065+0000 7fe3efdbf640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3e8156eb0 0x7fe3e8143580 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.065+0000 7fe3efdbf640 1 --2- 172.21.15.150:0/3299827542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3e80c7570 0x7fe3e81f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.065+0000 7fe3efdbf640 1 -- 172.21.15.150:0/3299827542 >> 172.21.15.150:0/3299827542 conn(0x7fe3e80c4a00 msgr2=0x7fe3e8155fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.065+0000 7fe3efdbf640 1 -- 172.21.15.150:0/3299827542 shutdown_connections 2024-09-17T04:58:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:48.065+0000 7fe3efdbf640 1 -- 172.21.15.150:0/3299827542 wait complete. 2024-09-17T04:58:48.082 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:58:50.083 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:50.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.249+0000 7f671851f640 1 -- 172.21.15.150:0/1727569537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6710157ca0 msgr2=0x7f67101580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:50.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.249+0000 7f671851f640 1 --2- 172.21.15.150:0/1727569537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6710157ca0 0x7f67101580b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f6708009bc0 tx=0x7f670801ca80 comp rx=0 tx=0).stop 2024-09-17T04:58:50.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.249+0000 7f671851f640 1 -- 172.21.15.150:0/1727569537 shutdown_connections 2024-09-17T04:58:50.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.249+0000 7f671851f640 1 --2- 172.21.15.150:0/1727569537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6710157ca0 0x7f67101580b0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:50.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.249+0000 7f671851f640 1 --2- 172.21.15.150:0/1727569537 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67100ad7f0 0x7f67100c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:50.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.249+0000 7f671851f640 1 -- 172.21.15.150:0/1727569537 >> 172.21.15.150:0/1727569537 conn(0x7f67100c9de0 msgr2=0x7f67100ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:50.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.249+0000 7f671851f640 1 -- 172.21.15.150:0/1727569537 shutdown_connections 2024-09-17T04:58:50.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.249+0000 7f671851f640 1 -- 172.21.15.150:0/1727569537 wait complete. 2024-09-17T04:58:50.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f671851f640 1 Processor -- start 2024-09-17T04:58:50.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f671851f640 1 -- start start 2024-09-17T04:58:50.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f671851f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67100ad7f0 0x7f6710147640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:50.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f671851f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6710151e10 0x7f67101466f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:50.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f671851f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6710157ca0 0x7f671014a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:50.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f671851f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6710158770 con 0x7f6710151e10 2024-09-17T04:58:50.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f671851f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f67101585f0 con 0x7f6710157ca0 2024-09-17T04:58:50.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f671851f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f67100c7ce0 con 0x7f67100ad7f0 2024-09-17T04:58:50.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6715aba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6710151e10 0x7f67101466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6715aba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6710151e10 0x7f67101466f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57740/0 (socket says 172.21.15.150:57740) 2024-09-17T04:58:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6715aba640 1 -- 172.21.15.150:0/2379409168 learned_addr learned my addr 172.21.15.150:0/2379409168 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f67162bb640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67100ad7f0 0x7f6710147640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6716abc640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6710157ca0 0x7f671014a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6715aba640 1 -- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67100ad7f0 msgr2=0x7f6710147640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6715aba640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67100ad7f0 0x7f6710147640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6715aba640 1 -- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6710157ca0 msgr2=0x7f671014a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6715aba640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6710157ca0 0x7f671014a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6715aba640 1 -- 172.21.15.150:0/2379409168 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6710148370 con 0x7f6710151e10 2024-09-17T04:58:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f67162bb640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67100ad7f0 0x7f6710147640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:58:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6715aba640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6710151e10 0x7f67101466f0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f670000c3f0 tx=0x7f670000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:50.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6706ffd640 1 -- 172.21.15.150:0/2379409168 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6700011790 con 0x7f6710151e10 2024-09-17T04:58:50.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6706ffd640 1 -- 172.21.15.150:0/2379409168 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6700017070 con 0x7f6710151e10 2024-09-17T04:58:50.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f671851f640 1 -- 172.21.15.150:0/2379409168 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67100c47e0 con 0x7f6710151e10 2024-09-17T04:58:50.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f6706ffd640 1 -- 172.21.15.150:0/2379409168 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6700011ce0 con 0x7f6710151e10 2024-09-17T04:58:50.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f671851f640 1 -- 172.21.15.150:0/2379409168 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67100c4da0 con 0x7f6710151e10 2024-09-17T04:58:50.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.253+0000 7f671851f640 1 -- 172.21.15.150:0/2379409168 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67100c6230 con 0x7f6710151e10 2024-09-17T04:58:50.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.257+0000 7f6706ffd640 1 -- 172.21.15.150:0/2379409168 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f670001c460 con 0x7f6710151e10 2024-09-17T04:58:50.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.257+0000 7f6706ffd640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f66f4073230 0x7f66f4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:50.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.257+0000 7f6706ffd640 1 -- 172.21.15.150:0/2379409168 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f6700094600 con 0x7f6710151e10 2024-09-17T04:58:50.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.257+0000 7f67162bb640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f66f4073230 0x7f66f4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:50.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.257+0000 7f67162bb640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f66f4073230 0x7f66f4075740 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f670c005fd0 tx=0x7f670c005f40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:50.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.261+0000 7f6706ffd640 1 -- 172.21.15.150:0/2379409168 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f670005efd0 con 0x7f6710151e10 2024-09-17T04:58:50.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.469+0000 7f671851f640 1 -- 172.21.15.150:0/2379409168 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67100c4530 con 0x7f6710151e10 2024-09-17T04:58:50.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.469+0000 7f6706ffd640 1 -- 172.21.15.150:0/2379409168 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+6536 (secure 0 0 0) 0x7f670005e8d0 con 0x7f6710151e10 2024-09-17T04:58:50.467 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:50.467 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":71,"btime":"2024-09-17T04:58:36:360546+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5258,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:03.946007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5841},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5841":{"gid":5841,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/1275062894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1275062894},{"type":"v1","addr":"172.21.15.150:6833","nonce":1275062894}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5841,"qdb_cluster":[5841]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 71 2024-09-17T04:58:50.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.469+0000 7f671851f640 1 -- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f66f4073230 msgr2=0x7f66f4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.469+0000 7f671851f640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f66f4073230 0x7f66f4075740 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f670c005fd0 tx=0x7f670c005f40 comp rx=0 tx=0).stop 2024-09-17T04:58:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.473+0000 7f671851f640 1 -- 172.21.15.150:0/2379409168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6710151e10 msgr2=0x7f67101466f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.473+0000 7f671851f640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6710151e10 0x7f67101466f0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f670000c3f0 tx=0x7f670000c8b0 comp rx=0 tx=0).stop 2024-09-17T04:58:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.473+0000 7f671851f640 1 -- 172.21.15.150:0/2379409168 shutdown_connections 2024-09-17T04:58:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.473+0000 7f671851f640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f66f4073230 0x7f66f4075740 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.473+0000 7f671851f640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6710157ca0 0x7f671014a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.473+0000 7f671851f640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6710151e10 0x7f67101466f0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.473+0000 7f671851f640 1 --2- 172.21.15.150:0/2379409168 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f67100ad7f0 0x7f6710147640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.473+0000 7f671851f640 1 -- 172.21.15.150:0/2379409168 >> 172.21.15.150:0/2379409168 conn(0x7f67100c9de0 msgr2=0x7f671015e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.473+0000 7f671851f640 1 -- 172.21.15.150:0/2379409168 shutdown_connections 2024-09-17T04:58:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:50.473+0000 7f671851f640 1 -- 172.21.15.150:0/2379409168 wait complete. 2024-09-17T04:58:50.490 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T04:58:52.491 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:52.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.661+0000 7f0248a52640 1 -- 172.21.15.150:0/2566266618 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02340ad440 con 0x7f0244157ca0 2024-09-17T04:58:52.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.661+0000 7f024bcb8640 1 -- 172.21.15.150:0/2566266618 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0244157ca0 msgr2=0x7f02441580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:52.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.661+0000 7f024bcb8640 1 --2- 172.21.15.150:0/2566266618 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0244157ca0 0x7f02441580b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0234009bc0 tx=0x7f023401ca80 comp rx=0 tx=0).stop 2024-09-17T04:58:52.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.661+0000 7f024bcb8640 1 -- 172.21.15.150:0/2566266618 shutdown_connections 2024-09-17T04:58:52.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.661+0000 7f024bcb8640 1 --2- 172.21.15.150:0/2566266618 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0244157ca0 0x7f02441580b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:52.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.661+0000 7f024bcb8640 1 --2- 172.21.15.150:0/2566266618 >> v2:172.21.15.176:3300/0 conn(0x7f02440ad7f0 0x7f02440c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:52.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.661+0000 7f024bcb8640 1 -- 172.21.15.150:0/2566266618 >> 172.21.15.150:0/2566266618 conn(0x7f02440c9de0 msgr2=0x7f02440ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:52.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.661+0000 7f024bcb8640 1 -- 172.21.15.150:0/2566266618 shutdown_connections 2024-09-17T04:58:52.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.661+0000 7f024bcb8640 1 -- 172.21.15.150:0/2566266618 wait complete. 2024-09-17T04:58:52.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f024bcb8640 1 Processor -- start 2024-09-17T04:58:52.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f024bcb8640 1 -- start start 2024-09-17T04:58:52.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f024bcb8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02440ad7f0 0x7f0244150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:52.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f024bcb8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0244151e10 0x7f0244071600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:52.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f024bcb8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0244157ca0 0x7f0244071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:52.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f024bcb8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0244158770 con 0x7f0244151e10 2024-09-17T04:58:52.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f024bcb8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f02441585f0 con 0x7f0244157ca0 2024-09-17T04:58:52.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f024bcb8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f02440c7ce0 con 0x7f02440ad7f0 2024-09-17T04:58:52.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f0249253640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0244151e10 0x7f0244071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:52.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f0249253640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0244151e10 0x7f0244071600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57742/0 (socket says 172.21.15.150:57742) 2024-09-17T04:58:52.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f0249253640 1 -- 172.21.15.150:0/855060188 learned_addr learned my addr 172.21.15.150:0/855060188 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:52.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f024a255640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0244157ca0 0x7f0244071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:52.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f0249a54640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02440ad7f0 0x7f0244150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:52.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f0249253640 1 -- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02440ad7f0 msgr2=0x7f0244150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:52.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f0249253640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02440ad7f0 0x7f0244150670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:52.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f0249253640 1 -- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0244157ca0 msgr2=0x7f0244071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:52.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f0249253640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0244157ca0 0x7f0244071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:52.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f0249253640 1 -- 172.21.15.150:0/855060188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02441510d0 con 0x7f0244151e10 2024-09-17T04:58:52.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f0249253640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0244151e10 0x7f0244071600 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f023800c430 tx=0x7f023800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:52.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f02427fc640 1 -- 172.21.15.150:0/855060188 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238011790 con 0x7f0244151e10 2024-09-17T04:58:52.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f02427fc640 1 -- 172.21.15.150:0/855060188 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0238017070 con 0x7f0244151e10 2024-09-17T04:58:52.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f024bcb8640 1 -- 172.21.15.150:0/855060188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0244150bb0 con 0x7f0244151e10 2024-09-17T04:58:52.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f02427fc640 1 -- 172.21.15.150:0/855060188 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238011ce0 con 0x7f0244151e10 2024-09-17T04:58:52.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.665+0000 7f024bcb8640 1 -- 172.21.15.150:0/855060188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0244151620 con 0x7f0244151e10 2024-09-17T04:58:52.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.669+0000 7f024bcb8640 1 -- 172.21.15.150:0/855060188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02440c6230 con 0x7f0244151e10 2024-09-17T04:58:52.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.669+0000 7f02427fc640 1 -- 172.21.15.150:0/855060188 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f023801c460 con 0x7f0244151e10 2024-09-17T04:58:52.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.669+0000 7f02427fc640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0230073230 0x7f0230075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:52.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.669+0000 7f02427fc640 1 -- 172.21.15.150:0/855060188 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f0238094600 con 0x7f0244151e10 2024-09-17T04:58:52.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.669+0000 7f0249a54640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0230073230 0x7f0230075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:52.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.669+0000 7f0249a54640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0230073230 0x7f0230075740 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f023c0095b0 tx=0x7f023c005f40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:52.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.673+0000 7f02427fc640 1 -- 172.21.15.150:0/855060188 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f023805e920 con 0x7f0244151e10 2024-09-17T04:58:52.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.880+0000 7f024bcb8640 1 -- 172.21.15.150:0/855060188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f024415e6b0 con 0x7f0244151e10 2024-09-17T04:58:52.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.880+0000 7f02427fc640 1 -- 172.21.15.150:0/855060188 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5704 (secure 0 0 0) 0x7f0238063350 con 0x7f0244151e10 2024-09-17T04:58:52.880 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:52.880 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":73,"btime":"2024-09-17T04:58:51:939608+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:51.365154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:reconnect","state_seq":5,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:52.881 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 73 2024-09-17T04:58:52.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.884+0000 7f024bcb8640 1 -- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0230073230 msgr2=0x7f0230075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.884+0000 7f024bcb8640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0230073230 0x7f0230075740 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f023c0095b0 tx=0x7f023c005f40 comp rx=0 tx=0).stop 2024-09-17T04:58:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.884+0000 7f024bcb8640 1 -- 172.21.15.150:0/855060188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0244151e10 msgr2=0x7f0244071600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.884+0000 7f024bcb8640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0244151e10 0x7f0244071600 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f023800c430 tx=0x7f023800c8f0 comp rx=0 tx=0).stop 2024-09-17T04:58:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.884+0000 7f024bcb8640 1 -- 172.21.15.150:0/855060188 shutdown_connections 2024-09-17T04:58:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.884+0000 7f024bcb8640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0230073230 0x7f0230075740 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.884+0000 7f024bcb8640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0244157ca0 0x7f0244071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.884+0000 7f024bcb8640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0244151e10 0x7f0244071600 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.884+0000 7f024bcb8640 1 --2- 172.21.15.150:0/855060188 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f02440ad7f0 0x7f0244150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.884+0000 7f024bcb8640 1 -- 172.21.15.150:0/855060188 >> 172.21.15.150:0/855060188 conn(0x7f02440c9de0 msgr2=0x7f02440c5330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.884+0000 7f024bcb8640 1 -- 172.21.15.150:0/855060188 shutdown_connections 2024-09-17T04:58:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:52.884+0000 7f024bcb8640 1 -- 172.21.15.150:0/855060188 wait complete. 2024-09-17T04:58:52.905 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-17T04:58:52.905 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T04:58:52.905 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.056+0000 7f2eabd25640 1 -- 172.21.15.150:0/3769730878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e90004d50 msgr2=0x7f2e90005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.056+0000 7f2eabd25640 1 --2- 172.21.15.150:0/3769730878 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e90004d50 0x7f2e90005160 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f2ea406fcc0 tx=0x7f2e9c01d590 comp rx=0 tx=0).stop 2024-09-17T04:58:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.056+0000 7f2eaa2c2640 1 -- 172.21.15.150:0/3769730878 reap_dead start 2024-09-17T04:58:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.056+0000 7f2eabd25640 1 -- 172.21.15.150:0/3769730878 shutdown_connections 2024-09-17T04:58:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.056+0000 7f2eabd25640 1 -- 172.21.15.150:0/3769730878 >> 172.21.15.150:0/3769730878 conn(0x7f2ea40c4a00 msgr2=0x7f2ea40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.056+0000 7f2eabd25640 1 -- 172.21.15.150:0/3769730878 shutdown_connections 2024-09-17T04:58:53.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.056+0000 7f2eabd25640 1 -- 172.21.15.150:0/3769730878 wait complete. 2024-09-17T04:58:53.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2eabd25640 1 Processor -- start 2024-09-17T04:58:53.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2eabd25640 1 -- start start 2024-09-17T04:58:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2eabd25640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e90004d50 0x7f2ea414e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2eabd25640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea414f1b0 0x7f2ea414f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2eabd25640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ea40ab620 0x7f2ea40aba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2eabd25640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea4070da0 con 0x7f2ea414f1b0 2024-09-17T04:58:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2eabd25640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea4070ac0 con 0x7f2ea40ab620 2024-09-17T04:58:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2eabd25640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea4153640 con 0x7f2e90004d50 2024-09-17T04:58:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2ea92c0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea414f1b0 0x7f2ea414f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2ea92c0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea414f1b0 0x7f2ea414f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57752/0 (socket says 172.21.15.150:57752) 2024-09-17T04:58:53.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2ea92c0640 1 -- 172.21.15.150:0/2437684555 learned_addr learned my addr 172.21.15.150:0/2437684555 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2ea9ac1640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e90004d50 0x7f2ea414e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2eaa2c2640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ea40ab620 0x7f2ea40aba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2ea92c0640 1 -- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e90004d50 msgr2=0x7f2ea414e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2ea92c0640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e90004d50 0x7f2ea414e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2ea92c0640 1 -- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ea40ab620 msgr2=0x7f2ea40aba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2ea92c0640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ea40ab620 0x7f2ea40aba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2ea92c0640 1 -- 172.21.15.150:0/2437684555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea4146a70 con 0x7f2ea414f1b0 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2ea92c0640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea414f1b0 0x7f2ea414f5c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400e660 tx=0x7f2e9400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2e9a7fc640 1 -- 172.21.15.150:0/2437684555 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94015070 con 0x7f2ea414f1b0 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2e9a7fc640 1 -- 172.21.15.150:0/2437684555 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e9400cb50 con 0x7f2ea414f1b0 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2eabd25640 1 -- 172.21.15.150:0/2437684555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea40ac940 con 0x7f2ea414f1b0 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2e9a7fc640 1 -- 172.21.15.150:0/2437684555 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94011460 con 0x7f2ea414f1b0 2024-09-17T04:58:53.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2eabd25640 1 -- 172.21.15.150:0/2437684555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea40ab2c0 con 0x7f2ea414f1b0 2024-09-17T04:58:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.060+0000 7f2eabd25640 1 -- 172.21.15.150:0/2437684555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea4153100 con 0x7f2ea414f1b0 2024-09-17T04:58:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.064+0000 7f2e9a7fc640 1 -- 172.21.15.150:0/2437684555 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2e94018070 con 0x7f2ea414f1b0 2024-09-17T04:58:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.064+0000 7f2e9a7fc640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2e88073230 0x7f2e88075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.064+0000 7f2e9a7fc640 1 -- 172.21.15.150:0/2437684555 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f2e94094da0 con 0x7f2ea414f1b0 2024-09-17T04:58:53.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.064+0000 7f2ea9ac1640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2e88073230 0x7f2e88075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:53.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.064+0000 7f2ea9ac1640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2e88073230 0x7f2e88075740 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009b60 tx=0x7f2ea00091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:53.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.068+0000 7f2e9a7fc640 1 -- 172.21.15.150:0/2437684555 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2e9405f030 con 0x7f2ea414f1b0 2024-09-17T04:58:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.276+0000 7f2eabd25640 1 -- 172.21.15.150:0/2437684555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ea41e8e80 con 0x7f2ea414f1b0 2024-09-17T04:58:53.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.276+0000 7f2e9a7fc640 1 -- 172.21.15.150:0/2437684555 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5701 (secure 0 0 0) 0x7f2e94063a60 con 0x7f2ea414f1b0 2024-09-17T04:58:53.277 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:53.277 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":74,"btime":"2024-09-17T04:58:52:950552+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:51.957364+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:53.277 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 74 2024-09-17T04:58:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.280+0000 7f2eabd25640 1 -- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2e88073230 msgr2=0x7f2e88075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.280+0000 7f2eabd25640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2e88073230 0x7f2e88075740 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009b60 tx=0x7f2ea00091e0 comp rx=0 tx=0).stop 2024-09-17T04:58:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.280+0000 7f2eabd25640 1 -- 172.21.15.150:0/2437684555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea414f1b0 msgr2=0x7f2ea414f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.280+0000 7f2eabd25640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea414f1b0 0x7f2ea414f5c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400e660 tx=0x7f2e9400eb20 comp rx=0 tx=0).stop 2024-09-17T04:58:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.280+0000 7f2eabd25640 1 -- 172.21.15.150:0/2437684555 shutdown_connections 2024-09-17T04:58:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.280+0000 7f2eabd25640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2e88073230 0x7f2e88075740 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.280+0000 7f2eabd25640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ea40ab620 0x7f2ea40aba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.280+0000 7f2eabd25640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ea414f1b0 0x7f2ea414f5c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.280+0000 7f2eabd25640 1 --2- 172.21.15.150:0/2437684555 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2e90004d50 0x7f2ea414e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.280+0000 7f2eabd25640 1 -- 172.21.15.150:0/2437684555 >> 172.21.15.150:0/2437684555 conn(0x7f2ea40c4a00 msgr2=0x7f2ea40c2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.280+0000 7f2eabd25640 1 -- 172.21.15.150:0/2437684555 shutdown_connections 2024-09-17T04:58:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:53.280+0000 7f2eabd25640 1 -- 172.21.15.150:0/2437684555 wait complete. 2024-09-17T04:58:53.302 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T04:58:52:950552+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 74, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 74, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5258": { "addr": "172.21.15.176:6837/713081612", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 713081612, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 713081612, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5258, "incarnation": 72, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:58:51.957364+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5258 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 70, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5901": { "addr": "172.21.15.150:6835/2081398022", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 2081398022, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2081398022, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5901, "incarnation": 67, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:58:33.945203+0000", "qdb_cluster": [ 5901 ], "qdb_leader": 5901, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5901 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:58:55.303 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:55.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 -- 172.21.15.150:0/2934333849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12200a8120 msgr2=0x7f1220149f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:55.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 --2- 172.21.15.150:0/2934333849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12200a8120 0x7f1220149f40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f12200c9e40 tx=0x7f121001c8c0 comp rx=0 tx=0).stop 2024-09-17T04:58:55.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 -- 172.21.15.150:0/2934333849 shutdown_connections 2024-09-17T04:58:55.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 --2- 172.21.15.150:0/2934333849 >> v2:172.21.15.176:3300/0 conn(0x7f12200c37b0 0x7f12201535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:55.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 --2- 172.21.15.150:0/2934333849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12200a8120 0x7f1220149f40 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:55.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 -- 172.21.15.150:0/2934333849 >> 172.21.15.150:0/2934333849 conn(0x7f12200aab60 msgr2=0x7f12200aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:55.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 -- 172.21.15.150:0/2934333849 shutdown_connections 2024-09-17T04:58:55.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 -- 172.21.15.150:0/2934333849 wait complete. 2024-09-17T04:58:55.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 Processor -- start 2024-09-17T04:58:55.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 -- start start 2024-09-17T04:58:55.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12200a8120 0x7f12200add10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:55.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12200c37b0 0x7f122014ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:55.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1220157810 0x7f1220148780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f12200c8a00 con 0x7f12200a8120 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1220072390 con 0x7f1220157810 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.480+0000 7f1224fae640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f12200c8b80 con 0x7f12200c37b0 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f121e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12200a8120 0x7f12200add10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f121e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12200a8120 0x7f12200add10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57770/0 (socket says 172.21.15.150:57770) 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f121e59c640 1 -- 172.21.15.150:0/1815447801 learned_addr learned my addr 172.21.15.150:0/1815447801 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f1215d9b640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12200c37b0 0x7f122014ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f121ed9d640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1220157810 0x7f1220148780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f121e59c640 1 -- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12200c37b0 msgr2=0x7f122014ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f121e59c640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12200c37b0 0x7f122014ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f121e59c640 1 -- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1220157810 msgr2=0x7f1220148780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f121e59c640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1220157810 0x7f1220148780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f121e59c640 1 -- 172.21.15.150:0/1815447801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12200aedc0 con 0x7f12200a8120 2024-09-17T04:58:55.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f121e59c640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12200a8120 0x7f12200add10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f12100194d0 tx=0x7f1210005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:55.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f12177fe640 1 -- 172.21.15.150:0/1815447801 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12100b2040 con 0x7f12200a8120 2024-09-17T04:58:55.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f12177fe640 1 -- 172.21.15.150:0/1815447801 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1210004330 con 0x7f12200a8120 2024-09-17T04:58:55.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f1224fae640 1 -- 172.21.15.150:0/1815447801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12201493d0 con 0x7f12200a8120 2024-09-17T04:58:55.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f12177fe640 1 -- 172.21.15.150:0/1815447801 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210005290 con 0x7f12200a8120 2024-09-17T04:58:55.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f1224fae640 1 -- 172.21.15.150:0/1815447801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12201ee850 con 0x7f12200a8120 2024-09-17T04:58:55.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f1224fae640 1 -- 172.21.15.150:0/1815447801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1220149f40 con 0x7f12200a8120 2024-09-17T04:58:55.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f12177fe640 1 -- 172.21.15.150:0/1815447801 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1210005430 con 0x7f12200a8120 2024-09-17T04:58:55.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.484+0000 7f12177fe640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1200073230 0x7f1200075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:55.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.488+0000 7f12177fe640 1 -- 172.21.15.150:0/1815447801 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f121012eb30 con 0x7f12200a8120 2024-09-17T04:58:55.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.488+0000 7f1215d9b640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1200073230 0x7f1200075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:55.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.488+0000 7f1215d9b640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1200073230 0x7f1200075740 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f12080058d0 tx=0x7f1208005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:55.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.492+0000 7f12177fe640 1 -- 172.21.15.150:0/1815447801 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f12100164b0 con 0x7f12200a8120 2024-09-17T04:58:55.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.748+0000 7f1224fae640 1 -- 172.21.15.150:0/1815447801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1220154330 con 0x7f12200a8120 2024-09-17T04:58:55.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.748+0000 7f12177fe640 1 -- 172.21.15.150:0/1815447801 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+5708 (secure 0 0 0) 0x7f12100f94a0 con 0x7f12200a8120 2024-09-17T04:58:55.748 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:55.748 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":75,"btime":"2024-09-17T04:58:53:961249+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:53.961234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5258,"qdb_cluster":[5258]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:55.749 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 75 2024-09-17T04:58:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.752+0000 7f1224fae640 1 -- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1200073230 msgr2=0x7f1200075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.752+0000 7f1224fae640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1200073230 0x7f1200075740 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f12080058d0 tx=0x7f1208005860 comp rx=0 tx=0).stop 2024-09-17T04:58:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.752+0000 7f1224fae640 1 -- 172.21.15.150:0/1815447801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12200a8120 msgr2=0x7f12200add10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.752+0000 7f1224fae640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12200a8120 0x7f12200add10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f12100194d0 tx=0x7f1210005e90 comp rx=0 tx=0).stop 2024-09-17T04:58:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.752+0000 7f1224fae640 1 -- 172.21.15.150:0/1815447801 shutdown_connections 2024-09-17T04:58:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.752+0000 7f1224fae640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1200073230 0x7f1200075740 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.752+0000 7f1224fae640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1220157810 0x7f1220148780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.752+0000 7f1224fae640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12200c37b0 0x7f122014ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.752+0000 7f1224fae640 1 --2- 172.21.15.150:0/1815447801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12200a8120 0x7f12200add10 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.752+0000 7f1224fae640 1 -- 172.21.15.150:0/1815447801 >> 172.21.15.150:0/1815447801 conn(0x7f12200aab60 msgr2=0x7f12200c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.752+0000 7f1224fae640 1 -- 172.21.15.150:0/1815447801 shutdown_connections 2024-09-17T04:58:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:55.752+0000 7f1224fae640 1 -- 172.21.15.150:0/1815447801 wait complete. 2024-09-17T04:58:55.778 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=5841 2024-09-17T04:58:55.779 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.a 2024-09-17T04:58:56.369 INFO:tasks.ceph.mds.d.smithi176.stderr:2024-09-17T04:58:56.365+0000 7f74fa7b4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:58:56.780 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-17T04:58:56.780 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:58:56.780 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:58:56.782 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:58:56.782 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:58:56.882 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T04:58:56.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.956+0000 7fb39f044640 1 -- 172.21.15.150:0/1899590380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3980c7eb0 msgr2=0x7fb3981540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:56.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.956+0000 7fb39f044640 1 --2- 172.21.15.150:0/1899590380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3980c7eb0 0x7fb3981540b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fb38c009900 tx=0x7fb38c01c8e0 comp rx=0 tx=0).stop 2024-09-17T04:58:56.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.956+0000 7fb39f044640 1 -- 172.21.15.150:0/1899590380 shutdown_connections 2024-09-17T04:58:56.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.956+0000 7fb39f044640 1 --2- 172.21.15.150:0/1899590380 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb398149d70 0x7fb39814a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:56.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.956+0000 7fb39f044640 1 --2- 172.21.15.150:0/1899590380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3980c7eb0 0x7fb3981540b0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:56.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.956+0000 7fb39f044640 1 -- 172.21.15.150:0/1899590380 >> 172.21.15.150:0/1899590380 conn(0x7fb3980aab60 msgr2=0x7fb3980aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:56.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.956+0000 7fb39f044640 1 -- 172.21.15.150:0/1899590380 shutdown_connections 2024-09-17T04:58:56.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.956+0000 7fb39f044640 1 -- 172.21.15.150:0/1899590380 wait complete. 2024-09-17T04:58:56.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39f044640 1 Processor -- start 2024-09-17T04:58:56.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39f044640 1 -- start start 2024-09-17T04:58:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39f044640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3980c7eb0 0x7fb398150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39f044640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3980c8330 0x7fb3981f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39f044640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb398149d70 0x7fb3980ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39f044640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb3981568d0 con 0x7fb398149d70 2024-09-17T04:58:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39f044640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb398156750 con 0x7fb3980c8330 2024-09-17T04:58:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39f044640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb3980c8b50 con 0x7fb3980c7eb0 2024-09-17T04:58:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39d5e1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb398149d70 0x7fb3980ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39d5e1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb398149d70 0x7fb3980ae0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57816/0 (socket says 172.21.15.150:57816) 2024-09-17T04:58:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39d5e1640 1 -- 172.21.15.150:0/3976430274 learned_addr learned my addr 172.21.15.150:0/3976430274 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:58:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39cde0640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3980c7eb0 0x7fb398150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb397fff640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3980c8330 0x7fb3981f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39d5e1640 1 -- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3980c7eb0 msgr2=0x7fb398150300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39d5e1640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3980c7eb0 0x7fb398150300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39d5e1640 1 -- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3980c8330 msgr2=0x7fb3981f05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39d5e1640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3980c8330 0x7fb3981f05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39d5e1640 1 -- 172.21.15.150:0/3976430274 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39814d2a0 con 0x7fb398149d70 2024-09-17T04:58:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.960+0000 7fb39d5e1640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb398149d70 0x7fb3980ae0c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb39000c760 tx=0x7fb39000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.964+0000 7fb3957fa640 1 -- 172.21.15.150:0/3976430274 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb390003ba0 con 0x7fb398149d70 2024-09-17T04:58:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.964+0000 7fb3957fa640 1 -- 172.21.15.150:0/3976430274 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb390003d60 con 0x7fb398149d70 2024-09-17T04:58:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.964+0000 7fb3957fa640 1 -- 172.21.15.150:0/3976430274 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39000aab0 con 0x7fb398149d70 2024-09-17T04:58:56.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.964+0000 7fb39f044640 1 -- 172.21.15.150:0/3976430274 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3981f43d0 con 0x7fb398149d70 2024-09-17T04:58:56.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.964+0000 7fb39f044640 1 -- 172.21.15.150:0/3976430274 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3980c2340 con 0x7fb398149d70 2024-09-17T04:58:56.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.964+0000 7fb39f044640 1 -- 172.21.15.150:0/3976430274 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3981f4200 con 0x7fb398149d70 2024-09-17T04:58:56.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.964+0000 7fb3957fa640 1 -- 172.21.15.150:0/3976430274 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb39000ac50 con 0x7fb398149d70 2024-09-17T04:58:56.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.964+0000 7fb3957fa640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb37c073200 0x7fb37c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:58:56.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.964+0000 7fb3957fa640 1 -- 172.21.15.150:0/3976430274 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7fb390094690 con 0x7fb398149d70 2024-09-17T04:58:56.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.964+0000 7fb39cde0640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb37c073200 0x7fb37c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:58:56.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.964+0000 7fb39cde0640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb37c073200 0x7fb37c075710 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0af000 tx=0x7fb38c01b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:58:56.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:56.972+0000 7fb3957fa640 1 -- 172.21.15.150:0/3976430274 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb39005ef70 con 0x7fb398149d70 2024-09-17T04:58:57.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.208+0000 7fb39f044640 1 -- 172.21.15.150:0/3976430274 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3981540b0 con 0x7fb398149d70 2024-09-17T04:58:57.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.208+0000 7fb3957fa640 1 -- 172.21.15.150:0/3976430274 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6536 (secure 0 0 0) 0x7fb39005e870 con 0x7fb398149d70 2024-09-17T04:58:57.209 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:58:57.209 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":76,"btime":"2024-09-17T04:58:56:975674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6075,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:53.961234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5258,"qdb_cluster":[5258]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:58:57.210 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 76 2024-09-17T04:58:57.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.212+0000 7fb39f044640 1 -- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb37c073200 msgr2=0x7fb37c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:57.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.212+0000 7fb39f044640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb37c073200 0x7fb37c075710 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0af000 tx=0x7fb38c01b120 comp rx=0 tx=0).stop 2024-09-17T04:58:57.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.212+0000 7fb39f044640 1 -- 172.21.15.150:0/3976430274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb398149d70 msgr2=0x7fb3980ae0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:58:57.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.212+0000 7fb39f044640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb398149d70 0x7fb3980ae0c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb39000c760 tx=0x7fb39000cc20 comp rx=0 tx=0).stop 2024-09-17T04:58:57.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.212+0000 7fb39f044640 1 -- 172.21.15.150:0/3976430274 shutdown_connections 2024-09-17T04:58:57.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.212+0000 7fb39f044640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb37c073200 0x7fb37c075710 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:57.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.212+0000 7fb39f044640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb3980c8330 0x7fb3981f05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:57.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.212+0000 7fb39f044640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb398149d70 0x7fb3980ae0c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:57.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.212+0000 7fb39f044640 1 --2- 172.21.15.150:0/3976430274 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb3980c7eb0 0x7fb398150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:58:57.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.212+0000 7fb39f044640 1 -- 172.21.15.150:0/3976430274 >> 172.21.15.150:0/3976430274 conn(0x7fb3980aab60 msgr2=0x7fb39815e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:58:57.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.212+0000 7fb39f044640 1 -- 172.21.15.150:0/3976430274 shutdown_connections 2024-09-17T04:58:57.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:58:57.212+0000 7fb39f044640 1 -- 172.21.15.150:0/3976430274 wait complete. 2024-09-17T04:58:57.231 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-17T04:58:57.231 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before thrashing 2024-09-17T04:59:00.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:59:00.692 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:59:01.232 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:01.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.388+0000 7f14d209b640 1 -- 172.21.15.150:0/2221840792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14cc0c37b0 msgr2=0x7f14cc1535e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:01.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.388+0000 7f14d209b640 1 --2- 172.21.15.150:0/2221840792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14cc0c37b0 0x7f14cc1535e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f14cc14dc20 tx=0x7f14b401c8c0 comp rx=0 tx=0).stop 2024-09-17T04:59:01.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.388+0000 7f14d209b640 1 -- 172.21.15.150:0/2221840792 shutdown_connections 2024-09-17T04:59:01.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.388+0000 7f14d209b640 1 --2- 172.21.15.150:0/2221840792 >> v2:172.21.15.176:3300/0 conn(0x7f14cc157810 0x7f14cc153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:01.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.388+0000 7f14d209b640 1 --2- 172.21.15.150:0/2221840792 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14cc0c37b0 0x7f14cc1535e0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:01.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.388+0000 7f14d209b640 1 -- 172.21.15.150:0/2221840792 >> 172.21.15.150:0/2221840792 conn(0x7f14cc0aab60 msgr2=0x7f14cc0aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:01.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.388+0000 7f14d209b640 1 -- 172.21.15.150:0/2221840792 shutdown_connections 2024-09-17T04:59:01.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.388+0000 7f14d209b640 1 -- 172.21.15.150:0/2221840792 wait complete. 2024-09-17T04:59:01.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.388+0000 7f14d209b640 1 Processor -- start 2024-09-17T04:59:01.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.388+0000 7f14d209b640 1 -- start start 2024-09-17T04:59:01.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14d209b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14cc0a8120 0x7f14cc1f0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:01.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14d209b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14cc0c37b0 0x7f14cc0c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:01.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14d209b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14cc157810 0x7f14cc1520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:01.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14d209b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc160300 con 0x7f14cc0c37b0 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14d209b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc160180 con 0x7f14cc157810 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14d209b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f14cc160480 con 0x7f14cc0a8120 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14cc0c37b0 0x7f14cc0c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14cc0c37b0 0x7f14cc0c23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46004/0 (socket says 172.21.15.150:46004) 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c3fff640 1 -- 172.21.15.150:0/268874371 learned_addr learned my addr 172.21.15.150:0/268874371 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14cb7fe640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14cc0a8120 0x7f14cc1f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14cbfff640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14cc157810 0x7f14cc1520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c3fff640 1 -- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14cc0a8120 msgr2=0x7f14cc1f0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c3fff640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14cc0a8120 0x7f14cc1f0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c3fff640 1 -- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14cc157810 msgr2=0x7f14cc1520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c3fff640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14cc157810 0x7f14cc1520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c3fff640 1 -- 172.21.15.150:0/268874371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc1506a0 con 0x7f14cc0c37b0 2024-09-17T04:59:01.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c3fff640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14cc0c37b0 0x7f14cc0c23b0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f14b40194d0 tx=0x7f14b40a5710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:01.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c8ff9640 1 -- 172.21.15.150:0/268874371 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40b2040 con 0x7f14cc0c37b0 2024-09-17T04:59:01.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c8ff9640 1 -- 172.21.15.150:0/268874371 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14b40042c0 con 0x7f14cc0c37b0 2024-09-17T04:59:01.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14d209b640 1 -- 172.21.15.150:0/268874371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14cc1491b0 con 0x7f14cc0c37b0 2024-09-17T04:59:01.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c8ff9640 1 -- 172.21.15.150:0/268874371 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4005290 con 0x7f14cc0c37b0 2024-09-17T04:59:01.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14d209b640 1 -- 172.21.15.150:0/268874371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14cc149ab0 con 0x7f14cc0c37b0 2024-09-17T04:59:01.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14d209b640 1 -- 172.21.15.150:0/268874371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14cc153210 con 0x7f14cc0c37b0 2024-09-17T04:59:01.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.392+0000 7f14c8ff9640 1 -- 172.21.15.150:0/268874371 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f14b4005430 con 0x7f14cc0c37b0 2024-09-17T04:59:01.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.396+0000 7f14c8ff9640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f14ac073230 0x7f14ac075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:01.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.396+0000 7f14c8ff9640 1 -- 172.21.15.150:0/268874371 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f14b412e570 con 0x7f14cc0c37b0 2024-09-17T04:59:01.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.396+0000 7f14cb7fe640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f14ac073230 0x7f14ac075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:01.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.396+0000 7f14cb7fe640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f14ac073230 0x7f14ac075740 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f14bc007fd0 tx=0x7f14bc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.400+0000 7f14c8ff9640 1 -- 172.21.15.150:0/268874371 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f14b40164b0 con 0x7f14cc0c37b0 2024-09-17T04:59:01.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.648+0000 7f14d209b640 1 -- 172.21.15.150:0/268874371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14cc149f40 con 0x7f14cc0c37b0 2024-09-17T04:59:01.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.648+0000 7f14c8ff9640 1 -- 172.21.15.150:0/268874371 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6536 (secure 0 0 0) 0x7f14b40044b0 con 0x7f14cc0c37b0 2024-09-17T04:59:01.648 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:01.648 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":76,"btime":"2024-09-17T04:58:56:975674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6075,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:53.961234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5258,"qdb_cluster":[5258]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:01.648 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 76 2024-09-17T04:59:01.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.652+0000 7f14d209b640 1 -- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f14ac073230 msgr2=0x7f14ac075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:01.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.652+0000 7f14d209b640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f14ac073230 0x7f14ac075740 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f14bc007fd0 tx=0x7f14bc009040 comp rx=0 tx=0).stop 2024-09-17T04:59:01.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.652+0000 7f14d209b640 1 -- 172.21.15.150:0/268874371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14cc0c37b0 msgr2=0x7f14cc0c23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:01.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.652+0000 7f14d209b640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14cc0c37b0 0x7f14cc0c23b0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f14b40194d0 tx=0x7f14b40a5710 comp rx=0 tx=0).stop 2024-09-17T04:59:01.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.652+0000 7f14d209b640 1 -- 172.21.15.150:0/268874371 shutdown_connections 2024-09-17T04:59:01.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.652+0000 7f14d209b640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f14ac073230 0x7f14ac075740 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:01.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.652+0000 7f14d209b640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14cc157810 0x7f14cc1520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:01.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.652+0000 7f14d209b640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14cc0c37b0 0x7f14cc0c23b0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:01.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.652+0000 7f14d209b640 1 --2- 172.21.15.150:0/268874371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14cc0a8120 0x7f14cc1f0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:01.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.652+0000 7f14d209b640 1 -- 172.21.15.150:0/268874371 >> 172.21.15.150:0/268874371 conn(0x7f14cc0aab60 msgr2=0x7f14cc1562e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:01.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.652+0000 7f14d209b640 1 -- 172.21.15.150:0/268874371 shutdown_connections 2024-09-17T04:59:01.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.652+0000 7f14d209b640 1 -- 172.21.15.150:0/268874371 wait complete. 2024-09-17T04:59:01.673 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-17T04:59:01.673 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:59:01.673 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:59:01.692 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:59:01.692 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:01.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.856+0000 7f731de99640 1 -- 172.21.15.150:0/2932451827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73180c5680 msgr2=0x7f73180c7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:01.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.856+0000 7f731de99640 1 --2- 172.21.15.150:0/2932451827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73180c5680 0x7f73180c7d60 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7300009840 tx=0x7f730001c820 comp rx=0 tx=0).stop 2024-09-17T04:59:01.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.856+0000 7f731de99640 1 -- 172.21.15.150:0/2932451827 shutdown_connections 2024-09-17T04:59:01.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.856+0000 7f731de99640 1 --2- 172.21.15.150:0/2932451827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73180c5680 0x7f73180c7d60 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:01.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.856+0000 7f731de99640 1 -- 172.21.15.150:0/2932451827 >> 172.21.15.150:0/2932451827 conn(0x7f73180ca460 msgr2=0x7f73180c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:01.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.856+0000 7f731de99640 1 -- 172.21.15.150:0/2932451827 shutdown_connections 2024-09-17T04:59:01.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.856+0000 7f731de99640 1 -- 172.21.15.150:0/2932451827 wait complete. 2024-09-17T04:59:01.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f731de99640 1 Processor -- start 2024-09-17T04:59:01.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f731de99640 1 -- start start 2024-09-17T04:59:01.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f731de99640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7318071670 0x7f73180bfc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:01.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f731de99640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73180c5680 0x7f7318150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f731de99640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7318157c00 0x7f7318145850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f731de99640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f73180c6740 con 0x7f7318157c00 2024-09-17T04:59:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f731de99640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f73180c65c0 con 0x7f73180c5680 2024-09-17T04:59:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f731de99640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7318072180 con 0x7f7318071670 2024-09-17T04:59:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f7317fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7318157c00 0x7f7318145850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f7317fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7318157c00 0x7f7318145850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46008/0 (socket says 172.21.15.150:46008) 2024-09-17T04:59:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f7317fff640 1 -- 172.21.15.150:0/633363522 learned_addr learned my addr 172.21.15.150:0/633363522 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f730ffff640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73180c5680 0x7f7318150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f73177fe640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7318071670 0x7f73180bfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f7317fff640 1 -- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7318071670 msgr2=0x7f73180bfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f7317fff640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7318071670 0x7f73180bfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f7317fff640 1 -- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73180c5680 msgr2=0x7f7318150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f7317fff640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73180c5680 0x7f7318150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f7317fff640 1 -- 172.21.15.150:0/633363522 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73181518b0 con 0x7f7318157c00 2024-09-17T04:59:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f7317fff640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7318157c00 0x7f7318145850 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f731000cb10 tx=0x7f731000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f7314ff9640 1 -- 172.21.15.150:0/633363522 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7310016020 con 0x7f7318157c00 2024-09-17T04:59:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f7314ff9640 1 -- 172.21.15.150:0/633363522 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f731000ce50 con 0x7f7318157c00 2024-09-17T04:59:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f731de99640 1 -- 172.21.15.150:0/633363522 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73180beda0 con 0x7f7318157c00 2024-09-17T04:59:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f7314ff9640 1 -- 172.21.15.150:0/633363522 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731000ad60 con 0x7f7318157c00 2024-09-17T04:59:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f731de99640 1 -- 172.21.15.150:0/633363522 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73181550f0 con 0x7f7318157c00 2024-09-17T04:59:01.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.860+0000 7f731de99640 1 -- 172.21.15.150:0/633363522 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73180c7d60 con 0x7f7318157c00 2024-09-17T04:59:01.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.864+0000 7f7314ff9640 1 -- 172.21.15.150:0/633363522 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7310019020 con 0x7f7318157c00 2024-09-17T04:59:01.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.864+0000 7f7314ff9640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72f8073200 0x7f72f8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:01.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.864+0000 7f7314ff9640 1 -- 172.21.15.150:0/633363522 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f7310094af0 con 0x7f7318157c00 2024-09-17T04:59:01.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.864+0000 7f73177fe640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72f8073200 0x7f72f8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:01.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.864+0000 7f73177fe640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72f8073200 0x7f72f8075710 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7308007fd0 tx=0x7f7308009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:01.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:01.868+0000 7f7314ff9640 1 -- 172.21.15.150:0/633363522 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7310020020 con 0x7f7318157c00 2024-09-17T04:59:02.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.124+0000 7f731de99640 1 -- 172.21.15.150:0/633363522 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f731814d830 con 0x7f7318157c00 2024-09-17T04:59:02.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f7314ff9640 1 -- 172.21.15.150:0/633363522 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6536 (secure 0 0 0) 0x7f731005ed90 con 0x7f7318157c00 2024-09-17T04:59:02.126 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:02.126 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":76,"btime":"2024-09-17T04:58:56:975674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6075,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:53.961234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5258,"qdb_cluster":[5258]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:02.126 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 76 2024-09-17T04:59:02.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f731de99640 1 -- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72f8073200 msgr2=0x7f72f8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:02.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f731de99640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72f8073200 0x7f72f8075710 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7308007fd0 tx=0x7f7308009040 comp rx=0 tx=0).stop 2024-09-17T04:59:02.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f731de99640 1 -- 172.21.15.150:0/633363522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7318157c00 msgr2=0x7f7318145850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:02.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f731de99640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7318157c00 0x7f7318145850 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f731000cb10 tx=0x7f731000cc00 comp rx=0 tx=0).stop 2024-09-17T04:59:02.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f731de99640 1 -- 172.21.15.150:0/633363522 shutdown_connections 2024-09-17T04:59:02.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f731de99640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72f8073200 0x7f72f8075710 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:02.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f731de99640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73180c5680 0x7f7318150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:02.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f731de99640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7318157c00 0x7f7318145850 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:02.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f731de99640 1 --2- 172.21.15.150:0/633363522 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7318071670 0x7f73180bfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:02.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f731de99640 1 -- 172.21.15.150:0/633363522 >> 172.21.15.150:0/633363522 conn(0x7f73180ca460 msgr2=0x7f7318148870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:02.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f731de99640 1 -- 172.21.15.150:0/633363522 shutdown_connections 2024-09-17T04:59:02.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:02.128+0000 7f731de99640 1 -- 172.21.15.150:0/633363522 wait complete. 2024-09-17T04:59:02.146 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:59:04.148 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.316+0000 7f72d20df640 1 -- 172.21.15.150:0/3290970025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72cc0c5490 msgr2=0x7f72cc153c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.316+0000 7f72d20df640 1 --2- 172.21.15.150:0/3290970025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72cc0c5490 0x7f72cc153c50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f72cc075090 tx=0x7f72c001c7a0 comp rx=0 tx=0).stop 2024-09-17T04:59:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.316+0000 7f72d20df640 1 -- 172.21.15.150:0/3290970025 shutdown_connections 2024-09-17T04:59:04.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.316+0000 7f72d20df640 1 --2- 172.21.15.150:0/3290970025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72cc0c5490 0x7f72cc153c50 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:04.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.316+0000 7f72d20df640 1 --2- 172.21.15.150:0/3290970025 >> v2:172.21.15.176:3300/0 conn(0x7f72cc06f920 0x7f72cc0c6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:04.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.316+0000 7f72d20df640 1 -- 172.21.15.150:0/3290970025 >> 172.21.15.150:0/3290970025 conn(0x7f72cc0abb00 msgr2=0x7f72cc0abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:04.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.316+0000 7f72d20df640 1 -- 172.21.15.150:0/3290970025 shutdown_connections 2024-09-17T04:59:04.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.316+0000 7f72d20df640 1 -- 172.21.15.150:0/3290970025 wait complete. 2024-09-17T04:59:04.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72d20df640 1 Processor -- start 2024-09-17T04:59:04.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72d20df640 1 -- start start 2024-09-17T04:59:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72d20df640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72cc06f920 0x7f72cc14f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72d20df640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72cc0c5490 0x7f72cc1f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72d20df640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72cc154190 0x7f72cc14e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72d20df640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc0c7140 con 0x7f72cc154190 2024-09-17T04:59:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72d20df640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc0c6fc0 con 0x7f72cc06f920 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72d20df640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f72cc1639d0 con 0x7f72cc0c5490 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72cbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72cc154190 0x7f72cc14e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72cbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72cc154190 0x7f72cc14e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46032/0 (socket says 172.21.15.150:46032) 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72cbfff640 1 -- 172.21.15.150:0/111454974 learned_addr learned my addr 172.21.15.150:0/111454974 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72cb7fe640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72cc06f920 0x7f72cc14f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72caffd640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72cc0c5490 0x7f72cc1f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72cbfff640 1 -- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72cc0c5490 msgr2=0x7f72cc1f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72cbfff640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72cc0c5490 0x7f72cc1f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72cbfff640 1 -- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72cc06f920 msgr2=0x7f72cc14f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72cbfff640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72cc06f920 0x7f72cc14f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72cbfff640 1 -- 172.21.15.150:0/111454974 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72cc0ae7a0 con 0x7f72cc154190 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72cbfff640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72cc154190 0x7f72cc14e590 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00b490 tx=0x7f72bc00b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72b3fff640 1 -- 172.21.15.150:0/111454974 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc015020 con 0x7f72cc154190 2024-09-17T04:59:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72b3fff640 1 -- 172.21.15.150:0/111454974 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72bc011920 con 0x7f72cc154190 2024-09-17T04:59:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72b3fff640 1 -- 172.21.15.150:0/111454974 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc011db0 con 0x7f72cc154190 2024-09-17T04:59:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72d20df640 1 -- 172.21.15.150:0/111454974 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72cc0aea40 con 0x7f72cc154190 2024-09-17T04:59:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.320+0000 7f72d20df640 1 -- 172.21.15.150:0/111454974 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72cc072e40 con 0x7f72cc154190 2024-09-17T04:59:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.324+0000 7f72d20df640 1 -- 172.21.15.150:0/111454974 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72cc0c6650 con 0x7f72cc154190 2024-09-17T04:59:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.324+0000 7f72b3fff640 1 -- 172.21.15.150:0/111454974 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f72bc018020 con 0x7f72cc154190 2024-09-17T04:59:04.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.324+0000 7f72b3fff640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72b8073230 0x7f72b8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:04.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.324+0000 7f72b3fff640 1 -- 172.21.15.150:0/111454974 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f72bc094450 con 0x7f72cc154190 2024-09-17T04:59:04.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.324+0000 7f72cb7fe640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72b8073230 0x7f72b8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:04.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.324+0000 7f72cb7fe640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72b8073230 0x7f72b8075740 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f72c4009e60 tx=0x7f72c40095b0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:04.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.328+0000 7f72b3fff640 1 -- 172.21.15.150:0/111454974 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f72bc01f030 con 0x7f72cc154190 2024-09-17T04:59:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.528+0000 7f72d20df640 1 -- 172.21.15.150:0/111454974 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72cc159260 con 0x7f72cc154190 2024-09-17T04:59:04.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72b3fff640 1 -- 172.21.15.150:0/111454974 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6536 (secure 0 0 0) 0x7f72bc05edc0 con 0x7f72cc154190 2024-09-17T04:59:04.530 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:04.530 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":76,"btime":"2024-09-17T04:58:56:975674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6075,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:53.961234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5258,"qdb_cluster":[5258]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:04.531 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 76 2024-09-17T04:59:04.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72d20df640 1 -- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72b8073230 msgr2=0x7f72b8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:04.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72d20df640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72b8073230 0x7f72b8075740 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f72c4009e60 tx=0x7f72c40095b0 comp rx=0 tx=0).stop 2024-09-17T04:59:04.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72d20df640 1 -- 172.21.15.150:0/111454974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72cc154190 msgr2=0x7f72cc14e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:04.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72d20df640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72cc154190 0x7f72cc14e590 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00b490 tx=0x7f72bc00b950 comp rx=0 tx=0).stop 2024-09-17T04:59:04.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72d20df640 1 -- 172.21.15.150:0/111454974 shutdown_connections 2024-09-17T04:59:04.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72d20df640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72b8073230 0x7f72b8075740 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:04.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72d20df640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72cc154190 0x7f72cc14e590 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:04.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72d20df640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72cc0c5490 0x7f72cc1f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:04.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72d20df640 1 --2- 172.21.15.150:0/111454974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72cc06f920 0x7f72cc14f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:04.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72d20df640 1 -- 172.21.15.150:0/111454974 >> 172.21.15.150:0/111454974 conn(0x7f72cc0abb00 msgr2=0x7f72cc0abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:04.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72d20df640 1 -- 172.21.15.150:0/111454974 shutdown_connections 2024-09-17T04:59:04.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:04.532+0000 7f72d20df640 1 -- 172.21.15.150:0/111454974 wait complete. 2024-09-17T04:59:04.551 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:59:06.551 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:06.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.720+0000 7f7118e1a640 1 -- 172.21.15.150:0/1215342927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71140ad7e0 msgr2=0x7f71140701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:06.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.720+0000 7f7118e1a640 1 --2- 172.21.15.150:0/1215342927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71140ad7e0 0x7f71140701c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f710c009c80 tx=0x7f710c01cda0 comp rx=0 tx=0).stop 2024-09-17T04:59:06.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.720+0000 7f7118e1a640 1 -- 172.21.15.150:0/1215342927 shutdown_connections 2024-09-17T04:59:06.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.720+0000 7f7118e1a640 1 --2- 172.21.15.150:0/1215342927 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71140ad7e0 0x7f71140701c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:06.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.720+0000 7f7118e1a640 1 --2- 172.21.15.150:0/1215342927 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71140c3db0 0x7f71140ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:06.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.720+0000 7f7118e1a640 1 -- 172.21.15.150:0/1215342927 >> 172.21.15.150:0/1215342927 conn(0x7f71140c4a00 msgr2=0x7f71140c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:06.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.720+0000 7f7118e1a640 1 -- 172.21.15.150:0/1215342927 shutdown_connections 2024-09-17T04:59:06.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.720+0000 7f7118e1a640 1 -- 172.21.15.150:0/1215342927 wait complete. 2024-09-17T04:59:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.720+0000 7f7118e1a640 1 Processor -- start 2024-09-17T04:59:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.720+0000 7f7118e1a640 1 -- start start 2024-09-17T04:59:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7118e1a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71140ad7e0 0x7f7114157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7118e1a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71140c3db0 0x7f711414a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7118e1a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71140c41c0 0x7f7114148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7118e1a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f71140a75f0 con 0x7f71140c3db0 2024-09-17T04:59:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7118e1a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f71140add70 con 0x7f71140ad7e0 2024-09-17T04:59:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7118e1a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f71140a7770 con 0x7f71140c41c0 2024-09-17T04:59:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7111d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71140c3db0 0x7f711414a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7111d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71140c3db0 0x7f711414a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46044/0 (socket says 172.21.15.150:46044) 2024-09-17T04:59:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7111d9b640 1 -- 172.21.15.150:0/810109278 learned_addr learned my addr 172.21.15.150:0/810109278 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:06.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f711259c640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71140ad7e0 0x7f7114157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:06.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7112d9d640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71140c41c0 0x7f7114148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:06.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7111d9b640 1 -- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71140c41c0 msgr2=0x7f7114148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:06.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7111d9b640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71140c41c0 0x7f7114148e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:06.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7111d9b640 1 -- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71140ad7e0 msgr2=0x7f7114157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:06.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7111d9b640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71140ad7e0 0x7f7114157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7111d9b640 1 -- 172.21.15.150:0/810109278 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7114156570 con 0x7f71140c3db0 2024-09-17T04:59:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7111d9b640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71140c3db0 0x7f711414a2a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7108001ec0 tx=0x7f710800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7102ffd640 1 -- 172.21.15.150:0/810109278 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7108017070 con 0x7f71140c3db0 2024-09-17T04:59:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7102ffd640 1 -- 172.21.15.150:0/810109278 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71080053c0 con 0x7f71140c3db0 2024-09-17T04:59:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7118e1a640 1 -- 172.21.15.150:0/810109278 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7114157330 con 0x7f71140c3db0 2024-09-17T04:59:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7102ffd640 1 -- 172.21.15.150:0/810109278 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7108003860 con 0x7f71140c3db0 2024-09-17T04:59:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7118e1a640 1 -- 172.21.15.150:0/810109278 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7114158530 con 0x7f71140c3db0 2024-09-17T04:59:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7118e1a640 1 -- 172.21.15.150:0/810109278 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71140ad2a0 con 0x7f71140c3db0 2024-09-17T04:59:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.724+0000 7f7102ffd640 1 -- 172.21.15.150:0/810109278 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f710801a070 con 0x7f71140c3db0 2024-09-17T04:59:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.728+0000 7f7102ffd640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f70f4073200 0x7f70f4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.728+0000 7f7102ffd640 1 -- 172.21.15.150:0/810109278 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f7108093d00 con 0x7f71140c3db0 2024-09-17T04:59:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.728+0000 7f711259c640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f70f4073200 0x7f70f4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:06.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.728+0000 7f711259c640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f70f4073200 0x7f70f4075710 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f71140c2500 tx=0x7f71040091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:06.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.732+0000 7f7102ffd640 1 -- 172.21.15.150:0/810109278 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f710805e670 con 0x7f71140c3db0 2024-09-17T04:59:06.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.940+0000 7f7118e1a640 1 -- 172.21.15.150:0/810109278 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71141570c0 con 0x7f71140c3db0 2024-09-17T04:59:06.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7102ffd640 1 -- 172.21.15.150:0/810109278 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6536 (secure 0 0 0) 0x7f710805df70 con 0x7f71140c3db0 2024-09-17T04:59:06.942 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:06.942 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":76,"btime":"2024-09-17T04:58:56:975674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6075,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:53.961234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5258,"qdb_cluster":[5258]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:06.942 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 76 2024-09-17T04:59:06.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7118e1a640 1 -- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f70f4073200 msgr2=0x7f70f4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:06.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7118e1a640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f70f4073200 0x7f70f4075710 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f71140c2500 tx=0x7f71040091e0 comp rx=0 tx=0).stop 2024-09-17T04:59:06.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7118e1a640 1 -- 172.21.15.150:0/810109278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71140c3db0 msgr2=0x7f711414a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:06.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7118e1a640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71140c3db0 0x7f711414a2a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7108001ec0 tx=0x7f710800c890 comp rx=0 tx=0).stop 2024-09-17T04:59:06.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7118e1a640 1 -- 172.21.15.150:0/810109278 shutdown_connections 2024-09-17T04:59:06.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7118e1a640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f70f4073200 0x7f70f4075710 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7118e1a640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f71140c41c0 0x7f7114148e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7118e1a640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f71140c3db0 0x7f711414a2a0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7118e1a640 1 --2- 172.21.15.150:0/810109278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71140ad7e0 0x7f7114157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7118e1a640 1 -- 172.21.15.150:0/810109278 >> 172.21.15.150:0/810109278 conn(0x7f71140c4a00 msgr2=0x7f7114153ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7118e1a640 1 -- 172.21.15.150:0/810109278 shutdown_connections 2024-09-17T04:59:06.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:06.944+0000 7f7118e1a640 1 -- 172.21.15.150:0/810109278 wait complete. 2024-09-17T04:59:06.963 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:59:08.964 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:09.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.136+0000 7fed17ea2640 1 -- 172.21.15.150:0/3863526402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed10073440 msgr2=0x7fed100a7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:09.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.136+0000 7fed17ea2640 1 --2- 172.21.15.150:0/3863526402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed10073440 0x7fed100a7300 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009840 tx=0x7fed0c01c790 comp rx=0 tx=0).stop 2024-09-17T04:59:09.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.136+0000 7fed17ea2640 1 -- 172.21.15.150:0/3863526402 shutdown_connections 2024-09-17T04:59:09.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.136+0000 7fed17ea2640 1 --2- 172.21.15.150:0/3863526402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed10073440 0x7fed100a7300 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:09.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.136+0000 7fed17ea2640 1 -- 172.21.15.150:0/3863526402 >> 172.21.15.150:0/3863526402 conn(0x7fed100c4a00 msgr2=0x7fed100c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:09.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.136+0000 7fed17ea2640 1 -- 172.21.15.150:0/3863526402 shutdown_connections 2024-09-17T04:59:09.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.136+0000 7fed17ea2640 1 -- 172.21.15.150:0/3863526402 wait complete. 2024-09-17T04:59:09.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed17ea2640 1 Processor -- start 2024-09-17T04:59:09.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed17ea2640 1 -- start start 2024-09-17T04:59:09.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed17ea2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed10070e50 0x7fed10156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:09.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed17ea2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed10073440 0x7fed100abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:09.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed17ea2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed10074450 0x7fed10145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:09.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed17ea2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fed10074f70 con 0x7fed10070e50 2024-09-17T04:59:09.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed17ea2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fed100adef0 con 0x7fed10073440 2024-09-17T04:59:09.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed17ea2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fed100750f0 con 0x7fed10074450 2024-09-17T04:59:09.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed15c3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed10070e50 0x7fed10156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:09.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed15c3e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed10070e50 0x7fed10156520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46058/0 (socket says 172.21.15.150:46058) 2024-09-17T04:59:09.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed15c3e640 1 -- 172.21.15.150:0/3131348206 learned_addr learned my addr 172.21.15.150:0/3131348206 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:09.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed1543d640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed10073440 0x7fed100abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:09.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed1643f640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed10074450 0x7fed10145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:09.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed15c3e640 1 -- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed10074450 msgr2=0x7fed10145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:09.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed15c3e640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed10074450 0x7fed10145f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:09.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed15c3e640 1 -- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed10073440 msgr2=0x7fed100abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:09.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed15c3e640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed10073440 0x7fed100abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:09.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed15c3e640 1 -- 172.21.15.150:0/3131348206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed101553f0 con 0x7fed10070e50 2024-09-17T04:59:09.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed15c3e640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed10070e50 0x7fed10156520 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fed0c098170 tx=0x7fed0c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:09.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed067fc640 1 -- 172.21.15.150:0/3131348206 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c005240 con 0x7fed10070e50 2024-09-17T04:59:09.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed17ea2640 1 -- 172.21.15.150:0/3131348206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1014cfe0 con 0x7fed10070e50 2024-09-17T04:59:09.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed067fc640 1 -- 172.21.15.150:0/3131348206 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed0c0be070 con 0x7fed10070e50 2024-09-17T04:59:09.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed067fc640 1 -- 172.21.15.150:0/3131348206 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c0a6640 con 0x7fed10070e50 2024-09-17T04:59:09.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed17ea2640 1 -- 172.21.15.150:0/3131348206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed100ac8b0 con 0x7fed10070e50 2024-09-17T04:59:09.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.140+0000 7fed17ea2640 1 -- 172.21.15.150:0/3131348206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed100ad840 con 0x7fed10070e50 2024-09-17T04:59:09.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.144+0000 7fed067fc640 1 -- 172.21.15.150:0/3131348206 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fed0c0046d0 con 0x7fed10070e50 2024-09-17T04:59:09.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.144+0000 7fed067fc640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fecf4073230 0x7fecf4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:09.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.144+0000 7fed067fc640 1 -- 172.21.15.150:0/3131348206 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7fed0c130f20 con 0x7fed10070e50 2024-09-17T04:59:09.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.144+0000 7fed1543d640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fecf4073230 0x7fecf4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:09.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.144+0000 7fed1543d640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fecf4073230 0x7fecf4075740 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fed00007fd0 tx=0x7fed00009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:09.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.148+0000 7fed067fc640 1 -- 172.21.15.150:0/3131348206 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fed0c0c7020 con 0x7fed10070e50 2024-09-17T04:59:09.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.352+0000 7fed17ea2640 1 -- 172.21.15.150:0/3131348206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed100c2510 con 0x7fed10070e50 2024-09-17T04:59:09.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed067fc640 1 -- 172.21.15.150:0/3131348206 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6536 (secure 0 0 0) 0x7fed0c0ac030 con 0x7fed10070e50 2024-09-17T04:59:09.354 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:09.354 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":76,"btime":"2024-09-17T04:58:56:975674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6075,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:53.961234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5258,"qdb_cluster":[5258]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:09.354 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 76 2024-09-17T04:59:09.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed17ea2640 1 -- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fecf4073230 msgr2=0x7fecf4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:09.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed17ea2640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fecf4073230 0x7fecf4075740 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fed00007fd0 tx=0x7fed00009040 comp rx=0 tx=0).stop 2024-09-17T04:59:09.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed17ea2640 1 -- 172.21.15.150:0/3131348206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed10070e50 msgr2=0x7fed10156520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:09.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed17ea2640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed10070e50 0x7fed10156520 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fed0c098170 tx=0x7fed0c005dc0 comp rx=0 tx=0).stop 2024-09-17T04:59:09.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed17ea2640 1 -- 172.21.15.150:0/3131348206 shutdown_connections 2024-09-17T04:59:09.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed17ea2640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fecf4073230 0x7fecf4075740 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:09.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed17ea2640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed10073440 0x7fed100abe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:09.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed17ea2640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed10074450 0x7fed10145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:09.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed17ea2640 1 --2- 172.21.15.150:0/3131348206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed10070e50 0x7fed10156520 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:09.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed17ea2640 1 -- 172.21.15.150:0/3131348206 >> 172.21.15.150:0/3131348206 conn(0x7fed100c4a00 msgr2=0x7fed10074860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:09.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed17ea2640 1 -- 172.21.15.150:0/3131348206 shutdown_connections 2024-09-17T04:59:09.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:09.356+0000 7fed17ea2640 1 -- 172.21.15.150:0/3131348206 wait complete. 2024-09-17T04:59:09.376 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:59:11.377 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:11.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.540+0000 7f9392f20640 1 -- 172.21.15.150:0/1097227108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c14da10 msgr2=0x7f938c1536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:11.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.540+0000 7f9392f20640 1 --2- 172.21.15.150:0/1097227108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c14da10 0x7f938c1536e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f938c0c4600 tx=0x7f938001c6e0 comp rx=0 tx=0).stop 2024-09-17T04:59:11.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.540+0000 7f9392f20640 1 -- 172.21.15.150:0/1097227108 shutdown_connections 2024-09-17T04:59:11.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.540+0000 7f9392f20640 1 --2- 172.21.15.150:0/1097227108 >> v2:172.21.15.176:3300/0 conn(0x7f938c153c20 0x7f938c154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:11.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.540+0000 7f9392f20640 1 --2- 172.21.15.150:0/1097227108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c14da10 0x7f938c1536e0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:11.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.540+0000 7f9392f20640 1 -- 172.21.15.150:0/1097227108 >> 172.21.15.150:0/1097227108 conn(0x7f938c070b50 msgr2=0x7f938c070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:11.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.540+0000 7f9392f20640 1 -- 172.21.15.150:0/1097227108 shutdown_connections 2024-09-17T04:59:11.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.540+0000 7f9392f20640 1 -- 172.21.15.150:0/1097227108 wait complete. 2024-09-17T04:59:11.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f9392f20640 1 Processor -- start 2024-09-17T04:59:11.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f9392f20640 1 -- start start 2024-09-17T04:59:11.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f9392f20640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f938c0ab9d0 0x7f938c0bf270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:11.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f9392f20640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c14da10 0x7f938c0bd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f9392f20640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f938c153c20 0x7f938c0ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f9392f20640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f938c069d70 con 0x7f938c14da10 2024-09-17T04:59:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f9392f20640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f938c069bf0 con 0x7f938c0ab9d0 2024-09-17T04:59:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f9392f20640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f938c1608b0 con 0x7f938c153c20 2024-09-17T04:59:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f938bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c14da10 0x7f938c0bd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f938bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c14da10 0x7f938c0bd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58298/0 (socket says 172.21.15.150:58298) 2024-09-17T04:59:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f938bfff640 1 -- 172.21.15.150:0/321705885 learned_addr learned my addr 172.21.15.150:0/321705885 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f9390cbc640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f938c0ab9d0 0x7f938c0bf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f938bfff640 1 -- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f938c153c20 msgr2=0x7f938c0ae060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f93914bd640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f938c153c20 0x7f938c0ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f938bfff640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f938c153c20 0x7f938c0ae060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:11.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f938bfff640 1 -- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f938c0ab9d0 msgr2=0x7f938c0bf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:11.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f938bfff640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f938c0ab9d0 0x7f938c0bf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:11.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f938bfff640 1 -- 172.21.15.150:0/321705885 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f938c0bf7b0 con 0x7f938c14da10 2024-09-17T04:59:11.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f9390cbc640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f938c0ab9d0 0x7f938c0bf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:59:11.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f938bfff640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c14da10 0x7f938c0bd880 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f938001a310 tx=0x7f93800a84a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:11.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f93897fa640 1 -- 172.21.15.150:0/321705885 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93800be070 con 0x7f938c14da10 2024-09-17T04:59:11.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f93897fa640 1 -- 172.21.15.150:0/321705885 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93800a8a50 con 0x7f938c14da10 2024-09-17T04:59:11.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f9392f20640 1 -- 172.21.15.150:0/321705885 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f938c145ae0 con 0x7f938c14da10 2024-09-17T04:59:11.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f93897fa640 1 -- 172.21.15.150:0/321705885 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9380005040 con 0x7f938c14da10 2024-09-17T04:59:11.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.544+0000 7f9392f20640 1 -- 172.21.15.150:0/321705885 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f938c0be210 con 0x7f938c14da10 2024-09-17T04:59:11.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.548+0000 7f9392f20640 1 -- 172.21.15.150:0/321705885 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f938c1536e0 con 0x7f938c14da10 2024-09-17T04:59:11.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.548+0000 7f93897fa640 1 -- 172.21.15.150:0/321705885 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f93800051e0 con 0x7f938c14da10 2024-09-17T04:59:11.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.548+0000 7f93897fa640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9378073200 0x7f9378075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:11.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.548+0000 7f93897fa640 1 -- 172.21.15.150:0/321705885 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f938012f000 con 0x7f938c14da10 2024-09-17T04:59:11.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.548+0000 7f9390cbc640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9378073200 0x7f9378075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:11.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.548+0000 7f9390cbc640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9378073200 0x7f9378075710 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f93840015b0 tx=0x7f9384009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:11.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.552+0000 7f93897fa640 1 -- 172.21.15.150:0/321705885 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f93800c7020 con 0x7f938c14da10 2024-09-17T04:59:11.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.756+0000 7f9392f20640 1 -- 172.21.15.150:0/321705885 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f938c075060 con 0x7f938c14da10 2024-09-17T04:59:11.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.760+0000 7f93897fa640 1 -- 172.21.15.150:0/321705885 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6536 (secure 0 0 0) 0x7f938c0bf7b0 con 0x7f938c14da10 2024-09-17T04:59:11.759 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:11.759 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":76,"btime":"2024-09-17T04:58:56:975674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6075,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:53.961234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5258,"qdb_cluster":[5258]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:11.759 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 76 2024-09-17T04:59:11.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.760+0000 7f9392f20640 1 -- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9378073200 msgr2=0x7f9378075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:11.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.760+0000 7f9392f20640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9378073200 0x7f9378075710 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f93840015b0 tx=0x7f9384009040 comp rx=0 tx=0).stop 2024-09-17T04:59:11.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.760+0000 7f9392f20640 1 -- 172.21.15.150:0/321705885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c14da10 msgr2=0x7f938c0bd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:11.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.760+0000 7f9392f20640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c14da10 0x7f938c0bd880 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f938001a310 tx=0x7f93800a84a0 comp rx=0 tx=0).stop 2024-09-17T04:59:11.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.764+0000 7f9392f20640 1 -- 172.21.15.150:0/321705885 shutdown_connections 2024-09-17T04:59:11.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.764+0000 7f9392f20640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9378073200 0x7f9378075710 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:11.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.764+0000 7f9392f20640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f938c153c20 0x7f938c0ae060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:11.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.764+0000 7f9392f20640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f938c14da10 0x7f938c0bd880 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:11.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.764+0000 7f9392f20640 1 --2- 172.21.15.150:0/321705885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f938c0ab9d0 0x7f938c0bf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:11.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.764+0000 7f9392f20640 1 -- 172.21.15.150:0/321705885 >> 172.21.15.150:0/321705885 conn(0x7f938c070b50 msgr2=0x7f938c0aa240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:11.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.764+0000 7f9392f20640 1 -- 172.21.15.150:0/321705885 shutdown_connections 2024-09-17T04:59:11.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:11.764+0000 7f9392f20640 1 -- 172.21.15.150:0/321705885 wait complete. 2024-09-17T04:59:11.783 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:59:13.785 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:13.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.952+0000 7f6dd487f640 1 -- 172.21.15.150:0/2079210827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db8004810 msgr2=0x7f6db8004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:13.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.952+0000 7f6dd487f640 1 --2- 172.21.15.150:0/2079210827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db8004810 0x7f6db8004c20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4009b60 tx=0x7f6dc401d270 comp rx=0 tx=0).stop 2024-09-17T04:59:13.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.952+0000 7f6dd487f640 1 -- 172.21.15.150:0/2079210827 shutdown_connections 2024-09-17T04:59:13.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.952+0000 7f6dd487f640 1 --2- 172.21.15.150:0/2079210827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db8004810 0x7f6db8004c20 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:13.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.952+0000 7f6dd487f640 1 -- 172.21.15.150:0/2079210827 >> 172.21.15.150:0/2079210827 conn(0x7f6dd00c4a00 msgr2=0x7f6dd00c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:13.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.952+0000 7f6dd487f640 1 -- 172.21.15.150:0/2079210827 shutdown_connections 2024-09-17T04:59:13.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.952+0000 7f6dd487f640 1 -- 172.21.15.150:0/2079210827 wait complete. 2024-09-17T04:59:13.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dd487f640 1 Processor -- start 2024-09-17T04:59:13.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dd487f640 1 -- start start 2024-09-17T04:59:13.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dd487f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6db8004810 0x7f6dd01e3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:13.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dd487f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd01eea40 0x7f6dd01f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:13.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dd487f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dd01f20e0 0x7f6dd014beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:13.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dd487f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd0074e70 con 0x7f6dd01eea40 2024-09-17T04:59:13.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dd487f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd00adef0 con 0x7f6db8004810 2024-09-17T04:59:13.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dd487f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6dd0074ff0 con 0x7f6dd01f20e0 2024-09-17T04:59:13.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dcdd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd01eea40 0x7f6dd01f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dcdd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd01eea40 0x7f6dd01f1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58324/0 (socket says 172.21.15.150:58324) 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dcdd9b640 1 -- 172.21.15.150:0/3296554801 learned_addr learned my addr 172.21.15.150:0/3296554801 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dced9d640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dd01f20e0 0x7f6dd014beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dce59c640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6db8004810 0x7f6dd01e3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dcdd9b640 1 -- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dd01f20e0 msgr2=0x7f6dd014beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dcdd9b640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dd01f20e0 0x7f6dd014beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dcdd9b640 1 -- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6db8004810 msgr2=0x7f6dd01e3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dcdd9b640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6db8004810 0x7f6dd01e3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dcdd9b640 1 -- 172.21.15.150:0/3296554801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd01ef4e0 con 0x7f6dd01eea40 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dcdd9b640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd01eea40 0x7f6dd01f1ba0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6dc401d0a0 tx=0x7f6dc40056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dbeffd640 1 -- 172.21.15.150:0/3296554801 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc40b1040 con 0x7f6dd01eea40 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dbeffd640 1 -- 172.21.15.150:0/3296554801 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6dc40ac7b0 con 0x7f6dd01eea40 2024-09-17T04:59:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dd487f640 1 -- 172.21.15.150:0/3296554801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd0155460 con 0x7f6dd01eea40 2024-09-17T04:59:13.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dbeffd640 1 -- 172.21.15.150:0/3296554801 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc40acc00 con 0x7f6dd01eea40 2024-09-17T04:59:13.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dd487f640 1 -- 172.21.15.150:0/3296554801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dd0154ea0 con 0x7f6dd01eea40 2024-09-17T04:59:13.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.956+0000 7f6dd487f640 1 -- 172.21.15.150:0/3296554801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dd01eef10 con 0x7f6dd01eea40 2024-09-17T04:59:13.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.960+0000 7f6dbeffd640 1 -- 172.21.15.150:0/3296554801 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6dc40be020 con 0x7f6dd01eea40 2024-09-17T04:59:13.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.960+0000 7f6dbeffd640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6db0073200 0x7f6db0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:13.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.960+0000 7f6dbeffd640 1 -- 172.21.15.150:0/3296554801 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f6dc412fd50 con 0x7f6dd01eea40 2024-09-17T04:59:13.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.960+0000 7f6dce59c640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6db0073200 0x7f6db0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:13.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.960+0000 7f6dce59c640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6db0073200 0x7f6db0075710 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0001870 tx=0x7f6dc0001680 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:13.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:13.964+0000 7f6dbeffd640 1 -- 172.21.15.150:0/3296554801 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6dc40c5020 con 0x7f6dd01eea40 2024-09-17T04:59:14.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.172+0000 7f6dd487f640 1 -- 172.21.15.150:0/3296554801 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6dd00ace00 con 0x7f6dd01eea40 2024-09-17T04:59:14.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.172+0000 7f6dbeffd640 1 -- 172.21.15.150:0/3296554801 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6536 (secure 0 0 0) 0x7f6dc4016f80 con 0x7f6dd01eea40 2024-09-17T04:59:14.172 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:14.172 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":76,"btime":"2024-09-17T04:58:56:975674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6075,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:53.961234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5258,"qdb_cluster":[5258]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 76 2024-09-17T04:59:14.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.176+0000 7f6dd487f640 1 -- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6db0073200 msgr2=0x7f6db0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:14.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.176+0000 7f6dd487f640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6db0073200 0x7f6db0075710 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0001870 tx=0x7f6dc0001680 comp rx=0 tx=0).stop 2024-09-17T04:59:14.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.176+0000 7f6dd487f640 1 -- 172.21.15.150:0/3296554801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd01eea40 msgr2=0x7f6dd01f1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:14.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.176+0000 7f6dd487f640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd01eea40 0x7f6dd01f1ba0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6dc401d0a0 tx=0x7f6dc40056a0 comp rx=0 tx=0).stop 2024-09-17T04:59:14.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.176+0000 7f6dd487f640 1 -- 172.21.15.150:0/3296554801 shutdown_connections 2024-09-17T04:59:14.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.176+0000 7f6dd487f640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6db0073200 0x7f6db0075710 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:14.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.176+0000 7f6dd487f640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6dd01f20e0 0x7f6dd014beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:14.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.176+0000 7f6dd487f640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dd01eea40 0x7f6dd01f1ba0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:14.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.176+0000 7f6dd487f640 1 --2- 172.21.15.150:0/3296554801 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6db8004810 0x7f6dd01e3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:14.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.176+0000 7f6dd487f640 1 -- 172.21.15.150:0/3296554801 >> 172.21.15.150:0/3296554801 conn(0x7f6dd00c4a00 msgr2=0x7f6dd00c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:14.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.176+0000 7f6dd487f640 1 -- 172.21.15.150:0/3296554801 shutdown_connections 2024-09-17T04:59:14.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:14.176+0000 7f6dd487f640 1 -- 172.21.15.150:0/3296554801 wait complete. 2024-09-17T04:59:14.195 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:59:16.196 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.364+0000 7f282f3f4640 1 -- 172.21.15.150:0/132572618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28280c35b0 msgr2=0x7f28280c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.364+0000 7f282f3f4640 1 --2- 172.21.15.150:0/132572618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28280c35b0 0x7f28280c8a80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f282808d8b0 tx=0x7f282401c890 comp rx=0 tx=0).stop 2024-09-17T04:59:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.364+0000 7f282f3f4640 1 -- 172.21.15.150:0/132572618 shutdown_connections 2024-09-17T04:59:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.364+0000 7f282f3f4640 1 --2- 172.21.15.150:0/132572618 >> v2:172.21.15.176:3300/0 conn(0x7f28280c8fc0 0x7f28280c20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.364+0000 7f282f3f4640 1 --2- 172.21.15.150:0/132572618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28280c35b0 0x7f28280c8a80 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.364+0000 7f282f3f4640 1 -- 172.21.15.150:0/132572618 >> 172.21.15.150:0/132572618 conn(0x7f2828074b80 msgr2=0x7f282806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.364+0000 7f282f3f4640 1 -- 172.21.15.150:0/132572618 shutdown_connections 2024-09-17T04:59:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.364+0000 7f282f3f4640 1 -- 172.21.15.150:0/132572618 wait complete. 2024-09-17T04:59:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282f3f4640 1 Processor -- start 2024-09-17T04:59:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282f3f4640 1 -- start start 2024-09-17T04:59:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282f3f4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28280c35b0 0x7f28281f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282f3f4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28280c8fc0 0x7f2828153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282f3f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28281568a0 0x7f2828153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282f3f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f28280c6b70 con 0x7f28281568a0 2024-09-17T04:59:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282f3f4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f28280c69f0 con 0x7f28280c8fc0 2024-09-17T04:59:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282f3f4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f28280b0fd0 con 0x7f28280c35b0 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282d991640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28281568a0 0x7f2828153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282d991640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28281568a0 0x7f2828153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58328/0 (socket says 172.21.15.150:58328) 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282d991640 1 -- 172.21.15.150:0/2061804315 learned_addr learned my addr 172.21.15.150:0/2061804315 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282d190640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28280c35b0 0x7f28281f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282c98f640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28280c8fc0 0x7f2828153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282d991640 1 -- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28280c35b0 msgr2=0x7f28281f3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282d991640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28280c35b0 0x7f28281f3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282d991640 1 -- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28280c8fc0 msgr2=0x7f2828153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282d991640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28280c8fc0 0x7f2828153040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282d991640 1 -- 172.21.15.150:0/2061804315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28281585b0 con 0x7f28281568a0 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282d991640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28281568a0 0x7f2828153580 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f282000ea80 tx=0x7f282000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f281dffb640 1 -- 172.21.15.150:0/2061804315 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2820004b30 con 0x7f28281568a0 2024-09-17T04:59:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f281dffb640 1 -- 172.21.15.150:0/2061804315 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2820004cf0 con 0x7f28281568a0 2024-09-17T04:59:16.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282f3f4640 1 -- 172.21.15.150:0/2061804315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28281564d0 con 0x7f28281568a0 2024-09-17T04:59:16.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f281dffb640 1 -- 172.21.15.150:0/2061804315 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2820005280 con 0x7f28281568a0 2024-09-17T04:59:16.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282f3f4640 1 -- 172.21.15.150:0/2061804315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28281f3030 con 0x7f28281568a0 2024-09-17T04:59:16.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.368+0000 7f282f3f4640 1 -- 172.21.15.150:0/2061804315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28280b14a0 con 0x7f28281568a0 2024-09-17T04:59:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.372+0000 7f281dffb640 1 -- 172.21.15.150:0/2061804315 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2820005420 con 0x7f28281568a0 2024-09-17T04:59:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.372+0000 7f281dffb640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f280c073200 0x7f280c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.372+0000 7f281dffb640 1 -- 172.21.15.150:0/2061804315 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f2820095a30 con 0x7f28281568a0 2024-09-17T04:59:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.372+0000 7f282d190640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f280c073200 0x7f280c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.372+0000 7f282d190640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f280c073200 0x7f280c075710 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2818001650 tx=0x7f2818009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.376+0000 7f281dffb640 1 -- 172.21.15.150:0/2061804315 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2820060480 con 0x7f28281568a0 2024-09-17T04:59:16.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.584+0000 7f282f3f4640 1 -- 172.21.15.150:0/2061804315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2828157790 con 0x7f28281568a0 2024-09-17T04:59:16.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.584+0000 7f281dffb640 1 -- 172.21.15.150:0/2061804315 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6536 (secure 0 0 0) 0x7f282005fd80 con 0x7f28281568a0 2024-09-17T04:59:16.584 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:16.584 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":76,"btime":"2024-09-17T04:58:56:975674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6075,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:53.961234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5258,"qdb_cluster":[5258]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:16.584 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 76 2024-09-17T04:59:16.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.588+0000 7f282f3f4640 1 -- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f280c073200 msgr2=0x7f280c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:16.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.588+0000 7f282f3f4640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f280c073200 0x7f280c075710 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2818001650 tx=0x7f2818009040 comp rx=0 tx=0).stop 2024-09-17T04:59:16.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.588+0000 7f282f3f4640 1 -- 172.21.15.150:0/2061804315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28281568a0 msgr2=0x7f2828153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:16.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.588+0000 7f282f3f4640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28281568a0 0x7f2828153580 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f282000ea80 tx=0x7f282000ef40 comp rx=0 tx=0).stop 2024-09-17T04:59:16.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.588+0000 7f282f3f4640 1 -- 172.21.15.150:0/2061804315 shutdown_connections 2024-09-17T04:59:16.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.588+0000 7f282f3f4640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f280c073200 0x7f280c075710 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:16.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.588+0000 7f282f3f4640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28280c8fc0 0x7f2828153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:16.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.588+0000 7f282f3f4640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28281568a0 0x7f2828153580 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:16.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.588+0000 7f282f3f4640 1 --2- 172.21.15.150:0/2061804315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28280c35b0 0x7f28281f3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:16.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.588+0000 7f282f3f4640 1 -- 172.21.15.150:0/2061804315 >> 172.21.15.150:0/2061804315 conn(0x7f2828074b80 msgr2=0x7f28281559e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:16.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.588+0000 7f282f3f4640 1 -- 172.21.15.150:0/2061804315 shutdown_connections 2024-09-17T04:59:16.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:16.588+0000 7f282f3f4640 1 -- 172.21.15.150:0/2061804315 wait complete. 2024-09-17T04:59:16.607 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:59:18.608 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:18.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.776+0000 7f4beb54d640 1 -- 172.21.15.150:0/2455295222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bd0004e70 msgr2=0x7f4bd00052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:18.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.776+0000 7f4beb54d640 1 --2- 172.21.15.150:0/2455295222 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4bd0004e70 0x7f4bd00052e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4be414dc30 tx=0x7f4bdc01caf0 comp rx=0 tx=0).stop 2024-09-17T04:59:18.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.776+0000 7f4be9aea640 1 -- 172.21.15.150:0/2455295222 reap_dead start 2024-09-17T04:59:18.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.776+0000 7f4beb54d640 1 -- 172.21.15.150:0/2455295222 shutdown_connections 2024-09-17T04:59:18.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.776+0000 7f4beb54d640 1 -- 172.21.15.150:0/2455295222 >> 172.21.15.150:0/2455295222 conn(0x7f4be40c4a00 msgr2=0x7f4be40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:18.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.776+0000 7f4beb54d640 1 -- 172.21.15.150:0/2455295222 shutdown_connections 2024-09-17T04:59:18.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.776+0000 7f4beb54d640 1 -- 172.21.15.150:0/2455295222 wait complete. 2024-09-17T04:59:18.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4beb54d640 1 Processor -- start 2024-09-17T04:59:18.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4beb54d640 1 -- start start 2024-09-17T04:59:18.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4beb54d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40aaba0 0x7f4be4146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:18.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4beb54d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be41f09d0 0x7f4be41f0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:18.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4beb54d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4be40a8e90 0x7f4be40a92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:18.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4beb54d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4be406f810 con 0x7f4be40aaba0 2024-09-17T04:59:18.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4beb54d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4be40adef0 con 0x7f4be40a8e90 2024-09-17T04:59:18.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4beb54d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4be4071260 con 0x7f4be41f09d0 2024-09-17T04:59:18.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4be92e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40aaba0 0x7f4be4146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:18.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4be92e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40aaba0 0x7f4be4146e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58344/0 (socket says 172.21.15.150:58344) 2024-09-17T04:59:18.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4be92e9640 1 -- 172.21.15.150:0/4289593022 learned_addr learned my addr 172.21.15.150:0/4289593022 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4be9aea640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4be40a8e90 0x7f4be40a92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4be8ae8640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be41f09d0 0x7f4be41f0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4be92e9640 1 -- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be41f09d0 msgr2=0x7f4be41f0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4be92e9640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be41f09d0 0x7f4be41f0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4be92e9640 1 -- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4be40a8e90 msgr2=0x7f4be40a92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4be92e9640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4be40a8e90 0x7f4be40a92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4be92e9640 1 -- 172.21.15.150:0/4289593022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be41df740 con 0x7f4be40aaba0 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4be9aea640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4be40a8e90 0x7f4be40a92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4be92e9640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40aaba0 0x7f4be4146e80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4be000e590 tx=0x7f4be000ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4bd9ffb640 1 -- 172.21.15.150:0/4289593022 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be0016070 con 0x7f4be40aaba0 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4bd9ffb640 1 -- 172.21.15.150:0/4289593022 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4be0012ae0 con 0x7f4be40aaba0 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4bd9ffb640 1 -- 172.21.15.150:0/4289593022 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be00113d0 con 0x7f4be40aaba0 2024-09-17T04:59:18.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4beb54d640 1 -- 172.21.15.150:0/4289593022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be41df9e0 con 0x7f4be40aaba0 2024-09-17T04:59:18.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4beb54d640 1 -- 172.21.15.150:0/4289593022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be40ac690 con 0x7f4be40aaba0 2024-09-17T04:59:18.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.780+0000 7f4beb54d640 1 -- 172.21.15.150:0/4289593022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be40a7300 con 0x7f4be40aaba0 2024-09-17T04:59:18.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.784+0000 7f4bd9ffb640 1 -- 172.21.15.150:0/4289593022 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4be000d070 con 0x7f4be40aaba0 2024-09-17T04:59:18.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.784+0000 7f4bd9ffb640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4bc8073230 0x7f4bc8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:18.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.784+0000 7f4bd9ffb640 1 -- 172.21.15.150:0/4289593022 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f4be0098210 con 0x7f4be40aaba0 2024-09-17T04:59:18.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.784+0000 7f4be8ae8640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4bc8073230 0x7f4bc8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:18.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.784+0000 7f4be8ae8640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4bc8073230 0x7f4bc8075740 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400cc30 tx=0x7f4bd4005f30 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:18.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.788+0000 7f4bd9ffb640 1 -- 172.21.15.150:0/4289593022 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4be0062b80 con 0x7f4be40aaba0 2024-09-17T04:59:18.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 -- 172.21.15.150:0/4289593022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4be40c60f0 con 0x7f4be40aaba0 2024-09-17T04:59:18.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4bd9ffb640 1 -- 172.21.15.150:0/4289593022 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+6536 (secure 0 0 0) 0x7f4be0062480 con 0x7f4be40aaba0 2024-09-17T04:59:18.994 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:18.995 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":76,"btime":"2024-09-17T04:58:56:975674+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6075,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:58:53.961234+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5258":{"gid":5258,"name":"d","rank":0,"incarnation":72,"state":"up:active","state_seq":7,"addr":"172.21.15.176:6837/713081612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":713081612},{"type":"v1","addr":"172.21.15.176:6837","nonce":713081612}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5258,"qdb_cluster":[5258]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:18.995 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 76 2024-09-17T04:59:18.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 -- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4bc8073230 msgr2=0x7f4bc8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4bc8073230 0x7f4bc8075740 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400cc30 tx=0x7f4bd4005f30 comp rx=0 tx=0).stop 2024-09-17T04:59:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 -- 172.21.15.150:0/4289593022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40aaba0 msgr2=0x7f4be4146e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40aaba0 0x7f4be4146e80 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4be000e590 tx=0x7f4be000ea50 comp rx=0 tx=0).stop 2024-09-17T04:59:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 -- 172.21.15.150:0/4289593022 shutdown_connections 2024-09-17T04:59:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4bc8073230 0x7f4bc8075740 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4be40a8e90 0x7f4be40a92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4be41f09d0 0x7f4be41f0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 --2- 172.21.15.150:0/4289593022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40aaba0 0x7f4be4146e80 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 -- 172.21.15.150:0/4289593022 >> 172.21.15.150:0/4289593022 conn(0x7f4be40c4a00 msgr2=0x7f4be40c5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 -- 172.21.15.150:0/4289593022 shutdown_connections 2024-09-17T04:59:18.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:18.996+0000 7f4beb54d640 1 -- 172.21.15.150:0/4289593022 wait complete. 2024-09-17T04:59:19.016 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T04:59:21.017 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:21.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.168+0000 7f6a85b30640 1 -- 172.21.15.150:0/2023265287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a800c3900 msgr2=0x7f6a80074920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:21.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.168+0000 7f6a85b30640 1 --2- 172.21.15.150:0/2023265287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a800c3900 0x7f6a80074920 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009840 tx=0x7f6a7401c7e0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.168+0000 7f6a85b30640 1 -- 172.21.15.150:0/2023265287 shutdown_connections 2024-09-17T04:59:21.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.168+0000 7f6a85b30640 1 --2- 172.21.15.150:0/2023265287 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a800c3900 0x7f6a80074920 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.168+0000 7f6a85b30640 1 --2- 172.21.15.150:0/2023265287 >> v2:172.21.15.176:3300/0 conn(0x7f6a800c4250 0x7f6a80153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.168+0000 7f6a85b30640 1 -- 172.21.15.150:0/2023265287 >> 172.21.15.150:0/2023265287 conn(0x7f6a8006e360 msgr2=0x7f6a8006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.168+0000 7f6a85b30640 1 -- 172.21.15.150:0/2023265287 shutdown_connections 2024-09-17T04:59:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.168+0000 7f6a85b30640 1 -- 172.21.15.150:0/2023265287 wait complete. 2024-09-17T04:59:21.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a85b30640 1 Processor -- start 2024-09-17T04:59:21.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a85b30640 1 -- start start 2024-09-17T04:59:21.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a85b30640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a80074e60 0x7f6a801efda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:21.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a85b30640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a800c3900 0x7f6a801592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:21.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a85b30640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6a800c4250 0x7f6a8014ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:21.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a85b30640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6a80072f30 con 0x7f6a800c3900 2024-09-17T04:59:21.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a85b30640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6a80072db0 con 0x7f6a80074e60 2024-09-17T04:59:21.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a85b30640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6a80160f90 con 0x7f6a800c4250 2024-09-17T04:59:21.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a7effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a800c3900 0x7f6a801592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a7effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a800c3900 0x7f6a801592e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47046/0 (socket says 172.21.15.150:47046) 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a7effd640 1 -- 172.21.15.150:0/3727132981 learned_addr learned my addr 172.21.15.150:0/3727132981 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a7f7fe640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a80074e60 0x7f6a801efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a7ffff640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6a800c4250 0x7f6a8014ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a7effd640 1 -- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6a800c4250 msgr2=0x7f6a8014ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a7effd640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6a800c4250 0x7f6a8014ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a7effd640 1 -- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a80074e60 msgr2=0x7f6a801efda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a7effd640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a80074e60 0x7f6a801efda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a7effd640 1 -- 172.21.15.150:0/3727132981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a8015a000 con 0x7f6a800c3900 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a7f7fe640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a80074e60 0x7f6a801efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a7effd640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a800c3900 0x7f6a801592e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009790 tx=0x7f6a740a6670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a67fff640 1 -- 172.21.15.150:0/3727132981 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a740a6a80 con 0x7f6a800c3900 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a67fff640 1 -- 172.21.15.150:0/3727132981 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a740a6d10 con 0x7f6a800c3900 2024-09-17T04:59:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a85b30640 1 -- 172.21.15.150:0/3727132981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a8015a210 con 0x7f6a800c3900 2024-09-17T04:59:21.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a67fff640 1 -- 172.21.15.150:0/3727132981 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a74004040 con 0x7f6a800c3900 2024-09-17T04:59:21.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.172+0000 7f6a85b30640 1 -- 172.21.15.150:0/3727132981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a80157e60 con 0x7f6a800c3900 2024-09-17T04:59:21.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.176+0000 7f6a85b30640 1 -- 172.21.15.150:0/3727132981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a801f0850 con 0x7f6a800c3900 2024-09-17T04:59:21.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.176+0000 7f6a67fff640 1 -- 172.21.15.150:0/3727132981 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6a740c1030 con 0x7f6a800c3900 2024-09-17T04:59:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.176+0000 7f6a67fff640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6a6c073200 0x7f6a6c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.176+0000 7f6a67fff640 1 -- 172.21.15.150:0/3727132981 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f6a7412ec00 con 0x7f6a800c3900 2024-09-17T04:59:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.176+0000 7f6a7f7fe640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6a6c073200 0x7f6a6c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.176+0000 7f6a7f7fe640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6a6c073200 0x7f6a6c075710 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000bbd0 tx=0x7f6a7000b5b0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:21.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.180+0000 7f6a67fff640 1 -- 172.21.15.150:0/3727132981 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6a74135080 con 0x7f6a800c3900 2024-09-17T04:59:21.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.400+0000 7f6a85b30640 1 -- 172.21.15.150:0/3727132981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a80074920 con 0x7f6a800c3900 2024-09-17T04:59:21.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a67fff640 1 -- 172.21.15.150:0/3727132981 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5704 (secure 0 0 0) 0x7f6a740c8060 con 0x7f6a800c3900 2024-09-17T04:59:21.402 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:21.402 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":77,"btime":"2024-09-17T04:59:20:945127+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:20.945117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:21.402 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 77 2024-09-17T04:59:21.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a85b30640 1 -- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6a6c073200 msgr2=0x7f6a6c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:21.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a85b30640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6a6c073200 0x7f6a6c075710 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000bbd0 tx=0x7f6a7000b5b0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a85b30640 1 -- 172.21.15.150:0/3727132981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a800c3900 msgr2=0x7f6a801592e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:21.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a85b30640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a800c3900 0x7f6a801592e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009790 tx=0x7f6a740a6670 comp rx=0 tx=0).stop 2024-09-17T04:59:21.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a85b30640 1 -- 172.21.15.150:0/3727132981 shutdown_connections 2024-09-17T04:59:21.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a85b30640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6a6c073200 0x7f6a6c075710 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a85b30640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6a800c4250 0x7f6a8014ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a85b30640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a800c3900 0x7f6a801592e0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a85b30640 1 --2- 172.21.15.150:0/3727132981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a80074e60 0x7f6a801efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a85b30640 1 -- 172.21.15.150:0/3727132981 >> 172.21.15.150:0/3727132981 conn(0x7f6a8006e360 msgr2=0x7f6a8015e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:21.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a85b30640 1 -- 172.21.15.150:0/3727132981 shutdown_connections 2024-09-17T04:59:21.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.404+0000 7f6a85b30640 1 -- 172.21.15.150:0/3727132981 wait complete. 2024-09-17T04:59:21.424 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-17T04:59:21.425 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T04:59:21.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:21.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.584+0000 7fb9c139a640 1 -- 172.21.15.150:0/768858774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc156820 msgr2=0x7fb9bc0c69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:21.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.584+0000 7fb9c139a640 1 --2- 172.21.15.150:0/768858774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc156820 0x7fb9bc0c69f0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0c1690 tx=0x7fb9ac01c8a0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.584+0000 7fb9c139a640 1 -- 172.21.15.150:0/768858774 shutdown_connections 2024-09-17T04:59:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.584+0000 7fb9c139a640 1 --2- 172.21.15.150:0/768858774 >> v2:172.21.15.176:3300/0 conn(0x7fb9bc0c2780 0x7fb9bc0c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.584+0000 7fb9c139a640 1 --2- 172.21.15.150:0/768858774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9bc1506d0 0x7fb9bc0c2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.584+0000 7fb9c139a640 1 --2- 172.21.15.150:0/768858774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc156820 0x7fb9bc0c69f0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.584+0000 7fb9c139a640 1 -- 172.21.15.150:0/768858774 >> 172.21.15.150:0/768858774 conn(0x7fb9bc074b80 msgr2=0x7fb9bc06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.584+0000 7fb9c139a640 1 -- 172.21.15.150:0/768858774 shutdown_connections 2024-09-17T04:59:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.584+0000 7fb9c139a640 1 -- 172.21.15.150:0/768858774 wait complete. 2024-09-17T04:59:21.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9c139a640 1 Processor -- start 2024-09-17T04:59:21.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9c139a640 1 -- start start 2024-09-17T04:59:21.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9c139a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc0c2780 0x7fb9bc1490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:21.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9c139a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9bc1506d0 0x7fb9bc153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:21.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9c139a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9bc156820 0x7fb9bc153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:21.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9c139a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc06ff00 con 0x7fb9bc0c2780 2024-09-17T04:59:21.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9c139a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc06fd80 con 0x7fb9bc1506d0 2024-09-17T04:59:21.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9c139a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb9bc0c5790 con 0x7fb9bc156820 2024-09-17T04:59:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9baffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc0c2780 0x7fb9bc1490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9baffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc0c2780 0x7fb9bc1490a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47072/0 (socket says 172.21.15.150:47072) 2024-09-17T04:59:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9baffd640 1 -- 172.21.15.150:0/2131546745 learned_addr learned my addr 172.21.15.150:0/2131546745 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9ba7fc640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9bc1506d0 0x7fb9bc153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9bb7fe640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9bc156820 0x7fb9bc153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9baffd640 1 -- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9bc156820 msgr2=0x7fb9bc153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9baffd640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9bc156820 0x7fb9bc153c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9baffd640 1 -- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9bc1506d0 msgr2=0x7fb9bc153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9baffd640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9bc1506d0 0x7fb9bc153700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9baffd640 1 -- 172.21.15.150:0/2131546745 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc147fb0 con 0x7fb9bc0c2780 2024-09-17T04:59:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9baffd640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc0c2780 0x7fb9bc1490a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac01af50 tx=0x7fb9ac0a55a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9a37fe640 1 -- 172.21.15.150:0/2131546745 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac0b2040 con 0x7fb9bc0c2780 2024-09-17T04:59:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9a37fe640 1 -- 172.21.15.150:0/2131546745 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9ac0043b0 con 0x7fb9bc0c2780 2024-09-17T04:59:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9c139a640 1 -- 172.21.15.150:0/2131546745 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9bc149d60 con 0x7fb9bc0c2780 2024-09-17T04:59:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9a37fe640 1 -- 172.21.15.150:0/2131546745 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ac003860 con 0x7fb9bc0c2780 2024-09-17T04:59:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9c139a640 1 -- 172.21.15.150:0/2131546745 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9bc152da0 con 0x7fb9bc0c2780 2024-09-17T04:59:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.588+0000 7fb9c139a640 1 -- 172.21.15.150:0/2131546745 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9bc0c5ce0 con 0x7fb9bc0c2780 2024-09-17T04:59:21.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.592+0000 7fb9a37fe640 1 -- 172.21.15.150:0/2131546745 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb9ac0bf060 con 0x7fb9bc0c2780 2024-09-17T04:59:21.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.592+0000 7fb9a37fe640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb9a8073230 0x7fb9a8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:21.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.592+0000 7fb9a37fe640 1 -- 172.21.15.150:0/2131546745 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7fb9ac12e4c0 con 0x7fb9bc0c2780 2024-09-17T04:59:21.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.592+0000 7fb9ba7fc640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb9a8073230 0x7fb9a8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:21.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.592+0000 7fb9ba7fc640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb9a8073230 0x7fb9a8075740 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0005970 tx=0x7fb9b0005900 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:21.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.596+0000 7fb9a37fe640 1 -- 172.21.15.150:0/2131546745 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb9ac016490 con 0x7fb9bc0c2780 2024-09-17T04:59:21.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.784+0000 7fb9c139a640 1 -- 172.21.15.150:0/2131546745 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9bc0b14f0 con 0x7fb9bc0c2780 2024-09-17T04:59:21.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.784+0000 7fb9a37fe640 1 -- 172.21.15.150:0/2131546745 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+5704 (secure 0 0 0) 0x7fb9ac0f8d60 con 0x7fb9bc0c2780 2024-09-17T04:59:21.785 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:21.785 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":77,"btime":"2024-09-17T04:59:20:945127+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:20.945117+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:21.785 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 77 2024-09-17T04:59:21.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.788+0000 7fb9c139a640 1 -- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb9a8073230 msgr2=0x7fb9a8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:21.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.788+0000 7fb9c139a640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb9a8073230 0x7fb9a8075740 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0005970 tx=0x7fb9b0005900 comp rx=0 tx=0).stop 2024-09-17T04:59:21.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.788+0000 7fb9c139a640 1 -- 172.21.15.150:0/2131546745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc0c2780 msgr2=0x7fb9bc1490a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:21.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.788+0000 7fb9c139a640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc0c2780 0x7fb9bc1490a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac01af50 tx=0x7fb9ac0a55a0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.788+0000 7fb9c139a640 1 -- 172.21.15.150:0/2131546745 shutdown_connections 2024-09-17T04:59:21.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.788+0000 7fb9c139a640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb9a8073230 0x7fb9a8075740 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.788+0000 7fb9c139a640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9bc1506d0 0x7fb9bc153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.788+0000 7fb9c139a640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb9bc156820 0x7fb9bc153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.788+0000 7fb9c139a640 1 --2- 172.21.15.150:0/2131546745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9bc0c2780 0x7fb9bc1490a0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:21.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.788+0000 7fb9c139a640 1 -- 172.21.15.150:0/2131546745 >> 172.21.15.150:0/2131546745 conn(0x7fb9bc074b80 msgr2=0x7fb9bc1554f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:21.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.788+0000 7fb9c139a640 1 -- 172.21.15.150:0/2131546745 shutdown_connections 2024-09-17T04:59:21.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:21.788+0000 7fb9c139a640 1 -- 172.21.15.150:0/2131546745 wait complete. 2024-09-17T04:59:21.806 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T04:59:20:945127+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 77, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 77, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6075": { "addr": "172.21.15.150:6833/2313391683", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2313391683, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 2313391683, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6075, "incarnation": 77, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:59:20.945117+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6075 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 70, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5901": { "addr": "172.21.15.150:6835/2081398022", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 2081398022, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2081398022, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5901, "incarnation": 67, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T04:58:33.945203+0000", "qdb_cluster": [ 5901 ], "qdb_leader": 5901, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5901 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T04:59:23.807 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:23.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.980+0000 7ff18e844640 1 -- 172.21.15.150:0/861913209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1881505a0 msgr2=0x7ff188152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.980+0000 7ff18e844640 1 --2- 172.21.15.150:0/861913209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1881505a0 0x7ff188152cf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff180009bc0 tx=0x7ff18001cba0 comp rx=0 tx=0).stop 2024-09-17T04:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.980+0000 7ff18e844640 1 -- 172.21.15.150:0/861913209 shutdown_connections 2024-09-17T04:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.980+0000 7ff18e844640 1 --2- 172.21.15.150:0/861913209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1881505a0 0x7ff188152cf0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.980+0000 7ff18e844640 1 --2- 172.21.15.150:0/861913209 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff1880a7350 0x7ff188150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.980+0000 7ff18e844640 1 -- 172.21.15.150:0/861913209 >> 172.21.15.150:0/861913209 conn(0x7ff1880c4a00 msgr2=0x7ff1880c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.980+0000 7ff18e844640 1 -- 172.21.15.150:0/861913209 shutdown_connections 2024-09-17T04:59:23.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.980+0000 7ff18e844640 1 -- 172.21.15.150:0/861913209 wait complete. 2024-09-17T04:59:23.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.980+0000 7ff18e844640 1 Processor -- start 2024-09-17T04:59:23.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff18e844640 1 -- start start 2024-09-17T04:59:23.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff18e844640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1880a7350 0x7ff1881f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:23.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff18e844640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff1880c42e0 0x7ff1880c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff18e844640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1881505a0 0x7ff188144fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff18e844640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff1881537c0 con 0x7ff1880a7350 2024-09-17T04:59:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff18e844640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff188153640 con 0x7ff1881505a0 2024-09-17T04:59:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff18e844640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff1881619a0 con 0x7ff1880c42e0 2024-09-17T04:59:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff187fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1880a7350 0x7ff1881f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff187fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1880a7350 0x7ff1881f3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47080/0 (socket says 172.21.15.150:47080) 2024-09-17T04:59:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff187fff640 1 -- 172.21.15.150:0/360199216 learned_addr learned my addr 172.21.15.150:0/360199216 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff1877fe640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff1880c42e0 0x7ff1880c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff18cde1640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1881505a0 0x7ff188144fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff187fff640 1 -- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff1880c42e0 msgr2=0x7ff1880c1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff187fff640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff1880c42e0 0x7ff1880c1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff187fff640 1 -- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1881505a0 msgr2=0x7ff188144fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:23.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff187fff640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1881505a0 0x7ff188144fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:23.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff187fff640 1 -- 172.21.15.150:0/360199216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff188147970 con 0x7ff1880a7350 2024-09-17T04:59:23.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff187fff640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1880a7350 0x7ff1881f3440 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff17000c400 tx=0x7ff17000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:23.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff184ff9640 1 -- 172.21.15.150:0/360199216 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff170011790 con 0x7ff1880a7350 2024-09-17T04:59:23.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff18e844640 1 -- 172.21.15.150:0/360199216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff188147c50 con 0x7ff1880a7350 2024-09-17T04:59:23.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff184ff9640 1 -- 172.21.15.150:0/360199216 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff170017070 con 0x7ff1880a7350 2024-09-17T04:59:23.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff184ff9640 1 -- 172.21.15.150:0/360199216 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff170011cc0 con 0x7ff1880a7350 2024-09-17T04:59:23.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff18e844640 1 -- 172.21.15.150:0/360199216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff188161180 con 0x7ff1880a7350 2024-09-17T04:59:23.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff18e844640 1 -- 172.21.15.150:0/360199216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff188150060 con 0x7ff1880a7350 2024-09-17T04:59:23.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.984+0000 7ff184ff9640 1 -- 172.21.15.150:0/360199216 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff17001c460 con 0x7ff1880a7350 2024-09-17T04:59:23.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.988+0000 7ff184ff9640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff168073230 0x7ff168075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:23.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.988+0000 7ff184ff9640 1 -- 172.21.15.150:0/360199216 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7ff1700948a0 con 0x7ff1880a7350 2024-09-17T04:59:23.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.988+0000 7ff1877fe640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff168073230 0x7ff168075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:23.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.988+0000 7ff1877fe640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff168073230 0x7ff168075740 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff188074eb0 tx=0x7ff1780091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:23.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:23.992+0000 7ff184ff9640 1 -- 172.21.15.150:0/360199216 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff17005f020 con 0x7ff1880a7350 2024-09-17T04:59:24.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.224+0000 7ff18e844640 1 -- 172.21.15.150:0/360199216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1880c68e0 con 0x7ff1880a7350 2024-09-17T04:59:24.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.224+0000 7ff184ff9640 1 -- 172.21.15.150:0/360199216 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5712 (secure 0 0 0) 0x7ff17005e920 con 0x7ff1880a7350 2024-09-17T04:59:24.224 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:24.224 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":80,"btime":"2024-09-17T04:59:24:099270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:24.224 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 80 2024-09-17T04:59:24.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.228+0000 7ff18e844640 1 -- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff168073230 msgr2=0x7ff168075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:24.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.228+0000 7ff18e844640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff168073230 0x7ff168075740 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff188074eb0 tx=0x7ff1780091e0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.228+0000 7ff18e844640 1 -- 172.21.15.150:0/360199216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1880a7350 msgr2=0x7ff1881f3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:24.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.228+0000 7ff18e844640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1880a7350 0x7ff1881f3440 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff17000c400 tx=0x7ff17000c8c0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.228+0000 7ff18e844640 1 -- 172.21.15.150:0/360199216 shutdown_connections 2024-09-17T04:59:24.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.228+0000 7ff18e844640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff168073230 0x7ff168075740 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.228+0000 7ff18e844640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff1881505a0 0x7ff188144fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.228+0000 7ff18e844640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff1880c42e0 0x7ff1880c1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.228+0000 7ff18e844640 1 --2- 172.21.15.150:0/360199216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1880a7350 0x7ff1881f3440 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.228+0000 7ff18e844640 1 -- 172.21.15.150:0/360199216 >> 172.21.15.150:0/360199216 conn(0x7ff1880c4a00 msgr2=0x7ff1881539a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:24.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.228+0000 7ff18e844640 1 -- 172.21.15.150:0/360199216 shutdown_connections 2024-09-17T04:59:24.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.228+0000 7ff18e844640 1 -- 172.21.15.150:0/360199216 wait complete. 2024-09-17T04:59:24.257 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=5258 2024-09-17T04:59:24.258 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.d 2024-09-17T04:59:24.258 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-17T04:59:24.258 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:59:24.258 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:59:24.261 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:59:24.261 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:24.368 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-09-17T04:59:24.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.416+0000 7f8c533b5640 1 -- 172.21.15.150:0/1904663900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c4c0c3900 msgr2=0x7f8c4c074920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:24.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.416+0000 7f8c533b5640 1 --2- 172.21.15.150:0/1904663900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c4c0c3900 0x7f8c4c074920 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c009870 tx=0x7f8c3c01c820 comp rx=0 tx=0).stop 2024-09-17T04:59:24.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.416+0000 7f8c533b5640 1 -- 172.21.15.150:0/1904663900 shutdown_connections 2024-09-17T04:59:24.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.416+0000 7f8c533b5640 1 --2- 172.21.15.150:0/1904663900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c4c0c3900 0x7f8c4c074920 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.416+0000 7f8c533b5640 1 -- 172.21.15.150:0/1904663900 >> 172.21.15.150:0/1904663900 conn(0x7f8c4c06e360 msgr2=0x7f8c4c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:24.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.416+0000 7f8c533b5640 1 -- 172.21.15.150:0/1904663900 shutdown_connections 2024-09-17T04:59:24.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.416+0000 7f8c533b5640 1 -- 172.21.15.150:0/1904663900 wait complete. 2024-09-17T04:59:24.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c533b5640 1 Processor -- start 2024-09-17T04:59:24.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c533b5640 1 -- start start 2024-09-17T04:59:24.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c533b5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c4c074e60 0x7f8c4c158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:24.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c533b5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c4c0c3900 0x7f8c4c0c2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:24.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c533b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c0c4250 0x7f8c4c1f03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:24.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c533b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c161eb0 con 0x7f8c4c0c4250 2024-09-17T04:59:24.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c533b5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c072fc0 con 0x7f8c4c0c3900 2024-09-17T04:59:24.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c533b5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8c4c162030 con 0x7f8c4c074e60 2024-09-17T04:59:24.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c51952640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c0c4250 0x7f8c4c1f03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:24.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c50950640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c4c0c3900 0x7f8c4c0c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c51952640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c0c4250 0x7f8c4c1f03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47096/0 (socket says 172.21.15.150:47096) 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c51952640 1 -- 172.21.15.150:0/4024418209 learned_addr learned my addr 172.21.15.150:0/4024418209 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c51151640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c4c074e60 0x7f8c4c158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c51952640 1 -- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c4c074e60 msgr2=0x7f8c4c158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c51952640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c4c074e60 0x7f8c4c158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c51952640 1 -- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c4c0c3900 msgr2=0x7f8c4c0c2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c51952640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c4c0c3900 0x7f8c4c0c2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c51952640 1 -- 172.21.15.150:0/4024418209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c0a8da0 con 0x7f8c4c0c4250 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c51952640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c0c4250 0x7f8c4c1f03e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f8c4400c760 tx=0x7f8c4400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c41ffb640 1 -- 172.21.15.150:0/4024418209 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44003db0 con 0x7f8c4c0c4250 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c41ffb640 1 -- 172.21.15.150:0/4024418209 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c4400ce10 con 0x7f8c4c0c4250 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c533b5640 1 -- 172.21.15.150:0/4024418209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c4c1ef920 con 0x7f8c4c0c4250 2024-09-17T04:59:24.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c41ffb640 1 -- 172.21.15.150:0/4024418209 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c440052d0 con 0x7f8c4c0c4250 2024-09-17T04:59:24.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c533b5640 1 -- 172.21.15.150:0/4024418209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c4c15f1a0 con 0x7f8c4c0c4250 2024-09-17T04:59:24.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.420+0000 7f8c533b5640 1 -- 172.21.15.150:0/4024418209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c4c15ec30 con 0x7f8c4c0c4250 2024-09-17T04:59:24.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.424+0000 7f8c41ffb640 1 -- 172.21.15.150:0/4024418209 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8c44005470 con 0x7f8c4c0c4250 2024-09-17T04:59:24.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.424+0000 7f8c41ffb640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c30073200 0x7f8c30075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:24.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.424+0000 7f8c41ffb640 1 -- 172.21.15.150:0/4024418209 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f8c44094610 con 0x7f8c4c0c4250 2024-09-17T04:59:24.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.424+0000 7f8c51151640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c30073200 0x7f8c30075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:24.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.424+0000 7f8c51151640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c30073200 0x7f8c30075710 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8c48001870 tx=0x7f8c48001380 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:24.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.428+0000 7f8c41ffb640 1 -- 172.21.15.150:0/4024418209 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8c4405e8e0 con 0x7f8c4c0c4250 2024-09-17T04:59:24.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.680+0000 7f8c533b5640 1 -- 172.21.15.150:0/4024418209 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c4c0c2760 con 0x7f8c4c0c4250 2024-09-17T04:59:24.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.680+0000 7f8c41ffb640 1 -- 172.21.15.150:0/4024418209 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5712 (secure 0 0 0) 0x7f8c4c0c2760 con 0x7f8c4c0c4250 2024-09-17T04:59:24.679 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:24.679 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":80,"btime":"2024-09-17T04:59:24:099270+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:24.679 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 80 2024-09-17T04:59:24.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.684+0000 7f8c533b5640 1 -- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c30073200 msgr2=0x7f8c30075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:24.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.684+0000 7f8c533b5640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c30073200 0x7f8c30075710 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8c48001870 tx=0x7f8c48001380 comp rx=0 tx=0).stop 2024-09-17T04:59:24.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.684+0000 7f8c533b5640 1 -- 172.21.15.150:0/4024418209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c0c4250 msgr2=0x7f8c4c1f03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:24.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.684+0000 7f8c533b5640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c0c4250 0x7f8c4c1f03e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f8c4400c760 tx=0x7f8c4400cc20 comp rx=0 tx=0).stop 2024-09-17T04:59:24.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.684+0000 7f8c533b5640 1 -- 172.21.15.150:0/4024418209 shutdown_connections 2024-09-17T04:59:24.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.684+0000 7f8c533b5640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c30073200 0x7f8c30075710 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.684+0000 7f8c533b5640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c4c0c3900 0x7f8c4c0c2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.684+0000 7f8c533b5640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c4c0c4250 0x7f8c4c1f03e0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.684+0000 7f8c533b5640 1 --2- 172.21.15.150:0/4024418209 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c4c074e60 0x7f8c4c158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:24.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.684+0000 7f8c533b5640 1 -- 172.21.15.150:0/4024418209 >> 172.21.15.150:0/4024418209 conn(0x7f8c4c06e360 msgr2=0x7f8c4c0c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:24.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.684+0000 7f8c533b5640 1 -- 172.21.15.150:0/4024418209 shutdown_connections 2024-09-17T04:59:24.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:24.684+0000 7f8c533b5640 1 -- 172.21.15.150:0/4024418209 wait complete. 2024-09-17T04:59:24.701 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-17T04:59:26.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:59:26.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.872+0000 7fc5b60aa640 1 -- 172.21.15.150:0/625438578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b00c3b40 msgr2=0x7fc5b0157c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:26.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.872+0000 7fc5b60aa640 1 --2- 172.21.15.150:0/625438578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b00c3b40 0x7fc5b0157c30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc598009840 tx=0x7fc59801c820 comp rx=0 tx=0).stop 2024-09-17T04:59:26.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.872+0000 7fc5b60aa640 1 -- 172.21.15.150:0/625438578 shutdown_connections 2024-09-17T04:59:26.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.872+0000 7fc5b60aa640 1 --2- 172.21.15.150:0/625438578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b00c3b40 0x7fc5b0157c30 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:26.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.872+0000 7fc5b60aa640 1 -- 172.21.15.150:0/625438578 >> 172.21.15.150:0/625438578 conn(0x7fc5b00c4a00 msgr2=0x7fc5b00c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:26.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.872+0000 7fc5b60aa640 1 -- 172.21.15.150:0/625438578 shutdown_connections 2024-09-17T04:59:26.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.872+0000 7fc5b60aa640 1 -- 172.21.15.150:0/625438578 wait complete. 2024-09-17T04:59:26.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.872+0000 7fc5b60aa640 1 Processor -- start 2024-09-17T04:59:26.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.872+0000 7fc5b60aa640 1 -- start start 2024-09-17T04:59:26.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5b60aa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5b00c3b40 0x7fc5b0074d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:26.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5b60aa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5b0151d00 0x7fc5b014d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:26.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5b60aa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b0158170 0x7fc5b00c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:26.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5b60aa640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b00c4630 con 0x7fc5b0158170 2024-09-17T04:59:26.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5b60aa640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b00c44b0 con 0x7fc5b00c3b40 2024-09-17T04:59:26.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5b60aa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b00c3f50 con 0x7fc5b0151d00 2024-09-17T04:59:26.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5affff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b0158170 0x7fc5b00c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5affff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b0158170 0x7fc5b00c1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47114/0 (socket says 172.21.15.150:47114) 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5affff640 1 -- 172.21.15.150:0/376780503 learned_addr learned my addr 172.21.15.150:0/376780503 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5af7fe640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5b00c3b40 0x7fc5b0074d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5a6ffd640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5b0151d00 0x7fc5b014d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5affff640 1 -- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5b0151d00 msgr2=0x7fc5b014d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5affff640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5b0151d00 0x7fc5b014d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5affff640 1 -- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5b00c3b40 msgr2=0x7fc5b0074d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5affff640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5b00c3b40 0x7fc5b0074d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5affff640 1 -- 172.21.15.150:0/376780503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b01600f0 con 0x7fc5b0158170 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5affff640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b0158170 0x7fc5b00c1330 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800c760 tx=0x7fc5a800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5acff9640 1 -- 172.21.15.150:0/376780503 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a8003db0 con 0x7fc5b0158170 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5acff9640 1 -- 172.21.15.150:0/376780503 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5a800ce10 con 0x7fc5b0158170 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5b60aa640 1 -- 172.21.15.150:0/376780503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b014da10 con 0x7fc5b0158170 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5acff9640 1 -- 172.21.15.150:0/376780503 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a80052d0 con 0x7fc5b0158170 2024-09-17T04:59:26.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5b60aa640 1 -- 172.21.15.150:0/376780503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b00740a0 con 0x7fc5b0158170 2024-09-17T04:59:26.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.876+0000 7fc5b60aa640 1 -- 172.21.15.150:0/376780503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5b0157c30 con 0x7fc5b0158170 2024-09-17T04:59:26.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.880+0000 7fc5acff9640 1 -- 172.21.15.150:0/376780503 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc5a8005470 con 0x7fc5b0158170 2024-09-17T04:59:26.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.880+0000 7fc5acff9640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc590073200 0x7fc590075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:59:26.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.880+0000 7fc5acff9640 1 -- 172.21.15.150:0/376780503 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7fc5a8094560 con 0x7fc5b0158170 2024-09-17T04:59:26.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.880+0000 7fc5af7fe640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc590073200 0x7fc590075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:59:26.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.880+0000 7fc5af7fe640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc590073200 0x7fc590075710 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0007fd0 tx=0x7fc5a0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:59:26.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:26.884+0000 7fc5acff9640 1 -- 172.21.15.150:0/376780503 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc5a805e830 con 0x7fc5b0158170 2024-09-17T04:59:26.988 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T04:59:26.988+0000 7f2ce6377640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:59:27.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.144+0000 7fc5b60aa640 1 -- 172.21.15.150:0/376780503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5b00c6380 con 0x7fc5b0158170 2024-09-17T04:59:27.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5acff9640 1 -- 172.21.15.150:0/376780503 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+6537 (secure 0 0 0) 0x7fc5a8063260 con 0x7fc5b0158170 2024-09-17T04:59:27.145 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T04:59:27.145 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":81,"btime":"2024-09-17T04:59:25:107413+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T04:59:27.145 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 81 2024-09-17T04:59:27.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5b60aa640 1 -- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc590073200 msgr2=0x7fc590075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:27.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5b60aa640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc590073200 0x7fc590075710 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0007fd0 tx=0x7fc5a0009040 comp rx=0 tx=0).stop 2024-09-17T04:59:27.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5b60aa640 1 -- 172.21.15.150:0/376780503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b0158170 msgr2=0x7fc5b00c1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:59:27.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5b60aa640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b0158170 0x7fc5b00c1330 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800c760 tx=0x7fc5a800cc20 comp rx=0 tx=0).stop 2024-09-17T04:59:27.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5b60aa640 1 -- 172.21.15.150:0/376780503 shutdown_connections 2024-09-17T04:59:27.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5b60aa640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc590073200 0x7fc590075710 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:27.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5b60aa640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5b0158170 0x7fc5b00c1330 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:27.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5b60aa640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5b0151d00 0x7fc5b014d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:27.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5b60aa640 1 --2- 172.21.15.150:0/376780503 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5b00c3b40 0x7fc5b0074d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:59:27.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5b60aa640 1 -- 172.21.15.150:0/376780503 >> 172.21.15.150:0/376780503 conn(0x7fc5b00c4a00 msgr2=0x7fc5b00c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:59:27.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5b60aa640 1 -- 172.21.15.150:0/376780503 shutdown_connections 2024-09-17T04:59:27.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T04:59:27.148+0000 7fc5b60aa640 1 -- 172.21.15.150:0/376780503 wait complete. 2024-09-17T04:59:27.169 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-17T04:59:27.170 INFO:tasks.mds_thrash.fs.[a]:waiting for 63 secs before thrashing 2024-09-17T04:59:30.826 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:59:30.829 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:00:00.949 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:00:00.952 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:00:05.460 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:05.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.636+0000 7f626d437640 1 -- 172.21.15.150:0/1680721651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6268073480 msgr2=0x7f62680c7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:05.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.636+0000 7f626d437640 1 --2- 172.21.15.150:0/1680721651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6268073480 0x7f62680c7180 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6260009c80 tx=0x7f626001cc80 comp rx=0 tx=0).stop 2024-09-17T05:00:05.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.636+0000 7f626d437640 1 -- 172.21.15.150:0/1680721651 shutdown_connections 2024-09-17T05:00:05.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.636+0000 7f626d437640 1 --2- 172.21.15.150:0/1680721651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6268073480 0x7f62680c7180 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:05.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.636+0000 7f626d437640 1 --2- 172.21.15.150:0/1680721651 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62680c47e0 0x7f626814b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:05.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.636+0000 7f626d437640 1 -- 172.21.15.150:0/1680721651 >> 172.21.15.150:0/1680721651 conn(0x7f62680abf10 msgr2=0x7f62680ac320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:05.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.636+0000 7f626d437640 1 -- 172.21.15.150:0/1680721651 shutdown_connections 2024-09-17T05:00:05.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.636+0000 7f626d437640 1 -- 172.21.15.150:0/1680721651 wait complete. 2024-09-17T05:00:05.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f626d437640 1 Processor -- start 2024-09-17T05:00:05.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f626d437640 1 -- start start 2024-09-17T05:00:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f626d437640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6268073480 0x7f62681f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f626d437640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62680c6d70 0x7f62680af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f626d437640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f626814ec20 0x7f62681506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f626d437640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f62680c9ce0 con 0x7f6268073480 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f626d437640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f62680c9b60 con 0x7f626814ec20 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f626d437640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f62681617c0 con 0x7f62680c6d70 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f6266ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6268073480 0x7f62681f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f6266ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6268073480 0x7f62681f13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59430/0 (socket says 172.21.15.150:59430) 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f6266ffd640 1 -- 172.21.15.150:0/2770132278 learned_addr learned my addr 172.21.15.150:0/2770132278 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f625ffff640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62680c6d70 0x7f62680af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f62677fe640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f626814ec20 0x7f62681506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f6266ffd640 1 -- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62680c6d70 msgr2=0x7f62680af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f6266ffd640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62680c6d70 0x7f62680af390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f6266ffd640 1 -- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f626814ec20 msgr2=0x7f62681506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f6266ffd640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f626814ec20 0x7f62681506a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.640+0000 7f6266ffd640 1 -- 172.21.15.150:0/2770132278 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6268146a80 con 0x7f6268073480 2024-09-17T05:00:05.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.644+0000 7f6266ffd640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6268073480 0x7f62681f13c0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f625800d750 tx=0x7f625800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:05.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.644+0000 7f625f7fe640 1 -- 172.21.15.150:0/2770132278 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258017070 con 0x7f6268073480 2024-09-17T05:00:05.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.644+0000 7f625f7fe640 1 -- 172.21.15.150:0/2770132278 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6258009d20 con 0x7f6268073480 2024-09-17T05:00:05.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.644+0000 7f626d437640 1 -- 172.21.15.150:0/2770132278 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62681f0bc0 con 0x7f6268073480 2024-09-17T05:00:05.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.644+0000 7f625f7fe640 1 -- 172.21.15.150:0/2770132278 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258005280 con 0x7f6268073480 2024-09-17T05:00:05.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.644+0000 7f626d437640 1 -- 172.21.15.150:0/2770132278 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f626814f060 con 0x7f6268073480 2024-09-17T05:00:05.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.644+0000 7f626d437640 1 -- 172.21.15.150:0/2770132278 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f626814f2c0 con 0x7f6268073480 2024-09-17T05:00:05.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.644+0000 7f625f7fe640 1 -- 172.21.15.150:0/2770132278 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6258005420 con 0x7f6268073480 2024-09-17T05:00:05.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.644+0000 7f625f7fe640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6248073230 0x7f6248075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:05.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.644+0000 7f625f7fe640 1 -- 172.21.15.150:0/2770132278 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f6258094a40 con 0x7f6268073480 2024-09-17T05:00:05.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.644+0000 7f625ffff640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6248073230 0x7f6248075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.648+0000 7f625ffff640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6248073230 0x7f6248075740 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f626815da60 tx=0x7f6250009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:05.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.652+0000 7f625f7fe640 1 -- 172.21.15.150:0/2770132278 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f625800de60 con 0x7f6268073480 2024-09-17T05:00:05.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.872+0000 7f626d437640 1 -- 172.21.15.150:0/2770132278 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62680c47e0 con 0x7f6268073480 2024-09-17T05:00:05.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.872+0000 7f625f7fe640 1 -- 172.21.15.150:0/2770132278 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+6537 (secure 0 0 0) 0x7f625805f360 con 0x7f6268073480 2024-09-17T05:00:05.872 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:05.872 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":81,"btime":"2024-09-17T04:59:25:107413+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T05:00:05.873 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 81 2024-09-17T05:00:05.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.876+0000 7f626d437640 1 -- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6248073230 msgr2=0x7f6248075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:05.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.876+0000 7f626d437640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6248073230 0x7f6248075740 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f626815da60 tx=0x7f6250009040 comp rx=0 tx=0).stop 2024-09-17T05:00:05.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.876+0000 7f626d437640 1 -- 172.21.15.150:0/2770132278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6268073480 msgr2=0x7f62681f13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:05.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.876+0000 7f626d437640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6268073480 0x7f62681f13c0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f625800d750 tx=0x7f625800dc10 comp rx=0 tx=0).stop 2024-09-17T05:00:05.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.876+0000 7f626d437640 1 -- 172.21.15.150:0/2770132278 shutdown_connections 2024-09-17T05:00:05.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.876+0000 7f626d437640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6248073230 0x7f6248075740 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:05.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.876+0000 7f626d437640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f626814ec20 0x7f62681506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:05.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.876+0000 7f626d437640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f62680c6d70 0x7f62680af390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:05.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.876+0000 7f626d437640 1 --2- 172.21.15.150:0/2770132278 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6268073480 0x7f62681f13c0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:05.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.876+0000 7f626d437640 1 -- 172.21.15.150:0/2770132278 >> 172.21.15.150:0/2770132278 conn(0x7f62680abf10 msgr2=0x7f626815e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:05.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.876+0000 7f626d437640 1 -- 172.21.15.150:0/2770132278 shutdown_connections 2024-09-17T05:00:05.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:05.876+0000 7f626d437640 1 -- 172.21.15.150:0/2770132278 wait complete. 2024-09-17T05:00:05.896 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-17T05:00:05.896 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T05:00:05.896 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:00:05.910 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T05:00:05.911 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:06.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.112+0000 7fec5dd09640 1 -- 172.21.15.150:0/908472065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec580c35b0 msgr2=0x7fec580c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:06.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.112+0000 7fec5dd09640 1 --2- 172.21.15.150:0/908472065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec580c35b0 0x7fec580c8a80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fec5808d880 tx=0x7fec4001c890 comp rx=0 tx=0).stop 2024-09-17T05:00:06.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.112+0000 7fec5dd09640 1 -- 172.21.15.150:0/908472065 shutdown_connections 2024-09-17T05:00:06.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.112+0000 7fec5dd09640 1 --2- 172.21.15.150:0/908472065 >> v2:172.21.15.176:3300/0 conn(0x7fec580c8fc0 0x7fec580c20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:06.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.112+0000 7fec5dd09640 1 --2- 172.21.15.150:0/908472065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec580c35b0 0x7fec580c8a80 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:06.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.112+0000 7fec5dd09640 1 --2- 172.21.15.150:0/908472065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec581568a0 0x7fec580c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:06.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.112+0000 7fec5dd09640 1 -- 172.21.15.150:0/908472065 >> 172.21.15.150:0/908472065 conn(0x7fec58074b80 msgr2=0x7fec5806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:06.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.112+0000 7fec5dd09640 1 -- 172.21.15.150:0/908472065 shutdown_connections 2024-09-17T05:00:06.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.112+0000 7fec5dd09640 1 -- 172.21.15.150:0/908472065 wait complete. 2024-09-17T05:00:06.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec5dd09640 1 Processor -- start 2024-09-17T05:00:06.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec5dd09640 1 -- start start 2024-09-17T05:00:06.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec5dd09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec580c35b0 0x7fec58150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:06.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec5dd09640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec580c8fc0 0x7fec58158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:06.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec5dd09640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec581568a0 0x7fec581f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec5dd09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fec580c69f0 con 0x7fec580c35b0 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec5dd09640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fec58154530 con 0x7fec580c8fc0 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec5dd09640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fec580c6b70 con 0x7fec581568a0 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec577fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec580c35b0 0x7fec58150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec577fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec580c35b0 0x7fec58150460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59444/0 (socket says 172.21.15.150:59444) 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec577fe640 1 -- 172.21.15.150:0/2245048860 learned_addr learned my addr 172.21.15.150:0/2245048860 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec57fff640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec581568a0 0x7fec581f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec4effd640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec580c8fc0 0x7fec58158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec577fe640 1 -- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec581568a0 msgr2=0x7fec581f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec577fe640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec581568a0 0x7fec581f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec577fe640 1 -- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec580c8fc0 msgr2=0x7fec58158e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec577fe640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec580c8fc0 0x7fec58158e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:06.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec577fe640 1 -- 172.21.15.150:0/2245048860 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec581593a0 con 0x7fec580c35b0 2024-09-17T05:00:06.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec577fe640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec580c35b0 0x7fec58150460 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fec4800ca20 tx=0x7fec4800cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:06.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec54ff9640 1 -- 172.21.15.150:0/2245048860 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec48007c70 con 0x7fec580c35b0 2024-09-17T05:00:06.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec54ff9640 1 -- 172.21.15.150:0/2245048860 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec4800f450 con 0x7fec580c35b0 2024-09-17T05:00:06.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec54ff9640 1 -- 172.21.15.150:0/2245048860 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4800f8a0 con 0x7fec580c35b0 2024-09-17T05:00:06.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec5dd09640 1 -- 172.21.15.150:0/2245048860 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec581f3ed0 con 0x7fec580c35b0 2024-09-17T05:00:06.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec5dd09640 1 -- 172.21.15.150:0/2245048860 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec580ca920 con 0x7fec580c35b0 2024-09-17T05:00:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.116+0000 7fec5dd09640 1 -- 172.21.15.150:0/2245048860 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec58159570 con 0x7fec580c35b0 2024-09-17T05:00:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.120+0000 7fec54ff9640 1 -- 172.21.15.150:0/2245048860 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fec4800fa40 con 0x7fec580c35b0 2024-09-17T05:00:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.120+0000 7fec54ff9640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec38073230 0x7fec38075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.120+0000 7fec54ff9640 1 -- 172.21.15.150:0/2245048860 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7fec48095120 con 0x7fec580c35b0 2024-09-17T05:00:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.120+0000 7fec4effd640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec38073230 0x7fec38075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:06.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.120+0000 7fec4effd640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec38073230 0x7fec38075740 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fec400af000 tx=0x7fec4001b0d0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.124+0000 7fec54ff9640 1 -- 172.21.15.150:0/2245048860 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fec48014030 con 0x7fec580c35b0 2024-09-17T05:00:06.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 -- 172.21.15.150:0/2245048860 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec580c20a0 con 0x7fec580c35b0 2024-09-17T05:00:06.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec54ff9640 1 -- 172.21.15.150:0/2245048860 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+6537 (secure 0 0 0) 0x7fec4805fa40 con 0x7fec580c35b0 2024-09-17T05:00:06.378 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:06.378 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":81,"btime":"2024-09-17T04:59:25:107413+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T05:00:06.378 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 81 2024-09-17T05:00:06.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 -- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec38073230 msgr2=0x7fec38075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:06.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec38073230 0x7fec38075740 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fec400af000 tx=0x7fec4001b0d0 comp rx=0 tx=0).stop 2024-09-17T05:00:06.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 -- 172.21.15.150:0/2245048860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec580c35b0 msgr2=0x7fec58150460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:06.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec580c35b0 0x7fec58150460 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fec4800ca20 tx=0x7fec4800cee0 comp rx=0 tx=0).stop 2024-09-17T05:00:06.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 -- 172.21.15.150:0/2245048860 shutdown_connections 2024-09-17T05:00:06.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec38073230 0x7fec38075740 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:06.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec580c8fc0 0x7fec58158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:06.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec581568a0 0x7fec581f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:06.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 --2- 172.21.15.150:0/2245048860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec580c35b0 0x7fec58150460 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:06.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 -- 172.21.15.150:0/2245048860 >> 172.21.15.150:0/2245048860 conn(0x7fec58074b80 msgr2=0x7fec58154710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:06.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 -- 172.21.15.150:0/2245048860 shutdown_connections 2024-09-17T05:00:06.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:06.380+0000 7fec5dd09640 1 -- 172.21.15.150:0/2245048860 wait complete. 2024-09-17T05:00:06.401 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:00:08.401 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:08.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.576+0000 7fdde882a640 1 -- 172.21.15.150:0/232537262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddcc004810 msgr2=0x7fddcc004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:08.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.576+0000 7fdde882a640 1 --2- 172.21.15.150:0/232537262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddcc004810 0x7fddcc004c20 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fddd8009b60 tx=0x7fddd801d270 comp rx=0 tx=0).stop 2024-09-17T05:00:08.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.576+0000 7fdde882a640 1 -- 172.21.15.150:0/232537262 shutdown_connections 2024-09-17T05:00:08.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.576+0000 7fdde882a640 1 --2- 172.21.15.150:0/232537262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddcc004810 0x7fddcc004c20 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:08.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.576+0000 7fdde882a640 1 -- 172.21.15.150:0/232537262 >> 172.21.15.150:0/232537262 conn(0x7fdde40ca460 msgr2=0x7fdde40c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:08.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.576+0000 7fdde882a640 1 -- 172.21.15.150:0/232537262 shutdown_connections 2024-09-17T05:00:08.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.576+0000 7fdde882a640 1 -- 172.21.15.150:0/232537262 wait complete. 2024-09-17T05:00:08.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde882a640 1 Processor -- start 2024-09-17T05:00:08.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde882a640 1 -- start start 2024-09-17T05:00:08.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde882a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddcc004810 0x7fdde4145ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:08.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde882a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde4142180 0x7fdde4144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:08.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde882a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde41413f0 0x7fdde41e1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:08.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde882a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdde40c62d0 con 0x7fddcc004810 2024-09-17T05:00:08.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde882a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdde40c51f0 con 0x7fdde4142180 2024-09-17T05:00:08.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde882a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdde40c65c0 con 0x7fdde41413f0 2024-09-17T05:00:08.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddcc004810 0x7fdde4145ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:08.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddcc004810 0x7fdde4145ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59464/0 (socket says 172.21.15.150:59464) 2024-09-17T05:00:08.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde259c640 1 -- 172.21.15.150:0/2619516950 learned_addr learned my addr 172.21.15.150:0/2619516950 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde1d9b640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde4142180 0x7fdde4144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde2d9d640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde41413f0 0x7fdde41e1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde259c640 1 -- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde41413f0 msgr2=0x7fdde41e1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde259c640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde41413f0 0x7fdde41e1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde259c640 1 -- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde4142180 msgr2=0x7fdde4144bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde259c640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde4142180 0x7fdde4144bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde259c640 1 -- 172.21.15.150:0/2619516950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde4141db0 con 0x7fddcc004810 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde1d9b640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde4142180 0x7fdde4144bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde259c640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddcc004810 0x7fdde4145ea0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fddd4001170 tx=0x7fddd400fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fddd2ffd640 1 -- 172.21.15.150:0/2619516950 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd4016070 con 0x7fddcc004810 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fddd2ffd640 1 -- 172.21.15.150:0/2619516950 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fddd4011990 con 0x7fddcc004810 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde882a640 1 -- 172.21.15.150:0/2619516950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde41eb2f0 con 0x7fddcc004810 2024-09-17T05:00:08.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fddd2ffd640 1 -- 172.21.15.150:0/2619516950 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd4011d60 con 0x7fddcc004810 2024-09-17T05:00:08.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.580+0000 7fdde882a640 1 -- 172.21.15.150:0/2619516950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde4146430 con 0x7fddcc004810 2024-09-17T05:00:08.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.584+0000 7fdde882a640 1 -- 172.21.15.150:0/2619516950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdde41eb0e0 con 0x7fddcc004810 2024-09-17T05:00:08.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.584+0000 7fddd2ffd640 1 -- 172.21.15.150:0/2619516950 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fddd4019070 con 0x7fddcc004810 2024-09-17T05:00:08.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.584+0000 7fddd2ffd640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fddc4073200 0x7fddc4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:08.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.584+0000 7fddd2ffd640 1 -- 172.21.15.150:0/2619516950 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7fddd4098150 con 0x7fddcc004810 2024-09-17T05:00:08.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.584+0000 7fdde1d9b640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fddc4073200 0x7fddc4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:08.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.584+0000 7fdde1d9b640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fddc4073200 0x7fddc4075710 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fddd80ae000 tx=0x7fddd801a010 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:08.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.588+0000 7fddd2ffd640 1 -- 172.21.15.150:0/2619516950 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fddd4062a70 con 0x7fddcc004810 2024-09-17T05:00:08.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.796+0000 7fdde882a640 1 -- 172.21.15.150:0/2619516950 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdde4072260 con 0x7fddcc004810 2024-09-17T05:00:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.796+0000 7fddd2ffd640 1 -- 172.21.15.150:0/2619516950 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+6537 (secure 0 0 0) 0x7fddd4062370 con 0x7fddcc004810 2024-09-17T05:00:08.796 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:08.796 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":81,"btime":"2024-09-17T04:59:25:107413+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T05:00:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 81 2024-09-17T05:00:08.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.800+0000 7fdde882a640 1 -- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fddc4073200 msgr2=0x7fddc4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:08.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.800+0000 7fdde882a640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fddc4073200 0x7fddc4075710 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fddd80ae000 tx=0x7fddd801a010 comp rx=0 tx=0).stop 2024-09-17T05:00:08.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.800+0000 7fdde882a640 1 -- 172.21.15.150:0/2619516950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddcc004810 msgr2=0x7fdde4145ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:08.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.800+0000 7fdde882a640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddcc004810 0x7fdde4145ea0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fddd4001170 tx=0x7fddd400fa30 comp rx=0 tx=0).stop 2024-09-17T05:00:08.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.800+0000 7fdde882a640 1 -- 172.21.15.150:0/2619516950 shutdown_connections 2024-09-17T05:00:08.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.800+0000 7fdde882a640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fddc4073200 0x7fddc4075710 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:08.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.800+0000 7fdde882a640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdde4142180 0x7fdde4144bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:08.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.800+0000 7fdde882a640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdde41413f0 0x7fdde41e1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:08.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.800+0000 7fdde882a640 1 --2- 172.21.15.150:0/2619516950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fddcc004810 0x7fdde4145ea0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:08.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.800+0000 7fdde882a640 1 -- 172.21.15.150:0/2619516950 >> 172.21.15.150:0/2619516950 conn(0x7fdde40ca460 msgr2=0x7fdde40c6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:08.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.800+0000 7fdde882a640 1 -- 172.21.15.150:0/2619516950 shutdown_connections 2024-09-17T05:00:08.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:08.800+0000 7fdde882a640 1 -- 172.21.15.150:0/2619516950 wait complete. 2024-09-17T05:00:08.820 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:00:10.821 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:10.996+0000 7fdcaf447640 1 -- 172.21.15.150:0/4025489832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca814e490 msgr2=0x7fdca80c7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:10.996+0000 7fdcaf447640 1 --2- 172.21.15.150:0/4025489832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca814e490 0x7fdca80c7d60 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdca8075260 tx=0x7fdc9801c870 comp rx=0 tx=0).stop 2024-09-17T05:00:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:10.996+0000 7fdcaf447640 1 -- 172.21.15.150:0/4025489832 shutdown_connections 2024-09-17T05:00:10.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:10.996+0000 7fdcaf447640 1 --2- 172.21.15.150:0/4025489832 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdca8157c00 0x7fdca8158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:10.996+0000 7fdcaf447640 1 --2- 172.21.15.150:0/4025489832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca814e490 0x7fdca80c7d60 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:10.996+0000 7fdcaf447640 1 --2- 172.21.15.150:0/4025489832 >> v2:172.21.15.176:3300/0 conn(0x7fdca8071670 0x7fdca80c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:10.996+0000 7fdcaf447640 1 -- 172.21.15.150:0/4025489832 >> 172.21.15.150:0/4025489832 conn(0x7fdca80ca460 msgr2=0x7fdca80c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:10.996+0000 7fdcaf447640 1 -- 172.21.15.150:0/4025489832 shutdown_connections 2024-09-17T05:00:10.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:10.996+0000 7fdcaf447640 1 -- 172.21.15.150:0/4025489832 wait complete. 2024-09-17T05:00:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:10.996+0000 7fdcaf447640 1 Processor -- start 2024-09-17T05:00:10.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:10.996+0000 7fdcaf447640 1 -- start start 2024-09-17T05:00:10.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcaf447640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdca8071670 0x7fdca8151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:10.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcaf447640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdca814e490 0x7fdca81457b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:10.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcaf447640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca8157c00 0x7fdca814eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:10.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcaf447640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdca80c65c0 con 0x7fdca8157c00 2024-09-17T05:00:10.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcaf447640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdca80c8350 con 0x7fdca8071670 2024-09-17T05:00:10.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcaf447640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdca80c6740 con 0x7fdca814e490 2024-09-17T05:00:10.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcad9e4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca8157c00 0x7fdca814eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcad9e4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca8157c00 0x7fdca814eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44468/0 (socket says 172.21.15.150:44468) 2024-09-17T05:00:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcad9e4640 1 -- 172.21.15.150:0/3079333306 learned_addr learned my addr 172.21.15.150:0/3079333306 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcad1e3640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdca8071670 0x7fdca8151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcac9e2640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdca814e490 0x7fdca81457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcad9e4640 1 -- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdca814e490 msgr2=0x7fdca81457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcad9e4640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdca814e490 0x7fdca81457b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcad9e4640 1 -- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdca8071670 msgr2=0x7fdca8151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcad9e4640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdca8071670 0x7fdca8151070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcad9e4640 1 -- 172.21.15.150:0/3079333306 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca8154200 con 0x7fdca8157c00 2024-09-17T05:00:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcad9e4640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca8157c00 0x7fdca814eb10 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fdca000e910 tx=0x7fdca000edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:11.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdc9dffb640 1 -- 172.21.15.150:0/3079333306 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca0004d30 con 0x7fdca8157c00 2024-09-17T05:00:11.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdc9dffb640 1 -- 172.21.15.150:0/3079333306 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdca000cb20 con 0x7fdca8157c00 2024-09-17T05:00:11.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdc9dffb640 1 -- 172.21.15.150:0/3079333306 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca0005020 con 0x7fdca8157c00 2024-09-17T05:00:11.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcaf447640 1 -- 172.21.15.150:0/3079333306 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca81f08d0 con 0x7fdca8157c00 2024-09-17T05:00:11.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcaf447640 1 -- 172.21.15.150:0/3079333306 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca81f0e90 con 0x7fdca8157c00 2024-09-17T05:00:11.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.000+0000 7fdcaf447640 1 -- 172.21.15.150:0/3079333306 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdca8072180 con 0x7fdca8157c00 2024-09-17T05:00:11.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.004+0000 7fdc9dffb640 1 -- 172.21.15.150:0/3079333306 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fdca00051c0 con 0x7fdca8157c00 2024-09-17T05:00:11.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.004+0000 7fdc9dffb640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdc8c073200 0x7fdc8c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:11.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.004+0000 7fdc9dffb640 1 -- 172.21.15.150:0/3079333306 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7fdca00951b0 con 0x7fdca8157c00 2024-09-17T05:00:11.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.004+0000 7fdcad1e3640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdc8c073200 0x7fdc8c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:11.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.004+0000 7fdcad1e3640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdc8c073200 0x7fdc8c075710 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdca4005fd0 tx=0x7fdca4005d10 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:11.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.008+0000 7fdc9dffb640 1 -- 172.21.15.150:0/3079333306 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdca0016030 con 0x7fdca8157c00 2024-09-17T05:00:11.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.316+0000 7fdcaf447640 1 -- 172.21.15.150:0/3079333306 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdca80c55f0 con 0x7fdca8157c00 2024-09-17T05:00:11.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.316+0000 7fdc9dffb640 1 -- 172.21.15.150:0/3079333306 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+6537 (secure 0 0 0) 0x7fdca005fa40 con 0x7fdca8157c00 2024-09-17T05:00:11.316 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:11.316 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":81,"btime":"2024-09-17T04:59:25:107413+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T05:00:11.316 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 81 2024-09-17T05:00:11.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.320+0000 7fdcaf447640 1 -- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdc8c073200 msgr2=0x7fdc8c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:11.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.320+0000 7fdcaf447640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdc8c073200 0x7fdc8c075710 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdca4005fd0 tx=0x7fdca4005d10 comp rx=0 tx=0).stop 2024-09-17T05:00:11.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.320+0000 7fdcaf447640 1 -- 172.21.15.150:0/3079333306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca8157c00 msgr2=0x7fdca814eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:11.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.320+0000 7fdcaf447640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca8157c00 0x7fdca814eb10 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fdca000e910 tx=0x7fdca000edd0 comp rx=0 tx=0).stop 2024-09-17T05:00:11.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.320+0000 7fdcaf447640 1 -- 172.21.15.150:0/3079333306 shutdown_connections 2024-09-17T05:00:11.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.320+0000 7fdcaf447640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdc8c073200 0x7fdc8c075710 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:11.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.320+0000 7fdcaf447640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdca8157c00 0x7fdca814eb10 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:11.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.320+0000 7fdcaf447640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdca814e490 0x7fdca81457b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:11.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.320+0000 7fdcaf447640 1 --2- 172.21.15.150:0/3079333306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdca8071670 0x7fdca8151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:11.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.320+0000 7fdcaf447640 1 -- 172.21.15.150:0/3079333306 >> 172.21.15.150:0/3079333306 conn(0x7fdca80ca460 msgr2=0x7fdca80c7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:11.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.320+0000 7fdcaf447640 1 -- 172.21.15.150:0/3079333306 shutdown_connections 2024-09-17T05:00:11.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:11.320+0000 7fdcaf447640 1 -- 172.21.15.150:0/3079333306 wait complete. 2024-09-17T05:00:11.340 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:00:13.341 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:13.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.524+0000 7efe4830c640 1 -- 172.21.15.150:0/3290131936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe2c004d50 msgr2=0x7efe2c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:13.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.524+0000 7efe4830c640 1 --2- 172.21.15.150:0/3290131936 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe2c004d50 0x7efe2c005160 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7efe38009eb0 tx=0x7efe3801d5f0 comp rx=0 tx=0).stop 2024-09-17T05:00:13.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.524+0000 7efe468a9640 1 -- 172.21.15.150:0/3290131936 reap_dead start 2024-09-17T05:00:13.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.524+0000 7efe4830c640 1 -- 172.21.15.150:0/3290131936 shutdown_connections 2024-09-17T05:00:13.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.524+0000 7efe4830c640 1 -- 172.21.15.150:0/3290131936 >> 172.21.15.150:0/3290131936 conn(0x7efe400c5ae0 msgr2=0x7efe40073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:13.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe4830c640 1 -- 172.21.15.150:0/3290131936 shutdown_connections 2024-09-17T05:00:13.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe4830c640 1 -- 172.21.15.150:0/3290131936 wait complete. 2024-09-17T05:00:13.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe4830c640 1 Processor -- start 2024-09-17T05:00:13.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe4830c640 1 -- start start 2024-09-17T05:00:13.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe4830c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe400c6210 0x7efe40150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:13.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe4830c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe40150f20 0x7efe400c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:13.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe4830c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe40155160 0x7efe401525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:13.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe4830c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efe400a9c30 con 0x7efe40155160 2024-09-17T05:00:13.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe4830c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efe400a9ab0 con 0x7efe400c6210 2024-09-17T05:00:13.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe4830c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efe400a9db0 con 0x7efe40150f20 2024-09-17T05:00:13.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe468a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe40155160 0x7efe401525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:13.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe468a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe40155160 0x7efe401525e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44486/0 (socket says 172.21.15.150:44486) 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe468a9640 1 -- 172.21.15.150:0/2215809082 learned_addr learned my addr 172.21.15.150:0/2215809082 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe458a7640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe40150f20 0x7efe400c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe460a8640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe400c6210 0x7efe40150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe468a9640 1 -- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe40150f20 msgr2=0x7efe400c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe468a9640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe40150f20 0x7efe400c7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe468a9640 1 -- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe400c6210 msgr2=0x7efe40150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe468a9640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe400c6210 0x7efe40150690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe468a9640 1 -- 172.21.15.150:0/2215809082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe40154c40 con 0x7efe40155160 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.528+0000 7efe458a7640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe40150f20 0x7efe400c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.532+0000 7efe468a9640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe40155160 0x7efe401525e0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7efe380ae040 tx=0x7efe38004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.532+0000 7efe36ffd640 1 -- 172.21.15.150:0/2215809082 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe38009ab0 con 0x7efe40155160 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.532+0000 7efe36ffd640 1 -- 172.21.15.150:0/2215809082 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe38003c70 con 0x7efe40155160 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.532+0000 7efe4830c640 1 -- 172.21.15.150:0/2215809082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe401ee140 con 0x7efe40155160 2024-09-17T05:00:13.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.532+0000 7efe36ffd640 1 -- 172.21.15.150:0/2215809082 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe38009200 con 0x7efe40155160 2024-09-17T05:00:13.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.532+0000 7efe4830c640 1 -- 172.21.15.150:0/2215809082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe40156920 con 0x7efe40155160 2024-09-17T05:00:13.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.532+0000 7efe4830c640 1 -- 172.21.15.150:0/2215809082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe4006fd30 con 0x7efe40155160 2024-09-17T05:00:13.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.532+0000 7efe36ffd640 1 -- 172.21.15.150:0/2215809082 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7efe38004240 con 0x7efe40155160 2024-09-17T05:00:13.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.536+0000 7efe36ffd640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efe24073200 0x7efe24075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:13.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.536+0000 7efe36ffd640 1 -- 172.21.15.150:0/2215809082 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7efe38130620 con 0x7efe40155160 2024-09-17T05:00:13.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.536+0000 7efe460a8640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efe24073200 0x7efe24075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:13.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.536+0000 7efe460a8640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efe24073200 0x7efe24075710 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7efe3c001ec0 tx=0x7efe3c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:13.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.536+0000 7efe36ffd640 1 -- 172.21.15.150:0/2215809082 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7efe380c6030 con 0x7efe40155160 2024-09-17T05:00:13.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.772+0000 7efe4830c640 1 -- 172.21.15.150:0/2215809082 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe40159250 con 0x7efe40155160 2024-09-17T05:00:13.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.772+0000 7efe36ffd640 1 -- 172.21.15.150:0/2215809082 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+6537 (secure 0 0 0) 0x7efe38017300 con 0x7efe40155160 2024-09-17T05:00:13.770 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:13.771 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":81,"btime":"2024-09-17T04:59:25:107413+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T05:00:13.771 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 81 2024-09-17T05:00:13.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.776+0000 7efe4830c640 1 -- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efe24073200 msgr2=0x7efe24075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:13.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.776+0000 7efe4830c640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efe24073200 0x7efe24075710 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7efe3c001ec0 tx=0x7efe3c009040 comp rx=0 tx=0).stop 2024-09-17T05:00:13.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.776+0000 7efe4830c640 1 -- 172.21.15.150:0/2215809082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe40155160 msgr2=0x7efe401525e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:13.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.776+0000 7efe4830c640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe40155160 0x7efe401525e0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7efe380ae040 tx=0x7efe38004f50 comp rx=0 tx=0).stop 2024-09-17T05:00:13.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.776+0000 7efe4830c640 1 -- 172.21.15.150:0/2215809082 shutdown_connections 2024-09-17T05:00:13.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.776+0000 7efe4830c640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efe24073200 0x7efe24075710 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.776+0000 7efe4830c640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe40155160 0x7efe401525e0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.776+0000 7efe4830c640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe40150f20 0x7efe400c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.776+0000 7efe4830c640 1 --2- 172.21.15.150:0/2215809082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe400c6210 0x7efe40150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.776+0000 7efe4830c640 1 -- 172.21.15.150:0/2215809082 >> 172.21.15.150:0/2215809082 conn(0x7efe400c5ae0 msgr2=0x7efe4014ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.776+0000 7efe4830c640 1 -- 172.21.15.150:0/2215809082 shutdown_connections 2024-09-17T05:00:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:13.776+0000 7efe4830c640 1 -- 172.21.15.150:0/2215809082 wait complete. 2024-09-17T05:00:13.792 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:00:15.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:15.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.988+0000 7f743087e640 1 -- 172.21.15.150:0/373800711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c06fca0 msgr2=0x7f742c153790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:15.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.988+0000 7f743087e640 1 --2- 172.21.15.150:0/373800711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c06fca0 0x7f742c153790 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f742c0c2be0 tx=0x7f741c01c770 comp rx=0 tx=0).stop 2024-09-17T05:00:15.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.988+0000 7f743087e640 1 -- 172.21.15.150:0/373800711 shutdown_connections 2024-09-17T05:00:15.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.988+0000 7f743087e640 1 --2- 172.21.15.150:0/373800711 >> v2:172.21.15.176:3300/0 conn(0x7f742c074570 0x7f742c0aaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:15.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.988+0000 7f743087e640 1 --2- 172.21.15.150:0/373800711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c06fca0 0x7f742c153790 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:15.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.988+0000 7f743087e640 1 -- 172.21.15.150:0/373800711 >> 172.21.15.150:0/373800711 conn(0x7f742c0c5ae0 msgr2=0x7f742c073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:15.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.988+0000 7f743087e640 1 -- 172.21.15.150:0/373800711 shutdown_connections 2024-09-17T05:00:15.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.988+0000 7f743087e640 1 -- 172.21.15.150:0/373800711 wait complete. 2024-09-17T05:00:15.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f743087e640 1 Processor -- start 2024-09-17T05:00:15.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f743087e640 1 -- start start 2024-09-17T05:00:15.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f743087e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c06fca0 0x7f742c145400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:15.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f743087e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f742c074570 0x7f742c14eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:15.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f743087e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f742c074980 0x7f742c14c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:15.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f743087e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f742c0c4ce0 con 0x7f742c06fca0 2024-09-17T05:00:15.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f743087e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f742c15f6a0 con 0x7f742c074980 2024-09-17T05:00:15.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f743087e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f742c154640 con 0x7f742c074570 2024-09-17T05:00:15.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f742a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c06fca0 0x7f742c145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f742a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c06fca0 0x7f742c145400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44504/0 (socket says 172.21.15.150:44504) 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f742a59c640 1 -- 172.21.15.150:0/3017368154 learned_addr learned my addr 172.21.15.150:0/3017368154 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f742ad9d640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f742c074980 0x7f742c14c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f7423fff640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f742c074570 0x7f742c14eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f742a59c640 1 -- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f742c074570 msgr2=0x7f742c14eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f742a59c640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f742c074570 0x7f742c14eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f742a59c640 1 -- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f742c074980 msgr2=0x7f742c14c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f742a59c640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f742c074980 0x7f742c14c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f742a59c640 1 -- 172.21.15.150:0/3017368154 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742c0c6860 con 0x7f742c06fca0 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f742a59c640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c06fca0 0x7f742c145400 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f741c009790 tx=0x7f741c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f7422ffd640 1 -- 172.21.15.150:0/3017368154 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c0b2040 con 0x7f742c06fca0 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f7422ffd640 1 -- 172.21.15.150:0/3017368154 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f741c0043e0 con 0x7f742c06fca0 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f7422ffd640 1 -- 172.21.15.150:0/3017368154 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c005270 con 0x7f742c06fca0 2024-09-17T05:00:15.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f743087e640 1 -- 172.21.15.150:0/3017368154 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742c14fb20 con 0x7f742c06fca0 2024-09-17T05:00:15.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.992+0000 7f743087e640 1 -- 172.21.15.150:0/3017368154 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f742c1571f0 con 0x7f742c06fca0 2024-09-17T05:00:15.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.996+0000 7f743087e640 1 -- 172.21.15.150:0/3017368154 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f742c153790 con 0x7f742c06fca0 2024-09-17T05:00:15.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.996+0000 7f7422ffd640 1 -- 172.21.15.150:0/3017368154 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f741c005410 con 0x7f742c06fca0 2024-09-17T05:00:15.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.996+0000 7f7422ffd640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f740c073230 0x7f740c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:15.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.996+0000 7f7422ffd640 1 -- 172.21.15.150:0/3017368154 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f741c12e1c0 con 0x7f742c06fca0 2024-09-17T05:00:15.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.996+0000 7f7423fff640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f740c073230 0x7f740c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:15.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:15.996+0000 7f7423fff640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f740c073230 0x7f740c075740 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f74140058d0 tx=0x7f7414005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:16.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.000+0000 7f7422ffd640 1 -- 172.21.15.150:0/3017368154 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f741c0163f0 con 0x7f742c06fca0 2024-09-17T05:00:16.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.268+0000 7f743087e640 1 -- 172.21.15.150:0/3017368154 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f742c149870 con 0x7f742c06fca0 2024-09-17T05:00:16.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.268+0000 7f7422ffd640 1 -- 172.21.15.150:0/3017368154 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+6537 (secure 0 0 0) 0x7f741c0f8940 con 0x7f742c06fca0 2024-09-17T05:00:16.268 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:16.268 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":81,"btime":"2024-09-17T04:59:25:107413+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T05:00:16.269 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 81 2024-09-17T05:00:16.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.272+0000 7f743087e640 1 -- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f740c073230 msgr2=0x7f740c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:16.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.272+0000 7f743087e640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f740c073230 0x7f740c075740 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f74140058d0 tx=0x7f7414005860 comp rx=0 tx=0).stop 2024-09-17T05:00:16.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.272+0000 7f743087e640 1 -- 172.21.15.150:0/3017368154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c06fca0 msgr2=0x7f742c145400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:16.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.272+0000 7f743087e640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c06fca0 0x7f742c145400 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f741c009790 tx=0x7f741c005dd0 comp rx=0 tx=0).stop 2024-09-17T05:00:16.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.272+0000 7f743087e640 1 -- 172.21.15.150:0/3017368154 shutdown_connections 2024-09-17T05:00:16.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.272+0000 7f743087e640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f740c073230 0x7f740c075740 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:16.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.272+0000 7f743087e640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f742c074980 0x7f742c14c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:16.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.272+0000 7f743087e640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f742c074570 0x7f742c14eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:16.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.272+0000 7f743087e640 1 --2- 172.21.15.150:0/3017368154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c06fca0 0x7f742c145400 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:16.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.272+0000 7f743087e640 1 -- 172.21.15.150:0/3017368154 >> 172.21.15.150:0/3017368154 conn(0x7f742c0c5ae0 msgr2=0x7f742c0c7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:16.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.272+0000 7f743087e640 1 -- 172.21.15.150:0/3017368154 shutdown_connections 2024-09-17T05:00:16.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:16.272+0000 7f743087e640 1 -- 172.21.15.150:0/3017368154 wait complete. 2024-09-17T05:00:16.292 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:00:18.293 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:18.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.476+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/2090083093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a4074e40 msgr2=0x7fd5a41570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:18.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.476+0000 7fd5aaa1c640 1 --2- 172.21.15.150:0/2090083093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a4074e40 0x7fd5a41570b0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4074030 tx=0x7fd59c01cc60 comp rx=0 tx=0).stop 2024-09-17T05:00:18.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.476+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/2090083093 shutdown_connections 2024-09-17T05:00:18.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.476+0000 7fd5aaa1c640 1 --2- 172.21.15.150:0/2090083093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a4074e40 0x7fd5a41570b0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:18.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.476+0000 7fd5aaa1c640 1 --2- 172.21.15.150:0/2090083093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd5a40a8120 0x7fd5a4074900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:18.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.476+0000 7fd5aaa1c640 1 --2- 172.21.15.150:0/2090083093 >> v2:172.21.15.176:3300/0 conn(0x7fd5a40c39d0 0x7fd5a4150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:18.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.476+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/2090083093 >> 172.21.15.150:0/2090083093 conn(0x7fd5a406e360 msgr2=0x7fd5a406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:18.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.476+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/2090083093 shutdown_connections 2024-09-17T05:00:18.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.476+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/2090083093 wait complete. 2024-09-17T05:00:18.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5aaa1c640 1 Processor -- start 2024-09-17T05:00:18.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5aaa1c640 1 -- start start 2024-09-17T05:00:18.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5aaa1c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a4074e40 0x7fd5a4158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:18.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5aaa1c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5a40a8120 0x7fd5a40c5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:18.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5aaa1c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd5a40c39d0 0x7fd5a414b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:18.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5aaa1c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a40a86b0 con 0x7fd5a4074e40 2024-09-17T05:00:18.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5aaa1c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a40a8530 con 0x7fd5a40a8120 2024-09-17T05:00:18.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5aaa1c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a40c74c0 con 0x7fd5a40c39d0 2024-09-17T05:00:18.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a4074e40 0x7fd5a4158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:18.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a4074e40 0x7fd5a4158f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44526/0 (socket says 172.21.15.150:44526) 2024-09-17T05:00:18.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a3fff640 1 -- 172.21.15.150:0/1313013052 learned_addr learned my addr 172.21.15.150:0/1313013052 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:18.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a8fb9640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd5a40c39d0 0x7fd5a414b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:18.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a37fe640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5a40a8120 0x7fd5a40c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:18.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a3fff640 1 -- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd5a40c39d0 msgr2=0x7fd5a414b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:18.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a3fff640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd5a40c39d0 0x7fd5a414b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:18.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a3fff640 1 -- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5a40a8120 msgr2=0x7fd5a40c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:18.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a3fff640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5a40a8120 0x7fd5a40c5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:18.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a3fff640 1 -- 172.21.15.150:0/1313013052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a4159460 con 0x7fd5a4074e40 2024-09-17T05:00:18.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a3fff640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a4074e40 0x7fd5a4158f20 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd59400d950 tx=0x7fd59400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:18.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a0ff9640 1 -- 172.21.15.150:0/1313013052 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd594017070 con 0x7fd5a4074e40 2024-09-17T05:00:18.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a0ff9640 1 -- 172.21.15.150:0/1313013052 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd594009d20 con 0x7fd5a4074e40 2024-09-17T05:00:18.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/1313013052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a415cc90 con 0x7fd5a4074e40 2024-09-17T05:00:18.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5a0ff9640 1 -- 172.21.15.150:0/1313013052 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5940106c0 con 0x7fd5a4074e40 2024-09-17T05:00:18.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.480+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/1313013052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a41f0840 con 0x7fd5a4074e40 2024-09-17T05:00:18.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.484+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/1313013052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a4157d80 con 0x7fd5a4074e40 2024-09-17T05:00:18.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.484+0000 7fd5a0ff9640 1 -- 172.21.15.150:0/1313013052 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd59401a070 con 0x7fd5a4074e40 2024-09-17T05:00:18.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.484+0000 7fd5a0ff9640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd590073200 0x7fd590075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:18.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.484+0000 7fd5a0ff9640 1 -- 172.21.15.150:0/1313013052 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7fd594094c40 con 0x7fd5a4074e40 2024-09-17T05:00:18.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.484+0000 7fd5a37fe640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd590073200 0x7fd590075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:18.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.484+0000 7fd5a37fe640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd590073200 0x7fd590075710 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40c8210 tx=0x7fd59800c460 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:18.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.488+0000 7fd5a0ff9640 1 -- 172.21.15.150:0/1313013052 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd5a4159460 con 0x7fd5a4074e40 2024-09-17T05:00:18.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.768+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/1313013052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5a40c79b0 con 0x7fd5a4074e40 2024-09-17T05:00:18.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5a0ff9640 1 -- 172.21.15.150:0/1313013052 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+6537 (secure 0 0 0) 0x7fd59405f4d0 con 0x7fd5a4074e40 2024-09-17T05:00:18.769 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:18.769 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":81,"btime":"2024-09-17T04:59:25:107413+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T04:58:33.945203+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5901":{"gid":5901,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6835/2081398022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2081398022},{"type":"v1","addr":"172.21.15.150:6835","nonce":2081398022}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5901,"qdb_cluster":[5901]},"id":2}]} 2024-09-17T05:00:18.769 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 81 2024-09-17T05:00:18.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd590073200 msgr2=0x7fd590075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:18.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5aaa1c640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd590073200 0x7fd590075710 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40c8210 tx=0x7fd59800c460 comp rx=0 tx=0).stop 2024-09-17T05:00:18.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/1313013052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a4074e40 msgr2=0x7fd5a4158f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:18.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5aaa1c640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a4074e40 0x7fd5a4158f20 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd59400d950 tx=0x7fd59400de10 comp rx=0 tx=0).stop 2024-09-17T05:00:18.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/1313013052 shutdown_connections 2024-09-17T05:00:18.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5aaa1c640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd590073200 0x7fd590075710 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:18.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5aaa1c640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5a40a8120 0x7fd5a40c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:18.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5aaa1c640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd5a40c39d0 0x7fd5a414b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:18.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5aaa1c640 1 --2- 172.21.15.150:0/1313013052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a4074e40 0x7fd5a4158f20 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:18.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/1313013052 >> 172.21.15.150:0/1313013052 conn(0x7fd5a406e360 msgr2=0x7fd5a4150400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:18.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/1313013052 shutdown_connections 2024-09-17T05:00:18.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:18.772+0000 7fd5aaa1c640 1 -- 172.21.15.150:0/1313013052 wait complete. 2024-09-17T05:00:18.791 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:00:20.792 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:20.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.972+0000 7f59ce4c1640 1 -- 172.21.15.150:0/2567680483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5df0 msgr2=0x7f59c8157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:20.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.972+0000 7f59ce4c1640 1 --2- 172.21.15.150:0/2567680483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5df0 0x7f59c8157330 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f59c80c4610 tx=0x7f59b801c650 comp rx=0 tx=0).stop 2024-09-17T05:00:20.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.972+0000 7f59ce4c1640 1 -- 172.21.15.150:0/2567680483 shutdown_connections 2024-09-17T05:00:20.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.972+0000 7f59ce4c1640 1 --2- 172.21.15.150:0/2567680483 >> v2:172.21.15.176:3300/0 conn(0x7f59c80c4b20 0x7f59c8153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:20.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.972+0000 7f59ce4c1640 1 --2- 172.21.15.150:0/2567680483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5df0 0x7f59c8157330 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:20.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.972+0000 7f59ce4c1640 1 -- 172.21.15.150:0/2567680483 >> 172.21.15.150:0/2567680483 conn(0x7f59c8074750 msgr2=0x7f59c8074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:20.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.972+0000 7f59ce4c1640 1 -- 172.21.15.150:0/2567680483 shutdown_connections 2024-09-17T05:00:20.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.972+0000 7f59ce4c1640 1 -- 172.21.15.150:0/2567680483 wait complete. 2024-09-17T05:00:20.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.976+0000 7f59ce4c1640 1 Processor -- start 2024-09-17T05:00:20.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.976+0000 7f59ce4c1640 1 -- start start 2024-09-17T05:00:20.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.976+0000 7f59ce4c1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59c80c4b20 0x7f59c8159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:20.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.976+0000 7f59ce4c1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5df0 0x7f59c814bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:20.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.976+0000 7f59ce4c1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59c80c6340 0x7f59c8149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:20.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.976+0000 7f59ce4c1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f59c8163d10 con 0x7f59c80c5df0 2024-09-17T05:00:20.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.976+0000 7f59ce4c1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f59c8163b90 con 0x7f59c80c6340 2024-09-17T05:00:20.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.976+0000 7f59ce4c1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f59c8163e90 con 0x7f59c80c4b20 2024-09-17T05:00:20.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.976+0000 7f59bffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5df0 0x7f59c814bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:20.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.976+0000 7f59bffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5df0 0x7f59c814bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32846/0 (socket says 172.21.15.150:32846) 2024-09-17T05:00:20.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.976+0000 7f59bffff640 1 -- 172.21.15.150:0/300439176 learned_addr learned my addr 172.21.15.150:0/300439176 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:20.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.976+0000 7f59c7fff640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59c80c4b20 0x7f59c8159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:20.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59cca5e640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59c80c6340 0x7f59c8149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:20.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59c7fff640 1 -- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59c80c6340 msgr2=0x7f59c8149dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:20.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59c7fff640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59c80c6340 0x7f59c8149dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:20.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59c7fff640 1 -- 172.21.15.150:0/300439176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5df0 msgr2=0x7f59c814bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:20.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59c7fff640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5df0 0x7f59c814bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:20.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59c7fff640 1 -- 172.21.15.150:0/300439176 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c81498f0 con 0x7f59c80c4b20 2024-09-17T05:00:20.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59c7fff640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59c80c4b20 0x7f59c8159a80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f59b80ac000 tx=0x7f59b80a86c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:20.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59c57fa640 1 -- 172.21.15.150:0/300439176 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b8004340 con 0x7f59c80c4b20 2024-09-17T05:00:20.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59c57fa640 1 -- 172.21.15.150:0/300439176 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59b8003600 con 0x7f59c80c4b20 2024-09-17T05:00:20.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59ce4c1640 1 -- 172.21.15.150:0/300439176 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c81f0710 con 0x7f59c80c4b20 2024-09-17T05:00:20.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59c57fa640 1 -- 172.21.15.150:0/300439176 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b8003a50 con 0x7f59c80c4b20 2024-09-17T05:00:20.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59ce4c1640 1 -- 172.21.15.150:0/300439176 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c81f0cd0 con 0x7f59c80c4b20 2024-09-17T05:00:20.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.980+0000 7f59ce4c1640 1 -- 172.21.15.150:0/300439176 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59c8153bc0 con 0x7f59c80c4b20 2024-09-17T05:00:20.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.984+0000 7f59c57fa640 1 -- 172.21.15.150:0/300439176 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f59b80a6040 con 0x7f59c80c4b20 2024-09-17T05:00:20.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.984+0000 7f59c57fa640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f59a8073230 0x7f59a8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:20.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.984+0000 7f59c57fa640 1 -- 172.21.15.150:0/300439176 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7f59b812f3b0 con 0x7f59c80c4b20 2024-09-17T05:00:20.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.984+0000 7f59bffff640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f59a8073230 0x7f59a8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:20.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.984+0000 7f59bffff640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f59a8073230 0x7f59a8075740 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f59b0006fd0 tx=0x7f59b0008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:20.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:20.988+0000 7f59c57fa640 1 -- 172.21.15.150:0/300439176 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f59b80ac030 con 0x7f59c80c4b20 2024-09-17T05:00:21.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.260+0000 7f59ce4c1640 1 -- 172.21.15.150:0/300439176 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59c8149380 con 0x7f59c80c4b20 2024-09-17T05:00:21.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.260+0000 7f59c57fa640 1 -- 172.21.15.150:0/300439176 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5702 (secure 0 0 0) 0x7f59c80b20f0 con 0x7f59c80c4b20 2024-09-17T05:00:21.258 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:21.258 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":82,"btime":"2024-09-17T05:00:20:961742+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:20.961736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:00:21.259 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 82 2024-09-17T05:00:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.264+0000 7f59ce4c1640 1 -- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f59a8073230 msgr2=0x7f59a8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.264+0000 7f59ce4c1640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f59a8073230 0x7f59a8075740 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f59b0006fd0 tx=0x7f59b0008040 comp rx=0 tx=0).stop 2024-09-17T05:00:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.264+0000 7f59ce4c1640 1 -- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59c80c4b20 msgr2=0x7f59c8159a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.264+0000 7f59ce4c1640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59c80c4b20 0x7f59c8159a80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f59b80ac000 tx=0x7f59b80a86c0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.264+0000 7f59ce4c1640 1 -- 172.21.15.150:0/300439176 shutdown_connections 2024-09-17T05:00:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.264+0000 7f59ce4c1640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f59a8073230 0x7f59a8075740 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.264+0000 7f59ce4c1640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59c80c6340 0x7f59c8149dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.264+0000 7f59ce4c1640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59c80c5df0 0x7f59c814bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.264+0000 7f59ce4c1640 1 --2- 172.21.15.150:0/300439176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59c80c4b20 0x7f59c8159a80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.264+0000 7f59ce4c1640 1 -- 172.21.15.150:0/300439176 >> 172.21.15.150:0/300439176 conn(0x7f59c8074750 msgr2=0x7f59c80c8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:21.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.264+0000 7f59ce4c1640 1 -- 172.21.15.150:0/300439176 shutdown_connections 2024-09-17T05:00:21.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.264+0000 7f59ce4c1640 1 -- 172.21.15.150:0/300439176 wait complete. 2024-09-17T05:00:21.284 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-17T05:00:21.285 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:00:21.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:21.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.448+0000 7f54e3d63640 1 -- 172.21.15.150:0/639956306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54dc0ad7f0 msgr2=0x7f54dc0c6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:21.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.448+0000 7f54e3d63640 1 --2- 172.21.15.150:0/639956306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54dc0ad7f0 0x7f54dc0c6230 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0c9cb0 tx=0x7f54d801c780 comp rx=0 tx=0).stop 2024-09-17T05:00:21.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.448+0000 7f54e3d63640 1 -- 172.21.15.150:0/639956306 shutdown_connections 2024-09-17T05:00:21.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.448+0000 7f54e3d63640 1 --2- 172.21.15.150:0/639956306 >> v2:172.21.15.176:3300/0 conn(0x7f54dc157ca0 0x7f54dc1580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.448+0000 7f54e3d63640 1 --2- 172.21.15.150:0/639956306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54dc0ad7f0 0x7f54dc0c6230 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.448+0000 7f54e3d63640 1 -- 172.21.15.150:0/639956306 >> 172.21.15.150:0/639956306 conn(0x7f54dc0c9de0 msgr2=0x7f54dc0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:21.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.448+0000 7f54e3d63640 1 -- 172.21.15.150:0/639956306 shutdown_connections 2024-09-17T05:00:21.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.448+0000 7f54e3d63640 1 -- 172.21.15.150:0/639956306 wait complete. 2024-09-17T05:00:21.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.448+0000 7f54e3d63640 1 Processor -- start 2024-09-17T05:00:21.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e3d63640 1 -- start start 2024-09-17T05:00:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e3d63640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54dc0ad7f0 0x7f54dc144dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e3d63640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54dc151e10 0x7f54dc1ed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e3d63640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54dc157ca0 0x7f54dc147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e3d63640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc158770 con 0x7f54dc157ca0 2024-09-17T05:00:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e3d63640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc1585f0 con 0x7f54dc151e10 2024-09-17T05:00:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e3d63640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f54dc0c7ce0 con 0x7f54dc0ad7f0 2024-09-17T05:00:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e2300640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54dc157ca0 0x7f54dc147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e2300640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54dc157ca0 0x7f54dc147410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32856/0 (socket says 172.21.15.150:32856) 2024-09-17T05:00:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e2300640 1 -- 172.21.15.150:0/2761970477 learned_addr learned my addr 172.21.15.150:0/2761970477 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e12fe640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54dc151e10 0x7f54dc1ed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e1aff640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54dc0ad7f0 0x7f54dc144dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e2300640 1 -- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54dc0ad7f0 msgr2=0x7f54dc144dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e2300640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54dc0ad7f0 0x7f54dc144dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e2300640 1 -- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54dc151e10 msgr2=0x7f54dc1ed7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e2300640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54dc151e10 0x7f54dc1ed7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e2300640 1 -- 172.21.15.150:0/2761970477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54dc14bd00 con 0x7f54dc157ca0 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e2300640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54dc157ca0 0x7f54dc147410 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f54d400e9a0 tx=0x7f54d400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54d27fc640 1 -- 172.21.15.150:0/2761970477 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d4004d30 con 0x7f54dc157ca0 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54d27fc640 1 -- 172.21.15.150:0/2761970477 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54d400cb20 con 0x7f54dc157ca0 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54d27fc640 1 -- 172.21.15.150:0/2761970477 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d4005020 con 0x7f54dc157ca0 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e3d63640 1 -- 172.21.15.150:0/2761970477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54dc14dfc0 con 0x7f54dc157ca0 2024-09-17T05:00:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e3d63640 1 -- 172.21.15.150:0/2761970477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54dc157630 con 0x7f54dc157ca0 2024-09-17T05:00:21.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.452+0000 7f54e3d63640 1 -- 172.21.15.150:0/2761970477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54dc0c6230 con 0x7f54dc157ca0 2024-09-17T05:00:21.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.456+0000 7f54d27fc640 1 -- 172.21.15.150:0/2761970477 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f54d40051c0 con 0x7f54dc157ca0 2024-09-17T05:00:21.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.456+0000 7f54d27fc640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f54c0073200 0x7f54c0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:21.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.456+0000 7f54d27fc640 1 -- 172.21.15.150:0/2761970477 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7f54d4095230 con 0x7f54dc157ca0 2024-09-17T05:00:21.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.456+0000 7f54e1aff640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f54c0073200 0x7f54c0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:21.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.456+0000 7f54e1aff640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f54c0073200 0x7f54c0075710 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f54d80af000 tx=0x7f54d801afc0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:21.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.460+0000 7f54d27fc640 1 -- 172.21.15.150:0/2761970477 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f54d4016030 con 0x7f54dc157ca0 2024-09-17T05:00:21.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.688+0000 7f54e3d63640 1 -- 172.21.15.150:0/2761970477 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54dc146cc0 con 0x7f54dc157ca0 2024-09-17T05:00:21.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.688+0000 7f54d27fc640 1 -- 172.21.15.150:0/2761970477 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+5702 (secure 0 0 0) 0x7f54d405f420 con 0x7f54dc157ca0 2024-09-17T05:00:21.686 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:21.686 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":82,"btime":"2024-09-17T05:00:20:961742+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:20.961736+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:replay","state_seq":1,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:00:21.687 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 82 2024-09-17T05:00:21.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.692+0000 7f54e3d63640 1 -- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f54c0073200 msgr2=0x7f54c0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:21.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.692+0000 7f54e3d63640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f54c0073200 0x7f54c0075710 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f54d80af000 tx=0x7f54d801afc0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.692+0000 7f54e3d63640 1 -- 172.21.15.150:0/2761970477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54dc157ca0 msgr2=0x7f54dc147410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:21.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.692+0000 7f54e3d63640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54dc157ca0 0x7f54dc147410 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f54d400e9a0 tx=0x7f54d400ee60 comp rx=0 tx=0).stop 2024-09-17T05:00:21.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.692+0000 7f54e3d63640 1 -- 172.21.15.150:0/2761970477 shutdown_connections 2024-09-17T05:00:21.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.692+0000 7f54e3d63640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f54c0073200 0x7f54c0075710 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.692+0000 7f54e3d63640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f54dc151e10 0x7f54dc1ed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.692+0000 7f54e3d63640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54dc157ca0 0x7f54dc147410 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.692+0000 7f54e3d63640 1 --2- 172.21.15.150:0/2761970477 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f54dc0ad7f0 0x7f54dc144dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:21.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.692+0000 7f54e3d63640 1 -- 172.21.15.150:0/2761970477 >> 172.21.15.150:0/2761970477 conn(0x7f54dc0c9de0 msgr2=0x7f54dc0c8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:21.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.692+0000 7f54e3d63640 1 -- 172.21.15.150:0/2761970477 shutdown_connections 2024-09-17T05:00:21.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:21.692+0000 7f54e3d63640 1 -- 172.21.15.150:0/2761970477 wait complete. 2024-09-17T05:00:21.713 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:00:20:961742+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 82, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 80, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6075": { "addr": "172.21.15.150:6833/2313391683", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2313391683, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 2313391683, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6075, "incarnation": 77, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T04:59:24.099259+0000", "qdb_cluster": [ 6075 ], "qdb_leader": 6075, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6075 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 82, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5375": { "addr": "172.21.15.176:6837/702608997", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 702608997, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 702608997, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5375, "incarnation": 82, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:00:20.961736+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5375 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:00:23.714 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:23.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.892+0000 7f3f233a4640 1 -- 172.21.15.150:0/4163772235 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3f1c06df60 msgr2=0x7f3f1c06e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:23.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.892+0000 7f3f233a4640 1 --2- 172.21.15.150:0/4163772235 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3f1c06df60 0x7f3f1c06e370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c009900 tx=0x7f3f0c01c7c0 comp rx=0 tx=0).stop 2024-09-17T05:00:23.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.892+0000 7f3f233a4640 1 -- 172.21.15.150:0/4163772235 shutdown_connections 2024-09-17T05:00:23.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.892+0000 7f3f233a4640 1 --2- 172.21.15.150:0/4163772235 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3f1c06df60 0x7f3f1c06e370 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:23.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.892+0000 7f3f233a4640 1 --2- 172.21.15.150:0/4163772235 >> v2:172.21.15.176:3300/0 conn(0x7f3f1c0ab9d0 0x7f3f1c14da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:23.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.892+0000 7f3f233a4640 1 -- 172.21.15.150:0/4163772235 >> 172.21.15.150:0/4163772235 conn(0x7f3f1c070b50 msgr2=0x7f3f1c070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:23.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.892+0000 7f3f233a4640 1 -- 172.21.15.150:0/4163772235 shutdown_connections 2024-09-17T05:00:23.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.892+0000 7f3f233a4640 1 -- 172.21.15.150:0/4163772235 wait complete. 2024-09-17T05:00:23.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f233a4640 1 Processor -- start 2024-09-17T05:00:23.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f233a4640 1 -- start start 2024-09-17T05:00:23.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f233a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f1c06df60 0x7f3f1c1f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:23.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f233a4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3f1c0ab9d0 0x7f3f1c0c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:23.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f233a4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f1c1536e0 0x7f3f1c146e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:23.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f233a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c154200 con 0x7f3f1c06df60 2024-09-17T05:00:23.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f233a4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c069e00 con 0x7f3f1c1536e0 2024-09-17T05:00:23.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f233a4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3f1c154380 con 0x7f3f1c0ab9d0 2024-09-17T05:00:23.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f21140640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f1c06df60 0x7f3f1c1f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:23.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f21140640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f1c06df60 0x7f3f1c1f0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32858/0 (socket says 172.21.15.150:32858) 2024-09-17T05:00:23.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f21140640 1 -- 172.21.15.150:0/2424650845 learned_addr learned my addr 172.21.15.150:0/2424650845 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:23.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f2093f640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3f1c0ab9d0 0x7f3f1c0c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:23.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f21941640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f1c1536e0 0x7f3f1c146e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:23.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f21140640 1 -- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3f1c0ab9d0 msgr2=0x7f3f1c0c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:23.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f21140640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3f1c0ab9d0 0x7f3f1c0c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:23.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f21140640 1 -- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f1c1536e0 msgr2=0x7f3f1c146e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:23.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f21140640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f1c1536e0 0x7f3f1c146e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:23.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f21140640 1 -- 172.21.15.150:0/2424650845 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f1c1503a0 con 0x7f3f1c06df60 2024-09-17T05:00:23.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f2093f640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3f1c0ab9d0 0x7f3f1c0c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:00:23.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f21140640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f1c06df60 0x7f3f1c1f0c90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800e7a0 tx=0x7f3f1800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:23.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.896+0000 7f3f11ffb640 1 -- 172.21.15.150:0/2424650845 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1800ccc0 con 0x7f3f1c06df60 2024-09-17T05:00:23.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.900+0000 7f3f11ffb640 1 -- 172.21.15.150:0/2424650845 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f18010430 con 0x7f3f1c06df60 2024-09-17T05:00:23.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.900+0000 7f3f11ffb640 1 -- 172.21.15.150:0/2424650845 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f18010860 con 0x7f3f1c06df60 2024-09-17T05:00:23.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.900+0000 7f3f233a4640 1 -- 172.21.15.150:0/2424650845 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f1c15d0f0 con 0x7f3f1c06df60 2024-09-17T05:00:23.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.900+0000 7f3f233a4640 1 -- 172.21.15.150:0/2424650845 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f1c15ce40 con 0x7f3f1c06df60 2024-09-17T05:00:23.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.900+0000 7f3f233a4640 1 -- 172.21.15.150:0/2424650845 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f1c06e370 con 0x7f3f1c06df60 2024-09-17T05:00:23.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.900+0000 7f3f11ffb640 1 -- 172.21.15.150:0/2424650845 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3f18010a00 con 0x7f3f1c06df60 2024-09-17T05:00:23.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.900+0000 7f3f11ffb640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3f00073230 0x7f3f00075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:23.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.900+0000 7f3f11ffb640 1 -- 172.21.15.150:0/2424650845 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7f3f180954b0 con 0x7f3f1c06df60 2024-09-17T05:00:23.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.900+0000 7f3f2093f640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3f00073230 0x7f3f00075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:23.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.904+0000 7f3f2093f640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3f00073230 0x7f3f00075740 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c0ae000 tx=0x7f3f0c01b000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:23.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:23.908+0000 7f3f11ffb640 1 -- 172.21.15.150:0/2424650845 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3f1805fcf0 con 0x7f3f1c06df60 2024-09-17T05:00:24.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.144+0000 7f3f233a4640 1 -- 172.21.15.150:0/2424650845 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f1c075060 con 0x7f3f1c06df60 2024-09-17T05:00:24.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.144+0000 7f3f11ffb640 1 -- 172.21.15.150:0/2424650845 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5710 (secure 0 0 0) 0x7f3f1805f5f0 con 0x7f3f1c06df60 2024-09-17T05:00:24.143 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:24.143 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":85,"btime":"2024-09-17T05:00:23:977041+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:24.143 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 85 2024-09-17T05:00:24.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.148+0000 7f3f233a4640 1 -- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3f00073230 msgr2=0x7f3f00075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:24.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.148+0000 7f3f233a4640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3f00073230 0x7f3f00075740 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c0ae000 tx=0x7f3f0c01b000 comp rx=0 tx=0).stop 2024-09-17T05:00:24.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.148+0000 7f3f233a4640 1 -- 172.21.15.150:0/2424650845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f1c06df60 msgr2=0x7f3f1c1f0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:24.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.148+0000 7f3f233a4640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f1c06df60 0x7f3f1c1f0c90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800e7a0 tx=0x7f3f1800ec60 comp rx=0 tx=0).stop 2024-09-17T05:00:24.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.148+0000 7f3f233a4640 1 -- 172.21.15.150:0/2424650845 shutdown_connections 2024-09-17T05:00:24.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.148+0000 7f3f233a4640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3f00073230 0x7f3f00075740 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:24.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.148+0000 7f3f233a4640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f1c1536e0 0x7f3f1c146e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:24.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.148+0000 7f3f233a4640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3f1c0ab9d0 0x7f3f1c0c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:24.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.148+0000 7f3f233a4640 1 --2- 172.21.15.150:0/2424650845 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f1c06df60 0x7f3f1c1f0c90 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:24.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.148+0000 7f3f233a4640 1 -- 172.21.15.150:0/2424650845 >> 172.21.15.150:0/2424650845 conn(0x7f3f1c070b50 msgr2=0x7f3f1c0aa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:24.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.148+0000 7f3f233a4640 1 -- 172.21.15.150:0/2424650845 shutdown_connections 2024-09-17T05:00:24.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:24.148+0000 7f3f233a4640 1 -- 172.21.15.150:0/2424650845 wait complete. 2024-09-17T05:00:24.166 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=5901 2024-09-17T05:00:24.166 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.c 2024-09-17T05:00:25.122 INFO:tasks.ceph.mds.d.smithi176.stderr:2024-09-17T05:00:25.116+0000 7f9350a5e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:00:26.168 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-17T05:00:26.168 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T05:00:26.168 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T05:00:26.171 INFO:tasks.ceph.mds.c:Started 2024-09-17T05:00:26.171 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:26.287 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T05:00:26.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.351+0000 7f2fdb3ee640 1 -- 172.21.15.150:0/1757001583 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fd406fca0 msgr2=0x7f2fd406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:26.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.351+0000 7f2fdb3ee640 1 --2- 172.21.15.150:0/1757001583 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fd406fca0 0x7f2fd406e090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009840 tx=0x7f2fc401c790 comp rx=0 tx=0).stop 2024-09-17T05:00:26.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.351+0000 7f2fdb3ee640 1 -- 172.21.15.150:0/1757001583 shutdown_connections 2024-09-17T05:00:26.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.351+0000 7f2fdb3ee640 1 --2- 172.21.15.150:0/1757001583 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fd406e5d0 0x7f2fd415dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:26.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.351+0000 7f2fdb3ee640 1 --2- 172.21.15.150:0/1757001583 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fd406fca0 0x7f2fd406e090 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:26.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.351+0000 7f2fdb3ee640 1 -- 172.21.15.150:0/1757001583 >> 172.21.15.150:0/1757001583 conn(0x7f2fd40c5ae0 msgr2=0x7f2fd4073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:26.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.351+0000 7f2fdb3ee640 1 -- 172.21.15.150:0/1757001583 shutdown_connections 2024-09-17T05:00:26.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.351+0000 7f2fdb3ee640 1 -- 172.21.15.150:0/1757001583 wait complete. 2024-09-17T05:00:26.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.351+0000 7f2fdb3ee640 1 Processor -- start 2024-09-17T05:00:26.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.351+0000 7f2fdb3ee640 1 -- start start 2024-09-17T05:00:26.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fdb3ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fd406e5d0 0x7f2fd40bf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:26.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fdb3ee640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fd406fca0 0x7f2fd41efc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:26.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fdb3ee640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fd40745c0 0x7f2fd40c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:26.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fdb3ee640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd4070610 con 0x7f2fd406e5d0 2024-09-17T05:00:26.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fdb3ee640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd4070790 con 0x7f2fd40745c0 2024-09-17T05:00:26.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fdb3ee640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2fd40ab140 con 0x7f2fd406fca0 2024-09-17T05:00:26.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fd918a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fd406e5d0 0x7f2fd40bf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:26.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fd918a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fd406e5d0 0x7f2fd40bf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32890/0 (socket says 172.21.15.150:32890) 2024-09-17T05:00:26.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fd918a640 1 -- 172.21.15.150:0/1473746062 learned_addr learned my addr 172.21.15.150:0/1473746062 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:26.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fd998b640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fd40745c0 0x7f2fd40c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:26.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fd8989640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fd406fca0 0x7f2fd41efc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:26.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fd918a640 1 -- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fd406fca0 msgr2=0x7f2fd41efc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:26.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fd918a640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fd406fca0 0x7f2fd41efc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:26.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fd918a640 1 -- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fd40745c0 msgr2=0x7f2fd40c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:26.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fd918a640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fd40745c0 0x7f2fd40c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:26.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fd918a640 1 -- 172.21.15.150:0/1473746062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd4153710 con 0x7f2fd406e5d0 2024-09-17T05:00:26.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fd8989640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fd406fca0 0x7f2fd41efc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:00:26.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fd918a640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fd406e5d0 0x7f2fd40bf090 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000c460 tx=0x7f2fd000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:26.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fc9ffb640 1 -- 172.21.15.150:0/1473746062 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd00052d0 con 0x7f2fd406e5d0 2024-09-17T05:00:26.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fc9ffb640 1 -- 172.21.15.150:0/1473746062 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2fd0005490 con 0x7f2fd406e5d0 2024-09-17T05:00:26.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fdb3ee640 1 -- 172.21.15.150:0/1473746062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd4152760 con 0x7f2fd406e5d0 2024-09-17T05:00:26.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fc9ffb640 1 -- 172.21.15.150:0/1473746062 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0012890 con 0x7f2fd406e5d0 2024-09-17T05:00:26.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fdb3ee640 1 -- 172.21.15.150:0/1473746062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd40bd9c0 con 0x7f2fd406e5d0 2024-09-17T05:00:26.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.355+0000 7f2fbf7fe640 1 -- 172.21.15.150:0/1473746062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fac0052a0 con 0x7f2fd406e5d0 2024-09-17T05:00:26.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.359+0000 7f2fc9ffb640 1 -- 172.21.15.150:0/1473746062 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2fd0012a30 con 0x7f2fd406e5d0 2024-09-17T05:00:26.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.359+0000 7f2fc9ffb640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2fb8073200 0x7f2fb8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:26.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.359+0000 7f2fc9ffb640 1 -- 172.21.15.150:0/1473746062 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7f2fd0094490 con 0x7f2fd406e5d0 2024-09-17T05:00:26.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.359+0000 7f2fd8989640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2fb8073200 0x7f2fb8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:26.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.359+0000 7f2fd8989640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2fb8073200 0x7f2fb8075710 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40af000 tx=0x7f2fc401afd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:26.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.363+0000 7f2fc9ffb640 1 -- 172.21.15.150:0/1473746062 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2fd005ed10 con 0x7f2fd406e5d0 2024-09-17T05:00:26.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 -- 172.21.15.150:0/1473746062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fac005860 con 0x7f2fd406e5d0 2024-09-17T05:00:26.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fc9ffb640 1 -- 172.21.15.150:0/1473746062 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5710 (secure 0 0 0) 0x7f2fd005e610 con 0x7f2fd406e5d0 2024-09-17T05:00:26.617 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:26.617 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":85,"btime":"2024-09-17T05:00:23:977041+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:26.618 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 85 2024-09-17T05:00:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 -- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2fb8073200 msgr2=0x7f2fb8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2fb8073200 0x7f2fb8075710 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40af000 tx=0x7f2fc401afd0 comp rx=0 tx=0).stop 2024-09-17T05:00:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 -- 172.21.15.150:0/1473746062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fd406e5d0 msgr2=0x7f2fd40bf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fd406e5d0 0x7f2fd40bf090 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000c460 tx=0x7f2fd000c920 comp rx=0 tx=0).stop 2024-09-17T05:00:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 -- 172.21.15.150:0/1473746062 shutdown_connections 2024-09-17T05:00:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2fb8073200 0x7f2fb8075710 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2fd40745c0 0x7f2fd40c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:26.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2fd406fca0 0x7f2fd41efc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:26.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 --2- 172.21.15.150:0/1473746062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fd406e5d0 0x7f2fd40bf090 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:26.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 -- 172.21.15.150:0/1473746062 >> 172.21.15.150:0/1473746062 conn(0x7f2fd40c5ae0 msgr2=0x7f2fd414c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:26.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 -- 172.21.15.150:0/1473746062 shutdown_connections 2024-09-17T05:00:26.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:26.619+0000 7f2fbf7fe640 1 -- 172.21.15.150:0/1473746062 wait complete. 2024-09-17T05:00:26.641 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-17T05:00:28.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:28.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.823+0000 7fb1d038b640 1 -- 172.21.15.150:0/1100856559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c80c5760 msgr2=0x7fb1c80c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:28.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.823+0000 7fb1d038b640 1 --2- 172.21.15.150:0/1100856559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c80c5760 0x7fb1c80c5b70 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8070370 tx=0x7fb1b801c830 comp rx=0 tx=0).stop 2024-09-17T05:00:28.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.823+0000 7fb1d038b640 1 -- 172.21.15.150:0/1100856559 shutdown_connections 2024-09-17T05:00:28.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.823+0000 7fb1d038b640 1 --2- 172.21.15.150:0/1100856559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c80c5760 0x7fb1c80c5b70 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:28.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.823+0000 7fb1d038b640 1 --2- 172.21.15.150:0/1100856559 >> v2:172.21.15.176:3300/0 conn(0x7fb1c81535b0 0x7fb1c8157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:28.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.823+0000 7fb1d038b640 1 -- 172.21.15.150:0/1100856559 >> 172.21.15.150:0/1100856559 conn(0x7fb1c80aaa80 msgr2=0x7fb1c80aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:28.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.823+0000 7fb1d038b640 1 -- 172.21.15.150:0/1100856559 shutdown_connections 2024-09-17T05:00:28.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.823+0000 7fb1d038b640 1 -- 172.21.15.150:0/1100856559 wait complete. 2024-09-17T05:00:28.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.823+0000 7fb1d038b640 1 Processor -- start 2024-09-17T05:00:28.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.823+0000 7fb1d038b640 1 -- start start 2024-09-17T05:00:28.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1d038b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c80a7f10 0x7fb1c814a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:28.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1d038b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1c80c5760 0x7fb1c814ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:28.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1d038b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1c81535b0 0x7fb1c81504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:28.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1d038b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c80718d0 con 0x7fb1c80a7f10 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1d038b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c80a8730 con 0x7fb1c80c5760 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1d038b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb1c80c5370 con 0x7fb1c81535b0 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1ce127640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c80a7f10 0x7fb1c814a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1ce127640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c80a7f10 0x7fb1c814a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32904/0 (socket says 172.21.15.150:32904) 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1ce127640 1 -- 172.21.15.150:0/395611499 learned_addr learned my addr 172.21.15.150:0/395611499 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1cd926640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1c80c5760 0x7fb1c814ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1ce928640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1c81535b0 0x7fb1c81504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1ce127640 1 -- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1c81535b0 msgr2=0x7fb1c81504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1ce127640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1c81535b0 0x7fb1c81504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1ce127640 1 -- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1c80c5760 msgr2=0x7fb1c814ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1ce127640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1c80c5760 0x7fb1c814ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1ce127640 1 -- 172.21.15.150:0/395611499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c80ac4e0 con 0x7fb1c80a7f10 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1ce127640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c80a7f10 0x7fb1c814a640 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009860 tx=0x7fb1c4009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:28.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1beffd640 1 -- 172.21.15.150:0/395611499 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4014070 con 0x7fb1c80a7f10 2024-09-17T05:00:28.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1beffd640 1 -- 172.21.15.150:0/395611499 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1c4009e50 con 0x7fb1c80a7f10 2024-09-17T05:00:28.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1d038b640 1 -- 172.21.15.150:0/395611499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c80ae930 con 0x7fb1c80a7f10 2024-09-17T05:00:28.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1beffd640 1 -- 172.21.15.150:0/395611499 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4007950 con 0x7fb1c80a7f10 2024-09-17T05:00:28.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1d038b640 1 -- 172.21.15.150:0/395611499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c814ad50 con 0x7fb1c80a7f10 2024-09-17T05:00:28.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1d038b640 1 -- 172.21.15.150:0/395611499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1c814ab80 con 0x7fb1c80a7f10 2024-09-17T05:00:28.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1beffd640 1 -- 172.21.15.150:0/395611499 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb1c400c070 con 0x7fb1c80a7f10 2024-09-17T05:00:28.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.827+0000 7fb1beffd640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1ac073200 0x7fb1ac075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:28.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.831+0000 7fb1beffd640 1 -- 172.21.15.150:0/395611499 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7fb1c4093d60 con 0x7fb1c80a7f10 2024-09-17T05:00:28.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.831+0000 7fb1cd926640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1ac073200 0x7fb1ac075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:28.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.831+0000 7fb1cd926640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1ac073200 0x7fb1ac075710 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80af000 tx=0x7fb1b801b070 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:28.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:28.835+0000 7fb1beffd640 1 -- 172.21.15.150:0/395611499 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb1c405df50 con 0x7fb1c80a7f10 2024-09-17T05:00:29.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.071+0000 7fb1d038b640 1 -- 172.21.15.150:0/395611499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1c81539c0 con 0x7fb1c80a7f10 2024-09-17T05:00:29.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.071+0000 7fb1beffd640 1 -- 172.21.15.150:0/395611499 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+6538 (secure 0 0 0) 0x7fb1c4062980 con 0x7fb1c80a7f10 2024-09-17T05:00:29.072 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:29.072 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":86,"btime":"2024-09-17T05:00:26:993373+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:29.072 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 86 2024-09-17T05:00:29.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.075+0000 7fb1d038b640 1 -- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1ac073200 msgr2=0x7fb1ac075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:29.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.075+0000 7fb1d038b640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1ac073200 0x7fb1ac075710 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb1b80af000 tx=0x7fb1b801b070 comp rx=0 tx=0).stop 2024-09-17T05:00:29.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.075+0000 7fb1d038b640 1 -- 172.21.15.150:0/395611499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c80a7f10 msgr2=0x7fb1c814a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:29.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.075+0000 7fb1d038b640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c80a7f10 0x7fb1c814a640 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009860 tx=0x7fb1c4009d20 comp rx=0 tx=0).stop 2024-09-17T05:00:29.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.075+0000 7fb1d038b640 1 -- 172.21.15.150:0/395611499 shutdown_connections 2024-09-17T05:00:29.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.075+0000 7fb1d038b640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1ac073200 0x7fb1ac075710 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:29.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.075+0000 7fb1d038b640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1c80c5760 0x7fb1c814ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:29.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.075+0000 7fb1d038b640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1c81535b0 0x7fb1c81504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:29.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.075+0000 7fb1d038b640 1 --2- 172.21.15.150:0/395611499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1c80a7f10 0x7fb1c814a640 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:29.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.075+0000 7fb1d038b640 1 -- 172.21.15.150:0/395611499 >> 172.21.15.150:0/395611499 conn(0x7fb1c80aaa80 msgr2=0x7fb1c815e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:29.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.075+0000 7fb1d038b640 1 -- 172.21.15.150:0/395611499 shutdown_connections 2024-09-17T05:00:29.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:29.075+0000 7fb1d038b640 1 -- 172.21.15.150:0/395611499 wait complete. 2024-09-17T05:00:29.096 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-17T05:00:29.096 INFO:tasks.mds_thrash.fs.[b]:waiting for 108 secs before thrashing 2024-09-17T05:00:30.171 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:30.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.327+0000 7fe20790f640 1 -- 172.21.15.150:0/2417633378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1ec004d50 msgr2=0x7fe1ec005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:30.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.327+0000 7fe20790f640 1 --2- 172.21.15.150:0/2417633378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe1ec004d50 0x7fe1ec005160 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe200075140 tx=0x7fe1f801d5b0 comp rx=0 tx=0).stop 2024-09-17T05:00:30.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.327+0000 7fe205eac640 1 -- 172.21.15.150:0/2417633378 reap_dead start 2024-09-17T05:00:30.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe20790f640 1 -- 172.21.15.150:0/2417633378 shutdown_connections 2024-09-17T05:00:30.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe20790f640 1 -- 172.21.15.150:0/2417633378 >> 172.21.15.150:0/2417633378 conn(0x7fe2000c91e0 msgr2=0x7fe2000c51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:30.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe20790f640 1 -- 172.21.15.150:0/2417633378 shutdown_connections 2024-09-17T05:00:30.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe20790f640 1 -- 172.21.15.150:0/2417633378 wait complete. 2024-09-17T05:00:30.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe20790f640 1 Processor -- start 2024-09-17T05:00:30.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe20790f640 1 -- start start 2024-09-17T05:00:30.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe20790f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1ec004d50 0x7fe200150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:30.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe20790f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20014fe90 0x7fe20014cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:30.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe20790f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2001463e0 0x7fe2001e0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:30.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe20790f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe2000c4d30 con 0x7fe20014fe90 2024-09-17T05:00:30.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe20790f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe200155670 con 0x7fe2001463e0 2024-09-17T05:00:30.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe20790f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe2000c4eb0 con 0x7fe1ec004d50 2024-09-17T05:00:30.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe204eaa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20014fe90 0x7fe20014cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:30.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe204eaa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20014fe90 0x7fe20014cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48570/0 (socket says 172.21.15.150:48570) 2024-09-17T05:00:30.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe204eaa640 1 -- 172.21.15.150:0/2194612144 learned_addr learned my addr 172.21.15.150:0/2194612144 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:30.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe2056ab640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1ec004d50 0x7fe200150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:30.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe205eac640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2001463e0 0x7fe2001e0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:30.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe204eaa640 1 -- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1ec004d50 msgr2=0x7fe200150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:30.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe204eaa640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1ec004d50 0x7fe200150590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:30.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe204eaa640 1 -- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2001463e0 msgr2=0x7fe2001e0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:30.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe204eaa640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2001463e0 0x7fe2001e0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:30.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe204eaa640 1 -- 172.21.15.150:0/2194612144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2001e1d00 con 0x7fe20014fe90 2024-09-17T05:00:30.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe2056ab640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1ec004d50 0x7fe200150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:00:30.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.331+0000 7fe204eaa640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20014fe90 0x7fe20014cf30 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000f420 tx=0x7fe1f000f8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:30.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.335+0000 7fe1f5ffb640 1 -- 172.21.15.150:0/2194612144 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0009cd0 con 0x7fe20014fe90 2024-09-17T05:00:30.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.335+0000 7fe1f5ffb640 1 -- 172.21.15.150:0/2194612144 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1f00080d0 con 0x7fe20014fe90 2024-09-17T05:00:30.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.335+0000 7fe20790f640 1 -- 172.21.15.150:0/2194612144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe20014d7e0 con 0x7fe20014fe90 2024-09-17T05:00:30.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.335+0000 7fe1f5ffb640 1 -- 172.21.15.150:0/2194612144 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0008500 con 0x7fe20014fe90 2024-09-17T05:00:30.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.335+0000 7fe20790f640 1 -- 172.21.15.150:0/2194612144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe200146c20 con 0x7fe20014fe90 2024-09-17T05:00:30.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.335+0000 7fe20790f640 1 -- 172.21.15.150:0/2194612144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2000c2a40 con 0x7fe20014fe90 2024-09-17T05:00:30.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.335+0000 7fe1f5ffb640 1 -- 172.21.15.150:0/2194612144 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe1f000a030 con 0x7fe20014fe90 2024-09-17T05:00:30.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.335+0000 7fe1f5ffb640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe1e4073230 0x7fe1e4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:30.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.335+0000 7fe1f5ffb640 1 -- 172.21.15.150:0/2194612144 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7fe1f00974f0 con 0x7fe20014fe90 2024-09-17T05:00:30.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.335+0000 7fe2056ab640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe1e4073230 0x7fe1e4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:30.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.335+0000 7fe2056ab640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe1e4073230 0x7fe1e4075740 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009b80 tx=0x7fe1fc0093a0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:30.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.343+0000 7fe1f5ffb640 1 -- 172.21.15.150:0/2194612144 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe1f0061db0 con 0x7fe20014fe90 2024-09-17T05:00:30.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.571+0000 7fe20790f640 1 -- 172.21.15.150:0/2194612144 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe200154650 con 0x7fe20014fe90 2024-09-17T05:00:30.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.571+0000 7fe1f5ffb640 1 -- 172.21.15.150:0/2194612144 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+6538 (secure 0 0 0) 0x7fe1f00616b0 con 0x7fe20014fe90 2024-09-17T05:00:30.570 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:30.570 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":86,"btime":"2024-09-17T05:00:26:993373+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:30.570 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 86 2024-09-17T05:00:30.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.575+0000 7fe20790f640 1 -- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe1e4073230 msgr2=0x7fe1e4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:30.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.575+0000 7fe20790f640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe1e4073230 0x7fe1e4075740 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009b80 tx=0x7fe1fc0093a0 comp rx=0 tx=0).stop 2024-09-17T05:00:30.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.575+0000 7fe20790f640 1 -- 172.21.15.150:0/2194612144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20014fe90 msgr2=0x7fe20014cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:30.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.575+0000 7fe20790f640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20014fe90 0x7fe20014cf30 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000f420 tx=0x7fe1f000f8e0 comp rx=0 tx=0).stop 2024-09-17T05:00:30.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.575+0000 7fe20790f640 1 -- 172.21.15.150:0/2194612144 shutdown_connections 2024-09-17T05:00:30.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.575+0000 7fe20790f640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe1e4073230 0x7fe1e4075740 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:30.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.575+0000 7fe20790f640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2001463e0 0x7fe2001e0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:30.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.575+0000 7fe20790f640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe20014fe90 0x7fe20014cf30 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:30.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.575+0000 7fe20790f640 1 --2- 172.21.15.150:0/2194612144 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe1ec004d50 0x7fe200150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:30.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.575+0000 7fe20790f640 1 -- 172.21.15.150:0/2194612144 >> 172.21.15.150:0/2194612144 conn(0x7fe2000c91e0 msgr2=0x7fe200146fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:30.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.575+0000 7fe20790f640 1 -- 172.21.15.150:0/2194612144 shutdown_connections 2024-09-17T05:00:30.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.575+0000 7fe20790f640 1 -- 172.21.15.150:0/2194612144 wait complete. 2024-09-17T05:00:30.596 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-17T05:00:30.596 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T05:00:30.596 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:00:30.610 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T05:00:30.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:30.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 -- 172.21.15.150:0/3271086657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f002c004d50 msgr2=0x7f002c005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:30.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 --2- 172.21.15.150:0/3271086657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f002c004d50 0x7f002c005220 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f00340013e0 tx=0x7f003401d2a0 comp rx=0 tx=0).stop 2024-09-17T05:00:30.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f004138e640 1 -- 172.21.15.150:0/3271086657 reap_dead start 2024-09-17T05:00:30.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 -- 172.21.15.150:0/3271086657 shutdown_connections 2024-09-17T05:00:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 -- 172.21.15.150:0/3271086657 >> 172.21.15.150:0/3271086657 conn(0x7f003c06e360 msgr2=0x7f003c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 -- 172.21.15.150:0/3271086657 shutdown_connections 2024-09-17T05:00:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 -- 172.21.15.150:0/3271086657 wait complete. 2024-09-17T05:00:30.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 Processor -- start 2024-09-17T05:00:30.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 -- start start 2024-09-17T05:00:30.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f003c14a3f0 0x7f003c06d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:30.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f003c1df9d0 0x7f003c1dfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:30.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f003c1eb680 0x7f003c1eba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:30.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f003c071120 con 0x7f003c1df9d0 2024-09-17T05:00:30.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f003c070fa0 con 0x7f003c1eb680 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f0042df1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f003c072db0 con 0x7f003c14a3f0 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f003bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f003c1df9d0 0x7f003c1dfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f003bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f003c1df9d0 0x7f003c1dfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48590/0 (socket says 172.21.15.150:48590) 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.815+0000 7f003bfff640 1 -- 172.21.15.150:0/693841820 learned_addr learned my addr 172.21.15.150:0/693841820 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f004138e640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f003c1eb680 0x7f003c1eba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f0040b8d640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f003c14a3f0 0x7f003c06d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f003bfff640 1 -- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f003c14a3f0 msgr2=0x7f003c06d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f003bfff640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f003c14a3f0 0x7f003c06d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f003bfff640 1 -- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f003c1eb680 msgr2=0x7f003c1eba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f003bfff640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f003c1eb680 0x7f003c1eba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f003bfff640 1 -- 172.21.15.150:0/693841820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003c14a110 con 0x7f003c1df9d0 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f003bfff640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f003c1df9d0 0x7f003c1dfde0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0028011850 tx=0x7f0028011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f00397fa640 1 -- 172.21.15.150:0/693841820 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028004440 con 0x7f003c1df9d0 2024-09-17T05:00:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f0042df1640 1 -- 172.21.15.150:0/693841820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f003c1ee3a0 con 0x7f003c1df9d0 2024-09-17T05:00:30.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f00397fa640 1 -- 172.21.15.150:0/693841820 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f002800fa30 con 0x7f003c1df9d0 2024-09-17T05:00:30.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f00397fa640 1 -- 172.21.15.150:0/693841820 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002800fe80 con 0x7f003c1df9d0 2024-09-17T05:00:30.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f0042df1640 1 -- 172.21.15.150:0/693841820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f003c06d560 con 0x7f003c1df9d0 2024-09-17T05:00:30.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f0042df1640 1 -- 172.21.15.150:0/693841820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f003c1ee190 con 0x7f003c1df9d0 2024-09-17T05:00:30.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f00397fa640 1 -- 172.21.15.150:0/693841820 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0028003600 con 0x7f003c1df9d0 2024-09-17T05:00:30.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f00397fa640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0020073200 0x7f0020075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:30.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f00397fa640 1 -- 172.21.15.150:0/693841820 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7f0028095620 con 0x7f003c1df9d0 2024-09-17T05:00:30.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.819+0000 7f0040b8d640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0020073200 0x7f0020075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:30.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.823+0000 7f0040b8d640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0020073200 0x7f0020075710 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f003c1ed6e0 tx=0x7f0030009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:30.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:30.823+0000 7f00397fa640 1 -- 172.21.15.150:0/693841820 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f002805fe70 con 0x7f003c1df9d0 2024-09-17T05:00:30.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:00:31.000 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:00:31.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.043+0000 7f0042df1640 1 -- 172.21.15.150:0/693841820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f003c0c8800 con 0x7f003c1df9d0 2024-09-17T05:00:31.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f00397fa640 1 -- 172.21.15.150:0/693841820 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+6538 (secure 0 0 0) 0x7f002805f770 con 0x7f003c1df9d0 2024-09-17T05:00:31.045 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:31.045 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":86,"btime":"2024-09-17T05:00:26:993373+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:31.046 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 86 2024-09-17T05:00:31.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f0042df1640 1 -- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0020073200 msgr2=0x7f0020075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:31.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f0042df1640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0020073200 0x7f0020075710 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f003c1ed6e0 tx=0x7f0030009040 comp rx=0 tx=0).stop 2024-09-17T05:00:31.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f0042df1640 1 -- 172.21.15.150:0/693841820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f003c1df9d0 msgr2=0x7f003c1dfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f0042df1640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f003c1df9d0 0x7f003c1dfde0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f0028011850 tx=0x7f0028011d10 comp rx=0 tx=0).stop 2024-09-17T05:00:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f0042df1640 1 -- 172.21.15.150:0/693841820 shutdown_connections 2024-09-17T05:00:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f0042df1640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0020073200 0x7f0020075710 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f0042df1640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f003c1eb680 0x7f003c1eba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f0042df1640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f003c1df9d0 0x7f003c1dfde0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f0042df1640 1 --2- 172.21.15.150:0/693841820 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f003c14a3f0 0x7f003c06d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f0042df1640 1 -- 172.21.15.150:0/693841820 >> 172.21.15.150:0/693841820 conn(0x7f003c06e360 msgr2=0x7f003c0a9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f0042df1640 1 -- 172.21.15.150:0/693841820 shutdown_connections 2024-09-17T05:00:31.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:31.047+0000 7f0042df1640 1 -- 172.21.15.150:0/693841820 wait complete. 2024-09-17T05:00:31.069 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:00:33.070 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:33.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.239+0000 7faa46e25640 1 -- 172.21.15.150:0/2415821297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa400c42e0 msgr2=0x7faa400c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.239+0000 7faa46e25640 1 --2- 172.21.15.150:0/2415821297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa400c42e0 0x7faa400c6690 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7faa30009900 tx=0x7faa3001c8e0 comp rx=0 tx=0).stop 2024-09-17T05:00:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.239+0000 7faa46e25640 1 -- 172.21.15.150:0/2415821297 shutdown_connections 2024-09-17T05:00:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.239+0000 7faa46e25640 1 --2- 172.21.15.150:0/2415821297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa400c42e0 0x7faa400c6690 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.239+0000 7faa46e25640 1 -- 172.21.15.150:0/2415821297 >> 172.21.15.150:0/2415821297 conn(0x7faa400c4a00 msgr2=0x7faa400c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.239+0000 7faa46e25640 1 -- 172.21.15.150:0/2415821297 shutdown_connections 2024-09-17T05:00:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.239+0000 7faa46e25640 1 -- 172.21.15.150:0/2415821297 wait complete. 2024-09-17T05:00:33.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.239+0000 7faa46e25640 1 Processor -- start 2024-09-17T05:00:33.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.239+0000 7faa46e25640 1 -- start start 2024-09-17T05:00:33.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa46e25640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faa400a7350 0x7faa4014c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:33.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa46e25640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa400c42e0 0x7faa401efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:33.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa46e25640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faa40152cf0 0x7faa40156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:33.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa46e25640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7faa400c6d50 con 0x7faa400c42e0 2024-09-17T05:00:33.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa46e25640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7faa400c6bd0 con 0x7faa40152cf0 2024-09-17T05:00:33.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa46e25640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faa40150770 con 0x7faa400a7350 2024-09-17T05:00:33.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa400c42e0 0x7faa401efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa400c42e0 0x7faa401efe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48608/0 (socket says 172.21.15.150:48608) 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3ffff640 1 -- 172.21.15.150:0/2091858277 learned_addr learned my addr 172.21.15.150:0/2091858277 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa453c2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faa40152cf0 0x7faa40156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa44bc1640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faa400a7350 0x7faa4014c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3ffff640 1 -- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faa400a7350 msgr2=0x7faa4014c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3ffff640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faa400a7350 0x7faa4014c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3ffff640 1 -- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faa40152cf0 msgr2=0x7faa40156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3ffff640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faa40152cf0 0x7faa40156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3ffff640 1 -- 172.21.15.150:0/2091858277 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa401ef920 con 0x7faa400c42e0 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa44bc1640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faa400a7350 0x7faa4014c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3ffff640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa400c42e0 0x7faa401efe00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7faa3400c3f0 tx=0x7faa3400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3d7fa640 1 -- 172.21.15.150:0/2091858277 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34017070 con 0x7faa400c42e0 2024-09-17T05:00:33.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3d7fa640 1 -- 172.21.15.150:0/2091858277 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa340053c0 con 0x7faa400c42e0 2024-09-17T05:00:33.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa46e25640 1 -- 172.21.15.150:0/2091858277 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa401543a0 con 0x7faa400c42e0 2024-09-17T05:00:33.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3d7fa640 1 -- 172.21.15.150:0/2091858277 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34003860 con 0x7faa400c42e0 2024-09-17T05:00:33.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa46e25640 1 -- 172.21.15.150:0/2091858277 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa40155f50 con 0x7faa400c42e0 2024-09-17T05:00:33.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa46e25640 1 -- 172.21.15.150:0/2091858277 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa400c6690 con 0x7faa400c42e0 2024-09-17T05:00:33.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3d7fa640 1 -- 172.21.15.150:0/2091858277 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7faa3401a070 con 0x7faa400c42e0 2024-09-17T05:00:33.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3d7fa640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faa2c073230 0x7faa2c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:33.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.243+0000 7faa3d7fa640 1 -- 172.21.15.150:0/2091858277 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7faa34093f80 con 0x7faa400c42e0 2024-09-17T05:00:33.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.247+0000 7faa44bc1640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faa2c073230 0x7faa2c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:33.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.247+0000 7faa44bc1640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faa2c073230 0x7faa2c075740 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faa300af000 tx=0x7faa3001b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:33.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.247+0000 7faa3d7fa640 1 -- 172.21.15.150:0/2091858277 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7faa34021020 con 0x7faa400c42e0 2024-09-17T05:00:33.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.463+0000 7faa46e25640 1 -- 172.21.15.150:0/2091858277 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa400c3770 con 0x7faa400c42e0 2024-09-17T05:00:33.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.463+0000 7faa3d7fa640 1 -- 172.21.15.150:0/2091858277 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+6538 (secure 0 0 0) 0x7faa3405e2e0 con 0x7faa400c42e0 2024-09-17T05:00:33.462 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:33.462 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":86,"btime":"2024-09-17T05:00:26:993373+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:33.463 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 86 2024-09-17T05:00:33.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.463+0000 7faa46e25640 1 -- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faa2c073230 msgr2=0x7faa2c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:33.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.463+0000 7faa46e25640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faa2c073230 0x7faa2c075740 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faa300af000 tx=0x7faa3001b120 comp rx=0 tx=0).stop 2024-09-17T05:00:33.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.463+0000 7faa46e25640 1 -- 172.21.15.150:0/2091858277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa400c42e0 msgr2=0x7faa401efe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:33.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.463+0000 7faa46e25640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa400c42e0 0x7faa401efe00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7faa3400c3f0 tx=0x7faa3400c8b0 comp rx=0 tx=0).stop 2024-09-17T05:00:33.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.467+0000 7faa46e25640 1 -- 172.21.15.150:0/2091858277 shutdown_connections 2024-09-17T05:00:33.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.467+0000 7faa46e25640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faa2c073230 0x7faa2c075740 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:33.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.467+0000 7faa46e25640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faa40152cf0 0x7faa40156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:33.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.467+0000 7faa46e25640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa400c42e0 0x7faa401efe00 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:33.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.467+0000 7faa46e25640 1 --2- 172.21.15.150:0/2091858277 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faa400a7350 0x7faa4014c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:33.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.467+0000 7faa46e25640 1 -- 172.21.15.150:0/2091858277 >> 172.21.15.150:0/2091858277 conn(0x7faa400c4a00 msgr2=0x7faa400744e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:33.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.467+0000 7faa46e25640 1 -- 172.21.15.150:0/2091858277 shutdown_connections 2024-09-17T05:00:33.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:33.467+0000 7faa46e25640 1 -- 172.21.15.150:0/2091858277 wait complete. 2024-09-17T05:00:33.484 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:00:35.484 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:35.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.651+0000 7ffa5ec14640 1 -- 172.21.15.150:0/3423160530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa40004d50 msgr2=0x7ffa40005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:35.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.651+0000 7ffa5ec14640 1 --2- 172.21.15.150:0/3423160530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa40004d50 0x7ffa40005200 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009e90 tx=0x7ffa5001d5d0 comp rx=0 tx=0).stop 2024-09-17T05:00:35.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.651+0000 7ffa5d1b1640 1 -- 172.21.15.150:0/3423160530 reap_dead start 2024-09-17T05:00:35.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.651+0000 7ffa5ec14640 1 -- 172.21.15.150:0/3423160530 shutdown_connections 2024-09-17T05:00:35.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.651+0000 7ffa5ec14640 1 -- 172.21.15.150:0/3423160530 >> 172.21.15.150:0/3423160530 conn(0x7ffa580c4a00 msgr2=0x7ffa580c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:35.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.651+0000 7ffa5ec14640 1 -- 172.21.15.150:0/3423160530 shutdown_connections 2024-09-17T05:00:35.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.651+0000 7ffa5ec14640 1 -- 172.21.15.150:0/3423160530 wait complete. 2024-09-17T05:00:35.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5ec14640 1 Processor -- start 2024-09-17T05:00:35.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5ec14640 1 -- start start 2024-09-17T05:00:35.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5ec14640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa581506f0 0x7ffa581edfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:35.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5ec14640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa581ee4f0 0x7ffa580ab5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:35.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5ec14640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa581ee900 0x7ffa580abaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:35.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5ec14640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffa580adce0 con 0x7ffa581ee900 2024-09-17T05:00:35.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5ec14640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5806f810 con 0x7ffa581ee4f0 2024-09-17T05:00:35.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5ec14640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffa580ade60 con 0x7ffa581506f0 2024-09-17T05:00:35.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5d1b1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa581ee900 0x7ffa580abaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:35.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5d1b1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa581ee900 0x7ffa580abaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48614/0 (socket says 172.21.15.150:48614) 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5d1b1640 1 -- 172.21.15.150:0/276395833 learned_addr learned my addr 172.21.15.150:0/276395833 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa57fff640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa581ee4f0 0x7ffa580ab5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5c9b0640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa581506f0 0x7ffa581edfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5d1b1640 1 -- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa581506f0 msgr2=0x7ffa581edfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5d1b1640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa581506f0 0x7ffa581edfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5d1b1640 1 -- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa581ee4f0 msgr2=0x7ffa580ab5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5d1b1640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa581ee4f0 0x7ffa580ab5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5d1b1640 1 -- 172.21.15.150:0/276395833 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa581566a0 con 0x7ffa581ee900 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5d1b1640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa581ee900 0x7ffa580abaf0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009e90 tx=0x7ffa50004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa557fa640 1 -- 172.21.15.150:0/276395833 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50009a90 con 0x7ffa581ee900 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa557fa640 1 -- 172.21.15.150:0/276395833 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa500a9510 con 0x7ffa581ee900 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa557fa640 1 -- 172.21.15.150:0/276395833 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa500a9b00 con 0x7ffa581ee900 2024-09-17T05:00:35.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5ec14640 1 -- 172.21.15.150:0/276395833 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa580ad050 con 0x7ffa581ee900 2024-09-17T05:00:35.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5ec14640 1 -- 172.21.15.150:0/276395833 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa58157870 con 0x7ffa581ee900 2024-09-17T05:00:35.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.655+0000 7ffa5ec14640 1 -- 172.21.15.150:0/276395833 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa58154020 con 0x7ffa581ee900 2024-09-17T05:00:35.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.659+0000 7ffa557fa640 1 -- 172.21.15.150:0/276395833 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ffa500a7040 con 0x7ffa581ee900 2024-09-17T05:00:35.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.659+0000 7ffa557fa640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa44073200 0x7ffa44075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:35.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.659+0000 7ffa557fa640 1 -- 172.21.15.150:0/276395833 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7ffa5012faa0 con 0x7ffa581ee900 2024-09-17T05:00:35.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.659+0000 7ffa5c9b0640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa44073200 0x7ffa44075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:35.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.659+0000 7ffa5c9b0640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa44073200 0x7ffa44075710 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffa48001250 tx=0x7ffa480011e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:35.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.663+0000 7ffa557fa640 1 -- 172.21.15.150:0/276395833 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ffa500ad070 con 0x7ffa581ee900 2024-09-17T05:00:35.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.871+0000 7ffa5ec14640 1 -- 172.21.15.150:0/276395833 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa580c3a70 con 0x7ffa581ee900 2024-09-17T05:00:35.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.871+0000 7ffa557fa640 1 -- 172.21.15.150:0/276395833 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+6538 (secure 0 0 0) 0x7ffa500172e0 con 0x7ffa581ee900 2024-09-17T05:00:35.870 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:35.870 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":86,"btime":"2024-09-17T05:00:26:993373+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:35.871 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 86 2024-09-17T05:00:35.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.875+0000 7ffa5ec14640 1 -- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa44073200 msgr2=0x7ffa44075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:35.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.875+0000 7ffa5ec14640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa44073200 0x7ffa44075710 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffa48001250 tx=0x7ffa480011e0 comp rx=0 tx=0).stop 2024-09-17T05:00:35.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.875+0000 7ffa5ec14640 1 -- 172.21.15.150:0/276395833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa581ee900 msgr2=0x7ffa580abaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:35.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.875+0000 7ffa5ec14640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa581ee900 0x7ffa580abaf0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ffa50009e90 tx=0x7ffa50004f50 comp rx=0 tx=0).stop 2024-09-17T05:00:35.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.875+0000 7ffa5ec14640 1 -- 172.21.15.150:0/276395833 shutdown_connections 2024-09-17T05:00:35.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.875+0000 7ffa5ec14640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa44073200 0x7ffa44075710 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:35.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.875+0000 7ffa5ec14640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa581ee4f0 0x7ffa580ab5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:35.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.875+0000 7ffa5ec14640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa581ee900 0x7ffa580abaf0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:35.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.875+0000 7ffa5ec14640 1 --2- 172.21.15.150:0/276395833 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa581506f0 0x7ffa581edfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:35.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.875+0000 7ffa5ec14640 1 -- 172.21.15.150:0/276395833 >> 172.21.15.150:0/276395833 conn(0x7ffa580c4a00 msgr2=0x7ffa580c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:35.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.875+0000 7ffa5ec14640 1 -- 172.21.15.150:0/276395833 shutdown_connections 2024-09-17T05:00:35.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:35.875+0000 7ffa5ec14640 1 -- 172.21.15.150:0/276395833 wait complete. 2024-09-17T05:00:35.893 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:00:37.894 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:38.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.067+0000 7f28f311c640 1 -- 172.21.15.150:0/3884857803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ec0c3b40 msgr2=0x7f28ec157c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:38.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.067+0000 7f28f311c640 1 --2- 172.21.15.150:0/3884857803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ec0c3b40 0x7f28ec157c30 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f28e0009900 tx=0x7f28e001c880 comp rx=0 tx=0).stop 2024-09-17T05:00:38.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.067+0000 7f28f311c640 1 -- 172.21.15.150:0/3884857803 shutdown_connections 2024-09-17T05:00:38.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.067+0000 7f28f311c640 1 --2- 172.21.15.150:0/3884857803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ec158170 0x7f28ec158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:38.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.067+0000 7f28f311c640 1 --2- 172.21.15.150:0/3884857803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ec0c3b40 0x7f28ec157c30 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:38.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.067+0000 7f28f311c640 1 -- 172.21.15.150:0/3884857803 >> 172.21.15.150:0/3884857803 conn(0x7f28ec0c4a00 msgr2=0x7f28ec0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.067+0000 7f28f311c640 1 -- 172.21.15.150:0/3884857803 shutdown_connections 2024-09-17T05:00:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.067+0000 7f28f311c640 1 -- 172.21.15.150:0/3884857803 wait complete. 2024-09-17T05:00:38.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28f311c640 1 Processor -- start 2024-09-17T05:00:38.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28f311c640 1 -- start start 2024-09-17T05:00:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28f311c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ec0c3b40 0x7f28ec1eda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28f311c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ec151d00 0x7f28ec14ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28f311c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28ec158170 0x7f28ec0c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28f311c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec0c3f50 con 0x7f28ec151d00 2024-09-17T05:00:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28f311c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec0adec0 con 0x7f28ec158170 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28f311c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f28ec0c40d0 con 0x7f28ec0c3b40 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28ebfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ec151d00 0x7f28ec14ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28ebfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ec151d00 0x7f28ec14ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48626/0 (socket says 172.21.15.150:48626) 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28ebfff640 1 -- 172.21.15.150:0/1253323189 learned_addr learned my addr 172.21.15.150:0/1253323189 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28f16b9640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28ec158170 0x7f28ec0c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28f0eb8640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ec0c3b40 0x7f28ec1eda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28ebfff640 1 -- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ec0c3b40 msgr2=0x7f28ec1eda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28ebfff640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ec0c3b40 0x7f28ec1eda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28ebfff640 1 -- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28ec158170 msgr2=0x7f28ec0c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28ebfff640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28ec158170 0x7f28ec0c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28ebfff640 1 -- 172.21.15.150:0/1253323189 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ec149750 con 0x7f28ec151d00 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28ebfff640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ec151d00 0x7f28ec14ec60 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f28e00b0040 tx=0x7f28e00a6780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28e97fa640 1 -- 172.21.15.150:0/1253323189 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e0004400 con 0x7f28ec151d00 2024-09-17T05:00:38.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28e97fa640 1 -- 172.21.15.150:0/1253323189 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28e00bc070 con 0x7f28ec151d00 2024-09-17T05:00:38.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28f311c640 1 -- 172.21.15.150:0/1253323189 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28ec1f11b0 con 0x7f28ec151d00 2024-09-17T05:00:38.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28e97fa640 1 -- 172.21.15.150:0/1253323189 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e0005290 con 0x7f28ec151d00 2024-09-17T05:00:38.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.071+0000 7f28f311c640 1 -- 172.21.15.150:0/1253323189 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28ec0c69a0 con 0x7f28ec151d00 2024-09-17T05:00:38.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.075+0000 7f28f311c640 1 -- 172.21.15.150:0/1253323189 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28ec1f0fa0 con 0x7f28ec151d00 2024-09-17T05:00:38.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.075+0000 7f28e97fa640 1 -- 172.21.15.150:0/1253323189 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f28e0005430 con 0x7f28ec151d00 2024-09-17T05:00:38.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.075+0000 7f28e97fa640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f28d8073230 0x7f28d8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:38.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.075+0000 7f28e97fa640 1 -- 172.21.15.150:0/1253323189 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7f28e012f8b0 con 0x7f28ec151d00 2024-09-17T05:00:38.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.075+0000 7f28f0eb8640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f28d8073230 0x7f28d8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:38.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.075+0000 7f28f0eb8640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f28d8073230 0x7f28d8075740 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00b810 tx=0x7f28dc008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:38.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.079+0000 7f28e97fa640 1 -- 172.21.15.150:0/1253323189 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f28e00164d0 con 0x7f28ec151d00 2024-09-17T05:00:38.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.283+0000 7f28f311c640 1 -- 172.21.15.150:0/1253323189 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28ec157c30 con 0x7f28ec151d00 2024-09-17T05:00:38.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28e97fa640 1 -- 172.21.15.150:0/1253323189 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+6538 (secure 0 0 0) 0x7f28e00fa070 con 0x7f28ec151d00 2024-09-17T05:00:38.285 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:38.285 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":86,"btime":"2024-09-17T05:00:26:993373+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:38.286 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 86 2024-09-17T05:00:38.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28f311c640 1 -- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f28d8073230 msgr2=0x7f28d8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:38.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28f311c640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f28d8073230 0x7f28d8075740 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00b810 tx=0x7f28dc008040 comp rx=0 tx=0).stop 2024-09-17T05:00:38.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28f311c640 1 -- 172.21.15.150:0/1253323189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ec151d00 msgr2=0x7f28ec14ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:38.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28f311c640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ec151d00 0x7f28ec14ec60 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f28e00b0040 tx=0x7f28e00a6780 comp rx=0 tx=0).stop 2024-09-17T05:00:38.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28f311c640 1 -- 172.21.15.150:0/1253323189 shutdown_connections 2024-09-17T05:00:38.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28f311c640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f28d8073230 0x7f28d8075740 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:38.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28f311c640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28ec158170 0x7f28ec0c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:38.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28f311c640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ec151d00 0x7f28ec14ec60 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:38.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28f311c640 1 --2- 172.21.15.150:0/1253323189 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ec0c3b40 0x7f28ec1eda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:38.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28f311c640 1 -- 172.21.15.150:0/1253323189 >> 172.21.15.150:0/1253323189 conn(0x7f28ec0c4a00 msgr2=0x7f28ec0c2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:38.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28f311c640 1 -- 172.21.15.150:0/1253323189 shutdown_connections 2024-09-17T05:00:38.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:38.287+0000 7f28f311c640 1 -- 172.21.15.150:0/1253323189 wait complete. 2024-09-17T05:00:38.308 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:00:40.310 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:40.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.479+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/3519505794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb40005360 msgr2=0x7fbb4000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:40.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.479+0000 7fbb5c9e3640 1 --2- 172.21.15.150:0/3519505794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb40005360 0x7fbb4000b9d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000c610 tx=0x7fbb5001fa00 comp rx=0 tx=0).stop 2024-09-17T05:00:40.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.479+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/3519505794 shutdown_connections 2024-09-17T05:00:40.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.479+0000 7fbb5c9e3640 1 --2- 172.21.15.150:0/3519505794 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb40005360 0x7fbb4000b9d0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:40.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.479+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/3519505794 >> 172.21.15.150:0/3519505794 conn(0x7fbb580aaa80 msgr2=0x7fbb580aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:40.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.479+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/3519505794 shutdown_connections 2024-09-17T05:00:40.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.479+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/3519505794 wait complete. 2024-09-17T05:00:40.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb5c9e3640 1 Processor -- start 2024-09-17T05:00:40.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb5c9e3640 1 -- start start 2024-09-17T05:00:40.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb5c9e3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb40005360 0x7fbb58074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:40.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb5c9e3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580c0350 0x7fbb5806ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:40.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb5c9e3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb580c0760 0x7fbb5806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb5c9e3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbb58151160 con 0x7fbb580c0350 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb5c9e3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbb58156c40 con 0x7fbb580c0760 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb5c9e3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbb580c5370 con 0x7fbb40005360 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb55d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580c0350 0x7fbb5806ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb55d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580c0350 0x7fbb5806ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45864/0 (socket says 172.21.15.150:45864) 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb55d9b640 1 -- 172.21.15.150:0/233166076 learned_addr learned my addr 172.21.15.150:0/233166076 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb5659c640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb40005360 0x7fbb58074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb56d9d640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb580c0760 0x7fbb5806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb55d9b640 1 -- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb40005360 msgr2=0x7fbb58074440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb55d9b640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb40005360 0x7fbb58074440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb55d9b640 1 -- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb580c0760 msgr2=0x7fbb5806d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb55d9b640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb580c0760 0x7fbb5806d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:40.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb55d9b640 1 -- 172.21.15.150:0/233166076 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb580c1ad0 con 0x7fbb580c0350 2024-09-17T05:00:40.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb55d9b640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580c0350 0x7fbb5806ce50 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00c3e0 tx=0x7fbb4c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:40.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb46ffd640 1 -- 172.21.15.150:0/233166076 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c017070 con 0x7fbb580c0350 2024-09-17T05:00:40.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb46ffd640 1 -- 172.21.15.150:0/233166076 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb4c0053c0 con 0x7fbb580c0350 2024-09-17T05:00:40.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/233166076 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb581edd20 con 0x7fbb580c0350 2024-09-17T05:00:40.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb46ffd640 1 -- 172.21.15.150:0/233166076 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c003840 con 0x7fbb580c0350 2024-09-17T05:00:40.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/233166076 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb58150440 con 0x7fbb580c0350 2024-09-17T05:00:40.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.483+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/233166076 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb581539c0 con 0x7fbb580c0350 2024-09-17T05:00:40.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.487+0000 7fbb46ffd640 1 -- 172.21.15.150:0/233166076 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbb4c01a070 con 0x7fbb580c0350 2024-09-17T05:00:40.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.487+0000 7fbb46ffd640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb38073230 0x7fbb38075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:40.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.487+0000 7fbb46ffd640 1 -- 172.21.15.150:0/233166076 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7fbb4c094260 con 0x7fbb580c0350 2024-09-17T05:00:40.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.487+0000 7fbb5659c640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb38073230 0x7fbb38075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:40.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.487+0000 7fbb5659c640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb38073230 0x7fbb38075740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009b40 tx=0x7fbb480091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:40.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.491+0000 7fbb46ffd640 1 -- 172.21.15.150:0/233166076 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbb4c021020 con 0x7fbb580c0350 2024-09-17T05:00:40.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.691+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/233166076 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb580c3070 con 0x7fbb580c0350 2024-09-17T05:00:40.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb46ffd640 1 -- 172.21.15.150:0/233166076 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+6538 (secure 0 0 0) 0x7fbb4c05ea80 con 0x7fbb580c0350 2024-09-17T05:00:40.693 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:40.693 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":86,"btime":"2024-09-17T05:00:26:993373+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:40.694 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 86 2024-09-17T05:00:40.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb38073230 msgr2=0x7fbb38075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:40.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb5c9e3640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb38073230 0x7fbb38075740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009b40 tx=0x7fbb480091e0 comp rx=0 tx=0).stop 2024-09-17T05:00:40.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/233166076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580c0350 msgr2=0x7fbb5806ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:40.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb5c9e3640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580c0350 0x7fbb5806ce50 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00c3e0 tx=0x7fbb4c00c8a0 comp rx=0 tx=0).stop 2024-09-17T05:00:40.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/233166076 shutdown_connections 2024-09-17T05:00:40.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb5c9e3640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb38073230 0x7fbb38075740 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:40.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb5c9e3640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb580c0760 0x7fbb5806d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:40.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb5c9e3640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb580c0350 0x7fbb5806ce50 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:40.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb5c9e3640 1 --2- 172.21.15.150:0/233166076 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb40005360 0x7fbb58074440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:40.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/233166076 >> 172.21.15.150:0/233166076 conn(0x7fbb580aaa80 msgr2=0x7fbb58155340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:40.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/233166076 shutdown_connections 2024-09-17T05:00:40.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:40.695+0000 7fbb5c9e3640 1 -- 172.21.15.150:0/233166076 wait complete. 2024-09-17T05:00:40.713 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:00:42.714 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:42.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.879+0000 7efc54ce4640 1 -- 172.21.15.150:0/549297000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc3c004810 msgr2=0x7efc3c004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:42.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.879+0000 7efc54ce4640 1 --2- 172.21.15.150:0/549297000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc3c004810 0x7efc3c004c20 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7efc380016f0 tx=0x7efc3801c790 comp rx=0 tx=0).stop 2024-09-17T05:00:42.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.879+0000 7efc54ce4640 1 -- 172.21.15.150:0/549297000 shutdown_connections 2024-09-17T05:00:42.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.879+0000 7efc54ce4640 1 --2- 172.21.15.150:0/549297000 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc3c004810 0x7efc3c004c20 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:42.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.879+0000 7efc54ce4640 1 -- 172.21.15.150:0/549297000 >> 172.21.15.150:0/549297000 conn(0x7efc500c4a00 msgr2=0x7efc500c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:42.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.879+0000 7efc54ce4640 1 -- 172.21.15.150:0/549297000 shutdown_connections 2024-09-17T05:00:42.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.879+0000 7efc54ce4640 1 -- 172.21.15.150:0/549297000 wait complete. 2024-09-17T05:00:42.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc54ce4640 1 Processor -- start 2024-09-17T05:00:42.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc54ce4640 1 -- start start 2024-09-17T05:00:42.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc54ce4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc3c004810 0x7efc501e2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:42.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc54ce4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc500aab30 0x7efc5014aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc54ce4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc5014b4f0 0x7efc500c1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc54ce4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efc50074860 con 0x7efc3c004810 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc54ce4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efc500adef0 con 0x7efc500aab30 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc54ce4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efc500749e0 con 0x7efc5014b4f0 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc4e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc3c004810 0x7efc501e2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc4e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc3c004810 0x7efc501e2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45896/0 (socket says 172.21.15.150:45896) 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc4e59c640 1 -- 172.21.15.150:0/1451752803 learned_addr learned my addr 172.21.15.150:0/1451752803 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc4dd9b640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc500aab30 0x7efc5014aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc4ed9d640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc5014b4f0 0x7efc500c1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc4e59c640 1 -- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc5014b4f0 msgr2=0x7efc500c1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc4e59c640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc5014b4f0 0x7efc500c1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc4e59c640 1 -- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc500aab30 msgr2=0x7efc5014aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc4e59c640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc500aab30 0x7efc5014aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:42.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc4e59c640 1 -- 172.21.15.150:0/1451752803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc5014e470 con 0x7efc3c004810 2024-09-17T05:00:42.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc4e59c640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc3c004810 0x7efc501e2950 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7efc4800fb30 tx=0x7efc4800d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:42.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc46ffd640 1 -- 172.21.15.150:0/1451752803 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4800de60 con 0x7efc3c004810 2024-09-17T05:00:42.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc46ffd640 1 -- 172.21.15.150:0/1451752803 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc4801b530 con 0x7efc3c004810 2024-09-17T05:00:42.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.883+0000 7efc46ffd640 1 -- 172.21.15.150:0/1451752803 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4801b960 con 0x7efc3c004810 2024-09-17T05:00:42.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.887+0000 7efc54ce4640 1 -- 172.21.15.150:0/1451752803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc500ac5e0 con 0x7efc3c004810 2024-09-17T05:00:42.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.887+0000 7efc54ce4640 1 -- 172.21.15.150:0/1451752803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc501f3d00 con 0x7efc3c004810 2024-09-17T05:00:42.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.887+0000 7efc54ce4640 1 -- 172.21.15.150:0/1451752803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc50074ea0 con 0x7efc3c004810 2024-09-17T05:00:42.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.887+0000 7efc46ffd640 1 -- 172.21.15.150:0/1451752803 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7efc4801bb00 con 0x7efc3c004810 2024-09-17T05:00:42.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.887+0000 7efc46ffd640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efc30073230 0x7efc30075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:42.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.887+0000 7efc46ffd640 1 -- 172.21.15.150:0/1451752803 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7efc480958a0 con 0x7efc3c004810 2024-09-17T05:00:42.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.887+0000 7efc4dd9b640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efc30073230 0x7efc30075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:42.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.887+0000 7efc4dd9b640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efc30073230 0x7efc30075740 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7efc380016c0 tx=0x7efc38019580 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:42.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:42.891+0000 7efc46ffd640 1 -- 172.21.15.150:0/1451752803 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7efc48060170 con 0x7efc3c004810 2024-09-17T05:00:43.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.095+0000 7efc54ce4640 1 -- 172.21.15.150:0/1451752803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc500c60f0 con 0x7efc3c004810 2024-09-17T05:00:43.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc46ffd640 1 -- 172.21.15.150:0/1451752803 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+6538 (secure 0 0 0) 0x7efc4805fa70 con 0x7efc3c004810 2024-09-17T05:00:43.097 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:43.097 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":86,"btime":"2024-09-17T05:00:26:993373+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:43.097 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 86 2024-09-17T05:00:43.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc54ce4640 1 -- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efc30073230 msgr2=0x7efc30075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:43.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc54ce4640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efc30073230 0x7efc30075740 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7efc380016c0 tx=0x7efc38019580 comp rx=0 tx=0).stop 2024-09-17T05:00:43.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc54ce4640 1 -- 172.21.15.150:0/1451752803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc3c004810 msgr2=0x7efc501e2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:43.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc54ce4640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc3c004810 0x7efc501e2950 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7efc4800fb30 tx=0x7efc4800d2e0 comp rx=0 tx=0).stop 2024-09-17T05:00:43.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc54ce4640 1 -- 172.21.15.150:0/1451752803 shutdown_connections 2024-09-17T05:00:43.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc54ce4640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efc30073230 0x7efc30075740 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:43.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc54ce4640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc500aab30 0x7efc5014aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:43.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc54ce4640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efc5014b4f0 0x7efc500c1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:43.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc54ce4640 1 --2- 172.21.15.150:0/1451752803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc3c004810 0x7efc501e2950 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:43.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc54ce4640 1 -- 172.21.15.150:0/1451752803 >> 172.21.15.150:0/1451752803 conn(0x7efc500c4a00 msgr2=0x7efc500c33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:43.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc54ce4640 1 -- 172.21.15.150:0/1451752803 shutdown_connections 2024-09-17T05:00:43.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:43.099+0000 7efc54ce4640 1 -- 172.21.15.150:0/1451752803 wait complete. 2024-09-17T05:00:43.119 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:00:45.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:45.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.287+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/1110418643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc08151d00 msgr2=0x7fcc080c3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:45.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.287+0000 7fcc0e9ba640 1 --2- 172.21.15.150:0/1110418643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc08151d00 0x7fcc080c3430 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcc080adcf0 tx=0x7fcbf801c8c0 comp rx=0 tx=0).stop 2024-09-17T05:00:45.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.287+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/1110418643 shutdown_connections 2024-09-17T05:00:45.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.287+0000 7fcc0e9ba640 1 --2- 172.21.15.150:0/1110418643 >> v2:172.21.15.176:3300/0 conn(0x7fcc080c3970 0x7fcc080c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:45.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.287+0000 7fcc0e9ba640 1 --2- 172.21.15.150:0/1110418643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc08151d00 0x7fcc080c3430 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:45.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.287+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/1110418643 >> 172.21.15.150:0/1110418643 conn(0x7fcc080c4a00 msgr2=0x7fcc080c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:45.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.287+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/1110418643 shutdown_connections 2024-09-17T05:00:45.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.287+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/1110418643 wait complete. 2024-09-17T05:00:45.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.287+0000 7fcc0e9ba640 1 Processor -- start 2024-09-17T05:00:45.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.287+0000 7fcc0e9ba640 1 -- start start 2024-09-17T05:00:45.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc0e9ba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc080c3970 0x7fcc0814ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:45.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc0e9ba640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc08151d00 0x7fcc08074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:45.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc0e9ba640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc08157c30 0x7fcc08074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:45.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc0e9ba640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcc081619b0 con 0x7fcc080c3970 2024-09-17T05:00:45.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc0e9ba640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcc08161830 con 0x7fcc08151d00 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc0e9ba640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcc08161b30 con 0x7fcc08157c30 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc07fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc080c3970 0x7fcc0814ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc07fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc080c3970 0x7fcc0814ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45920/0 (socket says 172.21.15.150:45920) 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc07fff640 1 -- 172.21.15.150:0/2012003352 learned_addr learned my addr 172.21.15.150:0/2012003352 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc0cf57640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc08157c30 0x7fcc08074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc077fe640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc08151d00 0x7fcc08074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc07fff640 1 -- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc08157c30 msgr2=0x7fcc08074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc07fff640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc08157c30 0x7fcc08074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc07fff640 1 -- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc08151d00 msgr2=0x7fcc08074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc07fff640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc08151d00 0x7fcc08074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc07fff640 1 -- 172.21.15.150:0/2012003352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0814e1a0 con 0x7fcc080c3970 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc07fff640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc080c3970 0x7fcc0814ebd0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fcbf801af70 tx=0x7fcbf80a5540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:45.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc04ff9640 1 -- 172.21.15.150:0/2012003352 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf80bb070 con 0x7fcc080c3970 2024-09-17T05:00:45.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc04ff9640 1 -- 172.21.15.150:0/2012003352 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbf80043b0 con 0x7fcc080c3970 2024-09-17T05:00:45.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/2012003352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc081f3c00 con 0x7fcc080c3970 2024-09-17T05:00:45.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc04ff9640 1 -- 172.21.15.150:0/2012003352 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf80b2040 con 0x7fcc080c3970 2024-09-17T05:00:45.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/2012003352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc0815e030 con 0x7fcc080c3970 2024-09-17T05:00:45.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/2012003352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc080c3d80 con 0x7fcc080c3970 2024-09-17T05:00:45.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.291+0000 7fcc04ff9640 1 -- 172.21.15.150:0/2012003352 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fcbf80be070 con 0x7fcc080c3970 2024-09-17T05:00:45.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.295+0000 7fcc04ff9640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbf4073200 0x7fcbf4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:45.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.295+0000 7fcc04ff9640 1 -- 172.21.15.150:0/2012003352 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7fcbf812e2f0 con 0x7fcc080c3970 2024-09-17T05:00:45.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.295+0000 7fcc077fe640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbf4073200 0x7fcbf4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:45.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.295+0000 7fcc077fe640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbf4073200 0x7fcbf4075710 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0058d0 tx=0x7fcbfc005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:45.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.299+0000 7fcc04ff9640 1 -- 172.21.15.150:0/2012003352 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcbf80164b0 con 0x7fcc080c3970 2024-09-17T05:00:45.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.507+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/2012003352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc080a7730 con 0x7fcc080c3970 2024-09-17T05:00:45.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc04ff9640 1 -- 172.21.15.150:0/2012003352 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+6538 (secure 0 0 0) 0x7fcbf80f8570 con 0x7fcc080c3970 2024-09-17T05:00:45.509 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:45.509 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":86,"btime":"2024-09-17T05:00:26:993373+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":86}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T04:59:24.099259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6075":{"gid":6075,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/2313391683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2313391683},{"type":"v1","addr":"172.21.15.150:6833","nonce":2313391683}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6075,"qdb_cluster":[6075]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:45.510 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 86 2024-09-17T05:00:45.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbf4073200 msgr2=0x7fcbf4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:45.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc0e9ba640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbf4073200 0x7fcbf4075710 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0058d0 tx=0x7fcbfc005860 comp rx=0 tx=0).stop 2024-09-17T05:00:45.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/2012003352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc080c3970 msgr2=0x7fcc0814ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:45.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc0e9ba640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc080c3970 0x7fcc0814ebd0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fcbf801af70 tx=0x7fcbf80a5540 comp rx=0 tx=0).stop 2024-09-17T05:00:45.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/2012003352 shutdown_connections 2024-09-17T05:00:45.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc0e9ba640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbf4073200 0x7fcbf4075710 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:45.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc0e9ba640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc08151d00 0x7fcc08074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:45.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc0e9ba640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcc08157c30 0x7fcc08074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:45.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc0e9ba640 1 --2- 172.21.15.150:0/2012003352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc080c3970 0x7fcc0814ebd0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:45.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/2012003352 >> 172.21.15.150:0/2012003352 conn(0x7fcc080c4a00 msgr2=0x7fcc080c1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:45.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/2012003352 shutdown_connections 2024-09-17T05:00:45.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:45.511+0000 7fcc0e9ba640 1 -- 172.21.15.150:0/2012003352 wait complete. 2024-09-17T05:00:45.531 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:00:47.532 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:47.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.699+0000 7f275c784640 1 -- 172.21.15.150:0/2640199337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27540a7350 msgr2=0x7f2754150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:47.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.699+0000 7f275c784640 1 --2- 172.21.15.150:0/2640199337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27540a7350 0x7f2754150060 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f27540c7010 tx=0x7f274401c8a0 comp rx=0 tx=0).stop 2024-09-17T05:00:47.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.699+0000 7f275c784640 1 -- 172.21.15.150:0/2640199337 shutdown_connections 2024-09-17T05:00:47.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.699+0000 7f275c784640 1 --2- 172.21.15.150:0/2640199337 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27541505a0 0x7f2754152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:47.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.699+0000 7f275c784640 1 --2- 172.21.15.150:0/2640199337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27540a7350 0x7f2754150060 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:47.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.699+0000 7f275c784640 1 --2- 172.21.15.150:0/2640199337 >> v2:172.21.15.176:3300/0 conn(0x7f27540c42e0 0x7f27540c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:47.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.699+0000 7f275c784640 1 -- 172.21.15.150:0/2640199337 >> 172.21.15.150:0/2640199337 conn(0x7f27540c4a00 msgr2=0x7f27540c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:47.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.699+0000 7f275c784640 1 -- 172.21.15.150:0/2640199337 shutdown_connections 2024-09-17T05:00:47.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.699+0000 7f275c784640 1 -- 172.21.15.150:0/2640199337 wait complete. 2024-09-17T05:00:47.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.699+0000 7f275c784640 1 Processor -- start 2024-09-17T05:00:47.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.699+0000 7f275c784640 1 -- start start 2024-09-17T05:00:47.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f275c784640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27540a7350 0x7f27540c0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:47.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f275c784640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27540c42e0 0x7f275415be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:47.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f275c784640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27541505a0 0x7f27540c2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:47.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f275c784640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f27540c6bd0 con 0x7f27540c42e0 2024-09-17T05:00:47.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f275c784640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2754070da0 con 0x7f27541505a0 2024-09-17T05:00:47.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f275c784640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f27540c6d50 con 0x7f27540a7350 2024-09-17T05:00:47.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f2759d1f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27540c42e0 0x7f275415be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:47.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f2759d1f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27540c42e0 0x7f275415be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45940/0 (socket says 172.21.15.150:45940) 2024-09-17T05:00:47.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f2759d1f640 1 -- 172.21.15.150:0/2709179709 learned_addr learned my addr 172.21.15.150:0/2709179709 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:47.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f275a520640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27540a7350 0x7f27540c0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:47.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f275ad21640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27541505a0 0x7f27540c2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:47.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f2759d1f640 1 -- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27540a7350 msgr2=0x7f27540c0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:47.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f2759d1f640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27540a7350 0x7f27540c0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:47.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f2759d1f640 1 -- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27541505a0 msgr2=0x7f27540c2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:47.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f2759d1f640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27541505a0 0x7f27540c2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:47.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f2759d1f640 1 -- 172.21.15.150:0/2709179709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2754157680 con 0x7f27540c42e0 2024-09-17T05:00:47.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f2759d1f640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27540c42e0 0x7f275415be80 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2744009850 tx=0x7f27440a57d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:47.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f274affd640 1 -- 172.21.15.150:0/2709179709 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27440b2040 con 0x7f27540c42e0 2024-09-17T05:00:47.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f274affd640 1 -- 172.21.15.150:0/2709179709 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2744004430 con 0x7f27540c42e0 2024-09-17T05:00:47.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f275c784640 1 -- 172.21.15.150:0/2709179709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27540be3d0 con 0x7f27540c42e0 2024-09-17T05:00:47.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f274affd640 1 -- 172.21.15.150:0/2709179709 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2744005290 con 0x7f27540c42e0 2024-09-17T05:00:47.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f275c784640 1 -- 172.21.15.150:0/2709179709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27541ee4a0 con 0x7f27540c42e0 2024-09-17T05:00:47.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f275c784640 1 -- 172.21.15.150:0/2709179709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f275414e690 con 0x7f27540c42e0 2024-09-17T05:00:47.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f274affd640 1 -- 172.21.15.150:0/2709179709 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2744005430 con 0x7f27540c42e0 2024-09-17T05:00:47.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f274affd640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2738073200 0x7f2738075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:47.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.703+0000 7f274affd640 1 -- 172.21.15.150:0/2709179709 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f274412e750 con 0x7f27540c42e0 2024-09-17T05:00:47.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.707+0000 7f275a520640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2738073200 0x7f2738075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:47.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.707+0000 7f275a520640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2738073200 0x7f2738075710 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f27500058d0 tx=0x7f2750005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:47.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.711+0000 7f274affd640 1 -- 172.21.15.150:0/2709179709 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f27540be3d0 con 0x7f27540c42e0 2024-09-17T05:00:47.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.923+0000 7f275c784640 1 -- 172.21.15.150:0/2709179709 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f275414e3b0 con 0x7f27540c42e0 2024-09-17T05:00:47.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.923+0000 7f274affd640 1 -- 172.21.15.150:0/2709179709 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5705 (secure 0 0 0) 0x7f2744016490 con 0x7f27540c42e0 2024-09-17T05:00:47.922 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:47.922 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":88,"btime":"2024-09-17T05:00:47:106814+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:46.316606+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:47.923 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 88 2024-09-17T05:00:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.927+0000 7f275c784640 1 -- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2738073200 msgr2=0x7f2738075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.927+0000 7f275c784640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2738073200 0x7f2738075710 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f27500058d0 tx=0x7f2750005860 comp rx=0 tx=0).stop 2024-09-17T05:00:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.927+0000 7f275c784640 1 -- 172.21.15.150:0/2709179709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27540c42e0 msgr2=0x7f275415be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.927+0000 7f275c784640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27540c42e0 0x7f275415be80 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2744009850 tx=0x7f27440a57d0 comp rx=0 tx=0).stop 2024-09-17T05:00:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.927+0000 7f275c784640 1 -- 172.21.15.150:0/2709179709 shutdown_connections 2024-09-17T05:00:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.927+0000 7f275c784640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2738073200 0x7f2738075710 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.927+0000 7f275c784640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27541505a0 0x7f27540c2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.927+0000 7f275c784640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27540c42e0 0x7f275415be80 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.927+0000 7f275c784640 1 --2- 172.21.15.150:0/2709179709 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27540a7350 0x7f27540c0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.927+0000 7f275c784640 1 -- 172.21.15.150:0/2709179709 >> 172.21.15.150:0/2709179709 conn(0x7f27540c4a00 msgr2=0x7f2754154aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:47.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.927+0000 7f275c784640 1 -- 172.21.15.150:0/2709179709 shutdown_connections 2024-09-17T05:00:47.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:47.927+0000 7f275c784640 1 -- 172.21.15.150:0/2709179709 wait complete. 2024-09-17T05:00:47.946 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-17T05:00:47.946 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:00:47.947 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:48.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.099+0000 7f4da9a3d640 1 -- 172.21.15.150:0/509161284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4156eb0 msgr2=0x7f4da41572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:48.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.099+0000 7f4da9a3d640 1 --2- 172.21.15.150:0/509161284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4156eb0 0x7f4da41572c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f4da40737f0 tx=0x7f4d8c01c830 comp rx=0 tx=0).stop 2024-09-17T05:00:48.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.099+0000 7f4da9a3d640 1 -- 172.21.15.150:0/509161284 shutdown_connections 2024-09-17T05:00:48.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.099+0000 7f4da9a3d640 1 --2- 172.21.15.150:0/509161284 >> v2:172.21.15.176:3300/0 conn(0x7f4da414cda0 0x7f4da414d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:48.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.099+0000 7f4da9a3d640 1 --2- 172.21.15.150:0/509161284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4156eb0 0x7f4da41572c0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:48.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.099+0000 7f4da9a3d640 1 -- 172.21.15.150:0/509161284 >> 172.21.15.150:0/509161284 conn(0x7f4da40c4a00 msgr2=0x7f4da40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:48.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.099+0000 7f4da9a3d640 1 -- 172.21.15.150:0/509161284 shutdown_connections 2024-09-17T05:00:48.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.099+0000 7f4da9a3d640 1 -- 172.21.15.150:0/509161284 wait complete. 2024-09-17T05:00:48.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.099+0000 7f4da9a3d640 1 Processor -- start 2024-09-17T05:00:48.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.099+0000 7f4da9a3d640 1 -- start start 2024-09-17T05:00:48.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da9a3d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4da40c7570 0x7f4da4143020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:48.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da9a3d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4da414cda0 0x7f4da4146310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:48.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da9a3d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4156eb0 0x7f4da40ac500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:48.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da9a3d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4da40c65c0 con 0x7f4da4156eb0 2024-09-17T05:00:48.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da9a3d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4da40c6440 con 0x7f4da414cda0 2024-09-17T05:00:48.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da9a3d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4da40c6740 con 0x7f4da40c7570 2024-09-17T05:00:48.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4156eb0 0x7f4da40ac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:48.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4156eb0 0x7f4da40ac500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45962/0 (socket says 172.21.15.150:45962) 2024-09-17T05:00:48.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da37fe640 1 -- 172.21.15.150:0/3247907715 learned_addr learned my addr 172.21.15.150:0/3247907715 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da2ffd640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4da40c7570 0x7f4da4143020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da27fc640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4da414cda0 0x7f4da4146310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da37fe640 1 -- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4da40c7570 msgr2=0x7f4da4143020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da37fe640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4da40c7570 0x7f4da4143020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da37fe640 1 -- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4da414cda0 msgr2=0x7f4da4146310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da37fe640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4da414cda0 0x7f4da4146310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da37fe640 1 -- 172.21.15.150:0/3247907715 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da40aafb0 con 0x7f4da4156eb0 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da37fe640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4156eb0 0x7f4da40ac500 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00c9d0 tx=0x7f4d9c00ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4d9bfff640 1 -- 172.21.15.150:0/3247907715 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c007bb0 con 0x7f4da4156eb0 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4d9bfff640 1 -- 172.21.15.150:0/3247907715 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d9c00f450 con 0x7f4da4156eb0 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4d9bfff640 1 -- 172.21.15.150:0/3247907715 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c007e60 con 0x7f4da4156eb0 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da9a3d640 1 -- 172.21.15.150:0/3247907715 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da414cb30 con 0x7f4da4156eb0 2024-09-17T05:00:48.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da9a3d640 1 -- 172.21.15.150:0/3247907715 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da40ad760 con 0x7f4da4156eb0 2024-09-17T05:00:48.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4da9a3d640 1 -- 172.21.15.150:0/3247907715 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da414d1b0 con 0x7f4da4156eb0 2024-09-17T05:00:48.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4d9bfff640 1 -- 172.21.15.150:0/3247907715 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4d9c00f8a0 con 0x7f4da4156eb0 2024-09-17T05:00:48.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.103+0000 7f4d9bfff640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d84073230 0x7f4d84075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:48.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.107+0000 7f4d9bfff640 1 -- 172.21.15.150:0/3247907715 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f4d9c0951d0 con 0x7f4da4156eb0 2024-09-17T05:00:48.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.107+0000 7f4da2ffd640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d84073230 0x7f4d84075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:48.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.107+0000 7f4da2ffd640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d84073230 0x7f4d84075740 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4d94001cf0 tx=0x7f4d94009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:48.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.111+0000 7f4d9bfff640 1 -- 172.21.15.150:0/3247907715 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4d9c014030 con 0x7f4da4156eb0 2024-09-17T05:00:48.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.319+0000 7f4da9a3d640 1 -- 172.21.15.150:0/3247907715 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4da40c2410 con 0x7f4da4156eb0 2024-09-17T05:00:48.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.319+0000 7f4d9bfff640 1 -- 172.21.15.150:0/3247907715 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+5702 (secure 0 0 0) 0x7f4d9c05fa40 con 0x7f4da4156eb0 2024-09-17T05:00:48.318 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:48.318 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":89,"btime":"2024-09-17T05:00:48:109133+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:47.114465+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:48.318 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 89 2024-09-17T05:00:48.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.319+0000 7f4da9a3d640 1 -- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d84073230 msgr2=0x7f4d84075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:48.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.319+0000 7f4da9a3d640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d84073230 0x7f4d84075740 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4d94001cf0 tx=0x7f4d94009040 comp rx=0 tx=0).stop 2024-09-17T05:00:48.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.319+0000 7f4da9a3d640 1 -- 172.21.15.150:0/3247907715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4156eb0 msgr2=0x7f4da40ac500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:48.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.319+0000 7f4da9a3d640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4156eb0 0x7f4da40ac500 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00c9d0 tx=0x7f4d9c00ce90 comp rx=0 tx=0).stop 2024-09-17T05:00:48.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.319+0000 7f4da9a3d640 1 -- 172.21.15.150:0/3247907715 shutdown_connections 2024-09-17T05:00:48.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.319+0000 7f4da9a3d640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d84073230 0x7f4d84075740 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:48.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.319+0000 7f4da9a3d640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4da414cda0 0x7f4da4146310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:48.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.319+0000 7f4da9a3d640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4156eb0 0x7f4da40ac500 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:48.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.319+0000 7f4da9a3d640 1 --2- 172.21.15.150:0/3247907715 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4da40c7570 0x7f4da4143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:48.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.323+0000 7f4da9a3d640 1 -- 172.21.15.150:0/3247907715 >> 172.21.15.150:0/3247907715 conn(0x7f4da40c4a00 msgr2=0x7f4da40a93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:48.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.323+0000 7f4da9a3d640 1 -- 172.21.15.150:0/3247907715 shutdown_connections 2024-09-17T05:00:48.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:48.323+0000 7f4da9a3d640 1 -- 172.21.15.150:0/3247907715 wait complete. 2024-09-17T05:00:48.341 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:00:48:109133+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 89, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6252": { "addr": "172.21.15.150:6835/2062728083", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 2062728083, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2062728083, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6252, "incarnation": 87, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:00:47.114465+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6252 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 85, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5375": { "addr": "172.21.15.176:6837/702608997", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 702608997, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 702608997, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5375, "incarnation": 82, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:00:23.977031+0000", "qdb_cluster": [ 5375 ], "qdb_leader": 5375, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5375 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:00:50.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:50.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.531+0000 7f419c73b640 1 -- 172.21.15.150:0/2310137204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41940a85b0 msgr2=0x7f41940c7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:50.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.531+0000 7f419c73b640 1 --2- 172.21.15.150:0/2310137204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41940a85b0 0x7f41940c7270 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f41940755f0 tx=0x7f418401c830 comp rx=0 tx=0).stop 2024-09-17T05:00:50.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.531+0000 7f419c73b640 1 -- 172.21.15.150:0/2310137204 shutdown_connections 2024-09-17T05:00:50.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.531+0000 7f419c73b640 1 --2- 172.21.15.150:0/2310137204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41940a85b0 0x7f41940c7270 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:50.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.531+0000 7f419c73b640 1 --2- 172.21.15.150:0/2310137204 >> v2:172.21.15.176:3300/0 conn(0x7f41940c6cb0 0x7f41940ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:50.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.531+0000 7f419c73b640 1 -- 172.21.15.150:0/2310137204 >> 172.21.15.150:0/2310137204 conn(0x7f41940aab60 msgr2=0x7f41940aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:50.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.531+0000 7f419c73b640 1 -- 172.21.15.150:0/2310137204 shutdown_connections 2024-09-17T05:00:50.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.531+0000 7f419c73b640 1 -- 172.21.15.150:0/2310137204 wait complete. 2024-09-17T05:00:50.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.531+0000 7f419c73b640 1 Processor -- start 2024-09-17T05:00:50.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.531+0000 7f419c73b640 1 -- start start 2024-09-17T05:00:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f419c73b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f41940a85b0 0x7f4194071070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f419c73b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41940c6cb0 0x7f41940a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f419c73b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940c77b0 0x7f4194144e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f419c73b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f41940c3460 con 0x7f41940c6cb0 2024-09-17T05:00:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f419c73b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f41940c32e0 con 0x7f41940c77b0 2024-09-17T05:00:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f419c73b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4194161890 con 0x7f41940a85b0 2024-09-17T05:00:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f4199cd6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41940c6cb0 0x7f41940a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:50.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f4199cd6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41940c6cb0 0x7f41940a9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49430/0 (socket says 172.21.15.150:49430) 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f4199cd6640 1 -- 172.21.15.150:0/67634273 learned_addr learned my addr 172.21.15.150:0/67634273 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f419acd8640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940c77b0 0x7f4194144e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f419a4d7640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f41940a85b0 0x7f4194071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f4199cd6640 1 -- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f41940a85b0 msgr2=0x7f4194071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f4199cd6640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f41940a85b0 0x7f4194071070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f4199cd6640 1 -- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940c77b0 msgr2=0x7f4194144e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f4199cd6640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940c77b0 0x7f4194144e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f4199cd6640 1 -- 172.21.15.150:0/67634273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4194071f60 con 0x7f41940c6cb0 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f419acd8640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940c77b0 0x7f4194144e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f4199cd6640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41940c6cb0 0x7f41940a9e20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4184019440 tx=0x7f4184005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f418affd640 1 -- 172.21.15.150:0/67634273 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41840b2040 con 0x7f41940c6cb0 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f418affd640 1 -- 172.21.15.150:0/67634273 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4184004330 con 0x7f41940c6cb0 2024-09-17T05:00:50.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f419c73b640 1 -- 172.21.15.150:0/67634273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419414f8d0 con 0x7f41940c6cb0 2024-09-17T05:00:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f418affd640 1 -- 172.21.15.150:0/67634273 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4184005290 con 0x7f41940c6cb0 2024-09-17T05:00:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f419c73b640 1 -- 172.21.15.150:0/67634273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f419414f410 con 0x7f41940c6cb0 2024-09-17T05:00:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f419c73b640 1 -- 172.21.15.150:0/67634273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41940c7270 con 0x7f41940c6cb0 2024-09-17T05:00:50.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f418affd640 1 -- 172.21.15.150:0/67634273 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4184005430 con 0x7f41940c6cb0 2024-09-17T05:00:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.535+0000 7f418affd640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4178073200 0x7f4178075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.539+0000 7f418affd640 1 -- 172.21.15.150:0/67634273 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f418412ebc0 con 0x7f41940c6cb0 2024-09-17T05:00:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.539+0000 7f419a4d7640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4178073200 0x7f4178075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:50.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.539+0000 7f419a4d7640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4178073200 0x7f4178075710 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f41900058d0 tx=0x7f4190005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.543+0000 7f418affd640 1 -- 172.21.15.150:0/67634273 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f41840164b0 con 0x7f41940c6cb0 2024-09-17T05:00:50.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.779+0000 7f419c73b640 1 -- 172.21.15.150:0/67634273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4194072130 con 0x7f41940c6cb0 2024-09-17T05:00:50.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.779+0000 7f418affd640 1 -- 172.21.15.150:0/67634273 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+5709 (secure 0 0 0) 0x7f4184118e20 con 0x7f41940c6cb0 2024-09-17T05:00:50.778 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:50.778 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":90,"btime":"2024-09-17T05:00:49:115379+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:50.778 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 90 2024-09-17T05:00:50.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.783+0000 7f419c73b640 1 -- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4178073200 msgr2=0x7f4178075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:50.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.783+0000 7f419c73b640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4178073200 0x7f4178075710 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f41900058d0 tx=0x7f4190005860 comp rx=0 tx=0).stop 2024-09-17T05:00:50.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.783+0000 7f419c73b640 1 -- 172.21.15.150:0/67634273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41940c6cb0 msgr2=0x7f41940a9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:50.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.783+0000 7f419c73b640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41940c6cb0 0x7f41940a9e20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4184019440 tx=0x7f4184005e70 comp rx=0 tx=0).stop 2024-09-17T05:00:50.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.783+0000 7f419c73b640 1 -- 172.21.15.150:0/67634273 shutdown_connections 2024-09-17T05:00:50.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.783+0000 7f419c73b640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4178073200 0x7f4178075710 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:50.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.783+0000 7f419c73b640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940c77b0 0x7f4194144e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:50.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.783+0000 7f419c73b640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41940c6cb0 0x7f41940a9e20 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:50.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.783+0000 7f419c73b640 1 --2- 172.21.15.150:0/67634273 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f41940a85b0 0x7f4194071070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:50.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.783+0000 7f419c73b640 1 -- 172.21.15.150:0/67634273 >> 172.21.15.150:0/67634273 conn(0x7f41940aab60 msgr2=0x7f41940c84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:50.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.783+0000 7f419c73b640 1 -- 172.21.15.150:0/67634273 shutdown_connections 2024-09-17T05:00:50.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:50.783+0000 7f419c73b640 1 -- 172.21.15.150:0/67634273 wait complete. 2024-09-17T05:00:50.805 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=6075 2024-09-17T05:00:50.806 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.a 2024-09-17T05:00:51.807 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-17T05:00:51.807 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T05:00:51.807 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T05:00:51.809 INFO:tasks.ceph.mds.a:Started 2024-09-17T05:00:51.810 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:00:51.916 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T05:00:51.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.991+0000 7f170de07640 1 -- 172.21.15.150:0/3547223097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f0004d50 msgr2=0x7f16f0005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:51.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.991+0000 7f170de07640 1 --2- 172.21.15.150:0/3547223097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16f0004d50 0x7f16f0005160 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00c260 tx=0x7f16fc01cf60 comp rx=0 tx=0).stop 2024-09-17T05:00:51.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.991+0000 7f1707fff640 1 -- 172.21.15.150:0/3547223097 reap_dead start 2024-09-17T05:00:51.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.991+0000 7f170de07640 1 -- 172.21.15.150:0/3547223097 shutdown_connections 2024-09-17T05:00:51.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.991+0000 7f170de07640 1 -- 172.21.15.150:0/3547223097 >> 172.21.15.150:0/3547223097 conn(0x7f1708074b80 msgr2=0x7f170806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:51.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f170de07640 1 -- 172.21.15.150:0/3547223097 shutdown_connections 2024-09-17T05:00:51.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f170de07640 1 -- 172.21.15.150:0/3547223097 wait complete. 2024-09-17T05:00:51.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f170de07640 1 Processor -- start 2024-09-17T05:00:51.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f170de07640 1 -- start start 2024-09-17T05:00:51.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f170de07640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16f0004d50 0x7f17080b0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:51.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f170de07640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f170814cd60 0x7f17080cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:51.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f170de07640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1708159080 0x7f1708159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:51.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f170de07640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1708154450 con 0x7f1708159080 2024-09-17T05:00:51.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f170de07640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f17080c39c0 con 0x7f170814cd60 2024-09-17T05:00:51.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f170de07640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f17080c3b40 con 0x7f16f0004d50 2024-09-17T05:00:51.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f1707fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1708159080 0x7f1708159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:51.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f1707fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1708159080 0x7f1708159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49468/0 (socket says 172.21.15.150:49468) 2024-09-17T05:00:51.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f1707fff640 1 -- 172.21.15.150:0/3425532686 learned_addr learned my addr 172.21.15.150:0/3425532686 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:00:51.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f17077fe640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16f0004d50 0x7f17080b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f1706ffd640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f170814cd60 0x7f17080cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f1707fff640 1 -- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16f0004d50 msgr2=0x7f17080b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f1707fff640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16f0004d50 0x7f17080b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f1707fff640 1 -- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f170814cd60 msgr2=0x7f17080cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f1707fff640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f170814cd60 0x7f17080cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f1707fff640 1 -- 172.21.15.150:0/3425532686 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170814caf0 con 0x7f1708159080 2024-09-17T05:00:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f1706ffd640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f170814cd60 0x7f17080cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:00:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.995+0000 7f1707fff640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1708159080 0x7f1708159490 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f16fc098b30 tx=0x7f16fc004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.999+0000 7f16effff640 1 -- 172.21.15.150:0/3425532686 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc006c40 con 0x7f1708159080 2024-09-17T05:00:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.999+0000 7f16effff640 1 -- 172.21.15.150:0/3425532686 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16fc006440 con 0x7f1708159080 2024-09-17T05:00:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.999+0000 7f16effff640 1 -- 172.21.15.150:0/3425532686 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc001cc0 con 0x7f1708159080 2024-09-17T05:00:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.999+0000 7f170de07640 1 -- 172.21.15.150:0/3425532686 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17081ed3a0 con 0x7f1708159080 2024-09-17T05:00:51.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.999+0000 7f170de07640 1 -- 172.21.15.150:0/3425532686 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17081efdf0 con 0x7f1708159080 2024-09-17T05:00:51.999 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T05:00:51.999+0000 7f0df29c2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:00:52.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.999+0000 7f170de07640 1 -- 172.21.15.150:0/3425532686 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17080c8a80 con 0x7f1708159080 2024-09-17T05:00:52.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.999+0000 7f16effff640 1 -- 172.21.15.150:0/3425532686 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f16fc004c50 con 0x7f1708159080 2024-09-17T05:00:52.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.999+0000 7f16effff640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f16f4073200 0x7f16f4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:00:52.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.999+0000 7f16effff640 1 -- 172.21.15.150:0/3425532686 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f16fc12efc0 con 0x7f1708159080 2024-09-17T05:00:52.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:51.999+0000 7f17077fe640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f16f4073200 0x7f16f4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:00:52.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.003+0000 7f17077fe640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f16f4073200 0x7f16f4075710 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1700011550 tx=0x7f1700009500 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:00:52.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.007+0000 7f16effff640 1 -- 172.21.15.150:0/3425532686 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f16fc016d10 con 0x7f1708159080 2024-09-17T05:00:52.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.235+0000 7f170de07640 1 -- 172.21.15.150:0/3425532686 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f170814d2b0 con 0x7f1708159080 2024-09-17T05:00:52.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.235+0000 7f16effff640 1 -- 172.21.15.150:0/3425532686 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6534 (secure 0 0 0) 0x7f16fc0f9160 con 0x7f1708159080 2024-09-17T05:00:52.234 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:00:52.235 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":91,"btime":"2024-09-17T05:00:52:139685+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6342,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:00:52.235 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 91 2024-09-17T05:00:52.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.239+0000 7f170de07640 1 -- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f16f4073200 msgr2=0x7f16f4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:52.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.239+0000 7f170de07640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f16f4073200 0x7f16f4075710 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1700011550 tx=0x7f1700009500 comp rx=0 tx=0).stop 2024-09-17T05:00:52.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.239+0000 7f170de07640 1 -- 172.21.15.150:0/3425532686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1708159080 msgr2=0x7f1708159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:00:52.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.239+0000 7f170de07640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1708159080 0x7f1708159490 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f16fc098b30 tx=0x7f16fc004b60 comp rx=0 tx=0).stop 2024-09-17T05:00:52.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.239+0000 7f170de07640 1 -- 172.21.15.150:0/3425532686 shutdown_connections 2024-09-17T05:00:52.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.239+0000 7f170de07640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f16f4073200 0x7f16f4075710 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:52.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.239+0000 7f170de07640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f170814cd60 0x7f17080cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:52.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.239+0000 7f170de07640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1708159080 0x7f1708159490 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:52.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.239+0000 7f170de07640 1 --2- 172.21.15.150:0/3425532686 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16f0004d50 0x7f17080b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:00:52.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.239+0000 7f170de07640 1 -- 172.21.15.150:0/3425532686 >> 172.21.15.150:0/3425532686 conn(0x7f1708074b80 msgr2=0x7f17081dff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:00:52.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.239+0000 7f170de07640 1 -- 172.21.15.150:0/3425532686 shutdown_connections 2024-09-17T05:00:52.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:00:52.239+0000 7f170de07640 1 -- 172.21.15.150:0/3425532686 wait complete. 2024-09-17T05:00:52.261 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-17T05:00:52.262 INFO:tasks.mds_thrash.fs.[a]:waiting for 113 secs before thrashing 2024-09-17T05:01:01.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:01:01.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:01:31.339 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:01:31.342 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:02:01.394 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:02:01.398 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:02:17.098 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:17.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.286+0000 7fcf262ca640 1 -- 172.21.15.150:0/2594812552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf200a8120 msgr2=0x7fcf20074900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:17.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.286+0000 7fcf262ca640 1 --2- 172.21.15.150:0/2594812552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf200a8120 0x7fcf20074900 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fcf20074030 tx=0x7fcf1401c3a0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.286+0000 7fcf262ca640 1 -- 172.21.15.150:0/2594812552 shutdown_connections 2024-09-17T05:02:17.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.286+0000 7fcf262ca640 1 --2- 172.21.15.150:0/2594812552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf200a8120 0x7fcf20074900 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.286+0000 7fcf262ca640 1 --2- 172.21.15.150:0/2594812552 >> v2:172.21.15.176:3300/0 conn(0x7fcf200c39d0 0x7fcf20150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.286+0000 7fcf262ca640 1 -- 172.21.15.150:0/2594812552 >> 172.21.15.150:0/2594812552 conn(0x7fcf2006e360 msgr2=0x7fcf2006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:17.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.286+0000 7fcf262ca640 1 -- 172.21.15.150:0/2594812552 shutdown_connections 2024-09-17T05:02:17.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.286+0000 7fcf262ca640 1 -- 172.21.15.150:0/2594812552 wait complete. 2024-09-17T05:02:17.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf262ca640 1 Processor -- start 2024-09-17T05:02:17.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf262ca640 1 -- start start 2024-09-17T05:02:17.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf262ca640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf20074e40 0x7fcf20156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:17.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf262ca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf200a8120 0x7fcf2014ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf262ca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf200c39d0 0x7fcf201549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf262ca640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcf201613f0 con 0x7fcf200a8120 2024-09-17T05:02:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf262ca640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcf20161270 con 0x7fcf20074e40 2024-09-17T05:02:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf262ca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcf20161570 con 0x7fcf200c39d0 2024-09-17T05:02:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf200a8120 0x7fcf2014ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf200a8120 0x7fcf2014ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44288/0 (socket says 172.21.15.150:44288) 2024-09-17T05:02:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1f7fe640 1 -- 172.21.15.150:0/1886490873 learned_addr learned my addr 172.21.15.150:0/1886490873 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1ffff640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf20074e40 0x7fcf20156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf24867640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf200c39d0 0x7fcf201549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1f7fe640 1 -- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf200c39d0 msgr2=0x7fcf201549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1f7fe640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf200c39d0 0x7fcf201549d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1f7fe640 1 -- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf20074e40 msgr2=0x7fcf20156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:17.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1f7fe640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf20074e40 0x7fcf20156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1f7fe640 1 -- 172.21.15.150:0/1886490873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf20156880 con 0x7fcf200a8120 2024-09-17T05:02:17.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf24867640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf200c39d0 0x7fcf201549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:02:17.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1f7fe640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf200a8120 0x7fcf2014ab30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fcf1401c880 tx=0x7fcf14004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:17.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1cff9640 1 -- 172.21.15.150:0/1886490873 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf140b9020 con 0x7fcf200a8120 2024-09-17T05:02:17.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1cff9640 1 -- 172.21.15.150:0/1886490873 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf14005320 con 0x7fcf200a8120 2024-09-17T05:02:17.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf262ca640 1 -- 172.21.15.150:0/1886490873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf201f4d10 con 0x7fcf200a8120 2024-09-17T05:02:17.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf1cff9640 1 -- 172.21.15.150:0/1886490873 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf140b1040 con 0x7fcf200a8120 2024-09-17T05:02:17.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.290+0000 7fcf262ca640 1 -- 172.21.15.150:0/1886490873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf2014c890 con 0x7fcf200a8120 2024-09-17T05:02:17.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.294+0000 7fcf262ca640 1 -- 172.21.15.150:0/1886490873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf2014db60 con 0x7fcf200a8120 2024-09-17T05:02:17.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.294+0000 7fcf1cff9640 1 -- 172.21.15.150:0/1886490873 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fcf140046d0 con 0x7fcf200a8120 2024-09-17T05:02:17.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.294+0000 7fcf1cff9640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf0c073230 0x7fcf0c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:17.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.294+0000 7fcf1cff9640 1 -- 172.21.15.150:0/1886490873 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7fcf1412dc40 con 0x7fcf200a8120 2024-09-17T05:02:17.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.294+0000 7fcf1ffff640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf0c073230 0x7fcf0c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:17.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.294+0000 7fcf1ffff640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf0c073230 0x7fcf0c075740 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000bb50 tx=0x7fcf1000b2e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:17.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.302+0000 7fcf1cff9640 1 -- 172.21.15.150:0/1886490873 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcf140c3060 con 0x7fcf200a8120 2024-09-17T05:02:17.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.530+0000 7fcf262ca640 1 -- 172.21.15.150:0/1886490873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf20074900 con 0x7fcf200a8120 2024-09-17T05:02:17.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.530+0000 7fcf1cff9640 1 -- 172.21.15.150:0/1886490873 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6534 (secure 0 0 0) 0x7fcf140163d0 con 0x7fcf200a8120 2024-09-17T05:02:17.532 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:17.532 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":91,"btime":"2024-09-17T05:00:52:139685+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6342,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:02:17.532 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 91 2024-09-17T05:02:17.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.534+0000 7fcf262ca640 1 -- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf0c073230 msgr2=0x7fcf0c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:17.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.534+0000 7fcf262ca640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf0c073230 0x7fcf0c075740 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000bb50 tx=0x7fcf1000b2e0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.534+0000 7fcf262ca640 1 -- 172.21.15.150:0/1886490873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf200a8120 msgr2=0x7fcf2014ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:17.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.534+0000 7fcf262ca640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf200a8120 0x7fcf2014ab30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fcf1401c880 tx=0x7fcf14004600 comp rx=0 tx=0).stop 2024-09-17T05:02:17.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.534+0000 7fcf262ca640 1 -- 172.21.15.150:0/1886490873 shutdown_connections 2024-09-17T05:02:17.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.534+0000 7fcf262ca640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf0c073230 0x7fcf0c075740 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.534+0000 7fcf262ca640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf200c39d0 0x7fcf201549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.534+0000 7fcf262ca640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf200a8120 0x7fcf2014ab30 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.534+0000 7fcf262ca640 1 --2- 172.21.15.150:0/1886490873 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf20074e40 0x7fcf20156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.534+0000 7fcf262ca640 1 -- 172.21.15.150:0/1886490873 >> 172.21.15.150:0/1886490873 conn(0x7fcf2006e360 msgr2=0x7fcf200c7420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:17.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.534+0000 7fcf262ca640 1 -- 172.21.15.150:0/1886490873 shutdown_connections 2024-09-17T05:02:17.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.534+0000 7fcf262ca640 1 -- 172.21.15.150:0/1886490873 wait complete. 2024-09-17T05:02:17.558 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-17T05:02:17.558 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T05:02:17.558 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:02:17.578 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T05:02:17.578 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:17.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.726+0000 7f1d959c1640 1 -- 172.21.15.150:0/536580056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d900a85b0 msgr2=0x7f1d900c7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:17.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.726+0000 7f1d959c1640 1 --2- 172.21.15.150:0/536580056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d900a85b0 0x7f1d900c7270 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1d900755f0 tx=0x7f1d7801c770 comp rx=0 tx=0).stop 2024-09-17T05:02:17.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.726+0000 7f1d959c1640 1 -- 172.21.15.150:0/536580056 shutdown_connections 2024-09-17T05:02:17.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.726+0000 7f1d959c1640 1 --2- 172.21.15.150:0/536580056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d900a85b0 0x7f1d900c7270 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.726+0000 7f1d959c1640 1 --2- 172.21.15.150:0/536580056 >> v2:172.21.15.176:3300/0 conn(0x7f1d900c6cb0 0x7f1d900ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.726+0000 7f1d959c1640 1 -- 172.21.15.150:0/536580056 >> 172.21.15.150:0/536580056 conn(0x7f1d900aab60 msgr2=0x7f1d900aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:17.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.726+0000 7f1d959c1640 1 -- 172.21.15.150:0/536580056 shutdown_connections 2024-09-17T05:02:17.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.726+0000 7f1d959c1640 1 -- 172.21.15.150:0/536580056 wait complete. 2024-09-17T05:02:17.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d959c1640 1 Processor -- start 2024-09-17T05:02:17.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d959c1640 1 -- start start 2024-09-17T05:02:17.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d959c1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d900a85b0 0x7f1d9014f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:17.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d959c1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d900c6cb0 0x7f1d900a9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:17.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d959c1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d900c77b0 0x7f1d90144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:17.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d959c1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1d900c3460 con 0x7f1d900c6cb0 2024-09-17T05:02:17.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d959c1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1d900c32e0 con 0x7f1d900c77b0 2024-09-17T05:02:17.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d959c1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1d90161890 con 0x7f1d900a85b0 2024-09-17T05:02:17.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d87fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d900c6cb0 0x7f1d900a9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:17.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d87fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d900c6cb0 0x7f1d900a9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44314/0 (socket says 172.21.15.150:44314) 2024-09-17T05:02:17.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d87fff640 1 -- 172.21.15.150:0/685653655 learned_addr learned my addr 172.21.15.150:0/685653655 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:17.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d8f7fe640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d900c77b0 0x7f1d90144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:17.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d8effd640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d900a85b0 0x7f1d9014f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:17.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d87fff640 1 -- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d900a85b0 msgr2=0x7f1d9014f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:17.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d87fff640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d900a85b0 0x7f1d9014f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d87fff640 1 -- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d900c77b0 msgr2=0x7f1d90144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:17.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d87fff640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d900c77b0 0x7f1d90144eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:17.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d87fff640 1 -- 172.21.15.150:0/685653655 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d9014d4f0 con 0x7f1d900c6cb0 2024-09-17T05:02:17.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d8f7fe640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d900c77b0 0x7f1d90144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:02:17.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d8effd640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d900a85b0 0x7f1d9014f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:02:17.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d87fff640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d900c6cb0 0x7f1d900a9cf0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1d7801a3a0 tx=0x7f1d780a6460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:17.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d8cff9640 1 -- 172.21.15.150:0/685653655 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d780be070 con 0x7f1d900c6cb0 2024-09-17T05:02:17.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d8cff9640 1 -- 172.21.15.150:0/685653655 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d78005040 con 0x7f1d900c6cb0 2024-09-17T05:02:17.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d959c1640 1 -- 172.21.15.150:0/685653655 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d9014d790 con 0x7f1d900c6cb0 2024-09-17T05:02:17.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d8cff9640 1 -- 172.21.15.150:0/685653655 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d78005470 con 0x7f1d900c6cb0 2024-09-17T05:02:17.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.730+0000 7f1d959c1640 1 -- 172.21.15.150:0/685653655 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d9015e010 con 0x7f1d900c6cb0 2024-09-17T05:02:17.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.734+0000 7f1d959c1640 1 -- 172.21.15.150:0/685653655 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d900c7270 con 0x7f1d900c6cb0 2024-09-17T05:02:17.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.734+0000 7f1d8cff9640 1 -- 172.21.15.150:0/685653655 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1d780046d0 con 0x7f1d900c6cb0 2024-09-17T05:02:17.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.734+0000 7f1d8cff9640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d70073200 0x7f1d70075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:17.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.734+0000 7f1d8cff9640 1 -- 172.21.15.150:0/685653655 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f1d7812e7f0 con 0x7f1d900c6cb0 2024-09-17T05:02:17.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.734+0000 7f1d8effd640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d70073200 0x7f1d70075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:17.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.734+0000 7f1d8effd640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d70073200 0x7f1d70075710 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1d80009b90 tx=0x7f1d800091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:17.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:17.742+0000 7f1d8cff9640 1 -- 172.21.15.150:0/685653655 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1d780ac030 con 0x7f1d900c6cb0 2024-09-17T05:02:18.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.002+0000 7f1d959c1640 1 -- 172.21.15.150:0/685653655 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d9015e580 con 0x7f1d900c6cb0 2024-09-17T05:02:18.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.002+0000 7f1d8cff9640 1 -- 172.21.15.150:0/685653655 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6534 (secure 0 0 0) 0x7f1d780163f0 con 0x7f1d900c6cb0 2024-09-17T05:02:18.003 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:18.003 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":91,"btime":"2024-09-17T05:00:52:139685+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6342,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:02:18.003 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 91 2024-09-17T05:02:18.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.006+0000 7f1d959c1640 1 -- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d70073200 msgr2=0x7f1d70075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:18.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.006+0000 7f1d959c1640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d70073200 0x7f1d70075710 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1d80009b90 tx=0x7f1d800091e0 comp rx=0 tx=0).stop 2024-09-17T05:02:18.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.006+0000 7f1d959c1640 1 -- 172.21.15.150:0/685653655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d900c6cb0 msgr2=0x7f1d900a9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:18.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.006+0000 7f1d959c1640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d900c6cb0 0x7f1d900a9cf0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1d7801a3a0 tx=0x7f1d780a6460 comp rx=0 tx=0).stop 2024-09-17T05:02:18.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.006+0000 7f1d959c1640 1 -- 172.21.15.150:0/685653655 shutdown_connections 2024-09-17T05:02:18.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.006+0000 7f1d959c1640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d70073200 0x7f1d70075710 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:18.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.006+0000 7f1d959c1640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d900c77b0 0x7f1d90144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:18.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.006+0000 7f1d959c1640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d900c6cb0 0x7f1d900a9cf0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:18.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.006+0000 7f1d959c1640 1 --2- 172.21.15.150:0/685653655 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d900a85b0 0x7f1d9014f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:18.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.006+0000 7f1d959c1640 1 -- 172.21.15.150:0/685653655 >> 172.21.15.150:0/685653655 conn(0x7f1d900aab60 msgr2=0x7f1d900c4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:18.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.006+0000 7f1d959c1640 1 -- 172.21.15.150:0/685653655 shutdown_connections 2024-09-17T05:02:18.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:18.006+0000 7f1d959c1640 1 -- 172.21.15.150:0/685653655 wait complete. 2024-09-17T05:02:18.024 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:02:20.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:20.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.202+0000 7efee9cbb640 1 -- 172.21.15.150:0/4266573491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efed0004d50 msgr2=0x7efed0005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:20.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.202+0000 7efee9cbb640 1 --2- 172.21.15.150:0/4266573491 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efed0004d50 0x7efed0005160 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7efedc009f10 tx=0x7efedc01d5f0 comp rx=0 tx=0).stop 2024-09-17T05:02:20.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.202+0000 7efee3fff640 1 -- 172.21.15.150:0/4266573491 reap_dead start 2024-09-17T05:02:20.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.202+0000 7efee9cbb640 1 -- 172.21.15.150:0/4266573491 shutdown_connections 2024-09-17T05:02:20.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.202+0000 7efee9cbb640 1 -- 172.21.15.150:0/4266573491 >> 172.21.15.150:0/4266573491 conn(0x7efee40aab60 msgr2=0x7efee40aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:20.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.202+0000 7efee9cbb640 1 -- 172.21.15.150:0/4266573491 shutdown_connections 2024-09-17T05:02:20.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.202+0000 7efee9cbb640 1 -- 172.21.15.150:0/4266573491 wait complete. 2024-09-17T05:02:20.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee9cbb640 1 Processor -- start 2024-09-17T05:02:20.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee9cbb640 1 -- start start 2024-09-17T05:02:20.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee9cbb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efed0004d50 0x7efee41547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:20.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee9cbb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efee414d6f0 0x7efee41e86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:20.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee9cbb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efee4157290 0x7efee4148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:20.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee9cbb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efee414a480 con 0x7efee414d6f0 2024-09-17T05:02:20.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee9cbb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efee4072500 con 0x7efee4157290 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee9cbb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efee41544c0 con 0x7efed0004d50 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efedbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efee414d6f0 0x7efee41e86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efedbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efee414d6f0 0x7efee41e86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35366/0 (socket says 172.21.15.150:35366) 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efedbfff640 1 -- 172.21.15.150:0/2778546261 learned_addr learned my addr 172.21.15.150:0/2778546261 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee37fe640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efed0004d50 0x7efee41547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee3fff640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efee4157290 0x7efee4148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efedbfff640 1 -- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efed0004d50 msgr2=0x7efee41547f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efedbfff640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efed0004d50 0x7efee41547f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efedbfff640 1 -- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efee4157290 msgr2=0x7efee4148d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efedbfff640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efee4157290 0x7efee4148d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efedbfff640 1 -- 172.21.15.150:0/2778546261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee4149ef0 con 0x7efee414d6f0 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee37fe640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efed0004d50 0x7efee41547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:02:20.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efedbfff640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efee414d6f0 0x7efee41e86d0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7efecc00e970 tx=0x7efecc00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:20.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee0ff9640 1 -- 172.21.15.150:0/2778546261 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc00cc60 con 0x7efee414d6f0 2024-09-17T05:02:20.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee0ff9640 1 -- 172.21.15.150:0/2778546261 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efecc00ce00 con 0x7efee414d6f0 2024-09-17T05:02:20.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee9cbb640 1 -- 172.21.15.150:0/2778546261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee4156730 con 0x7efee414d6f0 2024-09-17T05:02:20.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee0ff9640 1 -- 172.21.15.150:0/2778546261 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc005280 con 0x7efee414d6f0 2024-09-17T05:02:20.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee9cbb640 1 -- 172.21.15.150:0/2778546261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee41569d0 con 0x7efee414d6f0 2024-09-17T05:02:20.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.206+0000 7efee9cbb640 1 -- 172.21.15.150:0/2778546261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee4156ef0 con 0x7efee414d6f0 2024-09-17T05:02:20.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.210+0000 7efee0ff9640 1 -- 172.21.15.150:0/2778546261 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7efecc005420 con 0x7efee414d6f0 2024-09-17T05:02:20.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.210+0000 7efee0ff9640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efec4073200 0x7efec4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:20.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.210+0000 7efee0ff9640 1 -- 172.21.15.150:0/2778546261 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7efecc095420 con 0x7efee414d6f0 2024-09-17T05:02:20.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.210+0000 7efee37fe640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efec4073200 0x7efec4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:20.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.210+0000 7efee37fe640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efec4073200 0x7efec4075710 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7efed4001ec0 tx=0x7efed40016d0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:20.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.214+0000 7efee0ff9640 1 -- 172.21.15.150:0/2778546261 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7efecc05fc00 con 0x7efee414d6f0 2024-09-17T05:02:20.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.446+0000 7efee9cbb640 1 -- 172.21.15.150:0/2778546261 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efee40c8330 con 0x7efee414d6f0 2024-09-17T05:02:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.450+0000 7efee0ff9640 1 -- 172.21.15.150:0/2778546261 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6534 (secure 0 0 0) 0x7efecc05f500 con 0x7efee414d6f0 2024-09-17T05:02:20.448 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:20.448 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":91,"btime":"2024-09-17T05:00:52:139685+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6342,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:02:20.449 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 91 2024-09-17T05:02:20.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.450+0000 7efee9cbb640 1 -- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efec4073200 msgr2=0x7efec4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:20.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.450+0000 7efee9cbb640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efec4073200 0x7efec4075710 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7efed4001ec0 tx=0x7efed40016d0 comp rx=0 tx=0).stop 2024-09-17T05:02:20.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.450+0000 7efee9cbb640 1 -- 172.21.15.150:0/2778546261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efee414d6f0 msgr2=0x7efee41e86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:20.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.450+0000 7efee9cbb640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efee414d6f0 0x7efee41e86d0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7efecc00e970 tx=0x7efecc00ee30 comp rx=0 tx=0).stop 2024-09-17T05:02:20.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.450+0000 7efee9cbb640 1 -- 172.21.15.150:0/2778546261 shutdown_connections 2024-09-17T05:02:20.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.450+0000 7efee9cbb640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efec4073200 0x7efec4075710 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:20.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.454+0000 7efee9cbb640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efee4157290 0x7efee4148d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:20.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.454+0000 7efee9cbb640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efee414d6f0 0x7efee41e86d0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:20.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.454+0000 7efee9cbb640 1 --2- 172.21.15.150:0/2778546261 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efed0004d50 0x7efee41547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:20.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.454+0000 7efee9cbb640 1 -- 172.21.15.150:0/2778546261 >> 172.21.15.150:0/2778546261 conn(0x7efee40aab60 msgr2=0x7efee406e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:20.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.454+0000 7efee9cbb640 1 -- 172.21.15.150:0/2778546261 shutdown_connections 2024-09-17T05:02:20.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:20.454+0000 7efee9cbb640 1 -- 172.21.15.150:0/2778546261 wait complete. 2024-09-17T05:02:20.473 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:02:22.474 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:22.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.650+0000 7f2cd9f51640 1 -- 172.21.15.150:0/99975157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cbc004f50 msgr2=0x7f2cbc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:22.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.650+0000 7f2cd9f51640 1 --2- 172.21.15.150:0/99975157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cbc004f50 0x7f2cbc00b9d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4075420 tx=0x7f2ccc01d200 comp rx=0 tx=0).stop 2024-09-17T05:02:22.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.650+0000 7f2cd3fff640 1 -- 172.21.15.150:0/99975157 reap_dead start 2024-09-17T05:02:22.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.650+0000 7f2cd9f51640 1 -- 172.21.15.150:0/99975157 shutdown_connections 2024-09-17T05:02:22.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.650+0000 7f2cd9f51640 1 -- 172.21.15.150:0/99975157 >> 172.21.15.150:0/99975157 conn(0x7f2cd40c5ae0 msgr2=0x7f2cd4073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:22.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.650+0000 7f2cd9f51640 1 -- 172.21.15.150:0/99975157 shutdown_connections 2024-09-17T05:02:22.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.650+0000 7f2cd9f51640 1 -- 172.21.15.150:0/99975157 wait complete. 2024-09-17T05:02:22.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.650+0000 7f2cd9f51640 1 Processor -- start 2024-09-17T05:02:22.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.650+0000 7f2cd9f51640 1 -- start start 2024-09-17T05:02:22.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd9f51640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd40c1c20 0x7f2cd4143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:22.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd9f51640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cd40c2030 0x7f2cd4143670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:22.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd9f51640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cd41efe40 0x7f2cd41f0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd9f51640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd40c3660 con 0x7f2cd40c2030 2024-09-17T05:02:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd9f51640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd40c34e0 con 0x7f2cd41efe40 2024-09-17T05:02:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd9f51640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2cd41609b0 con 0x7f2cd40c1c20 2024-09-17T05:02:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cd40c2030 0x7f2cd4143670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cd40c2030 0x7f2cd4143670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35390/0 (socket says 172.21.15.150:35390) 2024-09-17T05:02:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd2ffd640 1 -- 172.21.15.150:0/440821252 learned_addr learned my addr 172.21.15.150:0/440821252 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd37fe640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd40c1c20 0x7f2cd4143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd3fff640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cd41efe40 0x7f2cd41f0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd2ffd640 1 -- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd40c1c20 msgr2=0x7f2cd4143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd2ffd640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd40c1c20 0x7f2cd4143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:22.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd2ffd640 1 -- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cd41efe40 msgr2=0x7f2cd41f0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:22.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd2ffd640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cd41efe40 0x7f2cd41f0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:22.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd2ffd640 1 -- 172.21.15.150:0/440821252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd4153f30 con 0x7f2cd40c2030 2024-09-17T05:02:22.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd2ffd640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cd40c2030 0x7f2cd4143670 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8004da0 tx=0x7f2cc8001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:22.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cbbfff640 1 -- 172.21.15.150:0/440821252 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8011660 con 0x7f2cd40c2030 2024-09-17T05:02:22.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cbbfff640 1 -- 172.21.15.150:0/440821252 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cc8016070 con 0x7f2cd40c2030 2024-09-17T05:02:22.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd9f51640 1 -- 172.21.15.150:0/440821252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cd4156c90 con 0x7f2cd40c2030 2024-09-17T05:02:22.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cbbfff640 1 -- 172.21.15.150:0/440821252 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8011b90 con 0x7f2cd40c2030 2024-09-17T05:02:22.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd9f51640 1 -- 172.21.15.150:0/440821252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cd40710f0 con 0x7f2cd40c2030 2024-09-17T05:02:22.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cd9f51640 1 -- 172.21.15.150:0/440821252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cd4156a80 con 0x7f2cd40c2030 2024-09-17T05:02:22.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cbbfff640 1 -- 172.21.15.150:0/440821252 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2cc801b460 con 0x7f2cd40c2030 2024-09-17T05:02:22.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cbbfff640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cc0073200 0x7f2cc0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:22.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.654+0000 7f2cbbfff640 1 -- 172.21.15.150:0/440821252 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f2cc80950b0 con 0x7f2cd40c2030 2024-09-17T05:02:22.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.658+0000 7f2cd37fe640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cc0073200 0x7f2cc0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:22.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.658+0000 7f2cd37fe640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cc0073200 0x7f2cc0075710 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4157d80 tx=0x7f2cc400d390 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:22.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.662+0000 7f2cbbfff640 1 -- 172.21.15.150:0/440821252 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2cc805f920 con 0x7f2cd40c2030 2024-09-17T05:02:22.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.882+0000 7f2cd9f51640 1 -- 172.21.15.150:0/440821252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2cd40743a0 con 0x7f2cd40c2030 2024-09-17T05:02:22.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.882+0000 7f2cbbfff640 1 -- 172.21.15.150:0/440821252 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6534 (secure 0 0 0) 0x7f2cc805f220 con 0x7f2cd40c2030 2024-09-17T05:02:22.883 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:22.883 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":91,"btime":"2024-09-17T05:00:52:139685+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6342,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:02:22.883 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 91 2024-09-17T05:02:22.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.886+0000 7f2cd9f51640 1 -- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cc0073200 msgr2=0x7f2cc0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:22.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.886+0000 7f2cd9f51640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cc0073200 0x7f2cc0075710 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4157d80 tx=0x7f2cc400d390 comp rx=0 tx=0).stop 2024-09-17T05:02:22.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.886+0000 7f2cd9f51640 1 -- 172.21.15.150:0/440821252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cd40c2030 msgr2=0x7f2cd4143670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:22.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.886+0000 7f2cd9f51640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cd40c2030 0x7f2cd4143670 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8004da0 tx=0x7f2cc8001640 comp rx=0 tx=0).stop 2024-09-17T05:02:22.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.886+0000 7f2cd9f51640 1 -- 172.21.15.150:0/440821252 shutdown_connections 2024-09-17T05:02:22.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.886+0000 7f2cd9f51640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cc0073200 0x7f2cc0075710 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:22.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.886+0000 7f2cd9f51640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2cd41efe40 0x7f2cd41f0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:22.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.886+0000 7f2cd9f51640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2cd40c2030 0x7f2cd4143670 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:22.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.886+0000 7f2cd9f51640 1 --2- 172.21.15.150:0/440821252 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2cd40c1c20 0x7f2cd4143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:22.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.886+0000 7f2cd9f51640 1 -- 172.21.15.150:0/440821252 >> 172.21.15.150:0/440821252 conn(0x7f2cd40c5ae0 msgr2=0x7f2cd41581c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:22.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.886+0000 7f2cd9f51640 1 -- 172.21.15.150:0/440821252 shutdown_connections 2024-09-17T05:02:22.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:22.886+0000 7f2cd9f51640 1 -- 172.21.15.150:0/440821252 wait complete. 2024-09-17T05:02:22.906 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:02:24.907 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:25.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.086+0000 7f7c73d78640 1 -- 172.21.15.150:0/3701057893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c6c157ca0 msgr2=0x7f7c6c1580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:25.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.086+0000 7f7c73d78640 1 --2- 172.21.15.150:0/3701057893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c6c157ca0 0x7f7c6c1580b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009bc0 tx=0x7f7c6401ca80 comp rx=0 tx=0).stop 2024-09-17T05:02:25.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.086+0000 7f7c73d78640 1 -- 172.21.15.150:0/3701057893 shutdown_connections 2024-09-17T05:02:25.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.086+0000 7f7c73d78640 1 --2- 172.21.15.150:0/3701057893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c6c157ca0 0x7f7c6c1580b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:25.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.086+0000 7f7c73d78640 1 --2- 172.21.15.150:0/3701057893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c6c0c6230 0x7f7c6c0c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:25.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.086+0000 7f7c73d78640 1 -- 172.21.15.150:0/3701057893 >> 172.21.15.150:0/3701057893 conn(0x7f7c6c0c9de0 msgr2=0x7f7c6c0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:25.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.086+0000 7f7c73d78640 1 -- 172.21.15.150:0/3701057893 shutdown_connections 2024-09-17T05:02:25.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.086+0000 7f7c73d78640 1 -- 172.21.15.150:0/3701057893 wait complete. 2024-09-17T05:02:25.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c73d78640 1 Processor -- start 2024-09-17T05:02:25.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c73d78640 1 -- start start 2024-09-17T05:02:25.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c73d78640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c6c0ad7f0 0x7f7c6c14c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:25.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c73d78640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c6c0c6230 0x7f7c6c1ed6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:25.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c73d78640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c6c157ca0 0x7f7c6c0727e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:25.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c73d78640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c0c59f0 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c73d78640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c0c5870 con 0x7f7c6c0c6230 2024-09-17T05:02:25.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c73d78640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7c6c0c7ce0 con 0x7f7c6c157ca0 2024-09-17T05:02:25.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c71b14640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c6c0ad7f0 0x7f7c6c14c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:25.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c71b14640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c6c0ad7f0 0x7f7c6c14c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35400/0 (socket says 172.21.15.150:35400) 2024-09-17T05:02:25.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c71b14640 1 -- 172.21.15.150:0/363147826 learned_addr learned my addr 172.21.15.150:0/363147826 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:25.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c71313640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c6c0c6230 0x7f7c6c1ed6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:25.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c72315640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c6c157ca0 0x7f7c6c0727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:25.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c71b14640 1 -- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c6c157ca0 msgr2=0x7f7c6c0727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:25.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c71b14640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c6c157ca0 0x7f7c6c0727e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c71b14640 1 -- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c6c0c6230 msgr2=0x7f7c6c1ed6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c71b14640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c6c0c6230 0x7f7c6c1ed6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c71b14640 1 -- 172.21.15.150:0/363147826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c6c15e050 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c71313640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c6c0c6230 0x7f7c6c1ed6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c71b14640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c6c0ad7f0 0x7f7c6c14c520 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800c3a0 tx=0x7f7c6800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c627fc640 1 -- 172.21.15.150:0/363147826 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68004020 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c627fc640 1 -- 172.21.15.150:0/363147826 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c680042b0 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c73d78640 1 -- 172.21.15.150:0/363147826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c6c147ba0 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c627fc640 1 -- 172.21.15.150:0/363147826 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c680038b0 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.090+0000 7f7c73d78640 1 -- 172.21.15.150:0/363147826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c6c14ba40 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.094+0000 7f7c73d78640 1 -- 172.21.15.150:0/363147826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c6c0c5330 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.094+0000 7f7c627fc640 1 -- 172.21.15.150:0/363147826 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7c68019020 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.094+0000 7f7c627fc640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c50073200 0x7f7c50075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:25.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.094+0000 7f7c627fc640 1 -- 172.21.15.150:0/363147826 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f7c68094240 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.094+0000 7f7c71313640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c50073200 0x7f7c50075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:25.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.094+0000 7f7c71313640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c50073200 0x7f7c50075710 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c14ef60 tx=0x7f7c5c0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:25.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.098+0000 7f7c627fc640 1 -- 172.21.15.150:0/363147826 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7c6805ea30 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.338+0000 7f7c73d78640 1 -- 172.21.15.150:0/363147826 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c6c0aa590 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.338+0000 7f7c627fc640 1 -- 172.21.15.150:0/363147826 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6534 (secure 0 0 0) 0x7f7c6805e330 con 0x7f7c6c0ad7f0 2024-09-17T05:02:25.334 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:25.335 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":91,"btime":"2024-09-17T05:00:52:139685+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6342,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:02:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 91 2024-09-17T05:02:25.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.342+0000 7f7c73d78640 1 -- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c50073200 msgr2=0x7f7c50075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.342+0000 7f7c73d78640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c50073200 0x7f7c50075710 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c14ef60 tx=0x7f7c5c0091e0 comp rx=0 tx=0).stop 2024-09-17T05:02:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.342+0000 7f7c73d78640 1 -- 172.21.15.150:0/363147826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c6c0ad7f0 msgr2=0x7f7c6c14c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.342+0000 7f7c73d78640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c6c0ad7f0 0x7f7c6c14c520 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800c3a0 tx=0x7f7c6800c860 comp rx=0 tx=0).stop 2024-09-17T05:02:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.342+0000 7f7c73d78640 1 -- 172.21.15.150:0/363147826 shutdown_connections 2024-09-17T05:02:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.342+0000 7f7c73d78640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c50073200 0x7f7c50075710 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.342+0000 7f7c73d78640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c6c0c6230 0x7f7c6c1ed6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.342+0000 7f7c73d78640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c6c157ca0 0x7f7c6c0727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.342+0000 7f7c73d78640 1 --2- 172.21.15.150:0/363147826 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c6c0ad7f0 0x7f7c6c14c520 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.342+0000 7f7c73d78640 1 -- 172.21.15.150:0/363147826 >> 172.21.15.150:0/363147826 conn(0x7f7c6c0c9de0 msgr2=0x7f7c6c0aaab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.342+0000 7f7c73d78640 1 -- 172.21.15.150:0/363147826 shutdown_connections 2024-09-17T05:02:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:25.342+0000 7f7c73d78640 1 -- 172.21.15.150:0/363147826 wait complete. 2024-09-17T05:02:25.360 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:02:27.361 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:27.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.534+0000 7f844f7e7640 1 -- 172.21.15.150:0/3027191009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84480c7eb0 msgr2=0x7f84481540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:27.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.534+0000 7f844f7e7640 1 --2- 172.21.15.150:0/3027191009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84480c7eb0 0x7f84481540b0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f84480755f0 tx=0x7f844401c8e0 comp rx=0 tx=0).stop 2024-09-17T05:02:27.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.534+0000 7f844f7e7640 1 -- 172.21.15.150:0/3027191009 shutdown_connections 2024-09-17T05:02:27.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.534+0000 7f844f7e7640 1 --2- 172.21.15.150:0/3027191009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8448149d70 0x7f844814a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:27.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.534+0000 7f844f7e7640 1 --2- 172.21.15.150:0/3027191009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84480c7eb0 0x7f84481540b0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:27.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.534+0000 7f844f7e7640 1 -- 172.21.15.150:0/3027191009 >> 172.21.15.150:0/3027191009 conn(0x7f84480aab60 msgr2=0x7f84480aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:27.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.534+0000 7f844f7e7640 1 -- 172.21.15.150:0/3027191009 shutdown_connections 2024-09-17T05:02:27.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.534+0000 7f844f7e7640 1 -- 172.21.15.150:0/3027191009 wait complete. 2024-09-17T05:02:27.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.534+0000 7f844f7e7640 1 Processor -- start 2024-09-17T05:02:27.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.534+0000 7f844f7e7640 1 -- start start 2024-09-17T05:02:27.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844f7e7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84480c7eb0 0x7f844814f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:27.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844f7e7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84480c8330 0x7f84480ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:27.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844f7e7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8448149d70 0x7f84481492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:27.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844f7e7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f84481568d0 con 0x7f84480c8330 2024-09-17T05:02:27.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844f7e7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8448156750 con 0x7f84480c7eb0 2024-09-17T05:02:27.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844f7e7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f84480c8b50 con 0x7f8448149d70 2024-09-17T05:02:27.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844cd82640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84480c8330 0x7f84480ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:27.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844cd82640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84480c8330 0x7f84480ae8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35422/0 (socket says 172.21.15.150:35422) 2024-09-17T05:02:27.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844cd82640 1 -- 172.21.15.150:0/3032494896 learned_addr learned my addr 172.21.15.150:0/3032494896 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:27.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844d583640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84480c7eb0 0x7f844814f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:27.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844cd82640 1 -- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8448149d70 msgr2=0x7f84481492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:27.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844dd84640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8448149d70 0x7f84481492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:27.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844cd82640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8448149d70 0x7f84481492b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:27.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844cd82640 1 -- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84480c7eb0 msgr2=0x7f844814f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:27.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844cd82640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84480c7eb0 0x7f844814f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:27.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844cd82640 1 -- 172.21.15.150:0/3032494896 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84480ad060 con 0x7f84480c8330 2024-09-17T05:02:27.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844d583640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84480c7eb0 0x7f844814f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:02:27.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844cd82640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84480c8330 0x7f84480ae8b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f843800c970 tx=0x7f843800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:27.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f843dffb640 1 -- 172.21.15.150:0/3032494896 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8438004220 con 0x7f84480c8330 2024-09-17T05:02:27.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f843dffb640 1 -- 172.21.15.150:0/3032494896 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84380079b0 con 0x7f84480c8330 2024-09-17T05:02:27.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f843dffb640 1 -- 172.21.15.150:0/3032494896 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84380044b0 con 0x7f84480c8330 2024-09-17T05:02:27.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844f7e7640 1 -- 172.21.15.150:0/3032494896 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84480ac1f0 con 0x7f84480c8330 2024-09-17T05:02:27.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844f7e7640 1 -- 172.21.15.150:0/3032494896 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8448150a90 con 0x7f84480c8330 2024-09-17T05:02:27.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f844f7e7640 1 -- 172.21.15.150:0/3032494896 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84480ac430 con 0x7f84480c8330 2024-09-17T05:02:27.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f843dffb640 1 -- 172.21.15.150:0/3032494896 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8438003600 con 0x7f84480c8330 2024-09-17T05:02:27.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f843dffb640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f842c073200 0x7f842c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:27.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.538+0000 7f843dffb640 1 -- 172.21.15.150:0/3032494896 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f8438094bf0 con 0x7f84480c8330 2024-09-17T05:02:27.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.542+0000 7f844d583640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f842c073200 0x7f842c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:27.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.542+0000 7f844d583640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f842c073200 0x7f842c075710 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f84440af000 tx=0x7f844401b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:27.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.546+0000 7f843dffb640 1 -- 172.21.15.150:0/3032494896 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f84480ac430 con 0x7f84480c8330 2024-09-17T05:02:27.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.806+0000 7f844f7e7640 1 -- 172.21.15.150:0/3032494896 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8448151640 con 0x7f84480c8330 2024-09-17T05:02:27.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f843dffb640 1 -- 172.21.15.150:0/3032494896 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6534 (secure 0 0 0) 0x7f8438014030 con 0x7f84480c8330 2024-09-17T05:02:27.804 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:27.804 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":91,"btime":"2024-09-17T05:00:52:139685+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6342,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:02:27.805 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 91 2024-09-17T05:02:27.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f844f7e7640 1 -- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f842c073200 msgr2=0x7f842c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:27.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f844f7e7640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f842c073200 0x7f842c075710 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f84440af000 tx=0x7f844401b120 comp rx=0 tx=0).stop 2024-09-17T05:02:27.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f844f7e7640 1 -- 172.21.15.150:0/3032494896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84480c8330 msgr2=0x7f84480ae8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:27.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f844f7e7640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84480c8330 0x7f84480ae8b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f843800c970 tx=0x7f843800ce30 comp rx=0 tx=0).stop 2024-09-17T05:02:27.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f844f7e7640 1 -- 172.21.15.150:0/3032494896 shutdown_connections 2024-09-17T05:02:27.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f844f7e7640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f842c073200 0x7f842c075710 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:27.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f844f7e7640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8448149d70 0x7f84481492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:27.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f844f7e7640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84480c8330 0x7f84480ae8b0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:27.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f844f7e7640 1 --2- 172.21.15.150:0/3032494896 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84480c7eb0 0x7f844814f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:27.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f844f7e7640 1 -- 172.21.15.150:0/3032494896 >> 172.21.15.150:0/3032494896 conn(0x7f84480aab60 msgr2=0x7f844815e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:27.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f844f7e7640 1 -- 172.21.15.150:0/3032494896 shutdown_connections 2024-09-17T05:02:27.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:27.810+0000 7f844f7e7640 1 -- 172.21.15.150:0/3032494896 wait complete. 2024-09-17T05:02:27.828 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:02:29.829 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:30.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.014+0000 7fedc40b3640 1 -- 172.21.15.150:0/3189347422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedbc0c4950 msgr2=0x7fedbc0aa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:30.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.014+0000 7fedc40b3640 1 --2- 172.21.15.150:0/3189347422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedbc0c4950 0x7fedbc0aa1d0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fedbc0aea20 tx=0x7fedb801db60 comp rx=0 tx=0).stop 2024-09-17T05:02:30.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.014+0000 7fedc40b3640 1 -- 172.21.15.150:0/3189347422 shutdown_connections 2024-09-17T05:02:30.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.014+0000 7fedc40b3640 1 --2- 172.21.15.150:0/3189347422 >> v2:172.21.15.176:3300/0 conn(0x7fedbc14cec0 0x7fedbc06ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:30.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.014+0000 7fedc40b3640 1 --2- 172.21.15.150:0/3189347422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedbc0c4950 0x7fedbc0aa1d0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:30.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.014+0000 7fedc40b3640 1 -- 172.21.15.150:0/3189347422 >> 172.21.15.150:0/3189347422 conn(0x7fedbc070b50 msgr2=0x7fedbc070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:30.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.014+0000 7fedc40b3640 1 -- 172.21.15.150:0/3189347422 shutdown_connections 2024-09-17T05:02:30.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.014+0000 7fedc40b3640 1 -- 172.21.15.150:0/3189347422 wait complete. 2024-09-17T05:02:30.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.014+0000 7fedc40b3640 1 Processor -- start 2024-09-17T05:02:30.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.014+0000 7fedc40b3640 1 -- start start 2024-09-17T05:02:30.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc40b3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedbc0aa7b0 0x7fedbc0ad790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:30.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc40b3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fedbc0c4950 0x7fedbc155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:30.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc40b3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedbc14cec0 0x7fedbc1419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:30.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc40b3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc06d510 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc40b3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc06d390 con 0x7fedbc14cec0 2024-09-17T05:02:30.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc40b3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fedbc15f6b0 con 0x7fedbc0c4950 2024-09-17T05:02:30.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc1e4f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedbc0aa7b0 0x7fedbc0ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc1e4f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedbc0aa7b0 0x7fedbc0ad790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57252/0 (socket says 172.21.15.150:57252) 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc1e4f640 1 -- 172.21.15.150:0/2893446523 learned_addr learned my addr 172.21.15.150:0/2893446523 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc2650640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedbc14cec0 0x7fedbc1419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc164e640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fedbc0c4950 0x7fedbc155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc1e4f640 1 -- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fedbc0c4950 msgr2=0x7fedbc155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc1e4f640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fedbc0c4950 0x7fedbc155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc1e4f640 1 -- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedbc14cec0 msgr2=0x7fedbc1419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc1e4f640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedbc14cec0 0x7fedbc1419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc1e4f640 1 -- 172.21.15.150:0/2893446523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedbc0adcd0 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc2650640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedbc14cec0 0x7fedbc1419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc1e4f640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedbc0aa7b0 0x7fedbc0ad790 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fedb801c210 tx=0x7fedb8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedb27fc640 1 -- 172.21.15.150:0/2893446523 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb80b3070 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedb27fc640 1 -- 172.21.15.150:0/2893446523 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fedb8005530 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedb27fc640 1 -- 172.21.15.150:0/2893446523 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8004930 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc40b3640 1 -- 172.21.15.150:0/2893446523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedbc155780 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc40b3640 1 -- 172.21.15.150:0/2893446523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedbc1542b0 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.018+0000 7fedc40b3640 1 -- 172.21.15.150:0/2893446523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedbc0aa1d0 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.022+0000 7fedb27fc640 1 -- 172.21.15.150:0/2893446523 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fedb80be060 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.022+0000 7fedb27fc640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feda0073230 0x7feda0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:30.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.022+0000 7fedb27fc640 1 -- 172.21.15.150:0/2893446523 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7fedb812ed30 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.022+0000 7fedc164e640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feda0073230 0x7feda0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:30.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.022+0000 7fedc164e640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feda0073230 0x7feda0075740 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fedac0058d0 tx=0x7fedac005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:30.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.026+0000 7fedb27fc640 1 -- 172.21.15.150:0/2893446523 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fedb80177e0 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.262+0000 7fedc40b3640 1 -- 172.21.15.150:0/2893446523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedbc071bf0 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.262+0000 7fedb27fc640 1 -- 172.21.15.150:0/2893446523 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+6534 (secure 0 0 0) 0x7fedb80f9490 con 0x7fedbc0aa7b0 2024-09-17T05:02:30.259 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:30.259 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":91,"btime":"2024-09-17T05:00:52:139685+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6342,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:00:23.977031+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5375},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5375":{"gid":5375,"name":"d","rank":0,"incarnation":82,"state":"up:active","state_seq":18,"addr":"172.21.15.176:6837/702608997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":702608997},{"type":"v1","addr":"172.21.15.176:6837","nonce":702608997}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5375,"qdb_cluster":[5375]},"id":2}]} 2024-09-17T05:02:30.259 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 91 2024-09-17T05:02:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.266+0000 7fedc40b3640 1 -- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feda0073230 msgr2=0x7feda0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.266+0000 7fedc40b3640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feda0073230 0x7feda0075740 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fedac0058d0 tx=0x7fedac005860 comp rx=0 tx=0).stop 2024-09-17T05:02:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.266+0000 7fedc40b3640 1 -- 172.21.15.150:0/2893446523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedbc0aa7b0 msgr2=0x7fedbc0ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.266+0000 7fedc40b3640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedbc0aa7b0 0x7fedbc0ad790 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fedb801c210 tx=0x7fedb8004600 comp rx=0 tx=0).stop 2024-09-17T05:02:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.266+0000 7fedc40b3640 1 -- 172.21.15.150:0/2893446523 shutdown_connections 2024-09-17T05:02:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.266+0000 7fedc40b3640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feda0073230 0x7feda0075740 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.266+0000 7fedc40b3640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedbc14cec0 0x7fedbc1419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.266+0000 7fedc40b3640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fedbc0c4950 0x7fedbc155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.266+0000 7fedc40b3640 1 --2- 172.21.15.150:0/2893446523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fedbc0aa7b0 0x7fedbc0ad790 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.266+0000 7fedc40b3640 1 -- 172.21.15.150:0/2893446523 >> 172.21.15.150:0/2893446523 conn(0x7fedbc070b50 msgr2=0x7fedbc151630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.266+0000 7fedc40b3640 1 -- 172.21.15.150:0/2893446523 shutdown_connections 2024-09-17T05:02:30.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:30.266+0000 7fedc40b3640 1 -- 172.21.15.150:0/2893446523 wait complete. 2024-09-17T05:02:30.287 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:02:31.477 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:02:31.480 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:02:32.288 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:32.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.454+0000 7f927b86b640 1 -- 172.21.15.150:0/4210494481 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92740c37b0 msgr2=0x7f92741535e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:32.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.454+0000 7f927b86b640 1 --2- 172.21.15.150:0/4210494481 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92740c37b0 0x7f92741535e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9264009900 tx=0x7f926401c8e0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.454+0000 7f927b86b640 1 -- 172.21.15.150:0/4210494481 shutdown_connections 2024-09-17T05:02:32.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.454+0000 7f927b86b640 1 --2- 172.21.15.150:0/4210494481 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92740c37b0 0x7f92741535e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.454+0000 7f927b86b640 1 --2- 172.21.15.150:0/4210494481 >> v2:172.21.15.176:3300/0 conn(0x7f92740a8120 0x7f9274149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.454+0000 7f927b86b640 1 -- 172.21.15.150:0/4210494481 >> 172.21.15.150:0/4210494481 conn(0x7f92740aab60 msgr2=0x7f92740aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:32.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.454+0000 7f927b86b640 1 -- 172.21.15.150:0/4210494481 shutdown_connections 2024-09-17T05:02:32.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.454+0000 7f927b86b640 1 -- 172.21.15.150:0/4210494481 wait complete. 2024-09-17T05:02:32.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.454+0000 7f927b86b640 1 Processor -- start 2024-09-17T05:02:32.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.454+0000 7f927b86b640 1 -- start start 2024-09-17T05:02:32.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f927b86b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92740a8120 0x7f92741f0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:32.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f927b86b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92740c37b0 0x7f927415c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:32.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f927b86b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9274157810 0x7f9274148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:32.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f927b86b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f92740c8a00 con 0x7f92740c37b0 2024-09-17T05:02:32.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f927b86b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9274072390 con 0x7f9274157810 2024-09-17T05:02:32.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f927b86b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f92740c8b80 con 0x7f92740a8120 2024-09-17T05:02:32.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9278e06640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92740c37b0 0x7f927415c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9278e06640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92740c37b0 0x7f927415c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57254/0 (socket says 172.21.15.150:57254) 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9278e06640 1 -- 172.21.15.150:0/1396908654 learned_addr learned my addr 172.21.15.150:0/1396908654 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9279e08640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9274157810 0x7f9274148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9279607640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92740a8120 0x7f92741f0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9278e06640 1 -- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92740a8120 msgr2=0x7f92741f0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9278e06640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92740a8120 0x7f92741f0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9278e06640 1 -- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9274157810 msgr2=0x7f9274148d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9278e06640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9274157810 0x7f9274148d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9278e06640 1 -- 172.21.15.150:0/1396908654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92741f42e0 con 0x7f92740c37b0 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9278e06640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92740c37b0 0x7f927415c300 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f92640b0040 tx=0x7f9264005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9269ffb640 1 -- 172.21.15.150:0/1396908654 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92640b2040 con 0x7f92740c37b0 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9269ffb640 1 -- 172.21.15.150:0/1396908654 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9264004330 con 0x7f92740c37b0 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f927b86b640 1 -- 172.21.15.150:0/1396908654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92741f45c0 con 0x7f92740c37b0 2024-09-17T05:02:32.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9269ffb640 1 -- 172.21.15.150:0/1396908654 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264005280 con 0x7f92740c37b0 2024-09-17T05:02:32.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f927b86b640 1 -- 172.21.15.150:0/1396908654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9274152d40 con 0x7f92740c37b0 2024-09-17T05:02:32.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f927b86b640 1 -- 172.21.15.150:0/1396908654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9274149f40 con 0x7f92740c37b0 2024-09-17T05:02:32.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9269ffb640 1 -- 172.21.15.150:0/1396908654 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9264005420 con 0x7f92740c37b0 2024-09-17T05:02:32.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9269ffb640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9258073230 0x7f9258075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:32.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.458+0000 7f9269ffb640 1 -- 172.21.15.150:0/1396908654 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f92640bd070 con 0x7f92740c37b0 2024-09-17T05:02:32.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.462+0000 7f9279607640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9258073230 0x7f9258075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:32.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.462+0000 7f9279607640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9258073230 0x7f9258075740 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9270009ad0 tx=0x7f92700091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:32.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.466+0000 7f9269ffb640 1 -- 172.21.15.150:0/1396908654 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f92640164d0 con 0x7f92740c37b0 2024-09-17T05:02:32.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.706+0000 7f927b86b640 1 -- 172.21.15.150:0/1396908654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92740aa640 con 0x7f92740c37b0 2024-09-17T05:02:32.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.706+0000 7f9269ffb640 1 -- 172.21.15.150:0/1396908654 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+5705 (secure 0 0 0) 0x7f92640f8fa0 con 0x7f92740c37b0 2024-09-17T05:02:32.703 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:32.703 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":93,"btime":"2024-09-17T05:02:32:001251+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:31.403454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:02:32.703 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 93 2024-09-17T05:02:32.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.710+0000 7f927b86b640 1 -- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9258073230 msgr2=0x7f9258075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:32.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.710+0000 7f927b86b640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9258073230 0x7f9258075740 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9270009ad0 tx=0x7f92700091e0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.710+0000 7f927b86b640 1 -- 172.21.15.150:0/1396908654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92740c37b0 msgr2=0x7f927415c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:32.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.710+0000 7f927b86b640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92740c37b0 0x7f927415c300 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f92640b0040 tx=0x7f9264005e00 comp rx=0 tx=0).stop 2024-09-17T05:02:32.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.710+0000 7f927b86b640 1 -- 172.21.15.150:0/1396908654 shutdown_connections 2024-09-17T05:02:32.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.710+0000 7f927b86b640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9258073230 0x7f9258075740 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.710+0000 7f927b86b640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9274157810 0x7f9274148d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.710+0000 7f927b86b640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92740c37b0 0x7f927415c300 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.710+0000 7f927b86b640 1 --2- 172.21.15.150:0/1396908654 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f92740a8120 0x7f92741f0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.710+0000 7f927b86b640 1 -- 172.21.15.150:0/1396908654 >> 172.21.15.150:0/1396908654 conn(0x7f92740aab60 msgr2=0x7f927406d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:32.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.710+0000 7f927b86b640 1 -- 172.21.15.150:0/1396908654 shutdown_connections 2024-09-17T05:02:32.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.710+0000 7f927b86b640 1 -- 172.21.15.150:0/1396908654 wait complete. 2024-09-17T05:02:32.723 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-17T05:02:32.723 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:02:32.724 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:32.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.870+0000 7fe8b130f640 1 -- 172.21.15.150:0/268723998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8ac074980 msgr2=0x7fe8ac0aaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:32.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.870+0000 7fe8b130f640 1 --2- 172.21.15.150:0/268723998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8ac074980 0x7fe8ac0aaf80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac073740 tx=0x7fe8a401ccf0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.870+0000 7fe8b130f640 1 -- 172.21.15.150:0/268723998 shutdown_connections 2024-09-17T05:02:32.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.870+0000 7fe8b130f640 1 --2- 172.21.15.150:0/268723998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8ac074980 0x7fe8ac0aaf80 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.870+0000 7fe8b130f640 1 --2- 172.21.15.150:0/268723998 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8ac074570 0x7fe8ac0aaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.870+0000 7fe8b130f640 1 --2- 172.21.15.150:0/268723998 >> v2:172.21.15.176:3300/0 conn(0x7fe8ac06fca0 0x7fe8ac153790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.870+0000 7fe8b130f640 1 -- 172.21.15.150:0/268723998 >> 172.21.15.150:0/268723998 conn(0x7fe8ac0c5ae0 msgr2=0x7fe8ac073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:32.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.870+0000 7fe8b130f640 1 -- 172.21.15.150:0/268723998 shutdown_connections 2024-09-17T05:02:32.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.870+0000 7fe8b130f640 1 -- 172.21.15.150:0/268723998 wait complete. 2024-09-17T05:02:32.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8b130f640 1 Processor -- start 2024-09-17T05:02:32.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8b130f640 1 -- start start 2024-09-17T05:02:32.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8b130f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8ac06fca0 0x7fe8ac1f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:32.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8b130f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8ac074570 0x7fe8ac14e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:32.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8b130f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8ac074980 0x7fe8ac1483c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:32.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8b130f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ac0c4b50 con 0x7fe8ac074980 2024-09-17T05:02:32.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8b130f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ac0c49d0 con 0x7fe8ac06fca0 2024-09-17T05:02:32.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8b130f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe8ac14a270 con 0x7fe8ac074570 2024-09-17T05:02:32.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8ab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8ac074980 0x7fe8ac1483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:32.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8ab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8ac074980 0x7fe8ac1483c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57284/0 (socket says 172.21.15.150:57284) 2024-09-17T05:02:32.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8ab7fe640 1 -- 172.21.15.150:0/3397473676 learned_addr learned my addr 172.21.15.150:0/3397473676 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:32.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8aaffd640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8ac06fca0 0x7fe8ac1f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:32.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8a27fc640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8ac074570 0x7fe8ac14e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:32.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8ab7fe640 1 -- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8ac074570 msgr2=0x7fe8ac14e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:32.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8ab7fe640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8ac074570 0x7fe8ac14e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8ab7fe640 1 -- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8ac06fca0 msgr2=0x7fe8ac1f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:32.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8ab7fe640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8ac06fca0 0x7fe8ac1f76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:32.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8ab7fe640 1 -- 172.21.15.150:0/3397473676 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8ac1f80f0 con 0x7fe8ac074980 2024-09-17T05:02:32.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8ab7fe640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8ac074980 0x7fe8ac1483c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe8a401b3a0 tx=0x7fe8a4003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:32.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8a3fff640 1 -- 172.21.15.150:0/3397473676 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a40b2040 con 0x7fe8ac074980 2024-09-17T05:02:32.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8b130f640 1 -- 172.21.15.150:0/3397473676 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8ac14f830 con 0x7fe8ac074980 2024-09-17T05:02:32.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8b130f640 1 -- 172.21.15.150:0/3397473676 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8ac0c7880 con 0x7fe8ac074980 2024-09-17T05:02:32.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8a3fff640 1 -- 172.21.15.150:0/3397473676 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe8a4004200 con 0x7fe8ac074980 2024-09-17T05:02:32.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.874+0000 7fe8a3fff640 1 -- 172.21.15.150:0/3397473676 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a40045d0 con 0x7fe8ac074980 2024-09-17T05:02:32.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.878+0000 7fe8b130f640 1 -- 172.21.15.150:0/3397473676 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8ac153790 con 0x7fe8ac074980 2024-09-17T05:02:32.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.878+0000 7fe8a3fff640 1 -- 172.21.15.150:0/3397473676 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe8a40ad440 con 0x7fe8ac074980 2024-09-17T05:02:32.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.878+0000 7fe8a3fff640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe88c073200 0x7fe88c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:32.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.878+0000 7fe8a3fff640 1 -- 172.21.15.150:0/3397473676 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7fe8a412f360 con 0x7fe8ac074980 2024-09-17T05:02:32.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.878+0000 7fe8aaffd640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe88c073200 0x7fe88c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:32.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.878+0000 7fe8aaffd640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe88c073200 0x7fe88c075710 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe89c005fd0 tx=0x7fe89c00c460 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:32.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:32.882+0000 7fe8a3fff640 1 -- 172.21.15.150:0/3397473676 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe8a4016970 con 0x7fe8ac074980 2024-09-17T05:02:33.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 -- 172.21.15.150:0/3397473676 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8ac154560 con 0x7fe8ac074980 2024-09-17T05:02:33.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8a3fff640 1 -- 172.21.15.150:0/3397473676 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+5702 (secure 0 0 0) 0x7fe8a40f9b80 con 0x7fe8ac074980 2024-09-17T05:02:33.121 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:33.121 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":94,"btime":"2024-09-17T05:02:33:011566+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:32.017315+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:02:33.121 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 94 2024-09-17T05:02:33.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 -- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe88c073200 msgr2=0x7fe88c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:33.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe88c073200 0x7fe88c075710 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe89c005fd0 tx=0x7fe89c00c460 comp rx=0 tx=0).stop 2024-09-17T05:02:33.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 -- 172.21.15.150:0/3397473676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8ac074980 msgr2=0x7fe8ac1483c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:33.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8ac074980 0x7fe8ac1483c0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe8a401b3a0 tx=0x7fe8a4003f80 comp rx=0 tx=0).stop 2024-09-17T05:02:33.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 -- 172.21.15.150:0/3397473676 shutdown_connections 2024-09-17T05:02:33.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe88c073200 0x7fe88c075710 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:33.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8ac074980 0x7fe8ac1483c0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:33.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8ac074570 0x7fe8ac14e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:33.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 --2- 172.21.15.150:0/3397473676 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8ac06fca0 0x7fe8ac1f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:33.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 -- 172.21.15.150:0/3397473676 >> 172.21.15.150:0/3397473676 conn(0x7fe8ac0c5ae0 msgr2=0x7fe8ac0ad5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:33.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 -- 172.21.15.150:0/3397473676 shutdown_connections 2024-09-17T05:02:33.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:33.126+0000 7fe8b130f640 1 -- 172.21.15.150:0/3397473676 wait complete. 2024-09-17T05:02:33.142 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:02:33:011566+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 90, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6252": { "addr": "172.21.15.150:6835/2062728083", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 2062728083, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 2062728083, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6252, "incarnation": 87, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:00:49.115368+0000", "qdb_cluster": [ 6252 ], "qdb_leader": 6252, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6252 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6342": { "addr": "172.21.15.150:6833/233311351", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 233311351, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 233311351, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6342, "incarnation": 92, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:02:32.017315+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6342 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:02:35.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:35.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.318+0000 7fc5248dc640 1 -- 172.21.15.150:0/1652395502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc520074570 msgr2=0x7fc5200aaa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:35.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.318+0000 7fc5248dc640 1 --2- 172.21.15.150:0/1652395502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc520074570 0x7fc5200aaa40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc51400aa10 tx=0x7fc51401dba0 comp rx=0 tx=0).stop 2024-09-17T05:02:35.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.318+0000 7fc5248dc640 1 -- 172.21.15.150:0/1652395502 shutdown_connections 2024-09-17T05:02:35.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.318+0000 7fc5248dc640 1 --2- 172.21.15.150:0/1652395502 >> v2:172.21.15.176:3300/0 conn(0x7fc520074980 0x7fc5200aaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:35.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.318+0000 7fc5248dc640 1 --2- 172.21.15.150:0/1652395502 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc520074570 0x7fc5200aaa40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:35.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.318+0000 7fc5248dc640 1 -- 172.21.15.150:0/1652395502 >> 172.21.15.150:0/1652395502 conn(0x7fc5200c5ae0 msgr2=0x7fc520073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:35.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.318+0000 7fc5248dc640 1 -- 172.21.15.150:0/1652395502 shutdown_connections 2024-09-17T05:02:35.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.318+0000 7fc5248dc640 1 -- 172.21.15.150:0/1652395502 wait complete. 2024-09-17T05:02:35.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc5248dc640 1 Processor -- start 2024-09-17T05:02:35.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc5248dc640 1 -- start start 2024-09-17T05:02:35.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc5248dc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc52006fca0 0x7fc5201564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:35.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc5248dc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc520074570 0x7fc52014c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc5248dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc520074980 0x7fc520145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc5248dc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc5200c4530 con 0x7fc520074980 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc5248dc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc5200c43b0 con 0x7fc52006fca0 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc5248dc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc5200c46b0 con 0x7fc520074570 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc51ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc520074980 0x7fc520145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc51ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc520074980 0x7fc520145270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57294/0 (socket says 172.21.15.150:57294) 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc51ed9d640 1 -- 172.21.15.150:0/2370519499 learned_addr learned my addr 172.21.15.150:0/2370519499 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc51dd9b640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc520074570 0x7fc52014c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc51e59c640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc52006fca0 0x7fc5201564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc51ed9d640 1 -- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc520074570 msgr2=0x7fc52014c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc51ed9d640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc520074570 0x7fc52014c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc51ed9d640 1 -- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc52006fca0 msgr2=0x7fc5201564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:35.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc51ed9d640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc52006fca0 0x7fc5201564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:35.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc51ed9d640 1 -- 172.21.15.150:0/2370519499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc520152a60 con 0x7fc520074980 2024-09-17T05:02:35.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc51e59c640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc52006fca0 0x7fc5201564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:02:35.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc51ed9d640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc520074980 0x7fc520145270 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc51800ea80 tx=0x7fc51800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:35.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.322+0000 7fc50effd640 1 -- 172.21.15.150:0/2370519499 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc518004b30 con 0x7fc520074980 2024-09-17T05:02:35.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.326+0000 7fc5248dc640 1 -- 172.21.15.150:0/2370519499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc52014a6c0 con 0x7fc520074980 2024-09-17T05:02:35.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.326+0000 7fc5248dc640 1 -- 172.21.15.150:0/2370519499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc52014a960 con 0x7fc520074980 2024-09-17T05:02:35.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.326+0000 7fc50effd640 1 -- 172.21.15.150:0/2370519499 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc518004cf0 con 0x7fc520074980 2024-09-17T05:02:35.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.326+0000 7fc50effd640 1 -- 172.21.15.150:0/2370519499 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc518005280 con 0x7fc520074980 2024-09-17T05:02:35.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.326+0000 7fc5248dc640 1 -- 172.21.15.150:0/2370519499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5040052a0 con 0x7fc520074980 2024-09-17T05:02:35.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.326+0000 7fc50effd640 1 -- 172.21.15.150:0/2370519499 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc518005420 con 0x7fc520074980 2024-09-17T05:02:35.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.326+0000 7fc50effd640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc500073200 0x7fc500075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:35.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.326+0000 7fc50effd640 1 -- 172.21.15.150:0/2370519499 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7fc518095b10 con 0x7fc520074980 2024-09-17T05:02:35.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.326+0000 7fc51e59c640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc500073200 0x7fc500075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:35.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.330+0000 7fc51e59c640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc500073200 0x7fc500075710 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc510000f70 tx=0x7fc510009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:35.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.334+0000 7fc50effd640 1 -- 172.21.15.150:0/2370519499 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc518060330 con 0x7fc520074980 2024-09-17T05:02:35.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.602+0000 7fc5248dc640 1 -- 172.21.15.150:0/2370519499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc504005590 con 0x7fc520074980 2024-09-17T05:02:35.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.602+0000 7fc50effd640 1 -- 172.21.15.150:0/2370519499 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) ==== 94+0+5709 (secure 0 0 0) 0x7fc51805fc30 con 0x7fc520074980 2024-09-17T05:02:35.599 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:35.599 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":95,"btime":"2024-09-17T05:02:34:022752+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:02:35.600 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 95 2024-09-17T05:02:35.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.606+0000 7fc5248dc640 1 -- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc500073200 msgr2=0x7fc500075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:35.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.606+0000 7fc5248dc640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc500073200 0x7fc500075710 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc510000f70 tx=0x7fc510009040 comp rx=0 tx=0).stop 2024-09-17T05:02:35.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.606+0000 7fc5248dc640 1 -- 172.21.15.150:0/2370519499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc520074980 msgr2=0x7fc520145270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:35.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.606+0000 7fc5248dc640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc520074980 0x7fc520145270 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc51800ea80 tx=0x7fc51800ef40 comp rx=0 tx=0).stop 2024-09-17T05:02:35.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.606+0000 7fc5248dc640 1 -- 172.21.15.150:0/2370519499 shutdown_connections 2024-09-17T05:02:35.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.606+0000 7fc5248dc640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc500073200 0x7fc500075710 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:35.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.606+0000 7fc5248dc640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc520074980 0x7fc520145270 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:35.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.606+0000 7fc5248dc640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc520074570 0x7fc52014c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:35.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.606+0000 7fc5248dc640 1 --2- 172.21.15.150:0/2370519499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc52006fca0 0x7fc5201564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:35.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.606+0000 7fc5248dc640 1 -- 172.21.15.150:0/2370519499 >> 172.21.15.150:0/2370519499 conn(0x7fc5200c5ae0 msgr2=0x7fc5200c78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:35.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.606+0000 7fc5248dc640 1 -- 172.21.15.150:0/2370519499 shutdown_connections 2024-09-17T05:02:35.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:35.606+0000 7fc5248dc640 1 -- 172.21.15.150:0/2370519499 wait complete. 2024-09-17T05:02:35.623 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5375 2024-09-17T05:02:35.623 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before reviving mds.d 2024-09-17T05:02:37.157 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T05:02:37.162+0000 7ff82f642640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:02:44.624 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-17T05:02:44.624 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T05:02:44.625 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T05:02:44.628 INFO:tasks.ceph.mds.d:Started 2024-09-17T05:02:44.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:44.738 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-09-17T05:02:44.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.802+0000 7f18f47be640 1 -- 172.21.15.150:0/841483976 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ec074280 msgr2=0x7f18ec15f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:44.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.802+0000 7f18f47be640 1 --2- 172.21.15.150:0/841483976 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ec074280 0x7f18ec15f160 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009bc0 tx=0x7f18e401cba0 comp rx=0 tx=0).stop 2024-09-17T05:02:44.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.802+0000 7f18f47be640 1 -- 172.21.15.150:0/841483976 shutdown_connections 2024-09-17T05:02:44.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.802+0000 7f18f47be640 1 --2- 172.21.15.150:0/841483976 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ec074280 0x7f18ec15f160 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:44.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.802+0000 7f18f47be640 1 --2- 172.21.15.150:0/841483976 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18ec0a9980 0x7f18ec073d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:44.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.802+0000 7f18f47be640 1 -- 172.21.15.150:0/841483976 >> 172.21.15.150:0/841483976 conn(0x7f18ec0c5220 msgr2=0x7f18ec0c5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:44.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.802+0000 7f18f47be640 1 -- 172.21.15.150:0/841483976 shutdown_connections 2024-09-17T05:02:44.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.802+0000 7f18f47be640 1 -- 172.21.15.150:0/841483976 wait complete. 2024-09-17T05:02:44.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.802+0000 7f18f47be640 1 Processor -- start 2024-09-17T05:02:44.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.802+0000 7f18f47be640 1 -- start start 2024-09-17T05:02:44.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f47be640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ec074280 0x7f18ec0c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:44.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f47be640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18ec0a9980 0x7f18ec141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:44.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f47be640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ec0c3930 0x7f18ec0c6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:44.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f47be640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f18ec0a9f10 con 0x7f18ec0c3930 2024-09-17T05:02:44.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f47be640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f18ec0a9d90 con 0x7f18ec074280 2024-09-17T05:02:44.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f47be640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f18ec075020 con 0x7f18ec0a9980 2024-09-17T05:02:44.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f2d5b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ec0c3930 0x7f18ec0c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:44.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f2d5b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ec0c3930 0x7f18ec0c6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44704/0 (socket says 172.21.15.150:44704) 2024-09-17T05:02:44.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f2d5b640 1 -- 172.21.15.150:0/539843867 learned_addr learned my addr 172.21.15.150:0/539843867 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:44.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f255a640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ec074280 0x7f18ec0c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:44.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f1d59640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18ec0a9980 0x7f18ec141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:44.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f2d5b640 1 -- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18ec0a9980 msgr2=0x7f18ec141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:44.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f2d5b640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18ec0a9980 0x7f18ec141690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:44.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f2d5b640 1 -- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ec074280 msgr2=0x7f18ec0c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:44.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f2d5b640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ec074280 0x7f18ec0c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:44.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f2d5b640 1 -- 172.21.15.150:0/539843867 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18ec1483c0 con 0x7f18ec0c3930 2024-09-17T05:02:44.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f2d5b640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ec0c3930 0x7f18ec0c6310 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f18e401c610 tx=0x7f18e4003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:44.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18e2ffd640 1 -- 172.21.15.150:0/539843867 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e40b2040 con 0x7f18ec0c3930 2024-09-17T05:02:44.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18e2ffd640 1 -- 172.21.15.150:0/539843867 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18e40bb070 con 0x7f18ec0c3930 2024-09-17T05:02:44.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f47be640 1 -- 172.21.15.150:0/539843867 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18ec146d00 con 0x7f18ec0c3930 2024-09-17T05:02:44.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18e2ffd640 1 -- 172.21.15.150:0/539843867 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e4004370 con 0x7f18ec0c3930 2024-09-17T05:02:44.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f47be640 1 -- 172.21.15.150:0/539843867 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18ec1480e0 con 0x7f18ec0c3930 2024-09-17T05:02:44.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.806+0000 7f18f47be640 1 -- 172.21.15.150:0/539843867 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18ec073d40 con 0x7f18ec0c3930 2024-09-17T05:02:44.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.810+0000 7f18e2ffd640 1 -- 172.21.15.150:0/539843867 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f18e4004510 con 0x7f18ec0c3930 2024-09-17T05:02:44.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.810+0000 7f18e2ffd640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f18d0073200 0x7f18d0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:44.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.810+0000 7f18e2ffd640 1 -- 172.21.15.150:0/539843867 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f18e412f2a0 con 0x7f18ec0c3930 2024-09-17T05:02:44.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.810+0000 7f18f255a640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f18d0073200 0x7f18d0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:44.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.810+0000 7f18f255a640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f18d0073200 0x7f18d0075710 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f18dc001870 tx=0x7f18dc001380 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:44.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:44.814+0000 7f18e2ffd640 1 -- 172.21.15.150:0/539843867 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f18ec1483c0 con 0x7f18ec0c3930 2024-09-17T05:02:45.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.110+0000 7f18f47be640 1 -- 172.21.15.150:0/539843867 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18ec142900 con 0x7f18ec0c3930 2024-09-17T05:02:45.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18e2ffd640 1 -- 172.21.15.150:0/539843867 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+6534 (secure 0 0 0) 0x7f18e40a5e20 con 0x7f18ec0c3930 2024-09-17T05:02:45.108 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:45.108 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":96,"btime":"2024-09-17T05:02:45:058000+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:02:45.109 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 96 2024-09-17T05:02:45.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18f47be640 1 -- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f18d0073200 msgr2=0x7f18d0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:45.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18f47be640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f18d0073200 0x7f18d0075710 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f18dc001870 tx=0x7f18dc001380 comp rx=0 tx=0).stop 2024-09-17T05:02:45.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18f47be640 1 -- 172.21.15.150:0/539843867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ec0c3930 msgr2=0x7f18ec0c6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:45.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18f47be640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ec0c3930 0x7f18ec0c6310 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f18e401c610 tx=0x7f18e4003f40 comp rx=0 tx=0).stop 2024-09-17T05:02:45.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18f47be640 1 -- 172.21.15.150:0/539843867 shutdown_connections 2024-09-17T05:02:45.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18f47be640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f18d0073200 0x7f18d0075710 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18f47be640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ec0c3930 0x7f18ec0c6310 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18f47be640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18ec0a9980 0x7f18ec141690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18f47be640 1 --2- 172.21.15.150:0/539843867 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ec074280 0x7f18ec0c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18f47be640 1 -- 172.21.15.150:0/539843867 >> 172.21.15.150:0/539843867 conn(0x7f18ec0c5220 msgr2=0x7f18ec148d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:45.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18f47be640 1 -- 172.21.15.150:0/539843867 shutdown_connections 2024-09-17T05:02:45.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.114+0000 7f18f47be640 1 -- 172.21.15.150:0/539843867 wait complete. 2024-09-17T05:02:45.131 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-17T05:02:45.131 INFO:tasks.mds_thrash.fs.[b]:waiting for 67 secs before thrashing 2024-09-17T05:02:45.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:45.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.430+0000 7fbf13c27640 1 -- 172.21.15.150:0/185046574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0a85b0 msgr2=0x7fbf0c0c7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:45.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.430+0000 7fbf13c27640 1 --2- 172.21.15.150:0/185046574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0a85b0 0x7fbf0c0c7270 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009900 tx=0x7fbefc01c850 comp rx=0 tx=0).stop 2024-09-17T05:02:45.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.430+0000 7fbf13c27640 1 -- 172.21.15.150:0/185046574 shutdown_connections 2024-09-17T05:02:45.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.430+0000 7fbf13c27640 1 --2- 172.21.15.150:0/185046574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf0c0c77b0 0x7fbf0c151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.430+0000 7fbf13c27640 1 --2- 172.21.15.150:0/185046574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0a85b0 0x7fbf0c0c7270 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.430+0000 7fbf13c27640 1 -- 172.21.15.150:0/185046574 >> 172.21.15.150:0/185046574 conn(0x7fbf0c0aab60 msgr2=0x7fbf0c0aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:45.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.430+0000 7fbf13c27640 1 -- 172.21.15.150:0/185046574 shutdown_connections 2024-09-17T05:02:45.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.430+0000 7fbf13c27640 1 -- 172.21.15.150:0/185046574 wait complete. 2024-09-17T05:02:45.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.430+0000 7fbf13c27640 1 Processor -- start 2024-09-17T05:02:45.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.430+0000 7fbf13c27640 1 -- start start 2024-09-17T05:02:45.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf13c27640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf0c0a85b0 0x7fbf0c1edc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:45.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf13c27640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0c6cb0 0x7fbf0c071a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:45.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf13c27640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf0c0c77b0 0x7fbf0c144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:45.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf13c27640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c0c3460 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf13c27640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbf0c0c32e0 con 0x7fbf0c0c77b0 2024-09-17T05:02:45.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf13c27640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbf0c161890 con 0x7fbf0c0a85b0 2024-09-17T05:02:45.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf111c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0c6cb0 0x7fbf0c071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:45.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf111c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0c6cb0 0x7fbf0c071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44726/0 (socket says 172.21.15.150:44726) 2024-09-17T05:02:45.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf111c2640 1 -- 172.21.15.150:0/2937562563 learned_addr learned my addr 172.21.15.150:0/2937562563 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:45.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf121c4640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf0c0c77b0 0x7fbf0c144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:45.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf119c3640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf0c0a85b0 0x7fbf0c1edc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:45.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf111c2640 1 -- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf0c0a85b0 msgr2=0x7fbf0c1edc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:45.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf111c2640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf0c0a85b0 0x7fbf0c1edc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf111c2640 1 -- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf0c0c77b0 msgr2=0x7fbf0c144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:45.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf111c2640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf0c0c77b0 0x7fbf0c144fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf111c2640 1 -- 172.21.15.150:0/2937562563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf0c148b50 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf121c4640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf0c0c77b0 0x7fbf0c144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:02:45.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf111c2640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0c6cb0 0x7fbf0c071a90 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0b0040 tx=0x7fbefc005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:45.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf027fc640 1 -- 172.21.15.150:0/2937562563 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc0ba020 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf027fc640 1 -- 172.21.15.150:0/2937562563 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbefc004330 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf13c27640 1 -- 172.21.15.150:0/2937562563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf0c1f1d50 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf027fc640 1 -- 172.21.15.150:0/2937562563 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbefc0b2040 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf13c27640 1 -- 172.21.15.150:0/2937562563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf0c14f560 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.434+0000 7fbf13c27640 1 -- 172.21.15.150:0/2937562563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf0c1f1b80 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.438+0000 7fbf027fc640 1 -- 172.21.15.150:0/2937562563 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbefc005280 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.438+0000 7fbf027fc640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbef0073200 0x7fbef0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:45.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.438+0000 7fbf027fc640 1 -- 172.21.15.150:0/2937562563 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7fbefc12e9f0 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.438+0000 7fbf119c3640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbef0073200 0x7fbef0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:45.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.438+0000 7fbf119c3640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbef0073200 0x7fbef0075710 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbf08007fd0 tx=0x7fbf08009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.442+0000 7fbf027fc640 1 -- 172.21.15.150:0/2937562563 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbefc0c4020 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.686+0000 7fbf13c27640 1 -- 172.21.15.150:0/2937562563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf0c0c7270 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.686+0000 7fbf027fc640 1 -- 172.21.15.150:0/2937562563 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+6534 (secure 0 0 0) 0x7fbefc0164d0 con 0x7fbf0c0c6cb0 2024-09-17T05:02:45.684 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:45.684 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":96,"btime":"2024-09-17T05:02:45:058000+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:02:45.685 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 96 2024-09-17T05:02:45.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.690+0000 7fbf13c27640 1 -- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbef0073200 msgr2=0x7fbef0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:45.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.690+0000 7fbf13c27640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbef0073200 0x7fbef0075710 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbf08007fd0 tx=0x7fbf08009040 comp rx=0 tx=0).stop 2024-09-17T05:02:45.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.690+0000 7fbf13c27640 1 -- 172.21.15.150:0/2937562563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0c6cb0 msgr2=0x7fbf0c071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:45.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.690+0000 7fbf13c27640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0c6cb0 0x7fbf0c071a90 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fbefc0b0040 tx=0x7fbefc005de0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.690+0000 7fbf13c27640 1 -- 172.21.15.150:0/2937562563 shutdown_connections 2024-09-17T05:02:45.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.690+0000 7fbf13c27640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbef0073200 0x7fbef0075710 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.690+0000 7fbf13c27640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf0c0c77b0 0x7fbf0c144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.690+0000 7fbf13c27640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf0c0c6cb0 0x7fbf0c071a90 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.690+0000 7fbf13c27640 1 --2- 172.21.15.150:0/2937562563 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf0c0a85b0 0x7fbf0c1edc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.690+0000 7fbf13c27640 1 -- 172.21.15.150:0/2937562563 >> 172.21.15.150:0/2937562563 conn(0x7fbf0c0aab60 msgr2=0x7fbf0c14e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:45.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.690+0000 7fbf13c27640 1 -- 172.21.15.150:0/2937562563 shutdown_connections 2024-09-17T05:02:45.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.690+0000 7fbf13c27640 1 -- 172.21.15.150:0/2937562563 wait complete. 2024-09-17T05:02:45.706 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-17T05:02:45.707 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T05:02:45.707 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:02:45.719 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T05:02:45.719 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:45.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.930+0000 7f7566e82640 1 -- 172.21.15.150:0/2350817179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f756014a180 msgr2=0x7f75600c7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:45.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.930+0000 7f7566e82640 1 --2- 172.21.15.150:0/2350817179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f756014a180 0x7f75600c7190 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7558009900 tx=0x7f755801c8e0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.930+0000 7f7566e82640 1 -- 172.21.15.150:0/2350817179 shutdown_connections 2024-09-17T05:02:45.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.930+0000 7f7566e82640 1 --2- 172.21.15.150:0/2350817179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f756014a180 0x7f75600c7190 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7566e82640 1 --2- 172.21.15.150:0/2350817179 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f75600c7eb0 0x7f75601540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7566e82640 1 -- 172.21.15.150:0/2350817179 >> 172.21.15.150:0/2350817179 conn(0x7f75600aab60 msgr2=0x7f75600aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:45.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7566e82640 1 -- 172.21.15.150:0/2350817179 shutdown_connections 2024-09-17T05:02:45.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7566e82640 1 -- 172.21.15.150:0/2350817179 wait complete. 2024-09-17T05:02:45.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7566e82640 1 Processor -- start 2024-09-17T05:02:45.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7566e82640 1 -- start start 2024-09-17T05:02:45.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7566e82640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75600c7eb0 0x7f75600ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:45.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7566e82640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7560149d70 0x7f75600be550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:45.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7566e82640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f756014a180 0x7f75601f10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:45.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7566e82640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f75600c7c60 con 0x7f75600c7eb0 2024-09-17T05:02:45.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7566e82640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f75600c7ae0 con 0x7f756014a180 2024-09-17T05:02:45.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7566e82640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7560156540 con 0x7f7560149d70 2024-09-17T05:02:45.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7564c1e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75600c7eb0 0x7f75600ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:45.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7564c1e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75600c7eb0 0x7f75600ac1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44742/0 (socket says 172.21.15.150:44742) 2024-09-17T05:02:45.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7564c1e640 1 -- 172.21.15.150:0/110304288 learned_addr learned my addr 172.21.15.150:0/110304288 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:45.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f756541f640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f756014a180 0x7f75601f10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:45.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f755ffff640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7560149d70 0x7f75600be550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:45.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7564c1e640 1 -- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7560149d70 msgr2=0x7f75600be550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:45.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7564c1e640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7560149d70 0x7f75600be550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7564c1e640 1 -- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f756014a180 msgr2=0x7f75601f10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:45.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7564c1e640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f756014a180 0x7f75601f10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:45.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7564c1e640 1 -- 172.21.15.150:0/110304288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f756015dd80 con 0x7f75600c7eb0 2024-09-17T05:02:45.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f756541f640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f756014a180 0x7f75601f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:02:45.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f7564c1e640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75600c7eb0 0x7f75600ac1b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f755000ebf0 tx=0x7f755000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:45.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.934+0000 7f755d7fa640 1 -- 172.21.15.150:0/110304288 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7550010630 con 0x7f75600c7eb0 2024-09-17T05:02:45.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.938+0000 7f755d7fa640 1 -- 172.21.15.150:0/110304288 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75500108c0 con 0x7f75600c7eb0 2024-09-17T05:02:45.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.938+0000 7f755d7fa640 1 -- 172.21.15.150:0/110304288 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7550010cf0 con 0x7f75600c7eb0 2024-09-17T05:02:45.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.938+0000 7f7566e82640 1 -- 172.21.15.150:0/110304288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7560148980 con 0x7f75600c7eb0 2024-09-17T05:02:45.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.938+0000 7f7566e82640 1 -- 172.21.15.150:0/110304288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75601f1db0 con 0x7f75600c7eb0 2024-09-17T05:02:45.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.938+0000 7f7566e82640 1 -- 172.21.15.150:0/110304288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75601540b0 con 0x7f75600c7eb0 2024-09-17T05:02:45.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.938+0000 7f755d7fa640 1 -- 172.21.15.150:0/110304288 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7550018420 con 0x7f75600c7eb0 2024-09-17T05:02:45.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.938+0000 7f755d7fa640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7540073230 0x7f7540075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:45.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.938+0000 7f755d7fa640 1 -- 172.21.15.150:0/110304288 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f7550095fb0 con 0x7f75600c7eb0 2024-09-17T05:02:45.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.938+0000 7f755ffff640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7540073230 0x7f7540075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:45.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.938+0000 7f755ffff640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7540073230 0x7f7540075740 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f75601506a0 tx=0x7f7548009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:45.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:45.942+0000 7f755d7fa640 1 -- 172.21.15.150:0/110304288 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f75500601d0 con 0x7f75600c7eb0 2024-09-17T05:02:46.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 -- 172.21.15.150:0/110304288 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f756006cff0 con 0x7f75600c7eb0 2024-09-17T05:02:46.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f755d7fa640 1 -- 172.21.15.150:0/110304288 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+6534 (secure 0 0 0) 0x7f7550064c00 con 0x7f75600c7eb0 2024-09-17T05:02:46.149 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:46.149 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":96,"btime":"2024-09-17T05:02:45:058000+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:02:46.149 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 96 2024-09-17T05:02:46.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 -- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7540073230 msgr2=0x7f7540075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:46.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7540073230 0x7f7540075740 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f75601506a0 tx=0x7f7548009040 comp rx=0 tx=0).stop 2024-09-17T05:02:46.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 -- 172.21.15.150:0/110304288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75600c7eb0 msgr2=0x7f75600ac1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:46.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75600c7eb0 0x7f75600ac1b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f755000ebf0 tx=0x7f755000c6a0 comp rx=0 tx=0).stop 2024-09-17T05:02:46.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 -- 172.21.15.150:0/110304288 shutdown_connections 2024-09-17T05:02:46.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7540073230 0x7f7540075740 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:46.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f756014a180 0x7f75601f10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:46.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7560149d70 0x7f75600be550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:46.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 --2- 172.21.15.150:0/110304288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75600c7eb0 0x7f75600ac1b0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:46.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 -- 172.21.15.150:0/110304288 >> 172.21.15.150:0/110304288 conn(0x7f75600aab60 msgr2=0x7f7560152d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:46.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 -- 172.21.15.150:0/110304288 shutdown_connections 2024-09-17T05:02:46.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:46.154+0000 7f7566e82640 1 -- 172.21.15.150:0/110304288 wait complete. 2024-09-17T05:02:46.168 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:02:48.170 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:48.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.342+0000 7f9d1fb18640 1 -- 172.21.15.150:0/1601722773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d00004810 msgr2=0x7f9d00004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:48.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.342+0000 7f9d1fb18640 1 --2- 172.21.15.150:0/1601722773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d00004810 0x7f9d00004c20 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0016f0 tx=0x7f9d0c01c790 comp rx=0 tx=0).stop 2024-09-17T05:02:48.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.342+0000 7f9d1fb18640 1 -- 172.21.15.150:0/1601722773 shutdown_connections 2024-09-17T05:02:48.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.342+0000 7f9d1fb18640 1 --2- 172.21.15.150:0/1601722773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d00004810 0x7f9d00004c20 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:48.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.342+0000 7f9d1fb18640 1 -- 172.21.15.150:0/1601722773 >> 172.21.15.150:0/1601722773 conn(0x7f9d180c4a00 msgr2=0x7f9d180c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:48.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.342+0000 7f9d1fb18640 1 -- 172.21.15.150:0/1601722773 shutdown_connections 2024-09-17T05:02:48.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.342+0000 7f9d1fb18640 1 -- 172.21.15.150:0/1601722773 wait complete. 2024-09-17T05:02:48.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1fb18640 1 Processor -- start 2024-09-17T05:02:48.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1fb18640 1 -- start start 2024-09-17T05:02:48.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1fb18640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d00004810 0x7f9d180abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:48.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1fb18640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d180aae50 0x7f9d180ab260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:48.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1fb18640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d18147110 0x7f9d18147520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:48.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1fb18640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9d18074860 con 0x7f9d00004810 2024-09-17T05:02:48.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1fb18640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9d180adef0 con 0x7f9d180aae50 2024-09-17T05:02:48.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1fb18640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9d180749e0 con 0x7f9d18147110 2024-09-17T05:02:48.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1d8b4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d00004810 0x7f9d180abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1d8b4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d00004810 0x7f9d180abcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44748/0 (socket says 172.21.15.150:44748) 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1d8b4640 1 -- 172.21.15.150:0/963310543 learned_addr learned my addr 172.21.15.150:0/963310543 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1d0b3640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d180aae50 0x7f9d180ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1e0b5640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d18147110 0x7f9d18147520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1d8b4640 1 -- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d18147110 msgr2=0x7f9d18147520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1d8b4640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d18147110 0x7f9d18147520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1d8b4640 1 -- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d180aae50 msgr2=0x7f9d180ab260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1d8b4640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d180aae50 0x7f9d180ab260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1d8b4640 1 -- 172.21.15.150:0/963310543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d180ac210 con 0x7f9d00004810 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1d8b4640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d00004810 0x7f9d180abcd0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9d0800d980 tx=0x7f9d0800de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d167fc640 1 -- 172.21.15.150:0/963310543 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d08016070 con 0x7f9d00004810 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d167fc640 1 -- 172.21.15.150:0/963310543 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d08009d20 con 0x7f9d00004810 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d167fc640 1 -- 172.21.15.150:0/963310543 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d080106c0 con 0x7f9d00004810 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1fb18640 1 -- 172.21.15.150:0/963310543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d181440b0 con 0x7f9d00004810 2024-09-17T05:02:48.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.346+0000 7f9d1fb18640 1 -- 172.21.15.150:0/963310543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d181e2ef0 con 0x7f9d00004810 2024-09-17T05:02:48.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.350+0000 7f9d1fb18640 1 -- 172.21.15.150:0/963310543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d18074f40 con 0x7f9d00004810 2024-09-17T05:02:48.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.350+0000 7f9d167fc640 1 -- 172.21.15.150:0/963310543 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9d08019070 con 0x7f9d00004810 2024-09-17T05:02:48.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.350+0000 7f9d167fc640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9d04073200 0x7f9d04075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.350+0000 7f9d167fc640 1 -- 172.21.15.150:0/963310543 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f9d08095280 con 0x7f9d00004810 2024-09-17T05:02:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.350+0000 7f9d1d0b3640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9d04073200 0x7f9d04075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:48.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.350+0000 7f9d1d0b3640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9d04073200 0x7f9d04075710 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0016c0 tx=0x7f9d0c019580 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:48.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.354+0000 7f9d167fc640 1 -- 172.21.15.150:0/963310543 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9d0805f530 con 0x7f9d00004810 2024-09-17T05:02:48.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.558+0000 7f9d1fb18640 1 -- 172.21.15.150:0/963310543 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d180c60f0 con 0x7f9d00004810 2024-09-17T05:02:48.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d167fc640 1 -- 172.21.15.150:0/963310543 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+6534 (secure 0 0 0) 0x7f9d08063f60 con 0x7f9d00004810 2024-09-17T05:02:48.556 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:48.557 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":96,"btime":"2024-09-17T05:02:45:058000+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:02:48.557 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 96 2024-09-17T05:02:48.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d1fb18640 1 -- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9d04073200 msgr2=0x7f9d04075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:48.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d1fb18640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9d04073200 0x7f9d04075710 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0016c0 tx=0x7f9d0c019580 comp rx=0 tx=0).stop 2024-09-17T05:02:48.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d1fb18640 1 -- 172.21.15.150:0/963310543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d00004810 msgr2=0x7f9d180abcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:48.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d1fb18640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d00004810 0x7f9d180abcd0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f9d0800d980 tx=0x7f9d0800de40 comp rx=0 tx=0).stop 2024-09-17T05:02:48.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d1fb18640 1 -- 172.21.15.150:0/963310543 shutdown_connections 2024-09-17T05:02:48.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d1fb18640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9d04073200 0x7f9d04075710 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:48.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d1fb18640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9d180aae50 0x7f9d180ab260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:48.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d1fb18640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9d18147110 0x7f9d18147520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:48.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d1fb18640 1 --2- 172.21.15.150:0/963310543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9d00004810 0x7f9d180abcd0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:48.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d1fb18640 1 -- 172.21.15.150:0/963310543 >> 172.21.15.150:0/963310543 conn(0x7f9d180c4a00 msgr2=0x7f9d181dfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:48.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d1fb18640 1 -- 172.21.15.150:0/963310543 shutdown_connections 2024-09-17T05:02:48.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:48.562+0000 7f9d1fb18640 1 -- 172.21.15.150:0/963310543 wait complete. 2024-09-17T05:02:48.577 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:02:50.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:50.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.742+0000 7f703bd11640 1 -- 172.21.15.150:0/1567309236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70340c3db0 msgr2=0x7f70340ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:50.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.742+0000 7f703bd11640 1 --2- 172.21.15.150:0/1567309236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70340c3db0 0x7f70340ad2a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f703406f7a0 tx=0x7f702401c7e0 comp rx=0 tx=0).stop 2024-09-17T05:02:50.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.742+0000 7f703bd11640 1 -- 172.21.15.150:0/1567309236 shutdown_connections 2024-09-17T05:02:50.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.742+0000 7f703bd11640 1 --2- 172.21.15.150:0/1567309236 >> v2:172.21.15.176:3300/0 conn(0x7f70340c41c0 0x7f70340ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:50.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.742+0000 7f703bd11640 1 --2- 172.21.15.150:0/1567309236 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70340c3db0 0x7f70340ad2a0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:50.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.742+0000 7f703bd11640 1 -- 172.21.15.150:0/1567309236 >> 172.21.15.150:0/1567309236 conn(0x7f70340c4a00 msgr2=0x7f70340c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:50.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.742+0000 7f703bd11640 1 -- 172.21.15.150:0/1567309236 shutdown_connections 2024-09-17T05:02:50.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.742+0000 7f703bd11640 1 -- 172.21.15.150:0/1567309236 wait complete. 2024-09-17T05:02:50.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f703bd11640 1 Processor -- start 2024-09-17T05:02:50.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f703bd11640 1 -- start start 2024-09-17T05:02:50.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f703bd11640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703406fab0 0x7f70340c1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:50.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f703bd11640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70340c3db0 0x7f7034073b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:50.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f703bd11640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70340c41c0 0x7f70341572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:50.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f703bd11640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7034070750 con 0x7f703406fab0 2024-09-17T05:02:50.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f703bd11640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f70340705d0 con 0x7f70340c41c0 2024-09-17T05:02:50.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f703bd11640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f70340a75f0 con 0x7f70340c3db0 2024-09-17T05:02:50.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f7039aad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703406fab0 0x7f70340c1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:50.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f7039aad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703406fab0 0x7f70340c1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49028/0 (socket says 172.21.15.150:49028) 2024-09-17T05:02:50.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f7039aad640 1 -- 172.21.15.150:0/4108032160 learned_addr learned my addr 172.21.15.150:0/4108032160 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f703a2ae640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70340c41c0 0x7f70341572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f70392ac640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70340c3db0 0x7f7034073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f7039aad640 1 -- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70340c3db0 msgr2=0x7f7034073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f7039aad640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70340c3db0 0x7f7034073b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f7039aad640 1 -- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70340c41c0 msgr2=0x7f70341572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f7039aad640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70340c41c0 0x7f70341572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f7039aad640 1 -- 172.21.15.150:0/4108032160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703415a080 con 0x7f703406fab0 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f703a2ae640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70340c41c0 0x7f70341572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f7039aad640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703406fab0 0x7f70340c1420 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f7024000c10 tx=0x7f7024004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f702a7fc640 1 -- 172.21.15.150:0/4108032160 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70240bc070 con 0x7f703406fab0 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f702a7fc640 1 -- 172.21.15.150:0/4108032160 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7024003600 con 0x7f703406fab0 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f702a7fc640 1 -- 172.21.15.150:0/4108032160 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70240b2040 con 0x7f703406fab0 2024-09-17T05:02:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f703bd11640 1 -- 172.21.15.150:0/4108032160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70341ef920 con 0x7f703406fab0 2024-09-17T05:02:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.746+0000 7f703bd11640 1 -- 172.21.15.150:0/4108032160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7034159da0 con 0x7f703406fab0 2024-09-17T05:02:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.750+0000 7f703bd11640 1 -- 172.21.15.150:0/4108032160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f703406fec0 con 0x7f703406fab0 2024-09-17T05:02:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.750+0000 7f702a7fc640 1 -- 172.21.15.150:0/4108032160 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f70240bf070 con 0x7f703406fab0 2024-09-17T05:02:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.750+0000 7f702a7fc640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7018073200 0x7f7018075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.750+0000 7f702a7fc640 1 -- 172.21.15.150:0/4108032160 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f702412e500 con 0x7f703406fab0 2024-09-17T05:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.750+0000 7f70392ac640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7018073200 0x7f7018075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.750+0000 7f70392ac640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7018073200 0x7f7018075710 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f70300058d0 tx=0x7f7030005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:50.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.754+0000 7f702a7fc640 1 -- 172.21.15.150:0/4108032160 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f703415a080 con 0x7f703406fab0 2024-09-17T05:02:50.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.966+0000 7f703bd11640 1 -- 172.21.15.150:0/4108032160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70340c1960 con 0x7f703406fab0 2024-09-17T05:02:50.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.966+0000 7f702a7fc640 1 -- 172.21.15.150:0/4108032160 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+6534 (secure 0 0 0) 0x7f70240163d0 con 0x7f703406fab0 2024-09-17T05:02:50.961 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:50.961 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":96,"btime":"2024-09-17T05:02:45:058000+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:02:50.962 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 96 2024-09-17T05:02:50.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.970+0000 7f703bd11640 1 -- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7018073200 msgr2=0x7f7018075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:50.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.970+0000 7f703bd11640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7018073200 0x7f7018075710 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f70300058d0 tx=0x7f7030005860 comp rx=0 tx=0).stop 2024-09-17T05:02:50.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.970+0000 7f703bd11640 1 -- 172.21.15.150:0/4108032160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703406fab0 msgr2=0x7f70340c1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:50.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.970+0000 7f703bd11640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703406fab0 0x7f70340c1420 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f7024000c10 tx=0x7f7024004040 comp rx=0 tx=0).stop 2024-09-17T05:02:50.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.970+0000 7f703bd11640 1 -- 172.21.15.150:0/4108032160 shutdown_connections 2024-09-17T05:02:50.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.970+0000 7f703bd11640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7018073200 0x7f7018075710 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:50.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.970+0000 7f703bd11640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70340c41c0 0x7f70341572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:50.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.970+0000 7f703bd11640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70340c3db0 0x7f7034073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:50.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.970+0000 7f703bd11640 1 --2- 172.21.15.150:0/4108032160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f703406fab0 0x7f70340c1420 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:50.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.970+0000 7f703bd11640 1 -- 172.21.15.150:0/4108032160 >> 172.21.15.150:0/4108032160 conn(0x7f70340c4a00 msgr2=0x7f70340a92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:50.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.970+0000 7f703bd11640 1 -- 172.21.15.150:0/4108032160 shutdown_connections 2024-09-17T05:02:50.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:50.970+0000 7f703bd11640 1 -- 172.21.15.150:0/4108032160 wait complete. 2024-09-17T05:02:50.982 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:02:52.984 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:53.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.157+0000 7ff0315bf640 1 -- 172.21.15.150:0/612927412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff02c0c3660 msgr2=0x7ff02c15dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:53.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.157+0000 7ff0315bf640 1 --2- 172.21.15.150:0/612927412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff02c0c3660 0x7ff02c15dfe0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff024009bc0 tx=0x7ff02401f3b0 comp rx=0 tx=0).stop 2024-09-17T05:02:53.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.157+0000 7ff0315bf640 1 -- 172.21.15.150:0/612927412 shutdown_connections 2024-09-17T05:02:53.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.157+0000 7ff0315bf640 1 --2- 172.21.15.150:0/612927412 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff02c0a98a0 0x7ff02c0c3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:53.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.157+0000 7ff0315bf640 1 --2- 172.21.15.150:0/612927412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff02c0c3660 0x7ff02c15dfe0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:53.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.157+0000 7ff0315bf640 1 -- 172.21.15.150:0/612927412 >> 172.21.15.150:0/612927412 conn(0x7ff02c070c20 msgr2=0x7ff02c071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:53.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.157+0000 7ff0315bf640 1 -- 172.21.15.150:0/612927412 shutdown_connections 2024-09-17T05:02:53.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.157+0000 7ff0315bf640 1 -- 172.21.15.150:0/612927412 wait complete. 2024-09-17T05:02:53.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0315bf640 1 Processor -- start 2024-09-17T05:02:53.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0315bf640 1 -- start start 2024-09-17T05:02:53.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0315bf640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff02c0a98a0 0x7ff02c14c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:53.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0315bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff02c0c3660 0x7ff02c0ac200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:53.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0315bf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff02c150d40 0x7ff02c1ecc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:53.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0315bf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff02c073390 con 0x7ff02c0c3660 2024-09-17T05:02:53.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0315bf640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff02c151210 con 0x7ff02c0a98a0 2024-09-17T05:02:53.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0315bf640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff02c06e170 con 0x7ff02c150d40 2024-09-17T05:02:53.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0227fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff02c0c3660 0x7ff02c0ac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:53.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0227fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff02c0c3660 0x7ff02c0ac200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49054/0 (socket says 172.21.15.150:49054) 2024-09-17T05:02:53.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0227fc640 1 -- 172.21.15.150:0/3969411499 learned_addr learned my addr 172.21.15.150:0/3969411499 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff02b7fe640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff02c150d40 0x7ff02c1ecc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff02affd640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff02c0a98a0 0x7ff02c14c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0227fc640 1 -- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff02c150d40 msgr2=0x7ff02c1ecc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0227fc640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff02c150d40 0x7ff02c1ecc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0227fc640 1 -- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff02c0a98a0 msgr2=0x7ff02c14c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0227fc640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff02c0a98a0 0x7ff02c14c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0227fc640 1 -- 172.21.15.150:0/3969411499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff02c143470 con 0x7ff02c0c3660 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0227fc640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff02c0c3660 0x7ff02c0ac200 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff0140058d0 tx=0x7ff01400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff023fff640 1 -- 172.21.15.150:0/3969411499 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014017070 con 0x7ff02c0c3660 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff023fff640 1 -- 172.21.15.150:0/3969411499 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff014009d20 con 0x7ff02c0c3660 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff023fff640 1 -- 172.21.15.150:0/3969411499 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0140106c0 con 0x7ff02c0c3660 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0315bf640 1 -- 172.21.15.150:0/3969411499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff02c143190 con 0x7ff02c0c3660 2024-09-17T05:02:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0315bf640 1 -- 172.21.15.150:0/3969411499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff02c1ea480 con 0x7ff02c0c3660 2024-09-17T05:02:53.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.161+0000 7ff0315bf640 1 -- 172.21.15.150:0/3969411499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff02c069cc0 con 0x7ff02c0c3660 2024-09-17T05:02:53.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.165+0000 7ff023fff640 1 -- 172.21.15.150:0/3969411499 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff01401a070 con 0x7ff02c0c3660 2024-09-17T05:02:53.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.165+0000 7ff023fff640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff00c073230 0x7ff00c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:53.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.165+0000 7ff023fff640 1 -- 172.21.15.150:0/3969411499 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7ff014094cf0 con 0x7ff02c0c3660 2024-09-17T05:02:53.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.165+0000 7ff02affd640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff00c073230 0x7ff00c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:53.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.165+0000 7ff02affd640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff00c073230 0x7ff00c075740 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff01c000f70 tx=0x7ff01c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:53.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.169+0000 7ff023fff640 1 -- 172.21.15.150:0/3969411499 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff014099070 con 0x7ff02c0c3660 2024-09-17T05:02:53.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.377+0000 7ff0315bf640 1 -- 172.21.15.150:0/3969411499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff02c1530a0 con 0x7ff02c0c3660 2024-09-17T05:02:53.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.377+0000 7ff023fff640 1 -- 172.21.15.150:0/3969411499 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+6534 (secure 0 0 0) 0x7ff01405f3f0 con 0x7ff02c0c3660 2024-09-17T05:02:53.373 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:53.373 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":96,"btime":"2024-09-17T05:02:45:058000+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:02:53.374 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 96 2024-09-17T05:02:53.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.377+0000 7ff0315bf640 1 -- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff00c073230 msgr2=0x7ff00c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:53.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.377+0000 7ff0315bf640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff00c073230 0x7ff00c075740 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff01c000f70 tx=0x7ff01c009040 comp rx=0 tx=0).stop 2024-09-17T05:02:53.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.377+0000 7ff0315bf640 1 -- 172.21.15.150:0/3969411499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff02c0c3660 msgr2=0x7ff02c0ac200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:53.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.377+0000 7ff0315bf640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff02c0c3660 0x7ff02c0ac200 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff0140058d0 tx=0x7ff01400dde0 comp rx=0 tx=0).stop 2024-09-17T05:02:53.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.377+0000 7ff0315bf640 1 -- 172.21.15.150:0/3969411499 shutdown_connections 2024-09-17T05:02:53.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.377+0000 7ff0315bf640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff00c073230 0x7ff00c075740 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:53.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.377+0000 7ff0315bf640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff02c150d40 0x7ff02c1ecc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:53.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.377+0000 7ff0315bf640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff02c0c3660 0x7ff02c0ac200 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:53.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.377+0000 7ff0315bf640 1 --2- 172.21.15.150:0/3969411499 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff02c0a98a0 0x7ff02c14c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:53.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.377+0000 7ff0315bf640 1 -- 172.21.15.150:0/3969411499 >> 172.21.15.150:0/3969411499 conn(0x7ff02c070c20 msgr2=0x7ff02c0ab050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:53.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.381+0000 7ff0315bf640 1 -- 172.21.15.150:0/3969411499 shutdown_connections 2024-09-17T05:02:53.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:53.381+0000 7ff0315bf640 1 -- 172.21.15.150:0/3969411499 wait complete. 2024-09-17T05:02:53.396 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:02:55.397 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:55.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.561+0000 7f751069f640 1 -- 172.21.15.150:0/3371120560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74f0004810 msgr2=0x7f74f0004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:55.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.561+0000 7f751069f640 1 --2- 172.21.15.150:0/3371120560 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74f0004810 0x7f74f0004c20 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f7508073d10 tx=0x7f74fc01d1d0 comp rx=0 tx=0).stop 2024-09-17T05:02:55.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.561+0000 7f750ec3c640 1 -- 172.21.15.150:0/3371120560 reap_dead start 2024-09-17T05:02:55.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.561+0000 7f751069f640 1 -- 172.21.15.150:0/3371120560 shutdown_connections 2024-09-17T05:02:55.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.561+0000 7f751069f640 1 -- 172.21.15.150:0/3371120560 >> 172.21.15.150:0/3371120560 conn(0x7f75080aab60 msgr2=0x7f75080aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:55.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.561+0000 7f751069f640 1 -- 172.21.15.150:0/3371120560 shutdown_connections 2024-09-17T05:02:55.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.561+0000 7f751069f640 1 -- 172.21.15.150:0/3371120560 wait complete. 2024-09-17T05:02:55.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f751069f640 1 Processor -- start 2024-09-17T05:02:55.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f751069f640 1 -- start start 2024-09-17T05:02:55.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f751069f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74f0004810 0x7f75080a9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:55.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f751069f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f750814f7b0 0x7f75081f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:55.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f751069f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f750814fbc0 0x7f7508142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f751069f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7508154560 con 0x7f74f0004810 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f751069f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f75080c6cb0 con 0x7f750814f7b0 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f751069f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f750806d9f0 con 0x7f750814fbc0 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f750e43b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74f0004810 0x7f75080a9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f750e43b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74f0004810 0x7f75080a9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49074/0 (socket says 172.21.15.150:49074) 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f750e43b640 1 -- 172.21.15.150:0/1733911986 learned_addr learned my addr 172.21.15.150:0/1733911986 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f750ec3c640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f750814fbc0 0x7f7508142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f750dc3a640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f750814f7b0 0x7f75081f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f750e43b640 1 -- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f750814fbc0 msgr2=0x7f7508142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f750e43b640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f750814fbc0 0x7f7508142020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f750e43b640 1 -- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f750814f7b0 msgr2=0x7f75081f35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f750e43b640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f750814f7b0 0x7f75081f35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:55.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f750e43b640 1 -- 172.21.15.150:0/1733911986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750814bd50 con 0x7f74f0004810 2024-09-17T05:02:55.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f750e43b640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74f0004810 0x7f75080a9580 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f750000e910 tx=0x7f750000edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:55.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f7506ffd640 1 -- 172.21.15.150:0/1733911986 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7500015070 con 0x7f74f0004810 2024-09-17T05:02:55.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f7506ffd640 1 -- 172.21.15.150:0/1733911986 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f750000ce20 con 0x7f74f0004810 2024-09-17T05:02:55.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f751069f640 1 -- 172.21.15.150:0/1733911986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7508147ca0 con 0x7f74f0004810 2024-09-17T05:02:55.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f7506ffd640 1 -- 172.21.15.150:0/1733911986 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75000116a0 con 0x7f74f0004810 2024-09-17T05:02:55.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.565+0000 7f751069f640 1 -- 172.21.15.150:0/1733911986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f750814de40 con 0x7f74f0004810 2024-09-17T05:02:55.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.569+0000 7f751069f640 1 -- 172.21.15.150:0/1733911986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f750806ce50 con 0x7f74f0004810 2024-09-17T05:02:55.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.569+0000 7f7506ffd640 1 -- 172.21.15.150:0/1733911986 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7500018070 con 0x7f74f0004810 2024-09-17T05:02:55.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.569+0000 7f7506ffd640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f74f4073200 0x7f74f4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:55.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.569+0000 7f7506ffd640 1 -- 172.21.15.150:0/1733911986 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f7500095590 con 0x7f74f0004810 2024-09-17T05:02:55.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.569+0000 7f750dc3a640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f74f4073200 0x7f74f4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:55.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.569+0000 7f750dc3a640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f74f4073200 0x7f74f4075710 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0ae000 tx=0x7f74fc019fd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:55.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.573+0000 7f7506ffd640 1 -- 172.21.15.150:0/1733911986 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f750005fe90 con 0x7f74f0004810 2024-09-17T05:02:55.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.765+0000 7f751069f640 1 -- 172.21.15.150:0/1733911986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75080c2560 con 0x7f74f0004810 2024-09-17T05:02:55.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.765+0000 7f7506ffd640 1 -- 172.21.15.150:0/1733911986 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+6534 (secure 0 0 0) 0x7f750005f790 con 0x7f74f0004810 2024-09-17T05:02:55.764 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:55.764 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":96,"btime":"2024-09-17T05:02:45:058000+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:02:55.764 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 96 2024-09-17T05:02:55.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.769+0000 7f751069f640 1 -- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f74f4073200 msgr2=0x7f74f4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:55.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.769+0000 7f751069f640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f74f4073200 0x7f74f4075710 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f74fc0ae000 tx=0x7f74fc019fd0 comp rx=0 tx=0).stop 2024-09-17T05:02:55.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.769+0000 7f751069f640 1 -- 172.21.15.150:0/1733911986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74f0004810 msgr2=0x7f75080a9580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:55.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.769+0000 7f751069f640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74f0004810 0x7f75080a9580 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f750000e910 tx=0x7f750000edd0 comp rx=0 tx=0).stop 2024-09-17T05:02:55.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.769+0000 7f751069f640 1 -- 172.21.15.150:0/1733911986 shutdown_connections 2024-09-17T05:02:55.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.769+0000 7f751069f640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f74f4073200 0x7f74f4075710 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:55.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.769+0000 7f751069f640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f750814f7b0 0x7f75081f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:55.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.769+0000 7f751069f640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f750814fbc0 0x7f7508142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:55.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.769+0000 7f751069f640 1 --2- 172.21.15.150:0/1733911986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74f0004810 0x7f75080a9580 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:55.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.769+0000 7f751069f640 1 -- 172.21.15.150:0/1733911986 >> 172.21.15.150:0/1733911986 conn(0x7f75080aab60 msgr2=0x7f7508150fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:55.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.769+0000 7f751069f640 1 -- 172.21.15.150:0/1733911986 shutdown_connections 2024-09-17T05:02:55.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:55.769+0000 7f751069f640 1 -- 172.21.15.150:0/1733911986 wait complete. 2024-09-17T05:02:55.784 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:02:57.785 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:02:57.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.953+0000 7fca472a2640 1 -- 172.21.15.150:0/2003019138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca400a9e80 msgr2=0x7fca40074990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:57.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.953+0000 7fca472a2640 1 --2- 172.21.15.150:0/2003019138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca400a9e80 0x7fca40074990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fca30009900 tx=0x7fca3001c820 comp rx=0 tx=0).stop 2024-09-17T05:02:57.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.953+0000 7fca472a2640 1 -- 172.21.15.150:0/2003019138 shutdown_connections 2024-09-17T05:02:57.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.953+0000 7fca472a2640 1 --2- 172.21.15.150:0/2003019138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca400a9e80 0x7fca40074990 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:57.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.953+0000 7fca472a2640 1 --2- 172.21.15.150:0/2003019138 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fca400708a0 0x7fca40150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:57.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.953+0000 7fca472a2640 1 -- 172.21.15.150:0/2003019138 >> 172.21.15.150:0/2003019138 conn(0x7fca400c4d50 msgr2=0x7fca400c5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:57.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.953+0000 7fca472a2640 1 -- 172.21.15.150:0/2003019138 shutdown_connections 2024-09-17T05:02:57.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.953+0000 7fca472a2640 1 -- 172.21.15.150:0/2003019138 wait complete. 2024-09-17T05:02:57.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca472a2640 1 Processor -- start 2024-09-17T05:02:57.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca472a2640 1 -- start start 2024-09-17T05:02:57.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca472a2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fca400708a0 0x7fca401ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:57.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca472a2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca400a9e80 0x7fca40156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:57.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca472a2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca400aa3d0 0x7fca40145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:57.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca472a2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fca40160c50 con 0x7fca400aa3d0 2024-09-17T05:02:57.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca472a2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fca40160ad0 con 0x7fca400a9e80 2024-09-17T05:02:57.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca472a2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fca40160dd0 con 0x7fca400708a0 2024-09-17T05:02:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca4583f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca400aa3d0 0x7fca40145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca4583f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca400aa3d0 0x7fca40145610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49084/0 (socket says 172.21.15.150:49084) 2024-09-17T05:02:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca4583f640 1 -- 172.21.15.150:0/2228363975 learned_addr learned my addr 172.21.15.150:0/2228363975 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:02:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca4503e640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fca400708a0 0x7fca401ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca4483d640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca400a9e80 0x7fca40156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca4583f640 1 -- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fca400708a0 msgr2=0x7fca401ee5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca4583f640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fca400708a0 0x7fca401ee5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca4583f640 1 -- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca400a9e80 msgr2=0x7fca40156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca4583f640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca400a9e80 0x7fca40156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca4583f640 1 -- 172.21.15.150:0/2228363975 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca401f26f0 con 0x7fca400aa3d0 2024-09-17T05:02:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca4583f640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca400aa3d0 0x7fca40145610 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fca3800ba10 tx=0x7fca3800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:57.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca35ffb640 1 -- 172.21.15.150:0/2228363975 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38019070 con 0x7fca400aa3d0 2024-09-17T05:02:57.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca35ffb640 1 -- 172.21.15.150:0/2228363975 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca3800cc30 con 0x7fca400aa3d0 2024-09-17T05:02:57.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca472a2640 1 -- 172.21.15.150:0/2228363975 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca40146170 con 0x7fca400aa3d0 2024-09-17T05:02:57.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca35ffb640 1 -- 172.21.15.150:0/2228363975 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3800ce60 con 0x7fca400aa3d0 2024-09-17T05:02:57.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca472a2640 1 -- 172.21.15.150:0/2228363975 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca40147950 con 0x7fca400aa3d0 2024-09-17T05:02:57.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.957+0000 7fca472a2640 1 -- 172.21.15.150:0/2228363975 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca401463b0 con 0x7fca400aa3d0 2024-09-17T05:02:57.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.961+0000 7fca35ffb640 1 -- 172.21.15.150:0/2228363975 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fca3801c070 con 0x7fca400aa3d0 2024-09-17T05:02:57.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.961+0000 7fca35ffb640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fca24073230 0x7fca24075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:02:57.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.961+0000 7fca35ffb640 1 -- 172.21.15.150:0/2228363975 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7fca38094ac0 con 0x7fca400aa3d0 2024-09-17T05:02:57.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.961+0000 7fca4503e640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fca24073230 0x7fca24075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:02:57.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.961+0000 7fca4503e640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fca24073230 0x7fca24075740 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0058d0 tx=0x7fca3c005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:02:57.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:57.965+0000 7fca35ffb640 1 -- 172.21.15.150:0/2228363975 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fca401463b0 con 0x7fca400aa3d0 2024-09-17T05:02:58.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.173+0000 7fca472a2640 1 -- 172.21.15.150:0/2228363975 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca40150dc0 con 0x7fca400aa3d0 2024-09-17T05:02:58.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.173+0000 7fca35ffb640 1 -- 172.21.15.150:0/2228363975 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+6534 (secure 0 0 0) 0x7fca3805f250 con 0x7fca400aa3d0 2024-09-17T05:02:58.170 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:02:58.170 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":96,"btime":"2024-09-17T05:02:45:058000+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:02:58.170 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 96 2024-09-17T05:02:58.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.173+0000 7fca472a2640 1 -- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fca24073230 msgr2=0x7fca24075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:58.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.173+0000 7fca472a2640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fca24073230 0x7fca24075740 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0058d0 tx=0x7fca3c005860 comp rx=0 tx=0).stop 2024-09-17T05:02:58.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.173+0000 7fca472a2640 1 -- 172.21.15.150:0/2228363975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca400aa3d0 msgr2=0x7fca40145610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:02:58.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.173+0000 7fca472a2640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca400aa3d0 0x7fca40145610 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fca3800ba10 tx=0x7fca3800bed0 comp rx=0 tx=0).stop 2024-09-17T05:02:58.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.177+0000 7fca472a2640 1 -- 172.21.15.150:0/2228363975 shutdown_connections 2024-09-17T05:02:58.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.177+0000 7fca472a2640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fca24073230 0x7fca24075740 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:58.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.177+0000 7fca472a2640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca400a9e80 0x7fca40156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:58.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.177+0000 7fca472a2640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca400aa3d0 0x7fca40145610 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:58.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.177+0000 7fca472a2640 1 --2- 172.21.15.150:0/2228363975 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fca400708a0 0x7fca401ee5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:02:58.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.177+0000 7fca472a2640 1 -- 172.21.15.150:0/2228363975 >> 172.21.15.150:0/2228363975 conn(0x7fca400c4d50 msgr2=0x7fca4014c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:02:58.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.177+0000 7fca472a2640 1 -- 172.21.15.150:0/2228363975 shutdown_connections 2024-09-17T05:02:58.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:02:58.177+0000 7fca472a2640 1 -- 172.21.15.150:0/2228363975 wait complete. 2024-09-17T05:02:58.191 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:03:00.192 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:03:00.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.361+0000 7f2b9371f640 1 -- 172.21.15.150:0/2261035797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b8c0c35b0 msgr2=0x7f2b8c0c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.361+0000 7f2b9371f640 1 --2- 172.21.15.150:0/2261035797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b8c0c35b0 0x7f2b8c0c8a80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009900 tx=0x7f2b7c01c8b0 comp rx=0 tx=0).stop 2024-09-17T05:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.361+0000 7f2b9371f640 1 -- 172.21.15.150:0/2261035797 shutdown_connections 2024-09-17T05:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.361+0000 7f2b9371f640 1 --2- 172.21.15.150:0/2261035797 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b8c0c8fc0 0x7f2b8c0c20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.361+0000 7f2b9371f640 1 --2- 172.21.15.150:0/2261035797 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b8c0c35b0 0x7f2b8c0c8a80 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.361+0000 7f2b9371f640 1 -- 172.21.15.150:0/2261035797 >> 172.21.15.150:0/2261035797 conn(0x7f2b8c074b80 msgr2=0x7f2b8c06f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.361+0000 7f2b9371f640 1 -- 172.21.15.150:0/2261035797 shutdown_connections 2024-09-17T05:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.361+0000 7f2b9371f640 1 -- 172.21.15.150:0/2261035797 wait complete. 2024-09-17T05:03:00.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.361+0000 7f2b9371f640 1 Processor -- start 2024-09-17T05:03:00.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.361+0000 7f2b9371f640 1 -- start start 2024-09-17T05:03:00.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b9371f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b8c0c35b0 0x7f2b8c150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b9371f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b8c0c8fc0 0x7f2b8c14fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b9371f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b8c1568a0 0x7f2b8c1f3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b9371f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c0c6b70 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b9371f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c0c69f0 con 0x7f2b8c1568a0 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b9371f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2b8c0b0fd0 con 0x7f2b8c0c8fc0 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b914bb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b8c0c35b0 0x7f2b8c150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b914bb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b8c0c35b0 0x7f2b8c150840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36990/0 (socket says 172.21.15.150:36990) 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b914bb640 1 -- 172.21.15.150:0/3863260663 learned_addr learned my addr 172.21.15.150:0/3863260663 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b91cbc640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b8c1568a0 0x7f2b8c1f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b914bb640 1 -- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b8c0c8fc0 msgr2=0x7f2b8c14fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b914bb640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b8c0c8fc0 0x7f2b8c14fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b914bb640 1 -- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b8c1568a0 msgr2=0x7f2b8c1f3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:00.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b914bb640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b8c1568a0 0x7f2b8c1f3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:00.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b914bb640 1 -- 172.21.15.150:0/3863260663 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8c1448c0 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b914bb640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b8c0c35b0 0x7f2b8c150840 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800b410 tx=0x7f2b8800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:00.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b81ffb640 1 -- 172.21.15.150:0/3863260663 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b88015030 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b81ffb640 1 -- 172.21.15.150:0/3863260663 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b88004020 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b9371f640 1 -- 172.21.15.150:0/3863260663 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8c155d10 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b81ffb640 1 -- 172.21.15.150:0/3863260663 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b880044b0 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b9371f640 1 -- 172.21.15.150:0/3863260663 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b8c1f4100 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.365+0000 7f2b9371f640 1 -- 172.21.15.150:0/3863260663 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b8c0b1500 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.369+0000 7f2b81ffb640 1 -- 172.21.15.150:0/3863260663 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2b88018070 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.369+0000 7f2b81ffb640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2b70073230 0x7f2b70075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:00.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.369+0000 7f2b81ffb640 1 -- 172.21.15.150:0/3863260663 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f2b88094720 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.369+0000 7f2b90cba640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2b70073230 0x7f2b70075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:00.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.369+0000 7f2b90cba640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2b70073230 0x7f2b70075740 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0af000 tx=0x7f2b7c019650 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:00.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.373+0000 7f2b81ffb640 1 -- 172.21.15.150:0/3863260663 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2b8805eea0 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.577+0000 7f2b9371f640 1 -- 172.21.15.150:0/3863260663 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b8c144c70 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.577+0000 7f2b81ffb640 1 -- 172.21.15.150:0/3863260663 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+6534 (secure 0 0 0) 0x7f2b8805e7a0 con 0x7f2b8c0c35b0 2024-09-17T05:03:00.574 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:03:00.574 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":96,"btime":"2024-09-17T05:02:45:058000+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6010,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:00:49.115368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6252},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6252":{"gid":6252,"name":"c","rank":0,"incarnation":87,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/2062728083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2062728083},{"type":"v1","addr":"172.21.15.150:6835","nonce":2062728083}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6252,"qdb_cluster":[6252]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:03:00.574 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 96 2024-09-17T05:03:00.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.581+0000 7f2b9371f640 1 -- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2b70073230 msgr2=0x7f2b70075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:00.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.581+0000 7f2b9371f640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2b70073230 0x7f2b70075740 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0af000 tx=0x7f2b7c019650 comp rx=0 tx=0).stop 2024-09-17T05:03:00.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.581+0000 7f2b9371f640 1 -- 172.21.15.150:0/3863260663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b8c0c35b0 msgr2=0x7f2b8c150840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:00.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.581+0000 7f2b9371f640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b8c0c35b0 0x7f2b8c150840 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f2b8800b410 tx=0x7f2b8800b8d0 comp rx=0 tx=0).stop 2024-09-17T05:03:00.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.581+0000 7f2b9371f640 1 -- 172.21.15.150:0/3863260663 shutdown_connections 2024-09-17T05:03:00.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.581+0000 7f2b9371f640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2b70073230 0x7f2b70075740 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:00.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.581+0000 7f2b9371f640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b8c1568a0 0x7f2b8c1f3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:00.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.581+0000 7f2b9371f640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b8c0c8fc0 0x7f2b8c14fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:00.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.581+0000 7f2b9371f640 1 --2- 172.21.15.150:0/3863260663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b8c0c35b0 0x7f2b8c150840 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:00.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.581+0000 7f2b9371f640 1 -- 172.21.15.150:0/3863260663 >> 172.21.15.150:0/3863260663 conn(0x7f2b8c074b80 msgr2=0x7f2b8c0c5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:00.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.581+0000 7f2b9371f640 1 -- 172.21.15.150:0/3863260663 shutdown_connections 2024-09-17T05:03:00.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:00.581+0000 7f2b9371f640 1 -- 172.21.15.150:0/3863260663 wait complete. 2024-09-17T05:03:00.595 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:03:01.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:03:01.570 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:03:02.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:03:02.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.769+0000 7f82c4447640 1 -- 172.21.15.150:0/1243246104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc073440 msgr2=0x7f82bc0a7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:02.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.769+0000 7f82c4447640 1 --2- 172.21.15.150:0/1243246104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc073440 0x7f82bc0a7300 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009840 tx=0x7f82b001c790 comp rx=0 tx=0).stop 2024-09-17T05:03:02.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.769+0000 7f82c4447640 1 -- 172.21.15.150:0/1243246104 shutdown_connections 2024-09-17T05:03:02.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.769+0000 7f82c4447640 1 --2- 172.21.15.150:0/1243246104 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82bc074450 0x7f82bc074860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:02.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.769+0000 7f82c4447640 1 --2- 172.21.15.150:0/1243246104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc073440 0x7f82bc0a7300 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:02.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.769+0000 7f82c4447640 1 -- 172.21.15.150:0/1243246104 >> 172.21.15.150:0/1243246104 conn(0x7f82bc0c4a00 msgr2=0x7f82bc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:02.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.769+0000 7f82c4447640 1 -- 172.21.15.150:0/1243246104 shutdown_connections 2024-09-17T05:03:02.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.769+0000 7f82c4447640 1 -- 172.21.15.150:0/1243246104 wait complete. 2024-09-17T05:03:02.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.769+0000 7f82c4447640 1 Processor -- start 2024-09-17T05:03:02.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.769+0000 7f82c4447640 1 -- start start 2024-09-17T05:03:02.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c4447640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc070e50 0x7f82bc144c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:02.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c4447640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82bc073440 0x7f82bc0ac450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:02.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c4447640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82bc074450 0x7f82bc14ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:02.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c4447640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc0adef0 con 0x7f82bc070e50 2024-09-17T05:03:02.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c4447640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc071a50 con 0x7f82bc073440 2024-09-17T05:03:02.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c4447640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f82bc0ad610 con 0x7f82bc074450 2024-09-17T05:03:02.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c21e3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc070e50 0x7f82bc144c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:02.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c21e3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc070e50 0x7f82bc144c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37006/0 (socket says 172.21.15.150:37006) 2024-09-17T05:03:02.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c21e3640 1 -- 172.21.15.150:0/2265421315 learned_addr learned my addr 172.21.15.150:0/2265421315 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:03:02.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c29e4640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82bc074450 0x7f82bc14ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:02.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c19e2640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82bc073440 0x7f82bc0ac450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:02.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c21e3640 1 -- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82bc074450 msgr2=0x7f82bc14ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:02.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c21e3640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82bc074450 0x7f82bc14ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:02.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c21e3640 1 -- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82bc073440 msgr2=0x7f82bc0ac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:02.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c21e3640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82bc073440 0x7f82bc0ac450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:02.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c21e3640 1 -- 172.21.15.150:0/2265421315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82bc1456d0 con 0x7f82bc070e50 2024-09-17T05:03:02.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c21e3640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc070e50 0x7f82bc144c90 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f82b0000c10 tx=0x7f82b00a64e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:02.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82aeffd640 1 -- 172.21.15.150:0/2265421315 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b00ac030 con 0x7f82bc070e50 2024-09-17T05:03:02.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82aeffd640 1 -- 172.21.15.150:0/2265421315 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82b00053b0 con 0x7f82bc070e50 2024-09-17T05:03:02.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82aeffd640 1 -- 172.21.15.150:0/2265421315 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b00046d0 con 0x7f82bc070e50 2024-09-17T05:03:02.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c4447640 1 -- 172.21.15.150:0/2265421315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82bc1f0d40 con 0x7f82bc070e50 2024-09-17T05:03:02.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c4447640 1 -- 172.21.15.150:0/2265421315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82bc152d20 con 0x7f82bc070e50 2024-09-17T05:03:02.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82c4447640 1 -- 172.21.15.150:0/2265421315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82bc0adab0 con 0x7f82bc070e50 2024-09-17T05:03:02.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.773+0000 7f82aeffd640 1 -- 172.21.15.150:0/2265421315 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f82b00af070 con 0x7f82bc070e50 2024-09-17T05:03:02.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.777+0000 7f82aeffd640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82a0073200 0x7f82a0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:02.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.777+0000 7f82aeffd640 1 -- 172.21.15.150:0/2265421315 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f82b012f390 con 0x7f82bc070e50 2024-09-17T05:03:02.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.777+0000 7f82c19e2640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82a0073200 0x7f82a0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:02.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.777+0000 7f82c19e2640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82a0073200 0x7f82a0075710 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f82bc150eb0 tx=0x7f82b8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:02.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.781+0000 7f82aeffd640 1 -- 172.21.15.150:0/2265421315 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f82b0016410 con 0x7f82bc070e50 2024-09-17T05:03:02.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.985+0000 7f82c4447640 1 -- 172.21.15.150:0/2265421315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82bc0c2410 con 0x7f82bc070e50 2024-09-17T05:03:02.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82aeffd640 1 -- 172.21.15.150:0/2265421315 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) ==== 94+0+5702 (secure 0 0 0) 0x7f82b00f9510 con 0x7f82bc070e50 2024-09-17T05:03:02.984 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:03:02.985 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":98,"btime":"2024-09-17T05:03:02:128810+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:01.447392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:03:02.985 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 98 2024-09-17T05:03:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82c4447640 1 -- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82a0073200 msgr2=0x7f82a0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82c4447640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82a0073200 0x7f82a0075710 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f82bc150eb0 tx=0x7f82b8009040 comp rx=0 tx=0).stop 2024-09-17T05:03:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82c4447640 1 -- 172.21.15.150:0/2265421315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc070e50 msgr2=0x7f82bc144c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82c4447640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc070e50 0x7f82bc144c90 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f82b0000c10 tx=0x7f82b00a64e0 comp rx=0 tx=0).stop 2024-09-17T05:03:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82c4447640 1 -- 172.21.15.150:0/2265421315 shutdown_connections 2024-09-17T05:03:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82c4447640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82a0073200 0x7f82a0075710 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82c4447640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82bc073440 0x7f82bc0ac450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82c4447640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82bc074450 0x7f82bc14ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82c4447640 1 --2- 172.21.15.150:0/2265421315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82bc070e50 0x7f82bc144c90 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82c4447640 1 -- 172.21.15.150:0/2265421315 >> 172.21.15.150:0/2265421315 conn(0x7f82bc0c4a00 msgr2=0x7f82bc14b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82c4447640 1 -- 172.21.15.150:0/2265421315 shutdown_connections 2024-09-17T05:03:02.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:02.989+0000 7f82c4447640 1 -- 172.21.15.150:0/2265421315 wait complete. 2024-09-17T05:03:03.007 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-17T05:03:03.007 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:03:03.007 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:03:03.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 -- 172.21.15.150:0/1781824813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c4073440 msgr2=0x7f86c40a7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:03.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 --2- 172.21.15.150:0/1781824813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c4073440 0x7f86c40a7300 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f86c4072370 tx=0x7f86b401ccf0 comp rx=0 tx=0).stop 2024-09-17T05:03:03.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 -- 172.21.15.150:0/1781824813 shutdown_connections 2024-09-17T05:03:03.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 --2- 172.21.15.150:0/1781824813 >> v2:172.21.15.150:3300/0 conn(0x7f86c4070e50 0x7f86c4071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:03.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 --2- 172.21.15.150:0/1781824813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c4073440 0x7f86c40a7300 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:03.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 -- 172.21.15.150:0/1781824813 >> 172.21.15.150:0/1781824813 conn(0x7f86c40c4a00 msgr2=0x7f86c40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:03.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 -- 172.21.15.150:0/1781824813 shutdown_connections 2024-09-17T05:03:03.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 -- 172.21.15.150:0/1781824813 wait complete. 2024-09-17T05:03:03.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 Processor -- start 2024-09-17T05:03:03.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 -- start start 2024-09-17T05:03:03.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86c4070e50 0x7f86c4158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:03.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c40717a0 0x7f86c4155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:03.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86c4073440 0x7f86c414a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:03.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f86c4074f70 con 0x7f86c4073440 2024-09-17T05:03:03.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f86c40adef0 con 0x7f86c40717a0 2024-09-17T05:03:03.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.169+0000 7f86c9853640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f86c40750f0 con 0x7f86c4070e50 2024-09-17T05:03:03.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86c4073440 0x7f86c414a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:03.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86c4073440 0x7f86c414a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37026/0 (socket says 172.21.15.150:37026) 2024-09-17T05:03:03.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c37fe640 1 -- 172.21.15.150:0/3409967808 learned_addr learned my addr 172.21.15.150:0/3409967808 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:03:03.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c27fc640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c40717a0 0x7f86c4155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:03.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c2ffd640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86c4070e50 0x7f86c4158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:03.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c37fe640 1 -- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86c4070e50 msgr2=0x7f86c4158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:03.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c37fe640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86c4070e50 0x7f86c4158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:03.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c37fe640 1 -- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c40717a0 msgr2=0x7f86c4155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:03.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c37fe640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c40717a0 0x7f86c4155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:03.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c37fe640 1 -- 172.21.15.150:0/3409967808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86c4154c60 con 0x7f86c4073440 2024-09-17T05:03:03.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c37fe640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86c4073440 0x7f86c414a560 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00c3f0 tx=0x7f86bc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:03.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86abfff640 1 -- 172.21.15.150:0/3409967808 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86bc017070 con 0x7f86c4073440 2024-09-17T05:03:03.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86abfff640 1 -- 172.21.15.150:0/3409967808 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86bc0053c0 con 0x7f86c4073440 2024-09-17T05:03:03.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86abfff640 1 -- 172.21.15.150:0/3409967808 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86bc003860 con 0x7f86c4073440 2024-09-17T05:03:03.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c9853640 1 -- 172.21.15.150:0/3409967808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86c41482a0 con 0x7f86c4073440 2024-09-17T05:03:03.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c9853640 1 -- 172.21.15.150:0/3409967808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86c4153360 con 0x7f86c4073440 2024-09-17T05:03:03.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c9853640 1 -- 172.21.15.150:0/3409967808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86c4148090 con 0x7f86c4073440 2024-09-17T05:03:03.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86abfff640 1 -- 172.21.15.150:0/3409967808 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f86bc01a070 con 0x7f86c4073440 2024-09-17T05:03:03.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86abfff640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f86b0073200 0x7f86b0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:03.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86abfff640 1 -- 172.21.15.150:0/3409967808 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f86bc093df0 con 0x7f86c4073440 2024-09-17T05:03:03.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.173+0000 7f86c2ffd640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f86b0073200 0x7f86b0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:03.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.177+0000 7f86c2ffd640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f86b0073200 0x7f86b0075710 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f86b40af000 tx=0x7f86b401b530 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:03.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.181+0000 7f86abfff640 1 -- 172.21.15.150:0/3409967808 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f86bc05df20 con 0x7f86c4073440 2024-09-17T05:03:03.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.385+0000 7f86c9853640 1 -- 172.21.15.150:0/3409967808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86c40ad8e0 con 0x7f86c4073440 2024-09-17T05:03:03.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.385+0000 7f86abfff640 1 -- 172.21.15.150:0/3409967808 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5699 (secure 0 0 0) 0x7f86bc062950 con 0x7f86c4073440 2024-09-17T05:03:03.384 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:03:03.384 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":99,"btime":"2024-09-17T05:03:03:137959+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:02.144094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:03:03.384 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 99 2024-09-17T05:03:03.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.389+0000 7f86c9853640 1 -- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f86b0073200 msgr2=0x7f86b0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:03.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.389+0000 7f86c9853640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f86b0073200 0x7f86b0075710 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f86b40af000 tx=0x7f86b401b530 comp rx=0 tx=0).stop 2024-09-17T05:03:03.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.389+0000 7f86c9853640 1 -- 172.21.15.150:0/3409967808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86c4073440 msgr2=0x7f86c414a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:03.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.389+0000 7f86c9853640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86c4073440 0x7f86c414a560 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00c3f0 tx=0x7f86bc00c8b0 comp rx=0 tx=0).stop 2024-09-17T05:03:03.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.389+0000 7f86c9853640 1 -- 172.21.15.150:0/3409967808 shutdown_connections 2024-09-17T05:03:03.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.389+0000 7f86c9853640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f86b0073200 0x7f86b0075710 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:03.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.389+0000 7f86c9853640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86c40717a0 0x7f86c4155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:03.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.389+0000 7f86c9853640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86c4073440 0x7f86c414a560 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:03.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.389+0000 7f86c9853640 1 --2- 172.21.15.150:0/3409967808 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86c4070e50 0x7f86c4158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:03.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.389+0000 7f86c9853640 1 -- 172.21.15.150:0/3409967808 >> 172.21.15.150:0/3409967808 conn(0x7f86c40c4a00 msgr2=0x7f86c414e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:03.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.389+0000 7f86c9853640 1 -- 172.21.15.150:0/3409967808 shutdown_connections 2024-09-17T05:03:03.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:03.389+0000 7f86c9853640 1 -- 172.21.15.150:0/3409967808 wait complete. 2024-09-17T05:03:03.409 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:03:03:137959+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 99, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 99, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6010": { "addr": "172.21.15.176:6837/903190072", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 903190072, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 903190072, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6010, "incarnation": 97, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:03:02.144094+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6010 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 95, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6342": { "addr": "172.21.15.150:6833/233311351", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 233311351, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 233311351, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6342, "incarnation": 92, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 28 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:02:34.022739+0000", "qdb_cluster": [ 6342 ], "qdb_leader": 6342, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6342 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:03:05.062 INFO:tasks.ceph.mds.d.smithi176.stderr:2024-09-17T05:03:05.063+0000 7fd926d03640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:03:05.410 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:03:05.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.597+0000 7fc9ac71b640 1 -- 172.21.15.150:0/1330817653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9a40c6340 msgr2=0x7fc9a4157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:05.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.597+0000 7fc9ac71b640 1 --2- 172.21.15.150:0/1330817653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9a40c6340 0x7fc9a4157330 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc994009840 tx=0x7fc99401c820 comp rx=0 tx=0).stop 2024-09-17T05:03:05.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.597+0000 7fc9ac71b640 1 -- 172.21.15.150:0/1330817653 shutdown_connections 2024-09-17T05:03:05.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.597+0000 7fc9ac71b640 1 --2- 172.21.15.150:0/1330817653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9a40c4b20 0x7fc9a4153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:05.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.597+0000 7fc9ac71b640 1 --2- 172.21.15.150:0/1330817653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9a40c6340 0x7fc9a4157330 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:05.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.597+0000 7fc9ac71b640 1 -- 172.21.15.150:0/1330817653 >> 172.21.15.150:0/1330817653 conn(0x7fc9a4074750 msgr2=0x7fc9a4074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:05.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.597+0000 7fc9ac71b640 1 -- 172.21.15.150:0/1330817653 shutdown_connections 2024-09-17T05:03:05.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.597+0000 7fc9ac71b640 1 -- 172.21.15.150:0/1330817653 wait complete. 2024-09-17T05:03:05.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9ac71b640 1 Processor -- start 2024-09-17T05:03:05.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9ac71b640 1 -- start start 2024-09-17T05:03:05.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9ac71b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9a40c4b20 0x7fc9a414a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:05.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9ac71b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9a40c5df0 0x7fc9a4151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:05.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9ac71b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9a40c6340 0x7fc9a4158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:05.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9ac71b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a4163d10 con 0x7fc9a40c5df0 2024-09-17T05:03:05.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9ac71b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a4163b90 con 0x7fc9a40c4b20 2024-09-17T05:03:05.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9ac71b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a4163e90 con 0x7fc9a40c6340 2024-09-17T05:03:05.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9a9cb6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9a40c5df0 0x7fc9a4151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:05.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9a9cb6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9a40c5df0 0x7fc9a4151960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37038/0 (socket says 172.21.15.150:37038) 2024-09-17T05:03:05.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9a9cb6640 1 -- 172.21.15.150:0/308857749 learned_addr learned my addr 172.21.15.150:0/308857749 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:03:05.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9aacb8640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9a40c6340 0x7fc9a4158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:05.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9aa4b7640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9a40c4b20 0x7fc9a414a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:05.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9a9cb6640 1 -- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9a40c6340 msgr2=0x7fc9a4158a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:05.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9a9cb6640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9a40c6340 0x7fc9a4158a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:05.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9a9cb6640 1 -- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9a40c4b20 msgr2=0x7fc9a414a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:05.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9a9cb6640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9a40c4b20 0x7fc9a414a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:05.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9a9cb6640 1 -- 172.21.15.150:0/308857749 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a4159a20 con 0x7fc9a40c5df0 2024-09-17T05:03:05.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9aa4b7640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9a40c4b20 0x7fc9a414a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:03:05.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9a9cb6640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9a40c5df0 0x7fc9a4151960 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc994009790 tx=0x7fc9940a5700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:05.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc99affd640 1 -- 172.21.15.150:0/308857749 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9940bb070 con 0x7fc9a40c5df0 2024-09-17T05:03:05.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9ac71b640 1 -- 172.21.15.150:0/308857749 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a4149b90 con 0x7fc9a40c5df0 2024-09-17T05:03:05.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc99affd640 1 -- 172.21.15.150:0/308857749 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc994005030 con 0x7fc9a40c5df0 2024-09-17T05:03:05.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc99affd640 1 -- 172.21.15.150:0/308857749 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9940b2040 con 0x7fc9a40c5df0 2024-09-17T05:03:05.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9ac71b640 1 -- 172.21.15.150:0/308857749 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a4158070 con 0x7fc9a40c5df0 2024-09-17T05:03:05.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.601+0000 7fc9ac71b640 1 -- 172.21.15.150:0/308857749 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9a4153bc0 con 0x7fc9a40c5df0 2024-09-17T05:03:05.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.605+0000 7fc99affd640 1 -- 172.21.15.150:0/308857749 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc994005460 con 0x7fc9a40c5df0 2024-09-17T05:03:05.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.605+0000 7fc99affd640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc988073200 0x7fc988075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:05.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.605+0000 7fc99affd640 1 -- 172.21.15.150:0/308857749 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7fc99412e470 con 0x7fc9a40c5df0 2024-09-17T05:03:05.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.605+0000 7fc9aa4b7640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc988073200 0x7fc988075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:05.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.605+0000 7fc9aa4b7640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc988073200 0x7fc988075710 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0000f70 tx=0x7fc9a0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:05.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.609+0000 7fc99affd640 1 -- 172.21.15.150:0/308857749 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc9940f1db0 con 0x7fc9a40c5df0 2024-09-17T05:03:05.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.885+0000 7fc9ac71b640 1 -- 172.21.15.150:0/308857749 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9a4159790 con 0x7fc9a40c5df0 2024-09-17T05:03:05.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc99affd640 1 -- 172.21.15.150:0/308857749 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) ==== 95+0+5708 (secure 0 0 0) 0x7fc994016410 con 0x7fc9a40c5df0 2024-09-17T05:03:05.884 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:03:05.884 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":100,"btime":"2024-09-17T05:03:04:151371+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 100 2024-09-17T05:03:05.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc9ac71b640 1 -- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc988073200 msgr2=0x7fc988075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:05.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc9ac71b640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc988073200 0x7fc988075710 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0000f70 tx=0x7fc9a0009040 comp rx=0 tx=0).stop 2024-09-17T05:03:05.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc9ac71b640 1 -- 172.21.15.150:0/308857749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9a40c5df0 msgr2=0x7fc9a4151960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:05.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc9ac71b640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9a40c5df0 0x7fc9a4151960 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc994009790 tx=0x7fc9940a5700 comp rx=0 tx=0).stop 2024-09-17T05:03:05.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc9ac71b640 1 -- 172.21.15.150:0/308857749 shutdown_connections 2024-09-17T05:03:05.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc9ac71b640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc988073200 0x7fc988075710 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:05.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc9ac71b640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9a40c6340 0x7fc9a4158a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:05.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc9ac71b640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9a40c5df0 0x7fc9a4151960 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:05.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc9ac71b640 1 --2- 172.21.15.150:0/308857749 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9a40c4b20 0x7fc9a414a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:05.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc9ac71b640 1 -- 172.21.15.150:0/308857749 >> 172.21.15.150:0/308857749 conn(0x7fc9a4074750 msgr2=0x7fc9a40c1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:05.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc9ac71b640 1 -- 172.21.15.150:0/308857749 shutdown_connections 2024-09-17T05:03:05.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:05.889+0000 7fc9ac71b640 1 -- 172.21.15.150:0/308857749 wait complete. 2024-09-17T05:03:05.914 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=6252 2024-09-17T05:03:05.914 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.c 2024-09-17T05:03:05.914 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-17T05:03:05.915 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T05:03:05.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T05:03:05.917 INFO:tasks.ceph.mds.c:Started 2024-09-17T05:03:05.917 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:03:06.016 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T05:03:06.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.089+0000 7fd3dadc2640 1 -- 172.21.15.150:0/1814527762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3bc0099c0 msgr2=0x7fd3bc00bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:06.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.089+0000 7fd3dadc2640 1 --2- 172.21.15.150:0/1814527762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3bc0099c0 0x7fd3bc00bde0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40c76c0 tx=0x7fd3cc01cc20 comp rx=0 tx=0).stop 2024-09-17T05:03:06.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.089+0000 7fd3dadc2640 1 -- 172.21.15.150:0/1814527762 shutdown_connections 2024-09-17T05:03:06.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.089+0000 7fd3dadc2640 1 --2- 172.21.15.150:0/1814527762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3bc0099c0 0x7fd3bc00bde0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:06.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.089+0000 7fd3dadc2640 1 -- 172.21.15.150:0/1814527762 >> 172.21.15.150:0/1814527762 conn(0x7fd3d40ca460 msgr2=0x7fd3d40c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:06.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.089+0000 7fd3dadc2640 1 -- 172.21.15.150:0/1814527762 shutdown_connections 2024-09-17T05:03:06.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.089+0000 7fd3dadc2640 1 -- 172.21.15.150:0/1814527762 wait complete. 2024-09-17T05:03:06.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3dadc2640 1 Processor -- start 2024-09-17T05:03:06.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3dadc2640 1 -- start start 2024-09-17T05:03:06.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3dadc2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3bc0099c0 0x7fd3d413fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:06.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3dadc2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3d4142290 0x7fd3d413c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:06.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3dadc2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3d40c7a30 0x7fd3d40c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:06.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3dadc2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d40c62d0 con 0x7fd3bc0099c0 2024-09-17T05:03:06.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3dadc2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d40c51f0 con 0x7fd3d40c7a30 2024-09-17T05:03:06.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3dadc2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d40c65c0 con 0x7fd3d4142290 2024-09-17T05:03:06.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3d8b5e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3bc0099c0 0x7fd3d413fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:06.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3d8b5e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3bc0099c0 0x7fd3d413fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37058/0 (socket says 172.21.15.150:37058) 2024-09-17T05:03:06.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3d8b5e640 1 -- 172.21.15.150:0/2085263761 learned_addr learned my addr 172.21.15.150:0/2085263761 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:03:06.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3cbfff640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3d4142290 0x7fd3d413c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:06.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3d935f640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3d40c7a30 0x7fd3d40c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:06.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3d8b5e640 1 -- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3d4142290 msgr2=0x7fd3d413c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:06.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3d8b5e640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3d4142290 0x7fd3d413c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:06.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3d8b5e640 1 -- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3d40c7a30 msgr2=0x7fd3d40c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:06.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3d8b5e640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3d40c7a30 0x7fd3d40c7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:06.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3d8b5e640 1 -- 172.21.15.150:0/2085263761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d40c9490 con 0x7fd3bc0099c0 2024-09-17T05:03:06.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3cbfff640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3d4142290 0x7fd3d413c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:03:06.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3d935f640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3d40c7a30 0x7fd3d40c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:03:06.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3d8b5e640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3bc0099c0 0x7fd3d413fe20 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0001780 tx=0x7fd3d000e6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:06.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3c97fa640 1 -- 172.21.15.150:0/2085263761 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0012850 con 0x7fd3bc0099c0 2024-09-17T05:03:06.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3c97fa640 1 -- 172.21.15.150:0/2085263761 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd3d0012ae0 con 0x7fd3bc0099c0 2024-09-17T05:03:06.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3dadc2640 1 -- 172.21.15.150:0/2085263761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d413cb50 con 0x7fd3bc0099c0 2024-09-17T05:03:06.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3c97fa640 1 -- 172.21.15.150:0/2085263761 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d001a400 con 0x7fd3bc0099c0 2024-09-17T05:03:06.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.093+0000 7fd3dadc2640 1 -- 172.21.15.150:0/2085263761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d41ee300 con 0x7fd3bc0099c0 2024-09-17T05:03:06.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.097+0000 7fd3dadc2640 1 -- 172.21.15.150:0/2085263761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3d4072180 con 0x7fd3bc0099c0 2024-09-17T05:03:06.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.097+0000 7fd3c97fa640 1 -- 172.21.15.150:0/2085263761 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd3d000d030 con 0x7fd3bc0099c0 2024-09-17T05:03:06.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.097+0000 7fd3c97fa640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd3c0073200 0x7fd3c0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:06.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.097+0000 7fd3c97fa640 1 -- 172.21.15.150:0/2085263761 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7fd3d0097e70 con 0x7fd3bc0099c0 2024-09-17T05:03:06.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.097+0000 7fd3cbfff640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd3c0073200 0x7fd3c0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:06.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.097+0000 7fd3cbfff640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd3c0073200 0x7fd3c0075710 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4158450 tx=0x7fd3c4005dd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:06.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.101+0000 7fd3c97fa640 1 -- 172.21.15.150:0/2085263761 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd3d0016030 con 0x7fd3bc0099c0 2024-09-17T05:03:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3dadc2640 1 -- 172.21.15.150:0/2085263761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3d414d2b0 con 0x7fd3bc0099c0 2024-09-17T05:03:06.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3c97fa640 1 -- 172.21.15.150:0/2085263761 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+6537 (secure 0 0 0) 0x7fd3d0061f70 con 0x7fd3bc0099c0 2024-09-17T05:03:06.345 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:03:06.345 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":101,"btime":"2024-09-17T05:03:06:161377+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:03:06.345 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 101 2024-09-17T05:03:06.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3dadc2640 1 -- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd3c0073200 msgr2=0x7fd3c0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3dadc2640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd3c0073200 0x7fd3c0075710 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4158450 tx=0x7fd3c4005dd0 comp rx=0 tx=0).stop 2024-09-17T05:03:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3dadc2640 1 -- 172.21.15.150:0/2085263761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3bc0099c0 msgr2=0x7fd3d413fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3dadc2640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3bc0099c0 0x7fd3d413fe20 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0001780 tx=0x7fd3d000e6b0 comp rx=0 tx=0).stop 2024-09-17T05:03:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3dadc2640 1 -- 172.21.15.150:0/2085263761 shutdown_connections 2024-09-17T05:03:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3dadc2640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd3c0073200 0x7fd3c0075710 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3dadc2640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3d40c7a30 0x7fd3d40c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3dadc2640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd3d4142290 0x7fd3d413c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3dadc2640 1 --2- 172.21.15.150:0/2085263761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3bc0099c0 0x7fd3d413fe20 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3dadc2640 1 -- 172.21.15.150:0/2085263761 >> 172.21.15.150:0/2085263761 conn(0x7fd3d40ca460 msgr2=0x7fd3d414bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.349+0000 7fd3dadc2640 1 -- 172.21.15.150:0/2085263761 shutdown_connections 2024-09-17T05:03:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:06.353+0000 7fd3dadc2640 1 -- 172.21.15.150:0/2085263761 wait complete. 2024-09-17T05:03:06.367 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-17T05:03:06.367 INFO:tasks.mds_thrash.fs.[a]:waiting for 73 secs before thrashing 2024-09-17T05:03:31.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:03:31.621 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:03:52.133 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:03:52.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.316+0000 7fa84367b640 1 -- 172.21.15.150:0/3022753598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa828004810 msgr2=0x7fa828004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:52.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.316+0000 7fa84367b640 1 --2- 172.21.15.150:0/3022753598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa828004810 0x7fa828004c20 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fa82c009b30 tx=0x7fa82c01d230 comp rx=0 tx=0).stop 2024-09-17T05:03:52.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.316+0000 7fa84367b640 1 -- 172.21.15.150:0/3022753598 shutdown_connections 2024-09-17T05:03:52.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.316+0000 7fa84367b640 1 --2- 172.21.15.150:0/3022753598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa828004810 0x7fa828004c20 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.316+0000 7fa84367b640 1 -- 172.21.15.150:0/3022753598 >> 172.21.15.150:0/3022753598 conn(0x7fa83c0aaa80 msgr2=0x7fa83c0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:52.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa84367b640 1 -- 172.21.15.150:0/3022753598 shutdown_connections 2024-09-17T05:03:52.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa84367b640 1 -- 172.21.15.150:0/3022753598 wait complete. 2024-09-17T05:03:52.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa84367b640 1 Processor -- start 2024-09-17T05:03:52.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa84367b640 1 -- start start 2024-09-17T05:03:52.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa84367b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa828004810 0x7fa83c0c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:52.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa84367b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83c158950 0x7fa83c157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:52.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa84367b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa83c157690 0x7fa83c0ad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:52.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa84367b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c074830 con 0x7fa83c158950 2024-09-17T05:03:52.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa84367b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c072320 con 0x7fa83c157690 2024-09-17T05:03:52.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa84367b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa83c0749b0 con 0x7fa828004810 2024-09-17T05:03:52.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa840c16640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83c158950 0x7fa83c157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:52.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa840c16640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83c158950 0x7fa83c157150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43896/0 (socket says 172.21.15.150:43896) 2024-09-17T05:03:52.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa840c16640 1 -- 172.21.15.150:0/1668450328 learned_addr learned my addr 172.21.15.150:0/1668450328 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:03:52.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa841417640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa828004810 0x7fa83c0c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:52.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.320+0000 7fa841c18640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa83c157690 0x7fa83c0ad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:52.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa840c16640 1 -- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa828004810 msgr2=0x7fa83c0c22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:52.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa840c16640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa828004810 0x7fa83c0c22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa840c16640 1 -- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa83c157690 msgr2=0x7fa83c0ad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:52.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa840c16640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa83c157690 0x7fa83c0ad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa840c16640 1 -- 172.21.15.150:0/1668450328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83c150e80 con 0x7fa83c158950 2024-09-17T05:03:52.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa840c16640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83c158950 0x7fa83c157150 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa82c009b30 tx=0x7fa82c0acd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:52.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa831ffb640 1 -- 172.21.15.150:0/1668450328 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c0bc030 con 0x7fa83c158950 2024-09-17T05:03:52.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa831ffb640 1 -- 172.21.15.150:0/1668450328 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa82c0ac400 con 0x7fa83c158950 2024-09-17T05:03:52.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa84367b640 1 -- 172.21.15.150:0/1668450328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa83c158340 con 0x7fa83c158950 2024-09-17T05:03:52.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa831ffb640 1 -- 172.21.15.150:0/1668450328 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c0b1040 con 0x7fa83c158950 2024-09-17T05:03:52.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa84367b640 1 -- 172.21.15.150:0/1668450328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa83c157bb0 con 0x7fa83c158950 2024-09-17T05:03:52.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa84367b640 1 -- 172.21.15.150:0/1668450328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa83c06ce50 con 0x7fa83c158950 2024-09-17T05:03:52.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa831ffb640 1 -- 172.21.15.150:0/1668450328 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa82c0bf070 con 0x7fa83c158950 2024-09-17T05:03:52.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa831ffb640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa820073200 0x7fa820075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:52.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa831ffb640 1 -- 172.21.15.150:0/1668450328 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7fa82c12fe20 con 0x7fa83c158950 2024-09-17T05:03:52.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.324+0000 7fa841417640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa820073200 0x7fa820075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:52.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.328+0000 7fa841417640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa820073200 0x7fa820075710 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa8380018b0 tx=0x7fa838000ee0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:52.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.332+0000 7fa831ffb640 1 -- 172.21.15.150:0/1668450328 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa82c016f40 con 0x7fa83c158950 2024-09-17T05:03:52.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.572+0000 7fa84367b640 1 -- 172.21.15.150:0/1668450328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa83c154d20 con 0x7fa83c158950 2024-09-17T05:03:52.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.572+0000 7fa831ffb640 1 -- 172.21.15.150:0/1668450328 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+6537 (secure 0 0 0) 0x7fa82c0ac5d0 con 0x7fa83c158950 2024-09-17T05:03:52.570 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:03:52.570 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":101,"btime":"2024-09-17T05:03:06:161377+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:03:52.570 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 101 2024-09-17T05:03:52.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.572+0000 7fa84367b640 1 -- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa820073200 msgr2=0x7fa820075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:52.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.572+0000 7fa84367b640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa820073200 0x7fa820075710 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa8380018b0 tx=0x7fa838000ee0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.572+0000 7fa84367b640 1 -- 172.21.15.150:0/1668450328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83c158950 msgr2=0x7fa83c157150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:52.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.572+0000 7fa84367b640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83c158950 0x7fa83c157150 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa82c009b30 tx=0x7fa82c0acd20 comp rx=0 tx=0).stop 2024-09-17T05:03:52.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.576+0000 7fa84367b640 1 -- 172.21.15.150:0/1668450328 shutdown_connections 2024-09-17T05:03:52.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.576+0000 7fa84367b640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa820073200 0x7fa820075710 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.576+0000 7fa84367b640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa83c157690 0x7fa83c0ad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.576+0000 7fa84367b640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83c158950 0x7fa83c157150 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.576+0000 7fa84367b640 1 --2- 172.21.15.150:0/1668450328 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa828004810 0x7fa83c0c22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.576+0000 7fa84367b640 1 -- 172.21.15.150:0/1668450328 >> 172.21.15.150:0/1668450328 conn(0x7fa83c0aaa80 msgr2=0x7fa83c1e29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:52.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.576+0000 7fa84367b640 1 -- 172.21.15.150:0/1668450328 shutdown_connections 2024-09-17T05:03:52.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.576+0000 7fa84367b640 1 -- 172.21.15.150:0/1668450328 wait complete. 2024-09-17T05:03:52.594 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-17T05:03:52.594 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T05:03:52.595 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:03:52.608 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T05:03:52.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:03:52.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 -- 172.21.15.150:0/4247721891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0072ea0 msgr2=0x7fb0d00732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:52.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 --2- 172.21.15.150:0/4247721891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0072ea0 0x7fb0d00732b0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009900 tx=0x7fb0bc01c8e0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 -- 172.21.15.150:0/4247721891 shutdown_connections 2024-09-17T05:03:52.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 --2- 172.21.15.150:0/4247721891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0072ea0 0x7fb0d00732b0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 --2- 172.21.15.150:0/4247721891 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0d00c97a0 0x7fb0d0151c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 -- 172.21.15.150:0/4247721891 >> 172.21.15.150:0/4247721891 conn(0x7fb0d00abb00 msgr2=0x7fb0d00abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:52.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 -- 172.21.15.150:0/4247721891 shutdown_connections 2024-09-17T05:03:52.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 -- 172.21.15.150:0/4247721891 wait complete. 2024-09-17T05:03:52.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 Processor -- start 2024-09-17T05:03:52.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 -- start start 2024-09-17T05:03:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0072ea0 0x7fb0d015d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0d00c97a0 0x7fb0d01efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0d0157be0 0x7fb0d014dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d0161790 con 0x7fb0d0072ea0 2024-09-17T05:03:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d00c4df0 con 0x7fb0d00c97a0 2024-09-17T05:03:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.820+0000 7fb0d6f58640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d0161910 con 0x7fb0d0157be0 2024-09-17T05:03:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d4cf4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0072ea0 0x7fb0d015d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d4cf4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0072ea0 0x7fb0d015d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43920/0 (socket says 172.21.15.150:43920) 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d4cf4640 1 -- 172.21.15.150:0/1524436331 learned_addr learned my addr 172.21.15.150:0/1524436331 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d54f5640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0d0157be0 0x7fb0d014dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0cffff640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0d00c97a0 0x7fb0d01efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d4cf4640 1 -- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0d0157be0 msgr2=0x7fb0d014dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d4cf4640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0d0157be0 0x7fb0d014dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d4cf4640 1 -- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0d00c97a0 msgr2=0x7fb0d01efcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d4cf4640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0d00c97a0 0x7fb0d01efcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d4cf4640 1 -- 172.21.15.150:0/1524436331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d014ed50 con 0x7fb0d0072ea0 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d4cf4640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0072ea0 0x7fb0d015d2f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400d750 tx=0x7fb0c400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0cd7fa640 1 -- 172.21.15.150:0/1524436331 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4017070 con 0x7fb0d0072ea0 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0cd7fa640 1 -- 172.21.15.150:0/1524436331 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0c4009d20 con 0x7fb0d0072ea0 2024-09-17T05:03:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0cd7fa640 1 -- 172.21.15.150:0/1524436331 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4005280 con 0x7fb0d0072ea0 2024-09-17T05:03:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d6f58640 1 -- 172.21.15.150:0/1524436331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d014eae0 con 0x7fb0d0072ea0 2024-09-17T05:03:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d6f58640 1 -- 172.21.15.150:0/1524436331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d014e800 con 0x7fb0d0072ea0 2024-09-17T05:03:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0d6f58640 1 -- 172.21.15.150:0/1524436331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0d00732b0 con 0x7fb0d0072ea0 2024-09-17T05:03:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0cd7fa640 1 -- 172.21.15.150:0/1524436331 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb0c4005420 con 0x7fb0d0072ea0 2024-09-17T05:03:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0cd7fa640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb0b4073200 0x7fb0b4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0cd7fa640 1 -- 172.21.15.150:0/1524436331 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7fb0c4094be0 con 0x7fb0d0072ea0 2024-09-17T05:03:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.824+0000 7fb0cffff640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb0b4073200 0x7fb0b4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.828+0000 7fb0cffff640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb0b4073200 0x7fb0b4075710 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0af000 tx=0x7fb0bc01b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:52.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:52.832+0000 7fb0cd7fa640 1 -- 172.21.15.150:0/1524436331 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb0c400de60 con 0x7fb0d0072ea0 2024-09-17T05:03:53.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.088+0000 7fb0d6f58640 1 -- 172.21.15.150:0/1524436331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0d00c40f0 con 0x7fb0d0072ea0 2024-09-17T05:03:53.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0cd7fa640 1 -- 172.21.15.150:0/1524436331 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+6537 (secure 0 0 0) 0x7fb0c405f3b0 con 0x7fb0d0072ea0 2024-09-17T05:03:53.088 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:03:53.089 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":101,"btime":"2024-09-17T05:03:06:161377+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:03:53.089 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 101 2024-09-17T05:03:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0d6f58640 1 -- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb0b4073200 msgr2=0x7fb0b4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0d6f58640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb0b4073200 0x7fb0b4075710 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0af000 tx=0x7fb0bc01b120 comp rx=0 tx=0).stop 2024-09-17T05:03:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0d6f58640 1 -- 172.21.15.150:0/1524436331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0072ea0 msgr2=0x7fb0d015d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0d6f58640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0072ea0 0x7fb0d015d2f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400d750 tx=0x7fb0c400dc10 comp rx=0 tx=0).stop 2024-09-17T05:03:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0d6f58640 1 -- 172.21.15.150:0/1524436331 shutdown_connections 2024-09-17T05:03:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0d6f58640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb0b4073200 0x7fb0b4075710 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0d6f58640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb0d00c97a0 0x7fb0d01efcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0d6f58640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb0d0157be0 0x7fb0d014dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0d6f58640 1 --2- 172.21.15.150:0/1524436331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0d0072ea0 0x7fb0d015d2f0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0d6f58640 1 -- 172.21.15.150:0/1524436331 >> 172.21.15.150:0/1524436331 conn(0x7fb0d00abb00 msgr2=0x7fb0d015e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0d6f58640 1 -- 172.21.15.150:0/1524436331 shutdown_connections 2024-09-17T05:03:53.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:53.092+0000 7fb0d6f58640 1 -- 172.21.15.150:0/1524436331 wait complete. 2024-09-17T05:03:53.114 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:03:55.114 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:03:55.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.292+0000 7f2f0e498640 1 -- 172.21.15.150:0/88267999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef4004810 msgr2=0x7f2ef4004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:55.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.292+0000 7f2f0e498640 1 --2- 172.21.15.150:0/88267999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef4004810 0x7f2ef4004c20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f2f080c8d80 tx=0x7f2ef001cf00 comp rx=0 tx=0).stop 2024-09-17T05:03:55.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.292+0000 7f2f0e498640 1 -- 172.21.15.150:0/88267999 shutdown_connections 2024-09-17T05:03:55.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.292+0000 7f2f0e498640 1 --2- 172.21.15.150:0/88267999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef4004810 0x7f2ef4004c20 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:55.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.292+0000 7f2f0e498640 1 -- 172.21.15.150:0/88267999 >> 172.21.15.150:0/88267999 conn(0x7f2f080c8db0 msgr2=0x7f2f080c91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:55.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.292+0000 7f2f0e498640 1 -- 172.21.15.150:0/88267999 shutdown_connections 2024-09-17T05:03:55.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.292+0000 7f2f0e498640 1 -- 172.21.15.150:0/88267999 wait complete. 2024-09-17T05:03:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.292+0000 7f2f0e498640 1 Processor -- start 2024-09-17T05:03:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.292+0000 7f2f0e498640 1 -- start start 2024-09-17T05:03:55.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f0e498640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef4004810 0x7f2f0814ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f0e498640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f0814cb20 0x7f2f081f1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f0e498640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f081f1580 0x7f2f081e3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f0e498640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2f080abe60 con 0x7f2ef4004810 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f0e498640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2f080c5840 con 0x7f2f081f1580 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f0e498640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2f080acb70 con 0x7f2f0814cb20 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f07fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef4004810 0x7f2f0814ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f07fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef4004810 0x7f2f0814ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43940/0 (socket says 172.21.15.150:43940) 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f07fff640 1 -- 172.21.15.150:0/2294157968 learned_addr learned my addr 172.21.15.150:0/2294157968 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f0ca35640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f081f1580 0x7f2f081e3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2efffff640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f0814cb20 0x7f2f081f1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f07fff640 1 -- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f0814cb20 msgr2=0x7f2f081f1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f07fff640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f0814cb20 0x7f2f081f1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:55.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f07fff640 1 -- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f081f1580 msgr2=0x7f2f081e3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:55.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f07fff640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f081f1580 0x7f2f081e3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:55.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f07fff640 1 -- 172.21.15.150:0/2294157968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f080ab870 con 0x7f2ef4004810 2024-09-17T05:03:55.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f07fff640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef4004810 0x7f2f0814ab50 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8010700 tx=0x7f2ef8010bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:55.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f057fa640 1 -- 172.21.15.150:0/2294157968 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef800ead0 con 0x7f2ef4004810 2024-09-17T05:03:55.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f057fa640 1 -- 172.21.15.150:0/2294157968 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ef800ec70 con 0x7f2ef4004810 2024-09-17T05:03:55.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f0e498640 1 -- 172.21.15.150:0/2294157968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f0814b3c0 con 0x7f2ef4004810 2024-09-17T05:03:55.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f057fa640 1 -- 172.21.15.150:0/2294157968 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8005280 con 0x7f2ef4004810 2024-09-17T05:03:55.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f0e498640 1 -- 172.21.15.150:0/2294157968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f081f1af0 con 0x7f2ef4004810 2024-09-17T05:03:55.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.296+0000 7f2f0e498640 1 -- 172.21.15.150:0/2294157968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f0806ce50 con 0x7f2ef4004810 2024-09-17T05:03:55.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.300+0000 7f2f057fa640 1 -- 172.21.15.150:0/2294157968 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2ef8005420 con 0x7f2ef4004810 2024-09-17T05:03:55.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.300+0000 7f2f057fa640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2ee8073200 0x7f2ee8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:55.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.300+0000 7f2f057fa640 1 -- 172.21.15.150:0/2294157968 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f2ef8095c60 con 0x7f2ef4004810 2024-09-17T05:03:55.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.300+0000 7f2efffff640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2ee8073200 0x7f2ee8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:55.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.300+0000 7f2efffff640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2ee8073200 0x7f2ee8075710 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00ab000 tx=0x7f2ef0019d40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:55.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.304+0000 7f2f057fa640 1 -- 172.21.15.150:0/2294157968 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2f080ab870 con 0x7f2ef4004810 2024-09-17T05:03:55.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.548+0000 7f2f0e498640 1 -- 172.21.15.150:0/2294157968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f080c38b0 con 0x7f2ef4004810 2024-09-17T05:03:55.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.548+0000 7f2f057fa640 1 -- 172.21.15.150:0/2294157968 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+6537 (secure 0 0 0) 0x7f2ef800ee60 con 0x7f2ef4004810 2024-09-17T05:03:55.546 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:03:55.547 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":101,"btime":"2024-09-17T05:03:06:161377+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:03:55.547 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 101 2024-09-17T05:03:55.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.552+0000 7f2f0e498640 1 -- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2ee8073200 msgr2=0x7f2ee8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:55.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.552+0000 7f2f0e498640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2ee8073200 0x7f2ee8075710 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2ef00ab000 tx=0x7f2ef0019d40 comp rx=0 tx=0).stop 2024-09-17T05:03:55.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.552+0000 7f2f0e498640 1 -- 172.21.15.150:0/2294157968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef4004810 msgr2=0x7f2f0814ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:55.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.552+0000 7f2f0e498640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef4004810 0x7f2f0814ab50 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8010700 tx=0x7f2ef8010bc0 comp rx=0 tx=0).stop 2024-09-17T05:03:55.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.552+0000 7f2f0e498640 1 -- 172.21.15.150:0/2294157968 shutdown_connections 2024-09-17T05:03:55.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.552+0000 7f2f0e498640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2ee8073200 0x7f2ee8075710 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:55.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.552+0000 7f2f0e498640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2f081f1580 0x7f2f081e3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:55.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.552+0000 7f2f0e498640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2f0814cb20 0x7f2f081f1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:55.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.552+0000 7f2f0e498640 1 --2- 172.21.15.150:0/2294157968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ef4004810 0x7f2f0814ab50 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:55.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.552+0000 7f2f0e498640 1 -- 172.21.15.150:0/2294157968 >> 172.21.15.150:0/2294157968 conn(0x7f2f080c8db0 msgr2=0x7f2f080ad9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:55.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.552+0000 7f2f0e498640 1 -- 172.21.15.150:0/2294157968 shutdown_connections 2024-09-17T05:03:55.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:55.552+0000 7f2f0e498640 1 -- 172.21.15.150:0/2294157968 wait complete. 2024-09-17T05:03:55.570 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:03:57.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:03:57.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.756+0000 7fec6ed40640 1 -- 172.21.15.150:0/2972885125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec680c3660 msgr2=0x7fec6815dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:57.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.756+0000 7fec6ed40640 1 --2- 172.21.15.150:0/2972885125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec680c3660 0x7fec6815dfe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fec60009c80 tx=0x7fec6001f5a0 comp rx=0 tx=0).stop 2024-09-17T05:03:57.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.756+0000 7fec6ed40640 1 -- 172.21.15.150:0/2972885125 shutdown_connections 2024-09-17T05:03:57.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.756+0000 7fec6ed40640 1 --2- 172.21.15.150:0/2972885125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec680c3660 0x7fec6815dfe0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:57.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.756+0000 7fec6ed40640 1 --2- 172.21.15.150:0/2972885125 >> v2:172.21.15.176:3300/0 conn(0x7fec680a98a0 0x7fec680c3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:57.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.756+0000 7fec6ed40640 1 -- 172.21.15.150:0/2972885125 >> 172.21.15.150:0/2972885125 conn(0x7fec68070c20 msgr2=0x7fec68071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:57.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.756+0000 7fec6ed40640 1 -- 172.21.15.150:0/2972885125 shutdown_connections 2024-09-17T05:03:57.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.756+0000 7fec6ed40640 1 -- 172.21.15.150:0/2972885125 wait complete. 2024-09-17T05:03:57.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.756+0000 7fec6ed40640 1 Processor -- start 2024-09-17T05:03:57.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.756+0000 7fec6ed40640 1 -- start start 2024-09-17T05:03:57.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6ed40640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec680a98a0 0x7fec680c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:57.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6ed40640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec680c3660 0x7fec681f0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:57.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6ed40640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec68150d40 0x7fec681ed5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:57.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6ed40640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fec68073390 con 0x7fec680a98a0 2024-09-17T05:03:57.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6ed40640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fec68151210 con 0x7fec68150d40 2024-09-17T05:03:57.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6ed40640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fec6806e170 con 0x7fec680c3660 2024-09-17T05:03:57.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6cadc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec680a98a0 0x7fec680c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:57.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6cadc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec680a98a0 0x7fec680c1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43948/0 (socket says 172.21.15.150:43948) 2024-09-17T05:03:57.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6cadc640 1 -- 172.21.15.150:0/2573700770 learned_addr learned my addr 172.21.15.150:0/2573700770 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:03:57.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6d2dd640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec68150d40 0x7fec681ed5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:57.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec67fff640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec680c3660 0x7fec681f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:57.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6cadc640 1 -- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec680c3660 msgr2=0x7fec681f0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:57.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6cadc640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec680c3660 0x7fec681f0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:57.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6cadc640 1 -- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec68150d40 msgr2=0x7fec681ed5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:57.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6cadc640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec68150d40 0x7fec681ed5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:57.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6cadc640 1 -- 172.21.15.150:0/2573700770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec680ad390 con 0x7fec680a98a0 2024-09-17T05:03:57.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6cadc640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec680a98a0 0x7fec680c1d20 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00c3c0 tx=0x7fec5c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:57.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec657fa640 1 -- 172.21.15.150:0/2573700770 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c017070 con 0x7fec680a98a0 2024-09-17T05:03:57.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec657fa640 1 -- 172.21.15.150:0/2573700770 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec5c0053c0 con 0x7fec680a98a0 2024-09-17T05:03:57.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6ed40640 1 -- 172.21.15.150:0/2573700770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec680ad800 con 0x7fec680a98a0 2024-09-17T05:03:57.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec657fa640 1 -- 172.21.15.150:0/2573700770 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c003860 con 0x7fec680a98a0 2024-09-17T05:03:57.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6ed40640 1 -- 172.21.15.150:0/2573700770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec681ecf30 con 0x7fec680a98a0 2024-09-17T05:03:57.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.760+0000 7fec6ed40640 1 -- 172.21.15.150:0/2573700770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec680ae0d0 con 0x7fec680a98a0 2024-09-17T05:03:57.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.764+0000 7fec657fa640 1 -- 172.21.15.150:0/2573700770 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fec5c01a070 con 0x7fec680a98a0 2024-09-17T05:03:57.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.764+0000 7fec657fa640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec4c073200 0x7fec4c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:03:57.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.764+0000 7fec657fa640 1 -- 172.21.15.150:0/2573700770 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7fec5c093e10 con 0x7fec680a98a0 2024-09-17T05:03:57.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.764+0000 7fec67fff640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec4c073200 0x7fec4c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:03:57.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.764+0000 7fec67fff640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec4c073200 0x7fec4c075710 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fec68071cd0 tx=0x7fec540091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:03:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.768+0000 7fec657fa640 1 -- 172.21.15.150:0/2573700770 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fec5c05e5e0 con 0x7fec680a98a0 2024-09-17T05:03:57.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.988+0000 7fec6ed40640 1 -- 172.21.15.150:0/2573700770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec68074670 con 0x7fec680a98a0 2024-09-17T05:03:57.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.988+0000 7fec657fa640 1 -- 172.21.15.150:0/2573700770 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+6537 (secure 0 0 0) 0x7fec5c05dee0 con 0x7fec680a98a0 2024-09-17T05:03:57.986 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:03:57.986 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":101,"btime":"2024-09-17T05:03:06:161377+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:03:57.986 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 101 2024-09-17T05:03:57.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.988+0000 7fec6ed40640 1 -- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec4c073200 msgr2=0x7fec4c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:57.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.988+0000 7fec6ed40640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec4c073200 0x7fec4c075710 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fec68071cd0 tx=0x7fec540091e0 comp rx=0 tx=0).stop 2024-09-17T05:03:57.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.988+0000 7fec6ed40640 1 -- 172.21.15.150:0/2573700770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec680a98a0 msgr2=0x7fec680c1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:03:57.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.988+0000 7fec6ed40640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec680a98a0 0x7fec680c1d20 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00c3c0 tx=0x7fec5c00c880 comp rx=0 tx=0).stop 2024-09-17T05:03:57.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.988+0000 7fec6ed40640 1 -- 172.21.15.150:0/2573700770 shutdown_connections 2024-09-17T05:03:57.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.988+0000 7fec6ed40640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fec4c073200 0x7fec4c075710 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:57.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.988+0000 7fec6ed40640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec68150d40 0x7fec681ed5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:57.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.992+0000 7fec6ed40640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fec680c3660 0x7fec681f0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:57.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.992+0000 7fec6ed40640 1 --2- 172.21.15.150:0/2573700770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec680a98a0 0x7fec680c1d20 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:03:57.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.992+0000 7fec6ed40640 1 -- 172.21.15.150:0/2573700770 >> 172.21.15.150:0/2573700770 conn(0x7fec68070c20 msgr2=0x7fec6814a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:03:57.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.992+0000 7fec6ed40640 1 -- 172.21.15.150:0/2573700770 shutdown_connections 2024-09-17T05:03:57.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:03:57.992+0000 7fec6ed40640 1 -- 172.21.15.150:0/2573700770 wait complete. 2024-09-17T05:03:58.011 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:04:00.012 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.188+0000 7f8b89ad8640 1 -- 172.21.15.150:0/12395905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b6c004f50 msgr2=0x7f8b6c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.188+0000 7f8b89ad8640 1 --2- 172.21.15.150:0/12395905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b6c004f50 0x7f8b6c00b9d0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f8b840c4640 tx=0x7f8b7c01d280 comp rx=0 tx=0).stop 2024-09-17T05:04:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.188+0000 7f8b83fff640 1 -- 172.21.15.150:0/12395905 reap_dead start 2024-09-17T05:04:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.188+0000 7f8b89ad8640 1 -- 172.21.15.150:0/12395905 shutdown_connections 2024-09-17T05:04:00.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.188+0000 7f8b89ad8640 1 -- 172.21.15.150:0/12395905 >> 172.21.15.150:0/12395905 conn(0x7f8b84070b50 msgr2=0x7f8b84070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:00.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.188+0000 7f8b89ad8640 1 -- 172.21.15.150:0/12395905 shutdown_connections 2024-09-17T05:04:00.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.188+0000 7f8b89ad8640 1 -- 172.21.15.150:0/12395905 wait complete. 2024-09-17T05:04:00.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.192+0000 7f8b89ad8640 1 Processor -- start 2024-09-17T05:04:00.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.192+0000 7f8b89ad8640 1 -- start start 2024-09-17T05:04:00.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.192+0000 7f8b89ad8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b840c24a0 0x7f8b841472b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:00.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b89ad8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b841e3310 0x7f8b841e3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b89ad8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b841f37b0 0x7f8b841f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b89ad8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8406e7f0 con 0x7f8b841f37b0 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b89ad8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8406e670 con 0x7f8b841e3310 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b89ad8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8b84069bf0 con 0x7f8b840c24a0 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b83fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b841f37b0 0x7f8b841f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b83fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b841f37b0 0x7f8b841f3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37978/0 (socket says 172.21.15.150:37978) 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b83fff640 1 -- 172.21.15.150:0/4227439331 learned_addr learned my addr 172.21.15.150:0/4227439331 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b837fe640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b840c24a0 0x7f8b841472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b82ffd640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b841e3310 0x7f8b841e3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b83fff640 1 -- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b840c24a0 msgr2=0x7f8b841472b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b83fff640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b840c24a0 0x7f8b841472b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b83fff640 1 -- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b841e3310 msgr2=0x7f8b841e3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b83fff640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b841e3310 0x7f8b841e3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b83fff640 1 -- 172.21.15.150:0/4227439331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b841e1120 con 0x7f8b841f37b0 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b82ffd640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b841e3310 0x7f8b841e3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b83fff640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b841f37b0 0x7f8b841f3bc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c098c30 tx=0x7f8b7c00c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b6bfff640 1 -- 172.21.15.150:0/4227439331 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c001410 con 0x7f8b841f37b0 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b6bfff640 1 -- 172.21.15.150:0/4227439331 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b7c003a70 con 0x7f8b841f37b0 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b89ad8640 1 -- 172.21.15.150:0/4227439331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b84150690 con 0x7f8b841f37b0 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b6bfff640 1 -- 172.21.15.150:0/4227439331 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c003e60 con 0x7f8b841f37b0 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b89ad8640 1 -- 172.21.15.150:0/4227439331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b84151070 con 0x7f8b841f37b0 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b89ad8640 1 -- 172.21.15.150:0/4227439331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b841e1320 con 0x7f8b841f37b0 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b6bfff640 1 -- 172.21.15.150:0/4227439331 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8b7c006070 con 0x7f8b841f37b0 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b6bfff640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b70073200 0x7f8b70075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b6bfff640 1 -- 172.21.15.150:0/4227439331 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f8b7c12f080 con 0x7f8b841f37b0 2024-09-17T05:04:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.196+0000 7f8b837fe640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b70073200 0x7f8b70075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:00.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.200+0000 7f8b837fe640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b70073200 0x7f8b70075710 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8b74001100 tx=0x7f8b74001090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:00.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.204+0000 7f8b6bfff640 1 -- 172.21.15.150:0/4227439331 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8b7c017030 con 0x7f8b841f37b0 2024-09-17T05:04:00.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.448+0000 7f8b89ad8640 1 -- 172.21.15.150:0/4227439331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b8406d820 con 0x7f8b841f37b0 2024-09-17T05:04:00.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.448+0000 7f8b6bfff640 1 -- 172.21.15.150:0/4227439331 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+6537 (secure 0 0 0) 0x7f8b7c0f9880 con 0x7f8b841f37b0 2024-09-17T05:04:00.445 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:00.446 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":101,"btime":"2024-09-17T05:03:06:161377+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:04:00.446 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 101 2024-09-17T05:04:00.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.448+0000 7f8b89ad8640 1 -- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b70073200 msgr2=0x7f8b70075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:00.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.448+0000 7f8b89ad8640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b70073200 0x7f8b70075710 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8b74001100 tx=0x7f8b74001090 comp rx=0 tx=0).stop 2024-09-17T05:04:00.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.448+0000 7f8b89ad8640 1 -- 172.21.15.150:0/4227439331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b841f37b0 msgr2=0x7f8b841f3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:00.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.448+0000 7f8b89ad8640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b841f37b0 0x7f8b841f3bc0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c098c30 tx=0x7f8b7c00c260 comp rx=0 tx=0).stop 2024-09-17T05:04:00.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.448+0000 7f8b89ad8640 1 -- 172.21.15.150:0/4227439331 shutdown_connections 2024-09-17T05:04:00.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.448+0000 7f8b89ad8640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b70073200 0x7f8b70075710 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:00.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.448+0000 7f8b89ad8640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b841e3310 0x7f8b841e3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:00.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.448+0000 7f8b89ad8640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b841f37b0 0x7f8b841f3bc0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:00.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.448+0000 7f8b89ad8640 1 --2- 172.21.15.150:0/4227439331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b840c24a0 0x7f8b841472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:00.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.448+0000 7f8b89ad8640 1 -- 172.21.15.150:0/4227439331 >> 172.21.15.150:0/4227439331 conn(0x7f8b84070b50 msgr2=0x7f8b84159490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.452+0000 7f8b89ad8640 1 -- 172.21.15.150:0/4227439331 shutdown_connections 2024-09-17T05:04:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:00.452+0000 7f8b89ad8640 1 -- 172.21.15.150:0/4227439331 wait complete. 2024-09-17T05:04:00.477 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:04:02.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:04:02.044 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:04:02.479 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:02.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.652+0000 7f8973c56640 1 -- 172.21.15.150:0/3924150167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89580039c0 msgr2=0x7f8958003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:02.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.652+0000 7f8973c56640 1 --2- 172.21.15.150:0/3924150167 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89580039c0 0x7f8958003e30 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8968009b40 tx=0x7f896801d270 comp rx=0 tx=0).stop 2024-09-17T05:04:02.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.652+0000 7f89721f3640 1 -- 172.21.15.150:0/3924150167 reap_dead start 2024-09-17T05:04:02.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.652+0000 7f8973c56640 1 -- 172.21.15.150:0/3924150167 shutdown_connections 2024-09-17T05:04:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.652+0000 7f8973c56640 1 -- 172.21.15.150:0/3924150167 >> 172.21.15.150:0/3924150167 conn(0x7f896c0abaa0 msgr2=0x7f896c0abeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.652+0000 7f8973c56640 1 -- 172.21.15.150:0/3924150167 shutdown_connections 2024-09-17T05:04:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.652+0000 7f8973c56640 1 -- 172.21.15.150:0/3924150167 wait complete. 2024-09-17T05:04:02.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.656+0000 7f8973c56640 1 Processor -- start 2024-09-17T05:04:02.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.656+0000 7f8973c56640 1 -- start start 2024-09-17T05:04:02.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.656+0000 7f8973c56640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89580039c0 0x7f896c1534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:02.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.656+0000 7f8973c56640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f896c152e70 0x7f896c0c3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:02.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.656+0000 7f8973c56640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f896c06e830 0x7f896c139060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:02.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.656+0000 7f8973c56640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f896c15f100 con 0x7f896c152e70 2024-09-17T05:04:02.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.656+0000 7f8973c56640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f896c15ef80 con 0x7f896c06e830 2024-09-17T05:04:02.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.656+0000 7f8973c56640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f896c15f280 con 0x7f89580039c0 2024-09-17T05:04:02.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.656+0000 7f89711f1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f896c152e70 0x7f896c0c3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:02.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89711f1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f896c152e70 0x7f896c0c3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37982/0 (socket says 172.21.15.150:37982) 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89711f1640 1 -- 172.21.15.150:0/3697269397 learned_addr learned my addr 172.21.15.150:0/3697269397 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89719f2640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89580039c0 0x7f896c1534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89721f3640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f896c06e830 0x7f896c139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89711f1640 1 -- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89580039c0 msgr2=0x7f896c1534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89711f1640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89580039c0 0x7f896c1534f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89711f1640 1 -- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f896c06e830 msgr2=0x7f896c139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89711f1640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f896c06e830 0x7f896c139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89711f1640 1 -- 172.21.15.150:0/3697269397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896c06fe90 con 0x7f896c152e70 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89719f2640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89580039c0 0x7f896c1534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89711f1640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f896c152e70 0x7f896c0c3410 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f895c00d730 tx=0x7f895c00dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89627fc640 1 -- 172.21.15.150:0/3697269397 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c019070 con 0x7f896c152e70 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89627fc640 1 -- 172.21.15.150:0/3697269397 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f895c00de40 con 0x7f896c152e70 2024-09-17T05:04:02.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f8973c56640 1 -- 172.21.15.150:0/3697269397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896c155030 con 0x7f896c152e70 2024-09-17T05:04:02.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89627fc640 1 -- 172.21.15.150:0/3697269397 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c00b6c0 con 0x7f896c152e70 2024-09-17T05:04:02.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f8973c56640 1 -- 172.21.15.150:0/3697269397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f896c06fbb0 con 0x7f896c152e70 2024-09-17T05:04:02.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f8973c56640 1 -- 172.21.15.150:0/3697269397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f896c0c7e80 con 0x7f896c152e70 2024-09-17T05:04:02.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.660+0000 7f89627fc640 1 -- 172.21.15.150:0/3697269397 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f895c01c070 con 0x7f896c152e70 2024-09-17T05:04:02.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.664+0000 7f89627fc640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8950073200 0x7f8950075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:02.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.664+0000 7f89627fc640 1 -- 172.21.15.150:0/3697269397 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f895c0947c0 con 0x7f896c152e70 2024-09-17T05:04:02.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.664+0000 7f89719f2640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8950073200 0x7f8950075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:02.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.664+0000 7f89719f2640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8950073200 0x7f8950075710 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f89680ad000 tx=0x7f896801a010 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:02.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.668+0000 7f89627fc640 1 -- 172.21.15.150:0/3697269397 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f895c023020 con 0x7f896c152e70 2024-09-17T05:04:02.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.908+0000 7f8973c56640 1 -- 172.21.15.150:0/3697269397 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f896c139c00 con 0x7f896c152e70 2024-09-17T05:04:02.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f89627fc640 1 -- 172.21.15.150:0/3697269397 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+6537 (secure 0 0 0) 0x7f895c05ef70 con 0x7f896c152e70 2024-09-17T05:04:02.908 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:02.909 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":101,"btime":"2024-09-17T05:03:06:161377+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:04:02.909 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 101 2024-09-17T05:04:02.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f8973c56640 1 -- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8950073200 msgr2=0x7f8950075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:02.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f8973c56640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8950073200 0x7f8950075710 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f89680ad000 tx=0x7f896801a010 comp rx=0 tx=0).stop 2024-09-17T05:04:02.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f8973c56640 1 -- 172.21.15.150:0/3697269397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f896c152e70 msgr2=0x7f896c0c3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:02.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f8973c56640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f896c152e70 0x7f896c0c3410 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f895c00d730 tx=0x7f895c00dbf0 comp rx=0 tx=0).stop 2024-09-17T05:04:02.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f8973c56640 1 -- 172.21.15.150:0/3697269397 shutdown_connections 2024-09-17T05:04:02.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f8973c56640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8950073200 0x7f8950075710 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:02.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f8973c56640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f896c06e830 0x7f896c139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:02.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f8973c56640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f896c152e70 0x7f896c0c3410 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:02.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f8973c56640 1 --2- 172.21.15.150:0/3697269397 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89580039c0 0x7f896c1534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:02.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f8973c56640 1 -- 172.21.15.150:0/3697269397 >> 172.21.15.150:0/3697269397 conn(0x7f896c0abaa0 msgr2=0x7f896c0abeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:02.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f8973c56640 1 -- 172.21.15.150:0/3697269397 shutdown_connections 2024-09-17T05:04:02.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:02.912+0000 7f8973c56640 1 -- 172.21.15.150:0/3697269397 wait complete. 2024-09-17T05:04:02.936 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:04:04.937 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:05.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 -- 172.21.15.150:0/2671944136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5280c3d10 msgr2=0x7fa52815df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:05.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 --2- 172.21.15.150:0/2671944136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5280c3d10 0x7fa52815df70 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa52000be10 tx=0x7fa52001e9b0 comp rx=0 tx=0).stop 2024-09-17T05:04:05.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 -- 172.21.15.150:0/2671944136 shutdown_connections 2024-09-17T05:04:05.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 --2- 172.21.15.150:0/2671944136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5280c3d10 0x7fa52815df70 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:05.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 --2- 172.21.15.150:0/2671944136 >> v2:172.21.15.176:3300/0 conn(0x7fa5280c4250 0x7fa528153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:05.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 -- 172.21.15.150:0/2671944136 >> 172.21.15.150:0/2671944136 conn(0x7fa52806e360 msgr2=0x7fa52806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:05.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 -- 172.21.15.150:0/2671944136 shutdown_connections 2024-09-17T05:04:05.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 -- 172.21.15.150:0/2671944136 wait complete. 2024-09-17T05:04:05.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 Processor -- start 2024-09-17T05:04:05.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 -- start start 2024-09-17T05:04:05.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5280c3900 0x7fa5280ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:05.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5280c3d10 0x7fa5281f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:05.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280c4250 0x7fa5281f0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:05.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa528071120 con 0x7fa5280c3d10 2024-09-17T05:04:05.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa528070fa0 con 0x7fa5280c4250 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa52f1ab640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa528072db0 con 0x7fa5280c3900 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa51ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5280c3d10 0x7fa5281f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa51ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5280c3d10 0x7fa5281f2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38004/0 (socket says 172.21.15.150:38004) 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.111+0000 7fa51ffff640 1 -- 172.21.15.150:0/1604145368 learned_addr learned my addr 172.21.15.150:0/1604145368 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa52d748640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280c4250 0x7fa5281f0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa52cf47640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5280c3900 0x7fa5280ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa51ffff640 1 -- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5280c3900 msgr2=0x7fa5280ad550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa51ffff640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5280c3900 0x7fa5280ad550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa51ffff640 1 -- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280c4250 msgr2=0x7fa5281f0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa51ffff640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280c4250 0x7fa5281f0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa51ffff640 1 -- 172.21.15.150:0/1604145368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5281f38b0 con 0x7fa5280c3d10 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa52cf47640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5280c3900 0x7fa5280ad550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa52d748640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280c4250 0x7fa5281f0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:04:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa51ffff640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5280c3d10 0x7fa5281f2520 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fa51800c3c0 tx=0x7fa51800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:05.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa51d7fa640 1 -- 172.21.15.150:0/1604145368 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518017070 con 0x7fa5280c3d10 2024-09-17T05:04:05.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa52f1ab640 1 -- 172.21.15.150:0/1604145368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5280ac3e0 con 0x7fa5280c3d10 2024-09-17T05:04:05.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa51d7fa640 1 -- 172.21.15.150:0/1604145368 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5180053c0 con 0x7fa5280c3d10 2024-09-17T05:04:05.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa51d7fa640 1 -- 172.21.15.150:0/1604145368 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518003840 con 0x7fa5280c3d10 2024-09-17T05:04:05.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa52f1ab640 1 -- 172.21.15.150:0/1604145368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5281f2a60 con 0x7fa5280c3d10 2024-09-17T05:04:05.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa52f1ab640 1 -- 172.21.15.150:0/1604145368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa528153ee0 con 0x7fa5280c3d10 2024-09-17T05:04:05.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa51d7fa640 1 -- 172.21.15.150:0/1604145368 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa51801a070 con 0x7fa5280c3d10 2024-09-17T05:04:05.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa51d7fa640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa514073230 0x7fa514075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:05.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa51d7fa640 1 -- 172.21.15.150:0/1604145368 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7fa518093dc0 con 0x7fa5280c3d10 2024-09-17T05:04:05.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.115+0000 7fa52cf47640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa514073230 0x7fa514075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:05.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.119+0000 7fa52cf47640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa514073230 0x7fa514075740 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa5240058d0 tx=0x7fa524005e30 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:05.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.123+0000 7fa51d7fa640 1 -- 172.21.15.150:0/1604145368 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa51805e510 con 0x7fa5280c3d10 2024-09-17T05:04:05.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 -- 172.21.15.150:0/1604145368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5280aa590 con 0x7fa5280c3d10 2024-09-17T05:04:05.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa51d7fa640 1 -- 172.21.15.150:0/1604145368 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+6537 (secure 0 0 0) 0x7fa51805de10 con 0x7fa5280c3d10 2024-09-17T05:04:05.393 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:05.393 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":101,"btime":"2024-09-17T05:03:06:161377+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:02:34.022739+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6342},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6342":{"gid":6342,"name":"a","rank":0,"incarnation":92,"state":"up:active","state_seq":28,"addr":"172.21.15.150:6833/233311351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":233311351},{"type":"v1","addr":"172.21.15.150:6833","nonce":233311351}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6342,"qdb_cluster":[6342]},"id":2}]} 2024-09-17T05:04:05.394 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 101 2024-09-17T05:04:05.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 -- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa514073230 msgr2=0x7fa514075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:05.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa514073230 0x7fa514075740 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa5240058d0 tx=0x7fa524005e30 comp rx=0 tx=0).stop 2024-09-17T05:04:05.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 -- 172.21.15.150:0/1604145368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5280c3d10 msgr2=0x7fa5281f2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:05.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5280c3d10 0x7fa5281f2520 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fa51800c3c0 tx=0x7fa51800c880 comp rx=0 tx=0).stop 2024-09-17T05:04:05.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 -- 172.21.15.150:0/1604145368 shutdown_connections 2024-09-17T05:04:05.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa514073230 0x7fa514075740 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:05.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa5280c4250 0x7fa5281f0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:05.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5280c3d10 0x7fa5281f2520 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:05.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 --2- 172.21.15.150:0/1604145368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa5280c3900 0x7fa5280ad550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:05.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 -- 172.21.15.150:0/1604145368 >> 172.21.15.150:0/1604145368 conn(0x7fa52806e360 msgr2=0x7fa528151460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:05.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 -- 172.21.15.150:0/1604145368 shutdown_connections 2024-09-17T05:04:05.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:05.395+0000 7fa52f1ab640 1 -- 172.21.15.150:0/1604145368 wait complete. 2024-09-17T05:04:05.416 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:04:07.417 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:07.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.603+0000 7fa30eaed640 1 -- 172.21.15.150:0/2455627874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f0005e60 msgr2=0x7fa2f0006310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:07.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.603+0000 7fa30eaed640 1 --2- 172.21.15.150:0/2455627874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f0005e60 0x7fa2f0006310 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fa3000013e0 tx=0x7fa30001fa00 comp rx=0 tx=0).stop 2024-09-17T05:04:07.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.603+0000 7fa30d08a640 1 -- 172.21.15.150:0/2455627874 reap_dead start 2024-09-17T05:04:07.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.603+0000 7fa30eaed640 1 -- 172.21.15.150:0/2455627874 shutdown_connections 2024-09-17T05:04:07.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.603+0000 7fa30eaed640 1 -- 172.21.15.150:0/2455627874 >> 172.21.15.150:0/2455627874 conn(0x7fa3080c5ae0 msgr2=0x7fa308073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:07.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.603+0000 7fa30eaed640 1 -- 172.21.15.150:0/2455627874 shutdown_connections 2024-09-17T05:04:07.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.603+0000 7fa30eaed640 1 -- 172.21.15.150:0/2455627874 wait complete. 2024-09-17T05:04:07.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30eaed640 1 Processor -- start 2024-09-17T05:04:07.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30eaed640 1 -- start start 2024-09-17T05:04:07.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30eaed640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa308149bb0 0x7fa3081f0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:07.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30eaed640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa308145500 0x7fa308154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30eaed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3080c1c20 0x7fa3081443d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30eaed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa3080ab150 con 0x7fa3080c1c20 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30eaed640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa3080c37b0 con 0x7fa308149bb0 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30eaed640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa3080ab2d0 con 0x7fa308145500 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30d08a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3080c1c20 0x7fa3081443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30d08a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3080c1c20 0x7fa3081443d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38028/0 (socket says 172.21.15.150:38028) 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30d08a640 1 -- 172.21.15.150:0/3464354803 learned_addr learned my addr 172.21.15.150:0/3464354803 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa2fffff640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa308145500 0x7fa308154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30c889640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa308149bb0 0x7fa3081f0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30d08a640 1 -- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa308145500 msgr2=0x7fa308154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30d08a640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa308145500 0x7fa308154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30d08a640 1 -- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa308149bb0 msgr2=0x7fa3081f0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:07.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30d08a640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa308149bb0 0x7fa3081f0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:07.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30d08a640 1 -- 172.21.15.150:0/3464354803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa308145910 con 0x7fa3080c1c20 2024-09-17T05:04:07.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.607+0000 7fa30d08a640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3080c1c20 0x7fa3081443d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fa30001e100 tx=0x7fa30000c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:07.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.611+0000 7fa2fd7fa640 1 -- 172.21.15.150:0/3464354803 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa300004b50 con 0x7fa3080c1c20 2024-09-17T05:04:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.611+0000 7fa30eaed640 1 -- 172.21.15.150:0/3464354803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa308145220 con 0x7fa3080c1c20 2024-09-17T05:04:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.611+0000 7fa2fd7fa640 1 -- 172.21.15.150:0/3464354803 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa300004de0 con 0x7fa3080c1c20 2024-09-17T05:04:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.611+0000 7fa2fd7fa640 1 -- 172.21.15.150:0/3464354803 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3000a9810 con 0x7fa3080c1c20 2024-09-17T05:04:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.611+0000 7fa30eaed640 1 -- 172.21.15.150:0/3464354803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa308154800 con 0x7fa3080c1c20 2024-09-17T05:04:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.611+0000 7fa30eaed640 1 -- 172.21.15.150:0/3464354803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa308156e40 con 0x7fa3080c1c20 2024-09-17T05:04:07.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.611+0000 7fa2fd7fa640 1 -- 172.21.15.150:0/3464354803 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa3000a99b0 con 0x7fa3080c1c20 2024-09-17T05:04:07.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.611+0000 7fa2fd7fa640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa2f4073200 0x7fa2f4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:07.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.611+0000 7fa2fd7fa640 1 -- 172.21.15.150:0/3464354803 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7fa300131fa0 con 0x7fa3080c1c20 2024-09-17T05:04:07.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.611+0000 7fa30c889640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa2f4073200 0x7fa2f4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:07.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.615+0000 7fa30c889640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa2f4073200 0x7fa2f4075710 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa3080c76a0 tx=0x7fa3040091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:07.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.619+0000 7fa2fd7fa640 1 -- 172.21.15.150:0/3464354803 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa3000b8070 con 0x7fa3080c1c20 2024-09-17T05:04:07.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.887+0000 7fa30eaed640 1 -- 172.21.15.150:0/3464354803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3081539e0 con 0x7fa3080c1c20 2024-09-17T05:04:07.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.887+0000 7fa2fd7fa640 1 -- 172.21.15.150:0/3464354803 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+5709 (secure 0 0 0) 0x7fa308153cd0 con 0x7fa3080c1c20 2024-09-17T05:04:07.887 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:07.887 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":103,"btime":"2024-09-17T05:04:07:488011+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:06.500799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:04:07.887 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 103 2024-09-17T05:04:07.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.891+0000 7fa30eaed640 1 -- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa2f4073200 msgr2=0x7fa2f4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:07.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.891+0000 7fa30eaed640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa2f4073200 0x7fa2f4075710 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa3080c76a0 tx=0x7fa3040091e0 comp rx=0 tx=0).stop 2024-09-17T05:04:07.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.891+0000 7fa30eaed640 1 -- 172.21.15.150:0/3464354803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3080c1c20 msgr2=0x7fa3081443d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:07.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.891+0000 7fa30eaed640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3080c1c20 0x7fa3081443d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fa30001e100 tx=0x7fa30000c450 comp rx=0 tx=0).stop 2024-09-17T05:04:07.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.891+0000 7fa30eaed640 1 -- 172.21.15.150:0/3464354803 shutdown_connections 2024-09-17T05:04:07.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.891+0000 7fa30eaed640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa2f4073200 0x7fa2f4075710 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:07.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.891+0000 7fa30eaed640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3080c1c20 0x7fa3081443d0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:07.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.891+0000 7fa30eaed640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa308145500 0x7fa308154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:07.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.891+0000 7fa30eaed640 1 --2- 172.21.15.150:0/3464354803 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa308149bb0 0x7fa3081f0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:07.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.891+0000 7fa30eaed640 1 -- 172.21.15.150:0/3464354803 >> 172.21.15.150:0/3464354803 conn(0x7fa3080c5ae0 msgr2=0x7fa308149290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:07.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.891+0000 7fa30eaed640 1 -- 172.21.15.150:0/3464354803 shutdown_connections 2024-09-17T05:04:07.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:07.891+0000 7fa30eaed640 1 -- 172.21.15.150:0/3464354803 wait complete. 2024-09-17T05:04:07.909 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-17T05:04:07.910 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:04:07.910 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:08.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.071+0000 7fecc383f640 1 -- 172.21.15.150:0/1407058702 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fecbc1568a0 msgr2=0x7fecbc0c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:08.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.071+0000 7fecc383f640 1 --2- 172.21.15.150:0/1407058702 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fecbc1568a0 0x7fecbc0c5810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fecb8009900 tx=0x7fecb801c8b0 comp rx=0 tx=0).stop 2024-09-17T05:04:08.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.071+0000 7fecc383f640 1 -- 172.21.15.150:0/1407058702 shutdown_connections 2024-09-17T05:04:08.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.071+0000 7fecc383f640 1 --2- 172.21.15.150:0/1407058702 >> v2:172.21.15.176:3300/0 conn(0x7fecbc0c20a0 0x7fecbc0c24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:08.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.071+0000 7fecc383f640 1 --2- 172.21.15.150:0/1407058702 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fecbc1568a0 0x7fecbc0c5810 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:08.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.071+0000 7fecc383f640 1 -- 172.21.15.150:0/1407058702 >> 172.21.15.150:0/1407058702 conn(0x7fecbc074b80 msgr2=0x7fecbc06f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:08.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.071+0000 7fecc383f640 1 -- 172.21.15.150:0/1407058702 shutdown_connections 2024-09-17T05:04:08.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.071+0000 7fecc383f640 1 -- 172.21.15.150:0/1407058702 wait complete. 2024-09-17T05:04:08.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc383f640 1 Processor -- start 2024-09-17T05:04:08.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc383f640 1 -- start start 2024-09-17T05:04:08.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc383f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecbc0c20a0 0x7fecbc152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:08.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc383f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fecbc0c35b0 0x7fecbc0c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:08.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc383f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fecbc1568a0 0x7fecbc153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:08.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc383f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc0c9310 con 0x7fecbc0c20a0 2024-09-17T05:04:08.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc383f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc0c9190 con 0x7fecbc0c35b0 2024-09-17T05:04:08.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc383f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fecbc0c39c0 con 0x7fecbc1568a0 2024-09-17T05:04:08.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc15db640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecbc0c20a0 0x7fecbc152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:08.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc15db640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecbc0c20a0 0x7fecbc152600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38038/0 (socket says 172.21.15.150:38038) 2024-09-17T05:04:08.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc15db640 1 -- 172.21.15.150:0/2874199226 learned_addr learned my addr 172.21.15.150:0/2874199226 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc1ddc640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fecbc1568a0 0x7fecbc153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc0dda640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fecbc0c35b0 0x7fecbc0c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc15db640 1 -- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fecbc1568a0 msgr2=0x7fecbc153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc15db640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fecbc1568a0 0x7fecbc153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc15db640 1 -- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fecbc0c35b0 msgr2=0x7fecbc0c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc15db640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fecbc0c35b0 0x7fecbc0c7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc15db640 1 -- 172.21.15.150:0/2874199226 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecbc151d00 con 0x7fecbc0c20a0 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc0dda640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fecbc0c35b0 0x7fecbc0c7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc15db640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecbc0c20a0 0x7fecbc152600 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fecb80b0040 tx=0x7fecb8005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecb1ffb640 1 -- 172.21.15.150:0/2874199226 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb80b2040 con 0x7fecbc0c20a0 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc383f640 1 -- 172.21.15.150:0/2874199226 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecbc152b40 con 0x7fecbc0c20a0 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecb1ffb640 1 -- 172.21.15.150:0/2874199226 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fecb8005310 con 0x7fecbc0c20a0 2024-09-17T05:04:08.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecb1ffb640 1 -- 172.21.15.150:0/2874199226 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb80046d0 con 0x7fecbc0c20a0 2024-09-17T05:04:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc383f640 1 -- 172.21.15.150:0/2874199226 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecbc151740 con 0x7fecbc0c20a0 2024-09-17T05:04:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.075+0000 7fecc383f640 1 -- 172.21.15.150:0/2874199226 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecbc0c8880 con 0x7fecbc0c20a0 2024-09-17T05:04:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.079+0000 7fecb1ffb640 1 -- 172.21.15.150:0/2874199226 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fecb8004890 con 0x7fecbc0c20a0 2024-09-17T05:04:08.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.079+0000 7fecb1ffb640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feca0073200 0x7feca0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:08.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.079+0000 7fecb1ffb640 1 -- 172.21.15.150:0/2874199226 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7fecb80bd070 con 0x7fecbc0c20a0 2024-09-17T05:04:08.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.079+0000 7fecc0dda640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feca0073200 0x7feca0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:08.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.079+0000 7fecc0dda640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feca0073200 0x7feca0075710 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fecbc14c1d0 tx=0x7fecac009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:08.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.083+0000 7fecb1ffb640 1 -- 172.21.15.150:0/2874199226 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fecb80164d0 con 0x7fecbc0c20a0 2024-09-17T05:04:08.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.351+0000 7fecc383f640 1 -- 172.21.15.150:0/2874199226 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fecbc151530 con 0x7fecbc0c20a0 2024-09-17T05:04:08.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.351+0000 7fecb1ffb640 1 -- 172.21.15.150:0/2874199226 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) ==== 95+0+5709 (secure 0 0 0) 0x7fecb80f9600 con 0x7fecbc0c20a0 2024-09-17T05:04:08.352 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:08.352 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":103,"btime":"2024-09-17T05:04:07:488011+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:06.500799+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:04:08.353 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 103 2024-09-17T05:04:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.355+0000 7fecc383f640 1 -- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feca0073200 msgr2=0x7feca0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.355+0000 7fecc383f640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feca0073200 0x7feca0075710 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fecbc14c1d0 tx=0x7fecac009040 comp rx=0 tx=0).stop 2024-09-17T05:04:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.355+0000 7fecc383f640 1 -- 172.21.15.150:0/2874199226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecbc0c20a0 msgr2=0x7fecbc152600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.355+0000 7fecc383f640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecbc0c20a0 0x7fecbc152600 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fecb80b0040 tx=0x7fecb8005de0 comp rx=0 tx=0).stop 2024-09-17T05:04:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.355+0000 7fecc383f640 1 -- 172.21.15.150:0/2874199226 shutdown_connections 2024-09-17T05:04:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.355+0000 7fecc383f640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feca0073200 0x7feca0075710 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.355+0000 7fecc383f640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fecbc0c35b0 0x7fecbc0c7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.355+0000 7fecc383f640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fecbc1568a0 0x7fecbc153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.355+0000 7fecc383f640 1 --2- 172.21.15.150:0/2874199226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecbc0c20a0 0x7fecbc152600 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:08.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.355+0000 7fecc383f640 1 -- 172.21.15.150:0/2874199226 >> 172.21.15.150:0/2874199226 conn(0x7fecbc074b80 msgr2=0x7fecbc147810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:08.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.355+0000 7fecc383f640 1 -- 172.21.15.150:0/2874199226 shutdown_connections 2024-09-17T05:04:08.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:08.355+0000 7fecc383f640 1 -- 172.21.15.150:0/2874199226 wait complete. 2024-09-17T05:04:08.377 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:04:07:488011+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 103, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 100, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6010": { "addr": "172.21.15.176:6837/903190072", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 903190072, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 903190072, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6010, "incarnation": 97, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:03:04.151355+0000", "qdb_cluster": [ 6010 ], "qdb_leader": 6010, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6010 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 103, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6495": { "addr": "172.21.15.150:6835/3807957687", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 3807957687, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 3807957687, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6495, "incarnation": 102, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:04:06.500799+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6495 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:04:10.378 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:10.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.555+0000 7f7306135640 1 -- 172.21.15.150:0/2643511296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000c4250 msgr2=0x7f7300153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:10.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.555+0000 7f7306135640 1 --2- 172.21.15.150:0/2643511296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000c4250 0x7f7300153ee0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009900 tx=0x7f72f001c8a0 comp rx=0 tx=0).stop 2024-09-17T05:04:10.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.555+0000 7f7306135640 1 -- 172.21.15.150:0/2643511296 shutdown_connections 2024-09-17T05:04:10.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.555+0000 7f7306135640 1 --2- 172.21.15.150:0/2643511296 >> v2:172.21.15.176:3300/0 conn(0x7f73000c3d10 0x7f730015df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:10.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.555+0000 7f7306135640 1 --2- 172.21.15.150:0/2643511296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000c4250 0x7f7300153ee0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:10.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.555+0000 7f7306135640 1 -- 172.21.15.150:0/2643511296 >> 172.21.15.150:0/2643511296 conn(0x7f730006e360 msgr2=0x7f730006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:10.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.555+0000 7f7306135640 1 -- 172.21.15.150:0/2643511296 shutdown_connections 2024-09-17T05:04:10.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.555+0000 7f7306135640 1 -- 172.21.15.150:0/2643511296 wait complete. 2024-09-17T05:04:10.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f7306135640 1 Processor -- start 2024-09-17T05:04:10.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f7306135640 1 -- start start 2024-09-17T05:04:10.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f7306135640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73000c3900 0x7f73000c2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:10.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f7306135640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000c3d10 0x7f73001f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:10.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f7306135640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73000c4250 0x7f730014dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:10.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f7306135640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7300071120 con 0x7f73000c3d10 2024-09-17T05:04:10.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f7306135640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7300070fa0 con 0x7f73000c4250 2024-09-17T05:04:10.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f7306135640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7300072db0 con 0x7f73000c3900 2024-09-17T05:04:10.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72f6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000c3d10 0x7f73001f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:10.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72f6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000c3d10 0x7f73001f34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59772/0 (socket says 172.21.15.150:59772) 2024-09-17T05:04:10.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72f6ffd640 1 -- 172.21.15.150:0/62447123 learned_addr learned my addr 172.21.15.150:0/62447123 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:10.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72ff7fe640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73000c3900 0x7f73000c2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:10.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72fffff640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73000c4250 0x7f730014dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:10.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72f6ffd640 1 -- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73000c3900 msgr2=0x7f73000c2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:10.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72f6ffd640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73000c3900 0x7f73000c2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:10.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72f6ffd640 1 -- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73000c4250 msgr2=0x7f730014dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:10.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72f6ffd640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73000c4250 0x7f730014dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:10.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72f6ffd640 1 -- 172.21.15.150:0/62447123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73001f0920 con 0x7f73000c3d10 2024-09-17T05:04:10.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72fffff640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73000c4250 0x7f730014dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:04:10.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72f6ffd640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000c3d10 0x7f73001f34b0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f72e800b3f0 tx=0x7f72e800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:10.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72fcff9640 1 -- 172.21.15.150:0/62447123 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e8015020 con 0x7f73000c3d10 2024-09-17T05:04:10.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72fcff9640 1 -- 172.21.15.150:0/62447123 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72e8011920 con 0x7f73000c3d10 2024-09-17T05:04:10.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f72fcff9640 1 -- 172.21.15.150:0/62447123 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e8011d50 con 0x7f73000c3d10 2024-09-17T05:04:10.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f7306135640 1 -- 172.21.15.150:0/62447123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73001f0bc0 con 0x7f73000c3d10 2024-09-17T05:04:10.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.559+0000 7f7306135640 1 -- 172.21.15.150:0/62447123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7300156140 con 0x7f73000c3d10 2024-09-17T05:04:10.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.563+0000 7f7306135640 1 -- 172.21.15.150:0/62447123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7300153ee0 con 0x7f73000c3d10 2024-09-17T05:04:10.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.563+0000 7f72fcff9640 1 -- 172.21.15.150:0/62447123 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f72e8018020 con 0x7f73000c3d10 2024-09-17T05:04:10.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.563+0000 7f72fcff9640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e0073200 0x7f72e0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:10.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.563+0000 7f72fcff9640 1 -- 172.21.15.150:0/62447123 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7f72e8094b20 con 0x7f73000c3d10 2024-09-17T05:04:10.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.563+0000 7f72ff7fe640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e0073200 0x7f72e0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:10.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.563+0000 7f72ff7fe640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e0073200 0x7f72e0075710 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f72f00af000 tx=0x7f72f001b0e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:10.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.567+0000 7f72fcff9640 1 -- 172.21.15.150:0/62447123 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f72e801f020 con 0x7f73000c3d10 2024-09-17T05:04:10.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 -- 172.21.15.150:0/62447123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7300157890 con 0x7f73000c3d10 2024-09-17T05:04:10.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f72fcff9640 1 -- 172.21.15.150:0/62447123 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5713 (secure 0 0 0) 0x7f72e805f1b0 con 0x7f73000c3d10 2024-09-17T05:04:10.793 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:10.793 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":105,"btime":"2024-09-17T05:04:09:506287+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:10.794 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 105 2024-09-17T05:04:10.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 -- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e0073200 msgr2=0x7f72e0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:10.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e0073200 0x7f72e0075710 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f72f00af000 tx=0x7f72f001b0e0 comp rx=0 tx=0).stop 2024-09-17T05:04:10.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 -- 172.21.15.150:0/62447123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000c3d10 msgr2=0x7f73001f34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:10.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000c3d10 0x7f73001f34b0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f72e800b3f0 tx=0x7f72e800b8b0 comp rx=0 tx=0).stop 2024-09-17T05:04:10.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 -- 172.21.15.150:0/62447123 shutdown_connections 2024-09-17T05:04:10.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e0073200 0x7f72e0075710 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:10.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73000c4250 0x7f730014dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:10.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73000c3d10 0x7f73001f34b0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:10.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 --2- 172.21.15.150:0/62447123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73000c3900 0x7f73000c2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:10.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 -- 172.21.15.150:0/62447123 >> 172.21.15.150:0/62447123 conn(0x7f730006e360 msgr2=0x7f730014f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:10.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 -- 172.21.15.150:0/62447123 shutdown_connections 2024-09-17T05:04:10.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:10.795+0000 7f7306135640 1 -- 172.21.15.150:0/62447123 wait complete. 2024-09-17T05:04:10.815 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=6342 2024-09-17T05:04:10.815 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.a 2024-09-17T05:04:11.168 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T05:04:11.167+0000 7fb1d9413640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:04:17.816 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-17T05:04:17.816 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T05:04:17.817 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T05:04:17.820 INFO:tasks.ceph.mds.a:Started 2024-09-17T05:04:17.820 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:17.936 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T05:04:17.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.995+0000 7f15b8e01640 1 -- 172.21.15.150:0/120886489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b40c7cd0 msgr2=0x7f15b40c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:17.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.995+0000 7f15b8e01640 1 --2- 172.21.15.150:0/120886489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b40c7cd0 0x7f15b40c80e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009bc0 tx=0x7f15ac01cba0 comp rx=0 tx=0).stop 2024-09-17T05:04:17.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.995+0000 7f15b8e01640 1 -- 172.21.15.150:0/120886489 shutdown_connections 2024-09-17T05:04:17.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.995+0000 7f15b8e01640 1 --2- 172.21.15.150:0/120886489 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15b4072ea0 0x7f15b40732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:17.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.995+0000 7f15b8e01640 1 --2- 172.21.15.150:0/120886489 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b40c7cd0 0x7f15b40c80e0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:17.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.995+0000 7f15b8e01640 1 -- 172.21.15.150:0/120886489 >> 172.21.15.150:0/120886489 conn(0x7f15b40abb00 msgr2=0x7f15b40abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:17.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.995+0000 7f15b8e01640 1 -- 172.21.15.150:0/120886489 shutdown_connections 2024-09-17T05:04:17.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.995+0000 7f15b8e01640 1 -- 172.21.15.150:0/120886489 wait complete. 2024-09-17T05:04:17.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b8e01640 1 Processor -- start 2024-09-17T05:04:17.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b8e01640 1 -- start start 2024-09-17T05:04:17.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b8e01640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15b4072ea0 0x7f15b40addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:17.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b8e01640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b40c7cd0 0x7f15b414c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:17.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b8e01640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15b40c97a0 0x7f15b414b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:17.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b8e01640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f15b4161910 con 0x7f15b40c7cd0 2024-09-17T05:04:17.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b8e01640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f15b4161790 con 0x7f15b4072ea0 2024-09-17T05:04:17.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b8e01640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f15b4161a90 con 0x7f15b40c97a0 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15abfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b40c7cd0 0x7f15b414c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15abfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b40c7cd0 0x7f15b414c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59804/0 (socket says 172.21.15.150:59804) 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15abfff640 1 -- 172.21.15.150:0/1437575893 learned_addr learned my addr 172.21.15.150:0/1437575893 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b259c640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15b4072ea0 0x7f15b40addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b2d9d640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15b40c97a0 0x7f15b414b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15abfff640 1 -- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15b40c97a0 msgr2=0x7f15b414b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15abfff640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15b40c97a0 0x7f15b414b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15abfff640 1 -- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15b4072ea0 msgr2=0x7f15b40addf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15abfff640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15b4072ea0 0x7f15b40addf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15abfff640 1 -- 172.21.15.150:0/1437575893 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b414dba0 con 0x7f15b40c7cd0 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b259c640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15b4072ea0 0x7f15b40addf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b2d9d640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15b40c97a0 0x7f15b414b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:04:18.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15abfff640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b40c7cd0 0x7f15b414c880 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f159c00d720 tx=0x7f159c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:18.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15aaffd640 1 -- 172.21.15.150:0/1437575893 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c017070 con 0x7f15b40c7cd0 2024-09-17T05:04:18.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15aaffd640 1 -- 172.21.15.150:0/1437575893 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f159c004310 con 0x7f15b40c7cd0 2024-09-17T05:04:18.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b8e01640 1 -- 172.21.15.150:0/1437575893 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b414d8c0 con 0x7f15b40c7cd0 2024-09-17T05:04:18.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15aaffd640 1 -- 172.21.15.150:0/1437575893 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c009a20 con 0x7f15b40c7cd0 2024-09-17T05:04:18.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b8e01640 1 -- 172.21.15.150:0/1437575893 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b41f1cf0 con 0x7f15b40c7cd0 2024-09-17T05:04:18.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:17.999+0000 7f15b8e01640 1 -- 172.21.15.150:0/1437575893 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b40732b0 con 0x7f15b40c7cd0 2024-09-17T05:04:18.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.003+0000 7f15aaffd640 1 -- 172.21.15.150:0/1437575893 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f159c009bc0 con 0x7f15b40c7cd0 2024-09-17T05:04:18.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.003+0000 7f15aaffd640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1594073230 0x7f1594075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:18.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.003+0000 7f15aaffd640 1 -- 172.21.15.150:0/1437575893 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7f159c094c20 con 0x7f15b40c7cd0 2024-09-17T05:04:18.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.003+0000 7f15b259c640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1594073230 0x7f1594075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:18.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.003+0000 7f15b259c640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1594073230 0x7f1594075740 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f15a4001630 tx=0x7f15a4009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:18.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.007+0000 7f15aaffd640 1 -- 172.21.15.150:0/1437575893 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f159c05f250 con 0x7f15b40c7cd0 2024-09-17T05:04:18.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.259+0000 7f15b8e01640 1 -- 172.21.15.150:0/1437575893 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15b414f760 con 0x7f15b40c7cd0 2024-09-17T05:04:18.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.259+0000 7f15aaffd640 1 -- 172.21.15.150:0/1437575893 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5713 (secure 0 0 0) 0x7f159c05eb50 con 0x7f15b40c7cd0 2024-09-17T05:04:18.258 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:18.258 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":105,"btime":"2024-09-17T05:04:09:506287+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:18.258 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 105 2024-09-17T05:04:18.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.263+0000 7f15b8e01640 1 -- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1594073230 msgr2=0x7f1594075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:18.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.263+0000 7f15b8e01640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1594073230 0x7f1594075740 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f15a4001630 tx=0x7f15a4009040 comp rx=0 tx=0).stop 2024-09-17T05:04:18.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.263+0000 7f15b8e01640 1 -- 172.21.15.150:0/1437575893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b40c7cd0 msgr2=0x7f15b414c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:18.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.263+0000 7f15b8e01640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b40c7cd0 0x7f15b414c880 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f159c00d720 tx=0x7f159c00dbe0 comp rx=0 tx=0).stop 2024-09-17T05:04:18.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.263+0000 7f15b8e01640 1 -- 172.21.15.150:0/1437575893 shutdown_connections 2024-09-17T05:04:18.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.263+0000 7f15b8e01640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1594073230 0x7f1594075740 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:18.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.263+0000 7f15b8e01640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f15b40c97a0 0x7f15b414b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:18.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.263+0000 7f15b8e01640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15b40c7cd0 0x7f15b414c880 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:18.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.263+0000 7f15b8e01640 1 --2- 172.21.15.150:0/1437575893 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15b4072ea0 0x7f15b40addf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:18.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.263+0000 7f15b8e01640 1 -- 172.21.15.150:0/1437575893 >> 172.21.15.150:0/1437575893 conn(0x7f15b40abb00 msgr2=0x7f15b4150e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:18.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.263+0000 7f15b8e01640 1 -- 172.21.15.150:0/1437575893 shutdown_connections 2024-09-17T05:04:18.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:18.263+0000 7f15b8e01640 1 -- 172.21.15.150:0/1437575893 wait complete. 2024-09-17T05:04:18.285 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-17T05:04:19.368 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:19.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.547+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/840865319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb80004d50 msgr2=0x7fbb800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:19.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.547+0000 7fbb9bfe7640 1 --2- 172.21.15.150:0/840865319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb80004d50 0x7fbb800051c0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009ec0 tx=0x7fbb8c01d5f0 comp rx=0 tx=0).stop 2024-09-17T05:04:19.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.547+0000 7fbb9a584640 1 -- 172.21.15.150:0/840865319 reap_dead start 2024-09-17T05:04:19.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.547+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/840865319 shutdown_connections 2024-09-17T05:04:19.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.547+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/840865319 >> 172.21.15.150:0/840865319 conn(0x7fbb940aab60 msgr2=0x7fbb940aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:19.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.547+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/840865319 shutdown_connections 2024-09-17T05:04:19.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.547+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/840865319 wait complete. 2024-09-17T05:04:19.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.547+0000 7fbb9bfe7640 1 Processor -- start 2024-09-17T05:04:19.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.547+0000 7fbb9bfe7640 1 -- start start 2024-09-17T05:04:19.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb9bfe7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb80004d50 0x7fbb9406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:19.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb9bfe7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb940a9340 0x7fbb941efe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:19.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb9bfe7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb941f0350 0x7fbb94146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:19.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb9bfe7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbb94153e70 con 0x7fbb80004d50 2024-09-17T05:04:19.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb9bfe7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbb94153cf0 con 0x7fbb940a9340 2024-09-17T05:04:19.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb9bfe7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbb940c8a00 con 0x7fbb941f0350 2024-09-17T05:04:19.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb99d83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb80004d50 0x7fbb9406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:19.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb99d83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb80004d50 0x7fbb9406d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35570/0 (socket says 172.21.15.150:35570) 2024-09-17T05:04:19.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb99d83640 1 -- 172.21.15.150:0/1663040876 learned_addr learned my addr 172.21.15.150:0/1663040876 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:19.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb99582640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb940a9340 0x7fbb941efe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:19.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb9a584640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb941f0350 0x7fbb94146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:19.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb99d83640 1 -- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb941f0350 msgr2=0x7fbb94146ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb99d83640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb941f0350 0x7fbb94146ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb99d83640 1 -- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb940a9340 msgr2=0x7fbb941efe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb99d83640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb940a9340 0x7fbb941efe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb99d83640 1 -- 172.21.15.150:0/1663040876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb94149690 con 0x7fbb80004d50 2024-09-17T05:04:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb99582640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb940a9340 0x7fbb941efe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:04:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb99d83640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb80004d50 0x7fbb9406d680 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fbb90012670 tx=0x7fbb90012b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:19.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb8a7fc640 1 -- 172.21.15.150:0/1663040876 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb90010ac0 con 0x7fbb80004d50 2024-09-17T05:04:19.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb8a7fc640 1 -- 172.21.15.150:0/1663040876 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb90017030 con 0x7fbb80004d50 2024-09-17T05:04:19.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/1663040876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb941ed0a0 con 0x7fbb80004d50 2024-09-17T05:04:19.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb8a7fc640 1 -- 172.21.15.150:0/1663040876 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb90012e70 con 0x7fbb80004d50 2024-09-17T05:04:19.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.551+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/1663040876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb94152190 con 0x7fbb80004d50 2024-09-17T05:04:19.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.555+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/1663040876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb94149f40 con 0x7fbb80004d50 2024-09-17T05:04:19.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.555+0000 7fbb8a7fc640 1 -- 172.21.15.150:0/1663040876 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbb90013450 con 0x7fbb80004d50 2024-09-17T05:04:19.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.555+0000 7fbb8a7fc640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb78073230 0x7fbb78075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:19.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.555+0000 7fbb8a7fc640 1 -- 172.21.15.150:0/1663040876 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7fbb900987d0 con 0x7fbb80004d50 2024-09-17T05:04:19.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.555+0000 7fbb99582640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb78073230 0x7fbb78075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:19.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.555+0000 7fbb99582640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb78073230 0x7fbb78075740 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbb9406de90 tx=0x7fbb84006ad0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:19.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.563+0000 7fbb8a7fc640 1 -- 172.21.15.150:0/1663040876 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbb90062e60 con 0x7fbb80004d50 2024-09-17T05:04:19.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.835+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/1663040876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb941efb70 con 0x7fbb80004d50 2024-09-17T05:04:19.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.835+0000 7fbb8a7fc640 1 -- 172.21.15.150:0/1663040876 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+6542 (secure 0 0 0) 0x7fbb90062760 con 0x7fbb80004d50 2024-09-17T05:04:19.835 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:19.835 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":106,"btime":"2024-09-17T05:04:18:543254+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6564,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:19.836 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 106 2024-09-17T05:04:19.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.839+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb78073230 msgr2=0x7fbb78075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:19.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.839+0000 7fbb9bfe7640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb78073230 0x7fbb78075740 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbb9406de90 tx=0x7fbb84006ad0 comp rx=0 tx=0).stop 2024-09-17T05:04:19.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.839+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/1663040876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb80004d50 msgr2=0x7fbb9406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:19.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.839+0000 7fbb9bfe7640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb80004d50 0x7fbb9406d680 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fbb90012670 tx=0x7fbb90012b30 comp rx=0 tx=0).stop 2024-09-17T05:04:19.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.839+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/1663040876 shutdown_connections 2024-09-17T05:04:19.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.839+0000 7fbb9bfe7640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbb78073230 0x7fbb78075740 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:19.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.839+0000 7fbb9bfe7640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbb940a9340 0x7fbb941efe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:19.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.839+0000 7fbb9bfe7640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbb941f0350 0x7fbb94146ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:19.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.839+0000 7fbb9bfe7640 1 --2- 172.21.15.150:0/1663040876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb80004d50 0x7fbb9406d680 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:19.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.839+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/1663040876 >> 172.21.15.150:0/1663040876 conn(0x7fbb940aab60 msgr2=0x7fbb940c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:19.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.839+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/1663040876 shutdown_connections 2024-09-17T05:04:19.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:19.839+0000 7fbb9bfe7640 1 -- 172.21.15.150:0/1663040876 wait complete. 2024-09-17T05:04:19.858 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-17T05:04:19.858 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T05:04:19.858 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:04:19.877 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T05:04:19.877 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:20.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.047+0000 7fed296f4640 1 -- 172.21.15.150:0/1787622162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed241568a0 msgr2=0x7fed240c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:20.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.047+0000 7fed296f4640 1 --2- 172.21.15.150:0/1787622162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed241568a0 0x7fed240c5810 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fed14009870 tx=0x7fed1401c830 comp rx=0 tx=0).stop 2024-09-17T05:04:20.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.047+0000 7fed296f4640 1 -- 172.21.15.150:0/1787622162 shutdown_connections 2024-09-17T05:04:20.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.047+0000 7fed296f4640 1 --2- 172.21.15.150:0/1787622162 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed241568a0 0x7fed240c5810 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.047+0000 7fed296f4640 1 -- 172.21.15.150:0/1787622162 >> 172.21.15.150:0/1787622162 conn(0x7fed24074b80 msgr2=0x7fed2406f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:20.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.047+0000 7fed296f4640 1 -- 172.21.15.150:0/1787622162 shutdown_connections 2024-09-17T05:04:20.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.047+0000 7fed296f4640 1 -- 172.21.15.150:0/1787622162 wait complete. 2024-09-17T05:04:20.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed296f4640 1 Processor -- start 2024-09-17T05:04:20.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed296f4640 1 -- start start 2024-09-17T05:04:20.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed296f4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed240c20a0 0x7fed2414b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:20.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed296f4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed240c35b0 0x7fed24159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:20.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed296f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed241568a0 0x7fed24145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:20.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed296f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fed240c9310 con 0x7fed241568a0 2024-09-17T05:04:20.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed296f4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fed240c9190 con 0x7fed240c35b0 2024-09-17T05:04:20.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed296f4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fed240c39c0 con 0x7fed240c20a0 2024-09-17T05:04:20.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed241568a0 0x7fed24145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:20.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed241568a0 0x7fed24145a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35586/0 (socket says 172.21.15.150:35586) 2024-09-17T05:04:20.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed237fe640 1 -- 172.21.15.150:0/3990288112 learned_addr learned my addr 172.21.15.150:0/3990288112 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:20.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed22ffd640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed240c20a0 0x7fed2414b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:20.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed1bfff640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed240c35b0 0x7fed24159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:20.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed237fe640 1 -- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed240c20a0 msgr2=0x7fed2414b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:20.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed237fe640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed240c20a0 0x7fed2414b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed237fe640 1 -- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed240c35b0 msgr2=0x7fed24159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:20.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed237fe640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed240c35b0 0x7fed24159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed237fe640 1 -- 172.21.15.150:0/3990288112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed241f0410 con 0x7fed241568a0 2024-09-17T05:04:20.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed237fe640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed241568a0 0x7fed24145a40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00caa0 tx=0x7fed1c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:20.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed1b7fe640 1 -- 172.21.15.150:0/3990288112 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c016020 con 0x7fed241568a0 2024-09-17T05:04:20.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed1b7fe640 1 -- 172.21.15.150:0/3990288112 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed1c00a8d0 con 0x7fed241568a0 2024-09-17T05:04:20.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed296f4640 1 -- 172.21.15.150:0/3990288112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed2414aab0 con 0x7fed241568a0 2024-09-17T05:04:20.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed1b7fe640 1 -- 172.21.15.150:0/3990288112 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c00ad20 con 0x7fed241568a0 2024-09-17T05:04:20.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.051+0000 7fed296f4640 1 -- 172.21.15.150:0/3990288112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed2414b070 con 0x7fed241568a0 2024-09-17T05:04:20.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.055+0000 7fed296f4640 1 -- 172.21.15.150:0/3990288112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed240c6aa0 con 0x7fed241568a0 2024-09-17T05:04:20.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.055+0000 7fed1b7fe640 1 -- 172.21.15.150:0/3990288112 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fed1c019020 con 0x7fed241568a0 2024-09-17T05:04:20.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.055+0000 7fed1b7fe640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fed04073230 0x7fed04075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:20.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.055+0000 7fed1b7fe640 1 -- 172.21.15.150:0/3990288112 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7fed1c094fb0 con 0x7fed241568a0 2024-09-17T05:04:20.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.055+0000 7fed22ffd640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fed04073230 0x7fed04075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:20.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.055+0000 7fed22ffd640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fed04073230 0x7fed04075740 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fed140af000 tx=0x7fed1401b070 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:20.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.059+0000 7fed1b7fe640 1 -- 172.21.15.150:0/3990288112 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fed1c020030 con 0x7fed241568a0 2024-09-17T05:04:20.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.275+0000 7fed296f4640 1 -- 172.21.15.150:0/3990288112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed2414c720 con 0x7fed241568a0 2024-09-17T05:04:20.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.275+0000 7fed1b7fe640 1 -- 172.21.15.150:0/3990288112 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+6542 (secure 0 0 0) 0x7fed1c05f080 con 0x7fed241568a0 2024-09-17T05:04:20.275 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:20.276 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":106,"btime":"2024-09-17T05:04:18:543254+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6564,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:20.276 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 106 2024-09-17T05:04:20.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.279+0000 7fed296f4640 1 -- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fed04073230 msgr2=0x7fed04075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:20.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.279+0000 7fed296f4640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fed04073230 0x7fed04075740 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fed140af000 tx=0x7fed1401b070 comp rx=0 tx=0).stop 2024-09-17T05:04:20.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.279+0000 7fed296f4640 1 -- 172.21.15.150:0/3990288112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed241568a0 msgr2=0x7fed24145a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:20.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.279+0000 7fed296f4640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed241568a0 0x7fed24145a40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00caa0 tx=0x7fed1c00cb90 comp rx=0 tx=0).stop 2024-09-17T05:04:20.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.279+0000 7fed296f4640 1 -- 172.21.15.150:0/3990288112 shutdown_connections 2024-09-17T05:04:20.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.279+0000 7fed296f4640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fed04073230 0x7fed04075740 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.279+0000 7fed296f4640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed240c35b0 0x7fed24159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.279+0000 7fed296f4640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed241568a0 0x7fed24145a40 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.279+0000 7fed296f4640 1 --2- 172.21.15.150:0/3990288112 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fed240c20a0 0x7fed2414b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.279+0000 7fed296f4640 1 -- 172.21.15.150:0/3990288112 >> 172.21.15.150:0/3990288112 conn(0x7fed24074b80 msgr2=0x7fed2414ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:20.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.279+0000 7fed296f4640 1 -- 172.21.15.150:0/3990288112 shutdown_connections 2024-09-17T05:04:20.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.279+0000 7fed296f4640 1 -- 172.21.15.150:0/3990288112 wait complete. 2024-09-17T05:04:20.286 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:20.297 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:04:20.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.443+0000 7f80ccc1c640 1 -- 172.21.15.150:0/2080809995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80c80a8120 msgr2=0x7f80c8149f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:20.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.443+0000 7f80ccc1c640 1 --2- 172.21.15.150:0/2080809995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80c80a8120 0x7f80c8149f40 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f80c80c9e40 tx=0x7f80b801d320 comp rx=0 tx=0).stop 2024-09-17T05:04:20.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.443+0000 7f80ccc1c640 1 -- 172.21.15.150:0/2080809995 shutdown_connections 2024-09-17T05:04:20.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.443+0000 7f80ccc1c640 1 --2- 172.21.15.150:0/2080809995 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80c8157810 0x7f80c8153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.443+0000 7f80ccc1c640 1 --2- 172.21.15.150:0/2080809995 >> v2:172.21.15.176:3300/0 conn(0x7f80c80c37b0 0x7f80c81535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.443+0000 7f80ccc1c640 1 --2- 172.21.15.150:0/2080809995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80c80a8120 0x7f80c8149f40 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.443+0000 7f80ccc1c640 1 -- 172.21.15.150:0/2080809995 >> 172.21.15.150:0/2080809995 conn(0x7f80c80aab60 msgr2=0x7f80c80aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:20.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.443+0000 7f80ccc1c640 1 -- 172.21.15.150:0/2080809995 shutdown_connections 2024-09-17T05:04:20.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.443+0000 7f80ccc1c640 1 -- 172.21.15.150:0/2080809995 wait complete. 2024-09-17T05:04:20.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80ccc1c640 1 Processor -- start 2024-09-17T05:04:20.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80ccc1c640 1 -- start start 2024-09-17T05:04:20.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80ccc1c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80c80a8120 0x7f80c814dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:20.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80ccc1c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80c80c37b0 0x7f80c8148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:20.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80ccc1c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80c8157810 0x7f80c80ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80ccc1c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f80c80c6e30 con 0x7f80c8157810 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80ccc1c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f80c80c6cb0 con 0x7f80c80a8120 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80ccc1c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f80c80c6fb0 con 0x7f80c80c37b0 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80c6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80c8157810 0x7f80c80ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80c659c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80c80a8120 0x7f80c814dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80c6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80c8157810 0x7f80c80ad750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35614/0 (socket says 172.21.15.150:35614) 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80c6d9d640 1 -- 172.21.15.150:0/3329126175 learned_addr learned my addr 172.21.15.150:0/3329126175 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80c659c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80c80a8120 0x7f80c814dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:33324/0 (socket says 172.21.15.150:33324) 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80c5d9b640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80c80c37b0 0x7f80c8148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80c6d9d640 1 -- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80c80c37b0 msgr2=0x7f80c8148620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80c6d9d640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80c80c37b0 0x7f80c8148620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80c6d9d640 1 -- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80c80a8120 msgr2=0x7f80c814dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80c6d9d640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80c80a8120 0x7f80c814dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80c6d9d640 1 -- 172.21.15.150:0/3329126175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c814e1e0 con 0x7f80c8157810 2024-09-17T05:04:20.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80c6d9d640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80c8157810 0x7f80c80ad750 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f80c000e970 tx=0x7f80c000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:20.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80b6ffd640 1 -- 172.21.15.150:0/3329126175 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c0004d30 con 0x7f80c8157810 2024-09-17T05:04:20.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80b6ffd640 1 -- 172.21.15.150:0/3329126175 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80c000cb20 con 0x7f80c8157810 2024-09-17T05:04:20.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80ccc1c640 1 -- 172.21.15.150:0/3329126175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c8148f00 con 0x7f80c8157810 2024-09-17T05:04:20.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80b6ffd640 1 -- 172.21.15.150:0/3329126175 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c0005020 con 0x7f80c8157810 2024-09-17T05:04:20.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80ccc1c640 1 -- 172.21.15.150:0/3329126175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c80ae630 con 0x7f80c8157810 2024-09-17T05:04:20.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.447+0000 7f80ccc1c640 1 -- 172.21.15.150:0/3329126175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80c8149140 con 0x7f80c8157810 2024-09-17T05:04:20.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.451+0000 7f80b6ffd640 1 -- 172.21.15.150:0/3329126175 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f80c00051c0 con 0x7f80c8157810 2024-09-17T05:04:20.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.451+0000 7f80b6ffd640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f80a8073200 0x7f80a8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:20.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.451+0000 7f80b6ffd640 1 -- 172.21.15.150:0/3329126175 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7f80c0095350 con 0x7f80c8157810 2024-09-17T05:04:20.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.451+0000 7f80c659c640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f80a8073200 0x7f80a8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:20.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.451+0000 7f80c659c640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f80a8073200 0x7f80a8075710 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f80b80af000 tx=0x7f80b801b100 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:20.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.455+0000 7f80b6ffd640 1 -- 172.21.15.150:0/3329126175 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f80c8149140 con 0x7f80c8157810 2024-09-17T05:04:20.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.663+0000 7f80ccc1c640 1 -- 172.21.15.150:0/3329126175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80c8149f40 con 0x7f80c8157810 2024-09-17T05:04:20.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80b6ffd640 1 -- 172.21.15.150:0/3329126175 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+6542 (secure 0 0 0) 0x7f80c0016030 con 0x7f80c8157810 2024-09-17T05:04:20.665 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:20.665 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":106,"btime":"2024-09-17T05:04:18:543254+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6564,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:20.666 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 106 2024-09-17T05:04:20.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80ccc1c640 1 -- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f80a8073200 msgr2=0x7f80a8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:20.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80ccc1c640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f80a8073200 0x7f80a8075710 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f80b80af000 tx=0x7f80b801b100 comp rx=0 tx=0).stop 2024-09-17T05:04:20.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80ccc1c640 1 -- 172.21.15.150:0/3329126175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80c8157810 msgr2=0x7f80c80ad750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:20.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80ccc1c640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80c8157810 0x7f80c80ad750 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f80c000e970 tx=0x7f80c000ee30 comp rx=0 tx=0).stop 2024-09-17T05:04:20.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80ccc1c640 1 -- 172.21.15.150:0/3329126175 shutdown_connections 2024-09-17T05:04:20.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80ccc1c640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f80a8073200 0x7f80a8075710 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80ccc1c640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80c8157810 0x7f80c80ad750 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80ccc1c640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f80c80c37b0 0x7f80c8148620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80ccc1c640 1 --2- 172.21.15.150:0/3329126175 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80c80a8120 0x7f80c814dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:20.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80ccc1c640 1 -- 172.21.15.150:0/3329126175 >> 172.21.15.150:0/3329126175 conn(0x7f80c80aab60 msgr2=0x7f80c80c5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:20.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80ccc1c640 1 -- 172.21.15.150:0/3329126175 shutdown_connections 2024-09-17T05:04:20.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:20.667+0000 7f80ccc1c640 1 -- 172.21.15.150:0/3329126175 wait complete. 2024-09-17T05:04:20.687 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-17T05:04:20.687 INFO:tasks.mds_thrash.fs.[b]:waiting for 79 secs before thrashing 2024-09-17T05:04:22.298 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:22.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.463+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/697648745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4cc06df60 msgr2=0x7fc4cc06e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:22.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.463+0000 7fc4d1fb4640 1 --2- 172.21.15.150:0/697648745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4cc06df60 0x7fc4cc06e370 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc06e970 tx=0x7fc4b401cc60 comp rx=0 tx=0).stop 2024-09-17T05:04:22.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.463+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/697648745 shutdown_connections 2024-09-17T05:04:22.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.463+0000 7fc4d1fb4640 1 --2- 172.21.15.150:0/697648745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4cc06df60 0x7fc4cc06e370 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:22.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.463+0000 7fc4d1fb4640 1 --2- 172.21.15.150:0/697648745 >> v2:172.21.15.176:3300/0 conn(0x7fc4cc0ab9d0 0x7fc4cc14da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:22.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.463+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/697648745 >> 172.21.15.150:0/697648745 conn(0x7fc4cc070b50 msgr2=0x7fc4cc070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:22.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.463+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/697648745 shutdown_connections 2024-09-17T05:04:22.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.463+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/697648745 wait complete. 2024-09-17T05:04:22.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4d1fb4640 1 Processor -- start 2024-09-17T05:04:22.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4d1fb4640 1 -- start start 2024-09-17T05:04:22.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4d1fb4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4cc06df60 0x7fc4cc1efc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:22.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4d1fb4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4cc0ab9d0 0x7fc4cc146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:22.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4d1fb4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4cc1536e0 0x7fc4cc1404e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:22.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4d1fb4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc154200 con 0x7fc4cc1536e0 2024-09-17T05:04:22.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4d1fb4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc069e00 con 0x7fc4cc0ab9d0 2024-09-17T05:04:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4d1fb4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc4cc154380 con 0x7fc4cc06df60 2024-09-17T05:04:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4cbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4cc1536e0 0x7fc4cc1404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4cbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4cc1536e0 0x7fc4cc1404e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35642/0 (socket says 172.21.15.150:35642) 2024-09-17T05:04:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4cbfff640 1 -- 172.21.15.150:0/2124033682 learned_addr learned my addr 172.21.15.150:0/2124033682 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4cb7fe640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4cc06df60 0x7fc4cc1efc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4caffd640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4cc0ab9d0 0x7fc4cc146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4cbfff640 1 -- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4cc06df60 msgr2=0x7fc4cc1efc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4cbfff640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4cc06df60 0x7fc4cc1efc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4cbfff640 1 -- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4cc0ab9d0 msgr2=0x7fc4cc146fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4cbfff640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4cc0ab9d0 0x7fc4cc146fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4cbfff640 1 -- 172.21.15.150:0/2124033682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4cc0c7df0 con 0x7fc4cc1536e0 2024-09-17T05:04:22.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4caffd640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4cc0ab9d0 0x7fc4cc146fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:04:22.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4cbfff640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4cc1536e0 0x7fc4cc1404e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc00c3c0 tx=0x7fc4bc00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:22.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4c3fff640 1 -- 172.21.15.150:0/2124033682 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc0052d0 con 0x7fc4cc1536e0 2024-09-17T05:04:22.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4c3fff640 1 -- 172.21.15.150:0/2124033682 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4bc007020 con 0x7fc4cc1536e0 2024-09-17T05:04:22.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4c3fff640 1 -- 172.21.15.150:0/2124033682 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc007450 con 0x7fc4cc1536e0 2024-09-17T05:04:22.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/2124033682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4cc15ce30 con 0x7fc4cc1536e0 2024-09-17T05:04:22.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/2124033682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4cc0adf00 con 0x7fc4cc1536e0 2024-09-17T05:04:22.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.467+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/2124033682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4cc0c8040 con 0x7fc4cc1536e0 2024-09-17T05:04:22.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.471+0000 7fc4c3fff640 1 -- 172.21.15.150:0/2124033682 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc4bc0067e0 con 0x7fc4cc1536e0 2024-09-17T05:04:22.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.471+0000 7fc4c3fff640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc4ac073230 0x7fc4ac075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:22.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.471+0000 7fc4c3fff640 1 -- 172.21.15.150:0/2124033682 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7fc4bc0940b0 con 0x7fc4cc1536e0 2024-09-17T05:04:22.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.471+0000 7fc4cb7fe640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc4ac073230 0x7fc4ac075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:22.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.471+0000 7fc4cb7fe640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc4ac073230 0x7fc4ac075740 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4005fd0 tx=0x7fc4c4005e40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:22.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.475+0000 7fc4c3fff640 1 -- 172.21.15.150:0/2124033682 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc4bc05e740 con 0x7fc4cc1536e0 2024-09-17T05:04:22.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.683+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/2124033682 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4cc075060 con 0x7fc4cc1536e0 2024-09-17T05:04:22.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4c3fff640 1 -- 172.21.15.150:0/2124033682 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+6542 (secure 0 0 0) 0x7fc4bc05e040 con 0x7fc4cc1536e0 2024-09-17T05:04:22.685 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:22.685 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":106,"btime":"2024-09-17T05:04:18:543254+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6564,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:22.685 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 106 2024-09-17T05:04:22.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc4ac073230 msgr2=0x7fc4ac075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4d1fb4640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc4ac073230 0x7fc4ac075740 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4005fd0 tx=0x7fc4c4005e40 comp rx=0 tx=0).stop 2024-09-17T05:04:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/2124033682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4cc1536e0 msgr2=0x7fc4cc1404e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4d1fb4640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4cc1536e0 0x7fc4cc1404e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc00c3c0 tx=0x7fc4bc00c7c0 comp rx=0 tx=0).stop 2024-09-17T05:04:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/2124033682 shutdown_connections 2024-09-17T05:04:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4d1fb4640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc4ac073230 0x7fc4ac075740 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4d1fb4640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4cc0ab9d0 0x7fc4cc146fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4d1fb4640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4cc1536e0 0x7fc4cc1404e0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4d1fb4640 1 --2- 172.21.15.150:0/2124033682 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc4cc06df60 0x7fc4cc1efc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/2124033682 >> 172.21.15.150:0/2124033682 conn(0x7fc4cc070b50 msgr2=0x7fc4cc153af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/2124033682 shutdown_connections 2024-09-17T05:04:22.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:22.687+0000 7fc4d1fb4640 1 -- 172.21.15.150:0/2124033682 wait complete. 2024-09-17T05:04:22.706 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:04:24.707 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:24.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.871+0000 7f5771ffb640 1 -- 172.21.15.150:0/3636749231 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57640ad850 con 0x7f5774156900 2024-09-17T05:04:24.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.871+0000 7f57798dc640 1 -- 172.21.15.150:0/3636749231 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5774156900 msgr2=0x7f57740c3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:24.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.871+0000 7f57798dc640 1 --2- 172.21.15.150:0/3636749231 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5774156900 0x7f57740c3670 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5764009c80 tx=0x7f576401cce0 comp rx=0 tx=0).stop 2024-09-17T05:04:24.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.871+0000 7f57798dc640 1 -- 172.21.15.150:0/3636749231 shutdown_connections 2024-09-17T05:04:24.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.871+0000 7f57798dc640 1 --2- 172.21.15.150:0/3636749231 >> v2:172.21.15.176:3300/0 conn(0x7f57740c8cc0 0x7f5774150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:24.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.871+0000 7f57798dc640 1 --2- 172.21.15.150:0/3636749231 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5774156900 0x7f57740c3670 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:24.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.871+0000 7f57798dc640 1 -- 172.21.15.150:0/3636749231 >> 172.21.15.150:0/3636749231 conn(0x7f577406f760 msgr2=0x7f5774074580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:24.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.871+0000 7f57798dc640 1 -- 172.21.15.150:0/3636749231 shutdown_connections 2024-09-17T05:04:24.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.871+0000 7f57798dc640 1 -- 172.21.15.150:0/3636749231 wait complete. 2024-09-17T05:04:24.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57798dc640 1 Processor -- start 2024-09-17T05:04:24.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57798dc640 1 -- start start 2024-09-17T05:04:24.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57798dc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57740c8cc0 0x7f57740cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:24.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57798dc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57740c90d0 0x7f5774153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:24.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57798dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5774156900 0x7f5774150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:24.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57798dc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f57740b1040 con 0x7f5774156900 2024-09-17T05:04:24.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57798dc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f57740c41f0 con 0x7f57740c90d0 2024-09-17T05:04:24.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57798dc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f57740b11c0 con 0x7f57740c8cc0 2024-09-17T05:04:24.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57737fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5774156900 0x7f5774150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:24.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57737fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5774156900 0x7f5774150f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35650/0 (socket says 172.21.15.150:35650) 2024-09-17T05:04:24.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57737fe640 1 -- 172.21.15.150:0/2502845713 learned_addr learned my addr 172.21.15.150:0/2502845713 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:24.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f5772ffd640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57740c8cc0 0x7f57740cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:24.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57727fc640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57740c90d0 0x7f5774153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:24.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57737fe640 1 -- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57740c8cc0 msgr2=0x7f57740cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:24.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57737fe640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57740c8cc0 0x7f57740cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:24.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57737fe640 1 -- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57740c90d0 msgr2=0x7f5774153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:24.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57737fe640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57740c90d0 0x7f5774153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:24.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57737fe640 1 -- 172.21.15.150:0/2502845713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57741f01b0 con 0x7f5774156900 2024-09-17T05:04:24.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f5772ffd640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57740c8cc0 0x7f57740cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:04:24.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57727fc640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57740c90d0 0x7f5774153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:04:24.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57737fe640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5774156900 0x7f5774150f80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c3c0 tx=0x7f576c001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:24.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f576bfff640 1 -- 172.21.15.150:0/2502845713 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c017070 con 0x7f5774156900 2024-09-17T05:04:24.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f576bfff640 1 -- 172.21.15.150:0/2502845713 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f576c0053e0 con 0x7f5774156900 2024-09-17T05:04:24.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57798dc640 1 -- 172.21.15.150:0/2502845713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f577414c650 con 0x7f5774156900 2024-09-17T05:04:24.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f576bfff640 1 -- 172.21.15.150:0/2502845713 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c003860 con 0x7f5774156900 2024-09-17T05:04:24.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.875+0000 7f57798dc640 1 -- 172.21.15.150:0/2502845713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f577414d6f0 con 0x7f5774156900 2024-09-17T05:04:24.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.879+0000 7f57798dc640 1 -- 172.21.15.150:0/2502845713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f577414d4e0 con 0x7f5774156900 2024-09-17T05:04:24.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.879+0000 7f576bfff640 1 -- 172.21.15.150:0/2502845713 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f576c01a070 con 0x7f5774156900 2024-09-17T05:04:24.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.879+0000 7f576bfff640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5754073200 0x7f5754075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:24.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.879+0000 7f576bfff640 1 -- 172.21.15.150:0/2502845713 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7f576c093f00 con 0x7f5774156900 2024-09-17T05:04:24.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.879+0000 7f5772ffd640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5754073200 0x7f5754075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:24.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.879+0000 7f5772ffd640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5754073200 0x7f5754075710 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f57640af000 tx=0x7f576401b520 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:24.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:24.883+0000 7f576bfff640 1 -- 172.21.15.150:0/2502845713 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f57740c3bb0 con 0x7f5774156900 2024-09-17T05:04:25.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.099+0000 7f57798dc640 1 -- 172.21.15.150:0/2502845713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5774150570 con 0x7f5774156900 2024-09-17T05:04:25.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.099+0000 7f576bfff640 1 -- 172.21.15.150:0/2502845713 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+6542 (secure 0 0 0) 0x7f576c05e620 con 0x7f5774156900 2024-09-17T05:04:25.098 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:25.098 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":106,"btime":"2024-09-17T05:04:18:543254+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6564,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:25.098 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 106 2024-09-17T05:04:25.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.099+0000 7f57798dc640 1 -- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5754073200 msgr2=0x7f5754075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:25.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.099+0000 7f57798dc640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5754073200 0x7f5754075710 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f57640af000 tx=0x7f576401b520 comp rx=0 tx=0).stop 2024-09-17T05:04:25.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.099+0000 7f57798dc640 1 -- 172.21.15.150:0/2502845713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5774156900 msgr2=0x7f5774150f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:25.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.099+0000 7f57798dc640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5774156900 0x7f5774150f80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c3c0 tx=0x7f576c001e80 comp rx=0 tx=0).stop 2024-09-17T05:04:25.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.099+0000 7f57798dc640 1 -- 172.21.15.150:0/2502845713 shutdown_connections 2024-09-17T05:04:25.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.099+0000 7f57798dc640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5754073200 0x7f5754075710 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:25.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.099+0000 7f57798dc640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57740c90d0 0x7f5774153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:25.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.099+0000 7f57798dc640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5774156900 0x7f5774150f80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:25.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.099+0000 7f57798dc640 1 --2- 172.21.15.150:0/2502845713 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57740c8cc0 0x7f57740cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:25.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.099+0000 7f57798dc640 1 -- 172.21.15.150:0/2502845713 >> 172.21.15.150:0/2502845713 conn(0x7f577406f760 msgr2=0x7f57740c5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:25.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.103+0000 7f57798dc640 1 -- 172.21.15.150:0/2502845713 shutdown_connections 2024-09-17T05:04:25.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:25.103+0000 7f57798dc640 1 -- 172.21.15.150:0/2502845713 wait complete. 2024-09-17T05:04:25.120 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:04:27.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:27.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.287+0000 7fb5766d3640 1 -- 172.21.15.150:0/1900702663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5700c5830 msgr2=0x7fb57014b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:27.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.287+0000 7fb5766d3640 1 --2- 172.21.15.150:0/1900702663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5700c5830 0x7fb57014b0b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb570072d50 tx=0x7fb56401c6e0 comp rx=0 tx=0).stop 2024-09-17T05:04:27.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.287+0000 7fb5766d3640 1 -- 172.21.15.150:0/1900702663 shutdown_connections 2024-09-17T05:04:27.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.287+0000 7fb5766d3640 1 --2- 172.21.15.150:0/1900702663 >> v2:172.21.15.176:3300/0 conn(0x7fb570157c60 0x7fb570158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:27.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.287+0000 7fb5766d3640 1 --2- 172.21.15.150:0/1900702663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb5700c5830 0x7fb57014b0b0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:27.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.287+0000 7fb5766d3640 1 -- 172.21.15.150:0/1900702663 >> 172.21.15.150:0/1900702663 conn(0x7fb5700ca460 msgr2=0x7fb5700c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:27.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.287+0000 7fb5766d3640 1 -- 172.21.15.150:0/1900702663 shutdown_connections 2024-09-17T05:04:27.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.287+0000 7fb5766d3640 1 -- 172.21.15.150:0/1900702663 wait complete. 2024-09-17T05:04:27.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb5766d3640 1 Processor -- start 2024-09-17T05:04:27.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb5766d3640 1 -- start start 2024-09-17T05:04:27.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb5766d3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb570071650 0x7fb5700bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:27.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb5766d3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5700c5830 0x7fb5701ed900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:27.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb5766d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb570157c60 0x7fb570145030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:27.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb5766d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb5700c6760 con 0x7fb570157c60 2024-09-17T05:04:27.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb5766d3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb5700c65e0 con 0x7fb5700c5830 2024-09-17T05:04:27.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb5766d3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb570072120 con 0x7fb570071650 2024-09-17T05:04:27.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb574c70640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb570157c60 0x7fb570145030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:27.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb574c70640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb570157c60 0x7fb570145030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35662/0 (socket says 172.21.15.150:35662) 2024-09-17T05:04:27.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb574c70640 1 -- 172.21.15.150:0/1062141448 learned_addr learned my addr 172.21.15.150:0/1062141448 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:27.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb56f7fe640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5700c5830 0x7fb5701ed900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:27.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb574c70640 1 -- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb570071650 msgr2=0x7fb5700bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:27.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb56ffff640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb570071650 0x7fb5700bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:27.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb574c70640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb570071650 0x7fb5700bfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:27.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb574c70640 1 -- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5700c5830 msgr2=0x7fb5701ed900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:27.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb574c70640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5700c5830 0x7fb5701ed900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:27.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.291+0000 7fb574c70640 1 -- 172.21.15.150:0/1062141448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5700bee40 con 0x7fb570157c60 2024-09-17T05:04:27.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.295+0000 7fb574c70640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb570157c60 0x7fb570145030 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb56800e940 tx=0x7fb56800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:27.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.295+0000 7fb56cff9640 1 -- 172.21.15.150:0/1062141448 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb568004d30 con 0x7fb570157c60 2024-09-17T05:04:27.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.295+0000 7fb56cff9640 1 -- 172.21.15.150:0/1062141448 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb56800cb20 con 0x7fb570157c60 2024-09-17T05:04:27.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.295+0000 7fb5766d3640 1 -- 172.21.15.150:0/1062141448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb570156460 con 0x7fb570157c60 2024-09-17T05:04:27.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.295+0000 7fb56cff9640 1 -- 172.21.15.150:0/1062141448 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb568005020 con 0x7fb570157c60 2024-09-17T05:04:27.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.295+0000 7fb5766d3640 1 -- 172.21.15.150:0/1062141448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5700c0160 con 0x7fb570157c60 2024-09-17T05:04:27.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.295+0000 7fb5766d3640 1 -- 172.21.15.150:0/1062141448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb57014b0b0 con 0x7fb570157c60 2024-09-17T05:04:27.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.295+0000 7fb56cff9640 1 -- 172.21.15.150:0/1062141448 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb5680051c0 con 0x7fb570157c60 2024-09-17T05:04:27.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.295+0000 7fb56cff9640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb55c073230 0x7fb55c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:27.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.295+0000 7fb56cff9640 1 -- 172.21.15.150:0/1062141448 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7fb5680954c0 con 0x7fb570157c60 2024-09-17T05:04:27.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.295+0000 7fb56ffff640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb55c073230 0x7fb55c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:27.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.295+0000 7fb56ffff640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb55c073230 0x7fb55c075740 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb560001570 tx=0x7fb560009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:27.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.299+0000 7fb56cff9640 1 -- 172.21.15.150:0/1062141448 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb568016030 con 0x7fb570157c60 2024-09-17T05:04:27.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.507+0000 7fb5766d3640 1 -- 172.21.15.150:0/1062141448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb570155150 con 0x7fb570157c60 2024-09-17T05:04:27.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.507+0000 7fb56cff9640 1 -- 172.21.15.150:0/1062141448 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+6542 (secure 0 0 0) 0x7fb56805faf0 con 0x7fb570157c60 2024-09-17T05:04:27.506 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:27.506 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":106,"btime":"2024-09-17T05:04:18:543254+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6564,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:27.507 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 106 2024-09-17T05:04:27.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.511+0000 7fb5766d3640 1 -- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb55c073230 msgr2=0x7fb55c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:27.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.511+0000 7fb5766d3640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb55c073230 0x7fb55c075740 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb560001570 tx=0x7fb560009040 comp rx=0 tx=0).stop 2024-09-17T05:04:27.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.511+0000 7fb5766d3640 1 -- 172.21.15.150:0/1062141448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb570157c60 msgr2=0x7fb570145030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:27.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.511+0000 7fb5766d3640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb570157c60 0x7fb570145030 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb56800e940 tx=0x7fb56800ee00 comp rx=0 tx=0).stop 2024-09-17T05:04:27.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.511+0000 7fb5766d3640 1 -- 172.21.15.150:0/1062141448 shutdown_connections 2024-09-17T05:04:27.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.511+0000 7fb5766d3640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb55c073230 0x7fb55c075740 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:27.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.511+0000 7fb5766d3640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5700c5830 0x7fb5701ed900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:27.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.511+0000 7fb5766d3640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb570157c60 0x7fb570145030 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:27.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.511+0000 7fb5766d3640 1 --2- 172.21.15.150:0/1062141448 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb570071650 0x7fb5700bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:27.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.511+0000 7fb5766d3640 1 -- 172.21.15.150:0/1062141448 >> 172.21.15.150:0/1062141448 conn(0x7fb5700ca460 msgr2=0x7fb570149120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:27.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.511+0000 7fb5766d3640 1 -- 172.21.15.150:0/1062141448 shutdown_connections 2024-09-17T05:04:27.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:27.511+0000 7fb5766d3640 1 -- 172.21.15.150:0/1062141448 wait complete. 2024-09-17T05:04:27.528 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:04:29.529 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:29.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.699+0000 7f01c01c4640 1 -- 172.21.15.150:0/1629119758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b8151d00 msgr2=0x7f01b80c3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:29.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.699+0000 7f01c01c4640 1 --2- 172.21.15.150:0/1629119758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b8151d00 0x7f01b80c3430 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f01b80adcf0 tx=0x7f01b401c8c0 comp rx=0 tx=0).stop 2024-09-17T05:04:29.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.699+0000 7f01c01c4640 1 -- 172.21.15.150:0/1629119758 shutdown_connections 2024-09-17T05:04:29.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.699+0000 7f01c01c4640 1 --2- 172.21.15.150:0/1629119758 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01b8157c30 0x7f01b8158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:29.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.699+0000 7f01c01c4640 1 --2- 172.21.15.150:0/1629119758 >> v2:172.21.15.176:3300/0 conn(0x7f01b80c3970 0x7f01b80c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:29.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.699+0000 7f01c01c4640 1 --2- 172.21.15.150:0/1629119758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b8151d00 0x7f01b80c3430 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:29.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.699+0000 7f01c01c4640 1 -- 172.21.15.150:0/1629119758 >> 172.21.15.150:0/1629119758 conn(0x7f01b80c4a00 msgr2=0x7f01b80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:29.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.699+0000 7f01c01c4640 1 -- 172.21.15.150:0/1629119758 shutdown_connections 2024-09-17T05:04:29.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.699+0000 7f01c01c4640 1 -- 172.21.15.150:0/1629119758 wait complete. 2024-09-17T05:04:29.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.699+0000 7f01c01c4640 1 Processor -- start 2024-09-17T05:04:29.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01c01c4640 1 -- start start 2024-09-17T05:04:29.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01c01c4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01b80c3970 0x7f01b8147ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:29.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01c01c4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01b8151d00 0x7f01b814cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:29.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01c01c4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b8157c30 0x7f01b8146480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:29.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01c01c4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f01b80c4630 con 0x7f01b8157c30 2024-09-17T05:04:29.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01c01c4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f01b80c44b0 con 0x7f01b8151d00 2024-09-17T05:04:29.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01c01c4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f01b8154ea0 con 0x7f01b80c3970 2024-09-17T05:04:29.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01be761640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b8157c30 0x7f01b8146480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:29.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01be761640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b8157c30 0x7f01b8146480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51328/0 (socket says 172.21.15.150:51328) 2024-09-17T05:04:29.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01be761640 1 -- 172.21.15.150:0/48967042 learned_addr learned my addr 172.21.15.150:0/48967042 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:29.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01bdf60640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01b80c3970 0x7f01b8147ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:29.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01bd75f640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01b8151d00 0x7f01b814cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:29.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01be761640 1 -- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01b80c3970 msgr2=0x7f01b8147ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:29.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01be761640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01b80c3970 0x7f01b8147ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:29.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01be761640 1 -- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01b8151d00 msgr2=0x7f01b814cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:29.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01be761640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01b8151d00 0x7f01b814cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:29.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01be761640 1 -- 172.21.15.150:0/48967042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b814e810 con 0x7f01b8157c30 2024-09-17T05:04:29.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01be761640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b8157c30 0x7f01b8146480 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f01b000ca30 tx=0x7f01b000cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:29.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01ae7fc640 1 -- 172.21.15.150:0/48967042 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b0007d90 con 0x7f01b8157c30 2024-09-17T05:04:29.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01ae7fc640 1 -- 172.21.15.150:0/48967042 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01b0004c20 con 0x7f01b8157c30 2024-09-17T05:04:29.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01c01c4640 1 -- 172.21.15.150:0/48967042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01b81eebe0 con 0x7f01b8157c30 2024-09-17T05:04:29.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01ae7fc640 1 -- 172.21.15.150:0/48967042 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b0005280 con 0x7f01b8157c30 2024-09-17T05:04:29.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01c01c4640 1 -- 172.21.15.150:0/48967042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01b8147180 con 0x7f01b8157c30 2024-09-17T05:04:29.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01c01c4640 1 -- 172.21.15.150:0/48967042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01b81552b0 con 0x7f01b8157c30 2024-09-17T05:04:29.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.703+0000 7f01ae7fc640 1 -- 172.21.15.150:0/48967042 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f01b0005420 con 0x7f01b8157c30 2024-09-17T05:04:29.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.707+0000 7f01ae7fc640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f019c073200 0x7f019c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:29.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.707+0000 7f01ae7fc640 1 -- 172.21.15.150:0/48967042 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7f01b00956c0 con 0x7f01b8157c30 2024-09-17T05:04:29.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.707+0000 7f01bdf60640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f019c073200 0x7f019c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:29.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.707+0000 7f01bdf60640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f019c073200 0x7f019c075710 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f01b40af000 tx=0x7f01b4019660 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:29.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.711+0000 7f01ae7fc640 1 -- 172.21.15.150:0/48967042 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f01b0014070 con 0x7f01b8157c30 2024-09-17T05:04:29.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.915+0000 7f01c01c4640 1 -- 172.21.15.150:0/48967042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01b8147480 con 0x7f01b8157c30 2024-09-17T05:04:29.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01ae7fc640 1 -- 172.21.15.150:0/48967042 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+6542 (secure 0 0 0) 0x7f01b005fd50 con 0x7f01b8157c30 2024-09-17T05:04:29.917 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:29.917 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":106,"btime":"2024-09-17T05:04:18:543254+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6564,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:29.917 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 106 2024-09-17T05:04:29.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01c01c4640 1 -- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f019c073200 msgr2=0x7f019c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:29.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01c01c4640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f019c073200 0x7f019c075710 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f01b40af000 tx=0x7f01b4019660 comp rx=0 tx=0).stop 2024-09-17T05:04:29.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01c01c4640 1 -- 172.21.15.150:0/48967042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b8157c30 msgr2=0x7f01b8146480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:29.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01c01c4640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b8157c30 0x7f01b8146480 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f01b000ca30 tx=0x7f01b000cef0 comp rx=0 tx=0).stop 2024-09-17T05:04:29.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01c01c4640 1 -- 172.21.15.150:0/48967042 shutdown_connections 2024-09-17T05:04:29.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01c01c4640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f019c073200 0x7f019c075710 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:29.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01c01c4640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01b8151d00 0x7f01b814cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:29.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01c01c4640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01b8157c30 0x7f01b8146480 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:29.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01c01c4640 1 --2- 172.21.15.150:0/48967042 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f01b80c3970 0x7f01b8147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:29.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01c01c4640 1 -- 172.21.15.150:0/48967042 >> 172.21.15.150:0/48967042 conn(0x7f01b80c4a00 msgr2=0x7f01b80a6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:29.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01c01c4640 1 -- 172.21.15.150:0/48967042 shutdown_connections 2024-09-17T05:04:29.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:29.919+0000 7f01c01c4640 1 -- 172.21.15.150:0/48967042 wait complete. 2024-09-17T05:04:29.938 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:04:31.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:32.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 -- 172.21.15.150:0/813035420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1b0003a10 msgr2=0x7fd1b0003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:32.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 --2- 172.21.15.150:0/813035420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1b0003a10 0x7fd1b0003e80 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc009b50 tx=0x7fd1bc01d290 comp rx=0 tx=0).stop 2024-09-17T05:04:32.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cd1a2640 1 -- 172.21.15.150:0/813035420 reap_dead start 2024-09-17T05:04:32.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 -- 172.21.15.150:0/813035420 shutdown_connections 2024-09-17T05:04:32.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 -- 172.21.15.150:0/813035420 >> 172.21.15.150:0/813035420 conn(0x7fd1c80c9de0 msgr2=0x7fd1c80ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:32.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 -- 172.21.15.150:0/813035420 shutdown_connections 2024-09-17T05:04:32.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 -- 172.21.15.150:0/813035420 wait complete. 2024-09-17T05:04:32.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 Processor -- start 2024-09-17T05:04:32.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 -- start start 2024-09-17T05:04:32.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1b0003a10 0x7fd1c815e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:32.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1c8150450 0x7fd1c8150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:32.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd1c8146800 0x7fd1c8146c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:32.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c80c7e60 con 0x7fd1b0003a10 2024-09-17T05:04:32.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c80c7ce0 con 0x7fd1c8150450 2024-09-17T05:04:32.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cec05640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c815f030 con 0x7fd1c8146800 2024-09-17T05:04:32.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cc9a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1b0003a10 0x7fd1c815e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:32.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cc9a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1b0003a10 0x7fd1c815e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51346/0 (socket says 172.21.15.150:51346) 2024-09-17T05:04:32.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cc9a1640 1 -- 172.21.15.150:0/1422074357 learned_addr learned my addr 172.21.15.150:0/1422074357 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:32.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1c7fff640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1c8150450 0x7fd1c8150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:32.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cd1a2640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd1c8146800 0x7fd1c8146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:32.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cc9a1640 1 -- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd1c8146800 msgr2=0x7fd1c8146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:32.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cc9a1640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd1c8146800 0x7fd1c8146c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:32.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cc9a1640 1 -- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1c8150450 msgr2=0x7fd1c8150860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:32.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cc9a1640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1c8150450 0x7fd1c8150860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:32.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.107+0000 7fd1cc9a1640 1 -- 172.21.15.150:0/1422074357 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c814c8e0 con 0x7fd1b0003a10 2024-09-17T05:04:32.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.111+0000 7fd1cc9a1640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1b0003a10 0x7fd1c815e510 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc009aa0 tx=0x7fd1bc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:32.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.111+0000 7fd1c57fa640 1 -- 172.21.15.150:0/1422074357 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc004d00 con 0x7fd1b0003a10 2024-09-17T05:04:32.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.111+0000 7fd1c57fa640 1 -- 172.21.15.150:0/1422074357 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1bc005030 con 0x7fd1b0003a10 2024-09-17T05:04:32.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.111+0000 7fd1cec05640 1 -- 172.21.15.150:0/1422074357 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c8157360 con 0x7fd1b0003a10 2024-09-17T05:04:32.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.111+0000 7fd1c57fa640 1 -- 172.21.15.150:0/1422074357 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc005440 con 0x7fd1b0003a10 2024-09-17T05:04:32.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.111+0000 7fd1cec05640 1 -- 172.21.15.150:0/1422074357 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c814c3c0 con 0x7fd1b0003a10 2024-09-17T05:04:32.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.111+0000 7fd1cec05640 1 -- 172.21.15.150:0/1422074357 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1c80c6230 con 0x7fd1b0003a10 2024-09-17T05:04:32.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.111+0000 7fd1c57fa640 1 -- 172.21.15.150:0/1422074357 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd1bc0bb440 con 0x7fd1b0003a10 2024-09-17T05:04:32.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.111+0000 7fd1c57fa640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd1b4073200 0x7fd1b4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:32.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.111+0000 7fd1c57fa640 1 -- 172.21.15.150:0/1422074357 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7fd1bc12f780 con 0x7fd1b0003a10 2024-09-17T05:04:32.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.111+0000 7fd1c7fff640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd1b4073200 0x7fd1b4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:32.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.115+0000 7fd1c7fff640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd1b4073200 0x7fd1b4075710 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80058d0 tx=0x7fd1b8005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:32.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.115+0000 7fd1c57fa640 1 -- 172.21.15.150:0/1422074357 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd1bc016fa0 con 0x7fd1b0003a10 2024-09-17T05:04:32.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:04:32.260 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:04:32.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.307+0000 7fd1cec05640 1 -- 172.21.15.150:0/1422074357 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1c81576f0 con 0x7fd1b0003a10 2024-09-17T05:04:32.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.307+0000 7fd1c57fa640 1 -- 172.21.15.150:0/1422074357 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+6542 (secure 0 0 0) 0x7fd1bc0f9e10 con 0x7fd1b0003a10 2024-09-17T05:04:32.307 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:32.308 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":106,"btime":"2024-09-17T05:04:18:543254+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6564,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:32.308 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 106 2024-09-17T05:04:32.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.311+0000 7fd1cec05640 1 -- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd1b4073200 msgr2=0x7fd1b4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:32.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.311+0000 7fd1cec05640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd1b4073200 0x7fd1b4075710 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80058d0 tx=0x7fd1b8005860 comp rx=0 tx=0).stop 2024-09-17T05:04:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.311+0000 7fd1cec05640 1 -- 172.21.15.150:0/1422074357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1b0003a10 msgr2=0x7fd1c815e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.311+0000 7fd1cec05640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1b0003a10 0x7fd1c815e510 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc009aa0 tx=0x7fd1bc005d50 comp rx=0 tx=0).stop 2024-09-17T05:04:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.311+0000 7fd1cec05640 1 -- 172.21.15.150:0/1422074357 shutdown_connections 2024-09-17T05:04:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.311+0000 7fd1cec05640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd1b4073200 0x7fd1b4075710 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.311+0000 7fd1cec05640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd1c8150450 0x7fd1c8150860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.311+0000 7fd1cec05640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd1c8146800 0x7fd1c8146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.311+0000 7fd1cec05640 1 --2- 172.21.15.150:0/1422074357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1b0003a10 0x7fd1c815e510 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.311+0000 7fd1cec05640 1 -- 172.21.15.150:0/1422074357 >> 172.21.15.150:0/1422074357 conn(0x7fd1c80c9de0 msgr2=0x7fd1c81580b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.311+0000 7fd1cec05640 1 -- 172.21.15.150:0/1422074357 shutdown_connections 2024-09-17T05:04:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:32.311+0000 7fd1cec05640 1 -- 172.21.15.150:0/1422074357 wait complete. 2024-09-17T05:04:32.328 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:04:34.329 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:34.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.495+0000 7fe4ab14d640 1 -- 172.21.15.150:0/2200936400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe48c004d50 msgr2=0x7fe48c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.495+0000 7fe4ab14d640 1 --2- 172.21.15.150:0/2200936400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe48c004d50 0x7fe48c005160 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fe49c009eb0 tx=0x7fe49c01d5d0 comp rx=0 tx=0).stop 2024-09-17T05:04:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.495+0000 7fe4a96ea640 1 -- 172.21.15.150:0/2200936400 reap_dead start 2024-09-17T05:04:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.495+0000 7fe4ab14d640 1 -- 172.21.15.150:0/2200936400 shutdown_connections 2024-09-17T05:04:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.495+0000 7fe4ab14d640 1 -- 172.21.15.150:0/2200936400 >> 172.21.15.150:0/2200936400 conn(0x7fe4a40c4a00 msgr2=0x7fe4a40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.495+0000 7fe4ab14d640 1 -- 172.21.15.150:0/2200936400 shutdown_connections 2024-09-17T05:04:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.495+0000 7fe4ab14d640 1 -- 172.21.15.150:0/2200936400 wait complete. 2024-09-17T05:04:34.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4ab14d640 1 Processor -- start 2024-09-17T05:04:34.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4ab14d640 1 -- start start 2024-09-17T05:04:34.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4ab14d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4a4073440 0x7fe4a4157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:34.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4ab14d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4a414daa0 0x7fe4a41ed0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:34.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4ab14d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4a4158550 0x7fe4a4158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:34.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4ab14d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a40add70 con 0x7fe4a414daa0 2024-09-17T05:04:34.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4ab14d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a40c3000 con 0x7fe4a4073440 2024-09-17T05:04:34.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4ab14d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe4a40a75f0 con 0x7fe4a4158550 2024-09-17T05:04:34.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4a414daa0 0x7fe4a41ed0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:34.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4a414daa0 0x7fe4a41ed0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51372/0 (socket says 172.21.15.150:51372) 2024-09-17T05:04:34.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a3fff640 1 -- 172.21.15.150:0/159562375 learned_addr learned my addr 172.21.15.150:0/159562375 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:34.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a8ee9640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4a4073440 0x7fe4a4157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:34.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a96ea640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4a4158550 0x7fe4a4158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:34.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a3fff640 1 -- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4a4158550 msgr2=0x7fe4a4158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:34.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a3fff640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4a4158550 0x7fe4a4158960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:34.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a3fff640 1 -- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4a4073440 msgr2=0x7fe4a4157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:34.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a3fff640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4a4073440 0x7fe4a4157a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:34.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a3fff640 1 -- 172.21.15.150:0/159562375 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a41ec8c0 con 0x7fe4a414daa0 2024-09-17T05:04:34.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a3fff640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4a414daa0 0x7fe4a41ed0a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fe494000fe0 tx=0x7fe49400df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:34.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a17fa640 1 -- 172.21.15.150:0/159562375 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe494018070 con 0x7fe4a414daa0 2024-09-17T05:04:34.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a17fa640 1 -- 172.21.15.150:0/159562375 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe4940052e0 con 0x7fe4a414daa0 2024-09-17T05:04:34.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4ab14d640 1 -- 172.21.15.150:0/159562375 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a414e470 con 0x7fe4a414daa0 2024-09-17T05:04:34.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.499+0000 7fe4a17fa640 1 -- 172.21.15.150:0/159562375 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4940036e0 con 0x7fe4a414daa0 2024-09-17T05:04:34.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.503+0000 7fe4ab14d640 1 -- 172.21.15.150:0/159562375 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a40c2140 con 0x7fe4a414daa0 2024-09-17T05:04:34.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.503+0000 7fe4ab14d640 1 -- 172.21.15.150:0/159562375 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4a41ecce0 con 0x7fe4a414daa0 2024-09-17T05:04:34.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.503+0000 7fe4a17fa640 1 -- 172.21.15.150:0/159562375 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe49401b070 con 0x7fe4a414daa0 2024-09-17T05:04:34.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.503+0000 7fe4a17fa640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe490073200 0x7fe490075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:34.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.503+0000 7fe4a17fa640 1 -- 172.21.15.150:0/159562375 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7fe494094250 con 0x7fe4a414daa0 2024-09-17T05:04:34.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.503+0000 7fe4a8ee9640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe490073200 0x7fe490075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:34.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.503+0000 7fe4a8ee9640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe490073200 0x7fe490075710 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe498005d10 tx=0x7fe498005c80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:34.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.507+0000 7fe4a17fa640 1 -- 172.21.15.150:0/159562375 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe49405e2c0 con 0x7fe4a414daa0 2024-09-17T05:04:34.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.711+0000 7fe4ab14d640 1 -- 172.21.15.150:0/159562375 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4a40ad2a0 con 0x7fe4a414daa0 2024-09-17T05:04:34.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.711+0000 7fe4a17fa640 1 -- 172.21.15.150:0/159562375 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+6542 (secure 0 0 0) 0x7fe494062cf0 con 0x7fe4a414daa0 2024-09-17T05:04:34.711 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:34.711 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":106,"btime":"2024-09-17T05:04:18:543254+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6564,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:03:04.151355+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6010":{"gid":6010,"name":"d","rank":0,"incarnation":97,"state":"up:active","state_seq":8,"addr":"172.21.15.176:6837/903190072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":903190072},{"type":"v1","addr":"172.21.15.176:6837","nonce":903190072}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6010,"qdb_cluster":[6010]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:34.711 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 106 2024-09-17T05:04:34.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.715+0000 7fe4ab14d640 1 -- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe490073200 msgr2=0x7fe490075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:34.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.715+0000 7fe4ab14d640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe490073200 0x7fe490075710 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe498005d10 tx=0x7fe498005c80 comp rx=0 tx=0).stop 2024-09-17T05:04:34.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.715+0000 7fe4ab14d640 1 -- 172.21.15.150:0/159562375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4a414daa0 msgr2=0x7fe4a41ed0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.715+0000 7fe4ab14d640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4a414daa0 0x7fe4a41ed0a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fe494000fe0 tx=0x7fe49400df80 comp rx=0 tx=0).stop 2024-09-17T05:04:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.715+0000 7fe4ab14d640 1 -- 172.21.15.150:0/159562375 shutdown_connections 2024-09-17T05:04:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.715+0000 7fe4ab14d640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe490073200 0x7fe490075710 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.715+0000 7fe4ab14d640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe4a4158550 0x7fe4a4158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.715+0000 7fe4ab14d640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4a414daa0 0x7fe4a41ed0a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.715+0000 7fe4ab14d640 1 --2- 172.21.15.150:0/159562375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4a4073440 0x7fe4a4157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.715+0000 7fe4ab14d640 1 -- 172.21.15.150:0/159562375 >> 172.21.15.150:0/159562375 conn(0x7fe4a40c4a00 msgr2=0x7fe4a40c5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.715+0000 7fe4ab14d640 1 -- 172.21.15.150:0/159562375 shutdown_connections 2024-09-17T05:04:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:34.715+0000 7fe4ab14d640 1 -- 172.21.15.150:0/159562375 wait complete. 2024-09-17T05:04:34.732 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:04:36.733 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:36.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.891+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/549075247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b4004d50 msgr2=0x7fd2b4005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:36.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.891+0000 7fd2cf3ec640 1 --2- 172.21.15.150:0/549075247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b4004d50 0x7fd2b4005160 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009eb0 tx=0x7fd2c001d5d0 comp rx=0 tx=0).stop 2024-09-17T05:04:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.891+0000 7fd2cd989640 1 -- 172.21.15.150:0/549075247 reap_dead start 2024-09-17T05:04:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.891+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/549075247 shutdown_connections 2024-09-17T05:04:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.891+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/549075247 >> 172.21.15.150:0/549075247 conn(0x7fd2c80c4a00 msgr2=0x7fd2c80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.891+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/549075247 shutdown_connections 2024-09-17T05:04:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.891+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/549075247 wait complete. 2024-09-17T05:04:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cf3ec640 1 Processor -- start 2024-09-17T05:04:36.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cf3ec640 1 -- start start 2024-09-17T05:04:36.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cf3ec640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b4004d50 0x7fd2c8157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:36.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cf3ec640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2c814dec0 0x7fd2c814ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:36.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cf3ec640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2c814f2d0 0x7fd2c8152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:36.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cf3ec640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c80add70 con 0x7fd2b4004d50 2024-09-17T05:04:36.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cf3ec640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c80c3000 con 0x7fd2c814dec0 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cf3ec640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c80ad9b0 con 0x7fd2c814f2d0 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cd188640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b4004d50 0x7fd2c8157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cd188640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b4004d50 0x7fd2c8157f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51396/0 (socket says 172.21.15.150:51396) 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cd188640 1 -- 172.21.15.150:0/2926374314 learned_addr learned my addr 172.21.15.150:0/2926374314 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cd989640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2c814f2d0 0x7fd2c8152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cc987640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2c814dec0 0x7fd2c814ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cd188640 1 -- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2c814f2d0 msgr2=0x7fd2c8152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cd188640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2c814f2d0 0x7fd2c8152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cd188640 1 -- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2c814dec0 msgr2=0x7fd2c814ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cd188640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2c814dec0 0x7fd2c814ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cd188640 1 -- 172.21.15.150:0/2926374314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c8148020 con 0x7fd2b4004d50 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cd188640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b4004d50 0x7fd2c8157f30 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80036b0 tx=0x7fd2b8003ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2bdffb640 1 -- 172.21.15.150:0/2926374314 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8015070 con 0x7fd2b4004d50 2024-09-17T05:04:36.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2bdffb640 1 -- 172.21.15.150:0/2926374314 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2b800f980 con 0x7fd2b4004d50 2024-09-17T05:04:36.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/2926374314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c8158bc0 con 0x7fd2b4004d50 2024-09-17T05:04:36.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2bdffb640 1 -- 172.21.15.150:0/2926374314 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b800fdd0 con 0x7fd2b4004d50 2024-09-17T05:04:36.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/2926374314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c81483f0 con 0x7fd2b4004d50 2024-09-17T05:04:36.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.895+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/2926374314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c81485c0 con 0x7fd2b4004d50 2024-09-17T05:04:36.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.899+0000 7fd2bdffb640 1 -- 172.21.15.150:0/2926374314 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd2b8011430 con 0x7fd2b4004d50 2024-09-17T05:04:36.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.899+0000 7fd2bdffb640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd2ac073200 0x7fd2ac075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:36.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.899+0000 7fd2bdffb640 1 -- 172.21.15.150:0/2926374314 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7fd2b8095980 con 0x7fd2b4004d50 2024-09-17T05:04:36.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.899+0000 7fd2cc987640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd2ac073200 0x7fd2ac075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:36.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.899+0000 7fd2cc987640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd2ac073200 0x7fd2ac075710 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd2c806d980 tx=0x7fd2c40091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:36.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:36.903+0000 7fd2bdffb640 1 -- 172.21.15.150:0/2926374314 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd2b805f970 con 0x7fd2b4004d50 2024-09-17T05:04:37.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.119+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/2926374314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2c80a75f0 con 0x7fd2b4004d50 2024-09-17T05:04:37.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.119+0000 7fd2bdffb640 1 -- 172.21.15.150:0/2926374314 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+5713 (secure 0 0 0) 0x7fd2b80643a0 con 0x7fd2b4004d50 2024-09-17T05:04:37.119 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:37.119 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":108,"btime":"2024-09-17T05:04:37:037207+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:36.414718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:37.119 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 108 2024-09-17T05:04:37.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.123+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd2ac073200 msgr2=0x7fd2ac075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:37.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.123+0000 7fd2cf3ec640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd2ac073200 0x7fd2ac075710 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd2c806d980 tx=0x7fd2c40091e0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.123+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/2926374314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b4004d50 msgr2=0x7fd2c8157f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:37.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.123+0000 7fd2cf3ec640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b4004d50 0x7fd2c8157f30 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80036b0 tx=0x7fd2b8003ab0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.123+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/2926374314 shutdown_connections 2024-09-17T05:04:37.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.123+0000 7fd2cf3ec640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd2ac073200 0x7fd2ac075710 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.123+0000 7fd2cf3ec640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2c814dec0 0x7fd2c814ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.123+0000 7fd2cf3ec640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2c814f2d0 0x7fd2c8152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.123+0000 7fd2cf3ec640 1 --2- 172.21.15.150:0/2926374314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2b4004d50 0x7fd2c8157f30 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.123+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/2926374314 >> 172.21.15.150:0/2926374314 conn(0x7fd2c80c4a00 msgr2=0x7fd2c80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:37.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.123+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/2926374314 shutdown_connections 2024-09-17T05:04:37.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.123+0000 7fd2cf3ec640 1 -- 172.21.15.150:0/2926374314 wait complete. 2024-09-17T05:04:37.143 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-17T05:04:37.143 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:04:37.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:37.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.295+0000 7f3c87807640 1 -- 172.21.15.150:0/1806124565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c005360 msgr2=0x7f3c6c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:37.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.295+0000 7f3c87807640 1 --2- 172.21.15.150:0/1806124565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c005360 0x7f3c6c00b9d0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3c78001dc0 tx=0x7f3c7801cb30 comp rx=0 tx=0).stop 2024-09-17T05:04:37.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.295+0000 7f3c87807640 1 -- 172.21.15.150:0/1806124565 shutdown_connections 2024-09-17T05:04:37.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.295+0000 7f3c87807640 1 --2- 172.21.15.150:0/1806124565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c005360 0x7f3c6c00b9d0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.295+0000 7f3c87807640 1 -- 172.21.15.150:0/1806124565 >> 172.21.15.150:0/1806124565 conn(0x7f3c800c4a00 msgr2=0x7f3c800c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:37.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.295+0000 7f3c87807640 1 -- 172.21.15.150:0/1806124565 shutdown_connections 2024-09-17T05:04:37.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.295+0000 7f3c87807640 1 -- 172.21.15.150:0/1806124565 wait complete. 2024-09-17T05:04:37.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c87807640 1 Processor -- start 2024-09-17T05:04:37.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c87807640 1 -- start start 2024-09-17T05:04:37.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c87807640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c005360 0x7f3c800abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:37.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c87807640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c800c1330 0x7f3c8014a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:37.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c87807640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c800c1740 0x7f3c8014a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:37.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c87807640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8006f250 con 0x7f3c6c005360 2024-09-17T05:04:37.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c87807640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3c800adef0 con 0x7f3c800c1740 2024-09-17T05:04:37.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c87807640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8006f810 con 0x7f3c800c1330 2024-09-17T05:04:37.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c855a3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c005360 0x7f3c800abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:37.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c855a3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c005360 0x7f3c800abb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51420/0 (socket says 172.21.15.150:51420) 2024-09-17T05:04:37.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c855a3640 1 -- 172.21.15.150:0/984759847 learned_addr learned my addr 172.21.15.150:0/984759847 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c85da4640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c800c1740 0x7f3c8014a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c84da2640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c800c1330 0x7f3c8014a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c855a3640 1 -- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c800c1330 msgr2=0x7f3c8014a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c855a3640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c800c1330 0x7f3c8014a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c855a3640 1 -- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c800c1740 msgr2=0x7f3c8014a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c855a3640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c800c1740 0x7f3c8014a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c855a3640 1 -- 172.21.15.150:0/984759847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8014bd10 con 0x7f3c6c005360 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c855a3640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c005360 0x7f3c800abb80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c012690 tx=0x7f3c7c012a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c75ffb640 1 -- 172.21.15.150:0/984759847 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c013920 con 0x7f3c6c005360 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c75ffb640 1 -- 172.21.15.150:0/984759847 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c7c013ac0 con 0x7f3c6c005360 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c87807640 1 -- 172.21.15.150:0/984759847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c8014adf0 con 0x7f3c6c005360 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c75ffb640 1 -- 172.21.15.150:0/984759847 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c010480 con 0x7f3c6c005360 2024-09-17T05:04:37.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.299+0000 7f3c87807640 1 -- 172.21.15.150:0/984759847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c800ac0c0 con 0x7f3c6c005360 2024-09-17T05:04:37.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.303+0000 7f3c87807640 1 -- 172.21.15.150:0/984759847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c800ad6d0 con 0x7f3c6c005360 2024-09-17T05:04:37.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.303+0000 7f3c75ffb640 1 -- 172.21.15.150:0/984759847 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3c7c010620 con 0x7f3c6c005360 2024-09-17T05:04:37.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.303+0000 7f3c75ffb640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c64073230 0x7f3c64075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:37.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.303+0000 7f3c75ffb640 1 -- 172.21.15.150:0/984759847 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f3c7c098aa0 con 0x7f3c6c005360 2024-09-17T05:04:37.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.303+0000 7f3c84da2640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c64073230 0x7f3c64075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:37.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.303+0000 7f3c84da2640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c64073230 0x7f3c64075740 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3c801ea2a0 tx=0x7f3c700091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:37.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.307+0000 7f3c75ffb640 1 -- 172.21.15.150:0/984759847 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3c7c062a90 con 0x7f3c6c005360 2024-09-17T05:04:37.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.503+0000 7f3c87807640 1 -- 172.21.15.150:0/984759847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c800c60f0 con 0x7f3c6c005360 2024-09-17T05:04:37.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.503+0000 7f3c75ffb640 1 -- 172.21.15.150:0/984759847 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+5713 (secure 0 0 0) 0x7f3c7c0674c0 con 0x7f3c6c005360 2024-09-17T05:04:37.503 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:37.503 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":108,"btime":"2024-09-17T05:04:37:037207+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:36.414718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:37.503 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 108 2024-09-17T05:04:37.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.507+0000 7f3c87807640 1 -- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c64073230 msgr2=0x7f3c64075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:37.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.507+0000 7f3c87807640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c64073230 0x7f3c64075740 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3c801ea2a0 tx=0x7f3c700091e0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.507+0000 7f3c87807640 1 -- 172.21.15.150:0/984759847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c005360 msgr2=0x7f3c800abb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:37.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.507+0000 7f3c87807640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c005360 0x7f3c800abb80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c012690 tx=0x7f3c7c012a90 comp rx=0 tx=0).stop 2024-09-17T05:04:37.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.507+0000 7f3c87807640 1 -- 172.21.15.150:0/984759847 shutdown_connections 2024-09-17T05:04:37.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.507+0000 7f3c87807640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c64073230 0x7f3c64075740 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.507+0000 7f3c87807640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c800c1740 0x7f3c8014a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.507+0000 7f3c87807640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c800c1330 0x7f3c8014a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.507+0000 7f3c87807640 1 --2- 172.21.15.150:0/984759847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c005360 0x7f3c800abb80 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:37.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.507+0000 7f3c87807640 1 -- 172.21.15.150:0/984759847 >> 172.21.15.150:0/984759847 conn(0x7f3c800c4a00 msgr2=0x7f3c801569a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:37.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.507+0000 7f3c87807640 1 -- 172.21.15.150:0/984759847 shutdown_connections 2024-09-17T05:04:37.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:37.507+0000 7f3c87807640 1 -- 172.21.15.150:0/984759847 wait complete. 2024-09-17T05:04:37.525 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:04:37:037207+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6564": { "addr": "172.21.15.150:6833/1984876738", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 1984876738, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 1984876738, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6564, "incarnation": 107, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:04:36.414718+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6564 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 105, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6495": { "addr": "172.21.15.150:6835/3807957687", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 3807957687, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 3807957687, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6495, "incarnation": 102, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:04:09.506271+0000", "qdb_cluster": [ 6495 ], "qdb_leader": 6495, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6495 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:04:39.527 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:39.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.703+0000 7fa9ec335640 1 -- 172.21.15.150:0/4174545176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9e40c5760 msgr2=0x7fa9e40c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:39.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.703+0000 7fa9ec335640 1 --2- 172.21.15.150:0/4174545176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9e40c5760 0x7fa9e40c5b70 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009840 tx=0x7fa9d401c700 comp rx=0 tx=0).stop 2024-09-17T05:04:39.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.703+0000 7fa9ec335640 1 -- 172.21.15.150:0/4174545176 shutdown_connections 2024-09-17T05:04:39.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.703+0000 7fa9ec335640 1 --2- 172.21.15.150:0/4174545176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9e40c5760 0x7fa9e40c5b70 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:39.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.703+0000 7fa9ec335640 1 -- 172.21.15.150:0/4174545176 >> 172.21.15.150:0/4174545176 conn(0x7fa9e40aaa80 msgr2=0x7fa9e40aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:39.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.703+0000 7fa9ec335640 1 -- 172.21.15.150:0/4174545176 shutdown_connections 2024-09-17T05:04:39.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.703+0000 7fa9ec335640 1 -- 172.21.15.150:0/4174545176 wait complete. 2024-09-17T05:04:39.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ec335640 1 Processor -- start 2024-09-17T05:04:39.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ec335640 1 -- start start 2024-09-17T05:04:39.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ec335640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9e40a7f10 0x7fa9e41f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:39.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ec335640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9e40c5760 0x7fa9e4155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:39.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ec335640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9e41535b0 0x7fa9e41529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:39.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ec335640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e40c5370 con 0x7fa9e40c5760 2024-09-17T05:04:39.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ec335640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e40718d0 con 0x7fa9e40a7f10 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ec335640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e40c54f0 con 0x7fa9e41535b0 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9e98d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9e40c5760 0x7fa9e4155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9e98d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9e40c5760 0x7fa9e4155f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38896/0 (socket says 172.21.15.150:38896) 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9e98d0640 1 -- 172.21.15.150:0/1517799355 learned_addr learned my addr 172.21.15.150:0/1517799355 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ea0d1640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9e40a7f10 0x7fa9e41f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ea8d2640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9e41535b0 0x7fa9e41529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9e98d0640 1 -- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9e41535b0 msgr2=0x7fa9e41529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9e98d0640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9e41535b0 0x7fa9e41529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9e98d0640 1 -- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9e40a7f10 msgr2=0x7fa9e41f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9e98d0640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9e40a7f10 0x7fa9e41f42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9e98d0640 1 -- 172.21.15.150:0/1517799355 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e4156450 con 0x7fa9e40c5760 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9e98d0640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9e40c5760 0x7fa9e4155f10 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4019310 tx=0x7fa9d40a6ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:39.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9daffd640 1 -- 172.21.15.150:0/1517799355 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40051c0 con 0x7fa9e40c5760 2024-09-17T05:04:39.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9daffd640 1 -- 172.21.15.150:0/1517799355 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa9d4005360 con 0x7fa9e40c5760 2024-09-17T05:04:39.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9daffd640 1 -- 172.21.15.150:0/1517799355 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40046d0 con 0x7fa9e40c5760 2024-09-17T05:04:39.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ec335640 1 -- 172.21.15.150:0/1517799355 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e4155170 con 0x7fa9e40c5760 2024-09-17T05:04:39.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ec335640 1 -- 172.21.15.150:0/1517799355 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e41f37c0 con 0x7fa9e40c5760 2024-09-17T05:04:39.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.707+0000 7fa9ec335640 1 -- 172.21.15.150:0/1517799355 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e40c6c50 con 0x7fa9e40c5760 2024-09-17T05:04:39.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.711+0000 7fa9daffd640 1 -- 172.21.15.150:0/1517799355 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa9d4004870 con 0x7fa9e40c5760 2024-09-17T05:04:39.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.711+0000 7fa9daffd640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa9c8073200 0x7fa9c8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:39.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.711+0000 7fa9daffd640 1 -- 172.21.15.150:0/1517799355 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7fa9d412ecf0 con 0x7fa9e40c5760 2024-09-17T05:04:39.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.711+0000 7fa9ea0d1640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa9c8073200 0x7fa9c8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:39.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.711+0000 7fa9ea0d1640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa9c8073200 0x7fa9c8075710 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0007fd0 tx=0x7fa9e0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:39.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.715+0000 7fa9daffd640 1 -- 172.21.15.150:0/1517799355 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa9d4133070 con 0x7fa9e40c5760 2024-09-17T05:04:39.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.943+0000 7fa9ec335640 1 -- 172.21.15.150:0/1517799355 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9e40c3070 con 0x7fa9e40c5760 2024-09-17T05:04:39.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9daffd640 1 -- 172.21.15.150:0/1517799355 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+5717 (secure 0 0 0) 0x7fa9d40ac030 con 0x7fa9e40c5760 2024-09-17T05:04:39.945 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:39.945 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":110,"btime":"2024-09-17T05:04:39:052857+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:39.052846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6564,"qdb_cluster":[6564]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:39.946 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 110 2024-09-17T05:04:39.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9ec335640 1 -- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa9c8073200 msgr2=0x7fa9c8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:39.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9ec335640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa9c8073200 0x7fa9c8075710 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0007fd0 tx=0x7fa9e0009040 comp rx=0 tx=0).stop 2024-09-17T05:04:39.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9ec335640 1 -- 172.21.15.150:0/1517799355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9e40c5760 msgr2=0x7fa9e4155f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:39.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9ec335640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9e40c5760 0x7fa9e4155f10 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4019310 tx=0x7fa9d40a6ff0 comp rx=0 tx=0).stop 2024-09-17T05:04:39.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9ec335640 1 -- 172.21.15.150:0/1517799355 shutdown_connections 2024-09-17T05:04:39.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9ec335640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa9c8073200 0x7fa9c8075710 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:39.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9ec335640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9e41535b0 0x7fa9e41529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:39.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9ec335640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9e40c5760 0x7fa9e4155f10 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:39.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9ec335640 1 --2- 172.21.15.150:0/1517799355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9e40a7f10 0x7fa9e41f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:39.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9ec335640 1 -- 172.21.15.150:0/1517799355 >> 172.21.15.150:0/1517799355 conn(0x7fa9e40aaa80 msgr2=0x7fa9e40c7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:39.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9ec335640 1 -- 172.21.15.150:0/1517799355 shutdown_connections 2024-09-17T05:04:39.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:39.947+0000 7fa9ec335640 1 -- 172.21.15.150:0/1517799355 wait complete. 2024-09-17T05:04:39.972 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6010 2024-09-17T05:04:39.973 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.d 2024-09-17T05:04:42.973 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-17T05:04:42.974 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T05:04:42.974 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T05:04:42.977 INFO:tasks.ceph.mds.d:Started 2024-09-17T05:04:42.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:43.093 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-09-17T05:04:43.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.170+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/818484842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0c815bfd0 msgr2=0x7fa0c815e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:43.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.170+0000 7fa0cf5b6640 1 --2- 172.21.15.150:0/818484842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0c815bfd0 0x7fa0c815e3f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009bc0 tx=0x7fa0c001f3b0 comp rx=0 tx=0).stop 2024-09-17T05:04:43.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.170+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/818484842 shutdown_connections 2024-09-17T05:04:43.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.170+0000 7fa0cf5b6640 1 --2- 172.21.15.150:0/818484842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0c815bfd0 0x7fa0c815e3f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:43.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.170+0000 7fa0cf5b6640 1 --2- 172.21.15.150:0/818484842 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0c8153b60 0x7fa0c8153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:43.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.170+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/818484842 >> 172.21.15.150:0/818484842 conn(0x7fa0c80c92d0 msgr2=0x7fa0c80c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:43.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.170+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/818484842 shutdown_connections 2024-09-17T05:04:43.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.170+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/818484842 wait complete. 2024-09-17T05:04:43.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.170+0000 7fa0cf5b6640 1 Processor -- start 2024-09-17T05:04:43.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.170+0000 7fa0cf5b6640 1 -- start start 2024-09-17T05:04:43.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cf5b6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0c8150bb0 0x7fa0c81ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:43.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cf5b6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0c8153b60 0x7fa0c8073e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:43.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cf5b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0c815bfd0 0x7fa0c80acf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cf5b6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c806ce50 con 0x7fa0c815bfd0 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cf5b6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c8151210 con 0x7fa0c8153b60 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cf5b6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa0c81544b0 con 0x7fa0c8150bb0 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cdb53640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0c815bfd0 0x7fa0c80acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cd352640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0c8150bb0 0x7fa0c81ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0ccb51640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0c8153b60 0x7fa0c8073e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cdb53640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0c815bfd0 0x7fa0c80acf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38906/0 (socket says 172.21.15.150:38906) 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cdb53640 1 -- 172.21.15.150:0/2089789897 learned_addr learned my addr 172.21.15.150:0/2089789897 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cd352640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0c8150bb0 0x7fa0c81ee700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:37840/0 (socket says 172.21.15.150:37840) 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cdb53640 1 -- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0c8150bb0 msgr2=0x7fa0c81ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cdb53640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0c8150bb0 0x7fa0c81ee700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cdb53640 1 -- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0c8153b60 msgr2=0x7fa0c8073e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:43.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cdb53640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0c8153b60 0x7fa0c8073e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:43.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cdb53640 1 -- 172.21.15.150:0/2089789897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c8153110 con 0x7fa0c815bfd0 2024-09-17T05:04:43.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cdb53640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0c815bfd0 0x7fa0c80acf00 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa0c001cfe0 tx=0x7fa0c0005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:43.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0bdffb640 1 -- 172.21.15.150:0/2089789897 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00b4040 con 0x7fa0c815bfd0 2024-09-17T05:04:43.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0bdffb640 1 -- 172.21.15.150:0/2089789897 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0c00af8a0 con 0x7fa0c815bfd0 2024-09-17T05:04:43.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/2089789897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c8147220 con 0x7fa0c815bfd0 2024-09-17T05:04:43.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0bdffb640 1 -- 172.21.15.150:0/2089789897 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00afc90 con 0x7fa0c815bfd0 2024-09-17T05:04:43.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/2089789897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c8146f40 con 0x7fa0c815bfd0 2024-09-17T05:04:43.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/2089789897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c8069820 con 0x7fa0c815bfd0 2024-09-17T05:04:43.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.174+0000 7fa0bdffb640 1 -- 172.21.15.150:0/2089789897 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa0c00c1020 con 0x7fa0c815bfd0 2024-09-17T05:04:43.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.178+0000 7fa0bdffb640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa0ac073230 0x7fa0ac075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:43.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.178+0000 7fa0bdffb640 1 -- 172.21.15.150:0/2089789897 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7fa0c01322c0 con 0x7fa0c815bfd0 2024-09-17T05:04:43.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.178+0000 7fa0cd352640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa0ac073230 0x7fa0ac075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:43.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.178+0000 7fa0cd352640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa0ac073230 0x7fa0ac075740 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4001870 tx=0x7fa0c4001380 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:43.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.182+0000 7fa0bdffb640 1 -- 172.21.15.150:0/2089789897 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa0c00c8020 con 0x7fa0c815bfd0 2024-09-17T05:04:43.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.410+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/2089789897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0c814cb10 con 0x7fa0c815bfd0 2024-09-17T05:04:43.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.410+0000 7fa0bdffb640 1 -- 172.21.15.150:0/2089789897 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+5717 (secure 0 0 0) 0x7fa0c0018f10 con 0x7fa0c815bfd0 2024-09-17T05:04:43.410 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:43.410 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":110,"btime":"2024-09-17T05:04:39:052857+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:39.052846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6564,"qdb_cluster":[6564]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:43.411 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 110 2024-09-17T05:04:43.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.410+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa0ac073230 msgr2=0x7fa0ac075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:43.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.410+0000 7fa0cf5b6640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa0ac073230 0x7fa0ac075740 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4001870 tx=0x7fa0c4001380 comp rx=0 tx=0).stop 2024-09-17T05:04:43.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.410+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/2089789897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0c815bfd0 msgr2=0x7fa0c80acf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:43.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.410+0000 7fa0cf5b6640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0c815bfd0 0x7fa0c80acf00 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa0c001cfe0 tx=0x7fa0c0005c60 comp rx=0 tx=0).stop 2024-09-17T05:04:43.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.410+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/2089789897 shutdown_connections 2024-09-17T05:04:43.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.410+0000 7fa0cf5b6640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa0ac073230 0x7fa0ac075740 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:43.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.410+0000 7fa0cf5b6640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0c8153b60 0x7fa0c8073e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:43.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.410+0000 7fa0cf5b6640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0c815bfd0 0x7fa0c80acf00 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:43.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.410+0000 7fa0cf5b6640 1 --2- 172.21.15.150:0/2089789897 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa0c8150bb0 0x7fa0c81ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:43.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.410+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/2089789897 >> 172.21.15.150:0/2089789897 conn(0x7fa0c80c92d0 msgr2=0x7fa0c806e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:43.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.414+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/2089789897 shutdown_connections 2024-09-17T05:04:43.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:43.414+0000 7fa0cf5b6640 1 -- 172.21.15.150:0/2089789897 wait complete. 2024-09-17T05:04:43.434 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-17T05:04:43.559 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T05:04:43.558+0000 7f3f4596e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:04:45.435 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:04:45.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.610+0000 7f185fe57640 1 -- 172.21.15.150:0/82876628 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18580c4da0 msgr2=0x7f1858150e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:45.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.610+0000 7f185fe57640 1 --2- 172.21.15.150:0/82876628 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18580c4da0 0x7f1858150e20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1854009870 tx=0x7f185401c790 comp rx=0 tx=0).stop 2024-09-17T05:04:45.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.610+0000 7f185fe57640 1 -- 172.21.15.150:0/82876628 shutdown_connections 2024-09-17T05:04:45.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.610+0000 7f185fe57640 1 --2- 172.21.15.150:0/82876628 >> v2:172.21.15.176:3300/0 conn(0x7f1858156b90 0x7f1858156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:45.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.610+0000 7f185fe57640 1 --2- 172.21.15.150:0/82876628 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18580c4da0 0x7f1858150e20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:45.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.610+0000 7f185fe57640 1 -- 172.21.15.150:0/82876628 >> 172.21.15.150:0/82876628 conn(0x7f18580c5220 msgr2=0x7f18580c5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:45.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.610+0000 7f185fe57640 1 -- 172.21.15.150:0/82876628 shutdown_connections 2024-09-17T05:04:45.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.610+0000 7f185fe57640 1 -- 172.21.15.150:0/82876628 wait complete. 2024-09-17T05:04:45.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.610+0000 7f185fe57640 1 Processor -- start 2024-09-17T05:04:45.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.610+0000 7f185fe57640 1 -- start start 2024-09-17T05:04:45.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185fe57640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18580aaf50 0x7f185814da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:45.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185fe57640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18580c4da0 0x7f18581f1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:45.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185fe57640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1858156b90 0x7f18581f1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:45.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185fe57640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1858069ac0 con 0x7f18580aaf50 2024-09-17T05:04:45.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185fe57640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1858069940 con 0x7f1858156b90 2024-09-17T05:04:45.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185fe57640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1858160740 con 0x7f18580c4da0 2024-09-17T05:04:45.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185dbf3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18580aaf50 0x7f185814da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:45.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185dbf3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18580aaf50 0x7f185814da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38912/0 (socket says 172.21.15.150:38912) 2024-09-17T05:04:45.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185dbf3640 1 -- 172.21.15.150:0/814988781 learned_addr learned my addr 172.21.15.150:0/814988781 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:04:45.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185e3f4640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1858156b90 0x7f18581f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:45.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185d3f2640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18580c4da0 0x7f18581f1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:45.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185dbf3640 1 -- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18580c4da0 msgr2=0x7f18581f1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:45.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185dbf3640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18580c4da0 0x7f18581f1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:45.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185dbf3640 1 -- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1858156b90 msgr2=0x7f18581f1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:45.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185dbf3640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1858156b90 0x7f18581f1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:45.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185dbf3640 1 -- 172.21.15.150:0/814988781 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185814df60 con 0x7f18580aaf50 2024-09-17T05:04:45.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185e3f4640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1858156b90 0x7f18581f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:04:45.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185dbf3640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18580aaf50 0x7f185814da20 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f185401a3c0 tx=0x7f18540a6460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:45.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f184e7fc640 1 -- 172.21.15.150:0/814988781 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18540ac070 con 0x7f18580aaf50 2024-09-17T05:04:45.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f184e7fc640 1 -- 172.21.15.150:0/814988781 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1854004320 con 0x7f18580aaf50 2024-09-17T05:04:45.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185fe57640 1 -- 172.21.15.150:0/814988781 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1858148230 con 0x7f18580aaf50 2024-09-17T05:04:45.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f184e7fc640 1 -- 172.21.15.150:0/814988781 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1854005320 con 0x7f18580aaf50 2024-09-17T05:04:45.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185fe57640 1 -- 172.21.15.150:0/814988781 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18580c6cc0 con 0x7f18580aaf50 2024-09-17T05:04:45.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.614+0000 7f185fe57640 1 -- 172.21.15.150:0/814988781 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1858150e20 con 0x7f18580aaf50 2024-09-17T05:04:45.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.618+0000 7f184e7fc640 1 -- 172.21.15.150:0/814988781 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f18540af070 con 0x7f18580aaf50 2024-09-17T05:04:45.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.618+0000 7f184e7fc640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f183c073230 0x7f183c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:04:45.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.618+0000 7f184e7fc640 1 -- 172.21.15.150:0/814988781 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f185412efe0 con 0x7f18580aaf50 2024-09-17T05:04:45.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.618+0000 7f185d3f2640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f183c073230 0x7f183c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:04:45.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.618+0000 7f185d3f2640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f183c073230 0x7f183c075740 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f18581564d0 tx=0x7f1848009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:04:45.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.622+0000 7f184e7fc640 1 -- 172.21.15.150:0/814988781 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1854016410 con 0x7f18580aaf50 2024-09-17T05:04:45.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.854+0000 7f185fe57640 1 -- 172.21.15.150:0/814988781 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18580ad1f0 con 0x7f18580aaf50 2024-09-17T05:04:45.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.854+0000 7f184e7fc640 1 -- 172.21.15.150:0/814988781 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+6546 (secure 0 0 0) 0x7f1854100e20 con 0x7f18580aaf50 2024-09-17T05:04:45.855 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:04:45.855 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":111,"btime":"2024-09-17T05:04:44:076711+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:39.052846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6564,"qdb_cluster":[6564]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:04:45.855 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 111 2024-09-17T05:04:45.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.858+0000 7f185fe57640 1 -- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f183c073230 msgr2=0x7f183c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:45.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.858+0000 7f185fe57640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f183c073230 0x7f183c075740 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f18581564d0 tx=0x7f1848009040 comp rx=0 tx=0).stop 2024-09-17T05:04:45.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.858+0000 7f185fe57640 1 -- 172.21.15.150:0/814988781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18580aaf50 msgr2=0x7f185814da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:04:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.858+0000 7f185fe57640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18580aaf50 0x7f185814da20 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f185401a3c0 tx=0x7f18540a6460 comp rx=0 tx=0).stop 2024-09-17T05:04:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.858+0000 7f185fe57640 1 -- 172.21.15.150:0/814988781 shutdown_connections 2024-09-17T05:04:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.858+0000 7f185fe57640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f183c073230 0x7f183c075740 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.858+0000 7f185fe57640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1858156b90 0x7f18581f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.858+0000 7f185fe57640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18580c4da0 0x7f18581f1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.858+0000 7f185fe57640 1 --2- 172.21.15.150:0/814988781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18580aaf50 0x7f185814da20 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:04:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.858+0000 7f185fe57640 1 -- 172.21.15.150:0/814988781 >> 172.21.15.150:0/814988781 conn(0x7f18580c5220 msgr2=0x7f18580c5630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:04:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.858+0000 7f185fe57640 1 -- 172.21.15.150:0/814988781 shutdown_connections 2024-09-17T05:04:45.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:04:45.858+0000 7f185fe57640 1 -- 172.21.15.150:0/814988781 wait complete. 2024-09-17T05:04:45.879 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-17T05:04:45.879 INFO:tasks.mds_thrash.fs.[a]:waiting for 17 secs before thrashing 2024-09-17T05:05:02.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:05:02.322 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:05:02.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:03.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.050+0000 7f061ff24640 1 -- 172.21.15.150:0/2342227580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06180aa9b0 msgr2=0x7f06180aadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:03.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.050+0000 7f061ff24640 1 --2- 172.21.15.150:0/2342227580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06180aa9b0 0x7f06180aadc0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f06180ab420 tx=0x7f061001cc60 comp rx=0 tx=0).stop 2024-09-17T05:05:03.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.050+0000 7f061ff24640 1 -- 172.21.15.150:0/2342227580 shutdown_connections 2024-09-17T05:05:03.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.050+0000 7f061ff24640 1 --2- 172.21.15.150:0/2342227580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06180aa9b0 0x7f06180aadc0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.050+0000 7f061ff24640 1 --2- 172.21.15.150:0/2342227580 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0618150e90 0x7f0618156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.050+0000 7f061ff24640 1 --2- 172.21.15.150:0/2342227580 >> v2:172.21.15.176:3300/0 conn(0x7f06180c4180 0x7f06181542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.050+0000 7f061ff24640 1 -- 172.21.15.150:0/2342227580 >> 172.21.15.150:0/2342227580 conn(0x7f06180c5ae0 msgr2=0x7f0618073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:03.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.050+0000 7f061ff24640 1 -- 172.21.15.150:0/2342227580 shutdown_connections 2024-09-17T05:05:03.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.050+0000 7f061ff24640 1 -- 172.21.15.150:0/2342227580 wait complete. 2024-09-17T05:05:03.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061ff24640 1 Processor -- start 2024-09-17T05:05:03.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061ff24640 1 -- start start 2024-09-17T05:05:03.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061ff24640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06180aa9b0 0x7f06181ed880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:03.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061ff24640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06180c4180 0x7f06181469f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:03.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061ff24640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0618150e90 0x7f061814ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:03.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061ff24640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0618157420 con 0x7f06180aa9b0 2024-09-17T05:05:03.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061ff24640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f06180adbc0 con 0x7f0618150e90 2024-09-17T05:05:03.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061ff24640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f061806e860 con 0x7f06180c4180 2024-09-17T05:05:03.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061dcc0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06180aa9b0 0x7f06181ed880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:03.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061dcc0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06180aa9b0 0x7f06181ed880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54766/0 (socket says 172.21.15.150:54766) 2024-09-17T05:05:03.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061dcc0640 1 -- 172.21.15.150:0/1565048124 learned_addr learned my addr 172.21.15.150:0/1565048124 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:03.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061d4bf640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06180c4180 0x7f06181469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:03.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061e4c1640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0618150e90 0x7f061814ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:03.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061dcc0640 1 -- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06180c4180 msgr2=0x7f06181469f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:03.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061dcc0640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06180c4180 0x7f06181469f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061dcc0640 1 -- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0618150e90 msgr2=0x7f061814ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:03.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061dcc0640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0618150e90 0x7f061814ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.054+0000 7f061dcc0640 1 -- 172.21.15.150:0/1565048124 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06180c81f0 con 0x7f06180aa9b0 2024-09-17T05:05:03.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.058+0000 7f061dcc0640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06180aa9b0 0x7f06181ed880 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f061400d700 tx=0x7f061400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:03.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.058+0000 7f060e7fc640 1 -- 172.21.15.150:0/1565048124 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614017070 con 0x7f06180aa9b0 2024-09-17T05:05:03.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.058+0000 7f060e7fc640 1 -- 172.21.15.150:0/1565048124 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0614009b10 con 0x7f06180aa9b0 2024-09-17T05:05:03.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.058+0000 7f060e7fc640 1 -- 172.21.15.150:0/1565048124 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614010460 con 0x7f06180aa9b0 2024-09-17T05:05:03.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.058+0000 7f061ff24640 1 -- 172.21.15.150:0/1565048124 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06180c76f0 con 0x7f06180aa9b0 2024-09-17T05:05:03.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.058+0000 7f061ff24640 1 -- 172.21.15.150:0/1565048124 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061815a860 con 0x7f06180aa9b0 2024-09-17T05:05:03.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.058+0000 7f061ff24640 1 -- 172.21.15.150:0/1565048124 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06181542c0 con 0x7f06180aa9b0 2024-09-17T05:05:03.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.058+0000 7f060e7fc640 1 -- 172.21.15.150:0/1565048124 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f061401a070 con 0x7f06180aa9b0 2024-09-17T05:05:03.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.058+0000 7f060e7fc640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f05fc073200 0x7f05fc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:03.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.062+0000 7f060e7fc640 1 -- 172.21.15.150:0/1565048124 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f0614094c60 con 0x7f06180aa9b0 2024-09-17T05:05:03.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.062+0000 7f061d4bf640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f05fc073200 0x7f05fc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.062+0000 7f061d4bf640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f05fc073200 0x7f05fc075710 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f06080058d0 tx=0x7f0608005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:03.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.066+0000 7f060e7fc640 1 -- 172.21.15.150:0/1565048124 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f061405f330 con 0x7f06180aa9b0 2024-09-17T05:05:03.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f061ff24640 1 -- 172.21.15.150:0/1565048124 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06180ab7f0 con 0x7f06180aa9b0 2024-09-17T05:05:03.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f060e7fc640 1 -- 172.21.15.150:0/1565048124 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+6546 (secure 0 0 0) 0x7f061405ec30 con 0x7f06180aa9b0 2024-09-17T05:05:03.322 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:03.322 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":111,"btime":"2024-09-17T05:04:44:076711+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:39.052846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6564,"qdb_cluster":[6564]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:03.322 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 111 2024-09-17T05:05:03.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f061ff24640 1 -- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f05fc073200 msgr2=0x7f05fc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:03.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f061ff24640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f05fc073200 0x7f05fc075710 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f06080058d0 tx=0x7f0608005860 comp rx=0 tx=0).stop 2024-09-17T05:05:03.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f061ff24640 1 -- 172.21.15.150:0/1565048124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06180aa9b0 msgr2=0x7f06181ed880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:03.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f061ff24640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06180aa9b0 0x7f06181ed880 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f061400d700 tx=0x7f061400dbc0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f061ff24640 1 -- 172.21.15.150:0/1565048124 shutdown_connections 2024-09-17T05:05:03.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f061ff24640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f05fc073200 0x7f05fc075710 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f061ff24640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0618150e90 0x7f061814ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f061ff24640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f06180c4180 0x7f06181469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f061ff24640 1 --2- 172.21.15.150:0/1565048124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06180aa9b0 0x7f06181ed880 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f061ff24640 1 -- 172.21.15.150:0/1565048124 >> 172.21.15.150:0/1565048124 conn(0x7f06180c5ae0 msgr2=0x7f0618073ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:03.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.326+0000 7f061ff24640 1 -- 172.21.15.150:0/1565048124 shutdown_connections 2024-09-17T05:05:03.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.330+0000 7f061ff24640 1 -- 172.21.15.150:0/1565048124 wait complete. 2024-09-17T05:05:03.346 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-17T05:05:03.346 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T05:05:03.347 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:05:03.361 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T05:05:03.361 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.570+0000 7fe0c71ef640 1 -- 172.21.15.150:0/3926364950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b0003fe0 msgr2=0x7fe0b0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:03.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.570+0000 7fe0c71ef640 1 --2- 172.21.15.150:0/3926364950 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b0003fe0 0x7fe0b0004810 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00c2e0 tx=0x7fe0ac01f6c0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.570+0000 7fe0c578c640 1 -- 172.21.15.150:0/3926364950 reap_dead start 2024-09-17T05:05:03.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.570+0000 7fe0c71ef640 1 -- 172.21.15.150:0/3926364950 shutdown_connections 2024-09-17T05:05:03.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.570+0000 7fe0c71ef640 1 -- 172.21.15.150:0/3926364950 >> 172.21.15.150:0/3926364950 conn(0x7fe0c006e360 msgr2=0x7fe0c006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:03.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.570+0000 7fe0c71ef640 1 -- 172.21.15.150:0/3926364950 shutdown_connections 2024-09-17T05:05:03.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.570+0000 7fe0c71ef640 1 -- 172.21.15.150:0/3926364950 wait complete. 2024-09-17T05:05:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.570+0000 7fe0c71ef640 1 Processor -- start 2024-09-17T05:05:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.570+0000 7fe0c71ef640 1 -- start start 2024-09-17T05:05:03.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0c71ef640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0b0003fe0 0x7fe0c0158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:03.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0c71ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00adb70 0x7fe0c014db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:03.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0c71ef640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0c014e0d0 0x7fe0c01efb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0c71ef640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c0072ed0 con 0x7fe0c00adb70 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0c71ef640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c015a310 con 0x7fe0b0003fe0 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0c71ef640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe0c015ac40 con 0x7fe0c014e0d0 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00adb70 0x7fe0c014db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00adb70 0x7fe0c014db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54780/0 (socket says 172.21.15.150:54780) 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bffff640 1 -- 172.21.15.150:0/1741176044 learned_addr learned my addr 172.21.15.150:0/1741176044 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0c4f8b640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0b0003fe0 0x7fe0c0158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0c578c640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0c014e0d0 0x7fe0c01efb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bffff640 1 -- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0c014e0d0 msgr2=0x7fe0c01efb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bffff640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0c014e0d0 0x7fe0c01efb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bffff640 1 -- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0b0003fe0 msgr2=0x7fe0c0158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bffff640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0b0003fe0 0x7fe0c0158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bffff640 1 -- 172.21.15.150:0/1741176044 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c014e5f0 con 0x7fe0c00adb70 2024-09-17T05:05:03.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bffff640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00adb70 0x7fe0c014db90 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac0b1040 tx=0x7fe0ac006cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:03.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bd7fa640 1 -- 172.21.15.150:0/1741176044 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0c1070 con 0x7fe0c00adb70 2024-09-17T05:05:03.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bd7fa640 1 -- 172.21.15.150:0/1741176044 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe0ac006270 con 0x7fe0c00adb70 2024-09-17T05:05:03.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0c71ef640 1 -- 172.21.15.150:0/1741176044 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c0145ba0 con 0x7fe0c00adb70 2024-09-17T05:05:03.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bd7fa640 1 -- 172.21.15.150:0/1741176044 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0b3030 con 0x7fe0c00adb70 2024-09-17T05:05:03.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0c71ef640 1 -- 172.21.15.150:0/1741176044 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c0159780 con 0x7fe0c00adb70 2024-09-17T05:05:03.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0c71ef640 1 -- 172.21.15.150:0/1741176044 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c014d460 con 0x7fe0c00adb70 2024-09-17T05:05:03.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.574+0000 7fe0bd7fa640 1 -- 172.21.15.150:0/1741176044 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe0ac003600 con 0x7fe0c00adb70 2024-09-17T05:05:03.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.578+0000 7fe0bd7fa640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe0a4073230 0x7fe0a4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:03.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.578+0000 7fe0bd7fa640 1 -- 172.21.15.150:0/1741176044 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7fe0ac131fc0 con 0x7fe0c00adb70 2024-09-17T05:05:03.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.578+0000 7fe0c4f8b640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe0a4073230 0x7fe0a4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:03.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.578+0000 7fe0c4f8b640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe0a4073230 0x7fe0a4075740 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4009be0 tx=0x7fe0b40091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:03.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.582+0000 7fe0bd7fa640 1 -- 172.21.15.150:0/1741176044 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe0ac019470 con 0x7fe0c00adb70 2024-09-17T05:05:03.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0c71ef640 1 -- 172.21.15.150:0/1741176044 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0c014cef0 con 0x7fe0c00adb70 2024-09-17T05:05:03.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0bd7fa640 1 -- 172.21.15.150:0/1741176044 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+6546 (secure 0 0 0) 0x7fe0ac0fbea0 con 0x7fe0c00adb70 2024-09-17T05:05:03.846 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:03.846 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":111,"btime":"2024-09-17T05:04:44:076711+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:39.052846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6564,"qdb_cluster":[6564]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:03.846 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 111 2024-09-17T05:05:03.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0c71ef640 1 -- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe0a4073230 msgr2=0x7fe0a4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:03.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0c71ef640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe0a4073230 0x7fe0a4075740 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4009be0 tx=0x7fe0b40091e0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0c71ef640 1 -- 172.21.15.150:0/1741176044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00adb70 msgr2=0x7fe0c014db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:03.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0c71ef640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00adb70 0x7fe0c014db90 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac0b1040 tx=0x7fe0ac006cf0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0c71ef640 1 -- 172.21.15.150:0/1741176044 shutdown_connections 2024-09-17T05:05:03.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0c71ef640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe0a4073230 0x7fe0a4075740 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0c71ef640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0c014e0d0 0x7fe0c01efb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0c71ef640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c00adb70 0x7fe0c014db90 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0c71ef640 1 --2- 172.21.15.150:0/1741176044 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0b0003fe0 0x7fe0c0158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:03.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0c71ef640 1 -- 172.21.15.150:0/1741176044 >> 172.21.15.150:0/1741176044 conn(0x7fe0c006e360 msgr2=0x7fe0c0071740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:03.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.850+0000 7fe0c71ef640 1 -- 172.21.15.150:0/1741176044 shutdown_connections 2024-09-17T05:05:03.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:03.854+0000 7fe0c71ef640 1 -- 172.21.15.150:0/1741176044 wait complete. 2024-09-17T05:05:03.878 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:05:05.879 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:06.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.050+0000 7f99a9a01640 1 -- 172.21.15.150:0/3907270606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a40c9520 msgr2=0x7f99a413c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:06.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.050+0000 7f99a9a01640 1 --2- 172.21.15.150:0/3907270606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a40c9520 0x7f99a413c8f0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f998c009900 tx=0x7f998c01c7c0 comp rx=0 tx=0).stop 2024-09-17T05:05:06.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.050+0000 7f99a9a01640 1 -- 172.21.15.150:0/3907270606 shutdown_connections 2024-09-17T05:05:06.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.050+0000 7f99a9a01640 1 --2- 172.21.15.150:0/3907270606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a40c9520 0x7f99a413c8f0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:06.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.050+0000 7f99a9a01640 1 --2- 172.21.15.150:0/3907270606 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99a413db50 0x7f99a4071680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:06.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.050+0000 7f99a9a01640 1 -- 172.21.15.150:0/3907270606 >> 172.21.15.150:0/3907270606 conn(0x7f99a40abaa0 msgr2=0x7f99a40abeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:06.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.050+0000 7f99a9a01640 1 -- 172.21.15.150:0/3907270606 shutdown_connections 2024-09-17T05:05:06.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.050+0000 7f99a9a01640 1 -- 172.21.15.150:0/3907270606 wait complete. 2024-09-17T05:05:06.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f99a9a01640 1 Processor -- start 2024-09-17T05:05:06.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f99a9a01640 1 -- start start 2024-09-17T05:05:06.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f99a9a01640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99a40c9520 0x7f99a4157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:06.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f99a9a01640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a413ce30 0x7f99a41448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:06.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f99a9a01640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99a413db50 0x7f99a40c3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:06.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f99a9a01640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f99a413d240 con 0x7f99a413ce30 2024-09-17T05:05:06.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f99a9a01640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f99a40c8bb0 con 0x7f99a40c9520 2024-09-17T05:05:06.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f99a9a01640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f99a413d3c0 con 0x7f99a413db50 2024-09-17T05:05:06.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f999bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a413ce30 0x7f99a41448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:06.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f999bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a413ce30 0x7f99a41448e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54796/0 (socket says 172.21.15.150:54796) 2024-09-17T05:05:06.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f999bfff640 1 -- 172.21.15.150:0/129122993 learned_addr learned my addr 172.21.15.150:0/129122993 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:06.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f99a37fe640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99a413db50 0x7f99a40c3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:06.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f99a2ffd640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99a40c9520 0x7f99a4157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:06.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f999bfff640 1 -- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99a413db50 msgr2=0x7f99a40c3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:06.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f999bfff640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99a413db50 0x7f99a40c3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:06.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f999bfff640 1 -- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99a40c9520 msgr2=0x7f99a4157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:06.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f999bfff640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99a40c9520 0x7f99a4157440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:06.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f999bfff640 1 -- 172.21.15.150:0/129122993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a40ae120 con 0x7f99a413ce30 2024-09-17T05:05:06.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.054+0000 7f999bfff640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a413ce30 0x7f99a41448e0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f998c0b0040 tx=0x7f998c005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:06.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.058+0000 7f99a0ff9640 1 -- 172.21.15.150:0/129122993 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c0b2040 con 0x7f99a413ce30 2024-09-17T05:05:06.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.058+0000 7f99a0ff9640 1 -- 172.21.15.150:0/129122993 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f998c004330 con 0x7f99a413ce30 2024-09-17T05:05:06.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.058+0000 7f99a9a01640 1 -- 172.21.15.150:0/129122993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99a4144e20 con 0x7f99a413ce30 2024-09-17T05:05:06.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.058+0000 7f99a0ff9640 1 -- 172.21.15.150:0/129122993 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998c005280 con 0x7f99a413ce30 2024-09-17T05:05:06.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.058+0000 7f99a9a01640 1 -- 172.21.15.150:0/129122993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99a4151940 con 0x7f99a413ce30 2024-09-17T05:05:06.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.058+0000 7f99a9a01640 1 -- 172.21.15.150:0/129122993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99a413c8f0 con 0x7f99a413ce30 2024-09-17T05:05:06.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.058+0000 7f99a0ff9640 1 -- 172.21.15.150:0/129122993 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f998c005420 con 0x7f99a413ce30 2024-09-17T05:05:06.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.058+0000 7f99a0ff9640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9988073200 0x7f9988075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:06.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.058+0000 7f99a0ff9640 1 -- 172.21.15.150:0/129122993 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f998c0bd070 con 0x7f99a413ce30 2024-09-17T05:05:06.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.058+0000 7f99a2ffd640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9988073200 0x7f9988075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:06.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.058+0000 7f99a2ffd640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9988073200 0x7f9988075710 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f99940058d0 tx=0x7f9994005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:06.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.062+0000 7f99a0ff9640 1 -- 172.21.15.150:0/129122993 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f998c0164d0 con 0x7f99a413ce30 2024-09-17T05:05:06.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.270+0000 7f99a9a01640 1 -- 172.21.15.150:0/129122993 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99a4152260 con 0x7f99a413ce30 2024-09-17T05:05:06.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.270+0000 7f99a0ff9640 1 -- 172.21.15.150:0/129122993 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+6546 (secure 0 0 0) 0x7f998c0f9050 con 0x7f99a413ce30 2024-09-17T05:05:06.268 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:06.268 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":111,"btime":"2024-09-17T05:04:44:076711+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:39.052846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6564,"qdb_cluster":[6564]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:06.268 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 111 2024-09-17T05:05:06.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.274+0000 7f99a9a01640 1 -- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9988073200 msgr2=0x7f9988075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:06.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.274+0000 7f99a9a01640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9988073200 0x7f9988075710 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f99940058d0 tx=0x7f9994005860 comp rx=0 tx=0).stop 2024-09-17T05:05:06.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.274+0000 7f99a9a01640 1 -- 172.21.15.150:0/129122993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a413ce30 msgr2=0x7f99a41448e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:06.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.274+0000 7f99a9a01640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a413ce30 0x7f99a41448e0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f998c0b0040 tx=0x7f998c005e90 comp rx=0 tx=0).stop 2024-09-17T05:05:06.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.274+0000 7f99a9a01640 1 -- 172.21.15.150:0/129122993 shutdown_connections 2024-09-17T05:05:06.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.274+0000 7f99a9a01640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9988073200 0x7f9988075710 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:06.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.274+0000 7f99a9a01640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f99a413db50 0x7f99a40c3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:06.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.274+0000 7f99a9a01640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99a413ce30 0x7f99a41448e0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:06.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.274+0000 7f99a9a01640 1 --2- 172.21.15.150:0/129122993 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99a40c9520 0x7f99a4157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:06.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.274+0000 7f99a9a01640 1 -- 172.21.15.150:0/129122993 >> 172.21.15.150:0/129122993 conn(0x7f99a40abaa0 msgr2=0x7f99a40c4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:06.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.274+0000 7f99a9a01640 1 -- 172.21.15.150:0/129122993 shutdown_connections 2024-09-17T05:05:06.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:06.274+0000 7f99a9a01640 1 -- 172.21.15.150:0/129122993 wait complete. 2024-09-17T05:05:06.290 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:05:08.291 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:08.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.462+0000 7f6be5a69640 1 -- 172.21.15.150:0/2065023425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be00c3b40 msgr2=0x7f6be0157c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.462+0000 7f6be5a69640 1 --2- 172.21.15.150:0/2065023425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be00c3b40 0x7f6be0157c30 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009900 tx=0x7f6bd401c880 comp rx=0 tx=0).stop 2024-09-17T05:05:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.462+0000 7f6be5a69640 1 -- 172.21.15.150:0/2065023425 shutdown_connections 2024-09-17T05:05:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.462+0000 7f6be5a69640 1 --2- 172.21.15.150:0/2065023425 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be0158170 0x7f6be0158580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.462+0000 7f6be5a69640 1 --2- 172.21.15.150:0/2065023425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be00c3b40 0x7f6be0157c30 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.462+0000 7f6be5a69640 1 -- 172.21.15.150:0/2065023425 >> 172.21.15.150:0/2065023425 conn(0x7f6be00c4a00 msgr2=0x7f6be00c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.462+0000 7f6be5a69640 1 -- 172.21.15.150:0/2065023425 shutdown_connections 2024-09-17T05:05:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.462+0000 7f6be5a69640 1 -- 172.21.15.150:0/2065023425 wait complete. 2024-09-17T05:05:08.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.462+0000 7f6be5a69640 1 Processor -- start 2024-09-17T05:05:08.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.462+0000 7f6be5a69640 1 -- start start 2024-09-17T05:05:08.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6be5a69640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6be00c3b40 0x7f6be015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:08.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6be5a69640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be0151d00 0x7f6be01f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6be5a69640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be0158170 0x7f6be0148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6be5a69640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6be00c40d0 con 0x7f6be0151d00 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6be5a69640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6be00c3f50 con 0x7f6be00c3b40 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6be5a69640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6be0161830 con 0x7f6be0158170 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bdeffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be0151d00 0x7f6be01f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bdeffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be0151d00 0x7f6be01f34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54820/0 (socket says 172.21.15.150:54820) 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bdeffd640 1 -- 172.21.15.150:0/1737188704 learned_addr learned my addr 172.21.15.150:0/1737188704 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bdffff640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be0158170 0x7f6be0148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bdf7fe640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6be00c3b40 0x7f6be015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bdeffd640 1 -- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be0158170 msgr2=0x7f6be0148ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bdeffd640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be0158170 0x7f6be0148ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bdeffd640 1 -- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6be00c3b40 msgr2=0x7f6be015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bdeffd640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6be00c3b40 0x7f6be015d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:08.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bdeffd640 1 -- 172.21.15.150:0/1737188704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be00c6d70 con 0x7f6be0151d00 2024-09-17T05:05:08.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bdffff640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be0158170 0x7f6be0148ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:05:08.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bdeffd640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be0151d00 0x7f6be01f34a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009850 tx=0x7f6bd40a5650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:08.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1737188704 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40b2040 con 0x7f6be0151d00 2024-09-17T05:05:08.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1737188704 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bd4004430 con 0x7f6be0151d00 2024-09-17T05:05:08.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6be5a69640 1 -- 172.21.15.150:0/1737188704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be0074010 con 0x7f6be0151d00 2024-09-17T05:05:08.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1737188704 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4003860 con 0x7f6be0151d00 2024-09-17T05:05:08.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6be5a69640 1 -- 172.21.15.150:0/1737188704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6be00745b0 con 0x7f6be0151d00 2024-09-17T05:05:08.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6be5a69640 1 -- 172.21.15.150:0/1737188704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6be0157c30 con 0x7f6be0151d00 2024-09-17T05:05:08.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1737188704 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6bd4003a00 con 0x7f6be0151d00 2024-09-17T05:05:08.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.466+0000 7f6bc7fff640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6bcc073200 0x7f6bcc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:08.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.470+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1737188704 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f6bd412eea0 con 0x7f6be0151d00 2024-09-17T05:05:08.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.470+0000 7f6bdf7fe640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6bcc073200 0x7f6bcc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:08.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.470+0000 7f6bdf7fe640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6bcc073200 0x7f6bcc075710 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0001650 tx=0x7f6bd0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:08.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.470+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1737188704 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6bd40164d0 con 0x7f6be0151d00 2024-09-17T05:05:08.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.682+0000 7f6be5a69640 1 -- 172.21.15.150:0/1737188704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6be0069550 con 0x7f6be0151d00 2024-09-17T05:05:08.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6bc7fff640 1 -- 172.21.15.150:0/1737188704 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+6546 (secure 0 0 0) 0x7f6bd40f96b0 con 0x7f6be0151d00 2024-09-17T05:05:08.681 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:08.681 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":111,"btime":"2024-09-17T05:04:44:076711+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:39.052846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6564,"qdb_cluster":[6564]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:08.681 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 111 2024-09-17T05:05:08.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6be5a69640 1 -- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6bcc073200 msgr2=0x7f6bcc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:08.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6be5a69640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6bcc073200 0x7f6bcc075710 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0001650 tx=0x7f6bd0009040 comp rx=0 tx=0).stop 2024-09-17T05:05:08.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6be5a69640 1 -- 172.21.15.150:0/1737188704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be0151d00 msgr2=0x7f6be01f34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:08.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6be5a69640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be0151d00 0x7f6be01f34a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009850 tx=0x7f6bd40a5650 comp rx=0 tx=0).stop 2024-09-17T05:05:08.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6be5a69640 1 -- 172.21.15.150:0/1737188704 shutdown_connections 2024-09-17T05:05:08.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6be5a69640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6bcc073200 0x7f6bcc075710 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:08.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6be5a69640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6be0158170 0x7f6be0148ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:08.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6be5a69640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6be0151d00 0x7f6be01f34a0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:08.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6be5a69640 1 --2- 172.21.15.150:0/1737188704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6be00c3b40 0x7f6be015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:08.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6be5a69640 1 -- 172.21.15.150:0/1737188704 >> 172.21.15.150:0/1737188704 conn(0x7f6be00c4a00 msgr2=0x7f6be00c2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:08.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6be5a69640 1 -- 172.21.15.150:0/1737188704 shutdown_connections 2024-09-17T05:05:08.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:08.686+0000 7f6be5a69640 1 -- 172.21.15.150:0/1737188704 wait complete. 2024-09-17T05:05:08.703 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:05:10.703 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:10.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.874+0000 7f7849889640 1 -- 172.21.15.150:0/3256329100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78440ad7f0 msgr2=0x7f78440c6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:10.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.874+0000 7f7849889640 1 --2- 172.21.15.150:0/3256329100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78440ad7f0 0x7f78440c6230 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f783c0098f0 tx=0x7f783c01c7c0 comp rx=0 tx=0).stop 2024-09-17T05:05:10.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.874+0000 7f7849889640 1 -- 172.21.15.150:0/3256329100 shutdown_connections 2024-09-17T05:05:10.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.874+0000 7f7849889640 1 --2- 172.21.15.150:0/3256329100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78440ad7f0 0x7f78440c6230 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:10.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.874+0000 7f7849889640 1 -- 172.21.15.150:0/3256329100 >> 172.21.15.150:0/3256329100 conn(0x7f78440c9de0 msgr2=0x7f78440ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:10.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.874+0000 7f7849889640 1 -- 172.21.15.150:0/3256329100 shutdown_connections 2024-09-17T05:05:10.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.874+0000 7f7849889640 1 -- 172.21.15.150:0/3256329100 wait complete. 2024-09-17T05:05:10.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f7849889640 1 Processor -- start 2024-09-17T05:05:10.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f7849889640 1 -- start start 2024-09-17T05:05:10.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f7849889640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f78440ad7f0 0x7f78440c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:10.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f7849889640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844151e10 0x7f784414b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:10.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f7849889640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7844157ca0 0x7f78440c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:10.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f7849889640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f78440ad210 con 0x7f7844151e10 2024-09-17T05:05:10.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f7849889640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f78440c5a40 con 0x7f7844157ca0 2024-09-17T05:05:10.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f7849889640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7844162860 con 0x7f78440ad7f0 2024-09-17T05:05:10.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f78427fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844151e10 0x7f784414b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:10.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f78427fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844151e10 0x7f784414b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47698/0 (socket says 172.21.15.150:47698) 2024-09-17T05:05:10.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f78427fc640 1 -- 172.21.15.150:0/525984549 learned_addr learned my addr 172.21.15.150:0/525984549 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:10.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f78437fe640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7844157ca0 0x7f78440c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:10.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f7842ffd640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f78440ad7f0 0x7f78440c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:10.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f78427fc640 1 -- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f78440ad7f0 msgr2=0x7f78440c2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:10.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f78427fc640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f78440ad7f0 0x7f78440c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:10.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f78427fc640 1 -- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7844157ca0 msgr2=0x7f78440c2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:10.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f78427fc640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7844157ca0 0x7f78440c2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:10.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f78427fc640 1 -- 172.21.15.150:0/525984549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78440c28c0 con 0x7f7844151e10 2024-09-17T05:05:10.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f78427fc640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844151e10 0x7f784414b3a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f782c00c790 tx=0x7f782c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:10.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f783bfff640 1 -- 172.21.15.150:0/525984549 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c003b80 con 0x7f7844151e10 2024-09-17T05:05:10.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f783bfff640 1 -- 172.21.15.150:0/525984549 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f782c003d20 con 0x7f7844151e10 2024-09-17T05:05:10.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f7849889640 1 -- 172.21.15.150:0/525984549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f784414d440 con 0x7f7844151e10 2024-09-17T05:05:10.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f783bfff640 1 -- 172.21.15.150:0/525984549 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c00aa90 con 0x7f7844151e10 2024-09-17T05:05:10.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f7849889640 1 -- 172.21.15.150:0/525984549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78440c3980 con 0x7f7844151e10 2024-09-17T05:05:10.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.878+0000 7f7849889640 1 -- 172.21.15.150:0/525984549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78440c3bc0 con 0x7f7844151e10 2024-09-17T05:05:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.882+0000 7f783bfff640 1 -- 172.21.15.150:0/525984549 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f782c00ac30 con 0x7f7844151e10 2024-09-17T05:05:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.882+0000 7f783bfff640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7824073200 0x7f7824075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.882+0000 7f783bfff640 1 -- 172.21.15.150:0/525984549 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f782c094b90 con 0x7f7844151e10 2024-09-17T05:05:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.882+0000 7f7842ffd640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7824073200 0x7f7824075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.882+0000 7f7842ffd640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7824073200 0x7f7824075710 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f783c0af000 tx=0x7f783c01b000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:10.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:10.886+0000 7f783bfff640 1 -- 172.21.15.150:0/525984549 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f78440c28c0 con 0x7f7844151e10 2024-09-17T05:05:11.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.094+0000 7f7849889640 1 -- 172.21.15.150:0/525984549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78440bed00 con 0x7f7844151e10 2024-09-17T05:05:11.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f783bfff640 1 -- 172.21.15.150:0/525984549 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+6546 (secure 0 0 0) 0x7f782c05f140 con 0x7f7844151e10 2024-09-17T05:05:11.093 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:11.093 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":111,"btime":"2024-09-17T05:04:44:076711+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:39.052846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6564,"qdb_cluster":[6564]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:11.094 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 111 2024-09-17T05:05:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f7849889640 1 -- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7824073200 msgr2=0x7f7824075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f7849889640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7824073200 0x7f7824075710 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f783c0af000 tx=0x7f783c01b000 comp rx=0 tx=0).stop 2024-09-17T05:05:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f7849889640 1 -- 172.21.15.150:0/525984549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844151e10 msgr2=0x7f784414b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f7849889640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844151e10 0x7f784414b3a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f782c00c790 tx=0x7f782c00cc50 comp rx=0 tx=0).stop 2024-09-17T05:05:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f7849889640 1 -- 172.21.15.150:0/525984549 shutdown_connections 2024-09-17T05:05:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f7849889640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7824073200 0x7f7824075710 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f7849889640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7844157ca0 0x7f78440c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f7849889640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7844151e10 0x7f784414b3a0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f7849889640 1 --2- 172.21.15.150:0/525984549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f78440ad7f0 0x7f78440c2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f7849889640 1 -- 172.21.15.150:0/525984549 >> 172.21.15.150:0/525984549 conn(0x7f78440c9de0 msgr2=0x7f78440c4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:11.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f7849889640 1 -- 172.21.15.150:0/525984549 shutdown_connections 2024-09-17T05:05:11.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:11.098+0000 7f7849889640 1 -- 172.21.15.150:0/525984549 wait complete. 2024-09-17T05:05:11.115 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:05:13.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:13.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.286+0000 7f0e0b87f640 1 -- 172.21.15.150:0/654253569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e04070e50 msgr2=0x7f0e04071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:13.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.286+0000 7f0e0b87f640 1 --2- 172.21.15.150:0/654253569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e04070e50 0x7f0e04071260 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009900 tx=0x7f0df401c850 comp rx=0 tx=0).stop 2024-09-17T05:05:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.286+0000 7f0e0b87f640 1 -- 172.21.15.150:0/654253569 shutdown_connections 2024-09-17T05:05:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.286+0000 7f0e0b87f640 1 --2- 172.21.15.150:0/654253569 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e040717a0 0x7f0e04074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.286+0000 7f0e0b87f640 1 --2- 172.21.15.150:0/654253569 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e04070e50 0x7f0e04071260 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.286+0000 7f0e0b87f640 1 -- 172.21.15.150:0/654253569 >> 172.21.15.150:0/654253569 conn(0x7f0e040c4a00 msgr2=0x7f0e040c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.286+0000 7f0e0b87f640 1 -- 172.21.15.150:0/654253569 shutdown_connections 2024-09-17T05:05:13.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.286+0000 7f0e0b87f640 1 -- 172.21.15.150:0/654253569 wait complete. 2024-09-17T05:05:13.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.286+0000 7f0e0b87f640 1 Processor -- start 2024-09-17T05:05:13.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.286+0000 7f0e0b87f640 1 -- start start 2024-09-17T05:05:13.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e0b87f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04070e50 0x7f0e04145a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:13.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e0b87f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e040717a0 0x7f0e04148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:13.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e0b87f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e04073440 0x7f0e041f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:13.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e0b87f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0e040ad190 con 0x7f0e04073440 2024-09-17T05:05:13.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e0b87f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0e040ad010 con 0x7f0e040717a0 2024-09-17T05:05:13.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e0b87f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0e040ad310 con 0x7f0e04070e50 2024-09-17T05:05:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e09e1c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e04073440 0x7f0e041f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e09e1c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e04073440 0x7f0e041f1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47710/0 (socket says 172.21.15.150:47710) 2024-09-17T05:05:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e09e1c640 1 -- 172.21.15.150:0/2101583795 learned_addr learned my addr 172.21.15.150:0/2101583795 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e0961b640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04070e50 0x7f0e04145a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e08e1a640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e040717a0 0x7f0e04148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e09e1c640 1 -- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04070e50 msgr2=0x7f0e04145a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e09e1c640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04070e50 0x7f0e04145a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e09e1c640 1 -- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e040717a0 msgr2=0x7f0e04148240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e09e1c640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e040717a0 0x7f0e04148240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e09e1c640 1 -- 172.21.15.150:0/2101583795 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e041f2290 con 0x7f0e04073440 2024-09-17T05:05:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e09e1c640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e04073440 0x7f0e041f1630 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00e970 tx=0x7f0dfc00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:13.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0df9ffb640 1 -- 172.21.15.150:0/2101583795 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc004d30 con 0x7f0e04073440 2024-09-17T05:05:13.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0df9ffb640 1 -- 172.21.15.150:0/2101583795 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0dfc00cac0 con 0x7f0e04073440 2024-09-17T05:05:13.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e0b87f640 1 -- 172.21.15.150:0/2101583795 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e04147a40 con 0x7f0e04073440 2024-09-17T05:05:13.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0df9ffb640 1 -- 172.21.15.150:0/2101583795 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc005020 con 0x7f0e04073440 2024-09-17T05:05:13.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e0b87f640 1 -- 172.21.15.150:0/2101583795 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e04145fb0 con 0x7f0e04073440 2024-09-17T05:05:13.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0e0b87f640 1 -- 172.21.15.150:0/2101583795 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e041464f0 con 0x7f0e04073440 2024-09-17T05:05:13.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0df9ffb640 1 -- 172.21.15.150:0/2101583795 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0dfc0051c0 con 0x7f0e04073440 2024-09-17T05:05:13.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.290+0000 7f0df9ffb640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0de8073200 0x7f0de8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:13.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.294+0000 7f0df9ffb640 1 -- 172.21.15.150:0/2101583795 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f0dfc095580 con 0x7f0e04073440 2024-09-17T05:05:13.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.294+0000 7f0e0961b640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0de8073200 0x7f0de8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:13.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.294+0000 7f0e0961b640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0de8073200 0x7f0de8075710 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0e041513e0 tx=0x7f0e00009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:13.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.298+0000 7f0df9ffb640 1 -- 172.21.15.150:0/2101583795 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0dfc016030 con 0x7f0e04073440 2024-09-17T05:05:13.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.502+0000 7f0e0b87f640 1 -- 172.21.15.150:0/2101583795 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e040ad7f0 con 0x7f0e04073440 2024-09-17T05:05:13.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.502+0000 7f0df9ffb640 1 -- 172.21.15.150:0/2101583795 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+6546 (secure 0 0 0) 0x7f0dfc05fbc0 con 0x7f0e04073440 2024-09-17T05:05:13.500 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:13.500 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":111,"btime":"2024-09-17T05:04:44:076711+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:39.052846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6564,"qdb_cluster":[6564]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:13.500 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 111 2024-09-17T05:05:13.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.506+0000 7f0e0b87f640 1 -- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0de8073200 msgr2=0x7f0de8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:13.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.506+0000 7f0e0b87f640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0de8073200 0x7f0de8075710 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0e041513e0 tx=0x7f0e00009040 comp rx=0 tx=0).stop 2024-09-17T05:05:13.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.506+0000 7f0e0b87f640 1 -- 172.21.15.150:0/2101583795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e04073440 msgr2=0x7f0e041f1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:13.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.506+0000 7f0e0b87f640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e04073440 0x7f0e041f1630 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00e970 tx=0x7f0dfc00ee30 comp rx=0 tx=0).stop 2024-09-17T05:05:13.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.506+0000 7f0e0b87f640 1 -- 172.21.15.150:0/2101583795 shutdown_connections 2024-09-17T05:05:13.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.506+0000 7f0e0b87f640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0de8073200 0x7f0de8075710 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:13.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.506+0000 7f0e0b87f640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e040717a0 0x7f0e04148240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:13.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.506+0000 7f0e0b87f640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e04073440 0x7f0e041f1630 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:13.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.506+0000 7f0e0b87f640 1 --2- 172.21.15.150:0/2101583795 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e04070e50 0x7f0e04145a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:13.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.506+0000 7f0e0b87f640 1 -- 172.21.15.150:0/2101583795 >> 172.21.15.150:0/2101583795 conn(0x7f0e040c4a00 msgr2=0x7f0e0415e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:13.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.506+0000 7f0e0b87f640 1 -- 172.21.15.150:0/2101583795 shutdown_connections 2024-09-17T05:05:13.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:13.506+0000 7f0e0b87f640 1 -- 172.21.15.150:0/2101583795 wait complete. 2024-09-17T05:05:13.522 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:05:15.524 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:15.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.690+0000 7f077ea8e640 1 -- 172.21.15.150:0/1138549838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0764005230 msgr2=0x7f076400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:15.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.690+0000 7f077ea8e640 1 --2- 172.21.15.150:0/1138549838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0764005230 0x7f076400b9d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f077000c310 tx=0x7f077001f720 comp rx=0 tx=0).stop 2024-09-17T05:05:15.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.690+0000 7f077d02b640 1 -- 172.21.15.150:0/1138549838 reap_dead start 2024-09-17T05:05:15.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.690+0000 7f077ea8e640 1 -- 172.21.15.150:0/1138549838 shutdown_connections 2024-09-17T05:05:15.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.690+0000 7f077ea8e640 1 -- 172.21.15.150:0/1138549838 >> 172.21.15.150:0/1138549838 conn(0x7f07780aab60 msgr2=0x7f07780aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:15.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.690+0000 7f077ea8e640 1 -- 172.21.15.150:0/1138549838 shutdown_connections 2024-09-17T05:05:15.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.690+0000 7f077ea8e640 1 -- 172.21.15.150:0/1138549838 wait complete. 2024-09-17T05:05:15.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077ea8e640 1 Processor -- start 2024-09-17T05:05:15.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077ea8e640 1 -- start start 2024-09-17T05:05:15.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077ea8e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0764005230 0x7f0778158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:15.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077ea8e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07780abe20 0x7f0778071260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:15.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077ea8e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07780717a0 0x7f077814cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:15.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077ea8e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f07781574a0 con 0x7f0764005230 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077ea8e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0778157320 con 0x7f07780abe20 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077ea8e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f077806d9f0 con 0x7f07780717a0 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077c82a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0764005230 0x7f0778158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077c82a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0764005230 0x7f0778158030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47720/0 (socket says 172.21.15.150:47720) 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077c82a640 1 -- 172.21.15.150:0/1333672527 learned_addr learned my addr 172.21.15.150:0/1333672527 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f0777fff640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07780abe20 0x7f0778071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077d02b640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07780717a0 0x7f077814cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077c82a640 1 -- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07780717a0 msgr2=0x7f077814cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077c82a640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07780717a0 0x7f077814cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077c82a640 1 -- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07780abe20 msgr2=0x7f0778071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077c82a640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07780abe20 0x7f0778071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077c82a640 1 -- 172.21.15.150:0/1333672527 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07781ecfd0 con 0x7f0764005230 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077c82a640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0764005230 0x7f0778158030 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f076000e9f0 tx=0x7f076000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:15.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f07757fa640 1 -- 172.21.15.150:0/1333672527 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076000cc60 con 0x7f0764005230 2024-09-17T05:05:15.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f07757fa640 1 -- 172.21.15.150:0/1333672527 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0760005020 con 0x7f0764005230 2024-09-17T05:05:15.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077ea8e640 1 -- 172.21.15.150:0/1333672527 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0778070780 con 0x7f0764005230 2024-09-17T05:05:15.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f07757fa640 1 -- 172.21.15.150:0/1333672527 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0760005450 con 0x7f0764005230 2024-09-17T05:05:15.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.694+0000 7f077ea8e640 1 -- 172.21.15.150:0/1333672527 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0778071cc0 con 0x7f0764005230 2024-09-17T05:05:15.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.698+0000 7f077ea8e640 1 -- 172.21.15.150:0/1333672527 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07780c69a0 con 0x7f0764005230 2024-09-17T05:05:15.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.698+0000 7f07757fa640 1 -- 172.21.15.150:0/1333672527 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f07600047b0 con 0x7f0764005230 2024-09-17T05:05:15.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.698+0000 7f07757fa640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0758073200 0x7f0758075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:15.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.698+0000 7f07757fa640 1 -- 172.21.15.150:0/1333672527 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f0760095560 con 0x7f0764005230 2024-09-17T05:05:15.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.698+0000 7f0777fff640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0758073200 0x7f0758075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:15.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.698+0000 7f0777fff640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0758073200 0x7f0758075710 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0768005c50 tx=0x7f07680094e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:15.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.702+0000 7f07757fa640 1 -- 172.21.15.150:0/1333672527 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0760016030 con 0x7f0764005230 2024-09-17T05:05:15.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.910+0000 7f077ea8e640 1 -- 172.21.15.150:0/1333672527 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07780c6660 con 0x7f0764005230 2024-09-17T05:05:15.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f07757fa640 1 -- 172.21.15.150:0/1333672527 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+6546 (secure 0 0 0) 0x7f076005fc30 con 0x7f0764005230 2024-09-17T05:05:15.910 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:15.910 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":111,"btime":"2024-09-17T05:04:44:076711+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:39.052846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6564,"qdb_cluster":[6564]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:15.910 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 111 2024-09-17T05:05:15.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f077ea8e640 1 -- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0758073200 msgr2=0x7f0758075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:15.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f077ea8e640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0758073200 0x7f0758075710 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0768005c50 tx=0x7f07680094e0 comp rx=0 tx=0).stop 2024-09-17T05:05:15.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f077ea8e640 1 -- 172.21.15.150:0/1333672527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0764005230 msgr2=0x7f0778158030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:15.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f077ea8e640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0764005230 0x7f0778158030 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f076000e9f0 tx=0x7f076000eeb0 comp rx=0 tx=0).stop 2024-09-17T05:05:15.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f077ea8e640 1 -- 172.21.15.150:0/1333672527 shutdown_connections 2024-09-17T05:05:15.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f077ea8e640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0758073200 0x7f0758075710 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:15.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f077ea8e640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07780abe20 0x7f0778071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:15.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f077ea8e640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07780717a0 0x7f077814cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:15.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f077ea8e640 1 --2- 172.21.15.150:0/1333672527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0764005230 0x7f0778158030 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:15.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f077ea8e640 1 -- 172.21.15.150:0/1333672527 >> 172.21.15.150:0/1333672527 conn(0x7f07780aab60 msgr2=0x7f077814fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:15.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f077ea8e640 1 -- 172.21.15.150:0/1333672527 shutdown_connections 2024-09-17T05:05:15.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:15.914+0000 7f077ea8e640 1 -- 172.21.15.150:0/1333672527 wait complete. 2024-09-17T05:05:15.933 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:05:17.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:18.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.098+0000 7f64187e0640 1 -- 172.21.15.150:0/382494157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6410073440 msgr2=0x7f64100a7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:18.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.098+0000 7f64187e0640 1 --2- 172.21.15.150:0/382494157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6410073440 0x7f64100a7300 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f640c009900 tx=0x7f640c01c850 comp rx=0 tx=0).stop 2024-09-17T05:05:18.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.098+0000 7f64187e0640 1 -- 172.21.15.150:0/382494157 shutdown_connections 2024-09-17T05:05:18.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.098+0000 7f64187e0640 1 --2- 172.21.15.150:0/382494157 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6410074450 0x7f6410074860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:18.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.098+0000 7f64187e0640 1 --2- 172.21.15.150:0/382494157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6410073440 0x7f64100a7300 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:18.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.098+0000 7f64187e0640 1 -- 172.21.15.150:0/382494157 >> 172.21.15.150:0/382494157 conn(0x7f64100c4a00 msgr2=0x7f64100c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:18.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.098+0000 7f64187e0640 1 -- 172.21.15.150:0/382494157 shutdown_connections 2024-09-17T05:05:18.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.098+0000 7f64187e0640 1 -- 172.21.15.150:0/382494157 wait complete. 2024-09-17T05:05:18.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f64187e0640 1 Processor -- start 2024-09-17T05:05:18.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f64187e0640 1 -- start start 2024-09-17T05:05:18.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f64187e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6410070e50 0x7f64101f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:18.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f64187e0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6410073440 0x7f64101457f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f64187e0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6410074450 0x7f6410149e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f64187e0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f64100adef0 con 0x7f6410074450 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f64187e0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6410071a50 con 0x7f6410073440 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f64187e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f64100ad610 con 0x7f6410070e50 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6416d7d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6410074450 0x7f6410149e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6416d7d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6410074450 0x7f6410149e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47736/0 (socket says 172.21.15.150:47736) 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6416d7d640 1 -- 172.21.15.150:0/1757542053 learned_addr learned my addr 172.21.15.150:0/1757542053 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f641657c640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6410070e50 0x7f64101f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6415d7b640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6410073440 0x7f64101457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6416d7d640 1 -- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6410070e50 msgr2=0x7f64101f13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6416d7d640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6410070e50 0x7f64101f13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6416d7d640 1 -- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6410073440 msgr2=0x7f64101457f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6416d7d640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6410073440 0x7f64101457f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:18.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6416d7d640 1 -- 172.21.15.150:0/1757542053 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64100c2090 con 0x7f6410074450 2024-09-17T05:05:18.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6415d7b640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6410073440 0x7f64101457f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:05:18.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6416d7d640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6410074450 0x7f6410149e40 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f640800e940 tx=0x7f640800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:18.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6406ffd640 1 -- 172.21.15.150:0/1757542053 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6408004d30 con 0x7f6410074450 2024-09-17T05:05:18.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6406ffd640 1 -- 172.21.15.150:0/1757542053 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f640800cac0 con 0x7f6410074450 2024-09-17T05:05:18.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f64187e0640 1 -- 172.21.15.150:0/1757542053 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6410156c20 con 0x7f6410074450 2024-09-17T05:05:18.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f6406ffd640 1 -- 172.21.15.150:0/1757542053 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6408005020 con 0x7f6410074450 2024-09-17T05:05:18.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f64187e0640 1 -- 172.21.15.150:0/1757542053 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64101574a0 con 0x7f6410074450 2024-09-17T05:05:18.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.102+0000 7f64187e0640 1 -- 172.21.15.150:0/1757542053 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64100a7300 con 0x7f6410074450 2024-09-17T05:05:18.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.106+0000 7f6406ffd640 1 -- 172.21.15.150:0/1757542053 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f64080051c0 con 0x7f6410074450 2024-09-17T05:05:18.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.106+0000 7f6406ffd640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f63f4073200 0x7f63f4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:18.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.106+0000 7f6406ffd640 1 -- 172.21.15.150:0/1757542053 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f6408095600 con 0x7f6410074450 2024-09-17T05:05:18.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.106+0000 7f641657c640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f63f4073200 0x7f63f4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:18.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.106+0000 7f641657c640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f63f4073200 0x7f63f4075710 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f640c0af000 tx=0x7f640c01b090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:18.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.110+0000 7f6406ffd640 1 -- 172.21.15.150:0/1757542053 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6408016030 con 0x7f6410074450 2024-09-17T05:05:18.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.314+0000 7f64187e0640 1 -- 172.21.15.150:0/1757542053 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6410150f50 con 0x7f6410074450 2024-09-17T05:05:18.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.314+0000 7f6406ffd640 1 -- 172.21.15.150:0/1757542053 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+6546 (secure 0 0 0) 0x7f6408066070 con 0x7f6410074450 2024-09-17T05:05:18.312 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:18.312 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":111,"btime":"2024-09-17T05:04:44:076711+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:39.052846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6564,"qdb_cluster":[6564]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:18.313 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 111 2024-09-17T05:05:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.318+0000 7f64187e0640 1 -- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f63f4073200 msgr2=0x7f63f4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.318+0000 7f64187e0640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f63f4073200 0x7f63f4075710 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f640c0af000 tx=0x7f640c01b090 comp rx=0 tx=0).stop 2024-09-17T05:05:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.318+0000 7f64187e0640 1 -- 172.21.15.150:0/1757542053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6410074450 msgr2=0x7f6410149e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.318+0000 7f64187e0640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6410074450 0x7f6410149e40 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f640800e940 tx=0x7f640800ee00 comp rx=0 tx=0).stop 2024-09-17T05:05:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.318+0000 7f64187e0640 1 -- 172.21.15.150:0/1757542053 shutdown_connections 2024-09-17T05:05:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.318+0000 7f64187e0640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f63f4073200 0x7f63f4075710 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.318+0000 7f64187e0640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6410073440 0x7f64101457f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.318+0000 7f64187e0640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6410074450 0x7f6410149e40 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.318+0000 7f64187e0640 1 --2- 172.21.15.150:0/1757542053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6410070e50 0x7f64101f13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.318+0000 7f64187e0640 1 -- 172.21.15.150:0/1757542053 >> 172.21.15.150:0/1757542053 conn(0x7f64100c4a00 msgr2=0x7f641015f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:18.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.318+0000 7f64187e0640 1 -- 172.21.15.150:0/1757542053 shutdown_connections 2024-09-17T05:05:18.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:18.318+0000 7f64187e0640 1 -- 172.21.15.150:0/1757542053 wait complete. 2024-09-17T05:05:18.334 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:05:20.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:20.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.502+0000 7f4a79d48640 1 -- 172.21.15.150:0/199637421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a74070e50 msgr2=0x7f4a74071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:20.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.502+0000 7f4a79d48640 1 --2- 172.21.15.150:0/199637421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a74070e50 0x7f4a74071260 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009900 tx=0x7f4a5c01c850 comp rx=0 tx=0).stop 2024-09-17T05:05:20.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.502+0000 7f4a79d48640 1 -- 172.21.15.150:0/199637421 shutdown_connections 2024-09-17T05:05:20.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.502+0000 7f4a79d48640 1 --2- 172.21.15.150:0/199637421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a740717a0 0x7f4a74074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:20.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.502+0000 7f4a79d48640 1 --2- 172.21.15.150:0/199637421 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a74070e50 0x7f4a74071260 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:20.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.502+0000 7f4a79d48640 1 -- 172.21.15.150:0/199637421 >> 172.21.15.150:0/199637421 conn(0x7f4a740c4a00 msgr2=0x7f4a740c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:20.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.502+0000 7f4a79d48640 1 -- 172.21.15.150:0/199637421 shutdown_connections 2024-09-17T05:05:20.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.502+0000 7f4a79d48640 1 -- 172.21.15.150:0/199637421 wait complete. 2024-09-17T05:05:20.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a79d48640 1 Processor -- start 2024-09-17T05:05:20.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a79d48640 1 -- start start 2024-09-17T05:05:20.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a79d48640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a74070e50 0x7f4a741f36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:20.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a79d48640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a740717a0 0x7f4a74151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:20.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a79d48640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a74073440 0x7f4a7414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:20.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a79d48640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4a740ad190 con 0x7f4a74070e50 2024-09-17T05:05:20.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a79d48640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4a740ad010 con 0x7f4a74073440 2024-09-17T05:05:20.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a79d48640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4a740ad310 con 0x7f4a740717a0 2024-09-17T05:05:20.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a737fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a74070e50 0x7f4a741f36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:20.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a737fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a74070e50 0x7f4a741f36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40156/0 (socket says 172.21.15.150:40156) 2024-09-17T05:05:20.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a737fe640 1 -- 172.21.15.150:0/2803926125 learned_addr learned my addr 172.21.15.150:0/2803926125 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:20.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a72ffd640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a740717a0 0x7f4a74151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:20.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a73fff640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a74073440 0x7f4a7414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:20.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a737fe640 1 -- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a740717a0 msgr2=0x7f4a74151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:20.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a737fe640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a740717a0 0x7f4a74151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:20.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a737fe640 1 -- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a74073440 msgr2=0x7f4a7414aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:20.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a737fe640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a74073440 0x7f4a7414aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:20.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a737fe640 1 -- 172.21.15.150:0/2803926125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a741f2c80 con 0x7f4a74070e50 2024-09-17T05:05:20.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a737fe640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a74070e50 0x7f4a741f36e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00c730 tx=0x7f4a6c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:20.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a6bfff640 1 -- 172.21.15.150:0/2803926125 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c016020 con 0x7f4a74070e50 2024-09-17T05:05:20.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a6bfff640 1 -- 172.21.15.150:0/2803926125 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a6c00ce40 con 0x7f4a74070e50 2024-09-17T05:05:20.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.506+0000 7f4a6bfff640 1 -- 172.21.15.150:0/2803926125 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c003c00 con 0x7f4a74070e50 2024-09-17T05:05:20.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.510+0000 7f4a79d48640 1 -- 172.21.15.150:0/2803926125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a74156430 con 0x7f4a74070e50 2024-09-17T05:05:20.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.510+0000 7f4a79d48640 1 -- 172.21.15.150:0/2803926125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a74149250 con 0x7f4a74070e50 2024-09-17T05:05:20.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.510+0000 7f4a79d48640 1 -- 172.21.15.150:0/2803926125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a741f2e50 con 0x7f4a74070e50 2024-09-17T05:05:20.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.510+0000 7f4a6bfff640 1 -- 172.21.15.150:0/2803926125 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4a6c00a7f0 con 0x7f4a74070e50 2024-09-17T05:05:20.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.510+0000 7f4a6bfff640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a54073230 0x7f4a54075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:20.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.510+0000 7f4a6bfff640 1 -- 172.21.15.150:0/2803926125 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f4a6c0946c0 con 0x7f4a74070e50 2024-09-17T05:05:20.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.510+0000 7f4a72ffd640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a54073230 0x7f4a54075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:20.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.510+0000 7f4a72ffd640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a54073230 0x7f4a54075740 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c0af000 tx=0x7f4a5c01b090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:20.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.514+0000 7f4a6bfff640 1 -- 172.21.15.150:0/2803926125 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4a6c020060 con 0x7f4a74070e50 2024-09-17T05:05:20.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.721+0000 7f4a79d48640 1 -- 172.21.15.150:0/2803926125 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a740ad7f0 con 0x7f4a74070e50 2024-09-17T05:05:20.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.721+0000 7f4a6bfff640 1 -- 172.21.15.150:0/2803926125 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+6546 (secure 0 0 0) 0x7f4a6c05ed00 con 0x7f4a74070e50 2024-09-17T05:05:20.721 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:20.721 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":111,"btime":"2024-09-17T05:04:44:076711+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:04:39.052846+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6564},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6564":{"gid":6564,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":8,"addr":"172.21.15.150:6833/1984876738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1984876738},{"type":"v1","addr":"172.21.15.150:6833","nonce":1984876738}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6564,"qdb_cluster":[6564]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:20.721 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 111 2024-09-17T05:05:20.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.725+0000 7f4a79d48640 1 -- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a54073230 msgr2=0x7f4a54075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:20.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.725+0000 7f4a79d48640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a54073230 0x7f4a54075740 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c0af000 tx=0x7f4a5c01b090 comp rx=0 tx=0).stop 2024-09-17T05:05:20.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.725+0000 7f4a79d48640 1 -- 172.21.15.150:0/2803926125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a74070e50 msgr2=0x7f4a741f36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:20.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.725+0000 7f4a79d48640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a74070e50 0x7f4a741f36e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00c730 tx=0x7f4a6c00cbf0 comp rx=0 tx=0).stop 2024-09-17T05:05:20.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.725+0000 7f4a79d48640 1 -- 172.21.15.150:0/2803926125 shutdown_connections 2024-09-17T05:05:20.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.725+0000 7f4a79d48640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a54073230 0x7f4a54075740 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:20.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.725+0000 7f4a79d48640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a74073440 0x7f4a7414aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:20.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.725+0000 7f4a79d48640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a740717a0 0x7f4a74151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:20.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.725+0000 7f4a79d48640 1 --2- 172.21.15.150:0/2803926125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a74070e50 0x7f4a741f36e0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:20.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.725+0000 7f4a79d48640 1 -- 172.21.15.150:0/2803926125 >> 172.21.15.150:0/2803926125 conn(0x7f4a740c4a00 msgr2=0x7f4a740c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:20.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.725+0000 7f4a79d48640 1 -- 172.21.15.150:0/2803926125 shutdown_connections 2024-09-17T05:05:20.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:20.725+0000 7f4a79d48640 1 -- 172.21.15.150:0/2803926125 wait complete. 2024-09-17T05:05:20.743 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:05:22.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:22.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.913+0000 7fd26849c640 1 -- 172.21.15.150:0/674755662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600ad7e0 msgr2=0x7fd2600701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:22.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.913+0000 7fd26849c640 1 --2- 172.21.15.150:0/674755662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600ad7e0 0x7fd2600701c0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fd2600c4740 tx=0x7fd25801cd80 comp rx=0 tx=0).stop 2024-09-17T05:05:22.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.913+0000 7fd26849c640 1 -- 172.21.15.150:0/674755662 shutdown_connections 2024-09-17T05:05:22.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.913+0000 7fd26849c640 1 --2- 172.21.15.150:0/674755662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600ad7e0 0x7fd2600701c0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:22.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.913+0000 7fd26849c640 1 --2- 172.21.15.150:0/674755662 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2600c3db0 0x7fd2600ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:22.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.913+0000 7fd26849c640 1 -- 172.21.15.150:0/674755662 >> 172.21.15.150:0/674755662 conn(0x7fd2600c4a00 msgr2=0x7fd2600c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:22.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.913+0000 7fd26849c640 1 -- 172.21.15.150:0/674755662 shutdown_connections 2024-09-17T05:05:22.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.913+0000 7fd26849c640 1 -- 172.21.15.150:0/674755662 wait complete. 2024-09-17T05:05:22.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd26849c640 1 Processor -- start 2024-09-17T05:05:22.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd26849c640 1 -- start start 2024-09-17T05:05:22.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd26849c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600ad7e0 0x7fd2601488d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:22.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd26849c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2600c3db0 0x7fd2601462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:22.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd26849c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2600c41c0 0x7fd26014c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:22.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd26849c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd2600a75f0 con 0x7fd2600ad7e0 2024-09-17T05:05:22.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd26849c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd2600add70 con 0x7fd2600c41c0 2024-09-17T05:05:22.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd26849c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd2600a7770 con 0x7fd2600c3db0 2024-09-17T05:05:22.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd266238640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600ad7e0 0x7fd2601488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:22.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd266238640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600ad7e0 0x7fd2601488d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40174/0 (socket says 172.21.15.150:40174) 2024-09-17T05:05:22.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd266238640 1 -- 172.21.15.150:0/2246236379 learned_addr learned my addr 172.21.15.150:0/2246236379 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:22.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd265a37640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2600c3db0 0x7fd2601462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:22.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd266a39640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2600c41c0 0x7fd26014c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:22.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd266238640 1 -- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2600c3db0 msgr2=0x7fd2601462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:22.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd266238640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2600c3db0 0x7fd2601462c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:22.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd266238640 1 -- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2600c41c0 msgr2=0x7fd26014c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:22.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd266238640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2600c41c0 0x7fd26014c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:22.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd266238640 1 -- 172.21.15.150:0/2246236379 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26015d3c0 con 0x7fd2600ad7e0 2024-09-17T05:05:22.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd266238640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600ad7e0 0x7fd2601488d0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd25c00c970 tx=0x7fd25c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:22.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd256ffd640 1 -- 172.21.15.150:0/2246236379 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25c007bb0 con 0x7fd2600ad7e0 2024-09-17T05:05:22.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd256ffd640 1 -- 172.21.15.150:0/2246236379 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd25c007e40 con 0x7fd2600ad7e0 2024-09-17T05:05:22.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd26849c640 1 -- 172.21.15.150:0/2246236379 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd26015d6a0 con 0x7fd2600ad7e0 2024-09-17T05:05:22.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd256ffd640 1 -- 172.21.15.150:0/2246236379 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25c00f6b0 con 0x7fd2600ad7e0 2024-09-17T05:05:22.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.917+0000 7fd26849c640 1 -- 172.21.15.150:0/2246236379 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd260146800 con 0x7fd2600ad7e0 2024-09-17T05:05:22.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.921+0000 7fd26849c640 1 -- 172.21.15.150:0/2246236379 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd260070d10 con 0x7fd2600ad7e0 2024-09-17T05:05:22.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.921+0000 7fd256ffd640 1 -- 172.21.15.150:0/2246236379 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd25c00f850 con 0x7fd2600ad7e0 2024-09-17T05:05:22.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.921+0000 7fd256ffd640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd244073200 0x7fd244075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:22.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.921+0000 7fd256ffd640 1 -- 172.21.15.150:0/2246236379 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7fd25c0954b0 con 0x7fd2600ad7e0 2024-09-17T05:05:22.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.921+0000 7fd265a37640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd244073200 0x7fd244075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:22.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.921+0000 7fd265a37640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd244073200 0x7fd244075710 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd260148ea0 tx=0x7fd250008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:22.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:22.925+0000 7fd256ffd640 1 -- 172.21.15.150:0/2246236379 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd260070d10 con 0x7fd2600ad7e0 2024-09-17T05:05:23.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.133+0000 7fd26849c640 1 -- 172.21.15.150:0/2246236379 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2600ad2a0 con 0x7fd2600ad7e0 2024-09-17T05:05:23.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.133+0000 7fd256ffd640 1 -- 172.21.15.150:0/2246236379 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) ==== 95+0+5714 (secure 0 0 0) 0x7fd25c014030 con 0x7fd2600ad7e0 2024-09-17T05:05:23.130 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:23.130 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":113,"btime":"2024-09-17T05:05:22:195327+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:21.497599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:23.131 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 113 2024-09-17T05:05:23.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.133+0000 7fd26849c640 1 -- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd244073200 msgr2=0x7fd244075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:23.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.133+0000 7fd26849c640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd244073200 0x7fd244075710 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd260148ea0 tx=0x7fd250008040 comp rx=0 tx=0).stop 2024-09-17T05:05:23.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.133+0000 7fd26849c640 1 -- 172.21.15.150:0/2246236379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600ad7e0 msgr2=0x7fd2601488d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:23.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.133+0000 7fd26849c640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600ad7e0 0x7fd2601488d0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd25c00c970 tx=0x7fd25c00ce30 comp rx=0 tx=0).stop 2024-09-17T05:05:23.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.133+0000 7fd26849c640 1 -- 172.21.15.150:0/2246236379 shutdown_connections 2024-09-17T05:05:23.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.133+0000 7fd26849c640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd244073200 0x7fd244075710 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.133+0000 7fd26849c640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2600c41c0 0x7fd26014c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.133+0000 7fd26849c640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2600c3db0 0x7fd2601462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.133+0000 7fd26849c640 1 --2- 172.21.15.150:0/2246236379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600ad7e0 0x7fd2601488d0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.133+0000 7fd26849c640 1 -- 172.21.15.150:0/2246236379 >> 172.21.15.150:0/2246236379 conn(0x7fd2600c4a00 msgr2=0x7fd260153ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:23.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.137+0000 7fd26849c640 1 -- 172.21.15.150:0/2246236379 shutdown_connections 2024-09-17T05:05:23.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.137+0000 7fd26849c640 1 -- 172.21.15.150:0/2246236379 wait complete. 2024-09-17T05:05:23.157 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-17T05:05:23.157 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:05:23.158 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:23.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.317+0000 7f38d90f1640 1 -- 172.21.15.150:0/953966424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38d40c41c0 msgr2=0x7f38d406fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:23.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.317+0000 7f38d90f1640 1 --2- 172.21.15.150:0/953966424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38d40c41c0 0x7f38d406fab0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009900 tx=0x7f38c401c8e0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.317+0000 7f38d90f1640 1 -- 172.21.15.150:0/953966424 shutdown_connections 2024-09-17T05:05:23.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.317+0000 7f38d90f1640 1 --2- 172.21.15.150:0/953966424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38d40c41c0 0x7f38d406fab0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.317+0000 7f38d90f1640 1 -- 172.21.15.150:0/953966424 >> 172.21.15.150:0/953966424 conn(0x7f38d40c4a00 msgr2=0x7f38d40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:23.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.317+0000 7f38d90f1640 1 -- 172.21.15.150:0/953966424 shutdown_connections 2024-09-17T05:05:23.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.317+0000 7f38d90f1640 1 -- 172.21.15.150:0/953966424 wait complete. 2024-09-17T05:05:23.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.317+0000 7f38d90f1640 1 Processor -- start 2024-09-17T05:05:23.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d90f1640 1 -- start start 2024-09-17T05:05:23.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d90f1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38d406fff0 0x7f38d4156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:23.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d90f1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38d40c3db0 0x7f38d4157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:23.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d90f1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38d40c41c0 0x7f38d4143240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:23.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d90f1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f38d40adb30 con 0x7f38d40c3db0 2024-09-17T05:05:23.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d90f1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f38d40ad9b0 con 0x7f38d40c41c0 2024-09-17T05:05:23.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d90f1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f38d40a75f0 con 0x7f38d406fff0 2024-09-17T05:05:23.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38d40c3db0 0x7f38d4157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:23.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38d40c3db0 0x7f38d4157e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40196/0 (socket says 172.21.15.150:40196) 2024-09-17T05:05:23.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d259c640 1 -- 172.21.15.150:0/841248956 learned_addr learned my addr 172.21.15.150:0/841248956 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:23.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d2d9d640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38d406fff0 0x7f38d4156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:23.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d359e640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38d40c41c0 0x7f38d4143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d259c640 1 -- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38d406fff0 msgr2=0x7f38d4156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d259c640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38d406fff0 0x7f38d4156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d259c640 1 -- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38d40c41c0 msgr2=0x7f38d4143240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d259c640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38d40c41c0 0x7f38d4143240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d259c640 1 -- 172.21.15.150:0/841248956 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d41f1eb0 con 0x7f38d40c3db0 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d2d9d640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38d406fff0 0x7f38d4156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d359e640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38d40c41c0 0x7f38d4143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d259c640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38d40c3db0 0x7f38d4157e60 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009850 tx=0x7f38c40a56b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38bb7fe640 1 -- 172.21.15.150:0/841248956 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c40b2040 con 0x7f38d40c3db0 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38bb7fe640 1 -- 172.21.15.150:0/841248956 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38c4004340 con 0x7f38d40c3db0 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d90f1640 1 -- 172.21.15.150:0/841248956 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d4140200 con 0x7f38d40c3db0 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38bb7fe640 1 -- 172.21.15.150:0/841248956 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c40044f0 con 0x7f38d40c3db0 2024-09-17T05:05:23.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d90f1640 1 -- 172.21.15.150:0/841248956 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d41f1bd0 con 0x7f38d40c3db0 2024-09-17T05:05:23.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.321+0000 7f38d90f1640 1 -- 172.21.15.150:0/841248956 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d41f19c0 con 0x7f38d40c3db0 2024-09-17T05:05:23.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.325+0000 7f38bb7fe640 1 -- 172.21.15.150:0/841248956 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f38c4003860 con 0x7f38d40c3db0 2024-09-17T05:05:23.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.325+0000 7f38bb7fe640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f38c0073200 0x7f38c0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:23.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.325+0000 7f38bb7fe640 1 -- 172.21.15.150:0/841248956 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7f38c412eba0 con 0x7f38d40c3db0 2024-09-17T05:05:23.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.325+0000 7f38d2d9d640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f38c0073200 0x7f38c0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:23.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.325+0000 7f38d2d9d640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f38c0073200 0x7f38c0075710 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f38c8000f70 tx=0x7f38c8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:23.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.329+0000 7f38bb7fe640 1 -- 172.21.15.150:0/841248956 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f38c40164d0 con 0x7f38d40c3db0 2024-09-17T05:05:23.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.525+0000 7f38d90f1640 1 -- 172.21.15.150:0/841248956 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38d406fab0 con 0x7f38d40c3db0 2024-09-17T05:05:23.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.525+0000 7f38bb7fe640 1 -- 172.21.15.150:0/841248956 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+5711 (secure 0 0 0) 0x7f38c40f9190 con 0x7f38d40c3db0 2024-09-17T05:05:23.524 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:23.524 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":114,"btime":"2024-09-17T05:05:23:204734+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:22.212254+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:23.524 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 114 2024-09-17T05:05:23.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.529+0000 7f38d90f1640 1 -- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f38c0073200 msgr2=0x7f38c0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:23.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.529+0000 7f38d90f1640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f38c0073200 0x7f38c0075710 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f38c8000f70 tx=0x7f38c8009040 comp rx=0 tx=0).stop 2024-09-17T05:05:23.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.529+0000 7f38d90f1640 1 -- 172.21.15.150:0/841248956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38d40c3db0 msgr2=0x7f38d4157e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:23.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.529+0000 7f38d90f1640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38d40c3db0 0x7f38d4157e60 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009850 tx=0x7f38c40a56b0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.529+0000 7f38d90f1640 1 -- 172.21.15.150:0/841248956 shutdown_connections 2024-09-17T05:05:23.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.529+0000 7f38d90f1640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f38c0073200 0x7f38c0075710 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.529+0000 7f38d90f1640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38d40c41c0 0x7f38d4143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.529+0000 7f38d90f1640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38d40c3db0 0x7f38d4157e60 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.529+0000 7f38d90f1640 1 --2- 172.21.15.150:0/841248956 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38d406fff0 0x7f38d4156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:23.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.529+0000 7f38d90f1640 1 -- 172.21.15.150:0/841248956 >> 172.21.15.150:0/841248956 conn(0x7f38d40c4a00 msgr2=0x7f38d406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:23.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.529+0000 7f38d90f1640 1 -- 172.21.15.150:0/841248956 shutdown_connections 2024-09-17T05:05:23.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:23.529+0000 7f38d90f1640 1 -- 172.21.15.150:0/841248956 wait complete. 2024-09-17T05:05:23.546 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:05:23:204734+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 114, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 114, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6229": { "addr": "172.21.15.176:6837/3940723333", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 3940723333, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 3940723333, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6229, "incarnation": 112, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:05:22.212254+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6229 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 105, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6495": { "addr": "172.21.15.150:6835/3807957687", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 3807957687, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 3807957687, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6495, "incarnation": 102, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:04:09.506271+0000", "qdb_cluster": [ 6495 ], "qdb_leader": 6495, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6495 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:05:25.547 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:25.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 -- 172.21.15.150:0/1919897773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c6340 msgr2=0x7f94ac157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:25.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 --2- 172.21.15.150:0/1919897773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c6340 0x7f94ac157330 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f94ac157020 tx=0x7f949401c850 comp rx=0 tx=0).stop 2024-09-17T05:05:25.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 -- 172.21.15.150:0/1919897773 shutdown_connections 2024-09-17T05:05:25.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 --2- 172.21.15.150:0/1919897773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c6340 0x7f94ac157330 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:25.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 -- 172.21.15.150:0/1919897773 >> 172.21.15.150:0/1919897773 conn(0x7f94ac074750 msgr2=0x7f94ac074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:25.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 -- 172.21.15.150:0/1919897773 shutdown_connections 2024-09-17T05:05:25.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 -- 172.21.15.150:0/1919897773 wait complete. 2024-09-17T05:05:25.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 Processor -- start 2024-09-17T05:05:25.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 -- start start 2024-09-17T05:05:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94ac0c4b20 0x7f94ac158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:25.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94ac0c5df0 0x7f94ac1f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c6340 0x7f94ac147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac164b10 con 0x7f94ac0c6340 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac164990 con 0x7f94ac0c5df0 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94b0de0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f94ac164c90 con 0x7f94ac0c4b20 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94aad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c6340 0x7f94ac147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94aad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c6340 0x7f94ac147ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40210/0 (socket says 172.21.15.150:40210) 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.733+0000 7f94aad9d640 1 -- 172.21.15.150:0/3850735705 learned_addr learned my addr 172.21.15.150:0/3850735705 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94a3fff640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94ac0c5df0 0x7f94ac1f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94aa59c640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94ac0c4b20 0x7f94ac158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94aad9d640 1 -- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94ac0c4b20 msgr2=0x7f94ac158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94aad9d640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94ac0c4b20 0x7f94ac158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94aad9d640 1 -- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94ac0c5df0 msgr2=0x7f94ac1f38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94aad9d640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94ac0c5df0 0x7f94ac1f38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:25.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94aad9d640 1 -- 172.21.15.150:0/3850735705 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94ac14a510 con 0x7f94ac0c6340 2024-09-17T05:05:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94aad9d640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c6340 0x7f94ac147ba0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f94a400c760 tx=0x7f94a400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94a2ffd640 1 -- 172.21.15.150:0/3850735705 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a4003b80 con 0x7f94ac0c6340 2024-09-17T05:05:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94a2ffd640 1 -- 172.21.15.150:0/3850735705 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94a4003d20 con 0x7f94ac0c6340 2024-09-17T05:05:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94a2ffd640 1 -- 172.21.15.150:0/3850735705 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a400aab0 con 0x7f94ac0c6340 2024-09-17T05:05:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94b0de0640 1 -- 172.21.15.150:0/3850735705 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94ac1f41a0 con 0x7f94ac0c6340 2024-09-17T05:05:25.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94b0de0640 1 -- 172.21.15.150:0/3850735705 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94ac1f4440 con 0x7f94ac0c6340 2024-09-17T05:05:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94b0de0640 1 -- 172.21.15.150:0/3850735705 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94ac157f40 con 0x7f94ac0c6340 2024-09-17T05:05:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.737+0000 7f94a2ffd640 1 -- 172.21.15.150:0/3850735705 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f94a400ac50 con 0x7f94ac0c6340 2024-09-17T05:05:25.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.741+0000 7f94a2ffd640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f948c073200 0x7f948c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.741+0000 7f94a2ffd640 1 -- 172.21.15.150:0/3850735705 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7f94a4094710 con 0x7f94ac0c6340 2024-09-17T05:05:25.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.741+0000 7f94aa59c640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f948c073200 0x7f948c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:25.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.741+0000 7f94aa59c640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f948c073200 0x7f948c075710 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f949c00b810 tx=0x7f949c008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:25.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.745+0000 7f94a2ffd640 1 -- 172.21.15.150:0/3850735705 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f94a4020060 con 0x7f94ac0c6340 2024-09-17T05:05:25.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 -- 172.21.15.150:0/3850735705 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94ac157330 con 0x7f94ac0c6340 2024-09-17T05:05:25.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94a2ffd640 1 -- 172.21.15.150:0/3850735705 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+5718 (secure 0 0 0) 0x7f94ac0b20f0 con 0x7f94ac0c6340 2024-09-17T05:05:25.982 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:25.982 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":115,"btime":"2024-09-17T05:05:24:217144+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:25.983 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 115 2024-09-17T05:05:25.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 -- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f948c073200 msgr2=0x7f948c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:25.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f948c073200 0x7f948c075710 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f949c00b810 tx=0x7f949c008040 comp rx=0 tx=0).stop 2024-09-17T05:05:25.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 -- 172.21.15.150:0/3850735705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c6340 msgr2=0x7f94ac147ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:25.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c6340 0x7f94ac147ba0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f94a400c760 tx=0x7f94a400cc20 comp rx=0 tx=0).stop 2024-09-17T05:05:25.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 -- 172.21.15.150:0/3850735705 shutdown_connections 2024-09-17T05:05:25.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f948c073200 0x7f948c075710 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:25.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94ac0c5df0 0x7f94ac1f38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:25.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94ac0c6340 0x7f94ac147ba0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:25.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 --2- 172.21.15.150:0/3850735705 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94ac0c4b20 0x7f94ac158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:25.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 -- 172.21.15.150:0/3850735705 >> 172.21.15.150:0/3850735705 conn(0x7f94ac074750 msgr2=0x7f94ac152180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:25.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 -- 172.21.15.150:0/3850735705 shutdown_connections 2024-09-17T05:05:25.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:25.985+0000 7f94b0de0640 1 -- 172.21.15.150:0/3850735705 wait complete. 2024-09-17T05:05:26.008 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=6564 2024-09-17T05:05:26.008 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.a 2024-09-17T05:05:29.089 INFO:tasks.ceph.mds.d.smithi176.stderr:2024-09-17T05:05:29.085+0000 7f1b53fd8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:05:32.373 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:05:32.377 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:05:33.009 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-17T05:05:33.034 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T05:05:33.035 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T05:05:33.038 INFO:tasks.ceph.mds.a:Started 2024-09-17T05:05:33.038 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:33.147 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T05:05:33.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.221+0000 7fbe978b1640 1 -- 172.21.15.150:0/3422176509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe900c4250 msgr2=0x7fbe90153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:33.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.221+0000 7fbe978b1640 1 --2- 172.21.15.150:0/3422176509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe900c4250 0x7fbe90153ee0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009900 tx=0x7fbe8c01c8e0 comp rx=0 tx=0).stop 2024-09-17T05:05:33.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.221+0000 7fbe978b1640 1 -- 172.21.15.150:0/3422176509 shutdown_connections 2024-09-17T05:05:33.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.221+0000 7fbe978b1640 1 --2- 172.21.15.150:0/3422176509 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe900c3900 0x7fbe90074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:33.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.221+0000 7fbe978b1640 1 --2- 172.21.15.150:0/3422176509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe900c4250 0x7fbe90153ee0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:33.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.221+0000 7fbe978b1640 1 -- 172.21.15.150:0/3422176509 >> 172.21.15.150:0/3422176509 conn(0x7fbe9006e360 msgr2=0x7fbe9006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:33.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.221+0000 7fbe978b1640 1 -- 172.21.15.150:0/3422176509 shutdown_connections 2024-09-17T05:05:33.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.221+0000 7fbe978b1640 1 -- 172.21.15.150:0/3422176509 wait complete. 2024-09-17T05:05:33.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.221+0000 7fbe978b1640 1 Processor -- start 2024-09-17T05:05:33.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.221+0000 7fbe978b1640 1 -- start start 2024-09-17T05:05:33.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe978b1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe90074e60 0x7fbe9006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:33.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe978b1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe900c3900 0x7fbe901efe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:33.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe978b1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe900c4250 0x7fbe900bed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:33.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe978b1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbe90072f30 con 0x7fbe90074e60 2024-09-17T05:05:33.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe978b1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbe90072db0 con 0x7fbe900c4250 2024-09-17T05:05:33.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe978b1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbe90160f90 con 0x7fbe900c3900 2024-09-17T05:05:33.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe9564d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe90074e60 0x7fbe9006d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:33.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe9564d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe90074e60 0x7fbe9006d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48200/0 (socket says 172.21.15.150:48200) 2024-09-17T05:05:33.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe9564d640 1 -- 172.21.15.150:0/2534048714 learned_addr learned my addr 172.21.15.150:0/2534048714 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:33.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe94e4c640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe900c3900 0x7fbe901efe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:33.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe95e4e640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe900c4250 0x7fbe900bed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:33.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe9564d640 1 -- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe900c3900 msgr2=0x7fbe901efe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:33.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe9564d640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe900c3900 0x7fbe901efe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:33.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe9564d640 1 -- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe900c4250 msgr2=0x7fbe900bed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:33.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe9564d640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe900c4250 0x7fbe900bed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:33.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe9564d640 1 -- 172.21.15.150:0/2534048714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe900ae7f0 con 0x7fbe90074e60 2024-09-17T05:05:33.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe95e4e640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe900c4250 0x7fbe900bed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:05:33.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe9564d640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe90074e60 0x7fbe9006d020 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c0b0040 tx=0x7fbe8c005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:33.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe85ffb640 1 -- 172.21.15.150:0/2534048714 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c0b2040 con 0x7fbe90074e60 2024-09-17T05:05:33.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe85ffb640 1 -- 172.21.15.150:0/2534048714 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe8c0a5c40 con 0x7fbe90074e60 2024-09-17T05:05:33.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe978b1640 1 -- 172.21.15.150:0/2534048714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe901ef380 con 0x7fbe90074e60 2024-09-17T05:05:33.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe85ffb640 1 -- 172.21.15.150:0/2534048714 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c005020 con 0x7fbe90074e60 2024-09-17T05:05:33.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe978b1640 1 -- 172.21.15.150:0/2534048714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe901ef940 con 0x7fbe90074e60 2024-09-17T05:05:33.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe978b1640 1 -- 172.21.15.150:0/2534048714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe90074920 con 0x7fbe90074e60 2024-09-17T05:05:33.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe85ffb640 1 -- 172.21.15.150:0/2534048714 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbe8c0bd060 con 0x7fbe90074e60 2024-09-17T05:05:33.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.225+0000 7fbe85ffb640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe740731f0 0x7fbe74075700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:33.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.229+0000 7fbe85ffb640 1 -- 172.21.15.150:0/2534048714 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7fbe8c12e490 con 0x7fbe90074e60 2024-09-17T05:05:33.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.229+0000 7fbe94e4c640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe740731f0 0x7fbe74075700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:33.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.229+0000 7fbe94e4c640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe740731f0 0x7fbe74075700 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbe900a88b0 tx=0x7fbe8000b2e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:33.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.233+0000 7fbe85ffb640 1 -- 172.21.15.150:0/2534048714 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbe8c0164d0 con 0x7fbe90074e60 2024-09-17T05:05:33.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.521+0000 7fbe978b1640 1 -- 172.21.15.150:0/2534048714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe90152fa0 con 0x7fbe90074e60 2024-09-17T05:05:33.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.521+0000 7fbe85ffb640 1 -- 172.21.15.150:0/2534048714 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+6547 (secure 0 0 0) 0x7fbe8c0f8a80 con 0x7fbe90074e60 2024-09-17T05:05:33.520 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:33.520 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":116,"btime":"2024-09-17T05:05:33:241748+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:33.521 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 116 2024-09-17T05:05:33.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.525+0000 7fbe978b1640 1 -- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe740731f0 msgr2=0x7fbe74075700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:33.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.525+0000 7fbe978b1640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe740731f0 0x7fbe74075700 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbe900a88b0 tx=0x7fbe8000b2e0 comp rx=0 tx=0).stop 2024-09-17T05:05:33.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.525+0000 7fbe978b1640 1 -- 172.21.15.150:0/2534048714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe90074e60 msgr2=0x7fbe9006d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:33.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.525+0000 7fbe978b1640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe90074e60 0x7fbe9006d020 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c0b0040 tx=0x7fbe8c005e00 comp rx=0 tx=0).stop 2024-09-17T05:05:33.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.525+0000 7fbe978b1640 1 -- 172.21.15.150:0/2534048714 shutdown_connections 2024-09-17T05:05:33.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.525+0000 7fbe978b1640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe740731f0 0x7fbe74075700 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:33.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.525+0000 7fbe978b1640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe900c4250 0x7fbe900bed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:33.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.525+0000 7fbe978b1640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe900c3900 0x7fbe901efe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:33.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.525+0000 7fbe978b1640 1 --2- 172.21.15.150:0/2534048714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe90074e60 0x7fbe9006d020 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:33.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.525+0000 7fbe978b1640 1 -- 172.21.15.150:0/2534048714 >> 172.21.15.150:0/2534048714 conn(0x7fbe9006e360 msgr2=0x7fbe9006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:33.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.525+0000 7fbe978b1640 1 -- 172.21.15.150:0/2534048714 shutdown_connections 2024-09-17T05:05:33.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:33.525+0000 7fbe978b1640 1 -- 172.21.15.150:0/2534048714 wait complete. 2024-09-17T05:05:33.543 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-17T05:05:33.544 INFO:tasks.mds_thrash.fs.[a]:waiting for 58 secs before thrashing 2024-09-17T05:05:39.688 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:39.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.865+0000 7fa876ce1640 1 -- 172.21.15.150:0/2596811922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8700c4250 msgr2=0x7fa870153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:39.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.865+0000 7fa876ce1640 1 --2- 172.21.15.150:0/2596811922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8700c4250 0x7fa870153ee0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fa864009930 tx=0x7fa86401c8e0 comp rx=0 tx=0).stop 2024-09-17T05:05:39.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.865+0000 7fa876ce1640 1 -- 172.21.15.150:0/2596811922 shutdown_connections 2024-09-17T05:05:39.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.865+0000 7fa876ce1640 1 --2- 172.21.15.150:0/2596811922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8700c4250 0x7fa870153ee0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:39.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.865+0000 7fa876ce1640 1 -- 172.21.15.150:0/2596811922 >> 172.21.15.150:0/2596811922 conn(0x7fa87006e360 msgr2=0x7fa87006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:39.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.865+0000 7fa876ce1640 1 -- 172.21.15.150:0/2596811922 shutdown_connections 2024-09-17T05:05:39.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.865+0000 7fa876ce1640 1 -- 172.21.15.150:0/2596811922 wait complete. 2024-09-17T05:05:39.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa876ce1640 1 Processor -- start 2024-09-17T05:05:39.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa876ce1640 1 -- start start 2024-09-17T05:05:39.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa876ce1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8700c3900 0x7fa8700c1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:39.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa876ce1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa8700c3d10 0x7fa87006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:39.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa876ce1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8700c4250 0x7fa87014d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:39.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa876ce1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa870071120 con 0x7fa8700c4250 2024-09-17T05:05:39.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa876ce1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa870070fa0 con 0x7fa8700c3900 2024-09-17T05:05:39.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa876ce1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa870072db0 con 0x7fa8700c3d10 2024-09-17T05:05:39.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa87527e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8700c4250 0x7fa87014d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:39.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa87527e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8700c4250 0x7fa87014d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42958/0 (socket says 172.21.15.150:42958) 2024-09-17T05:05:39.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa87527e640 1 -- 172.21.15.150:0/911046283 learned_addr learned my addr 172.21.15.150:0/911046283 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:39.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa874a7d640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8700c3900 0x7fa8700c1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:39.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.869+0000 7fa86ffff640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa8700c3d10 0x7fa87006d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:39.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa87527e640 1 -- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa8700c3d10 msgr2=0x7fa87006d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:39.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa87527e640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa8700c3d10 0x7fa87006d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:39.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa87527e640 1 -- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8700c3900 msgr2=0x7fa8700c1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:39.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa87527e640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8700c3900 0x7fa8700c1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:39.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa87527e640 1 -- 172.21.15.150:0/911046283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8700c2550 con 0x7fa8700c4250 2024-09-17T05:05:39.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa87527e640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8700c4250 0x7fa87014d0d0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fa86800ba40 tx=0x7fa86800bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:39.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa86d7fa640 1 -- 172.21.15.150:0/911046283 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868003b80 con 0x7fa8700c4250 2024-09-17T05:05:39.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa86d7fa640 1 -- 172.21.15.150:0/911046283 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa868003e10 con 0x7fa8700c4250 2024-09-17T05:05:39.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa876ce1640 1 -- 172.21.15.150:0/911046283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8700c3210 con 0x7fa8700c4250 2024-09-17T05:05:39.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa86d7fa640 1 -- 172.21.15.150:0/911046283 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86800cbe0 con 0x7fa8700c4250 2024-09-17T05:05:39.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa876ce1640 1 -- 172.21.15.150:0/911046283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8700c34f0 con 0x7fa8700c4250 2024-09-17T05:05:39.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa876ce1640 1 -- 172.21.15.150:0/911046283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa870153ee0 con 0x7fa8700c4250 2024-09-17T05:05:39.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa86d7fa640 1 -- 172.21.15.150:0/911046283 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa868006d10 con 0x7fa8700c4250 2024-09-17T05:05:39.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa86d7fa640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa854073230 0x7fa854075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:39.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa86d7fa640 1 -- 172.21.15.150:0/911046283 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7fa868094880 con 0x7fa8700c4250 2024-09-17T05:05:39.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.873+0000 7fa874a7d640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa854073230 0x7fa854075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:39.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.877+0000 7fa874a7d640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa854073230 0x7fa854075740 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa8640af000 tx=0x7fa86401b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:39.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:39.881+0000 7fa86d7fa640 1 -- 172.21.15.150:0/911046283 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa86805edd0 con 0x7fa8700c4250 2024-09-17T05:05:40.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.133+0000 7fa876ce1640 1 -- 172.21.15.150:0/911046283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa870072690 con 0x7fa8700c4250 2024-09-17T05:05:40.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.133+0000 7fa86d7fa640 1 -- 172.21.15.150:0/911046283 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+6547 (secure 0 0 0) 0x7fa86805e6d0 con 0x7fa8700c4250 2024-09-17T05:05:40.132 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:40.133 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":116,"btime":"2024-09-17T05:05:33:241748+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:40.133 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 116 2024-09-17T05:05:40.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.137+0000 7fa876ce1640 1 -- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa854073230 msgr2=0x7fa854075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:40.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.137+0000 7fa876ce1640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa854073230 0x7fa854075740 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa8640af000 tx=0x7fa86401b120 comp rx=0 tx=0).stop 2024-09-17T05:05:40.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.137+0000 7fa876ce1640 1 -- 172.21.15.150:0/911046283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8700c4250 msgr2=0x7fa87014d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:40.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.137+0000 7fa876ce1640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8700c4250 0x7fa87014d0d0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fa86800ba40 tx=0x7fa86800bf00 comp rx=0 tx=0).stop 2024-09-17T05:05:40.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.137+0000 7fa876ce1640 1 -- 172.21.15.150:0/911046283 shutdown_connections 2024-09-17T05:05:40.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.137+0000 7fa876ce1640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa854073230 0x7fa854075740 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.137+0000 7fa876ce1640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8700c4250 0x7fa87014d0d0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.137+0000 7fa876ce1640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa8700c3d10 0x7fa87006d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.137+0000 7fa876ce1640 1 --2- 172.21.15.150:0/911046283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8700c3900 0x7fa8700c1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.137+0000 7fa876ce1640 1 -- 172.21.15.150:0/911046283 >> 172.21.15.150:0/911046283 conn(0x7fa87006e360 msgr2=0x7fa8700c68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:40.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.137+0000 7fa876ce1640 1 -- 172.21.15.150:0/911046283 shutdown_connections 2024-09-17T05:05:40.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.137+0000 7fa876ce1640 1 -- 172.21.15.150:0/911046283 wait complete. 2024-09-17T05:05:40.157 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-17T05:05:40.158 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T05:05:40.158 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:05:40.172 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T05:05:40.172 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:40.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.381+0000 7fdd8db3d640 1 -- 172.21.15.150:0/1541036365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd880c4250 msgr2=0x7fdd88153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:40.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.381+0000 7fdd8db3d640 1 --2- 172.21.15.150:0/1541036365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd880c4250 0x7fdd88153ee0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fdd78009900 tx=0x7fdd7801c8e0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.381+0000 7fdd8db3d640 1 -- 172.21.15.150:0/1541036365 shutdown_connections 2024-09-17T05:05:40.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.381+0000 7fdd8db3d640 1 --2- 172.21.15.150:0/1541036365 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd880c3d10 0x7fdd8815df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.381+0000 7fdd8db3d640 1 --2- 172.21.15.150:0/1541036365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd880c4250 0x7fdd88153ee0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.381+0000 7fdd8db3d640 1 -- 172.21.15.150:0/1541036365 >> 172.21.15.150:0/1541036365 conn(0x7fdd8806e360 msgr2=0x7fdd8806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:40.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.381+0000 7fdd8db3d640 1 -- 172.21.15.150:0/1541036365 shutdown_connections 2024-09-17T05:05:40.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.381+0000 7fdd8db3d640 1 -- 172.21.15.150:0/1541036365 wait complete. 2024-09-17T05:05:40.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd8db3d640 1 Processor -- start 2024-09-17T05:05:40.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd8db3d640 1 -- start start 2024-09-17T05:05:40.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd8db3d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880c3900 0x7fdd8814f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:40.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd8db3d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd880c3d10 0x7fdd880bf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:40.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd8db3d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd880c4250 0x7fdd881f13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:40.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd8db3d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdd88071120 con 0x7fdd880c3d10 2024-09-17T05:05:40.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd8db3d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd88070fa0 con 0x7fdd880c3900 2024-09-17T05:05:40.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd8db3d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd88072db0 con 0x7fdd880c4250 2024-09-17T05:05:40.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd7ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd880c3d10 0x7fdd880bf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:40.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd7ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd880c3d10 0x7fdd880bf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42980/0 (socket says 172.21.15.150:42980) 2024-09-17T05:05:40.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd7ffff640 1 -- 172.21.15.150:0/2865687123 learned_addr learned my addr 172.21.15.150:0/2865687123 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:40.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd877fe640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880c3900 0x7fdd8814f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:40.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd87fff640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd880c4250 0x7fdd881f13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:40.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd7ffff640 1 -- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd880c4250 msgr2=0x7fdd881f13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:40.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd7ffff640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd880c4250 0x7fdd881f13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd7ffff640 1 -- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880c3900 msgr2=0x7fdd8814f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:40.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd7ffff640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880c3900 0x7fdd8814f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd7ffff640 1 -- 172.21.15.150:0/2865687123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8814b020 con 0x7fdd880c3d10 2024-09-17T05:05:40.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd877fe640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880c3900 0x7fdd8814f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:05:40.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd7ffff640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd880c3d10 0x7fdd880bf340 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000c3c0 tx=0x7fdd7000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:40.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd84ff9640 1 -- 172.21.15.150:0/2865687123 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd70017070 con 0x7fdd880c3d10 2024-09-17T05:05:40.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd8db3d640 1 -- 172.21.15.150:0/2865687123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd880be6f0 con 0x7fdd880c3d10 2024-09-17T05:05:40.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd84ff9640 1 -- 172.21.15.150:0/2865687123 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd700053c0 con 0x7fdd880c3d10 2024-09-17T05:05:40.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd84ff9640 1 -- 172.21.15.150:0/2865687123 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd70003840 con 0x7fdd880c3d10 2024-09-17T05:05:40.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd8db3d640 1 -- 172.21.15.150:0/2865687123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd880be9d0 con 0x7fdd880c3d10 2024-09-17T05:05:40.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.385+0000 7fdd8db3d640 1 -- 172.21.15.150:0/2865687123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd88153ee0 con 0x7fdd880c3d10 2024-09-17T05:05:40.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.389+0000 7fdd84ff9640 1 -- 172.21.15.150:0/2865687123 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fdd7001a070 con 0x7fdd880c3d10 2024-09-17T05:05:40.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.389+0000 7fdd84ff9640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd68073200 0x7fdd68075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:40.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.389+0000 7fdd84ff9640 1 -- 172.21.15.150:0/2865687123 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7fdd70093ed0 con 0x7fdd880c3d10 2024-09-17T05:05:40.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.389+0000 7fdd877fe640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd68073200 0x7fdd68075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:40.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.389+0000 7fdd877fe640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd68073200 0x7fdd68075710 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdd780af000 tx=0x7fdd7801b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:40.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.393+0000 7fdd84ff9640 1 -- 172.21.15.150:0/2865687123 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdd7005de70 con 0x7fdd880c3d10 2024-09-17T05:05:40.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.629+0000 7fdd8db3d640 1 -- 172.21.15.150:0/2865687123 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd880a8060 con 0x7fdd880c3d10 2024-09-17T05:05:40.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.629+0000 7fdd84ff9640 1 -- 172.21.15.150:0/2865687123 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+6547 (secure 0 0 0) 0x7fdd700628a0 con 0x7fdd880c3d10 2024-09-17T05:05:40.627 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:40.627 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":116,"btime":"2024-09-17T05:05:33:241748+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:40.628 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 116 2024-09-17T05:05:40.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.633+0000 7fdd8db3d640 1 -- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd68073200 msgr2=0x7fdd68075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:40.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.633+0000 7fdd8db3d640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd68073200 0x7fdd68075710 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdd780af000 tx=0x7fdd7801b120 comp rx=0 tx=0).stop 2024-09-17T05:05:40.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.633+0000 7fdd8db3d640 1 -- 172.21.15.150:0/2865687123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd880c3d10 msgr2=0x7fdd880bf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:40.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.633+0000 7fdd8db3d640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd880c3d10 0x7fdd880bf340 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fdd7000c3c0 tx=0x7fdd7000c880 comp rx=0 tx=0).stop 2024-09-17T05:05:40.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.633+0000 7fdd8db3d640 1 -- 172.21.15.150:0/2865687123 shutdown_connections 2024-09-17T05:05:40.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.633+0000 7fdd8db3d640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd68073200 0x7fdd68075710 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.633+0000 7fdd8db3d640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd880c4250 0x7fdd881f13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.633+0000 7fdd8db3d640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd880c3d10 0x7fdd880bf340 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.633+0000 7fdd8db3d640 1 --2- 172.21.15.150:0/2865687123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880c3900 0x7fdd8814f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:40.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.633+0000 7fdd8db3d640 1 -- 172.21.15.150:0/2865687123 >> 172.21.15.150:0/2865687123 conn(0x7fdd8806e360 msgr2=0x7fdd880c7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:40.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.633+0000 7fdd8db3d640 1 -- 172.21.15.150:0/2865687123 shutdown_connections 2024-09-17T05:05:40.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:40.633+0000 7fdd8db3d640 1 -- 172.21.15.150:0/2865687123 wait complete. 2024-09-17T05:05:40.652 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:05:42.653 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:42.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.833+0000 7fc9f1863640 1 -- 172.21.15.150:0/315974246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d4005360 msgr2=0x7fc9d400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:42.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.833+0000 7fc9f1863640 1 --2- 172.21.15.150:0/315974246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d4005360 0x7fc9d400b9d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400c5c0 tx=0x7fc9e401fa40 comp rx=0 tx=0).stop 2024-09-17T05:05:42.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.833+0000 7fc9f1863640 1 -- 172.21.15.150:0/315974246 shutdown_connections 2024-09-17T05:05:42.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.833+0000 7fc9f1863640 1 --2- 172.21.15.150:0/315974246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9d4005360 0x7fc9d400b9d0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:42.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.833+0000 7fc9f1863640 1 -- 172.21.15.150:0/315974246 >> 172.21.15.150:0/315974246 conn(0x7fc9ec06e360 msgr2=0x7fc9ec06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:42.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.833+0000 7fc9f1863640 1 -- 172.21.15.150:0/315974246 shutdown_connections 2024-09-17T05:05:42.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.833+0000 7fc9f1863640 1 -- 172.21.15.150:0/315974246 wait complete. 2024-09-17T05:05:42.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9f1863640 1 Processor -- start 2024-09-17T05:05:42.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9f1863640 1 -- start start 2024-09-17T05:05:42.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9f1863640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9d4005360 0x7fc9ec0c2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:42.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9f1863640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ec0c27a0 0x7fc9ec151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:42.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9f1863640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ec1522a0 0x7fc9ec1edb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:42.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9f1863640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec157520 con 0x7fc9ec0c27a0 2024-09-17T05:05:42.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9f1863640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec1573a0 con 0x7fc9d4005360 2024-09-17T05:05:42.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9f1863640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc9ec157ae0 con 0x7fc9ec1522a0 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9ea7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ec0c27a0 0x7fc9ec151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9ea7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ec0c27a0 0x7fc9ec151d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42998/0 (socket says 172.21.15.150:42998) 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9ea7fc640 1 -- 172.21.15.150:0/281950650 learned_addr learned my addr 172.21.15.150:0/281950650 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9eaffd640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9d4005360 0x7fc9ec0c2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9eb7fe640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ec1522a0 0x7fc9ec1edb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9ea7fc640 1 -- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ec1522a0 msgr2=0x7fc9ec1edb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9ea7fc640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ec1522a0 0x7fc9ec1edb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9ea7fc640 1 -- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9d4005360 msgr2=0x7fc9ec0c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9ea7fc640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9d4005360 0x7fc9ec0c2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9ea7fc640 1 -- 172.21.15.150:0/281950650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ec0a8330 con 0x7fc9ec0c27a0 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9eb7fe640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ec1522a0 0x7fc9ec1edb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9ea7fc640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ec0c27a0 0x7fc9ec151d60 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0011820 tx=0x7fc9e0011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9d3fff640 1 -- 172.21.15.150:0/281950650 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e001f070 con 0x7fc9ec0c27a0 2024-09-17T05:05:42.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9d3fff640 1 -- 172.21.15.150:0/281950650 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9e000fe40 con 0x7fc9ec0c27a0 2024-09-17T05:05:42.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9f1863640 1 -- 172.21.15.150:0/281950650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9ec1ee770 con 0x7fc9ec0c27a0 2024-09-17T05:05:42.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9d3fff640 1 -- 172.21.15.150:0/281950650 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e00126b0 con 0x7fc9ec0c27a0 2024-09-17T05:05:42.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.837+0000 7fc9f1863640 1 -- 172.21.15.150:0/281950650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9ec1e09a0 con 0x7fc9ec0c27a0 2024-09-17T05:05:42.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.841+0000 7fc9f1863640 1 -- 172.21.15.150:0/281950650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9ec157fd0 con 0x7fc9ec0c27a0 2024-09-17T05:05:42.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.841+0000 7fc9d3fff640 1 -- 172.21.15.150:0/281950650 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc9e0022070 con 0x7fc9ec0c27a0 2024-09-17T05:05:42.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.841+0000 7fc9d3fff640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc9d8073200 0x7fc9d8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:42.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.841+0000 7fc9d3fff640 1 -- 172.21.15.150:0/281950650 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7fc9e0095dd0 con 0x7fc9ec0c27a0 2024-09-17T05:05:42.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.841+0000 7fc9eaffd640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc9d8073200 0x7fc9d8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:42.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.841+0000 7fc9eaffd640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc9d8073200 0x7fc9d8075710 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc001100 tx=0x7fc9dc006a10 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:42.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:42.845+0000 7fc9d3fff640 1 -- 172.21.15.150:0/281950650 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc9e005fe70 con 0x7fc9ec0c27a0 2024-09-17T05:05:43.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.081+0000 7fc9f1863640 1 -- 172.21.15.150:0/281950650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9ec06dab0 con 0x7fc9ec0c27a0 2024-09-17T05:05:43.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.081+0000 7fc9d3fff640 1 -- 172.21.15.150:0/281950650 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+6547 (secure 0 0 0) 0x7fc9e00648a0 con 0x7fc9ec0c27a0 2024-09-17T05:05:43.080 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:43.080 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":116,"btime":"2024-09-17T05:05:33:241748+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:43.080 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 116 2024-09-17T05:05:43.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.085+0000 7fc9f1863640 1 -- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc9d8073200 msgr2=0x7fc9d8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:43.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.085+0000 7fc9f1863640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc9d8073200 0x7fc9d8075710 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc001100 tx=0x7fc9dc006a10 comp rx=0 tx=0).stop 2024-09-17T05:05:43.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.085+0000 7fc9f1863640 1 -- 172.21.15.150:0/281950650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ec0c27a0 msgr2=0x7fc9ec151d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:43.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.085+0000 7fc9f1863640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ec0c27a0 0x7fc9ec151d60 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0011820 tx=0x7fc9e0011ce0 comp rx=0 tx=0).stop 2024-09-17T05:05:43.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.085+0000 7fc9f1863640 1 -- 172.21.15.150:0/281950650 shutdown_connections 2024-09-17T05:05:43.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.085+0000 7fc9f1863640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc9d8073200 0x7fc9d8075710 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:43.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.085+0000 7fc9f1863640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ec1522a0 0x7fc9ec1edb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:43.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.085+0000 7fc9f1863640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9ec0c27a0 0x7fc9ec151d60 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:43.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.085+0000 7fc9f1863640 1 --2- 172.21.15.150:0/281950650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9d4005360 0x7fc9ec0c2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:43.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.085+0000 7fc9f1863640 1 -- 172.21.15.150:0/281950650 >> 172.21.15.150:0/281950650 conn(0x7fc9ec06e360 msgr2=0x7fc9ec0adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:43.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.085+0000 7fc9f1863640 1 -- 172.21.15.150:0/281950650 shutdown_connections 2024-09-17T05:05:43.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:43.085+0000 7fc9f1863640 1 -- 172.21.15.150:0/281950650 wait complete. 2024-09-17T05:05:43.104 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:05:45.106 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:45.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.297+0000 7f3b43fff640 1 -- 172.21.15.150:0/161892326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b440c2540 msgr2=0x7f3b44154080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:45.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.297+0000 7f3b43fff640 1 --2- 172.21.15.150:0/161892326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b440c2540 0x7f3b44154080 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3b440c2b40 tx=0x7f3b3401c800 comp rx=0 tx=0).stop 2024-09-17T05:05:45.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.297+0000 7f3b43fff640 1 -- 172.21.15.150:0/161892326 shutdown_connections 2024-09-17T05:05:45.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.297+0000 7f3b43fff640 1 --2- 172.21.15.150:0/161892326 >> v2:172.21.15.176:3300/0 conn(0x7f3b44074660 0x7f3b44071190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:45.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.297+0000 7f3b43fff640 1 --2- 172.21.15.150:0/161892326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b440c2540 0x7f3b44154080 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:45.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.297+0000 7f3b43fff640 1 -- 172.21.15.150:0/161892326 >> 172.21.15.150:0/161892326 conn(0x7f3b44073970 msgr2=0x7f3b44073d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:45.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.297+0000 7f3b43fff640 1 -- 172.21.15.150:0/161892326 shutdown_connections 2024-09-17T05:05:45.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.297+0000 7f3b43fff640 1 -- 172.21.15.150:0/161892326 wait complete. 2024-09-17T05:05:45.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.297+0000 7f3b43fff640 1 Processor -- start 2024-09-17T05:05:45.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.297+0000 7f3b43fff640 1 -- start start 2024-09-17T05:05:45.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b43fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b440716d0 0x7f3b441571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:45.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b43fff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b44074660 0x7f3b440c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:45.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b43fff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b440c2540 0x7f3b44144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:45.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b43fff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3b440acc60 con 0x7f3b440716d0 2024-09-17T05:05:45.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b43fff640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3b440acde0 con 0x7f3b44074660 2024-09-17T05:05:45.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b43fff640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3b440acf60 con 0x7f3b440c2540 2024-09-17T05:05:45.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b42ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b440716d0 0x7f3b441571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:45.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b42ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b440716d0 0x7f3b441571b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43012/0 (socket says 172.21.15.150:43012) 2024-09-17T05:05:45.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b42ffd640 1 -- 172.21.15.150:0/1271482404 learned_addr learned my addr 172.21.15.150:0/1271482404 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:45.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b427fc640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b44074660 0x7f3b440c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:45.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b437fe640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b440c2540 0x7f3b44144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:45.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b42ffd640 1 -- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b440c2540 msgr2=0x7f3b44144b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:45.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b42ffd640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b440c2540 0x7f3b44144b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:45.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b42ffd640 1 -- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b44074660 msgr2=0x7f3b440c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:45.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b42ffd640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b44074660 0x7f3b440c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:45.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b42ffd640 1 -- 172.21.15.150:0/1271482404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b44143a70 con 0x7f3b440716d0 2024-09-17T05:05:45.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b427fc640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b44074660 0x7f3b440c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:05:45.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b42ffd640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b440716d0 0x7f3b441571b0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009790 tx=0x7f3b34004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:45.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b2bfff640 1 -- 172.21.15.150:0/1271482404 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b340bb070 con 0x7f3b440716d0 2024-09-17T05:05:45.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b2bfff640 1 -- 172.21.15.150:0/1271482404 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b34005030 con 0x7f3b440716d0 2024-09-17T05:05:45.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b43fff640 1 -- 172.21.15.150:0/1271482404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b44143790 con 0x7f3b440716d0 2024-09-17T05:05:45.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b2bfff640 1 -- 172.21.15.150:0/1271482404 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b340b2040 con 0x7f3b440716d0 2024-09-17T05:05:45.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b43fff640 1 -- 172.21.15.150:0/1271482404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b44144200 con 0x7f3b440716d0 2024-09-17T05:05:45.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.301+0000 7f3b43fff640 1 -- 172.21.15.150:0/1271482404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b440c4b10 con 0x7f3b440716d0 2024-09-17T05:05:45.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.305+0000 7f3b2bfff640 1 -- 172.21.15.150:0/1271482404 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3b340be070 con 0x7f3b440716d0 2024-09-17T05:05:45.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.305+0000 7f3b2bfff640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3b30073200 0x7f3b30075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:45.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.305+0000 7f3b2bfff640 1 -- 172.21.15.150:0/1271482404 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7f3b3412e860 con 0x7f3b440716d0 2024-09-17T05:05:45.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.305+0000 7f3b427fc640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3b30073200 0x7f3b30075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:45.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.305+0000 7f3b427fc640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3b30073200 0x7f3b30075710 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3b44142020 tx=0x7f3b38005e30 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:45.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.309+0000 7f3b2bfff640 1 -- 172.21.15.150:0/1271482404 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3b340163f0 con 0x7f3b440716d0 2024-09-17T05:05:45.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.553+0000 7f3b43fff640 1 -- 172.21.15.150:0/1271482404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b441425c0 con 0x7f3b440716d0 2024-09-17T05:05:45.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.553+0000 7f3b2bfff640 1 -- 172.21.15.150:0/1271482404 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+6547 (secure 0 0 0) 0x7f3b340f8e40 con 0x7f3b440716d0 2024-09-17T05:05:45.551 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:45.551 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":116,"btime":"2024-09-17T05:05:33:241748+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:45.552 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 116 2024-09-17T05:05:45.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.557+0000 7f3b43fff640 1 -- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3b30073200 msgr2=0x7f3b30075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:45.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.557+0000 7f3b43fff640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3b30073200 0x7f3b30075710 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f3b44142020 tx=0x7f3b38005e30 comp rx=0 tx=0).stop 2024-09-17T05:05:45.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.557+0000 7f3b43fff640 1 -- 172.21.15.150:0/1271482404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b440716d0 msgr2=0x7f3b441571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:45.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.557+0000 7f3b43fff640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b440716d0 0x7f3b441571b0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009790 tx=0x7f3b34004040 comp rx=0 tx=0).stop 2024-09-17T05:05:45.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.557+0000 7f3b43fff640 1 -- 172.21.15.150:0/1271482404 shutdown_connections 2024-09-17T05:05:45.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.557+0000 7f3b43fff640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3b30073200 0x7f3b30075710 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:45.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.557+0000 7f3b43fff640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b44074660 0x7f3b440c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:45.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.557+0000 7f3b43fff640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b440c2540 0x7f3b44144b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:45.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.557+0000 7f3b43fff640 1 --2- 172.21.15.150:0/1271482404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b440716d0 0x7f3b441571b0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:45.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.557+0000 7f3b43fff640 1 -- 172.21.15.150:0/1271482404 >> 172.21.15.150:0/1271482404 conn(0x7f3b44073970 msgr2=0x7f3b44069500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:45.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.557+0000 7f3b43fff640 1 -- 172.21.15.150:0/1271482404 shutdown_connections 2024-09-17T05:05:45.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:45.557+0000 7f3b43fff640 1 -- 172.21.15.150:0/1271482404 wait complete. 2024-09-17T05:05:45.576 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:05:47.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:47.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.753+0000 7f1f76a83640 1 -- 172.21.15.150:0/917081422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f700c4250 msgr2=0x7f1f70153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:47.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.753+0000 7f1f76a83640 1 --2- 172.21.15.150:0/917081422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f700c4250 0x7f1f70153ee0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f1f64009840 tx=0x7f1f6401c820 comp rx=0 tx=0).stop 2024-09-17T05:05:47.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.753+0000 7f1f76a83640 1 -- 172.21.15.150:0/917081422 shutdown_connections 2024-09-17T05:05:47.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.753+0000 7f1f76a83640 1 --2- 172.21.15.150:0/917081422 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f700c3900 0x7f1f70074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:47.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.753+0000 7f1f76a83640 1 --2- 172.21.15.150:0/917081422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f700c4250 0x7f1f70153ee0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:47.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.753+0000 7f1f76a83640 1 -- 172.21.15.150:0/917081422 >> 172.21.15.150:0/917081422 conn(0x7f1f7006e360 msgr2=0x7f1f7006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:47.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.753+0000 7f1f76a83640 1 -- 172.21.15.150:0/917081422 shutdown_connections 2024-09-17T05:05:47.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.753+0000 7f1f76a83640 1 -- 172.21.15.150:0/917081422 wait complete. 2024-09-17T05:05:47.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f76a83640 1 Processor -- start 2024-09-17T05:05:47.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f76a83640 1 -- start start 2024-09-17T05:05:47.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f76a83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f70074e60 0x7f1f70156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:47.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f76a83640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f700c3900 0x7f1f7014eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:47.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f76a83640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f700c4250 0x7f1f701f1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:47.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f76a83640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1f70072f30 con 0x7f1f70074e60 2024-09-17T05:05:47.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f76a83640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1f70072db0 con 0x7f1f700c3900 2024-09-17T05:05:47.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f76a83640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1f70160f90 con 0x7f1f700c4250 2024-09-17T05:05:47.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f7481f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f70074e60 0x7f1f70156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:47.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f7481f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f70074e60 0x7f1f70156420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43032/0 (socket says 172.21.15.150:43032) 2024-09-17T05:05:47.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f7481f640 1 -- 172.21.15.150:0/1751825476 learned_addr learned my addr 172.21.15.150:0/1751825476 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f75020640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f700c4250 0x7f1f701f1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f6ffff640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f700c3900 0x7f1f7014eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f7481f640 1 -- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f700c4250 msgr2=0x7f1f701f1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f7481f640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f700c4250 0x7f1f701f1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f7481f640 1 -- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f700c3900 msgr2=0x7f1f7014eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f7481f640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f700c3900 0x7f1f7014eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f7481f640 1 -- 172.21.15.150:0/1751825476 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f701ee680 con 0x7f1f70074e60 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f6ffff640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f700c3900 0x7f1f7014eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.757+0000 7f1f7481f640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f70074e60 0x7f1f70156420 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f1f64098200 tx=0x7f1f64005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.761+0000 7f1f6d7fa640 1 -- 172.21.15.150:0/1751825476 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64004240 con 0x7f1f70074e60 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.761+0000 7f1f6d7fa640 1 -- 172.21.15.150:0/1751825476 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f640bf070 con 0x7f1f70074e60 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.761+0000 7f1f76a83640 1 -- 172.21.15.150:0/1751825476 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f70157dd0 con 0x7f1f70074e60 2024-09-17T05:05:47.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.761+0000 7f1f6d7fa640 1 -- 172.21.15.150:0/1751825476 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64003800 con 0x7f1f70074e60 2024-09-17T05:05:47.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.761+0000 7f1f76a83640 1 -- 172.21.15.150:0/1751825476 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f701f0e40 con 0x7f1f70074e60 2024-09-17T05:05:47.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.761+0000 7f1f76a83640 1 -- 172.21.15.150:0/1751825476 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f70074920 con 0x7f1f70074e60 2024-09-17T05:05:47.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.761+0000 7f1f6d7fa640 1 -- 172.21.15.150:0/1751825476 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1f640039a0 con 0x7f1f70074e60 2024-09-17T05:05:47.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.761+0000 7f1f6d7fa640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1f54073200 0x7f1f54075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:47.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.761+0000 7f1f6d7fa640 1 -- 172.21.15.150:0/1751825476 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7f1f64130ab0 con 0x7f1f70074e60 2024-09-17T05:05:47.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.761+0000 7f1f6ffff640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1f54073200 0x7f1f54075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:47.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.765+0000 7f1f6ffff640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1f54073200 0x7f1f54075710 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c005fd0 tx=0x7f1f5c005d40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:47.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:47.765+0000 7f1f6d7fa640 1 -- 172.21.15.150:0/1751825476 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1f64016410 con 0x7f1f70074e60 2024-09-17T05:05:48.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.021+0000 7f1f76a83640 1 -- 172.21.15.150:0/1751825476 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f70070890 con 0x7f1f70074e60 2024-09-17T05:05:48.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.021+0000 7f1f6d7fa640 1 -- 172.21.15.150:0/1751825476 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+6547 (secure 0 0 0) 0x7f1f640fb110 con 0x7f1f70074e60 2024-09-17T05:05:48.019 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:48.019 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":116,"btime":"2024-09-17T05:05:33:241748+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:48.020 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 116 2024-09-17T05:05:48.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.025+0000 7f1f76a83640 1 -- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1f54073200 msgr2=0x7f1f54075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:48.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.025+0000 7f1f76a83640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1f54073200 0x7f1f54075710 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c005fd0 tx=0x7f1f5c005d40 comp rx=0 tx=0).stop 2024-09-17T05:05:48.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.025+0000 7f1f76a83640 1 -- 172.21.15.150:0/1751825476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f70074e60 msgr2=0x7f1f70156420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:48.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.025+0000 7f1f76a83640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f70074e60 0x7f1f70156420 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f1f64098200 tx=0x7f1f64005db0 comp rx=0 tx=0).stop 2024-09-17T05:05:48.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.025+0000 7f1f76a83640 1 -- 172.21.15.150:0/1751825476 shutdown_connections 2024-09-17T05:05:48.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.025+0000 7f1f76a83640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1f54073200 0x7f1f54075710 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:48.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.025+0000 7f1f76a83640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f700c3900 0x7f1f7014eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:48.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.025+0000 7f1f76a83640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1f700c4250 0x7f1f701f1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:48.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.025+0000 7f1f76a83640 1 --2- 172.21.15.150:0/1751825476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f70074e60 0x7f1f70156420 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:48.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.025+0000 7f1f76a83640 1 -- 172.21.15.150:0/1751825476 >> 172.21.15.150:0/1751825476 conn(0x7f1f7006e360 msgr2=0x7f1f70152de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:48.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.025+0000 7f1f76a83640 1 -- 172.21.15.150:0/1751825476 shutdown_connections 2024-09-17T05:05:48.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:48.025+0000 7f1f76a83640 1 -- 172.21.15.150:0/1751825476 wait complete. 2024-09-17T05:05:48.044 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:05:50.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:50.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.229+0000 7fa45754d640 1 -- 172.21.15.150:0/3815815018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45006e5d0 msgr2=0x7fa45015dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:50.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.229+0000 7fa45754d640 1 --2- 172.21.15.150:0/3815815018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45006e5d0 0x7fa45015dfa0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa448009c80 tx=0x7fa44801cd10 comp rx=0 tx=0).stop 2024-09-17T05:05:50.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.229+0000 7fa45754d640 1 -- 172.21.15.150:0/3815815018 shutdown_connections 2024-09-17T05:05:50.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.229+0000 7fa45754d640 1 --2- 172.21.15.150:0/3815815018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45006e5d0 0x7fa45015dfa0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:50.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.229+0000 7fa45754d640 1 --2- 172.21.15.150:0/3815815018 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa45006fca0 0x7fa45006e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:50.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.229+0000 7fa45754d640 1 -- 172.21.15.150:0/3815815018 >> 172.21.15.150:0/3815815018 conn(0x7fa4500c5ae0 msgr2=0x7fa450073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:50.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.229+0000 7fa45754d640 1 -- 172.21.15.150:0/3815815018 shutdown_connections 2024-09-17T05:05:50.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.229+0000 7fa45754d640 1 -- 172.21.15.150:0/3815815018 wait complete. 2024-09-17T05:05:50.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.229+0000 7fa45754d640 1 Processor -- start 2024-09-17T05:05:50.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.229+0000 7fa45754d640 1 -- start start 2024-09-17T05:05:50.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa45754d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45006e5d0 0x7fa4500ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:50.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa45754d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa45006fca0 0x7fa45014be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:50.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa45754d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4500745c0 0x7fa4500c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:50.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa45754d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa450070610 con 0x7fa45006e5d0 2024-09-17T05:05:50.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa45754d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa450070790 con 0x7fa4500745c0 2024-09-17T05:05:50.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa45754d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa4500ab140 con 0x7fa45006fca0 2024-09-17T05:05:50.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa4552e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45006e5d0 0x7fa4500ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:50.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa4552e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45006e5d0 0x7fa4500ab5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35190/0 (socket says 172.21.15.150:35190) 2024-09-17T05:05:50.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa4552e9640 1 -- 172.21.15.150:0/145371629 learned_addr learned my addr 172.21.15.150:0/145371629 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:50.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa454ae8640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa45006fca0 0x7fa45014be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:50.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa455aea640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4500745c0 0x7fa4500c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:50.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa4552e9640 1 -- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa45006fca0 msgr2=0x7fa45014be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:50.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa4552e9640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa45006fca0 0x7fa45014be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:50.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa4552e9640 1 -- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4500745c0 msgr2=0x7fa4500c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:50.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa4552e9640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4500745c0 0x7fa4500c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:50.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa4552e9640 1 -- 172.21.15.150:0/145371629 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa450150570 con 0x7fa45006e5d0 2024-09-17T05:05:50.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa455aea640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4500745c0 0x7fa4500c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:05:50.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa4552e9640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45006e5d0 0x7fa4500ab5c0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fa44c00c3c0 tx=0x7fa44c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:50.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa445ffb640 1 -- 172.21.15.150:0/145371629 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c017070 con 0x7fa45006e5d0 2024-09-17T05:05:50.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa445ffb640 1 -- 172.21.15.150:0/145371629 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa44c0053c0 con 0x7fa45006e5d0 2024-09-17T05:05:50.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa45754d640 1 -- 172.21.15.150:0/145371629 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa450069d10 con 0x7fa45006e5d0 2024-09-17T05:05:50.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa445ffb640 1 -- 172.21.15.150:0/145371629 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c003860 con 0x7fa45006e5d0 2024-09-17T05:05:50.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa45754d640 1 -- 172.21.15.150:0/145371629 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa450142240 con 0x7fa45006e5d0 2024-09-17T05:05:50.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.233+0000 7fa45754d640 1 -- 172.21.15.150:0/145371629 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa45006e090 con 0x7fa45006e5d0 2024-09-17T05:05:50.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.237+0000 7fa445ffb640 1 -- 172.21.15.150:0/145371629 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa44c01a070 con 0x7fa45006e5d0 2024-09-17T05:05:50.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.237+0000 7fa445ffb640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa434073200 0x7fa434075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:50.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.237+0000 7fa445ffb640 1 -- 172.21.15.150:0/145371629 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7fa44c093f90 con 0x7fa45006e5d0 2024-09-17T05:05:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.237+0000 7fa454ae8640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa434073200 0x7fa434075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.237+0000 7fa454ae8640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa434073200 0x7fa434075710 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa4501493d0 tx=0x7fa44000b2e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.241+0000 7fa445ffb640 1 -- 172.21.15.150:0/145371629 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa44c05de90 con 0x7fa45006e5d0 2024-09-17T05:05:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.465+0000 7fa45754d640 1 -- 172.21.15.150:0/145371629 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa45015dfd0 con 0x7fa45006e5d0 2024-09-17T05:05:50.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.465+0000 7fa445ffb640 1 -- 172.21.15.150:0/145371629 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+6547 (secure 0 0 0) 0x7fa44c0628c0 con 0x7fa45006e5d0 2024-09-17T05:05:50.463 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:50.463 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":116,"btime":"2024-09-17T05:05:33:241748+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:50.463 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 116 2024-09-17T05:05:50.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.469+0000 7fa45754d640 1 -- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa434073200 msgr2=0x7fa434075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:50.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.469+0000 7fa45754d640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa434073200 0x7fa434075710 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa4501493d0 tx=0x7fa44000b2e0 comp rx=0 tx=0).stop 2024-09-17T05:05:50.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.469+0000 7fa45754d640 1 -- 172.21.15.150:0/145371629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45006e5d0 msgr2=0x7fa4500ab5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:50.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.469+0000 7fa45754d640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45006e5d0 0x7fa4500ab5c0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fa44c00c3c0 tx=0x7fa44c00c880 comp rx=0 tx=0).stop 2024-09-17T05:05:50.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.469+0000 7fa45754d640 1 -- 172.21.15.150:0/145371629 shutdown_connections 2024-09-17T05:05:50.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.469+0000 7fa45754d640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa434073200 0x7fa434075710 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:50.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.469+0000 7fa45754d640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4500745c0 0x7fa4500c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:50.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.469+0000 7fa45754d640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa45006fca0 0x7fa45014be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:50.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.469+0000 7fa45754d640 1 --2- 172.21.15.150:0/145371629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45006e5d0 0x7fa4500ab5c0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:50.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.469+0000 7fa45754d640 1 -- 172.21.15.150:0/145371629 >> 172.21.15.150:0/145371629 conn(0x7fa4500c5ae0 msgr2=0x7fa45014fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:50.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.469+0000 7fa45754d640 1 -- 172.21.15.150:0/145371629 shutdown_connections 2024-09-17T05:05:50.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:50.469+0000 7fa45754d640 1 -- 172.21.15.150:0/145371629 wait complete. 2024-09-17T05:05:50.485 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:05:52.486 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.677+0000 7f1349aca640 1 -- 172.21.15.150:0/3733501683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1344071670 msgr2=0x7f13440c5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.677+0000 7f1349aca640 1 --2- 172.21.15.150:0/3733501683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1344071670 0x7f13440c5680 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f1334009900 tx=0x7f133401c8e0 comp rx=0 tx=0).stop 2024-09-17T05:05:52.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.677+0000 7f1349aca640 1 -- 172.21.15.150:0/3733501683 shutdown_connections 2024-09-17T05:05:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.677+0000 7f1349aca640 1 --2- 172.21.15.150:0/3733501683 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f134414e490 0x7f13440c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.677+0000 7f1349aca640 1 --2- 172.21.15.150:0/3733501683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1344071670 0x7f13440c5680 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.677+0000 7f1349aca640 1 -- 172.21.15.150:0/3733501683 >> 172.21.15.150:0/3733501683 conn(0x7f13440ca460 msgr2=0x7f13440c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.677+0000 7f1349aca640 1 -- 172.21.15.150:0/3733501683 shutdown_connections 2024-09-17T05:05:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.677+0000 7f1349aca640 1 -- 172.21.15.150:0/3733501683 wait complete. 2024-09-17T05:05:52.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.677+0000 7f1349aca640 1 Processor -- start 2024-09-17T05:05:52.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.677+0000 7f1349aca640 1 -- start start 2024-09-17T05:05:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1349aca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1344071670 0x7f1344153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1349aca640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f134414e490 0x7f134414c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1349aca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1344157c00 0x7f13441f1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1349aca640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f13440c6740 con 0x7f1344157c00 2024-09-17T05:05:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1349aca640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f13440c65c0 con 0x7f134414e490 2024-09-17T05:05:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1349aca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f134414e8a0 con 0x7f1344071670 2024-09-17T05:05:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1343fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1344157c00 0x7f13441f1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1343fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1344157c00 0x7f13441f1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35206/0 (socket says 172.21.15.150:35206) 2024-09-17T05:05:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1343fff640 1 -- 172.21.15.150:0/2831576 learned_addr learned my addr 172.21.15.150:0/2831576 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f13437fe640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1344071670 0x7f1344153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:52.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1342ffd640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f134414e490 0x7f134414c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:52.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1343fff640 1 -- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1344071670 msgr2=0x7f1344153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:52.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1343fff640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1344071670 0x7f1344153f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:52.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1343fff640 1 -- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f134414e490 msgr2=0x7f134414c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:52.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1343fff640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f134414e490 0x7f134414c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:52.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1343fff640 1 -- 172.21.15.150:0/2831576 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1344154850 con 0x7f1344157c00 2024-09-17T05:05:52.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f13437fe640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1344071670 0x7f1344153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:05:52.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1343fff640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1344157c00 0x7f13441f1530 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f133c00be90 tx=0x7f133c00bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:52.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f132bfff640 1 -- 172.21.15.150:0/2831576 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c019070 con 0x7f1344157c00 2024-09-17T05:05:52.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f132bfff640 1 -- 172.21.15.150:0/2831576 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f133c014440 con 0x7f1344157c00 2024-09-17T05:05:52.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1349aca640 1 -- 172.21.15.150:0/2831576 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134414ca20 con 0x7f1344157c00 2024-09-17T05:05:52.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f132bfff640 1 -- 172.21.15.150:0/2831576 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0148d0 con 0x7f1344157c00 2024-09-17T05:05:52.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1349aca640 1 -- 172.21.15.150:0/2831576 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13441f1f90 con 0x7f1344157c00 2024-09-17T05:05:52.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f1349aca640 1 -- 172.21.15.150:0/2831576 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13440c5680 con 0x7f1344157c00 2024-09-17T05:05:52.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f132bfff640 1 -- 172.21.15.150:0/2831576 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f133c01c070 con 0x7f1344157c00 2024-09-17T05:05:52.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.681+0000 7f132bfff640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1330073200 0x7f1330075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:52.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.685+0000 7f132bfff640 1 -- 172.21.15.150:0/2831576 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7f133c094c60 con 0x7f1344157c00 2024-09-17T05:05:52.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.685+0000 7f13437fe640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1330073200 0x7f1330075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:52.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.685+0000 7f13437fe640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1330073200 0x7f1330075710 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f13340af000 tx=0x7f133401b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:52.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.689+0000 7f132bfff640 1 -- 172.21.15.150:0/2831576 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f133c05f240 con 0x7f1344157c00 2024-09-17T05:05:52.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.917+0000 7f1349aca640 1 -- 172.21.15.150:0/2831576 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13441f1150 con 0x7f1344157c00 2024-09-17T05:05:52.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.917+0000 7f132bfff640 1 -- 172.21.15.150:0/2831576 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+6547 (secure 0 0 0) 0x7f133c05eb40 con 0x7f1344157c00 2024-09-17T05:05:52.917 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:52.917 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":116,"btime":"2024-09-17T05:05:33:241748+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:52.917 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 116 2024-09-17T05:05:52.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.921+0000 7f1349aca640 1 -- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1330073200 msgr2=0x7f1330075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:52.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.921+0000 7f1349aca640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1330073200 0x7f1330075710 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f13340af000 tx=0x7f133401b120 comp rx=0 tx=0).stop 2024-09-17T05:05:52.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.921+0000 7f1349aca640 1 -- 172.21.15.150:0/2831576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1344157c00 msgr2=0x7f13441f1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:52.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.921+0000 7f1349aca640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1344157c00 0x7f13441f1530 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f133c00be90 tx=0x7f133c00bf80 comp rx=0 tx=0).stop 2024-09-17T05:05:52.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.921+0000 7f1349aca640 1 -- 172.21.15.150:0/2831576 shutdown_connections 2024-09-17T05:05:52.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.921+0000 7f1349aca640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1330073200 0x7f1330075710 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:52.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.921+0000 7f1349aca640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f134414e490 0x7f134414c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:52.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.921+0000 7f1349aca640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1344157c00 0x7f13441f1530 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:52.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.921+0000 7f1349aca640 1 --2- 172.21.15.150:0/2831576 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1344071670 0x7f1344153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:52.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.921+0000 7f1349aca640 1 -- 172.21.15.150:0/2831576 >> 172.21.15.150:0/2831576 conn(0x7f13440ca460 msgr2=0x7f1344148870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:52.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.921+0000 7f1349aca640 1 -- 172.21.15.150:0/2831576 shutdown_connections 2024-09-17T05:05:52.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:52.921+0000 7f1349aca640 1 -- 172.21.15.150:0/2831576 wait complete. 2024-09-17T05:05:52.939 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:05:54.941 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.121+0000 7f42c146e640 1 -- 172.21.15.150:0/1985956872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0c77b0 msgr2=0x7f42bc151980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.121+0000 7f42c146e640 1 --2- 172.21.15.150:0/1985956872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0c77b0 0x7f42bc151980 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009bc0 tx=0x7f42b401cb10 comp rx=0 tx=0).stop 2024-09-17T05:05:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.121+0000 7f42c146e640 1 -- 172.21.15.150:0/1985956872 shutdown_connections 2024-09-17T05:05:55.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.121+0000 7f42c146e640 1 --2- 172.21.15.150:0/1985956872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0c77b0 0x7f42bc151980 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:55.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.121+0000 7f42c146e640 1 --2- 172.21.15.150:0/1985956872 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc0a85b0 0x7f42bc0c7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:55.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.121+0000 7f42c146e640 1 -- 172.21.15.150:0/1985956872 >> 172.21.15.150:0/1985956872 conn(0x7f42bc0aab60 msgr2=0x7f42bc0aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:55.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.121+0000 7f42c146e640 1 -- 172.21.15.150:0/1985956872 shutdown_connections 2024-09-17T05:05:55.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.121+0000 7f42c146e640 1 -- 172.21.15.150:0/1985956872 wait complete. 2024-09-17T05:05:55.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.121+0000 7f42c146e640 1 Processor -- start 2024-09-17T05:05:55.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.121+0000 7f42c146e640 1 -- start start 2024-09-17T05:05:55.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42c146e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0a85b0 0x7f42bc150100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:55.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42c146e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc0c6cb0 0x7f42bc0c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:55.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42c146e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42bc0c77b0 0x7f42bc1eff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:55.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42c146e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc0c3460 con 0x7f42bc0c77b0 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42c146e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f42bc0c32e0 con 0x7f42bc0a85b0 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42c146e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f42bc161890 con 0x7f42bc0c6cb0 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42bb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42bc0c77b0 0x7f42bc1eff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42bb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42bc0c77b0 0x7f42bc1eff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35214/0 (socket says 172.21.15.150:35214) 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42bb7fe640 1 -- 172.21.15.150:0/1715668781 learned_addr learned my addr 172.21.15.150:0/1715668781 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42ba7fc640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc0c6cb0 0x7f42bc0c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42baffd640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0a85b0 0x7f42bc150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42bb7fe640 1 -- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc0c6cb0 msgr2=0x7f42bc0c23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42bb7fe640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc0c6cb0 0x7f42bc0c23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42bb7fe640 1 -- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0a85b0 msgr2=0x7f42bc150100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42bb7fe640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0a85b0 0x7f42bc150100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42bb7fe640 1 -- 172.21.15.150:0/1715668781 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42bc1ef730 con 0x7f42bc0c77b0 2024-09-17T05:05:55.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42bb7fe640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42bc0c77b0 0x7f42bc1eff10 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009b10 tx=0x7f42b4003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:55.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42a37fe640 1 -- 172.21.15.150:0/1715668781 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b40bb070 con 0x7f42bc0c77b0 2024-09-17T05:05:55.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42a37fe640 1 -- 172.21.15.150:0/1715668781 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42b4004d40 con 0x7f42bc0c77b0 2024-09-17T05:05:55.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42c146e640 1 -- 172.21.15.150:0/1715668781 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42bc14d6c0 con 0x7f42bc0c77b0 2024-09-17T05:05:55.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42a37fe640 1 -- 172.21.15.150:0/1715668781 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b40b2040 con 0x7f42bc0c77b0 2024-09-17T05:05:55.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42c146e640 1 -- 172.21.15.150:0/1715668781 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42bc14dc80 con 0x7f42bc0c77b0 2024-09-17T05:05:55.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42a17fa640 1 -- 172.21.15.150:0/1715668781 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42940052a0 con 0x7f42bc0c77b0 2024-09-17T05:05:55.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42a37fe640 1 -- 172.21.15.150:0/1715668781 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f42b40be070 con 0x7f42bc0c77b0 2024-09-17T05:05:55.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.125+0000 7f42a37fe640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f42a8073200 0x7f42a8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:55.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.129+0000 7f42a37fe640 1 -- 172.21.15.150:0/1715668781 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7f42b412e940 con 0x7f42bc0c77b0 2024-09-17T05:05:55.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.129+0000 7f42baffd640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f42a8073200 0x7f42a8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:55.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.129+0000 7f42baffd640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f42a8073200 0x7f42a8075710 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f42ac001640 tx=0x7f42ac009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:55.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.133+0000 7f42a37fe640 1 -- 172.21.15.150:0/1715668781 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f42b4016790 con 0x7f42bc0c77b0 2024-09-17T05:05:55.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 -- 172.21.15.150:0/1715668781 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4294005590 con 0x7f42bc0c77b0 2024-09-17T05:05:55.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a37fe640 1 -- 172.21.15.150:0/1715668781 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+6547 (secure 0 0 0) 0x7f42b40f8f20 con 0x7f42bc0c77b0 2024-09-17T05:05:55.394 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:55.394 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":116,"btime":"2024-09-17T05:05:33:241748+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6747,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:04:09.506271+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3807957687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3807957687},{"type":"v1","addr":"172.21.15.150:6835","nonce":3807957687}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":2}]} 2024-09-17T05:05:55.394 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 116 2024-09-17T05:05:55.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 -- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f42a8073200 msgr2=0x7f42a8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:55.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f42a8073200 0x7f42a8075710 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f42ac001640 tx=0x7f42ac009040 comp rx=0 tx=0).stop 2024-09-17T05:05:55.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 -- 172.21.15.150:0/1715668781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42bc0c77b0 msgr2=0x7f42bc1eff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:55.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42bc0c77b0 0x7f42bc1eff10 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009b10 tx=0x7f42b4003f60 comp rx=0 tx=0).stop 2024-09-17T05:05:55.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 -- 172.21.15.150:0/1715668781 shutdown_connections 2024-09-17T05:05:55.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f42a8073200 0x7f42a8075710 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:55.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42bc0c77b0 0x7f42bc1eff10 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:55.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42bc0c6cb0 0x7f42bc0c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:55.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 --2- 172.21.15.150:0/1715668781 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42bc0a85b0 0x7f42bc150100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:55.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 -- 172.21.15.150:0/1715668781 >> 172.21.15.150:0/1715668781 conn(0x7f42bc0aab60 msgr2=0x7f42bc0c4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:55.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 -- 172.21.15.150:0/1715668781 shutdown_connections 2024-09-17T05:05:55.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:55.397+0000 7f42a17fa640 1 -- 172.21.15.150:0/1715668781 wait complete. 2024-09-17T05:05:55.417 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:05:57.418 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:57.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.577+0000 7f8bfcc73640 1 -- 172.21.15.150:0/132735687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be40040c0 msgr2=0x7f8be4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:57.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.577+0000 7f8bfcc73640 1 --2- 172.21.15.150:0/132735687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be40040c0 0x7f8be4004810 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f8be00016c0 tx=0x7f8be001c790 comp rx=0 tx=0).stop 2024-09-17T05:05:57.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.577+0000 7f8bf6d9d640 1 -- 172.21.15.150:0/132735687 reap_dead start 2024-09-17T05:05:57.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.577+0000 7f8bfcc73640 1 -- 172.21.15.150:0/132735687 shutdown_connections 2024-09-17T05:05:57.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.577+0000 7f8bfcc73640 1 -- 172.21.15.150:0/132735687 >> 172.21.15.150:0/132735687 conn(0x7f8bf8074750 msgr2=0x7f8bf8074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:57.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.577+0000 7f8bfcc73640 1 -- 172.21.15.150:0/132735687 shutdown_connections 2024-09-17T05:05:57.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.577+0000 7f8bfcc73640 1 -- 172.21.15.150:0/132735687 wait complete. 2024-09-17T05:05:57.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bfcc73640 1 Processor -- start 2024-09-17T05:05:57.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bfcc73640 1 -- start start 2024-09-17T05:05:57.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bfcc73640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be40040c0 0x7f8bf81e1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:57.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bfcc73640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8bf8146d80 0x7f8bf8146590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:57.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bfcc73640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8bf80cad40 0x7f8bf80cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:57.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bfcc73640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf8156b30 con 0x7f8be40040c0 2024-09-17T05:05:57.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bfcc73640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf80c84c0 con 0x7f8bf8146d80 2024-09-17T05:05:57.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bfcc73640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8bf81567e0 con 0x7f8bf80cad40 2024-09-17T05:05:57.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bf659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be40040c0 0x7f8bf81e1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:57.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bf659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be40040c0 0x7f8bf81e1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35232/0 (socket says 172.21.15.150:35232) 2024-09-17T05:05:57.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bf659c640 1 -- 172.21.15.150:0/2214492532 learned_addr learned my addr 172.21.15.150:0/2214492532 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:57.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bf659c640 1 -- 172.21.15.150:0/2214492532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8bf80cad40 msgr2=0x7f8bf80cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:57.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bf659c640 1 --2- 172.21.15.150:0/2214492532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8bf80cad40 0x7f8bf80cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:57.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bf659c640 1 -- 172.21.15.150:0/2214492532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8bf8146d80 msgr2=0x7f8bf8146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:57.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bf659c640 1 --2- 172.21.15.150:0/2214492532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8bf8146d80 0x7f8bf8146590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:57.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bf659c640 1 -- 172.21.15.150:0/2214492532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf81543d0 con 0x7f8be40040c0 2024-09-17T05:05:57.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bf659c640 1 --2- 172.21.15.150:0/2214492532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be40040c0 0x7f8bf81e1330 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f8be8011990 tx=0x7f8be8011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:57.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8beeffd640 1 -- 172.21.15.150:0/2214492532 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be8017070 con 0x7f8be40040c0 2024-09-17T05:05:57.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8beeffd640 1 -- 172.21.15.150:0/2214492532 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8be800fb60 con 0x7f8be40040c0 2024-09-17T05:05:57.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bfcc73640 1 -- 172.21.15.150:0/2214492532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf81f0c10 con 0x7f8be40040c0 2024-09-17T05:05:57.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8beeffd640 1 -- 172.21.15.150:0/2214492532 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be8013450 con 0x7f8be40040c0 2024-09-17T05:05:57.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.581+0000 7f8bfcc73640 1 -- 172.21.15.150:0/2214492532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf81e0050 con 0x7f8be40040c0 2024-09-17T05:05:57.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.585+0000 7f8bfcc73640 1 -- 172.21.15.150:0/2214492532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bf8148180 con 0x7f8be40040c0 2024-09-17T05:05:57.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.585+0000 7f8beeffd640 1 -- 172.21.15.150:0/2214492532 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8be80135f0 con 0x7f8be40040c0 2024-09-17T05:05:57.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.585+0000 7f8beeffd640 1 --2- 172.21.15.150:0/2214492532 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8bd8073200 0x7f8bd8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:57.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.585+0000 7f8beeffd640 1 -- 172.21.15.150:0/2214492532 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f8be8098d30 con 0x7f8be40040c0 2024-09-17T05:05:57.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.585+0000 7f8beffff640 1 --2- 172.21.15.150:0/2214492532 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8bd8073200 0x7f8bd8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:57.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.585+0000 7f8beffff640 1 --2- 172.21.15.150:0/2214492532 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8bd8073200 0x7f8bd8075710 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8bf808e1c0 tx=0x7f8be4005840 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:57.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.589+0000 7f8beeffd640 1 -- 172.21.15.150:0/2214492532 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8bf80b20f0 con 0x7f8be40040c0 2024-09-17T05:05:57.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.841+0000 7f8bfcc73640 1 -- 172.21.15.150:0/2214492532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bf80c1300 con 0x7f8be40040c0 2024-09-17T05:05:57.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.841+0000 7f8beeffd640 1 -- 172.21.15.150:0/2214492532 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+5713 (secure 0 0 0) 0x7f8be80632c0 con 0x7f8be40040c0 2024-09-17T05:05:57.841 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:57.841 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":118,"btime":"2024-09-17T05:05:57:061183+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:05:56.521499+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6747":{"gid":6747,"name":"a","rank":0,"incarnation":117,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:05:57.841 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 118 2024-09-17T05:05:57.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.845+0000 7f8bfcc73640 1 -- 172.21.15.150:0/2214492532 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8bd8073200 msgr2=0x7f8bd8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:57.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.845+0000 7f8bfcc73640 1 --2- 172.21.15.150:0/2214492532 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8bd8073200 0x7f8bd8075710 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8bf808e1c0 tx=0x7f8be4005840 comp rx=0 tx=0).stop 2024-09-17T05:05:57.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.845+0000 7f8bfcc73640 1 -- 172.21.15.150:0/2214492532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be40040c0 msgr2=0x7f8bf81e1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:57.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.845+0000 7f8bfcc73640 1 --2- 172.21.15.150:0/2214492532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be40040c0 0x7f8bf81e1330 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f8be8011990 tx=0x7f8be8011e50 comp rx=0 tx=0).stop 2024-09-17T05:05:57.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.845+0000 7f8bfcc73640 1 -- 172.21.15.150:0/2214492532 shutdown_connections 2024-09-17T05:05:57.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.845+0000 7f8bfcc73640 1 --2- 172.21.15.150:0/2214492532 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8bd8073200 0x7f8bd8075710 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:57.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.845+0000 7f8bfcc73640 1 --2- 172.21.15.150:0/2214492532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8bf8146d80 0x7f8bf8146590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:57.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.845+0000 7f8bfcc73640 1 --2- 172.21.15.150:0/2214492532 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8bf80cad40 0x7f8bf80cb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:57.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.845+0000 7f8bfcc73640 1 --2- 172.21.15.150:0/2214492532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be40040c0 0x7f8bf81e1330 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:57.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.845+0000 7f8bfcc73640 1 -- 172.21.15.150:0/2214492532 >> 172.21.15.150:0/2214492532 conn(0x7f8bf8074750 msgr2=0x7f8bf80c8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:57.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.845+0000 7f8bfcc73640 1 -- 172.21.15.150:0/2214492532 shutdown_connections 2024-09-17T05:05:57.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:57.845+0000 7f8bfcc73640 1 -- 172.21.15.150:0/2214492532 wait complete. 2024-09-17T05:05:57.863 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-17T05:05:57.863 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:05:57.863 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:05:58.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.025+0000 7f522f4ff640 1 -- 172.21.15.150:0/1103684114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5214004810 msgr2=0x7f5214004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:58.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.025+0000 7f522f4ff640 1 --2- 172.21.15.150:0/1103684114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5214004810 0x7f5214004c20 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f521800bb10 tx=0x7f521801cda0 comp rx=0 tx=0).stop 2024-09-17T05:05:58.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.025+0000 7f522f4ff640 1 -- 172.21.15.150:0/1103684114 shutdown_connections 2024-09-17T05:05:58.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.025+0000 7f522f4ff640 1 --2- 172.21.15.150:0/1103684114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5214004810 0x7f5214004c20 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:58.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.025+0000 7f522f4ff640 1 -- 172.21.15.150:0/1103684114 >> 172.21.15.150:0/1103684114 conn(0x7f52280ca460 msgr2=0x7f52280c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:58.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.025+0000 7f522f4ff640 1 -- 172.21.15.150:0/1103684114 shutdown_connections 2024-09-17T05:05:58.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.025+0000 7f522f4ff640 1 -- 172.21.15.150:0/1103684114 wait complete. 2024-09-17T05:05:58.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.025+0000 7f522f4ff640 1 Processor -- start 2024-09-17T05:05:58.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.025+0000 7f522f4ff640 1 -- start start 2024-09-17T05:05:58.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522f4ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5214004810 0x7f52281ee740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:58.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522f4ff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52280c2f30 0x7f52281edee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:58.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522f4ff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52280aa1b0 0x7f52280aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:58.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522f4ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f52280c62d0 con 0x7f5214004810 2024-09-17T05:05:58.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522f4ff640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f52280c51f0 con 0x7f52280aa1b0 2024-09-17T05:05:58.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522f4ff640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f52280c65c0 con 0x7f52280c2f30 2024-09-17T05:05:58.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522d29b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5214004810 0x7f52281ee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522d29b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5214004810 0x7f52281ee740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35242/0 (socket says 172.21.15.150:35242) 2024-09-17T05:05:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522d29b640 1 -- 172.21.15.150:0/57719751 learned_addr learned my addr 172.21.15.150:0/57719751 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:05:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522ca9a640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52280c2f30 0x7f52281edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522da9c640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52280aa1b0 0x7f52280aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522d29b640 1 -- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52280c2f30 msgr2=0x7f52281edee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522d29b640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52280c2f30 0x7f52281edee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522d29b640 1 -- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52280aa1b0 msgr2=0x7f52280aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522d29b640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52280aa1b0 0x7f52280aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522d29b640 1 -- 172.21.15.150:0/57719751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5228147d40 con 0x7f5214004810 2024-09-17T05:05:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522d29b640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5214004810 0x7f52281ee740 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5224000fc0 tx=0x7f522400e6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f521dffb640 1 -- 172.21.15.150:0/57719751 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5224012850 con 0x7f5214004810 2024-09-17T05:05:58.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f521dffb640 1 -- 172.21.15.150:0/57719751 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5224012ae0 con 0x7f5214004810 2024-09-17T05:05:58.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522f4ff640 1 -- 172.21.15.150:0/57719751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5228147fb0 con 0x7f5214004810 2024-09-17T05:05:58.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f521dffb640 1 -- 172.21.15.150:0/57719751 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522401a400 con 0x7f5214004810 2024-09-17T05:05:58.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522f4ff640 1 -- 172.21.15.150:0/57719751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52281ee420 con 0x7f5214004810 2024-09-17T05:05:58.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f522f4ff640 1 -- 172.21.15.150:0/57719751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5228072260 con 0x7f5214004810 2024-09-17T05:05:58.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.029+0000 7f521dffb640 1 -- 172.21.15.150:0/57719751 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f522400d030 con 0x7f5214004810 2024-09-17T05:05:58.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.033+0000 7f521dffb640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f520c073200 0x7f520c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:05:58.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.033+0000 7f521dffb640 1 -- 172.21.15.150:0/57719751 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f5224097f00 con 0x7f5214004810 2024-09-17T05:05:58.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.033+0000 7f522ca9a640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f520c073200 0x7f520c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:05:58.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.033+0000 7f522ca9a640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f520c073200 0x7f520c075710 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f52180b0000 tx=0x7f5218019b40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:05:58.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.037+0000 7f521dffb640 1 -- 172.21.15.150:0/57719751 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5224016030 con 0x7f5214004810 2024-09-17T05:05:58.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.249+0000 7f522f4ff640 1 -- 172.21.15.150:0/57719751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52281579c0 con 0x7f5214004810 2024-09-17T05:05:58.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f521dffb640 1 -- 172.21.15.150:0/57719751 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+5710 (secure 0 0 0) 0x7f5224062490 con 0x7f5214004810 2024-09-17T05:05:58.249 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:05:58.249 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":119,"btime":"2024-09-17T05:05:58:063308+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:05:57.067445+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6747":{"gid":6747,"name":"a","rank":0,"incarnation":117,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:05:58.250 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 119 2024-09-17T05:05:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f522f4ff640 1 -- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f520c073200 msgr2=0x7f520c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f522f4ff640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f520c073200 0x7f520c075710 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f52180b0000 tx=0x7f5218019b40 comp rx=0 tx=0).stop 2024-09-17T05:05:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f522f4ff640 1 -- 172.21.15.150:0/57719751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5214004810 msgr2=0x7f52281ee740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:05:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f522f4ff640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5214004810 0x7f52281ee740 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f5224000fc0 tx=0x7f522400e6b0 comp rx=0 tx=0).stop 2024-09-17T05:05:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f522f4ff640 1 -- 172.21.15.150:0/57719751 shutdown_connections 2024-09-17T05:05:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f522f4ff640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f520c073200 0x7f520c075710 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f522f4ff640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52280aa1b0 0x7f52280aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f522f4ff640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52280c2f30 0x7f52281edee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f522f4ff640 1 --2- 172.21.15.150:0/57719751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5214004810 0x7f52281ee740 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:05:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f522f4ff640 1 -- 172.21.15.150:0/57719751 >> 172.21.15.150:0/57719751 conn(0x7f52280ca460 msgr2=0x7f522814d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:05:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f522f4ff640 1 -- 172.21.15.150:0/57719751 shutdown_connections 2024-09-17T05:05:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:05:58.253+0000 7f522f4ff640 1 -- 172.21.15.150:0/57719751 wait complete. 2024-09-17T05:05:58.278 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:05:58:063308+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 119, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 115, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6229": { "addr": "172.21.15.176:6837/3940723333", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 3940723333, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 3940723333, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6229, "incarnation": 112, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:05:24.217124+0000", "qdb_cluster": [ 6229 ], "qdb_leader": 6229, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6229 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 119, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6747": { "addr": "172.21.15.150:6833/1237450980", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 1237450980, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 1237450980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6747, "incarnation": 117, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:05:57.067445+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6747 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:06:00.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:00.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.453+0000 7f2afcb63640 1 -- 172.21.15.150:0/3026660337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c3fa0 msgr2=0x7f2af81547f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.457+0000 7f2afcb63640 1 --2- 172.21.15.150:0/3026660337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c3fa0 0x7f2af81547f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2af80c4b50 tx=0x7f2aec01c770 comp rx=0 tx=0).stop 2024-09-17T05:06:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.457+0000 7f2afcb63640 1 -- 172.21.15.150:0/3026660337 shutdown_connections 2024-09-17T05:06:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.457+0000 7f2afcb63640 1 --2- 172.21.15.150:0/3026660337 >> v2:172.21.15.176:3300/0 conn(0x7f2af80c43b0 0x7f2af80743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.457+0000 7f2afcb63640 1 --2- 172.21.15.150:0/3026660337 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c3fa0 0x7f2af81547f0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.457+0000 7f2afcb63640 1 -- 172.21.15.150:0/3026660337 >> 172.21.15.150:0/3026660337 conn(0x7f2af80c5ae0 msgr2=0x7f2af8073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.457+0000 7f2afcb63640 1 -- 172.21.15.150:0/3026660337 shutdown_connections 2024-09-17T05:06:00.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.457+0000 7f2afcb63640 1 -- 172.21.15.150:0/3026660337 wait complete. 2024-09-17T05:06:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.457+0000 7f2afcb63640 1 Processor -- start 2024-09-17T05:06:00.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.457+0000 7f2afcb63640 1 -- start start 2024-09-17T05:06:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2afcb63640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c38c0 0x7f2af813f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2afcb63640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2af80c3fa0 0x7f2af81442d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2afcb63640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2af80c43b0 0x7f2af80c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2afcb63640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2af80c4d50 con 0x7f2af80c38c0 2024-09-17T05:06:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2afcb63640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2af8154f40 con 0x7f2af80c43b0 2024-09-17T05:06:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2afcb63640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2af80c4ed0 con 0x7f2af80c3fa0 2024-09-17T05:06:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2af659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c38c0 0x7f2af813f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2af659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c38c0 0x7f2af813f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54838/0 (socket says 172.21.15.150:54838) 2024-09-17T05:06:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2af659c640 1 -- 172.21.15.150:0/1770762613 learned_addr learned my addr 172.21.15.150:0/1770762613 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2af5d9b640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2af80c3fa0 0x7f2af81442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2af6d9d640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2af80c43b0 0x7f2af80c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2af659c640 1 -- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2af80c3fa0 msgr2=0x7f2af81442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2af659c640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2af80c3fa0 0x7f2af81442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2af659c640 1 -- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2af80c43b0 msgr2=0x7f2af80c1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2af659c640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2af80c43b0 0x7f2af80c1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2af659c640 1 -- 172.21.15.150:0/1770762613 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af81f3f50 con 0x7f2af80c38c0 2024-09-17T05:06:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2af6d9d640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2af80c43b0 0x7f2af80c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:06:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2af659c640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c38c0 0x7f2af813f570 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800c3f0 tx=0x7f2ae800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2ae6ffd640 1 -- 172.21.15.150:0/1770762613 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8011790 con 0x7f2af80c38c0 2024-09-17T05:06:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2ae6ffd640 1 -- 172.21.15.150:0/1770762613 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ae8017070 con 0x7f2af80c38c0 2024-09-17T05:06:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2ae6ffd640 1 -- 172.21.15.150:0/1770762613 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8011ce0 con 0x7f2af80c38c0 2024-09-17T05:06:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2afcb63640 1 -- 172.21.15.150:0/1770762613 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2af813ff20 con 0x7f2af80c38c0 2024-09-17T05:06:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2afcb63640 1 -- 172.21.15.150:0/1770762613 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2af8145610 con 0x7f2af80c38c0 2024-09-17T05:06:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.461+0000 7f2afcb63640 1 -- 172.21.15.150:0/1770762613 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2af81547f0 con 0x7f2af80c38c0 2024-09-17T05:06:00.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.465+0000 7f2ae6ffd640 1 -- 172.21.15.150:0/1770762613 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2ae801c460 con 0x7f2af80c38c0 2024-09-17T05:06:00.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.465+0000 7f2ae6ffd640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2ad8073230 0x7f2ad8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:00.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.465+0000 7f2ae6ffd640 1 -- 172.21.15.150:0/1770762613 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f2ae80954c0 con 0x7f2af80c38c0 2024-09-17T05:06:00.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.465+0000 7f2af5d9b640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2ad8073230 0x7f2ad8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:00.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.465+0000 7f2af5d9b640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2ad8073230 0x7f2ad8075740 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0af000 tx=0x7f2aec01afb0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:00.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.469+0000 7f2ae6ffd640 1 -- 172.21.15.150:0/1770762613 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2ae805fad0 con 0x7f2af80c38c0 2024-09-17T05:06:00.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.681+0000 7f2afcb63640 1 -- 172.21.15.150:0/1770762613 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2af814ec80 con 0x7f2af80c38c0 2024-09-17T05:06:00.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.681+0000 7f2ae6ffd640 1 -- 172.21.15.150:0/1770762613 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) ==== 95+0+5717 (secure 0 0 0) 0x7f2ae805f3d0 con 0x7f2af80c38c0 2024-09-17T05:06:00.681 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:00.681 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":120,"btime":"2024-09-17T05:05:59:072160+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:05:59.072147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6747":{"gid":6747,"name":"a","rank":0,"incarnation":117,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6747,"qdb_cluster":[6747]},"id":2}]} 2024-09-17T05:06:00.681 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 120 2024-09-17T05:06:00.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.685+0000 7f2afcb63640 1 -- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2ad8073230 msgr2=0x7f2ad8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:00.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.685+0000 7f2afcb63640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2ad8073230 0x7f2ad8075740 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2aec0af000 tx=0x7f2aec01afb0 comp rx=0 tx=0).stop 2024-09-17T05:06:00.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.685+0000 7f2afcb63640 1 -- 172.21.15.150:0/1770762613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c38c0 msgr2=0x7f2af813f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:00.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.685+0000 7f2afcb63640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c38c0 0x7f2af813f570 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800c3f0 tx=0x7f2ae800c8b0 comp rx=0 tx=0).stop 2024-09-17T05:06:00.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.685+0000 7f2afcb63640 1 -- 172.21.15.150:0/1770762613 shutdown_connections 2024-09-17T05:06:00.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.685+0000 7f2afcb63640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2ad8073230 0x7f2ad8075740 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:00.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.685+0000 7f2afcb63640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2af80c43b0 0x7f2af80c1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:00.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.685+0000 7f2afcb63640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2af80c3fa0 0x7f2af81442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:00.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.685+0000 7f2afcb63640 1 --2- 172.21.15.150:0/1770762613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2af80c38c0 0x7f2af813f570 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:00.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.685+0000 7f2afcb63640 1 -- 172.21.15.150:0/1770762613 >> 172.21.15.150:0/1770762613 conn(0x7f2af80c5ae0 msgr2=0x7f2af80aa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:00.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.685+0000 7f2afcb63640 1 -- 172.21.15.150:0/1770762613 shutdown_connections 2024-09-17T05:06:00.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:00.685+0000 7f2afcb63640 1 -- 172.21.15.150:0/1770762613 wait complete. 2024-09-17T05:06:00.710 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=6495 2024-09-17T05:06:00.711 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before reviving mds.c 2024-09-17T05:06:02.489 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:06:02.492 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:06:03.246 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T05:06:03.249+0000 7fd8bbd6e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:06:09.711 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-17T05:06:09.712 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T05:06:09.712 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T05:06:09.715 INFO:tasks.ceph.mds.c:Started 2024-09-17T05:06:09.715 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:09.814 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T05:06:09.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.896+0000 7fb9a27d6640 1 -- 172.21.15.150:0/2704748371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb99c0c35b0 msgr2=0x7fb99c0c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:09.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.896+0000 7fb9a27d6640 1 --2- 172.21.15.150:0/2704748371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb99c0c35b0 0x7fb99c0c8a80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb990009840 tx=0x7fb99001c7f0 comp rx=0 tx=0).stop 2024-09-17T05:06:09.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.896+0000 7fb9a27d6640 1 -- 172.21.15.150:0/2704748371 shutdown_connections 2024-09-17T05:06:09.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.896+0000 7fb9a27d6640 1 --2- 172.21.15.150:0/2704748371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb99c0c35b0 0x7fb99c0c8a80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:09.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.896+0000 7fb9a27d6640 1 --2- 172.21.15.150:0/2704748371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb99c1568a0 0x7fb99c0c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:09.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.896+0000 7fb9a27d6640 1 -- 172.21.15.150:0/2704748371 >> 172.21.15.150:0/2704748371 conn(0x7fb99c074b80 msgr2=0x7fb99c06f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:09.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.896+0000 7fb9a27d6640 1 -- 172.21.15.150:0/2704748371 shutdown_connections 2024-09-17T05:06:09.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.896+0000 7fb9a27d6640 1 -- 172.21.15.150:0/2704748371 wait complete. 2024-09-17T05:06:09.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.896+0000 7fb9a27d6640 1 Processor -- start 2024-09-17T05:06:09.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.896+0000 7fb9a27d6640 1 -- start start 2024-09-17T05:06:09.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb9a27d6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb99c0c35b0 0x7fb99c147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:09.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb9a27d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb99c0c8fc0 0x7fb99c1f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:09.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb9a27d6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb99c1568a0 0x7fb99c14ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:09.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb9a27d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c0c69f0 con 0x7fb99c0c8fc0 2024-09-17T05:06:09.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb9a27d6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c154590 con 0x7fb99c0c35b0 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb9a27d6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb99c0c6b70 con 0x7fb99c1568a0 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb99b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb99c0c8fc0 0x7fb99c1f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb99b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb99c0c8fc0 0x7fb99c1f3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47800/0 (socket says 172.21.15.150:47800) 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb99b7fe640 1 -- 172.21.15.150:0/1653332727 learned_addr learned my addr 172.21.15.150:0/1653332727 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb99bfff640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb99c0c35b0 0x7fb99c147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb9a0d73640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb99c1568a0 0x7fb99c14ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb99b7fe640 1 -- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb99c1568a0 msgr2=0x7fb99c14ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb99b7fe640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb99c1568a0 0x7fb99c14ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb99b7fe640 1 -- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb99c0c35b0 msgr2=0x7fb99c147870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb99b7fe640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb99c0c35b0 0x7fb99c147870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb99b7fe640 1 -- 172.21.15.150:0/1653332727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb99c0c9de0 con 0x7fb99c0c8fc0 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb9a0d73640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb99c1568a0 0x7fb99c14ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:06:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb99bfff640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb99c0c35b0 0x7fb99c147870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:06:09.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb99b7fe640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb99c0c8fc0 0x7fb99c1f3670 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb990009790 tx=0x7fb9900a57a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:09.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb998ff9640 1 -- 172.21.15.150:0/1653332727 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9900b2040 con 0x7fb99c0c8fc0 2024-09-17T05:06:09.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb998ff9640 1 -- 172.21.15.150:0/1653332727 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb990005030 con 0x7fb99c0c8fc0 2024-09-17T05:06:09.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb998ff9640 1 -- 172.21.15.150:0/1653332727 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9900044f0 con 0x7fb99c0c8fc0 2024-09-17T05:06:09.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb9a27d6640 1 -- 172.21.15.150:0/1653332727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb99c1e9bb0 con 0x7fb99c0c8fc0 2024-09-17T05:06:09.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb9a27d6640 1 -- 172.21.15.150:0/1653332727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb99c159eb0 con 0x7fb99c0c8fc0 2024-09-17T05:06:09.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.900+0000 7fb9a27d6640 1 -- 172.21.15.150:0/1653332727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb99c1e9df0 con 0x7fb99c0c8fc0 2024-09-17T05:06:09.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.904+0000 7fb998ff9640 1 -- 172.21.15.150:0/1653332727 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb990005460 con 0x7fb99c0c8fc0 2024-09-17T05:06:09.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.904+0000 7fb998ff9640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb988073230 0x7fb988075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:09.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.904+0000 7fb998ff9640 1 -- 172.21.15.150:0/1653332727 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7fb99012ed30 con 0x7fb99c0c8fc0 2024-09-17T05:06:09.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.904+0000 7fb99bfff640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb988073230 0x7fb988075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:09.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.904+0000 7fb99bfff640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb988073230 0x7fb988075740 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009b90 tx=0x7fb98c0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:09.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:09.908+0000 7fb998ff9640 1 -- 172.21.15.150:0/1653332727 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb99c1e9df0 con 0x7fb99c0c8fc0 2024-09-17T05:06:10.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.176+0000 7fb9a27d6640 1 -- 172.21.15.150:0/1653332727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb99c0c2920 con 0x7fb99c0c8fc0 2024-09-17T05:06:10.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.176+0000 7fb998ff9640 1 -- 172.21.15.150:0/1653332727 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6540 (secure 0 0 0) 0x7fb990016410 con 0x7fb99c0c8fc0 2024-09-17T05:06:10.176 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:10.177 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":121,"btime":"2024-09-17T05:06:10:113102+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6831,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:05:59.072147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6747":{"gid":6747,"name":"a","rank":0,"incarnation":117,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6747,"qdb_cluster":[6747]},"id":2}]} 2024-09-17T05:06:10.177 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 121 2024-09-17T05:06:10.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.180+0000 7fb9a27d6640 1 -- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb988073230 msgr2=0x7fb988075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:10.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.180+0000 7fb9a27d6640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb988073230 0x7fb988075740 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009b90 tx=0x7fb98c0091e0 comp rx=0 tx=0).stop 2024-09-17T05:06:10.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.180+0000 7fb9a27d6640 1 -- 172.21.15.150:0/1653332727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb99c0c8fc0 msgr2=0x7fb99c1f3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:10.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.180+0000 7fb9a27d6640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb99c0c8fc0 0x7fb99c1f3670 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb990009790 tx=0x7fb9900a57a0 comp rx=0 tx=0).stop 2024-09-17T05:06:10.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.180+0000 7fb9a27d6640 1 -- 172.21.15.150:0/1653332727 shutdown_connections 2024-09-17T05:06:10.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.180+0000 7fb9a27d6640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb988073230 0x7fb988075740 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:10.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.180+0000 7fb9a27d6640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb99c1568a0 0x7fb99c14ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:10.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.180+0000 7fb9a27d6640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb99c0c8fc0 0x7fb99c1f3670 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:10.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.180+0000 7fb9a27d6640 1 --2- 172.21.15.150:0/1653332727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb99c0c35b0 0x7fb99c147870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:10.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.180+0000 7fb9a27d6640 1 -- 172.21.15.150:0/1653332727 >> 172.21.15.150:0/1653332727 conn(0x7fb99c074b80 msgr2=0x7fb99c14a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:10.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.180+0000 7fb9a27d6640 1 -- 172.21.15.150:0/1653332727 shutdown_connections 2024-09-17T05:06:10.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:10.180+0000 7fb9a27d6640 1 -- 172.21.15.150:0/1653332727 wait complete. 2024-09-17T05:06:10.204 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-17T05:06:10.204 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before thrashing 2024-09-17T05:06:16.205 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.384+0000 7fc9b306c640 1 -- 172.21.15.150:0/2357551391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c004d50 msgr2=0x7fc99c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.384+0000 7fc9b306c640 1 --2- 172.21.15.150:0/2357551391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c004d50 0x7fc99c005160 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0c9cd0 tx=0x7fc9a401d2c0 comp rx=0 tx=0).stop 2024-09-17T05:06:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.384+0000 7fc9b1609640 1 -- 172.21.15.150:0/2357551391 reap_dead start 2024-09-17T05:06:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.384+0000 7fc9b306c640 1 -- 172.21.15.150:0/2357551391 shutdown_connections 2024-09-17T05:06:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.384+0000 7fc9b306c640 1 -- 172.21.15.150:0/2357551391 >> 172.21.15.150:0/2357551391 conn(0x7fc9ac0c9de0 msgr2=0x7fc9ac0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.384+0000 7fc9b306c640 1 -- 172.21.15.150:0/2357551391 shutdown_connections 2024-09-17T05:06:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.384+0000 7fc9b306c640 1 -- 172.21.15.150:0/2357551391 wait complete. 2024-09-17T05:06:16.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.384+0000 7fc9b306c640 1 Processor -- start 2024-09-17T05:06:16.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.384+0000 7fc9b306c640 1 -- start start 2024-09-17T05:06:16.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b306c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c004d50 0x7fc9ac1ed2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:16.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b306c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ac1570f0 0x7fc9ac14b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:16.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b306c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9ac148920 0x7fc9ac148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:16.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b306c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac0c7e60 con 0x7fc99c004d50 2024-09-17T05:06:16.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b306c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac0c7ce0 con 0x7fc9ac148920 2024-09-17T05:06:16.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b306c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc9ac15f030 con 0x7fc9ac1570f0 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b0e08640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c004d50 0x7fc9ac1ed2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b0e08640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c004d50 0x7fc9ac1ed2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47808/0 (socket says 172.21.15.150:47808) 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b0e08640 1 -- 172.21.15.150:0/163735356 learned_addr learned my addr 172.21.15.150:0/163735356 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9abfff640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ac1570f0 0x7fc9ac14b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b1609640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9ac148920 0x7fc9ac148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b0e08640 1 -- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ac1570f0 msgr2=0x7fc9ac14b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b0e08640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ac1570f0 0x7fc9ac14b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b0e08640 1 -- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9ac148920 msgr2=0x7fc9ac148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b0e08640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9ac148920 0x7fc9ac148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b0e08640 1 -- 172.21.15.150:0/163735356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ac1510a0 con 0x7fc99c004d50 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b1609640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9ac148920 0x7fc9ac148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b0e08640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c004d50 0x7fc9ac1ed2f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0011a80 tx=0x7fc9a0011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9a97fa640 1 -- 172.21.15.150:0/163735356 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a000fe10 con 0x7fc99c004d50 2024-09-17T05:06:16.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9a97fa640 1 -- 172.21.15.150:0/163735356 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9a0004030 con 0x7fc99c004d50 2024-09-17T05:06:16.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b306c640 1 -- 172.21.15.150:0/163735356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9ac1495a0 con 0x7fc99c004d50 2024-09-17T05:06:16.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9a97fa640 1 -- 172.21.15.150:0/163735356 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0004480 con 0x7fc99c004d50 2024-09-17T05:06:16.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b306c640 1 -- 172.21.15.150:0/163735356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9ac149270 con 0x7fc99c004d50 2024-09-17T05:06:16.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.388+0000 7fc9b306c640 1 -- 172.21.15.150:0/163735356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9ac0c6230 con 0x7fc99c004d50 2024-09-17T05:06:16.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.392+0000 7fc9a97fa640 1 -- 172.21.15.150:0/163735356 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc9a0003600 con 0x7fc99c004d50 2024-09-17T05:06:16.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.392+0000 7fc9a97fa640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc990073230 0x7fc990075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:16.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.392+0000 7fc9a97fa640 1 -- 172.21.15.150:0/163735356 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7fc9a00959d0 con 0x7fc99c004d50 2024-09-17T05:06:16.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.392+0000 7fc9abfff640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc990073230 0x7fc990075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:16.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.392+0000 7fc9abfff640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc990073230 0x7fc990075740 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac1f1920 tx=0x7fc9980091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:16.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.396+0000 7fc9a97fa640 1 -- 172.21.15.150:0/163735356 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc9a005fff0 con 0x7fc99c004d50 2024-09-17T05:06:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.620+0000 7fc9b306c640 1 -- 172.21.15.150:0/163735356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9ac06a6e0 con 0x7fc99c004d50 2024-09-17T05:06:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.620+0000 7fc9a97fa640 1 -- 172.21.15.150:0/163735356 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6540 (secure 0 0 0) 0x7fc9a005f8f0 con 0x7fc99c004d50 2024-09-17T05:06:16.620 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:16.620 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":121,"btime":"2024-09-17T05:06:10:113102+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6831,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:05:59.072147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6747":{"gid":6747,"name":"a","rank":0,"incarnation":117,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6747,"qdb_cluster":[6747]},"id":2}]} 2024-09-17T05:06:16.620 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 121 2024-09-17T05:06:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.624+0000 7fc9b306c640 1 -- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc990073230 msgr2=0x7fc990075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.624+0000 7fc9b306c640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc990073230 0x7fc990075740 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac1f1920 tx=0x7fc9980091e0 comp rx=0 tx=0).stop 2024-09-17T05:06:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.624+0000 7fc9b306c640 1 -- 172.21.15.150:0/163735356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c004d50 msgr2=0x7fc9ac1ed2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.624+0000 7fc9b306c640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c004d50 0x7fc9ac1ed2f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0011a80 tx=0x7fc9a0011f40 comp rx=0 tx=0).stop 2024-09-17T05:06:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.624+0000 7fc9b306c640 1 -- 172.21.15.150:0/163735356 shutdown_connections 2024-09-17T05:06:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.624+0000 7fc9b306c640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc990073230 0x7fc990075740 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.624+0000 7fc9b306c640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9ac148920 0x7fc9ac148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.624+0000 7fc9b306c640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9ac1570f0 0x7fc9ac14b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.624+0000 7fc9b306c640 1 --2- 172.21.15.150:0/163735356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c004d50 0x7fc9ac1ed2f0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:16.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.624+0000 7fc9b306c640 1 -- 172.21.15.150:0/163735356 >> 172.21.15.150:0/163735356 conn(0x7fc9ac0c9de0 msgr2=0x7fc9ac0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:16.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.624+0000 7fc9b306c640 1 -- 172.21.15.150:0/163735356 shutdown_connections 2024-09-17T05:06:16.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.624+0000 7fc9b306c640 1 -- 172.21.15.150:0/163735356 wait complete. 2024-09-17T05:06:16.642 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-17T05:06:16.642 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T05:06:16.642 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:06:16.655 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T05:06:16.655 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:16.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.852+0000 7fe3cfae4640 1 -- 172.21.15.150:0/1176875616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c806d0d0 msgr2=0x7fe3c80a8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:16.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.852+0000 7fe3cfae4640 1 --2- 172.21.15.150:0/1176875616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c806d0d0 0x7fe3c80a8500 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009900 tx=0x7fe3c401c850 comp rx=0 tx=0).stop 2024-09-17T05:06:16.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.852+0000 7fe3cfae4640 1 -- 172.21.15.150:0/1176875616 shutdown_connections 2024-09-17T05:06:16.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.852+0000 7fe3cfae4640 1 --2- 172.21.15.150:0/1176875616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c806d0d0 0x7fe3c80a8500 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:16.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.852+0000 7fe3cfae4640 1 -- 172.21.15.150:0/1176875616 >> 172.21.15.150:0/1176875616 conn(0x7fe3c80aab60 msgr2=0x7fe3c80aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:16.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.852+0000 7fe3cfae4640 1 -- 172.21.15.150:0/1176875616 shutdown_connections 2024-09-17T05:06:16.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.852+0000 7fe3cfae4640 1 -- 172.21.15.150:0/1176875616 wait complete. 2024-09-17T05:06:16.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3cfae4640 1 Processor -- start 2024-09-17T05:06:16.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3cfae4640 1 -- start start 2024-09-17T05:06:16.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3cfae4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c806d0d0 0x7fe3c81ee4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:16.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3cfae4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c80c6cb0 0x7fe3c80a9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:16.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3cfae4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c81546f0 0x7fe3c814f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:16.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3cfae4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c8162c10 con 0x7fe3c81546f0 2024-09-17T05:06:16.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3cfae4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c8162a90 con 0x7fe3c806d0d0 2024-09-17T05:06:16.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3cfae4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c8162d90 con 0x7fe3c80c6cb0 2024-09-17T05:06:16.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3ce081640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c81546f0 0x7fe3c814f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:16.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3ce081640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c81546f0 0x7fe3c814f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47822/0 (socket says 172.21.15.150:47822) 2024-09-17T05:06:16.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3ce081640 1 -- 172.21.15.150:0/548795631 learned_addr learned my addr 172.21.15.150:0/548795631 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:16.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3cd07f640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c80c6cb0 0x7fe3c80a9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:16.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.856+0000 7fe3cd880640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c806d0d0 0x7fe3c81ee4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:16.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3ce081640 1 -- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c80c6cb0 msgr2=0x7fe3c80a9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:16.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3ce081640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c80c6cb0 0x7fe3c80a9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:16.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3ce081640 1 -- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c806d0d0 msgr2=0x7fe3c81ee4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:16.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3ce081640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c806d0d0 0x7fe3c81ee4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:16.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3ce081640 1 -- 172.21.15.150:0/548795631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c8071b90 con 0x7fe3c81546f0 2024-09-17T05:06:16.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3ce081640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c81546f0 0x7fe3c814f9d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe3c000ba10 tx=0x7fe3c000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:16.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3be7fc640 1 -- 172.21.15.150:0/548795631 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0019070 con 0x7fe3c81546f0 2024-09-17T05:06:16.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3be7fc640 1 -- 172.21.15.150:0/548795631 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3c000cc30 con 0x7fe3c81546f0 2024-09-17T05:06:16.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3cfae4640 1 -- 172.21.15.150:0/548795631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c8071e30 con 0x7fe3c81546f0 2024-09-17T05:06:16.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3be7fc640 1 -- 172.21.15.150:0/548795631 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c000ce60 con 0x7fe3c81546f0 2024-09-17T05:06:16.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3cfae4640 1 -- 172.21.15.150:0/548795631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c8150c30 con 0x7fe3c81546f0 2024-09-17T05:06:16.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3cfae4640 1 -- 172.21.15.150:0/548795631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3c80a8500 con 0x7fe3c81546f0 2024-09-17T05:06:16.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3be7fc640 1 -- 172.21.15.150:0/548795631 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe3c001c070 con 0x7fe3c81546f0 2024-09-17T05:06:16.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.860+0000 7fe3be7fc640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3ac073200 0x7fe3ac075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:16.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.864+0000 7fe3be7fc640 1 -- 172.21.15.150:0/548795631 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7fe3c0094bc0 con 0x7fe3c81546f0 2024-09-17T05:06:16.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.864+0000 7fe3cd880640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3ac073200 0x7fe3ac075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:16.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.864+0000 7fe3cd880640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3ac073200 0x7fe3ac075710 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8007fd0 tx=0x7fe3b8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:16.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:16.864+0000 7fe3be7fc640 1 -- 172.21.15.150:0/548795631 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe3c005eb80 con 0x7fe3c81546f0 2024-09-17T05:06:17.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.104+0000 7fe3cfae4640 1 -- 172.21.15.150:0/548795631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3c806d590 con 0x7fe3c81546f0 2024-09-17T05:06:17.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.104+0000 7fe3be7fc640 1 -- 172.21.15.150:0/548795631 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6540 (secure 0 0 0) 0x7fe3c00635b0 con 0x7fe3c81546f0 2024-09-17T05:06:17.103 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:17.103 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":121,"btime":"2024-09-17T05:06:10:113102+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6831,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:05:59.072147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6747":{"gid":6747,"name":"a","rank":0,"incarnation":117,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6747,"qdb_cluster":[6747]},"id":2}]} 2024-09-17T05:06:17.104 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 121 2024-09-17T05:06:17.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.108+0000 7fe3cfae4640 1 -- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3ac073200 msgr2=0x7fe3ac075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:17.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.108+0000 7fe3cfae4640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3ac073200 0x7fe3ac075710 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8007fd0 tx=0x7fe3b8009040 comp rx=0 tx=0).stop 2024-09-17T05:06:17.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.108+0000 7fe3cfae4640 1 -- 172.21.15.150:0/548795631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c81546f0 msgr2=0x7fe3c814f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:17.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.108+0000 7fe3cfae4640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c81546f0 0x7fe3c814f9d0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe3c000ba10 tx=0x7fe3c000bed0 comp rx=0 tx=0).stop 2024-09-17T05:06:17.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.108+0000 7fe3cfae4640 1 -- 172.21.15.150:0/548795631 shutdown_connections 2024-09-17T05:06:17.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.108+0000 7fe3cfae4640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3ac073200 0x7fe3ac075710 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:17.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.108+0000 7fe3cfae4640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c81546f0 0x7fe3c814f9d0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:17.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.108+0000 7fe3cfae4640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c80c6cb0 0x7fe3c80a9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:17.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.108+0000 7fe3cfae4640 1 --2- 172.21.15.150:0/548795631 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c806d0d0 0x7fe3c81ee4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:17.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.108+0000 7fe3cfae4640 1 -- 172.21.15.150:0/548795631 >> 172.21.15.150:0/548795631 conn(0x7fe3c80aab60 msgr2=0x7fe3c80c5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:17.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.108+0000 7fe3cfae4640 1 -- 172.21.15.150:0/548795631 shutdown_connections 2024-09-17T05:06:17.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:17.108+0000 7fe3cfae4640 1 -- 172.21.15.150:0/548795631 wait complete. 2024-09-17T05:06:17.125 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:06:19.126 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:19.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.292+0000 7f730ccaf640 1 -- 172.21.15.150:0/4038212049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73080745c0 msgr2=0x7f7308150d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:19.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.292+0000 7f730ccaf640 1 --2- 172.21.15.150:0/4038212049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73080745c0 0x7f7308150d00 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f73080c2ea0 tx=0x7f72f801c830 comp rx=0 tx=0).stop 2024-09-17T05:06:19.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.292+0000 7f730ccaf640 1 -- 172.21.15.150:0/4038212049 shutdown_connections 2024-09-17T05:06:19.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.292+0000 7f730ccaf640 1 --2- 172.21.15.150:0/4038212049 >> v2:172.21.15.176:3300/0 conn(0x7f73080700b0 0x7f730815dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:19.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.292+0000 7f730ccaf640 1 --2- 172.21.15.150:0/4038212049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73080745c0 0x7f7308150d00 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:19.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.292+0000 7f730ccaf640 1 -- 172.21.15.150:0/4038212049 >> 172.21.15.150:0/4038212049 conn(0x7f73080c5ae0 msgr2=0x7f7308073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:19.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.292+0000 7f730ccaf640 1 -- 172.21.15.150:0/4038212049 shutdown_connections 2024-09-17T05:06:19.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.292+0000 7f730ccaf640 1 -- 172.21.15.150:0/4038212049 wait complete. 2024-09-17T05:06:19.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.292+0000 7f730ccaf640 1 Processor -- start 2024-09-17T05:06:19.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.292+0000 7f730ccaf640 1 -- start start 2024-09-17T05:06:19.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730ccaf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f730806fca0 0x7f7308153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:19.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730ccaf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73080700b0 0x7f73081efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:19.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730ccaf640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73080745c0 0x7f73080c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:19.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730ccaf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f73080c31a0 con 0x7f730806fca0 2024-09-17T05:06:19.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730ccaf640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f73080c3020 con 0x7f73080745c0 2024-09-17T05:06:19.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730ccaf640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7308070610 con 0x7f73080700b0 2024-09-17T05:06:19.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f730806fca0 0x7f7308153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f730806fca0 0x7f7308153fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53124/0 (socket says 172.21.15.150:53124) 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730659c640 1 -- 172.21.15.150:0/2383050461 learned_addr learned my addr 172.21.15.150:0/2383050461 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f7306d9d640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73080745c0 0x7f73080c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f72fdd9b640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73080700b0 0x7f73081efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730659c640 1 -- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73080700b0 msgr2=0x7f73081efcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730659c640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73080700b0 0x7f73081efcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730659c640 1 -- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73080745c0 msgr2=0x7f73080c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730659c640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73080745c0 0x7f73080c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730659c640 1 -- 172.21.15.150:0/2383050461 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7308149f80 con 0x7f730806fca0 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730659c640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f730806fca0 0x7f7308153fe0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f72f801aee0 tx=0x7f72f80a5640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f72ff7fe640 1 -- 172.21.15.150:0/2383050461 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f80bb070 con 0x7f730806fca0 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f72ff7fe640 1 -- 172.21.15.150:0/2383050461 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72f80043b0 con 0x7f730806fca0 2024-09-17T05:06:19.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730ccaf640 1 -- 172.21.15.150:0/2383050461 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73081f34c0 con 0x7f730806fca0 2024-09-17T05:06:19.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f72ff7fe640 1 -- 172.21.15.150:0/2383050461 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f80b2040 con 0x7f730806fca0 2024-09-17T05:06:19.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730ccaf640 1 -- 172.21.15.150:0/2383050461 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73081f3a80 con 0x7f730806fca0 2024-09-17T05:06:19.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f730ccaf640 1 -- 172.21.15.150:0/2383050461 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73081f32b0 con 0x7f730806fca0 2024-09-17T05:06:19.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.296+0000 7f72ff7fe640 1 -- 172.21.15.150:0/2383050461 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f72f80be070 con 0x7f730806fca0 2024-09-17T05:06:19.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.300+0000 7f72ff7fe640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e8073230 0x7f72e8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:19.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.300+0000 7f72ff7fe640 1 -- 172.21.15.150:0/2383050461 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f72f812e500 con 0x7f730806fca0 2024-09-17T05:06:19.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.300+0000 7f72fdd9b640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e8073230 0x7f72e8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:19.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.300+0000 7f72fdd9b640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e8073230 0x7f72e8075740 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f730814af60 tx=0x7f72f0008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:19.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.304+0000 7f72ff7fe640 1 -- 172.21.15.150:0/2383050461 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f72f80164b0 con 0x7f730806fca0 2024-09-17T05:06:19.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.548+0000 7f730ccaf640 1 -- 172.21.15.150:0/2383050461 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73080ab140 con 0x7f730806fca0 2024-09-17T05:06:19.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.548+0000 7f72ff7fe640 1 -- 172.21.15.150:0/2383050461 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6540 (secure 0 0 0) 0x7f72f80f8aa0 con 0x7f730806fca0 2024-09-17T05:06:19.547 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:19.548 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":121,"btime":"2024-09-17T05:06:10:113102+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6831,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:05:59.072147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6747":{"gid":6747,"name":"a","rank":0,"incarnation":117,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6747,"qdb_cluster":[6747]},"id":2}]} 2024-09-17T05:06:19.548 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 121 2024-09-17T05:06:19.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.552+0000 7f730ccaf640 1 -- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e8073230 msgr2=0x7f72e8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:19.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.552+0000 7f730ccaf640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e8073230 0x7f72e8075740 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f730814af60 tx=0x7f72f0008040 comp rx=0 tx=0).stop 2024-09-17T05:06:19.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.552+0000 7f730ccaf640 1 -- 172.21.15.150:0/2383050461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f730806fca0 msgr2=0x7f7308153fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:19.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.552+0000 7f730ccaf640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f730806fca0 0x7f7308153fe0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f72f801aee0 tx=0x7f72f80a5640 comp rx=0 tx=0).stop 2024-09-17T05:06:19.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.552+0000 7f730ccaf640 1 -- 172.21.15.150:0/2383050461 shutdown_connections 2024-09-17T05:06:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.552+0000 7f730ccaf640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e8073230 0x7f72e8075740 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.552+0000 7f730ccaf640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73080745c0 0x7f73080c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.552+0000 7f730ccaf640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73080700b0 0x7f73081efcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.552+0000 7f730ccaf640 1 --2- 172.21.15.150:0/2383050461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f730806fca0 0x7f7308153fe0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.552+0000 7f730ccaf640 1 -- 172.21.15.150:0/2383050461 >> 172.21.15.150:0/2383050461 conn(0x7f73080c5ae0 msgr2=0x7f73080aa450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.552+0000 7f730ccaf640 1 -- 172.21.15.150:0/2383050461 shutdown_connections 2024-09-17T05:06:19.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:19.552+0000 7f730ccaf640 1 -- 172.21.15.150:0/2383050461 wait complete. 2024-09-17T05:06:19.571 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:06:21.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:21.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.732+0000 7f18b0b11640 1 -- 172.21.15.150:0/2996464585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1894005260 msgr2=0x7f189400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:21.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.732+0000 7f18b0b11640 1 --2- 172.21.15.150:0/2996464585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1894005260 0x7f189400b9d0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f18a00013e0 tx=0x7f18a001d2a0 comp rx=0 tx=0).stop 2024-09-17T05:06:21.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.732+0000 7f18b0b11640 1 -- 172.21.15.150:0/2996464585 shutdown_connections 2024-09-17T05:06:21.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.732+0000 7f18b0b11640 1 --2- 172.21.15.150:0/2996464585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1894005260 0x7f189400b9d0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:21.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.732+0000 7f18b0b11640 1 -- 172.21.15.150:0/2996464585 >> 172.21.15.150:0/2996464585 conn(0x7f18ac06e360 msgr2=0x7f18ac06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:21.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.732+0000 7f18b0b11640 1 -- 172.21.15.150:0/2996464585 shutdown_connections 2024-09-17T05:06:21.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.732+0000 7f18b0b11640 1 -- 172.21.15.150:0/2996464585 wait complete. 2024-09-17T05:06:21.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18b0b11640 1 Processor -- start 2024-09-17T05:06:21.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18b0b11640 1 -- start start 2024-09-17T05:06:21.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18b0b11640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1894005260 0x7f18ac14fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:21.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18b0b11640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ac0c21a0 0x7f18ac0c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:21.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18b0b11640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18ac1ece00 0x7f18ac1ed210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:21.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18b0b11640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac070fa0 con 0x7f18ac0c21a0 2024-09-17T05:06:21.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18b0b11640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac0c8720 con 0x7f1894005260 2024-09-17T05:06:21.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18b0b11640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f18ac071120 con 0x7f18ac1ece00 2024-09-17T05:06:21.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18a9d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ac0c21a0 0x7f18ac0c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:21.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18a9d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ac0c21a0 0x7f18ac0c25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53142/0 (socket says 172.21.15.150:53142) 2024-09-17T05:06:21.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18a9d9b640 1 -- 172.21.15.150:0/1848664940 learned_addr learned my addr 172.21.15.150:0/1848664940 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18aa59c640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1894005260 0x7f18ac14fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18aad9d640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18ac1ece00 0x7f18ac1ed210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18a9d9b640 1 -- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18ac1ece00 msgr2=0x7f18ac1ed210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18a9d9b640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18ac1ece00 0x7f18ac1ed210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18a9d9b640 1 -- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1894005260 msgr2=0x7f18ac14fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18a9d9b640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1894005260 0x7f18ac14fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18a9d9b640 1 -- 172.21.15.150:0/1848664940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18ac0a8130 con 0x7f18ac0c21a0 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18aad9d640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18ac1ece00 0x7f18ac1ed210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18a9d9b640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ac0c21a0 0x7f18ac0c25b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f189c011750 tx=0x7f189c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f189affd640 1 -- 172.21.15.150:0/1848664940 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c004460 con 0x7f18ac0c21a0 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f189affd640 1 -- 172.21.15.150:0/1848664940 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f189c016070 con 0x7f18ac0c21a0 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18b0b11640 1 -- 172.21.15.150:0/1848664940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18ac150650 con 0x7f18ac0c21a0 2024-09-17T05:06:21.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f189affd640 1 -- 172.21.15.150:0/1848664940 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c00fba0 con 0x7f18ac0c21a0 2024-09-17T05:06:21.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18b0b11640 1 -- 172.21.15.150:0/1848664940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18ac154c70 con 0x7f18ac0c21a0 2024-09-17T05:06:21.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.736+0000 7f18b0b11640 1 -- 172.21.15.150:0/1848664940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18ac1547b0 con 0x7f18ac0c21a0 2024-09-17T05:06:21.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.740+0000 7f189affd640 1 -- 172.21.15.150:0/1848664940 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f189c018030 con 0x7f18ac0c21a0 2024-09-17T05:06:21.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.740+0000 7f189affd640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f188c073200 0x7f188c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:21.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.740+0000 7f189affd640 1 -- 172.21.15.150:0/1848664940 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f189c095440 con 0x7f18ac0c21a0 2024-09-17T05:06:21.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.740+0000 7f18aa59c640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f188c073200 0x7f188c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:21.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.740+0000 7f18aa59c640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f188c073200 0x7f188c075710 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f18a40065f0 tx=0x7f18a4006b80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:21.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.744+0000 7f189affd640 1 -- 172.21.15.150:0/1848664940 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f18ac1547b0 con 0x7f18ac0c21a0 2024-09-17T05:06:21.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.968+0000 7f18b0b11640 1 -- 172.21.15.150:0/1848664940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18ac072db0 con 0x7f18ac0c21a0 2024-09-17T05:06:21.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.968+0000 7f189affd640 1 -- 172.21.15.150:0/1848664940 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6540 (secure 0 0 0) 0x7f189c05fa60 con 0x7f18ac0c21a0 2024-09-17T05:06:21.967 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:21.967 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":121,"btime":"2024-09-17T05:06:10:113102+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6831,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:05:59.072147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6747":{"gid":6747,"name":"a","rank":0,"incarnation":117,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6747,"qdb_cluster":[6747]},"id":2}]} 2024-09-17T05:06:21.967 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 121 2024-09-17T05:06:21.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.972+0000 7f18b0b11640 1 -- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f188c073200 msgr2=0x7f188c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:21.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.972+0000 7f18b0b11640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f188c073200 0x7f188c075710 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f18a40065f0 tx=0x7f18a4006b80 comp rx=0 tx=0).stop 2024-09-17T05:06:21.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.972+0000 7f18b0b11640 1 -- 172.21.15.150:0/1848664940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ac0c21a0 msgr2=0x7f18ac0c25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:21.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.972+0000 7f18b0b11640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ac0c21a0 0x7f18ac0c25b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f189c011750 tx=0x7f189c011c10 comp rx=0 tx=0).stop 2024-09-17T05:06:21.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.972+0000 7f18b0b11640 1 -- 172.21.15.150:0/1848664940 shutdown_connections 2024-09-17T05:06:21.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.972+0000 7f18b0b11640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f188c073200 0x7f188c075710 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:21.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.972+0000 7f18b0b11640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f18ac1ece00 0x7f18ac1ed210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:21.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.972+0000 7f18b0b11640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f18ac0c21a0 0x7f18ac0c25b0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:21.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.972+0000 7f18b0b11640 1 --2- 172.21.15.150:0/1848664940 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1894005260 0x7f18ac14fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:21.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.972+0000 7f18b0b11640 1 -- 172.21.15.150:0/1848664940 >> 172.21.15.150:0/1848664940 conn(0x7f18ac06e360 msgr2=0x7f18ac074730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:21.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.972+0000 7f18b0b11640 1 -- 172.21.15.150:0/1848664940 shutdown_connections 2024-09-17T05:06:21.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:21.972+0000 7f18b0b11640 1 -- 172.21.15.150:0/1848664940 wait complete. 2024-09-17T05:06:21.994 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:06:23.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:24.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.176+0000 7f7c703b5640 1 -- 172.21.15.150:0/3706745213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c54003a60 msgr2=0x7f7c54003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:24.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.176+0000 7f7c703b5640 1 --2- 172.21.15.150:0/3706745213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c54003a60 0x7f7c54003ed0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f7c60006fd0 tx=0x7f7c6001cf20 comp rx=0 tx=0).stop 2024-09-17T05:06:24.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.176+0000 7f7c703b5640 1 -- 172.21.15.150:0/3706745213 shutdown_connections 2024-09-17T05:06:24.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.176+0000 7f7c703b5640 1 --2- 172.21.15.150:0/3706745213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c54003a60 0x7f7c54003ed0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:24.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.176+0000 7f7c703b5640 1 -- 172.21.15.150:0/3706745213 >> 172.21.15.150:0/3706745213 conn(0x7f7c680aaa80 msgr2=0x7f7c680aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:24.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.176+0000 7f7c703b5640 1 -- 172.21.15.150:0/3706745213 shutdown_connections 2024-09-17T05:06:24.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.176+0000 7f7c703b5640 1 -- 172.21.15.150:0/3706745213 wait complete. 2024-09-17T05:06:24.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c703b5640 1 Processor -- start 2024-09-17T05:06:24.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c703b5640 1 -- start start 2024-09-17T05:06:24.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c703b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c54003a60 0x7f7c680aa380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:24.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c703b5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c681486c0 0x7f7c681f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:24.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c703b5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c68147080 0x7f7c68147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c703b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7c680747b0 con 0x7f7c54003a60 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c703b5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7c68074630 con 0x7f7c681486c0 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c703b5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7c68074930 con 0x7f7c68147080 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c6e151640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c54003a60 0x7f7c680aa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c6e151640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c54003a60 0x7f7c680aa380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53158/0 (socket says 172.21.15.150:53158) 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c6e151640 1 -- 172.21.15.150:0/971846787 learned_addr learned my addr 172.21.15.150:0/971846787 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c6d950640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c681486c0 0x7f7c681f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c6e952640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c68147080 0x7f7c68147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c6e151640 1 -- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c68147080 msgr2=0x7f7c68147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c6e151640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c68147080 0x7f7c68147490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c6e151640 1 -- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c681486c0 msgr2=0x7f7c681f0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c6e151640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c681486c0 0x7f7c681f0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c6e151640 1 -- 172.21.15.150:0/971846787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c681490e0 con 0x7f7c54003a60 2024-09-17T05:06:24.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c6e151640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c54003a60 0x7f7c680aa380 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f7c60007010 tx=0x7f7c60006d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:24.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c5effd640 1 -- 172.21.15.150:0/971846787 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c60001520 con 0x7f7c54003a60 2024-09-17T05:06:24.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c5effd640 1 -- 172.21.15.150:0/971846787 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c600016c0 con 0x7f7c54003a60 2024-09-17T05:06:24.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c703b5640 1 -- 172.21.15.150:0/971846787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c681493b0 con 0x7f7c54003a60 2024-09-17T05:06:24.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c5effd640 1 -- 172.21.15.150:0/971846787 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c60006330 con 0x7f7c54003a60 2024-09-17T05:06:24.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.180+0000 7f7c703b5640 1 -- 172.21.15.150:0/971846787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c681f14b0 con 0x7f7c54003a60 2024-09-17T05:06:24.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.184+0000 7f7c703b5640 1 -- 172.21.15.150:0/971846787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c680c8230 con 0x7f7c54003a60 2024-09-17T05:06:24.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.184+0000 7f7c5effd640 1 -- 172.21.15.150:0/971846787 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7c600a5c70 con 0x7f7c54003a60 2024-09-17T05:06:24.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.184+0000 7f7c5effd640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c4c073200 0x7f7c4c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:24.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.184+0000 7f7c5effd640 1 -- 172.21.15.150:0/971846787 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f7c6012e7d0 con 0x7f7c54003a60 2024-09-17T05:06:24.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.184+0000 7f7c6d950640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c4c073200 0x7f7c4c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:24.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.184+0000 7f7c6d950640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c4c073200 0x7f7c4c075710 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7c68146300 tx=0x7f7c580091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:24.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.192+0000 7f7c5effd640 1 -- 172.21.15.150:0/971846787 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7c600be060 con 0x7f7c54003a60 2024-09-17T05:06:24.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.440+0000 7f7c703b5640 1 -- 172.21.15.150:0/971846787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c6806f880 con 0x7f7c54003a60 2024-09-17T05:06:24.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.440+0000 7f7c5effd640 1 -- 172.21.15.150:0/971846787 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6540 (secure 0 0 0) 0x7f7c60016cd0 con 0x7f7c54003a60 2024-09-17T05:06:24.439 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:24.440 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":121,"btime":"2024-09-17T05:06:10:113102+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6831,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:05:59.072147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6747":{"gid":6747,"name":"a","rank":0,"incarnation":117,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6747,"qdb_cluster":[6747]},"id":2}]} 2024-09-17T05:06:24.440 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 121 2024-09-17T05:06:24.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.444+0000 7f7c703b5640 1 -- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c4c073200 msgr2=0x7f7c4c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:24.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.444+0000 7f7c703b5640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c4c073200 0x7f7c4c075710 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7c68146300 tx=0x7f7c580091e0 comp rx=0 tx=0).stop 2024-09-17T05:06:24.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.444+0000 7f7c703b5640 1 -- 172.21.15.150:0/971846787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c54003a60 msgr2=0x7f7c680aa380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:24.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.444+0000 7f7c703b5640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c54003a60 0x7f7c680aa380 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f7c60007010 tx=0x7f7c60006d50 comp rx=0 tx=0).stop 2024-09-17T05:06:24.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.444+0000 7f7c703b5640 1 -- 172.21.15.150:0/971846787 shutdown_connections 2024-09-17T05:06:24.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.444+0000 7f7c703b5640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c4c073200 0x7f7c4c075710 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:24.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.444+0000 7f7c703b5640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c681486c0 0x7f7c681f0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:24.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.444+0000 7f7c703b5640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c68147080 0x7f7c68147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:24.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.444+0000 7f7c703b5640 1 --2- 172.21.15.150:0/971846787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c54003a60 0x7f7c680aa380 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:24.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.444+0000 7f7c703b5640 1 -- 172.21.15.150:0/971846787 >> 172.21.15.150:0/971846787 conn(0x7f7c680aaa80 msgr2=0x7f7c6806d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:24.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.444+0000 7f7c703b5640 1 -- 172.21.15.150:0/971846787 shutdown_connections 2024-09-17T05:06:24.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:24.444+0000 7f7c703b5640 1 -- 172.21.15.150:0/971846787 wait complete. 2024-09-17T05:06:24.468 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:06:26.469 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:26.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.660+0000 7f5dde94d640 1 -- 172.21.15.150:0/970087005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dc0003a60 msgr2=0x7f5dc0003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:26.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.660+0000 7f5dde94d640 1 --2- 172.21.15.150:0/970087005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dc0003a60 0x7f5dc0003ed0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80016f0 tx=0x7f5dc801c750 comp rx=0 tx=0).stop 2024-09-17T05:06:26.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.660+0000 7f5dde94d640 1 -- 172.21.15.150:0/970087005 shutdown_connections 2024-09-17T05:06:26.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.660+0000 7f5dde94d640 1 --2- 172.21.15.150:0/970087005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dc0003a60 0x7f5dc0003ed0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:26.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.660+0000 7f5dde94d640 1 -- 172.21.15.150:0/970087005 >> 172.21.15.150:0/970087005 conn(0x7f5dd80aab60 msgr2=0x7f5dd80aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:26.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.660+0000 7f5dde94d640 1 -- 172.21.15.150:0/970087005 shutdown_connections 2024-09-17T05:06:26.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.660+0000 7f5dde94d640 1 -- 172.21.15.150:0/970087005 wait complete. 2024-09-17T05:06:26.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dde94d640 1 Processor -- start 2024-09-17T05:06:26.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dde94d640 1 -- start start 2024-09-17T05:06:26.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dde94d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5dc0003a60 0x7f5dd81f1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:26.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dde94d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd81f18d0 0x7f5dd814bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:26.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dde94d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5dd814c100 0x7f5dd8146d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:26.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dde94d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd80c7190 con 0x7f5dd81f18d0 2024-09-17T05:06:26.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dde94d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd80c6cb0 con 0x7f5dd814c100 2024-09-17T05:06:26.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dde94d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd80c7310 con 0x7f5dc0003a60 2024-09-17T05:06:26.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dd77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd81f18d0 0x7f5dd814bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dd77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd81f18d0 0x7f5dd814bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53188/0 (socket says 172.21.15.150:53188) 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dd77fe640 1 -- 172.21.15.150:0/3658074857 learned_addr learned my addr 172.21.15.150:0/3658074857 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dd7fff640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5dc0003a60 0x7f5dd81f1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5ddceea640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5dd814c100 0x7f5dd8146d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dd77fe640 1 -- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5dc0003a60 msgr2=0x7f5dd81f1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dd77fe640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5dc0003a60 0x7f5dd81f1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dd77fe640 1 -- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5dd814c100 msgr2=0x7f5dd8146d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dd77fe640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5dd814c100 0x7f5dd8146d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dd77fe640 1 -- 172.21.15.150:0/3658074857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd81f20a0 con 0x7f5dd81f18d0 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5ddceea640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5dd814c100 0x7f5dd8146d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dd77fe640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd81f18d0 0x7f5dd814bbc0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc011a50 tx=0x7f5dcc011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.664+0000 7f5dd4ff9640 1 -- 172.21.15.150:0/3658074857 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc01f070 con 0x7f5dd81f18d0 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.668+0000 7f5dde94d640 1 -- 172.21.15.150:0/3658074857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd814e0b0 con 0x7f5dd81f18d0 2024-09-17T05:06:26.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.668+0000 7f5dd4ff9640 1 -- 172.21.15.150:0/3658074857 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5dcc00fe80 con 0x7f5dd81f18d0 2024-09-17T05:06:26.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.668+0000 7f5dd4ff9640 1 -- 172.21.15.150:0/3658074857 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc012690 con 0x7f5dd81f18d0 2024-09-17T05:06:26.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.668+0000 7f5dde94d640 1 -- 172.21.15.150:0/3658074857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd80ac9b0 con 0x7f5dd81f18d0 2024-09-17T05:06:26.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.668+0000 7f5dde94d640 1 -- 172.21.15.150:0/3658074857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd814dd90 con 0x7f5dd81f18d0 2024-09-17T05:06:26.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.668+0000 7f5dd4ff9640 1 -- 172.21.15.150:0/3658074857 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5dcc022070 con 0x7f5dd81f18d0 2024-09-17T05:06:26.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.668+0000 7f5dd4ff9640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5dc4073200 0x7f5dc4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:26.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.668+0000 7f5dd4ff9640 1 -- 172.21.15.150:0/3658074857 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f5dcc095f50 con 0x7f5dd81f18d0 2024-09-17T05:06:26.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.668+0000 7f5dd7fff640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5dc4073200 0x7f5dc4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:26.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.672+0000 7f5dd7fff640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5dc4073200 0x7f5dc4075710 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80016c0 tx=0x7f5dc8019540 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:26.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.672+0000 7f5dd4ff9640 1 -- 172.21.15.150:0/3658074857 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5dcc05fe90 con 0x7f5dd81f18d0 2024-09-17T05:06:26.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.904+0000 7f5dde94d640 1 -- 172.21.15.150:0/3658074857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5dd80c77e0 con 0x7f5dd81f18d0 2024-09-17T05:06:26.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dd4ff9640 1 -- 172.21.15.150:0/3658074857 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6540 (secure 0 0 0) 0x7f5dcc0648c0 con 0x7f5dd81f18d0 2024-09-17T05:06:26.905 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:26.905 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":121,"btime":"2024-09-17T05:06:10:113102+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6831,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:05:59.072147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6747":{"gid":6747,"name":"a","rank":0,"incarnation":117,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6747,"qdb_cluster":[6747]},"id":2}]} 2024-09-17T05:06:26.906 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 121 2024-09-17T05:06:26.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dde94d640 1 -- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5dc4073200 msgr2=0x7f5dc4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:26.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dde94d640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5dc4073200 0x7f5dc4075710 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80016c0 tx=0x7f5dc8019540 comp rx=0 tx=0).stop 2024-09-17T05:06:26.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dde94d640 1 -- 172.21.15.150:0/3658074857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd81f18d0 msgr2=0x7f5dd814bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:26.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dde94d640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd81f18d0 0x7f5dd814bbc0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc011a50 tx=0x7f5dcc011f10 comp rx=0 tx=0).stop 2024-09-17T05:06:26.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dde94d640 1 -- 172.21.15.150:0/3658074857 shutdown_connections 2024-09-17T05:06:26.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dde94d640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5dc4073200 0x7f5dc4075710 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:26.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dde94d640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5dd814c100 0x7f5dd8146d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:26.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dde94d640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd81f18d0 0x7f5dd814bbc0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:26.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dde94d640 1 --2- 172.21.15.150:0/3658074857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5dc0003a60 0x7f5dd81f1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:26.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dde94d640 1 -- 172.21.15.150:0/3658074857 >> 172.21.15.150:0/3658074857 conn(0x7f5dd80aab60 msgr2=0x7f5dd806ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:26.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dde94d640 1 -- 172.21.15.150:0/3658074857 shutdown_connections 2024-09-17T05:06:26.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:26.908+0000 7f5dde94d640 1 -- 172.21.15.150:0/3658074857 wait complete. 2024-09-17T05:06:26.930 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:06:28.931 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:29.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.088+0000 7fb23eebe640 1 -- 172.21.15.150:0/2552969134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380c7eb0 msgr2=0x7fb2381540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:29.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.088+0000 7fb23eebe640 1 --2- 172.21.15.150:0/2552969134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380c7eb0 0x7fb2381540b0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb2380755f0 tx=0x7fb22c01c8e0 comp rx=0 tx=0).stop 2024-09-17T05:06:29.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.088+0000 7fb23eebe640 1 -- 172.21.15.150:0/2552969134 shutdown_connections 2024-09-17T05:06:29.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.088+0000 7fb23eebe640 1 --2- 172.21.15.150:0/2552969134 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb238149d70 0x7fb23814a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:29.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.088+0000 7fb23eebe640 1 --2- 172.21.15.150:0/2552969134 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380c7eb0 0x7fb2381540b0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:29.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.088+0000 7fb23eebe640 1 -- 172.21.15.150:0/2552969134 >> 172.21.15.150:0/2552969134 conn(0x7fb2380aab60 msgr2=0x7fb2380aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:29.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.088+0000 7fb23eebe640 1 -- 172.21.15.150:0/2552969134 shutdown_connections 2024-09-17T05:06:29.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.088+0000 7fb23eebe640 1 -- 172.21.15.150:0/2552969134 wait complete. 2024-09-17T05:06:29.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb23eebe640 1 Processor -- start 2024-09-17T05:06:29.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb23eebe640 1 -- start start 2024-09-17T05:06:29.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb23eebe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2380c7eb0 0x7fb2380c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:29.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb23eebe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380c8330 0x7fb2380c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:29.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb23eebe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb238149d70 0x7fb238149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:29.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb23eebe640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb2381568d0 con 0x7fb2380c8330 2024-09-17T05:06:29.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb23eebe640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb238156750 con 0x7fb238149d70 2024-09-17T05:06:29.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb23eebe640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb23814f460 con 0x7fb2380c7eb0 2024-09-17T05:06:29.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb237fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380c8330 0x7fb2380c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:29.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb237fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380c8330 0x7fb2380c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53200/0 (socket says 172.21.15.150:53200) 2024-09-17T05:06:29.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb237fff640 1 -- 172.21.15.150:0/1537523398 learned_addr learned my addr 172.21.15.150:0/1537523398 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:29.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb23d45b640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb238149d70 0x7fb238149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:29.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb23cc5a640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2380c7eb0 0x7fb2380c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:29.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb237fff640 1 -- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2380c7eb0 msgr2=0x7fb2380c23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:29.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb237fff640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2380c7eb0 0x7fb2380c23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:29.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb237fff640 1 -- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb238149d70 msgr2=0x7fb238149220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:29.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb237fff640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb238149d70 0x7fb238149220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:29.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb237fff640 1 -- 172.21.15.150:0/1537523398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23814f250 con 0x7fb2380c8330 2024-09-17T05:06:29.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb237fff640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380c8330 0x7fb2380c4910 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fb22400d950 tx=0x7fb22400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:29.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb2357fa640 1 -- 172.21.15.150:0/1537523398 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224017070 con 0x7fb2380c8330 2024-09-17T05:06:29.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb2357fa640 1 -- 172.21.15.150:0/1537523398 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb224009d20 con 0x7fb2380c8330 2024-09-17T05:06:29.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb23eebe640 1 -- 172.21.15.150:0/1537523398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2380c54c0 con 0x7fb2380c8330 2024-09-17T05:06:29.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb2357fa640 1 -- 172.21.15.150:0/1537523398 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2240106c0 con 0x7fb2380c8330 2024-09-17T05:06:29.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.092+0000 7fb23eebe640 1 -- 172.21.15.150:0/1537523398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2380c4110 con 0x7fb2380c8330 2024-09-17T05:06:29.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.096+0000 7fb23eebe640 1 -- 172.21.15.150:0/1537523398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2381540b0 con 0x7fb2380c8330 2024-09-17T05:06:29.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.096+0000 7fb2357fa640 1 -- 172.21.15.150:0/1537523398 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb22401a070 con 0x7fb2380c8330 2024-09-17T05:06:29.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.096+0000 7fb2357fa640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb21c073230 0x7fb21c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:29.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.096+0000 7fb2357fa640 1 -- 172.21.15.150:0/1537523398 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7fb224094e50 con 0x7fb2380c8330 2024-09-17T05:06:29.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.096+0000 7fb23cc5a640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb21c073230 0x7fb21c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:29.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.096+0000 7fb23cc5a640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb21c073230 0x7fb21c075740 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0af000 tx=0x7fb22c01b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:29.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.100+0000 7fb2357fa640 1 -- 172.21.15.150:0/1537523398 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb22405f470 con 0x7fb2380c8330 2024-09-17T05:06:29.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.352+0000 7fb23eebe640 1 -- 172.21.15.150:0/1537523398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb23806a6e0 con 0x7fb2380c8330 2024-09-17T05:06:29.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.352+0000 7fb2357fa640 1 -- 172.21.15.150:0/1537523398 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+6540 (secure 0 0 0) 0x7fb22405ed70 con 0x7fb2380c8330 2024-09-17T05:06:29.353 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:29.353 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":121,"btime":"2024-09-17T05:06:10:113102+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6831,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":121}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:05:59.072147+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6747},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6747":{"gid":6747,"name":"a","rank":0,"incarnation":117,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6833/1237450980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":1237450980},{"type":"v1","addr":"172.21.15.150:6833","nonce":1237450980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6747,"qdb_cluster":[6747]},"id":2}]} 2024-09-17T05:06:29.353 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 121 2024-09-17T05:06:29.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.356+0000 7fb23eebe640 1 -- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb21c073230 msgr2=0x7fb21c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:29.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.356+0000 7fb23eebe640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb21c073230 0x7fb21c075740 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0af000 tx=0x7fb22c01b120 comp rx=0 tx=0).stop 2024-09-17T05:06:29.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.356+0000 7fb23eebe640 1 -- 172.21.15.150:0/1537523398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380c8330 msgr2=0x7fb2380c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:29.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.356+0000 7fb23eebe640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380c8330 0x7fb2380c4910 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fb22400d950 tx=0x7fb22400de10 comp rx=0 tx=0).stop 2024-09-17T05:06:29.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.356+0000 7fb23eebe640 1 -- 172.21.15.150:0/1537523398 shutdown_connections 2024-09-17T05:06:29.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.356+0000 7fb23eebe640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb21c073230 0x7fb21c075740 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:29.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.356+0000 7fb23eebe640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb238149d70 0x7fb238149220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:29.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.356+0000 7fb23eebe640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2380c8330 0x7fb2380c4910 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:29.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.356+0000 7fb23eebe640 1 --2- 172.21.15.150:0/1537523398 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2380c7eb0 0x7fb2380c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:29.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.356+0000 7fb23eebe640 1 -- 172.21.15.150:0/1537523398 >> 172.21.15.150:0/1537523398 conn(0x7fb2380aab60 msgr2=0x7fb238156b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:29.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.356+0000 7fb23eebe640 1 -- 172.21.15.150:0/1537523398 shutdown_connections 2024-09-17T05:06:29.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:29.356+0000 7fb23eebe640 1 -- 172.21.15.150:0/1537523398 wait complete. 2024-09-17T05:06:29.375 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:06:31.376 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:31.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.544+0000 7f371ea08640 1 -- 172.21.15.150:0/3407986209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3700004030 msgr2=0x7f3700004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:31.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.544+0000 7f371ea08640 1 --2- 172.21.15.150:0/3407986209 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3700004030 0x7f3700004810 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f37180c4bc0 tx=0x7f370c01f420 comp rx=0 tx=0).stop 2024-09-17T05:06:31.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.544+0000 7f371cfa5640 1 -- 172.21.15.150:0/3407986209 reap_dead start 2024-09-17T05:06:31.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.544+0000 7f371ea08640 1 -- 172.21.15.150:0/3407986209 shutdown_connections 2024-09-17T05:06:31.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.544+0000 7f371ea08640 1 -- 172.21.15.150:0/3407986209 >> 172.21.15.150:0/3407986209 conn(0x7f371806e360 msgr2=0x7f371806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:31.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.544+0000 7f371ea08640 1 -- 172.21.15.150:0/3407986209 shutdown_connections 2024-09-17T05:06:31.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.544+0000 7f371ea08640 1 -- 172.21.15.150:0/3407986209 wait complete. 2024-09-17T05:06:31.545 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:31.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f371ea08640 1 Processor -- start 2024-09-17T05:06:31.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f371ea08640 1 -- start start 2024-09-17T05:06:31.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f371ea08640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3700004030 0x7f37180c2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:31.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f371ea08640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f37180c2c10 0x7f37181efc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:31.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f371ea08640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37181f0190 0x7f37180a79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:31.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f371ea08640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3718071120 con 0x7f3700004030 2024-09-17T05:06:31.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f371ea08640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3718070fa0 con 0x7f37181f0190 2024-09-17T05:06:31.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f371ea08640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3718072db0 con 0x7f37180c2c10 2024-09-17T05:06:31.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f3717fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3700004030 0x7f37180c2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:31.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f3717fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3700004030 0x7f37180c2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58636/0 (socket says 172.21.15.150:58636) 2024-09-17T05:06:31.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f3717fff640 1 -- 172.21.15.150:0/329842790 learned_addr learned my addr 172.21.15.150:0/329842790 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:31.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f37177fe640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f37180c2c10 0x7f37181efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:31.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f371cfa5640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37181f0190 0x7f37180a79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:31.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f3717fff640 1 -- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f37180c2c10 msgr2=0x7f37181efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:31.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f3717fff640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f37180c2c10 0x7f37181efc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f3717fff640 1 -- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37181f0190 msgr2=0x7f37180a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:31.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f3717fff640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37181f0190 0x7f37180a79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f3717fff640 1 -- 172.21.15.150:0/329842790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37181ed2b0 con 0x7f3700004030 2024-09-17T05:06:31.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.548+0000 7f37177fe640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f37180c2c10 0x7f37181efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:06:31.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.552+0000 7f3717fff640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3700004030 0x7f37180c2340 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f37080017d0 tx=0x7f370800e6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:31.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.552+0000 7f3714ff9640 1 -- 172.21.15.150:0/329842790 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3708016070 con 0x7f3700004030 2024-09-17T05:06:31.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.552+0000 7f3714ff9640 1 -- 172.21.15.150:0/329842790 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3708003600 con 0x7f3700004030 2024-09-17T05:06:31.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.552+0000 7f371ea08640 1 -- 172.21.15.150:0/329842790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3718154950 con 0x7f3700004030 2024-09-17T05:06:31.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.552+0000 7f3714ff9640 1 -- 172.21.15.150:0/329842790 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3708003a50 con 0x7f3700004030 2024-09-17T05:06:31.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.552+0000 7f371ea08640 1 -- 172.21.15.150:0/329842790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37180c2880 con 0x7f3700004030 2024-09-17T05:06:31.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.552+0000 7f3714ff9640 1 -- 172.21.15.150:0/329842790 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f370800d070 con 0x7f3700004030 2024-09-17T05:06:31.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.552+0000 7f3714ff9640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3704073200 0x7f3704075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:31.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.552+0000 7f3714ff9640 1 -- 172.21.15.150:0/329842790 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f3708098090 con 0x7f3700004030 2024-09-17T05:06:31.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.552+0000 7f371ea08640 1 -- 172.21.15.150:0/329842790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37180c3d10 con 0x7f3700004030 2024-09-17T05:06:31.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.552+0000 7f37177fe640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3704073200 0x7f3704075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:31.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.556+0000 7f37177fe640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3704073200 0x7f3704075710 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f370c0b2000 tx=0x7f370c01c1c0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:31.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.560+0000 7f3714ff9640 1 -- 172.21.15.150:0/329842790 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3708062550 con 0x7f3700004030 2024-09-17T05:06:31.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.712+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3689850119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13d00c7eb0 msgr2=0x7f13d01540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:31.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.712+0000 7f13d4b2d640 1 --2- 172.21.15.150:0/3689850119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13d00c7eb0 0x7f13d01540b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009900 tx=0x7f13c001c8e0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.712+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3689850119 shutdown_connections 2024-09-17T05:06:31.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.712+0000 7f13d4b2d640 1 --2- 172.21.15.150:0/3689850119 >> v2:172.21.15.176:3300/0 conn(0x7f13d014a180 0x7f13d00c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.712+0000 7f13d4b2d640 1 --2- 172.21.15.150:0/3689850119 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13d00c7eb0 0x7f13d01540b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.712+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3689850119 >> 172.21.15.150:0/3689850119 conn(0x7f13d00aab60 msgr2=0x7f13d00aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:31.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.712+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3689850119 shutdown_connections 2024-09-17T05:06:31.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.712+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3689850119 wait complete. 2024-09-17T05:06:31.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13d4b2d640 1 Processor -- start 2024-09-17T05:06:31.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13d4b2d640 1 -- start start 2024-09-17T05:06:31.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13d4b2d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13d00c7eb0 0x7f13d01eef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:31.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13d4b2d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13d0149d70 0x7f13d0150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13d4b2d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13d014a180 0x7f13d00ac360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13d4b2d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f13d00c7c60 con 0x7f13d0149d70 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13d4b2d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f13d00c7ae0 con 0x7f13d00c7eb0 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13d4b2d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f13d0156540 con 0x7f13d014a180 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13cdd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13d0149d70 0x7f13d0150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13cdd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13d0149d70 0x7f13d0150620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58646/0 (socket says 172.21.15.150:58646) 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13cdd9b640 1 -- 172.21.15.150:0/3230146635 learned_addr learned my addr 172.21.15.150:0/3230146635 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13ce59c640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13d00c7eb0 0x7f13d01eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13ced9d640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13d014a180 0x7f13d00ac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13cdd9b640 1 -- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13d014a180 msgr2=0x7f13d00ac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13cdd9b640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13d014a180 0x7f13d00ac360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13cdd9b640 1 -- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13d00c7eb0 msgr2=0x7f13d01eef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13cdd9b640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13d00c7eb0 0x7f13d01eef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13cdd9b640 1 -- 172.21.15.150:0/3230146635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d00bddd0 con 0x7f13d0149d70 2024-09-17T05:06:31.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13cdd9b640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13d0149d70 0x7f13d0150620 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f13c400c3f0 tx=0x7f13c400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:31.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13beffd640 1 -- 172.21.15.150:0/3230146635 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c40052d0 con 0x7f13d0149d70 2024-09-17T05:06:31.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13beffd640 1 -- 172.21.15.150:0/3230146635 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13c4007020 con 0x7f13d0149d70 2024-09-17T05:06:31.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3230146635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d0149b00 con 0x7f13d0149d70 2024-09-17T05:06:31.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13beffd640 1 -- 172.21.15.150:0/3230146635 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c4007470 con 0x7f13d0149d70 2024-09-17T05:06:31.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3230146635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d014fe20 con 0x7f13d0149d70 2024-09-17T05:06:31.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.716+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3230146635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13b40052a0 con 0x7f13d0149d70 2024-09-17T05:06:31.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.720+0000 7f13beffd640 1 -- 172.21.15.150:0/3230146635 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f13c40067e0 con 0x7f13d0149d70 2024-09-17T05:06:31.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.720+0000 7f13beffd640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f13b0073200 0x7f13b0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:31.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.720+0000 7f13beffd640 1 -- 172.21.15.150:0/3230146635 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f13c4094250 con 0x7f13d0149d70 2024-09-17T05:06:31.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.720+0000 7f13ce59c640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f13b0073200 0x7f13b0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:31.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.720+0000 7f13ce59c640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f13b0073200 0x7f13b0075710 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f13c00af000 tx=0x7f13c001b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:31.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.724+0000 7f13beffd640 1 -- 172.21.15.150:0/3230146635 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f13c405e710 con 0x7f13d0149d70 2024-09-17T05:06:31.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.796+0000 7f371ea08640 1 -- 172.21.15.150:0/329842790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37181f05a0 con 0x7f3700004030 2024-09-17T05:06:31.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.796+0000 7f3714ff9640 1 -- 172.21.15.150:0/329842790 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5704 (secure 0 0 0) 0x7f37181f05a0 con 0x7f3700004030 2024-09-17T05:06:31.795 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:31.795 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":122,"btime":"2024-09-17T05:06:31:077786+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:31.077779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:06:31.795 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 122 2024-09-17T05:06:31.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.800+0000 7f371ea08640 1 -- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3704073200 msgr2=0x7f3704075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:31.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.800+0000 7f371ea08640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3704073200 0x7f3704075710 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f370c0b2000 tx=0x7f370c01c1c0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.800+0000 7f371ea08640 1 -- 172.21.15.150:0/329842790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3700004030 msgr2=0x7f37180c2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:31.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.800+0000 7f371ea08640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3700004030 0x7f37180c2340 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f37080017d0 tx=0x7f370800e6b0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.800+0000 7f371ea08640 1 -- 172.21.15.150:0/329842790 shutdown_connections 2024-09-17T05:06:31.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.800+0000 7f371ea08640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3704073200 0x7f3704075710 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.800+0000 7f371ea08640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f37181f0190 0x7f37180a79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.800+0000 7f371ea08640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f37180c2c10 0x7f37181efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.800+0000 7f371ea08640 1 --2- 172.21.15.150:0/329842790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3700004030 0x7f37180c2340 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.800+0000 7f371ea08640 1 -- 172.21.15.150:0/329842790 >> 172.21.15.150:0/329842790 conn(0x7f371806e360 msgr2=0x7f371806d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:31.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.800+0000 7f371ea08640 1 -- 172.21.15.150:0/329842790 shutdown_connections 2024-09-17T05:06:31.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.800+0000 7f371ea08640 1 -- 172.21.15.150:0/329842790 wait complete. 2024-09-17T05:06:31.826 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-17T05:06:31.826 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:06:31.826 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:31.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.960+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3230146635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13b4005590 con 0x7f13d0149d70 2024-09-17T05:06:31.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.960+0000 7f13beffd640 1 -- 172.21.15.150:0/3230146635 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+5704 (secure 0 0 0) 0x7f13c4001090 con 0x7f13d0149d70 2024-09-17T05:06:31.960 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:31.960 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":122,"btime":"2024-09-17T05:06:31:077786+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:31.077779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:06:31.960 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 122 2024-09-17T05:06:31.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.964+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f13b0073200 msgr2=0x7f13b0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:31.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.964+0000 7f13d4b2d640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f13b0073200 0x7f13b0075710 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f13c00af000 tx=0x7f13c001b120 comp rx=0 tx=0).stop 2024-09-17T05:06:31.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.964+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3230146635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13d0149d70 msgr2=0x7f13d0150620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:31.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.964+0000 7f13d4b2d640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13d0149d70 0x7f13d0150620 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f13c400c3f0 tx=0x7f13c400c8b0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.964+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3230146635 shutdown_connections 2024-09-17T05:06:31.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.964+0000 7f13d4b2d640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f13b0073200 0x7f13b0075710 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.964+0000 7f13d4b2d640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13d014a180 0x7f13d00ac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.964+0000 7f13d4b2d640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13d0149d70 0x7f13d0150620 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.964+0000 7f13d4b2d640 1 --2- 172.21.15.150:0/3230146635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13d00c7eb0 0x7f13d01eef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:31.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.964+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3230146635 >> 172.21.15.150:0/3230146635 conn(0x7f13d00aab60 msgr2=0x7f13d00a9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:31.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.964+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3230146635 shutdown_connections 2024-09-17T05:06:31.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:31.964+0000 7f13d4b2d640 1 -- 172.21.15.150:0/3230146635 wait complete. 2024-09-17T05:06:31.981 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-17T05:06:31.982 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T05:06:31.982 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:06:32.001 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T05:06:32.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:32.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.020+0000 7f3c502ea640 1 -- 172.21.15.150:0/2921787122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c480c43b0 msgr2=0x7f3c480743a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:32.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.020+0000 7f3c502ea640 1 --2- 172.21.15.150:0/2921787122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c480c43b0 0x7f3c480743a0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f3c480c2e00 tx=0x7f3c4001ccc0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.020+0000 7f3c502ea640 1 -- 172.21.15.150:0/2921787122 shutdown_connections 2024-09-17T05:06:32.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.020+0000 7f3c502ea640 1 --2- 172.21.15.150:0/2921787122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c480c43b0 0x7f3c480743a0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.020+0000 7f3c502ea640 1 --2- 172.21.15.150:0/2921787122 >> v2:172.21.15.176:3300/0 conn(0x7f3c480c3fa0 0x7f3c481547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.020+0000 7f3c502ea640 1 --2- 172.21.15.150:0/2921787122 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c480c38c0 0x7f3c48151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.020+0000 7f3c502ea640 1 -- 172.21.15.150:0/2921787122 >> 172.21.15.150:0/2921787122 conn(0x7f3c480c5ae0 msgr2=0x7f3c48073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:32.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.020+0000 7f3c502ea640 1 -- 172.21.15.150:0/2921787122 shutdown_connections 2024-09-17T05:06:32.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.020+0000 7f3c502ea640 1 -- 172.21.15.150:0/2921787122 wait complete. 2024-09-17T05:06:32.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.020+0000 7f3c502ea640 1 Processor -- start 2024-09-17T05:06:32.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.020+0000 7f3c502ea640 1 -- start start 2024-09-17T05:06:32.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c502ea640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c480c38c0 0x7f3c4814f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:32.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c502ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c480c3fa0 0x7f3c48150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:32.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c502ea640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c480c43b0 0x7f3c480c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:32.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c502ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3c48154eb0 con 0x7f3c480c3fa0 2024-09-17T05:06:32.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c502ea640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3c48154d30 con 0x7f3c480c43b0 2024-09-17T05:06:32.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c502ea640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3c480c4d50 con 0x7f3c480c38c0 2024-09-17T05:06:32.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c4d885640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c480c3fa0 0x7f3c48150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c4d885640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c480c3fa0 0x7f3c48150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58676/0 (socket says 172.21.15.150:58676) 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c4d885640 1 -- 172.21.15.150:0/2990264283 learned_addr learned my addr 172.21.15.150:0/2990264283 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c4e086640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c480c38c0 0x7f3c4814f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c4e887640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c480c43b0 0x7f3c480c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c4d885640 1 -- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c480c38c0 msgr2=0x7f3c4814f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c4d885640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c480c38c0 0x7f3c4814f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c4d885640 1 -- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c480c43b0 msgr2=0x7f3c480c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c4d885640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c480c43b0 0x7f3c480c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c4d885640 1 -- 172.21.15.150:0/2990264283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4814f800 con 0x7f3c480c3fa0 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c4d885640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c480c3fa0 0x7f3c48150350 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f3c3800d750 tx=0x7f3c3800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c3effd640 1 -- 172.21.15.150:0/2990264283 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c38017070 con 0x7f3c480c3fa0 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c502ea640 1 -- 172.21.15.150:0/2990264283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c48145790 con 0x7f3c480c3fa0 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c3effd640 1 -- 172.21.15.150:0/2990264283 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c38009cc0 con 0x7f3c480c3fa0 2024-09-17T05:06:32.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c3effd640 1 -- 172.21.15.150:0/2990264283 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c380106c0 con 0x7f3c480c3fa0 2024-09-17T05:06:32.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c502ea640 1 -- 172.21.15.150:0/2990264283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c480c6ec0 con 0x7f3c480c3fa0 2024-09-17T05:06:32.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c502ea640 1 -- 172.21.15.150:0/2990264283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c481547f0 con 0x7f3c480c3fa0 2024-09-17T05:06:32.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c3effd640 1 -- 172.21.15.150:0/2990264283 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3c3801a070 con 0x7f3c480c3fa0 2024-09-17T05:06:32.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c3effd640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c2c073230 0x7f3c2c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:32.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.024+0000 7f3c3effd640 1 -- 172.21.15.150:0/2990264283 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f3c38094d70 con 0x7f3c480c3fa0 2024-09-17T05:06:32.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.028+0000 7f3c4e086640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c2c073230 0x7f3c2c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:32.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.028+0000 7f3c4e086640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c2c073230 0x7f3c2c075740 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3c48073460 tx=0x7f3c4400b260 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:32.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.032+0000 7f3c3effd640 1 -- 172.21.15.150:0/2990264283 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3c3800de60 con 0x7f3c480c3fa0 2024-09-17T05:06:32.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.152+0000 7f1d93d17640 1 -- 172.21.15.150:0/2048180184 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d8c0ae350 msgr2=0x7f1d8c074a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:32.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.152+0000 7f1d93d17640 1 --2- 172.21.15.150:0/2048180184 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d8c0ae350 0x7f1d8c074a60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009900 tx=0x7f1d8801c850 comp rx=0 tx=0).stop 2024-09-17T05:06:32.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.152+0000 7f1d93d17640 1 -- 172.21.15.150:0/2048180184 shutdown_connections 2024-09-17T05:06:32.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.152+0000 7f1d93d17640 1 --2- 172.21.15.150:0/2048180184 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d8c155730 0x7f1d8c158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.152+0000 7f1d93d17640 1 --2- 172.21.15.150:0/2048180184 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d8c0ae350 0x7f1d8c074a60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.152+0000 7f1d93d17640 1 -- 172.21.15.150:0/2048180184 >> 172.21.15.150:0/2048180184 conn(0x7f1d8c06e360 msgr2=0x7f1d8c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:32.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.152+0000 7f1d93d17640 1 -- 172.21.15.150:0/2048180184 shutdown_connections 2024-09-17T05:06:32.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.152+0000 7f1d93d17640 1 -- 172.21.15.150:0/2048180184 wait complete. 2024-09-17T05:06:32.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.152+0000 7f1d93d17640 1 Processor -- start 2024-09-17T05:06:32.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.152+0000 7f1d93d17640 1 -- start start 2024-09-17T05:06:32.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d93d17640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d8c0ae350 0x7f1d8c14c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:32.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d93d17640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d8c0c3740 0x7f1d8c14d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:32.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d93d17640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d8c155730 0x7f1d8c0aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:32.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d93d17640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8c158ae0 con 0x7f1d8c0ae350 2024-09-17T05:06:32.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d93d17640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8c0c8ac0 con 0x7f1d8c0c3740 2024-09-17T05:06:32.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d93d17640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1d8c1492a0 con 0x7f1d8c155730 2024-09-17T05:06:32.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d91ab3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d8c0ae350 0x7f1d8c14c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:32.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d91ab3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d8c0ae350 0x7f1d8c14c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58686/0 (socket says 172.21.15.150:58686) 2024-09-17T05:06:32.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d91ab3640 1 -- 172.21.15.150:0/2719418528 learned_addr learned my addr 172.21.15.150:0/2719418528 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:32.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d912b2640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d8c0c3740 0x7f1d8c14d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:32.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d922b4640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d8c155730 0x7f1d8c0aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:32.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d91ab3640 1 -- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d8c155730 msgr2=0x7f1d8c0aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:32.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d91ab3640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d8c155730 0x7f1d8c0aa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d91ab3640 1 -- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d8c0c3740 msgr2=0x7f1d8c14d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:32.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d91ab3640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d8c0c3740 0x7f1d8c14d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d91ab3640 1 -- 172.21.15.150:0/2719418528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d8c0c3050 con 0x7f1d8c0ae350 2024-09-17T05:06:32.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d912b2640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d8c0c3740 0x7f1d8c14d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:06:32.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d91ab3640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d8c0ae350 0x7f1d8c14c520 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009850 tx=0x7f1d880a6640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:32.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d827fc640 1 -- 172.21.15.150:0/2719418528 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d88004300 con 0x7f1d8c0ae350 2024-09-17T05:06:32.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d93d17640 1 -- 172.21.15.150:0/2719418528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d8c14aa50 con 0x7f1d8c0ae350 2024-09-17T05:06:32.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d827fc640 1 -- 172.21.15.150:0/2719418528 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d880bc070 con 0x7f1d8c0ae350 2024-09-17T05:06:32.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d827fc640 1 -- 172.21.15.150:0/2719418528 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d88005290 con 0x7f1d8c0ae350 2024-09-17T05:06:32.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d93d17640 1 -- 172.21.15.150:0/2719418528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d8c1528b0 con 0x7f1d8c0ae350 2024-09-17T05:06:32.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d6ffff640 1 -- 172.21.15.150:0/2719418528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d680052a0 con 0x7f1d8c0ae350 2024-09-17T05:06:32.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.156+0000 7f1d827fc640 1 -- 172.21.15.150:0/2719418528 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1d88005430 con 0x7f1d8c0ae350 2024-09-17T05:06:32.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.160+0000 7f1d827fc640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d70073200 0x7f1d70075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:32.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.160+0000 7f1d827fc640 1 -- 172.21.15.150:0/2719418528 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f1d8812f9c0 con 0x7f1d8c0ae350 2024-09-17T05:06:32.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.160+0000 7f1d912b2640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d70073200 0x7f1d70075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:32.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.160+0000 7f1d912b2640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d70073200 0x7f1d70075710 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0c3550 tx=0x7f1d7c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:32.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.164+0000 7f1d827fc640 1 -- 172.21.15.150:0/2719418528 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1d88131e20 con 0x7f1d8c0ae350 2024-09-17T05:06:32.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-09-17T05:06:32.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 -- 172.21.15.150:0/2990264283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c480c2440 con 0x7f3c480c3fa0 2024-09-17T05:06:32.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c3effd640 1 -- 172.21.15.150:0/2990264283 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+5707 (secure 0 0 0) 0x7f3c3805ec70 con 0x7f3c480c3fa0 2024-09-17T05:06:32.306 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:32.307 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":123,"btime":"2024-09-17T05:06:32:181170+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:31.587396+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:06:32.307 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 123 2024-09-17T05:06:32.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 -- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c2c073230 msgr2=0x7f3c2c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:32.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c2c073230 0x7f3c2c075740 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3c48073460 tx=0x7f3c4400b260 comp rx=0 tx=0).stop 2024-09-17T05:06:32.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 -- 172.21.15.150:0/2990264283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c480c3fa0 msgr2=0x7f3c48150350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:32.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c480c3fa0 0x7f3c48150350 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f3c3800d750 tx=0x7f3c3800dc10 comp rx=0 tx=0).stop 2024-09-17T05:06:32.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 -- 172.21.15.150:0/2990264283 shutdown_connections 2024-09-17T05:06:32.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c2c073230 0x7f3c2c075740 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c480c43b0 0x7f3c480c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c480c3fa0 0x7f3c48150350 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 --2- 172.21.15.150:0/2990264283 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c480c38c0 0x7f3c4814f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 -- 172.21.15.150:0/2990264283 >> 172.21.15.150:0/2990264283 conn(0x7f3c480c5ae0 msgr2=0x7f3c48149070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 -- 172.21.15.150:0/2990264283 shutdown_connections 2024-09-17T05:06:32.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.308+0000 7f3c502ea640 1 -- 172.21.15.150:0/2990264283 wait complete. 2024-09-17T05:06:32.329 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:06:32:181170+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 123, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 115, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6229": { "addr": "172.21.15.176:6837/3940723333", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 3940723333, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 3940723333, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6229, "incarnation": 112, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:05:24.217124+0000", "qdb_cluster": [ 6229 ], "qdb_leader": 6229, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6229 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 123, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6831": { "addr": "172.21.15.150:6835/92695211", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 92695211, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 92695211, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6831, "incarnation": 122, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:06:31.587396+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6831 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:06:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.420+0000 7f1d6ffff640 1 -- 172.21.15.150:0/2719418528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d68005590 con 0x7f1d8c0ae350 2024-09-17T05:06:32.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d827fc640 1 -- 172.21.15.150:0/2719418528 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+5707 (secure 0 0 0) 0x7f1d880164d0 con 0x7f1d8c0ae350 2024-09-17T05:06:32.421 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:32.421 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":123,"btime":"2024-09-17T05:06:32:181170+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:31.587396+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:06:32.421 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 123 2024-09-17T05:06:32.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d6ffff640 1 -- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d70073200 msgr2=0x7f1d70075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:32.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d6ffff640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d70073200 0x7f1d70075710 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0c3550 tx=0x7f1d7c009040 comp rx=0 tx=0).stop 2024-09-17T05:06:32.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d6ffff640 1 -- 172.21.15.150:0/2719418528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d8c0ae350 msgr2=0x7f1d8c14c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:32.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d6ffff640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d8c0ae350 0x7f1d8c14c520 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009850 tx=0x7f1d880a6640 comp rx=0 tx=0).stop 2024-09-17T05:06:32.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d6ffff640 1 -- 172.21.15.150:0/2719418528 shutdown_connections 2024-09-17T05:06:32.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d6ffff640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1d70073200 0x7f1d70075710 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d6ffff640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1d8c0c3740 0x7f1d8c14d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d6ffff640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1d8c155730 0x7f1d8c0aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d6ffff640 1 --2- 172.21.15.150:0/2719418528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d8c0ae350 0x7f1d8c14c520 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:32.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d6ffff640 1 -- 172.21.15.150:0/2719418528 >> 172.21.15.150:0/2719418528 conn(0x7f1d8c06e360 msgr2=0x7f1d8c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:32.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d6ffff640 1 -- 172.21.15.150:0/2719418528 shutdown_connections 2024-09-17T05:06:32.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:32.424+0000 7f1d6ffff640 1 -- 172.21.15.150:0/2719418528 wait complete. 2024-09-17T05:06:32.443 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:06:32.539 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:06:32.541 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:06:34.330 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:34.444 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:34.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.492+0000 7f85affca640 1 -- 172.21.15.150:0/2970913911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a8157c00 msgr2=0x7f85a8158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:34.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.492+0000 7f85affca640 1 --2- 172.21.15.150:0/2970913911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a8157c00 0x7f85a8158010 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f85a4009900 tx=0x7f85a401c8e0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.492+0000 7f85affca640 1 -- 172.21.15.150:0/2970913911 shutdown_connections 2024-09-17T05:06:34.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.492+0000 7f85affca640 1 --2- 172.21.15.150:0/2970913911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a8157c00 0x7f85a8158010 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.492+0000 7f85affca640 1 -- 172.21.15.150:0/2970913911 >> 172.21.15.150:0/2970913911 conn(0x7f85a80ca460 msgr2=0x7f85a80c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:34.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.492+0000 7f85affca640 1 -- 172.21.15.150:0/2970913911 shutdown_connections 2024-09-17T05:06:34.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.492+0000 7f85affca640 1 -- 172.21.15.150:0/2970913911 wait complete. 2024-09-17T05:06:34.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.492+0000 7f85affca640 1 Processor -- start 2024-09-17T05:06:34.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.492+0000 7f85affca640 1 -- start start 2024-09-17T05:06:34.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85affca640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85a8071670 0x7f85a8151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:34.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85affca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a80c5680 0x7f85a81548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:34.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85affca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85a8157c00 0x7f85a8157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:34.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85affca640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f85a8072300 con 0x7f85a80c5680 2024-09-17T05:06:34.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85affca640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f85a8072180 con 0x7f85a8071670 2024-09-17T05:06:34.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85affca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f85a81627c0 con 0x7f85a8157c00 2024-09-17T05:06:34.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85ad565640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a80c5680 0x7f85a81548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:34.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85ad565640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a80c5680 0x7f85a81548f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58704/0 (socket says 172.21.15.150:58704) 2024-09-17T05:06:34.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85ad565640 1 -- 172.21.15.150:0/3009117093 learned_addr learned my addr 172.21.15.150:0/3009117093 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85add66640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85a8071670 0x7f85a8151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85ae567640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85a8157c00 0x7f85a8157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85ad565640 1 -- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85a8157c00 msgr2=0x7f85a8157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85ad565640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85a8157c00 0x7f85a8157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85ad565640 1 -- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85a8071670 msgr2=0x7f85a8151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85ad565640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85a8071670 0x7f85a8151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85ad565640 1 -- 172.21.15.150:0/3009117093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a8154e30 con 0x7f85a80c5680 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85add66640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85a8071670 0x7f85a8151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85ad565640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a80c5680 0x7f85a81548f0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f859800c420 tx=0x7f859800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f859e7fc640 1 -- 172.21.15.150:0/3009117093 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598017070 con 0x7f85a80c5680 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f859e7fc640 1 -- 172.21.15.150:0/3009117093 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85980053c0 con 0x7f85a80c5680 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85affca640 1 -- 172.21.15.150:0/3009117093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a81ee590 con 0x7f85a80c5680 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f859e7fc640 1 -- 172.21.15.150:0/3009117093 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598003860 con 0x7f85a80c5680 2024-09-17T05:06:34.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85affca640 1 -- 172.21.15.150:0/3009117093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a8144cc0 con 0x7f85a80c5680 2024-09-17T05:06:34.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.496+0000 7f85affca640 1 -- 172.21.15.150:0/3009117093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a8153690 con 0x7f85a80c5680 2024-09-17T05:06:34.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.500+0000 7f859e7fc640 1 -- 172.21.15.150:0/3009117093 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f859801a070 con 0x7f85a80c5680 2024-09-17T05:06:34.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.500+0000 7f859e7fc640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f858c073200 0x7f858c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:34.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.500+0000 7f859e7fc640 1 -- 172.21.15.150:0/3009117093 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f8598094110 con 0x7f85a80c5680 2024-09-17T05:06:34.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.500+0000 7f85add66640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f858c073200 0x7f858c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:34.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.500+0000 7f85add66640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f858c073200 0x7f858c075710 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f85a0001db0 tx=0x7f85a0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:34.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.504+0000 7f859e7fc640 1 -- 172.21.15.150:0/3009117093 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f85a8154e30 con 0x7f85a80c5680 2024-09-17T05:06:34.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.604+0000 7fc80811b640 1 -- 172.21.15.150:0/3023162095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8000aa9b0 msgr2=0x7fc8000aadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:34.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.604+0000 7fc80811b640 1 --2- 172.21.15.150:0/3023162095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8000aa9b0 0x7fc8000aadc0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fc8000c2e70 tx=0x7fc7f801c7a0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.604+0000 7fc80811b640 1 -- 172.21.15.150:0/3023162095 shutdown_connections 2024-09-17T05:06:34.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.604+0000 7fc80811b640 1 --2- 172.21.15.150:0/3023162095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8000aa9b0 0x7fc8000aadc0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.604+0000 7fc80811b640 1 --2- 172.21.15.150:0/3023162095 >> v2:172.21.15.176:3300/0 conn(0x7fc8001542c0 0x7fc800156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.604+0000 7fc80811b640 1 -- 172.21.15.150:0/3023162095 >> 172.21.15.150:0/3023162095 conn(0x7fc8000c5ae0 msgr2=0x7fc800073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:34.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.604+0000 7fc80811b640 1 -- 172.21.15.150:0/3023162095 shutdown_connections 2024-09-17T05:06:34.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.604+0000 7fc80811b640 1 -- 172.21.15.150:0/3023162095 wait complete. 2024-09-17T05:06:34.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc80811b640 1 Processor -- start 2024-09-17T05:06:34.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc80811b640 1 -- start start 2024-09-17T05:06:34.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc80811b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8000aa9b0 0x7fc8000c0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:34.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc80811b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8000c4180 0x7fc8001f0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:34.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc80811b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8001542c0 0x7fc8000be770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:34.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc80811b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc800157420 con 0x7fc8001542c0 2024-09-17T05:06:34.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc80811b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc8000adbc0 con 0x7fc8000c4180 2024-09-17T05:06:34.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc80811b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc80006e860 con 0x7fc8000aa9b0 2024-09-17T05:06:34.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc8066b8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8001542c0 0x7fc8000be770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:34.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc8066b8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8001542c0 0x7fc8000be770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58716/0 (socket says 172.21.15.150:58716) 2024-09-17T05:06:34.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc8066b8640 1 -- 172.21.15.150:0/4286304498 learned_addr learned my addr 172.21.15.150:0/4286304498 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:34.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc8056b6640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8000c4180 0x7fc8001f0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:34.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc805eb7640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8000aa9b0 0x7fc8000c0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:34.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc8066b8640 1 -- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8000aa9b0 msgr2=0x7fc8000c0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:34.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc8066b8640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8000aa9b0 0x7fc8000c0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc8066b8640 1 -- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8000c4180 msgr2=0x7fc8001f0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:34.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc8066b8640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8000c4180 0x7fc8001f0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc8066b8640 1 -- 172.21.15.150:0/4286304498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8001f19a0 con 0x7fc8001542c0 2024-09-17T05:06:34.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc805eb7640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8000aa9b0 0x7fc8000c0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:06:34.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc8066b8640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8001542c0 0x7fc8000be770 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80b0040 tx=0x7fc7f80a5530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:34.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc7fe7fc640 1 -- 172.21.15.150:0/4286304498 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80bb070 con 0x7fc8001542c0 2024-09-17T05:06:34.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc7fe7fc640 1 -- 172.21.15.150:0/4286304498 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7f8004490 con 0x7fc8001542c0 2024-09-17T05:06:34.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc80811b640 1 -- 172.21.15.150:0/4286304498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80015c720 con 0x7fc8001542c0 2024-09-17T05:06:34.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc7fe7fc640 1 -- 172.21.15.150:0/4286304498 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80b2040 con 0x7fc8001542c0 2024-09-17T05:06:34.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc80811b640 1 -- 172.21.15.150:0/4286304498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8000bf130 con 0x7fc8001542c0 2024-09-17T05:06:34.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.608+0000 7fc80811b640 1 -- 172.21.15.150:0/4286304498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8000aadc0 con 0x7fc8001542c0 2024-09-17T05:06:34.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.612+0000 7fc7fe7fc640 1 -- 172.21.15.150:0/4286304498 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc7f80be070 con 0x7fc8001542c0 2024-09-17T05:06:34.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.612+0000 7fc7fe7fc640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7ec073230 0x7fc7ec075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:34.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.612+0000 7fc7fe7fc640 1 -- 172.21.15.150:0/4286304498 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7fc7f812e530 con 0x7fc8001542c0 2024-09-17T05:06:34.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.612+0000 7fc805eb7640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7ec073230 0x7fc7ec075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:34.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.612+0000 7fc805eb7640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7ec073230 0x7fc7ec075740 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0007e90 tx=0x7fc7f00073a0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:34.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.616+0000 7fc7fe7fc640 1 -- 172.21.15.150:0/4286304498 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc7f80164b0 con 0x7fc8001542c0 2024-09-17T05:06:34.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.716+0000 7f85affca640 1 -- 172.21.15.150:0/3009117093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85a8153d80 con 0x7f85a80c5680 2024-09-17T05:06:34.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.716+0000 7f859e7fc640 1 -- 172.21.15.150:0/3009117093 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) ==== 95+0+5711 (secure 0 0 0) 0x7f859805e650 con 0x7f85a80c5680 2024-09-17T05:06:34.717 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:34.717 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":125,"btime":"2024-09-17T05:06:34:202599+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:34.717 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 125 2024-09-17T05:06:34.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.720+0000 7f85affca640 1 -- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f858c073200 msgr2=0x7f858c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:34.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.720+0000 7f85affca640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f858c073200 0x7f858c075710 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f85a0001db0 tx=0x7f85a0009040 comp rx=0 tx=0).stop 2024-09-17T05:06:34.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.720+0000 7f85affca640 1 -- 172.21.15.150:0/3009117093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a80c5680 msgr2=0x7f85a81548f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:34.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.720+0000 7f85affca640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a80c5680 0x7f85a81548f0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f859800c420 tx=0x7f859800c8e0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.720+0000 7f85affca640 1 -- 172.21.15.150:0/3009117093 shutdown_connections 2024-09-17T05:06:34.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.720+0000 7f85affca640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f858c073200 0x7f858c075710 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.720+0000 7f85affca640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f85a8157c00 0x7f85a8157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.720+0000 7f85affca640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a80c5680 0x7f85a81548f0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.720+0000 7f85affca640 1 --2- 172.21.15.150:0/3009117093 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85a8071670 0x7f85a8151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.720+0000 7f85affca640 1 -- 172.21.15.150:0/3009117093 >> 172.21.15.150:0/3009117093 conn(0x7f85a80ca460 msgr2=0x7f85a8068eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:34.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.720+0000 7f85affca640 1 -- 172.21.15.150:0/3009117093 shutdown_connections 2024-09-17T05:06:34.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.720+0000 7f85affca640 1 -- 172.21.15.150:0/3009117093 wait complete. 2024-09-17T05:06:34.738 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=6747 2024-09-17T05:06:34.739 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.a 2024-09-17T05:06:34.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.824+0000 7fc80811b640 1 -- 172.21.15.150:0/4286304498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc800071860 con 0x7fc8001542c0 2024-09-17T05:06:34.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.824+0000 7fc7fe7fc640 1 -- 172.21.15.150:0/4286304498 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) ==== 95+0+5711 (secure 0 0 0) 0x7fc7f80f8b00 con 0x7fc8001542c0 2024-09-17T05:06:34.824 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:34.824 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":125,"btime":"2024-09-17T05:06:34:202599+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:34.825 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 125 2024-09-17T05:06:34.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.828+0000 7fc80811b640 1 -- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7ec073230 msgr2=0x7fc7ec075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:34.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.828+0000 7fc80811b640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7ec073230 0x7fc7ec075740 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0007e90 tx=0x7fc7f00073a0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.828+0000 7fc80811b640 1 -- 172.21.15.150:0/4286304498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8001542c0 msgr2=0x7fc8000be770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:34.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.828+0000 7fc80811b640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8001542c0 0x7fc8000be770 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80b0040 tx=0x7fc7f80a5530 comp rx=0 tx=0).stop 2024-09-17T05:06:34.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.828+0000 7fc80811b640 1 -- 172.21.15.150:0/4286304498 shutdown_connections 2024-09-17T05:06:34.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.828+0000 7fc80811b640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7ec073230 0x7fc7ec075740 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.828+0000 7fc80811b640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8000c4180 0x7fc8001f0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.828+0000 7fc80811b640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8001542c0 0x7fc8000be770 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.828+0000 7fc80811b640 1 --2- 172.21.15.150:0/4286304498 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8000aa9b0 0x7fc8000c0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:34.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.828+0000 7fc80811b640 1 -- 172.21.15.150:0/4286304498 >> 172.21.15.150:0/4286304498 conn(0x7fc8000c5ae0 msgr2=0x7fc800073b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:34.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.828+0000 7fc80811b640 1 -- 172.21.15.150:0/4286304498 shutdown_connections 2024-09-17T05:06:34.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:34.828+0000 7fc80811b640 1 -- 172.21.15.150:0/4286304498 wait complete. 2024-09-17T05:06:34.846 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:06:35.120 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T05:06:35.120+0000 7f6086931640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:06:36.740 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-17T05:06:36.740 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T05:06:36.740 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T05:06:36.743 INFO:tasks.ceph.mds.a:Started 2024-09-17T05:06:36.743 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:36.844 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T05:06:36.847 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.904+0000 7f1b6158d640 1 -- 172.21.15.150:0/1316172948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b44003a60 msgr2=0x7f1b44003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.904+0000 7f1b6158d640 1 --2- 172.21.15.150:0/1316172948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b44003a60 0x7f1b44003ed0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f1b5400c610 tx=0x7f1b5401d2c0 comp rx=0 tx=0).stop 2024-09-17T05:06:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.904+0000 7f1b6158d640 1 -- 172.21.15.150:0/1316172948 shutdown_connections 2024-09-17T05:06:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.904+0000 7f1b6158d640 1 --2- 172.21.15.150:0/1316172948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b44003a60 0x7f1b44003ed0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.904+0000 7f1b6158d640 1 -- 172.21.15.150:0/1316172948 >> 172.21.15.150:0/1316172948 conn(0x7f1b5c0aab60 msgr2=0x7f1b5c0aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.904+0000 7f1b6158d640 1 -- 172.21.15.150:0/1316172948 shutdown_connections 2024-09-17T05:06:36.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.904+0000 7f1b6158d640 1 -- 172.21.15.150:0/1316172948 wait complete. 2024-09-17T05:06:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b6158d640 1 Processor -- start 2024-09-17T05:06:36.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b6158d640 1 -- start start 2024-09-17T05:06:36.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b6158d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b44003a60 0x7f1b5c151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:36.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b6158d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b5c0abea0 0x7f1b5c0ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:36.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b6158d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b5c146ed0 0x7f1b5c1472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:36.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b6158d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c071a70 con 0x7f1b5c0abea0 2024-09-17T05:06:36.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b6158d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c0c8c10 con 0x7f1b44003a60 2024-09-17T05:06:36.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b6158d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1b5c071bf0 con 0x7f1b5c146ed0 2024-09-17T05:06:36.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b5a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b5c0abea0 0x7f1b5c0ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:36.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b5a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b5c0abea0 0x7f1b5c0ac2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58740/0 (socket says 172.21.15.150:58740) 2024-09-17T05:06:36.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b5a7fc640 1 -- 172.21.15.150:0/4041292565 learned_addr learned my addr 172.21.15.150:0/4041292565 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:36.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b5b7fe640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b5c146ed0 0x7f1b5c1472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:36.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b5affd640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b44003a60 0x7f1b5c151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:36.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b5a7fc640 1 -- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b5c146ed0 msgr2=0x7f1b5c1472e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:36.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b5a7fc640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b5c146ed0 0x7f1b5c1472e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:36.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b5a7fc640 1 -- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b44003a60 msgr2=0x7f1b5c151c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:36.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b5a7fc640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b44003a60 0x7f1b5c151c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:36.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b5a7fc640 1 -- 172.21.15.150:0/4041292565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5c1f2ea0 con 0x7f1b5c0abea0 2024-09-17T05:06:36.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b5a7fc640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b5c0abea0 0x7f1b5c0ac2b0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000c560 tx=0x7f1b5000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:36.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b437fe640 1 -- 172.21.15.150:0/4041292565 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b50017070 con 0x7f1b5c0abea0 2024-09-17T05:06:36.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b437fe640 1 -- 172.21.15.150:0/4041292565 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b500053e0 con 0x7f1b5c0abea0 2024-09-17T05:06:36.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b6158d640 1 -- 172.21.15.150:0/4041292565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5c0ad950 con 0x7f1b5c0abea0 2024-09-17T05:06:36.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b437fe640 1 -- 172.21.15.150:0/4041292565 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b50012890 con 0x7f1b5c0abea0 2024-09-17T05:06:36.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b6158d640 1 -- 172.21.15.150:0/4041292565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b5c152140 con 0x7f1b5c0abea0 2024-09-17T05:06:36.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.908+0000 7f1b6158d640 1 -- 172.21.15.150:0/4041292565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b5c072150 con 0x7f1b5c0abea0 2024-09-17T05:06:36.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.912+0000 7f1b437fe640 1 -- 172.21.15.150:0/4041292565 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1b5001a070 con 0x7f1b5c0abea0 2024-09-17T05:06:36.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.912+0000 7f1b437fe640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b48073200 0x7f1b48075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:36.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.912+0000 7f1b437fe640 1 -- 172.21.15.150:0/4041292565 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f1b500942c0 con 0x7f1b5c0abea0 2024-09-17T05:06:36.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.912+0000 7f1b5affd640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b48073200 0x7f1b48075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:36.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.912+0000 7f1b5affd640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b48073200 0x7f1b48075710 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1b540ac000 tx=0x7f1b540b1040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:36.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:36.920+0000 7f1b437fe640 1 -- 172.21.15.150:0/4041292565 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1b50021020 con 0x7f1b5c0abea0 2024-09-17T05:06:37.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 -- 172.21.15.150:0/2061218196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c074e60 msgr2=0x7f5d3c15df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:37.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 --2- 172.21.15.150:0/2061218196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c074e60 0x7f5d3c15df70 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009bc0 tx=0x7f5d3401cba0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 -- 172.21.15.150:0/2061218196 shutdown_connections 2024-09-17T05:06:37.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 --2- 172.21.15.150:0/2061218196 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d3c0c3900 0x7f5d3c074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 --2- 172.21.15.150:0/2061218196 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c074e60 0x7f5d3c15df70 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 -- 172.21.15.150:0/2061218196 >> 172.21.15.150:0/2061218196 conn(0x7f5d3c06e360 msgr2=0x7f5d3c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:37.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 -- 172.21.15.150:0/2061218196 shutdown_connections 2024-09-17T05:06:37.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 -- 172.21.15.150:0/2061218196 wait complete. 2024-09-17T05:06:37.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 Processor -- start 2024-09-17T05:06:37.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 -- start start 2024-09-17T05:06:37.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d3c074e60 0x7f5d3c1f1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:37.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0c3900 0x7f5d3c0adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:37.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.008+0000 7f5d42940640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d3c0c4250 0x7f5d3c0bff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:37.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d42940640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c160cd0 con 0x7f5d3c0c3900 2024-09-17T05:06:37.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d42940640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c072fc0 con 0x7f5d3c074e60 2024-09-17T05:06:37.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d42940640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5d3c160e50 con 0x7f5d3c0c4250 2024-09-17T05:06:37.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d3b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0c3900 0x7f5d3c0adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:37.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d3b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0c3900 0x7f5d3c0adfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58760/0 (socket says 172.21.15.150:58760) 2024-09-17T05:06:37.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d3b7fe640 1 -- 172.21.15.150:0/1795359011 learned_addr learned my addr 172.21.15.150:0/1795359011 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:37.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d40edd640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d3c0c4250 0x7f5d3c0bff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:37.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d3bfff640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d3c074e60 0x7f5d3c1f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:37.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d3b7fe640 1 -- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d3c0c4250 msgr2=0x7f5d3c0bff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:37.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d3b7fe640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d3c0c4250 0x7f5d3c0bff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d3b7fe640 1 -- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d3c074e60 msgr2=0x7f5d3c1f1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:37.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d3b7fe640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d3c074e60 0x7f5d3c1f1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d3b7fe640 1 -- 172.21.15.150:0/1795359011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3c1504d0 con 0x7f5d3c0c3900 2024-09-17T05:06:37.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d3b7fe640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0c3900 0x7f5d3c0adfc0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000d6b0 tx=0x7f5d3000db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:37.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d38ff9640 1 -- 172.21.15.150:0/1795359011 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30017070 con 0x7f5d3c0c3900 2024-09-17T05:06:37.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d38ff9640 1 -- 172.21.15.150:0/1795359011 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d30004310 con 0x7f5d3c0c3900 2024-09-17T05:06:37.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d42940640 1 -- 172.21.15.150:0/1795359011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d3c147ec0 con 0x7f5d3c0c3900 2024-09-17T05:06:37.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d38ff9640 1 -- 172.21.15.150:0/1795359011 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30009a80 con 0x7f5d3c0c3900 2024-09-17T05:06:37.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d42940640 1 -- 172.21.15.150:0/1795359011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d3c0be4c0 con 0x7f5d3c0c3900 2024-09-17T05:06:37.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d42940640 1 -- 172.21.15.150:0/1795359011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d3c14f690 con 0x7f5d3c0c3900 2024-09-17T05:06:37.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d38ff9640 1 -- 172.21.15.150:0/1795359011 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5d30009c20 con 0x7f5d3c0c3900 2024-09-17T05:06:37.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d38ff9640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5d28073230 0x7f5d28075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:37.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d38ff9640 1 -- 172.21.15.150:0/1795359011 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f5d30095130 con 0x7f5d3c0c3900 2024-09-17T05:06:37.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.012+0000 7f5d3bfff640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5d28073230 0x7f5d28075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:37.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.016+0000 7f5d3bfff640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5d28073230 0x7f5d28075740 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c001640 tx=0x7f5d2c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:37.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.020+0000 7f5d38ff9640 1 -- 172.21.15.150:0/1795359011 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5d3c1504d0 con 0x7f5d3c0c3900 2024-09-17T05:06:37.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 -- 172.21.15.150:0/4041292565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b5c0adb20 con 0x7f1b5c0abea0 2024-09-17T05:06:37.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b437fe640 1 -- 172.21.15.150:0/4041292565 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) ==== 95+0+5711 (secure 0 0 0) 0x7f1b5005e2c0 con 0x7f1b5c0abea0 2024-09-17T05:06:37.134 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:37.134 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":125,"btime":"2024-09-17T05:06:34:202599+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:37.134 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 125 2024-09-17T05:06:37.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 -- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b48073200 msgr2=0x7f1b48075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:37.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b48073200 0x7f1b48075710 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1b540ac000 tx=0x7f1b540b1040 comp rx=0 tx=0).stop 2024-09-17T05:06:37.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 -- 172.21.15.150:0/4041292565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b5c0abea0 msgr2=0x7f1b5c0ac2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:37.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b5c0abea0 0x7f1b5c0ac2b0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000c560 tx=0x7f1b5000c960 comp rx=0 tx=0).stop 2024-09-17T05:06:37.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 -- 172.21.15.150:0/4041292565 shutdown_connections 2024-09-17T05:06:37.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b48073200 0x7f1b48075710 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b5c146ed0 0x7f1b5c1472e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b5c0abea0 0x7f1b5c0ac2b0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 --2- 172.21.15.150:0/4041292565 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b44003a60 0x7f1b5c151c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 -- 172.21.15.150:0/4041292565 >> 172.21.15.150:0/4041292565 conn(0x7f1b5c0aab60 msgr2=0x7f1b5c149290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:37.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 -- 172.21.15.150:0/4041292565 shutdown_connections 2024-09-17T05:06:37.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.136+0000 7f1b6158d640 1 -- 172.21.15.150:0/4041292565 wait complete. 2024-09-17T05:06:37.153 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-17T05:06:37.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.236+0000 7f5d42940640 1 -- 172.21.15.150:0/1795359011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d3c074920 con 0x7f5d3c0c3900 2024-09-17T05:06:37.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.236+0000 7f5d38ff9640 1 -- 172.21.15.150:0/1795359011 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+6537 (secure 0 0 0) 0x7f5d3005f010 con 0x7f5d3c0c3900 2024-09-17T05:06:37.235 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:37.235 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":126,"btime":"2024-09-17T05:06:37:216871+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6912,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:37.235 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 126 2024-09-17T05:06:37.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.236+0000 7f5d42940640 1 -- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5d28073230 msgr2=0x7f5d28075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:37.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.236+0000 7f5d42940640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5d28073230 0x7f5d28075740 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c001640 tx=0x7f5d2c009040 comp rx=0 tx=0).stop 2024-09-17T05:06:37.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.236+0000 7f5d42940640 1 -- 172.21.15.150:0/1795359011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0c3900 msgr2=0x7f5d3c0adfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:37.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.236+0000 7f5d42940640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0c3900 0x7f5d3c0adfc0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000d6b0 tx=0x7f5d3000db70 comp rx=0 tx=0).stop 2024-09-17T05:06:37.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.236+0000 7f5d42940640 1 -- 172.21.15.150:0/1795359011 shutdown_connections 2024-09-17T05:06:37.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.236+0000 7f5d42940640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5d28073230 0x7f5d28075740 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.236+0000 7f5d42940640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5d3c0c4250 0x7f5d3c0bff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.236+0000 7f5d42940640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0c3900 0x7f5d3c0adfc0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.236+0000 7f5d42940640 1 --2- 172.21.15.150:0/1795359011 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5d3c074e60 0x7f5d3c1f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:37.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.236+0000 7f5d42940640 1 -- 172.21.15.150:0/1795359011 >> 172.21.15.150:0/1795359011 conn(0x7f5d3c06e360 msgr2=0x7f5d3c152e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:37.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.240+0000 7f5d42940640 1 -- 172.21.15.150:0/1795359011 shutdown_connections 2024-09-17T05:06:37.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:37.240+0000 7f5d42940640 1 -- 172.21.15.150:0/1795359011 wait complete. 2024-09-17T05:06:37.256 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:06:38.816 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-09-17T05:06:39.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:39.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:39.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 -- 172.21.15.150:0/4071562425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373c14e490 msgr2=0x7f373c0c7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:39.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 --2- 172.21.15.150:0/4071562425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373c14e490 0x7f373c0c7d60 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f373c075260 tx=0x7f373001c870 comp rx=0 tx=0).stop 2024-09-17T05:06:39.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 -- 172.21.15.150:0/4071562425 shutdown_connections 2024-09-17T05:06:39.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 --2- 172.21.15.150:0/4071562425 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f373c157c00 0x7f373c158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 --2- 172.21.15.150:0/4071562425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373c14e490 0x7f373c0c7d60 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 --2- 172.21.15.150:0/4071562425 >> v2:172.21.15.176:3300/0 conn(0x7f373c071670 0x7f373c0c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 -- 172.21.15.150:0/4071562425 >> 172.21.15.150:0/4071562425 conn(0x7f373c0ca460 msgr2=0x7f373c0c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:39.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 -- 172.21.15.150:0/4071562425 shutdown_connections 2024-09-17T05:06:39.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 -- 172.21.15.150:0/4071562425 wait complete. 2024-09-17T05:06:39.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 Processor -- start 2024-09-17T05:06:39.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 -- start start 2024-09-17T05:06:39.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373c071670 0x7f373c147b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:39.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f373c14e490 0x7f373c15afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:39.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f373c157c00 0x7f373c15b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:39.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f373c0c6740 con 0x7f373c071670 2024-09-17T05:06:39.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f373c0c65c0 con 0x7f373c157c00 2024-09-17T05:06:39.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f3742c34640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f373c14e8a0 con 0x7f373c14e490 2024-09-17T05:06:39.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f37409d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373c071670 0x7f373c147b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:39.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f37409d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373c071670 0x7f373c147b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54876/0 (socket says 172.21.15.150:54876) 2024-09-17T05:06:39.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.320+0000 7f37409d0640 1 -- 172.21.15.150:0/2983008922 learned_addr learned my addr 172.21.15.150:0/2983008922 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:39.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f373bfff640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f373c14e490 0x7f373c15afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:39.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f37411d1640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f373c157c00 0x7f373c15b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:39.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f37409d0640 1 -- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f373c14e490 msgr2=0x7f373c15afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:39.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f37409d0640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f373c14e490 0x7f373c15afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f37409d0640 1 -- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f373c157c00 msgr2=0x7f373c15b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:39.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f37409d0640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f373c157c00 0x7f373c15b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f37409d0640 1 -- 172.21.15.150:0/2983008922 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373c15dc80 con 0x7f373c071670 2024-09-17T05:06:39.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f37411d1640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f373c157c00 0x7f373c15b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:06:39.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f37409d0640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373c071670 0x7f373c147b60 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f373400cce0 tx=0x7f373400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:39.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f37397fa640 1 -- 172.21.15.150:0/2983008922 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3734004b30 con 0x7f373c071670 2024-09-17T05:06:39.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f37397fa640 1 -- 172.21.15.150:0/2983008922 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3734004dc0 con 0x7f373c071670 2024-09-17T05:06:39.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f37397fa640 1 -- 172.21.15.150:0/2983008922 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3734005280 con 0x7f373c071670 2024-09-17T05:06:39.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f3742c34640 1 -- 172.21.15.150:0/2983008922 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f373c147080 con 0x7f373c071670 2024-09-17T05:06:39.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f3742c34640 1 -- 172.21.15.150:0/2983008922 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f373c1480a0 con 0x7f373c071670 2024-09-17T05:06:39.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f3742c34640 1 -- 172.21.15.150:0/2983008922 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f373c072180 con 0x7f373c071670 2024-09-17T05:06:39.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.324+0000 7f37397fa640 1 -- 172.21.15.150:0/2983008922 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3734005420 con 0x7f373c071670 2024-09-17T05:06:39.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.328+0000 7f37397fa640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3728073230 0x7f3728075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:39.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.328+0000 7f37397fa640 1 -- 172.21.15.150:0/2983008922 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f37340954b0 con 0x7f373c071670 2024-09-17T05:06:39.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.328+0000 7f373bfff640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3728073230 0x7f3728075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:39.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.328+0000 7f373bfff640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3728073230 0x7f3728075740 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f37300af000 tx=0x7f3730019610 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:39.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.332+0000 7f37397fa640 1 -- 172.21.15.150:0/2983008922 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3734014030 con 0x7f373c071670 2024-09-17T05:06:39.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.424+0000 7f424fc6f640 1 -- 172.21.15.150:0/3294519886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4234003a40 msgr2=0x7f4234003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:39.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.424+0000 7f424fc6f640 1 --2- 172.21.15.150:0/3294519886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4234003a40 0x7f4234003eb0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f42440013e0 tx=0x7f424401d2a0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.424+0000 7f424e20c640 1 -- 172.21.15.150:0/3294519886 reap_dead start 2024-09-17T05:06:39.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.424+0000 7f424fc6f640 1 -- 172.21.15.150:0/3294519886 shutdown_connections 2024-09-17T05:06:39.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.424+0000 7f424fc6f640 1 -- 172.21.15.150:0/3294519886 >> 172.21.15.150:0/3294519886 conn(0x7f42480abb00 msgr2=0x7f42480abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:39.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.424+0000 7f424fc6f640 1 -- 172.21.15.150:0/3294519886 shutdown_connections 2024-09-17T05:06:39.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.424+0000 7f424fc6f640 1 -- 172.21.15.150:0/3294519886 wait complete. 2024-09-17T05:06:39.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424fc6f640 1 Processor -- start 2024-09-17T05:06:39.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424fc6f640 1 -- start start 2024-09-17T05:06:39.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424fc6f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4234003a40 0x7f42481574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:39.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424fc6f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3830 0x7f42481f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:39.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424fc6f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42480c3c40 0x7f424814d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:39.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424fc6f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f424815ff80 con 0x7f4234003a40 2024-09-17T05:06:39.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424fc6f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f42480c4df0 con 0x7f42480c3c40 2024-09-17T05:06:39.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424fc6f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4248160100 con 0x7f42480c3830 2024-09-17T05:06:39.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424da0b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4234003a40 0x7f42481574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:39.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424da0b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4234003a40 0x7f42481574e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54884/0 (socket says 172.21.15.150:54884) 2024-09-17T05:06:39.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424da0b640 1 -- 172.21.15.150:0/870465761 learned_addr learned my addr 172.21.15.150:0/870465761 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:39.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424d20a640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3830 0x7f42481f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:39.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424e20c640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42480c3c40 0x7f424814d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:39.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424da0b640 1 -- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3830 msgr2=0x7f42481f0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:39.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424da0b640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3830 0x7f42481f0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424da0b640 1 -- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42480c3c40 msgr2=0x7f424814d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:39.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424da0b640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42480c3c40 0x7f424814d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424da0b640 1 -- 172.21.15.150:0/870465761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4248156750 con 0x7f4234003a40 2024-09-17T05:06:39.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424da0b640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4234003a40 0x7f42481574e0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f424401b9a0 tx=0x7f424400c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:39.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f423e7fc640 1 -- 172.21.15.150:0/870465761 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42440ae440 con 0x7f4234003a40 2024-09-17T05:06:39.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f423e7fc640 1 -- 172.21.15.150:0/870465761 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42440ae6f0 con 0x7f4234003a40 2024-09-17T05:06:39.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f423e7fc640 1 -- 172.21.15.150:0/870465761 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42440aeb40 con 0x7f4234003a40 2024-09-17T05:06:39.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424fc6f640 1 -- 172.21.15.150:0/870465761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f424814e6e0 con 0x7f4234003a40 2024-09-17T05:06:39.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.428+0000 7f424fc6f640 1 -- 172.21.15.150:0/870465761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42481f17d0 con 0x7f4234003a40 2024-09-17T05:06:39.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.432+0000 7f424fc6f640 1 -- 172.21.15.150:0/870465761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42480c7cd0 con 0x7f4234003a40 2024-09-17T05:06:39.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.432+0000 7f423e7fc640 1 -- 172.21.15.150:0/870465761 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f42440aece0 con 0x7f4234003a40 2024-09-17T05:06:39.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.432+0000 7f423e7fc640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f422c073230 0x7f422c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:39.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.432+0000 7f423e7fc640 1 -- 172.21.15.150:0/870465761 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f424412fef0 con 0x7f4234003a40 2024-09-17T05:06:39.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.432+0000 7f424d20a640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f422c073230 0x7f422c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:39.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.432+0000 7f424d20a640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f422c073230 0x7f422c075740 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f42480af080 tx=0x7f4238006f00 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:39.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.440+0000 7f423e7fc640 1 -- 172.21.15.150:0/870465761 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f42480736c0 con 0x7f4234003a40 2024-09-17T05:06:39.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.548+0000 7f3742c34640 1 -- 172.21.15.150:0/2983008922 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f373c158310 con 0x7f373c071670 2024-09-17T05:06:39.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.548+0000 7f37397fa640 1 -- 172.21.15.150:0/2983008922 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+6537 (secure 0 0 0) 0x7f373405fa80 con 0x7f373c071670 2024-09-17T05:06:39.547 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:39.548 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":126,"btime":"2024-09-17T05:06:37:216871+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6912,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:39.548 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 126 2024-09-17T05:06:39.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.552+0000 7f3742c34640 1 -- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3728073230 msgr2=0x7f3728075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:39.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.552+0000 7f3742c34640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3728073230 0x7f3728075740 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f37300af000 tx=0x7f3730019610 comp rx=0 tx=0).stop 2024-09-17T05:06:39.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.552+0000 7f3742c34640 1 -- 172.21.15.150:0/2983008922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373c071670 msgr2=0x7f373c147b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:39.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.552+0000 7f3742c34640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373c071670 0x7f373c147b60 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f373400cce0 tx=0x7f373400a620 comp rx=0 tx=0).stop 2024-09-17T05:06:39.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.552+0000 7f3742c34640 1 -- 172.21.15.150:0/2983008922 shutdown_connections 2024-09-17T05:06:39.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.552+0000 7f3742c34640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3728073230 0x7f3728075740 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.552+0000 7f3742c34640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f373c157c00 0x7f373c15b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.552+0000 7f3742c34640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f373c14e490 0x7f373c15afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.552+0000 7f3742c34640 1 --2- 172.21.15.150:0/2983008922 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f373c071670 0x7f373c147b60 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.552+0000 7f3742c34640 1 -- 172.21.15.150:0/2983008922 >> 172.21.15.150:0/2983008922 conn(0x7f373c0ca460 msgr2=0x7f373c153d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:39.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.552+0000 7f3742c34640 1 -- 172.21.15.150:0/2983008922 shutdown_connections 2024-09-17T05:06:39.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.552+0000 7f3742c34640 1 -- 172.21.15.150:0/2983008922 wait complete. 2024-09-17T05:06:39.577 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-17T05:06:39.577 INFO:tasks.mds_thrash.fs.[b]:waiting for 78 secs before thrashing 2024-09-17T05:06:39.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.656+0000 7f424fc6f640 1 -- 172.21.15.150:0/870465761 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42480ac090 con 0x7f4234003a40 2024-09-17T05:06:39.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.656+0000 7f423e7fc640 1 -- 172.21.15.150:0/870465761 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+6537 (secure 0 0 0) 0x7f42440f9d50 con 0x7f4234003a40 2024-09-17T05:06:39.657 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:39.657 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":126,"btime":"2024-09-17T05:06:37:216871+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6912,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:39.657 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 126 2024-09-17T05:06:39.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.660+0000 7f424fc6f640 1 -- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f422c073230 msgr2=0x7f422c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:39.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.660+0000 7f424fc6f640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f422c073230 0x7f422c075740 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f42480af080 tx=0x7f4238006f00 comp rx=0 tx=0).stop 2024-09-17T05:06:39.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.660+0000 7f424fc6f640 1 -- 172.21.15.150:0/870465761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4234003a40 msgr2=0x7f42481574e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:39.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.660+0000 7f424fc6f640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4234003a40 0x7f42481574e0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f424401b9a0 tx=0x7f424400c470 comp rx=0 tx=0).stop 2024-09-17T05:06:39.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.660+0000 7f424fc6f640 1 -- 172.21.15.150:0/870465761 shutdown_connections 2024-09-17T05:06:39.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.660+0000 7f424fc6f640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f422c073230 0x7f422c075740 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.660+0000 7f424fc6f640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42480c3c40 0x7f424814d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.660+0000 7f424fc6f640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3830 0x7f42481f0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.660+0000 7f424fc6f640 1 --2- 172.21.15.150:0/870465761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4234003a40 0x7f42481574e0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:39.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.660+0000 7f424fc6f640 1 -- 172.21.15.150:0/870465761 >> 172.21.15.150:0/870465761 conn(0x7f42480abb00 msgr2=0x7f42480c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:39.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.660+0000 7f424fc6f640 1 -- 172.21.15.150:0/870465761 shutdown_connections 2024-09-17T05:06:39.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:39.660+0000 7f424fc6f640 1 -- 172.21.15.150:0/870465761 wait complete. 2024-09-17T05:06:39.677 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:06:41.679 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:41.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.844+0000 7f979ffff640 1 -- 172.21.15.150:0/3479986911 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97980a5ce0 con 0x7f97a8151d00 2024-09-17T05:06:41.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.844+0000 7f97af6ca640 1 -- 172.21.15.150:0/3479986911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97a8151d00 msgr2=0x7f97a80c3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:41.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.844+0000 7f97af6ca640 1 --2- 172.21.15.150:0/3479986911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97a8151d00 0x7f97a80c3430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9798009900 tx=0x7f979801c8e0 comp rx=0 tx=0).stop 2024-09-17T05:06:41.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.844+0000 7f97af6ca640 1 -- 172.21.15.150:0/3479986911 shutdown_connections 2024-09-17T05:06:41.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.844+0000 7f97af6ca640 1 --2- 172.21.15.150:0/3479986911 >> v2:172.21.15.176:3300/0 conn(0x7f97a80c3970 0x7f97a80c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:41.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.844+0000 7f97af6ca640 1 --2- 172.21.15.150:0/3479986911 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97a8151d00 0x7f97a80c3430 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:41.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.844+0000 7f97af6ca640 1 -- 172.21.15.150:0/3479986911 >> 172.21.15.150:0/3479986911 conn(0x7f97a80c4a00 msgr2=0x7f97a80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:41.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.844+0000 7f97af6ca640 1 -- 172.21.15.150:0/3479986911 shutdown_connections 2024-09-17T05:06:41.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.844+0000 7f97af6ca640 1 -- 172.21.15.150:0/3479986911 wait complete. 2024-09-17T05:06:41.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97af6ca640 1 Processor -- start 2024-09-17T05:06:41.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97af6ca640 1 -- start start 2024-09-17T05:06:41.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97af6ca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97a80c3970 0x7f97a81efd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:41.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97af6ca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97a8151d00 0x7f97a814a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:41.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97af6ca640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97a8157c30 0x7f97a80c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:41.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97af6ca640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f97a81619b0 con 0x7f97a80c3970 2024-09-17T05:06:41.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97af6ca640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f97a8161830 con 0x7f97a8157c30 2024-09-17T05:06:41.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97af6ca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f97a8161b30 con 0x7f97a8151d00 2024-09-17T05:06:41.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97ad466640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97a80c3970 0x7f97a81efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:41.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97ad466640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97a80c3970 0x7f97a81efd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54896/0 (socket says 172.21.15.150:54896) 2024-09-17T05:06:41.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97ad466640 1 -- 172.21.15.150:0/3408102693 learned_addr learned my addr 172.21.15.150:0/3408102693 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:41.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97acc65640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97a8151d00 0x7f97a814a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:41.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97adc67640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97a8157c30 0x7f97a80c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:41.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97ad466640 1 -- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97a8151d00 msgr2=0x7f97a814a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:41.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97ad466640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97a8151d00 0x7f97a814a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:41.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97ad466640 1 -- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97a8157c30 msgr2=0x7f97a80c6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:41.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97ad466640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97a8157c30 0x7f97a80c6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:41.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97ad466640 1 -- 172.21.15.150:0/3408102693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a80a7d50 con 0x7f97a80c3970 2024-09-17T05:06:41.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97ad466640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97a80c3970 0x7f97a81efd70 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f97980b0040 tx=0x7f9798005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:41.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f979dffb640 1 -- 172.21.15.150:0/3408102693 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97980b2040 con 0x7f97a80c3970 2024-09-17T05:06:41.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f979dffb640 1 -- 172.21.15.150:0/3408102693 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9798004310 con 0x7f97a80c3970 2024-09-17T05:06:41.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97af6ca640 1 -- 172.21.15.150:0/3408102693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a80a7a70 con 0x7f97a80c3970 2024-09-17T05:06:41.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f979dffb640 1 -- 172.21.15.150:0/3408102693 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798005250 con 0x7f97a80c3970 2024-09-17T05:06:41.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.848+0000 7f97af6ca640 1 -- 172.21.15.150:0/3408102693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a80a8990 con 0x7f97a80c3970 2024-09-17T05:06:41.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.852+0000 7f97af6ca640 1 -- 172.21.15.150:0/3408102693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a80c3d80 con 0x7f97a80c3970 2024-09-17T05:06:41.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.852+0000 7f979dffb640 1 -- 172.21.15.150:0/3408102693 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f97980053f0 con 0x7f97a80c3970 2024-09-17T05:06:41.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.852+0000 7f979dffb640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f978c073200 0x7f978c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:41.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.852+0000 7f979dffb640 1 -- 172.21.15.150:0/3408102693 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f979812e860 con 0x7f97a80c3970 2024-09-17T05:06:41.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.852+0000 7f97acc65640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f978c073200 0x7f978c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:41.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.852+0000 7f97acc65640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f978c073200 0x7f978c075710 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f97a80755d0 tx=0x7f97a40091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:41.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:41.856+0000 7f979dffb640 1 -- 172.21.15.150:0/3408102693 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f97980164d0 con 0x7f97a80c3970 2024-09-17T05:06:42.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.064+0000 7f97af6ca640 1 -- 172.21.15.150:0/3408102693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97a80a7730 con 0x7f97a80c3970 2024-09-17T05:06:42.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.064+0000 7f979dffb640 1 -- 172.21.15.150:0/3408102693 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+6537 (secure 0 0 0) 0x7f97980f8740 con 0x7f97a80c3970 2024-09-17T05:06:42.064 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:42.065 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":126,"btime":"2024-09-17T05:06:37:216871+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6912,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:42.065 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 126 2024-09-17T05:06:42.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.068+0000 7f97af6ca640 1 -- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f978c073200 msgr2=0x7f978c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:42.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.068+0000 7f97af6ca640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f978c073200 0x7f978c075710 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f97a80755d0 tx=0x7f97a40091e0 comp rx=0 tx=0).stop 2024-09-17T05:06:42.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.068+0000 7f97af6ca640 1 -- 172.21.15.150:0/3408102693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97a80c3970 msgr2=0x7f97a81efd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:42.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.068+0000 7f97af6ca640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97a80c3970 0x7f97a81efd70 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f97980b0040 tx=0x7f9798005e70 comp rx=0 tx=0).stop 2024-09-17T05:06:42.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.068+0000 7f97af6ca640 1 -- 172.21.15.150:0/3408102693 shutdown_connections 2024-09-17T05:06:42.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.068+0000 7f97af6ca640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f978c073200 0x7f978c075710 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:42.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.068+0000 7f97af6ca640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97a8157c30 0x7f97a80c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.068+0000 7f97af6ca640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97a8151d00 0x7f97a814a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.068+0000 7f97af6ca640 1 --2- 172.21.15.150:0/3408102693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97a80c3970 0x7f97a81efd70 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.068+0000 7f97af6ca640 1 -- 172.21.15.150:0/3408102693 >> 172.21.15.150:0/3408102693 conn(0x7f97a80c4a00 msgr2=0x7f97a8157390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.068+0000 7f97af6ca640 1 -- 172.21.15.150:0/3408102693 shutdown_connections 2024-09-17T05:06:42.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:42.068+0000 7f97af6ca640 1 -- 172.21.15.150:0/3408102693 wait complete. 2024-09-17T05:06:42.086 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:06:44.086 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.252+0000 7fa87b20a640 1 -- 172.21.15.150:0/2947531159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa85c003a10 msgr2=0x7fa85c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.252+0000 7fa87b20a640 1 --2- 172.21.15.150:0/2947531159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa85c003a10 0x7fa85c003e80 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009f00 tx=0x7fa86c01d5e0 comp rx=0 tx=0).stop 2024-09-17T05:06:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.252+0000 7fa8797a7640 1 -- 172.21.15.150:0/2947531159 reap_dead start 2024-09-17T05:06:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.252+0000 7fa87b20a640 1 -- 172.21.15.150:0/2947531159 shutdown_connections 2024-09-17T05:06:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.252+0000 7fa87b20a640 1 -- 172.21.15.150:0/2947531159 >> 172.21.15.150:0/2947531159 conn(0x7fa8740c9de0 msgr2=0x7fa8740ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.252+0000 7fa87b20a640 1 -- 172.21.15.150:0/2947531159 shutdown_connections 2024-09-17T05:06:44.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.252+0000 7fa87b20a640 1 -- 172.21.15.150:0/2947531159 wait complete. 2024-09-17T05:06:44.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa87b20a640 1 Processor -- start 2024-09-17T05:06:44.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa87b20a640 1 -- start start 2024-09-17T05:06:44.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa87b20a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa85c003a10 0x7fa8741f0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:44.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa87b20a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8741f34a0 0x7fa8740c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:44.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa87b20a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa87414b7c0 0x7fa87414bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:44.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa87b20a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa8740c7e60 con 0x7fa87414b7c0 2024-09-17T05:06:44.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa87b20a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa8740c7ce0 con 0x7fa8741f34a0 2024-09-17T05:06:44.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa87b20a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa87415f030 con 0x7fa85c003a10 2024-09-17T05:06:44.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa8797a7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa87414b7c0 0x7fa87414bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:44.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa8797a7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa87414b7c0 0x7fa87414bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54912/0 (socket says 172.21.15.150:54912) 2024-09-17T05:06:44.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa8797a7640 1 -- 172.21.15.150:0/4093704473 learned_addr learned my addr 172.21.15.150:0/4093704473 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:44.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa878fa6640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa85c003a10 0x7fa8741f0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:44.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa873fff640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8741f34a0 0x7fa8740c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:44.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa8797a7640 1 -- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa85c003a10 msgr2=0x7fa8741f0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:44.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa8797a7640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa85c003a10 0x7fa8741f0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:44.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa8797a7640 1 -- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8741f34a0 msgr2=0x7fa8740c32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:44.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa8797a7640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8741f34a0 0x7fa8740c32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:44.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa8797a7640 1 -- 172.21.15.150:0/4093704473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa874153240 con 0x7fa87414b7c0 2024-09-17T05:06:44.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa878fa6640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa85c003a10 0x7fa8741f0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:06:44.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa873fff640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8741f34a0 0x7fa8740c32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:06:44.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa8797a7640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa87414b7c0 0x7fa87414bbd0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fa8740c9cd0 tx=0x7fa864011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:44.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa8717fa640 1 -- 172.21.15.150:0/4093704473 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864003600 con 0x7fa87414b7c0 2024-09-17T05:06:44.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa8717fa640 1 -- 172.21.15.150:0/4093704473 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa864016070 con 0x7fa87414b7c0 2024-09-17T05:06:44.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa87b20a640 1 -- 172.21.15.150:0/4093704473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa87406f9b0 con 0x7fa87414b7c0 2024-09-17T05:06:44.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa8717fa640 1 -- 172.21.15.150:0/4093704473 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864003b30 con 0x7fa87414b7c0 2024-09-17T05:06:44.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.256+0000 7fa87b20a640 1 -- 172.21.15.150:0/4093704473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8741f2ca0 con 0x7fa87414b7c0 2024-09-17T05:06:44.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.260+0000 7fa87b20a640 1 -- 172.21.15.150:0/4093704473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8740c6230 con 0x7fa87414b7c0 2024-09-17T05:06:44.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.260+0000 7fa8717fa640 1 -- 172.21.15.150:0/4093704473 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa864003cd0 con 0x7fa87414b7c0 2024-09-17T05:06:44.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.260+0000 7fa8717fa640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa860073230 0x7fa860075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:44.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.260+0000 7fa8717fa640 1 -- 172.21.15.150:0/4093704473 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7fa864095530 con 0x7fa87414b7c0 2024-09-17T05:06:44.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.260+0000 7fa878fa6640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa860073230 0x7fa860075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:44.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.260+0000 7fa878fa6640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa860073230 0x7fa860075740 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0af000 tx=0x7fa86c01a380 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:44.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.264+0000 7fa8717fa640 1 -- 172.21.15.150:0/4093704473 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa86405fb00 con 0x7fa87414b7c0 2024-09-17T05:06:44.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.472+0000 7fa87b20a640 1 -- 172.21.15.150:0/4093704473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8740ab5f0 con 0x7fa87414b7c0 2024-09-17T05:06:44.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.472+0000 7fa8717fa640 1 -- 172.21.15.150:0/4093704473 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+6537 (secure 0 0 0) 0x7fa86405f400 con 0x7fa87414b7c0 2024-09-17T05:06:44.472 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:44.472 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":126,"btime":"2024-09-17T05:06:37:216871+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6912,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:44.473 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 126 2024-09-17T05:06:44.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.476+0000 7fa87b20a640 1 -- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa860073230 msgr2=0x7fa860075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:44.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.476+0000 7fa87b20a640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa860073230 0x7fa860075740 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0af000 tx=0x7fa86c01a380 comp rx=0 tx=0).stop 2024-09-17T05:06:44.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.476+0000 7fa87b20a640 1 -- 172.21.15.150:0/4093704473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa87414b7c0 msgr2=0x7fa87414bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:44.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.476+0000 7fa87b20a640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa87414b7c0 0x7fa87414bbd0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fa8740c9cd0 tx=0x7fa864011aa0 comp rx=0 tx=0).stop 2024-09-17T05:06:44.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.476+0000 7fa87b20a640 1 -- 172.21.15.150:0/4093704473 shutdown_connections 2024-09-17T05:06:44.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.476+0000 7fa87b20a640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa860073230 0x7fa860075740 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:44.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.476+0000 7fa87b20a640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8741f34a0 0x7fa8740c32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:44.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.476+0000 7fa87b20a640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa87414b7c0 0x7fa87414bbd0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:44.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.476+0000 7fa87b20a640 1 --2- 172.21.15.150:0/4093704473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa85c003a10 0x7fa8741f0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:44.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.476+0000 7fa87b20a640 1 -- 172.21.15.150:0/4093704473 >> 172.21.15.150:0/4093704473 conn(0x7fa8740c9de0 msgr2=0x7fa8740c9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:44.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.476+0000 7fa87b20a640 1 -- 172.21.15.150:0/4093704473 shutdown_connections 2024-09-17T05:06:44.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:44.476+0000 7fa87b20a640 1 -- 172.21.15.150:0/4093704473 wait complete. 2024-09-17T05:06:44.495 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:06:46.496 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:46.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.659+0000 7f9732082640 1 -- 172.21.15.150:0/3030332458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f972c14cec0 msgr2=0x7f972c0aa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:46.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.659+0000 7f9732082640 1 --2- 172.21.15.150:0/3030332458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f972c14cec0 0x7f972c0aa1d0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f9714009900 tx=0x7f971401c850 comp rx=0 tx=0).stop 2024-09-17T05:06:46.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.659+0000 7f9732082640 1 -- 172.21.15.150:0/3030332458 shutdown_connections 2024-09-17T05:06:46.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.659+0000 7f9732082640 1 --2- 172.21.15.150:0/3030332458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f972c0aa7b0 0x7f972c0aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:46.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.659+0000 7f9732082640 1 --2- 172.21.15.150:0/3030332458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f972c14cec0 0x7f972c0aa1d0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:46.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.659+0000 7f9732082640 1 -- 172.21.15.150:0/3030332458 >> 172.21.15.150:0/3030332458 conn(0x7f972c070b50 msgr2=0x7f972c070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:46.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.659+0000 7f9732082640 1 -- 172.21.15.150:0/3030332458 shutdown_connections 2024-09-17T05:06:46.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.659+0000 7f9732082640 1 -- 172.21.15.150:0/3030332458 wait complete. 2024-09-17T05:06:46.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9732082640 1 Processor -- start 2024-09-17T05:06:46.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9732082640 1 -- start start 2024-09-17T05:06:46.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9732082640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f972c0aa7b0 0x7f972c1f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:46.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9732082640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f972c0c4950 0x7f972c0c1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:46.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9732082640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f972c14cec0 0x7f972c13b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:46.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9732082640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f972c160040 con 0x7f972c0aa7b0 2024-09-17T05:06:46.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9732082640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f972c15fec0 con 0x7f972c14cec0 2024-09-17T05:06:46.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9732082640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f972c1601c0 con 0x7f972c0c4950 2024-09-17T05:06:46.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f972b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f972c0aa7b0 0x7f972c1f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:46.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f972b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f972c0aa7b0 0x7f972c1f3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54934/0 (socket says 172.21.15.150:54934) 2024-09-17T05:06:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f972b7fe640 1 -- 172.21.15.150:0/1775862493 learned_addr learned my addr 172.21.15.150:0/1775862493 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f972affd640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f972c0c4950 0x7f972c0c1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f972bfff640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f972c14cec0 0x7f972c13b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f972b7fe640 1 -- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f972c0c4950 msgr2=0x7f972c0c1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f972b7fe640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f972c0c4950 0x7f972c0c1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f972b7fe640 1 -- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f972c14cec0 msgr2=0x7f972c13b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f972b7fe640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f972c14cec0 0x7f972c13b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f972b7fe640 1 -- 172.21.15.150:0/1775862493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f972c14e9e0 con 0x7f972c0aa7b0 2024-09-17T05:06:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f972b7fe640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f972c0aa7b0 0x7f972c1f3bd0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f971c00c3c0 tx=0x7f971c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9723fff640 1 -- 172.21.15.150:0/1775862493 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c017070 con 0x7f972c0aa7b0 2024-09-17T05:06:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9723fff640 1 -- 172.21.15.150:0/1775862493 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f971c0053c0 con 0x7f972c0aa7b0 2024-09-17T05:06:46.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9732082640 1 -- 172.21.15.150:0/1775862493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f972c14ff50 con 0x7f972c0aa7b0 2024-09-17T05:06:46.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9723fff640 1 -- 172.21.15.150:0/1775862493 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c003860 con 0x7f972c0aa7b0 2024-09-17T05:06:46.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.663+0000 7f9732082640 1 -- 172.21.15.150:0/1775862493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f972c14fa40 con 0x7f972c0aa7b0 2024-09-17T05:06:46.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.667+0000 7f9732082640 1 -- 172.21.15.150:0/1775862493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f972c0aabc0 con 0x7f972c0aa7b0 2024-09-17T05:06:46.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.667+0000 7f9723fff640 1 -- 172.21.15.150:0/1775862493 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f971c01a070 con 0x7f972c0aa7b0 2024-09-17T05:06:46.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.667+0000 7f9723fff640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f970c073200 0x7f970c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:46.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.667+0000 7f9723fff640 1 -- 172.21.15.150:0/1775862493 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f971c094060 con 0x7f972c0aa7b0 2024-09-17T05:06:46.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.667+0000 7f972affd640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f970c073200 0x7f970c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:46.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.667+0000 7f972affd640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f970c073200 0x7f970c075710 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f97140af000 tx=0x7f971401b090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:46.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.675+0000 7f9723fff640 1 -- 172.21.15.150:0/1775862493 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f971c05e630 con 0x7f972c0aa7b0 2024-09-17T05:06:46.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.879+0000 7f9732082640 1 -- 172.21.15.150:0/1775862493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f972c071bf0 con 0x7f972c0aa7b0 2024-09-17T05:06:46.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.879+0000 7f9723fff640 1 -- 172.21.15.150:0/1775862493 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+6537 (secure 0 0 0) 0x7f971c05df30 con 0x7f972c0aa7b0 2024-09-17T05:06:46.879 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:46.879 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":126,"btime":"2024-09-17T05:06:37:216871+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6912,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:46.880 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 126 2024-09-17T05:06:46.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.883+0000 7f9732082640 1 -- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f970c073200 msgr2=0x7f970c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:46.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.883+0000 7f9732082640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f970c073200 0x7f970c075710 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f97140af000 tx=0x7f971401b090 comp rx=0 tx=0).stop 2024-09-17T05:06:46.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.883+0000 7f9732082640 1 -- 172.21.15.150:0/1775862493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f972c0aa7b0 msgr2=0x7f972c1f3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:46.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.883+0000 7f9732082640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f972c0aa7b0 0x7f972c1f3bd0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f971c00c3c0 tx=0x7f971c00c880 comp rx=0 tx=0).stop 2024-09-17T05:06:46.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.883+0000 7f9732082640 1 -- 172.21.15.150:0/1775862493 shutdown_connections 2024-09-17T05:06:46.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.883+0000 7f9732082640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f970c073200 0x7f970c075710 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:46.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.883+0000 7f9732082640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f972c14cec0 0x7f972c13b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:46.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.883+0000 7f9732082640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f972c0c4950 0x7f972c0c1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:46.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.883+0000 7f9732082640 1 --2- 172.21.15.150:0/1775862493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f972c0aa7b0 0x7f972c1f3bd0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:46.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.883+0000 7f9732082640 1 -- 172.21.15.150:0/1775862493 >> 172.21.15.150:0/1775862493 conn(0x7f972c070b50 msgr2=0x7f972c143430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:46.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.883+0000 7f9732082640 1 -- 172.21.15.150:0/1775862493 shutdown_connections 2024-09-17T05:06:46.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:46.883+0000 7f9732082640 1 -- 172.21.15.150:0/1775862493 wait complete. 2024-09-17T05:06:46.902 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:06:48.904 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:49.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.071+0000 7f138a78e640 1 -- 172.21.15.150:0/2506704324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1384074e60 msgr2=0x7f138415df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:49.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.071+0000 7f138a78e640 1 --2- 172.21.15.150:0/2506704324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1384074e60 0x7f138415df70 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f137c009bc0 tx=0x7f137c01cba0 comp rx=0 tx=0).stop 2024-09-17T05:06:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.071+0000 7f138a78e640 1 -- 172.21.15.150:0/2506704324 shutdown_connections 2024-09-17T05:06:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.071+0000 7f138a78e640 1 --2- 172.21.15.150:0/2506704324 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13840c3900 0x7f1384074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.071+0000 7f138a78e640 1 --2- 172.21.15.150:0/2506704324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1384074e60 0x7f138415df70 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.071+0000 7f138a78e640 1 -- 172.21.15.150:0/2506704324 >> 172.21.15.150:0/2506704324 conn(0x7f138406e360 msgr2=0x7f138406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.071+0000 7f138a78e640 1 -- 172.21.15.150:0/2506704324 shutdown_connections 2024-09-17T05:06:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.071+0000 7f138a78e640 1 -- 172.21.15.150:0/2506704324 wait complete. 2024-09-17T05:06:49.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.071+0000 7f138a78e640 1 Processor -- start 2024-09-17T05:06:49.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.071+0000 7f138a78e640 1 -- start start 2024-09-17T05:06:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f138a78e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1384074e60 0x7f138414f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f138a78e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13840c3900 0x7f138415dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f138a78e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840c4250 0x7f138414e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f138a78e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1384160cd0 con 0x7f13840c4250 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f138a78e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1384072fc0 con 0x7f1384074e60 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f138a78e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1384160e50 con 0x7f13840c3900 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1388d2b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840c4250 0x7f138414e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1388d2b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840c4250 0x7f138414e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54954/0 (socket says 172.21.15.150:54954) 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1388d2b640 1 -- 172.21.15.150:0/3242734104 learned_addr learned my addr 172.21.15.150:0/3242734104 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1383fff640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1384074e60 0x7f138414f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f13837fe640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13840c3900 0x7f138415dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1388d2b640 1 -- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13840c3900 msgr2=0x7f138415dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1388d2b640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13840c3900 0x7f138415dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1388d2b640 1 -- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1384074e60 msgr2=0x7f138414f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1388d2b640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1384074e60 0x7f138414f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1388d2b640 1 -- 172.21.15.150:0/3242734104 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138414fd30 con 0x7f13840c4250 2024-09-17T05:06:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1388d2b640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840c4250 0x7f138414e3b0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f137c009b10 tx=0x7f137c003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1380ff9640 1 -- 172.21.15.150:0/3242734104 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c0b2040 con 0x7f13840c4250 2024-09-17T05:06:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1380ff9640 1 -- 172.21.15.150:0/3242734104 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f137c004c50 con 0x7f13840c4250 2024-09-17T05:06:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f138a78e640 1 -- 172.21.15.150:0/3242734104 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f138414a820 con 0x7f13840c4250 2024-09-17T05:06:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1380ff9640 1 -- 172.21.15.150:0/3242734104 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c003fd0 con 0x7f13840c4250 2024-09-17T05:06:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f138a78e640 1 -- 172.21.15.150:0/3242734104 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f138414ade0 con 0x7f13840c4250 2024-09-17T05:06:49.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f138a78e640 1 -- 172.21.15.150:0/3242734104 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1384074920 con 0x7f13840c4250 2024-09-17T05:06:49.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.075+0000 7f1380ff9640 1 -- 172.21.15.150:0/3242734104 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f137c0a5d10 con 0x7f13840c4250 2024-09-17T05:06:49.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.079+0000 7f1380ff9640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1370073230 0x7f1370075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:49.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.079+0000 7f1380ff9640 1 -- 172.21.15.150:0/3242734104 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f137c12fcf0 con 0x7f13840c4250 2024-09-17T05:06:49.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.079+0000 7f1383fff640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1370073230 0x7f1370075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:49.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.079+0000 7f1383fff640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1370073230 0x7f1370075740 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f13740015f0 tx=0x7f1374009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:49.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.083+0000 7f1380ff9640 1 -- 172.21.15.150:0/3242734104 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f137c016790 con 0x7f13840c4250 2024-09-17T05:06:49.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.279+0000 7f138a78e640 1 -- 172.21.15.150:0/3242734104 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f138406e770 con 0x7f13840c4250 2024-09-17T05:06:49.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f1380ff9640 1 -- 172.21.15.150:0/3242734104 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+6537 (secure 0 0 0) 0x7f137c0fa340 con 0x7f13840c4250 2024-09-17T05:06:49.282 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:49.282 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":126,"btime":"2024-09-17T05:06:37:216871+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6912,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":126}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:05:24.217124+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6229":{"gid":6229,"name":"d","rank":0,"incarnation":112,"state":"up:active","state_seq":13,"addr":"172.21.15.176:6837/3940723333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":3940723333},{"type":"v1","addr":"172.21.15.176:6837","nonce":3940723333}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6229,"qdb_cluster":[6229]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:49.282 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 126 2024-09-17T05:06:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f138a78e640 1 -- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1370073230 msgr2=0x7f1370075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f138a78e640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1370073230 0x7f1370075740 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f13740015f0 tx=0x7f1374009040 comp rx=0 tx=0).stop 2024-09-17T05:06:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f138a78e640 1 -- 172.21.15.150:0/3242734104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840c4250 msgr2=0x7f138414e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f138a78e640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840c4250 0x7f138414e3b0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f137c009b10 tx=0x7f137c003f60 comp rx=0 tx=0).stop 2024-09-17T05:06:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f138a78e640 1 -- 172.21.15.150:0/3242734104 shutdown_connections 2024-09-17T05:06:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f138a78e640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1370073230 0x7f1370075740 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f138a78e640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840c4250 0x7f138414e3b0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f138a78e640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13840c3900 0x7f138415dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f138a78e640 1 --2- 172.21.15.150:0/3242734104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1384074e60 0x7f138414f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f138a78e640 1 -- 172.21.15.150:0/3242734104 >> 172.21.15.150:0/3242734104 conn(0x7f138406e360 msgr2=0x7f138414cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:49.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f138a78e640 1 -- 172.21.15.150:0/3242734104 shutdown_connections 2024-09-17T05:06:49.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:49.283+0000 7f138a78e640 1 -- 172.21.15.150:0/3242734104 wait complete. 2024-09-17T05:06:49.303 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:06:51.304 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:51.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.471+0000 7fcfdb325640 1 -- 172.21.15.150:0/466707708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd4074e60 msgr2=0x7fcfd415df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:51.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.471+0000 7fcfdb325640 1 --2- 172.21.15.150:0/466707708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd4074e60 0x7fcfd415df70 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4009840 tx=0x7fcfc401c7e0 comp rx=0 tx=0).stop 2024-09-17T05:06:51.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.471+0000 7fcfdb325640 1 -- 172.21.15.150:0/466707708 shutdown_connections 2024-09-17T05:06:51.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.471+0000 7fcfdb325640 1 --2- 172.21.15.150:0/466707708 >> v2:172.21.15.176:3300/0 conn(0x7fcfd40c3900 0x7fcfd4074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:51.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.471+0000 7fcfdb325640 1 --2- 172.21.15.150:0/466707708 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd4074e60 0x7fcfd415df70 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:51.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.471+0000 7fcfdb325640 1 -- 172.21.15.150:0/466707708 >> 172.21.15.150:0/466707708 conn(0x7fcfd406e360 msgr2=0x7fcfd406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:51.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.471+0000 7fcfdb325640 1 -- 172.21.15.150:0/466707708 shutdown_connections 2024-09-17T05:06:51.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.471+0000 7fcfdb325640 1 -- 172.21.15.150:0/466707708 wait complete. 2024-09-17T05:06:51.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfdb325640 1 Processor -- start 2024-09-17T05:06:51.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfdb325640 1 -- start start 2024-09-17T05:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfdb325640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfd4074e60 0x7fcfd41491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfdb325640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfd40c3900 0x7fcfd41ec450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfdb325640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd40c4250 0x7fcfd4143ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfdb325640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd4160cd0 con 0x7fcfd40c4250 2024-09-17T05:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfdb325640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd4072fc0 con 0x7fcfd4074e60 2024-09-17T05:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfdb325640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd4160e50 con 0x7fcfd40c3900 2024-09-17T05:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfd98c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd40c4250 0x7fcfd4143ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfd98c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd40c4250 0x7fcfd4143ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34152/0 (socket says 172.21.15.150:34152) 2024-09-17T05:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfd98c2640 1 -- 172.21.15.150:0/2783608850 learned_addr learned my addr 172.21.15.150:0/2783608850 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfd90c1640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfd4074e60 0x7fcfd41491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfd88c0640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfd40c3900 0x7fcfd41ec450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfd98c2640 1 -- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfd40c3900 msgr2=0x7fcfd41ec450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfd98c2640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfd40c3900 0x7fcfd41ec450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfd98c2640 1 -- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfd4074e60 msgr2=0x7fcfd41491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfd98c2640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfd4074e60 0x7fcfd41491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfd98c2640 1 -- 172.21.15.150:0/2783608850 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd40aea30 con 0x7fcfd40c4250 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfd90c1640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfd4074e60 0x7fcfd41491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfd98c2640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd40c4250 0x7fcfd4143ea0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fcfc40b0040 tx=0x7fcfc40a6690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfc9ffb640 1 -- 172.21.15.150:0/2783608850 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc40a6b70 con 0x7fcfd40c4250 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfc9ffb640 1 -- 172.21.15.150:0/2783608850 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfc40bf070 con 0x7fcfd40c4250 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfdb325640 1 -- 172.21.15.150:0/2783608850 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd415e130 con 0x7fcfd40c4250 2024-09-17T05:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfc9ffb640 1 -- 172.21.15.150:0/2783608850 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4004040 con 0x7fcfd40c4250 2024-09-17T05:06:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.475+0000 7fcfdb325640 1 -- 172.21.15.150:0/2783608850 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd4149720 con 0x7fcfd40c4250 2024-09-17T05:06:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.479+0000 7fcfdb325640 1 -- 172.21.15.150:0/2783608850 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd4074920 con 0x7fcfd40c4250 2024-09-17T05:06:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.479+0000 7fcfc9ffb640 1 -- 172.21.15.150:0/2783608850 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fcfc40041e0 con 0x7fcfd40c4250 2024-09-17T05:06:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.479+0000 7fcfc9ffb640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcfb8073200 0x7fcfb8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.479+0000 7fcfc9ffb640 1 -- 172.21.15.150:0/2783608850 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7fcfc412f740 con 0x7fcfd40c4250 2024-09-17T05:06:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.479+0000 7fcfd90c1640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcfb8073200 0x7fcfb8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.479+0000 7fcfd90c1640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcfb8073200 0x7fcfb8075710 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00012f0 tx=0x7fcfd0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:51.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.483+0000 7fcfc9ffb640 1 -- 172.21.15.150:0/2783608850 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcfc40163d0 con 0x7fcfd40c4250 2024-09-17T05:06:51.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.687+0000 7fcfdb325640 1 -- 172.21.15.150:0/2783608850 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfd40c32c0 con 0x7fcfd40c4250 2024-09-17T05:06:51.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfc9ffb640 1 -- 172.21.15.150:0/2783608850 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+5700 (secure 0 0 0) 0x7fcfc40f9b90 con 0x7fcfd40c4250 2024-09-17T05:06:51.690 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:51.690 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":127,"btime":"2024-09-17T05:06:51:084396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:06:51.084385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6912},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6912":{"gid":6912,"name":"a","rank":0,"incarnation":127,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:51.690 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 127 2024-09-17T05:06:51.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfdb325640 1 -- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcfb8073200 msgr2=0x7fcfb8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:51.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfdb325640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcfb8073200 0x7fcfb8075710 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00012f0 tx=0x7fcfd0009040 comp rx=0 tx=0).stop 2024-09-17T05:06:51.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfdb325640 1 -- 172.21.15.150:0/2783608850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd40c4250 msgr2=0x7fcfd4143ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:51.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfdb325640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd40c4250 0x7fcfd4143ea0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fcfc40b0040 tx=0x7fcfc40a6690 comp rx=0 tx=0).stop 2024-09-17T05:06:51.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfdb325640 1 -- 172.21.15.150:0/2783608850 shutdown_connections 2024-09-17T05:06:51.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfdb325640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcfb8073200 0x7fcfb8075710 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:51.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfdb325640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcfd40c4250 0x7fcfd4143ea0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:51.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfdb325640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcfd40c3900 0x7fcfd41ec450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:51.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfdb325640 1 --2- 172.21.15.150:0/2783608850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfd4074e60 0x7fcfd41491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:51.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfdb325640 1 -- 172.21.15.150:0/2783608850 >> 172.21.15.150:0/2783608850 conn(0x7fcfd406e360 msgr2=0x7fcfd4068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:51.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfdb325640 1 -- 172.21.15.150:0/2783608850 shutdown_connections 2024-09-17T05:06:51.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.691+0000 7fcfdb325640 1 -- 172.21.15.150:0/2783608850 wait complete. 2024-09-17T05:06:51.714 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-17T05:06:51.714 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:06:51.715 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:51.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.871+0000 7f2b75c20640 1 -- 172.21.15.150:0/1882453391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b700c3db0 msgr2=0x7f2b700ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:51.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.871+0000 7f2b75c20640 1 --2- 172.21.15.150:0/1882453391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b700c3db0 0x7f2b700ad2a0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f2b70072ef0 tx=0x7f2b6801cb10 comp rx=0 tx=0).stop 2024-09-17T05:06:51.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.871+0000 7f2b75c20640 1 -- 172.21.15.150:0/1882453391 shutdown_connections 2024-09-17T05:06:51.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.871+0000 7f2b75c20640 1 --2- 172.21.15.150:0/1882453391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b700c3db0 0x7f2b700ad2a0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:51.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.871+0000 7f2b75c20640 1 -- 172.21.15.150:0/1882453391 >> 172.21.15.150:0/1882453391 conn(0x7f2b700c4a00 msgr2=0x7f2b700c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:51.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.871+0000 7f2b75c20640 1 -- 172.21.15.150:0/1882453391 shutdown_connections 2024-09-17T05:06:51.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.871+0000 7f2b75c20640 1 -- 172.21.15.150:0/1882453391 wait complete. 2024-09-17T05:06:51.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.871+0000 7f2b75c20640 1 Processor -- start 2024-09-17T05:06:51.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.871+0000 7f2b75c20640 1 -- start start 2024-09-17T05:06:51.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b75c20640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b700ad7e0 0x7f2b7014b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b75c20640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b700c3db0 0x7f2b701532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b75c20640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b700c41c0 0x7f2b701f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b75c20640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2b700a7770 con 0x7f2b700c41c0 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b75c20640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2b700a75f0 con 0x7f2b700c3db0 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b75c20640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2b70162880 con 0x7f2b700ad7e0 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b6ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b700c41c0 0x7f2b701f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b6ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b700c41c0 0x7f2b701f2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34164/0 (socket says 172.21.15.150:34164) 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b6ffff640 1 -- 172.21.15.150:0/1529071319 learned_addr learned my addr 172.21.15.150:0/1529071319 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b6f7fe640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b700ad7e0 0x7f2b7014b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b6effd640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b700c3db0 0x7f2b701532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b6ffff640 1 -- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b700ad7e0 msgr2=0x7f2b7014b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b6ffff640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b700ad7e0 0x7f2b7014b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:51.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b6ffff640 1 -- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b700c3db0 msgr2=0x7f2b701532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:51.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b6ffff640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b700c3db0 0x7f2b701532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:51.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b6ffff640 1 -- 172.21.15.150:0/1529071319 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b70073570 con 0x7f2b700c41c0 2024-09-17T05:06:51.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b6ffff640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b700c41c0 0x7f2b701f2aa0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000c390 tx=0x7f2b6000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:51.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b67fff640 1 -- 172.21.15.150:0/1529071319 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60017070 con 0x7f2b700c41c0 2024-09-17T05:06:51.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b67fff640 1 -- 172.21.15.150:0/1529071319 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b60003600 con 0x7f2b700c41c0 2024-09-17T05:06:51.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b75c20640 1 -- 172.21.15.150:0/1529071319 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b70152320 con 0x7f2b700c41c0 2024-09-17T05:06:51.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b67fff640 1 -- 172.21.15.150:0/1529071319 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60003a30 con 0x7f2b700c41c0 2024-09-17T05:06:51.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b75c20640 1 -- 172.21.15.150:0/1529071319 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b701f3500 con 0x7f2b700c41c0 2024-09-17T05:06:51.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b75c20640 1 -- 172.21.15.150:0/1529071319 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b701f3760 con 0x7f2b700c41c0 2024-09-17T05:06:51.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b67fff640 1 -- 172.21.15.150:0/1529071319 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2b6001a070 con 0x7f2b700c41c0 2024-09-17T05:06:51.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.875+0000 7f2b67fff640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2b50073200 0x7f2b50075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:51.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.879+0000 7f2b67fff640 1 -- 172.21.15.150:0/1529071319 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7f2b60093f10 con 0x7f2b700c41c0 2024-09-17T05:06:51.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.879+0000 7f2b6f7fe640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2b50073200 0x7f2b50075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:51.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.879+0000 7f2b6f7fe640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2b50073200 0x7f2b50075710 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2b680af000 tx=0x7f2b6801b350 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:51.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:51.883+0000 7f2b67fff640 1 -- 172.21.15.150:0/1529071319 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2b6005e400 con 0x7f2b700c41c0 2024-09-17T05:06:52.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.091+0000 7f2b75c20640 1 -- 172.21.15.150:0/1529071319 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b700ad2a0 con 0x7f2b700c41c0 2024-09-17T05:06:52.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.091+0000 7f2b67fff640 1 -- 172.21.15.150:0/1529071319 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+5700 (secure 0 0 0) 0x7f2b6005dd00 con 0x7f2b700c41c0 2024-09-17T05:06:52.093 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:52.093 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":127,"btime":"2024-09-17T05:06:51:084396+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:06:51.084385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6912},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6912":{"gid":6912,"name":"a","rank":0,"incarnation":127,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:52.093 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 127 2024-09-17T05:06:52.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.095+0000 7f2b75c20640 1 -- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2b50073200 msgr2=0x7f2b50075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:52.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.095+0000 7f2b75c20640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2b50073200 0x7f2b50075710 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2b680af000 tx=0x7f2b6801b350 comp rx=0 tx=0).stop 2024-09-17T05:06:52.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.095+0000 7f2b75c20640 1 -- 172.21.15.150:0/1529071319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b700c41c0 msgr2=0x7f2b701f2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:52.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.095+0000 7f2b75c20640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b700c41c0 0x7f2b701f2aa0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000c390 tx=0x7f2b6000c790 comp rx=0 tx=0).stop 2024-09-17T05:06:52.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.095+0000 7f2b75c20640 1 -- 172.21.15.150:0/1529071319 shutdown_connections 2024-09-17T05:06:52.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.095+0000 7f2b75c20640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2b50073200 0x7f2b50075710 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:52.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.095+0000 7f2b75c20640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b700c3db0 0x7f2b701532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:52.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.095+0000 7f2b75c20640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2b700c41c0 0x7f2b701f2aa0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:52.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.095+0000 7f2b75c20640 1 --2- 172.21.15.150:0/1529071319 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2b700ad7e0 0x7f2b7014b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:52.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.095+0000 7f2b75c20640 1 -- 172.21.15.150:0/1529071319 >> 172.21.15.150:0/1529071319 conn(0x7f2b700c4a00 msgr2=0x7f2b700c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:52.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.095+0000 7f2b75c20640 1 -- 172.21.15.150:0/1529071319 shutdown_connections 2024-09-17T05:06:52.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:52.095+0000 7f2b75c20640 1 -- 172.21.15.150:0/1529071319 wait complete. 2024-09-17T05:06:52.118 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:06:51:084396+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 127, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 127, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6912": { "addr": "172.21.15.150:6833/523364553", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 523364553, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 523364553, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6912, "incarnation": 127, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:06:51.084385+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6912 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 125, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6831": { "addr": "172.21.15.150:6835/92695211", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 92695211, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 92695211, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6831, "incarnation": 122, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:06:34.202583+0000", "qdb_cluster": [ 6831 ], "qdb_leader": 6831, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6831 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:06:54.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:54.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.287+0000 7fa8c1efe640 1 -- 172.21.15.150:0/248847692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8a8003a60 msgr2=0x7fa8a8003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:54.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.287+0000 7fa8c1efe640 1 --2- 172.21.15.150:0/248847692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8a8003a60 0x7fa8a8003ed0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009b30 tx=0x7fa8ac01d230 comp rx=0 tx=0).stop 2024-09-17T05:06:54.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.287+0000 7fa8c1efe640 1 -- 172.21.15.150:0/248847692 shutdown_connections 2024-09-17T05:06:54.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.287+0000 7fa8c1efe640 1 --2- 172.21.15.150:0/248847692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8a8003a60 0x7fa8a8003ed0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:54.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.287+0000 7fa8c1efe640 1 -- 172.21.15.150:0/248847692 >> 172.21.15.150:0/248847692 conn(0x7fa8bc0c4a00 msgr2=0x7fa8bc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:54.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.287+0000 7fa8c1efe640 1 -- 172.21.15.150:0/248847692 shutdown_connections 2024-09-17T05:06:54.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.287+0000 7fa8c1efe640 1 -- 172.21.15.150:0/248847692 wait complete. 2024-09-17T05:06:54.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.287+0000 7fa8c1efe640 1 Processor -- start 2024-09-17T05:06:54.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8c1efe640 1 -- start start 2024-09-17T05:06:54.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8c1efe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa8a8003a60 0x7fa8bc0ad600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:54.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8c1efe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8bc1f4220 0x7fa8bc0c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:54.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8c1efe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8bc0aab30 0x7fa8bc0ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:54.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8c1efe640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc074860 con 0x7fa8bc1f4220 2024-09-17T05:06:54.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8c1efe640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc0adef0 con 0x7fa8bc0aab30 2024-09-17T05:06:54.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8c1efe640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa8bc0749e0 con 0x7fa8a8003a60 2024-09-17T05:06:54.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8baffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8bc1f4220 0x7fa8bc0c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:54.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8baffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8bc1f4220 0x7fa8bc0c1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34194/0 (socket says 172.21.15.150:34194) 2024-09-17T05:06:54.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8baffd640 1 -- 172.21.15.150:0/3582606612 learned_addr learned my addr 172.21.15.150:0/3582606612 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:54.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8bbfff640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8bc0aab30 0x7fa8bc0ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:54.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8bb7fe640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa8a8003a60 0x7fa8bc0ad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:54.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8baffd640 1 -- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa8a8003a60 msgr2=0x7fa8bc0ad600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:54.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8baffd640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa8a8003a60 0x7fa8bc0ad600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:54.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8baffd640 1 -- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8bc0aab30 msgr2=0x7fa8bc0ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:54.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8baffd640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8bc0aab30 0x7fa8bc0ab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:54.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8baffd640 1 -- 172.21.15.150:0/3582606612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8bc14ce00 con 0x7fa8bc1f4220 2024-09-17T05:06:54.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8bb7fe640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa8a8003a60 0x7fa8bc0ad600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:06:54.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8baffd640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8bc1f4220 0x7fa8bc0c1330 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400d5b0 tx=0x7fa8a400da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:54.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8b3fff640 1 -- 172.21.15.150:0/3582606612 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a40064e0 con 0x7fa8bc1f4220 2024-09-17T05:06:54.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8b3fff640 1 -- 172.21.15.150:0/3582606612 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8a4003710 con 0x7fa8bc1f4220 2024-09-17T05:06:54.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8c1efe640 1 -- 172.21.15.150:0/3582606612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8bc0ac430 con 0x7fa8bc1f4220 2024-09-17T05:06:54.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8b3fff640 1 -- 172.21.15.150:0/3582606612 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4003b60 con 0x7fa8bc1f4220 2024-09-17T05:06:54.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8c1efe640 1 -- 172.21.15.150:0/3582606612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8bc1f3740 con 0x7fa8bc1f4220 2024-09-17T05:06:54.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8c1efe640 1 -- 172.21.15.150:0/3582606612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8bc0ac600 con 0x7fa8bc1f4220 2024-09-17T05:06:54.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.291+0000 7fa8b3fff640 1 -- 172.21.15.150:0/3582606612 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa8a4003d00 con 0x7fa8bc1f4220 2024-09-17T05:06:54.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.295+0000 7fa8b3fff640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa89c0733a0 0x7fa89c0758b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:54.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.295+0000 7fa8b3fff640 1 -- 172.21.15.150:0/3582606612 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7fa8a4095190 con 0x7fa8bc1f4220 2024-09-17T05:06:54.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.295+0000 7fa8bb7fe640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa89c0733a0 0x7fa89c0758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:54.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.295+0000 7fa8bb7fe640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa89c0733a0 0x7fa89c0758b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac0ae000 tx=0x7fa8ac019fd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:54.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.299+0000 7fa8b3fff640 1 -- 172.21.15.150:0/3582606612 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa8bc0ac600 con 0x7fa8bc1f4220 2024-09-17T05:06:54.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.559+0000 7fa8c1efe640 1 -- 172.21.15.150:0/3582606612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8bc074ea0 con 0x7fa8bc1f4220 2024-09-17T05:06:54.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8b3fff640 1 -- 172.21.15.150:0/3582606612 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) ==== 95+0+5707 (secure 0 0 0) 0x7fa8a405f740 con 0x7fa8bc1f4220 2024-09-17T05:06:54.562 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:54.562 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":130,"btime":"2024-09-17T05:06:54:320185+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:06:54.320176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6912},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6912":{"gid":6912,"name":"a","rank":0,"incarnation":127,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6912,"qdb_cluster":[6912]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:54.562 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 130 2024-09-17T05:06:54.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8c1efe640 1 -- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa89c0733a0 msgr2=0x7fa89c0758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:54.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8c1efe640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa89c0733a0 0x7fa89c0758b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac0ae000 tx=0x7fa8ac019fd0 comp rx=0 tx=0).stop 2024-09-17T05:06:54.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8c1efe640 1 -- 172.21.15.150:0/3582606612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8bc1f4220 msgr2=0x7fa8bc0c1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:54.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8c1efe640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8bc1f4220 0x7fa8bc0c1330 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400d5b0 tx=0x7fa8a400da70 comp rx=0 tx=0).stop 2024-09-17T05:06:54.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8c1efe640 1 -- 172.21.15.150:0/3582606612 shutdown_connections 2024-09-17T05:06:54.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8c1efe640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa89c0733a0 0x7fa89c0758b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:54.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8c1efe640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8bc0aab30 0x7fa8bc0ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:54.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8c1efe640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8bc1f4220 0x7fa8bc0c1330 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:54.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8c1efe640 1 --2- 172.21.15.150:0/3582606612 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa8a8003a60 0x7fa8bc0ad600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:54.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8c1efe640 1 -- 172.21.15.150:0/3582606612 >> 172.21.15.150:0/3582606612 conn(0x7fa8bc0c4a00 msgr2=0x7fa8bc0a8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:54.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8c1efe640 1 -- 172.21.15.150:0/3582606612 shutdown_connections 2024-09-17T05:06:54.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:54.563+0000 7fa8c1efe640 1 -- 172.21.15.150:0/3582606612 wait complete. 2024-09-17T05:06:54.586 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6229 2024-09-17T05:06:54.586 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.d 2024-09-17T05:06:57.227 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T05:06:57.227+0000 7f8a9b913640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:06:57.587 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-17T05:06:57.587 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T05:06:57.587 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T05:06:57.590 INFO:tasks.ceph.mds.d:Started 2024-09-17T05:06:57.590 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:06:57.709 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-09-17T05:06:57.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.775+0000 7ff288e56640 1 -- 172.21.15.150:0/1873565559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2840c4c50 msgr2=0x7ff2840741b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:57.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.775+0000 7ff288e56640 1 --2- 172.21.15.150:0/1873565559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2840c4c50 0x7ff2840741b0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ff284073570 tx=0x7ff27c01ccf0 comp rx=0 tx=0).stop 2024-09-17T05:06:57.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.775+0000 7ff288e56640 1 -- 172.21.15.150:0/1873565559 shutdown_connections 2024-09-17T05:06:57.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.775+0000 7ff288e56640 1 --2- 172.21.15.150:0/1873565559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2840c4c50 0x7ff2840741b0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:57.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.775+0000 7ff288e56640 1 --2- 172.21.15.150:0/1873565559 >> v2:172.21.15.176:3300/0 conn(0x7ff28406db80 0x7ff2841537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:57.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.775+0000 7ff288e56640 1 -- 172.21.15.150:0/1873565559 >> 172.21.15.150:0/1873565559 conn(0x7ff2840c91b0 msgr2=0x7ff2840c5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:57.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.775+0000 7ff288e56640 1 -- 172.21.15.150:0/1873565559 shutdown_connections 2024-09-17T05:06:57.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.775+0000 7ff288e56640 1 -- 172.21.15.150:0/1873565559 wait complete. 2024-09-17T05:06:57.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.775+0000 7ff288e56640 1 Processor -- start 2024-09-17T05:06:57.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.775+0000 7ff288e56640 1 -- start start 2024-09-17T05:06:57.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff288e56640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28406db80 0x7ff2840c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:57.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff288e56640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff284073da0 0x7ff284143530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:57.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff288e56640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2840c4c50 0x7ff284141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:57.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff288e56640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff28406e450 con 0x7ff284073da0 2024-09-17T05:06:57.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff288e56640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff2840c2d60 con 0x7ff28406db80 2024-09-17T05:06:57.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff288e56640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff28406e5d0 con 0x7ff2840c4c50 2024-09-17T05:06:57.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff281d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff284073da0 0x7ff284143530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:57.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff28259c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28406db80 0x7ff2840c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:57.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff281d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff284073da0 0x7ff284143530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34208/0 (socket says 172.21.15.150:34208) 2024-09-17T05:06:57.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff281d9b640 1 -- 172.21.15.150:0/608249037 learned_addr learned my addr 172.21.15.150:0/608249037 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:06:57.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff282d9d640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2840c4c50 0x7ff284141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:57.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff281d9b640 1 -- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2840c4c50 msgr2=0x7ff284141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:57.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff281d9b640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2840c4c50 0x7ff284141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:57.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff281d9b640 1 -- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28406db80 msgr2=0x7ff2840c1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:57.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff281d9b640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28406db80 0x7ff2840c1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:57.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff281d9b640 1 -- 172.21.15.150:0/608249037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2840ac270 con 0x7ff284073da0 2024-09-17T05:06:57.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff281d9b640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff284073da0 0x7ff284143530 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ff27800b410 tx=0x7ff27800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:57.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff272ffd640 1 -- 172.21.15.150:0/608249037 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278015030 con 0x7ff284073da0 2024-09-17T05:06:57.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff288e56640 1 -- 172.21.15.150:0/608249037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff284141c30 con 0x7ff284073da0 2024-09-17T05:06:57.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff272ffd640 1 -- 172.21.15.150:0/608249037 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff278004020 con 0x7ff284073da0 2024-09-17T05:06:57.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff272ffd640 1 -- 172.21.15.150:0/608249037 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278004470 con 0x7ff284073da0 2024-09-17T05:06:57.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff288e56640 1 -- 172.21.15.150:0/608249037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff284142300 con 0x7ff284073da0 2024-09-17T05:06:57.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff288e56640 1 -- 172.21.15.150:0/608249037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2841537f0 con 0x7ff284073da0 2024-09-17T05:06:57.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.779+0000 7ff272ffd640 1 -- 172.21.15.150:0/608249037 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff278018070 con 0x7ff284073da0 2024-09-17T05:06:57.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.783+0000 7ff272ffd640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff264073230 0x7ff264075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:06:57.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.783+0000 7ff272ffd640 1 -- 172.21.15.150:0/608249037 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7ff2780947a0 con 0x7ff284073da0 2024-09-17T05:06:57.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.783+0000 7ff28259c640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff264073230 0x7ff264075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:06:57.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.783+0000 7ff28259c640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff264073230 0x7ff264075740 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff27400ab40 tx=0x7ff27400a250 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:06:57.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:57.787+0000 7ff272ffd640 1 -- 172.21.15.150:0/608249037 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff27805e640 con 0x7ff284073da0 2024-09-17T05:06:58.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.071+0000 7ff288e56640 1 -- 172.21.15.150:0/608249037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff284142b50 con 0x7ff284073da0 2024-09-17T05:06:58.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.071+0000 7ff272ffd640 1 -- 172.21.15.150:0/608249037 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) ==== 95+0+5707 (secure 0 0 0) 0x7ff278063070 con 0x7ff284073da0 2024-09-17T05:06:58.071 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:06:58.072 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":130,"btime":"2024-09-17T05:06:54:320185+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:06:54.320176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6912},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6912":{"gid":6912,"name":"a","rank":0,"incarnation":127,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6912,"qdb_cluster":[6912]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:06:58.072 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 130 2024-09-17T05:06:58.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.075+0000 7ff288e56640 1 -- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff264073230 msgr2=0x7ff264075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:58.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.075+0000 7ff288e56640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff264073230 0x7ff264075740 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff27400ab40 tx=0x7ff27400a250 comp rx=0 tx=0).stop 2024-09-17T05:06:58.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.075+0000 7ff288e56640 1 -- 172.21.15.150:0/608249037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff284073da0 msgr2=0x7ff284143530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:06:58.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.075+0000 7ff288e56640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff284073da0 0x7ff284143530 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ff27800b410 tx=0x7ff27800b8d0 comp rx=0 tx=0).stop 2024-09-17T05:06:58.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.075+0000 7ff288e56640 1 -- 172.21.15.150:0/608249037 shutdown_connections 2024-09-17T05:06:58.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.075+0000 7ff288e56640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff264073230 0x7ff264075740 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:58.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.075+0000 7ff288e56640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2840c4c50 0x7ff284141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:58.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.075+0000 7ff288e56640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff284073da0 0x7ff284143530 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:58.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.075+0000 7ff288e56640 1 --2- 172.21.15.150:0/608249037 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff28406db80 0x7ff2840c1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:06:58.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.075+0000 7ff288e56640 1 -- 172.21.15.150:0/608249037 >> 172.21.15.150:0/608249037 conn(0x7ff2840c91b0 msgr2=0x7ff2840c6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:06:58.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.075+0000 7ff288e56640 1 -- 172.21.15.150:0/608249037 shutdown_connections 2024-09-17T05:06:58.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:06:58.075+0000 7ff288e56640 1 -- 172.21.15.150:0/608249037 wait complete. 2024-09-17T05:06:58.098 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-17T05:07:00.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:00.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.271+0000 7f4a72720640 1 -- 172.21.15.150:0/744449877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a6c06fca0 msgr2=0x7f4a6c153790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:00.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.271+0000 7f4a72720640 1 --2- 172.21.15.150:0/744449877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a6c06fca0 0x7f4a6c153790 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009840 tx=0x7f4a5c01c790 comp rx=0 tx=0).stop 2024-09-17T05:07:00.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.271+0000 7f4a72720640 1 -- 172.21.15.150:0/744449877 shutdown_connections 2024-09-17T05:07:00.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.271+0000 7f4a72720640 1 --2- 172.21.15.150:0/744449877 >> v2:172.21.15.176:3300/0 conn(0x7f4a6c074570 0x7f4a6c0aaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:00.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.271+0000 7f4a72720640 1 --2- 172.21.15.150:0/744449877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a6c06fca0 0x7f4a6c153790 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:00.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.271+0000 7f4a72720640 1 -- 172.21.15.150:0/744449877 >> 172.21.15.150:0/744449877 conn(0x7f4a6c0c5ae0 msgr2=0x7f4a6c073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:00.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.271+0000 7f4a72720640 1 -- 172.21.15.150:0/744449877 shutdown_connections 2024-09-17T05:07:00.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.271+0000 7f4a72720640 1 -- 172.21.15.150:0/744449877 wait complete. 2024-09-17T05:07:00.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.271+0000 7f4a72720640 1 Processor -- start 2024-09-17T05:07:00.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.271+0000 7f4a72720640 1 -- start start 2024-09-17T05:07:00.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a72720640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c06fca0 0x7f4a6c1eb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:00.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a72720640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a6c074570 0x7f4a6c0ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:00.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a72720640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a6c074980 0x7f4a6c155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:00.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a72720640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c0c4c50 con 0x7f4a6c06fca0 2024-09-17T05:07:00.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a72720640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6c0c4ad0 con 0x7f4a6c074570 2024-09-17T05:07:00.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a72720640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4a6c15f6a0 con 0x7f4a6c074980 2024-09-17T05:07:00.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a6bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c06fca0 0x7f4a6c1eb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:00.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a6bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c06fca0 0x7f4a6c1eb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36714/0 (socket says 172.21.15.150:36714) 2024-09-17T05:07:00.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a6bfff640 1 -- 172.21.15.150:0/3511148490 learned_addr learned my addr 172.21.15.150:0/3511148490 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:00.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a70cbd640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a6c074980 0x7f4a6c155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:00.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a6b7fe640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a6c074570 0x7f4a6c0ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:00.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a6bfff640 1 -- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a6c074980 msgr2=0x7f4a6c155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:00.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a6bfff640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a6c074980 0x7f4a6c155510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:00.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a6bfff640 1 -- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a6c074570 msgr2=0x7f4a6c0ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:00.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a6bfff640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a6c074570 0x7f4a6c0ab9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:00.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a6bfff640 1 -- 172.21.15.150:0/3511148490 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a6c0ac5c0 con 0x7f4a6c06fca0 2024-09-17T05:07:00.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a6bfff640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c06fca0 0x7f4a6c1eb860 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c01ae40 tx=0x7f4a5c005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:00.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a68ff9640 1 -- 172.21.15.150:0/3511148490 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c0b2040 con 0x7f4a6c06fca0 2024-09-17T05:07:00.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a72720640 1 -- 172.21.15.150:0/3511148490 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a6c0ac2e0 con 0x7f4a6c06fca0 2024-09-17T05:07:00.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a68ff9640 1 -- 172.21.15.150:0/3511148490 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a5c0042f0 con 0x7f4a6c06fca0 2024-09-17T05:07:00.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a68ff9640 1 -- 172.21.15.150:0/3511148490 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c0052d0 con 0x7f4a6c06fca0 2024-09-17T05:07:00.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a72720640 1 -- 172.21.15.150:0/3511148490 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a6c0acc70 con 0x7f4a6c06fca0 2024-09-17T05:07:00.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a72720640 1 -- 172.21.15.150:0/3511148490 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a6c153790 con 0x7f4a6c06fca0 2024-09-17T05:07:00.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a68ff9640 1 -- 172.21.15.150:0/3511148490 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4a5c005470 con 0x7f4a6c06fca0 2024-09-17T05:07:00.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.275+0000 7f4a68ff9640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a58073200 0x7f4a58075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:00.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.279+0000 7f4a68ff9640 1 -- 172.21.15.150:0/3511148490 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7f4a5c12eab0 con 0x7f4a6c06fca0 2024-09-17T05:07:00.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.279+0000 7f4a6b7fe640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a58073200 0x7f4a58075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:00.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.279+0000 7f4a6b7fe640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a58073200 0x7f4a58075710 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000a710 tx=0x7f4a60010040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:00.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.283+0000 7f4a68ff9640 1 -- 172.21.15.150:0/3511148490 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4a5c016410 con 0x7f4a6c06fca0 2024-09-17T05:07:00.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.519+0000 7f4a72720640 1 -- 172.21.15.150:0/3511148490 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a6c069200 con 0x7f4a6c06fca0 2024-09-17T05:07:00.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a68ff9640 1 -- 172.21.15.150:0/3511148490 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+6533 (secure 0 0 0) 0x7f4a5c0f8950 con 0x7f4a6c06fca0 2024-09-17T05:07:00.522 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:00.522 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":131,"btime":"2024-09-17T05:06:58:539213+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5945,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:06:54.320176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6912},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6912":{"gid":6912,"name":"a","rank":0,"incarnation":127,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6912,"qdb_cluster":[6912]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:00.522 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 131 2024-09-17T05:07:00.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a72720640 1 -- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a58073200 msgr2=0x7f4a58075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:00.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a72720640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a58073200 0x7f4a58075710 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000a710 tx=0x7f4a60010040 comp rx=0 tx=0).stop 2024-09-17T05:07:00.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a72720640 1 -- 172.21.15.150:0/3511148490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c06fca0 msgr2=0x7f4a6c1eb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:00.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a72720640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c06fca0 0x7f4a6c1eb860 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c01ae40 tx=0x7f4a5c005d90 comp rx=0 tx=0).stop 2024-09-17T05:07:00.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a72720640 1 -- 172.21.15.150:0/3511148490 shutdown_connections 2024-09-17T05:07:00.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a72720640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a58073200 0x7f4a58075710 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:00.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a72720640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a6c074570 0x7f4a6c0ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:00.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a72720640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a6c074980 0x7f4a6c155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:00.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a72720640 1 --2- 172.21.15.150:0/3511148490 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a6c06fca0 0x7f4a6c1eb860 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:00.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a72720640 1 -- 172.21.15.150:0/3511148490 >> 172.21.15.150:0/3511148490 conn(0x7f4a6c0c5ae0 msgr2=0x7f4a6c0c64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:00.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a72720640 1 -- 172.21.15.150:0/3511148490 shutdown_connections 2024-09-17T05:07:00.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:00.523+0000 7f4a72720640 1 -- 172.21.15.150:0/3511148490 wait complete. 2024-09-17T05:07:00.546 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-17T05:07:00.546 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before thrashing 2024-09-17T05:07:02.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:07:02.617 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:07:08.547 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.711+0000 7f708c122640 1 -- 172.21.15.150:0/3504302743 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70840c5140 msgr2=0x7f70840748d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.711+0000 7f708c122640 1 --2- 172.21.15.150:0/3504302743 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70840c5140 0x7f70840748d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f70840c46e0 tx=0x7f707401c700 comp rx=0 tx=0).stop 2024-09-17T05:07:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.711+0000 7f708c122640 1 -- 172.21.15.150:0/3504302743 shutdown_connections 2024-09-17T05:07:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.711+0000 7f708c122640 1 --2- 172.21.15.150:0/3504302743 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70840c5140 0x7f70840748d0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.711+0000 7f708c122640 1 -- 172.21.15.150:0/3504302743 >> 172.21.15.150:0/3504302743 conn(0x7f708406e360 msgr2=0x7f708406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.711+0000 7f708c122640 1 -- 172.21.15.150:0/3504302743 shutdown_connections 2024-09-17T05:07:08.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.711+0000 7f708c122640 1 -- 172.21.15.150:0/3504302743 wait complete. 2024-09-17T05:07:08.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.715+0000 7f708c122640 1 Processor -- start 2024-09-17T05:07:08.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.715+0000 7f708c122640 1 -- start start 2024-09-17T05:07:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.715+0000 7f708c122640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7084074e10 0x7f708414ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.715+0000 7f708c122640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70840c3740 0x7f7084152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.715+0000 7f708c122640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70840c5140 0x7f70840be4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.715+0000 7f708c122640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7084162990 con 0x7f70840c3740 2024-09-17T05:07:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.715+0000 7f708c122640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f70840c7b50 con 0x7f70840c5140 2024-09-17T05:07:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.715+0000 7f708c122640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7084162b10 con 0x7f7084074e10 2024-09-17T05:07:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.715+0000 7f70896bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70840c3740 0x7f7084152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.715+0000 7f70896bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70840c3740 0x7f7084152fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36738/0 (socket says 172.21.15.150:36738) 2024-09-17T05:07:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.715+0000 7f70896bd640 1 -- 172.21.15.150:0/222841299 learned_addr learned my addr 172.21.15.150:0/222841299 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f7089ebe640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7084074e10 0x7f708414ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f708a6bf640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70840c5140 0x7f70840be4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:08.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f70896bd640 1 -- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7084074e10 msgr2=0x7f708414ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:08.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f70896bd640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7084074e10 0x7f708414ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:08.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f70896bd640 1 -- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70840c5140 msgr2=0x7f70840be4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:08.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f70896bd640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70840c5140 0x7f70840be4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:08.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f70896bd640 1 -- 172.21.15.150:0/222841299 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70840bdd60 con 0x7f70840c3740 2024-09-17T05:07:08.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f7089ebe640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7084074e10 0x7f708414ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:07:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f70896bd640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70840c3740 0x7f7084152fc0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f707401cbe0 tx=0x7f70740a67b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f707a7fc640 1 -- 172.21.15.150:0/222841299 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70740bd020 con 0x7f70840c3740 2024-09-17T05:07:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f708c122640 1 -- 172.21.15.150:0/222841299 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7084151cb0 con 0x7f70840c3740 2024-09-17T05:07:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f707a7fc640 1 -- 172.21.15.150:0/222841299 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70740044b0 con 0x7f70840c3740 2024-09-17T05:07:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f707a7fc640 1 -- 172.21.15.150:0/222841299 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7074003820 con 0x7f70840c3740 2024-09-17T05:07:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f708c122640 1 -- 172.21.15.150:0/222841299 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7084151f90 con 0x7f70840c3740 2024-09-17T05:07:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f708c122640 1 -- 172.21.15.150:0/222841299 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70840748d0 con 0x7f70840c3740 2024-09-17T05:07:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.719+0000 7f707a7fc640 1 -- 172.21.15.150:0/222841299 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f70740c0020 con 0x7f70840c3740 2024-09-17T05:07:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.723+0000 7f707a7fc640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7068073200 0x7f7068075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.723+0000 7f707a7fc640 1 -- 172.21.15.150:0/222841299 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7f70741305d0 con 0x7f70840c3740 2024-09-17T05:07:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.723+0000 7f7089ebe640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7068073200 0x7f7068075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.723+0000 7f7089ebe640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7068073200 0x7f7068075710 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7080000f70 tx=0x7f7080009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.727+0000 7f707a7fc640 1 -- 172.21.15.150:0/222841299 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f70740c7030 con 0x7f70840c3740 2024-09-17T05:07:08.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 -- 172.21.15.150:0/222841299 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70840c3420 con 0x7f70840c3740 2024-09-17T05:07:08.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f707a7fc640 1 -- 172.21.15.150:0/222841299 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+6533 (secure 0 0 0) 0x7f7074016410 con 0x7f70840c3740 2024-09-17T05:07:08.950 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:08.950 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":131,"btime":"2024-09-17T05:06:58:539213+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5945,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:06:54.320176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6912},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6912":{"gid":6912,"name":"a","rank":0,"incarnation":127,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6912,"qdb_cluster":[6912]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:08.951 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 131 2024-09-17T05:07:08.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 -- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7068073200 msgr2=0x7f7068075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:08.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7068073200 0x7f7068075710 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7080000f70 tx=0x7f7080009040 comp rx=0 tx=0).stop 2024-09-17T05:07:08.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 -- 172.21.15.150:0/222841299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70840c3740 msgr2=0x7f7084152fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:08.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70840c3740 0x7f7084152fc0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f707401cbe0 tx=0x7f70740a67b0 comp rx=0 tx=0).stop 2024-09-17T05:07:08.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 -- 172.21.15.150:0/222841299 shutdown_connections 2024-09-17T05:07:08.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7068073200 0x7f7068075710 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:08.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70840c5140 0x7f70840be4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:08.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70840c3740 0x7f7084152fc0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:08.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 --2- 172.21.15.150:0/222841299 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7084074e10 0x7f708414ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:08.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 -- 172.21.15.150:0/222841299 >> 172.21.15.150:0/222841299 conn(0x7f708406e360 msgr2=0x7f708406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:08.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 -- 172.21.15.150:0/222841299 shutdown_connections 2024-09-17T05:07:08.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:08.951+0000 7f708c122640 1 -- 172.21.15.150:0/222841299 wait complete. 2024-09-17T05:07:08.976 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-17T05:07:08.976 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T05:07:08.976 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:07:08.989 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T05:07:08.989 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:09.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.199+0000 7f1a13391640 1 -- 172.21.15.150:0/1138364685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19f8004810 msgr2=0x7f19f8004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:09.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.199+0000 7f1a13391640 1 --2- 172.21.15.150:0/1138364685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19f8004810 0x7f19f8004c20 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0c74e0 tx=0x7f19fc01d250 comp rx=0 tx=0).stop 2024-09-17T05:07:09.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.199+0000 7f1a1192e640 1 -- 172.21.15.150:0/1138364685 reap_dead start 2024-09-17T05:07:09.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.199+0000 7f1a13391640 1 -- 172.21.15.150:0/1138364685 shutdown_connections 2024-09-17T05:07:09.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.199+0000 7f1a13391640 1 -- 172.21.15.150:0/1138364685 >> 172.21.15.150:0/1138364685 conn(0x7f1a0c0ca460 msgr2=0x7f1a0c0c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:09.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.199+0000 7f1a13391640 1 -- 172.21.15.150:0/1138364685 shutdown_connections 2024-09-17T05:07:09.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.199+0000 7f1a13391640 1 -- 172.21.15.150:0/1138364685 wait complete. 2024-09-17T05:07:09.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.199+0000 7f1a13391640 1 Processor -- start 2024-09-17T05:07:09.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.199+0000 7f1a13391640 1 -- start start 2024-09-17T05:07:09.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.199+0000 7f1a13391640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c1df210 0x7f1a0c149ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a13391640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a0c143720 0x7f1a0c143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a13391640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c1ee260 0x7f1a0c1ee670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a13391640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0c0c62d0 con 0x7f1a0c143720 2024-09-17T05:07:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a13391640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0c158a10 con 0x7f1a0c1df210 2024-09-17T05:07:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a13391640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1a0c0c65c0 con 0x7f1a0c1ee260 2024-09-17T05:07:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a1092c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a0c143720 0x7f1a0c143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a1092c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a0c143720 0x7f1a0c143b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36764/0 (socket says 172.21.15.150:36764) 2024-09-17T05:07:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a1092c640 1 -- 172.21.15.150:0/110038298 learned_addr learned my addr 172.21.15.150:0/110038298 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a1112d640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c1df210 0x7f1a0c149ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a1192e640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c1ee260 0x7f1a0c1ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a1092c640 1 -- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c1ee260 msgr2=0x7f1a0c1ee670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a1092c640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c1ee260 0x7f1a0c1ee670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:09.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a1092c640 1 -- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c1df210 msgr2=0x7f1a0c149ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:09.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a1092c640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c1df210 0x7f1a0c149ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:09.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a1092c640 1 -- 172.21.15.150:0/110038298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a0c1de640 con 0x7f1a0c143720 2024-09-17T05:07:09.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a1092c640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a0c143720 0x7f1a0c143b30 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009a80 tx=0x7f19fc0055d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:09.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a01ffb640 1 -- 172.21.15.150:0/110038298 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc0b1040 con 0x7f1a0c143720 2024-09-17T05:07:09.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a01ffb640 1 -- 172.21.15.150:0/110038298 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f19fc0ac400 con 0x7f1a0c143720 2024-09-17T05:07:09.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a13391640 1 -- 172.21.15.150:0/110038298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a0c144070 con 0x7f1a0c143720 2024-09-17T05:07:09.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a01ffb640 1 -- 172.21.15.150:0/110038298 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc004290 con 0x7f1a0c143720 2024-09-17T05:07:09.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a13391640 1 -- 172.21.15.150:0/110038298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a0c1ddda0 con 0x7f1a0c143720 2024-09-17T05:07:09.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a13391640 1 -- 172.21.15.150:0/110038298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a0c1edeb0 con 0x7f1a0c143720 2024-09-17T05:07:09.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a01ffb640 1 -- 172.21.15.150:0/110038298 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f19fc0bd020 con 0x7f1a0c143720 2024-09-17T05:07:09.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a01ffb640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f19f0073200 0x7f19f0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:09.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a01ffb640 1 -- 172.21.15.150:0/110038298 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7f19fc12f3c0 con 0x7f1a0c143720 2024-09-17T05:07:09.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.203+0000 7f1a1112d640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f19f0073200 0x7f19f0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:09.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.207+0000 7f1a1112d640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f19f0073200 0x7f19f0075710 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c1e8480 tx=0x7f1a080091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:09.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.211+0000 7f1a01ffb640 1 -- 172.21.15.150:0/110038298 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1a0c1edeb0 con 0x7f1a0c143720 2024-09-17T05:07:09.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 -- 172.21.15.150:0/110038298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a0c14d3d0 con 0x7f1a0c143720 2024-09-17T05:07:09.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a01ffb640 1 -- 172.21.15.150:0/110038298 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+6533 (secure 0 0 0) 0x7f19fc0fee20 con 0x7f1a0c143720 2024-09-17T05:07:09.450 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:09.451 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":131,"btime":"2024-09-17T05:06:58:539213+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5945,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:06:54.320176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6912},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6912":{"gid":6912,"name":"a","rank":0,"incarnation":127,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6912,"qdb_cluster":[6912]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:09.451 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 131 2024-09-17T05:07:09.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 -- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f19f0073200 msgr2=0x7f19f0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:09.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f19f0073200 0x7f19f0075710 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c1e8480 tx=0x7f1a080091e0 comp rx=0 tx=0).stop 2024-09-17T05:07:09.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 -- 172.21.15.150:0/110038298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a0c143720 msgr2=0x7f1a0c143b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:09.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a0c143720 0x7f1a0c143b30 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009a80 tx=0x7f19fc0055d0 comp rx=0 tx=0).stop 2024-09-17T05:07:09.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 -- 172.21.15.150:0/110038298 shutdown_connections 2024-09-17T05:07:09.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f19f0073200 0x7f19f0075710 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:09.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a0c1ee260 0x7f1a0c1ee670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:09.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a0c143720 0x7f1a0c143b30 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:09.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 --2- 172.21.15.150:0/110038298 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a0c1df210 0x7f1a0c149ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:09.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 -- 172.21.15.150:0/110038298 >> 172.21.15.150:0/110038298 conn(0x7f1a0c0ca460 msgr2=0x7f1a0c0c46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:09.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 -- 172.21.15.150:0/110038298 shutdown_connections 2024-09-17T05:07:09.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:09.451+0000 7f1a13391640 1 -- 172.21.15.150:0/110038298 wait complete. 2024-09-17T05:07:09.479 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:07:11.479 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:11.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.651+0000 7ff487566640 1 -- 172.21.15.150:0/1203611144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4800c2780 msgr2=0x7ff4800c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:11.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.651+0000 7ff487566640 1 --2- 172.21.15.150:0/1203611144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4800c2780 0x7ff4800c8a80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff478009900 tx=0x7ff47801c8e0 comp rx=0 tx=0).stop 2024-09-17T05:07:11.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.651+0000 7ff487566640 1 -- 172.21.15.150:0/1203611144 shutdown_connections 2024-09-17T05:07:11.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.651+0000 7ff487566640 1 --2- 172.21.15.150:0/1203611144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4800c2780 0x7ff4800c8a80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:11.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.651+0000 7ff487566640 1 --2- 172.21.15.150:0/1203611144 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4800c69f0 0x7ff4800c2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:11.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.651+0000 7ff487566640 1 -- 172.21.15.150:0/1203611144 >> 172.21.15.150:0/1203611144 conn(0x7ff480074b80 msgr2=0x7ff48006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:11.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.651+0000 7ff487566640 1 -- 172.21.15.150:0/1203611144 shutdown_connections 2024-09-17T05:07:11.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.651+0000 7ff487566640 1 -- 172.21.15.150:0/1203611144 wait complete. 2024-09-17T05:07:11.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff487566640 1 Processor -- start 2024-09-17T05:07:11.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff487566640 1 -- start start 2024-09-17T05:07:11.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff487566640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4800c2780 0x7ff480153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:11.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff487566640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4800c69f0 0x7ff480152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:11.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff487566640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff480156820 0x7ff480149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff487566640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff4800c5790 con 0x7ff480156820 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff487566640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff48006ff90 con 0x7ff4800c69f0 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff487566640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff4800c5910 con 0x7ff4800c2780 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff485b03640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff480156820 0x7ff480149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff485b03640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff480156820 0x7ff480149df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56892/0 (socket says 172.21.15.150:56892) 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff484b01640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4800c69f0 0x7ff480152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff485b03640 1 -- 172.21.15.150:0/1462818668 learned_addr learned my addr 172.21.15.150:0/1462818668 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff485302640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4800c2780 0x7ff480153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff485b03640 1 -- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4800c2780 msgr2=0x7ff480153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff485b03640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4800c2780 0x7ff480153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff485b03640 1 -- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4800c69f0 msgr2=0x7ff480152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff485b03640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4800c69f0 0x7ff480152c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff485b03640 1 -- 172.21.15.150:0/1462818668 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4801490d0 con 0x7ff480156820 2024-09-17T05:07:11.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff485b03640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff480156820 0x7ff480149df0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7ff4780b0040 tx=0x7ff478005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:11.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff475ffb640 1 -- 172.21.15.150:0/1462818668 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4780b2040 con 0x7ff480156820 2024-09-17T05:07:11.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff475ffb640 1 -- 172.21.15.150:0/1462818668 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff478004330 con 0x7ff480156820 2024-09-17T05:07:11.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff487566640 1 -- 172.21.15.150:0/1462818668 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff480149410 con 0x7ff480156820 2024-09-17T05:07:11.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff475ffb640 1 -- 172.21.15.150:0/1462818668 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff478003860 con 0x7ff480156820 2024-09-17T05:07:11.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.655+0000 7ff487566640 1 -- 172.21.15.150:0/1462818668 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4801f0f40 con 0x7ff480156820 2024-09-17T05:07:11.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.659+0000 7ff487566640 1 -- 172.21.15.150:0/1462818668 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4800c8a80 con 0x7ff480156820 2024-09-17T05:07:11.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.659+0000 7ff475ffb640 1 -- 172.21.15.150:0/1462818668 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff4780bf060 con 0x7ff480156820 2024-09-17T05:07:11.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.659+0000 7ff475ffb640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff464073200 0x7ff464075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:11.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.659+0000 7ff475ffb640 1 -- 172.21.15.150:0/1462818668 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7ff47812e920 con 0x7ff480156820 2024-09-17T05:07:11.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.659+0000 7ff485302640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff464073200 0x7ff464075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:11.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.659+0000 7ff485302640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff464073200 0x7ff464075710 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff47c001e40 tx=0x7ff47c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:11.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.663+0000 7ff475ffb640 1 -- 172.21.15.150:0/1462818668 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff4780164d0 con 0x7ff480156820 2024-09-17T05:07:11.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.879+0000 7ff487566640 1 -- 172.21.15.150:0/1462818668 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff480149610 con 0x7ff480156820 2024-09-17T05:07:11.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.879+0000 7ff475ffb640 1 -- 172.21.15.150:0/1462818668 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+6533 (secure 0 0 0) 0x7ff4780f8840 con 0x7ff480156820 2024-09-17T05:07:11.881 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:11.881 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":131,"btime":"2024-09-17T05:06:58:539213+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5945,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:06:54.320176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6912},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6912":{"gid":6912,"name":"a","rank":0,"incarnation":127,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6912,"qdb_cluster":[6912]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:11.881 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 131 2024-09-17T05:07:11.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.883+0000 7ff487566640 1 -- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff464073200 msgr2=0x7ff464075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:11.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.883+0000 7ff487566640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff464073200 0x7ff464075710 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff47c001e40 tx=0x7ff47c009040 comp rx=0 tx=0).stop 2024-09-17T05:07:11.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.883+0000 7ff487566640 1 -- 172.21.15.150:0/1462818668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff480156820 msgr2=0x7ff480149df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:11.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.883+0000 7ff487566640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff480156820 0x7ff480149df0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7ff4780b0040 tx=0x7ff478005de0 comp rx=0 tx=0).stop 2024-09-17T05:07:11.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.883+0000 7ff487566640 1 -- 172.21.15.150:0/1462818668 shutdown_connections 2024-09-17T05:07:11.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.883+0000 7ff487566640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff464073200 0x7ff464075710 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:11.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.883+0000 7ff487566640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4800c69f0 0x7ff480152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:11.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.883+0000 7ff487566640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff480156820 0x7ff480149df0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:11.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.883+0000 7ff487566640 1 --2- 172.21.15.150:0/1462818668 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4800c2780 0x7ff480153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:11.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.883+0000 7ff487566640 1 -- 172.21.15.150:0/1462818668 >> 172.21.15.150:0/1462818668 conn(0x7ff480074b80 msgr2=0x7ff480154900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:11.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.883+0000 7ff487566640 1 -- 172.21.15.150:0/1462818668 shutdown_connections 2024-09-17T05:07:11.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:11.883+0000 7ff487566640 1 -- 172.21.15.150:0/1462818668 wait complete. 2024-09-17T05:07:11.902 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:07:13.903 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:14.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.067+0000 7fc56548c640 1 -- 172.21.15.150:0/2566549217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5600ad7e0 msgr2=0x7fc5600701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:14.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.067+0000 7fc56548c640 1 --2- 172.21.15.150:0/2566549217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5600ad7e0 0x7fc5600701c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fc56006f7a0 tx=0x7fc55801cd60 comp rx=0 tx=0).stop 2024-09-17T05:07:14.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.067+0000 7fc56548c640 1 -- 172.21.15.150:0/2566549217 shutdown_connections 2024-09-17T05:07:14.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.067+0000 7fc56548c640 1 --2- 172.21.15.150:0/2566549217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5600ad7e0 0x7fc5600701c0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.067+0000 7fc56548c640 1 --2- 172.21.15.150:0/2566549217 >> v2:172.21.15.176:3300/0 conn(0x7fc5600c3db0 0x7fc5600ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.067+0000 7fc56548c640 1 -- 172.21.15.150:0/2566549217 >> 172.21.15.150:0/2566549217 conn(0x7fc5600c4a00 msgr2=0x7fc5600c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.067+0000 7fc56548c640 1 -- 172.21.15.150:0/2566549217 shutdown_connections 2024-09-17T05:07:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.067+0000 7fc56548c640 1 -- 172.21.15.150:0/2566549217 wait complete. 2024-09-17T05:07:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.067+0000 7fc56548c640 1 Processor -- start 2024-09-17T05:07:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.067+0000 7fc56548c640 1 -- start start 2024-09-17T05:07:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc56548c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5600ad7e0 0x7fc560157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc56548c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5600c3db0 0x7fc56014af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc56548c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5600c41c0 0x7fc5601495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc56548c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc5600a75f0 con 0x7fc5600ad7e0 2024-09-17T05:07:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc56548c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc5600add70 con 0x7fc5600c3db0 2024-09-17T05:07:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc56548c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc5600a7770 con 0x7fc5600c41c0 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc55effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5600ad7e0 0x7fc560157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc55effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5600ad7e0 0x7fc560157070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56908/0 (socket says 172.21.15.150:56908) 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc55effd640 1 -- 172.21.15.150:0/992611485 learned_addr learned my addr 172.21.15.150:0/992611485 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc55f7fe640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5600c41c0 0x7fc5601495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc55e7fc640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5600c3db0 0x7fc56014af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc55effd640 1 -- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5600c41c0 msgr2=0x7fc5601495b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc55effd640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5600c41c0 0x7fc5601495b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc55effd640 1 -- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5600c3db0 msgr2=0x7fc56014af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc55effd640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5600c3db0 0x7fc56014af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc55effd640 1 -- 172.21.15.150:0/992611485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56014b480 con 0x7fc5600ad7e0 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc55effd640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5600ad7e0 0x7fc560157070 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fc55400d700 tx=0x7fc55400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc5477fe640 1 -- 172.21.15.150:0/992611485 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc554017070 con 0x7fc5600ad7e0 2024-09-17T05:07:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc5477fe640 1 -- 172.21.15.150:0/992611485 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc554009b10 con 0x7fc5600ad7e0 2024-09-17T05:07:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc56548c640 1 -- 172.21.15.150:0/992611485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc560154430 con 0x7fc5600ad7e0 2024-09-17T05:07:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc5477fe640 1 -- 172.21.15.150:0/992611485 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc554010460 con 0x7fc5600ad7e0 2024-09-17T05:07:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc56548c640 1 -- 172.21.15.150:0/992611485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc56014a740 con 0x7fc5600ad7e0 2024-09-17T05:07:14.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc56548c640 1 -- 172.21.15.150:0/992611485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5600ad2a0 con 0x7fc5600ad7e0 2024-09-17T05:07:14.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.071+0000 7fc5477fe640 1 -- 172.21.15.150:0/992611485 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc55401a070 con 0x7fc5600ad7e0 2024-09-17T05:07:14.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.075+0000 7fc5477fe640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc54c073200 0x7fc54c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:14.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.075+0000 7fc5477fe640 1 -- 172.21.15.150:0/992611485 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7fc5540953e0 con 0x7fc5600ad7e0 2024-09-17T05:07:14.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.075+0000 7fc55e7fc640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc54c073200 0x7fc54c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:14.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.075+0000 7fc55e7fc640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc54c073200 0x7fc54c075710 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc5600c52d0 tx=0x7fc550009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:14.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.079+0000 7fc5477fe640 1 -- 172.21.15.150:0/992611485 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc55405f8b0 con 0x7fc5600ad7e0 2024-09-17T05:07:14.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.283+0000 7fc56548c640 1 -- 172.21.15.150:0/992611485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5601575b0 con 0x7fc5600ad7e0 2024-09-17T05:07:14.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.283+0000 7fc5477fe640 1 -- 172.21.15.150:0/992611485 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+6533 (secure 0 0 0) 0x7fc55405f1b0 con 0x7fc5600ad7e0 2024-09-17T05:07:14.284 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:14.285 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":131,"btime":"2024-09-17T05:06:58:539213+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5945,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:06:54.320176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6912},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6912":{"gid":6912,"name":"a","rank":0,"incarnation":127,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6912,"qdb_cluster":[6912]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:14.285 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 131 2024-09-17T05:07:14.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.287+0000 7fc56548c640 1 -- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc54c073200 msgr2=0x7fc54c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:14.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.287+0000 7fc56548c640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc54c073200 0x7fc54c075710 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc5600c52d0 tx=0x7fc550009040 comp rx=0 tx=0).stop 2024-09-17T05:07:14.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.287+0000 7fc56548c640 1 -- 172.21.15.150:0/992611485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5600ad7e0 msgr2=0x7fc560157070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:14.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.287+0000 7fc56548c640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5600ad7e0 0x7fc560157070 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fc55400d700 tx=0x7fc55400dbc0 comp rx=0 tx=0).stop 2024-09-17T05:07:14.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.287+0000 7fc56548c640 1 -- 172.21.15.150:0/992611485 shutdown_connections 2024-09-17T05:07:14.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.287+0000 7fc56548c640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc54c073200 0x7fc54c075710 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:14.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.287+0000 7fc56548c640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5600c3db0 0x7fc56014af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:14.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.287+0000 7fc56548c640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5600c41c0 0x7fc5601495b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:14.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.287+0000 7fc56548c640 1 --2- 172.21.15.150:0/992611485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5600ad7e0 0x7fc560157070 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:14.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.287+0000 7fc56548c640 1 -- 172.21.15.150:0/992611485 >> 172.21.15.150:0/992611485 conn(0x7fc5600c4a00 msgr2=0x7fc5600c5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:14.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.287+0000 7fc56548c640 1 -- 172.21.15.150:0/992611485 shutdown_connections 2024-09-17T05:07:14.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:14.287+0000 7fc56548c640 1 -- 172.21.15.150:0/992611485 wait complete. 2024-09-17T05:07:14.305 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:07:16.306 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:16.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.483+0000 7fd92964f640 1 -- 172.21.15.150:0/3458201635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd924153bd0 msgr2=0x7fd92415ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:16.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.483+0000 7fd92964f640 1 --2- 172.21.15.150:0/3458201635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd924153bd0 0x7fd92415ee50 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fd91c009bc0 tx=0x7fd91c01cba0 comp rx=0 tx=0).stop 2024-09-17T05:07:16.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.483+0000 7fd92964f640 1 -- 172.21.15.150:0/3458201635 shutdown_connections 2024-09-17T05:07:16.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.483+0000 7fd92964f640 1 --2- 172.21.15.150:0/3458201635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd924153bd0 0x7fd92415ee50 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:16.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.483+0000 7fd92964f640 1 -- 172.21.15.150:0/3458201635 >> 172.21.15.150:0/3458201635 conn(0x7fd9240c5970 msgr2=0x7fd924073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:16.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.483+0000 7fd92964f640 1 -- 172.21.15.150:0/3458201635 shutdown_connections 2024-09-17T05:07:16.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.483+0000 7fd92964f640 1 -- 172.21.15.150:0/3458201635 wait complete. 2024-09-17T05:07:16.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd92964f640 1 Processor -- start 2024-09-17T05:07:16.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd92964f640 1 -- start start 2024-09-17T05:07:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd92964f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd924074db0 0x7fd9241f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd92964f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd92414d430 0x7fd9241425d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd92964f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd924153bd0 0x7fd924144ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd92964f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd9240ab240 con 0x7fd924074db0 2024-09-17T05:07:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd92964f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd9240ab0c0 con 0x7fd92414d430 2024-09-17T05:07:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd92964f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd924153fe0 con 0x7fd924153bd0 2024-09-17T05:07:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd922ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd924074db0 0x7fd9241f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd922ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd924074db0 0x7fd9241f0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56912/0 (socket says 172.21.15.150:56912) 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd922ffd640 1 -- 172.21.15.150:0/4228225587 learned_addr learned my addr 172.21.15.150:0/4228225587 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd91bfff640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd92414d430 0x7fd9241425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd9237fe640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd924153bd0 0x7fd924144ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd922ffd640 1 -- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd924153bd0 msgr2=0x7fd924144ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd922ffd640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd924153bd0 0x7fd924144ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd922ffd640 1 -- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd92414d430 msgr2=0x7fd9241425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd922ffd640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd92414d430 0x7fd9241425d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd922ffd640 1 -- 172.21.15.150:0/4228225587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9241459b0 con 0x7fd924074db0 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd922ffd640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd924074db0 0x7fd9241f0fe0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd91400c490 tx=0x7fd91400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd91b7fe640 1 -- 172.21.15.150:0/4228225587 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd914011790 con 0x7fd924074db0 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd91b7fe640 1 -- 172.21.15.150:0/4228225587 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd914017070 con 0x7fd924074db0 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd92964f640 1 -- 172.21.15.150:0/4228225587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd924145c90 con 0x7fd924074db0 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd91b7fe640 1 -- 172.21.15.150:0/4228225587 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd914011ce0 con 0x7fd924074db0 2024-09-17T05:07:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd92964f640 1 -- 172.21.15.150:0/4228225587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9241f0ac0 con 0x7fd924074db0 2024-09-17T05:07:16.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.487+0000 7fd92964f640 1 -- 172.21.15.150:0/4228225587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9241f05d0 con 0x7fd924074db0 2024-09-17T05:07:16.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.491+0000 7fd91b7fe640 1 -- 172.21.15.150:0/4228225587 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd914026430 con 0x7fd924074db0 2024-09-17T05:07:16.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.491+0000 7fd91b7fe640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd904073200 0x7fd904075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:16.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.491+0000 7fd91b7fe640 1 -- 172.21.15.150:0/4228225587 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7fd914095560 con 0x7fd924074db0 2024-09-17T05:07:16.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.491+0000 7fd91bfff640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd904073200 0x7fd904075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:16.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.491+0000 7fd91bfff640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd904073200 0x7fd904075710 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd92414ae30 tx=0x7fd90c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:16.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.495+0000 7fd91b7fe640 1 -- 172.21.15.150:0/4228225587 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd91405fc10 con 0x7fd924074db0 2024-09-17T05:07:16.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.703+0000 7fd92964f640 1 -- 172.21.15.150:0/4228225587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd924071c40 con 0x7fd924074db0 2024-09-17T05:07:16.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.703+0000 7fd91b7fe640 1 -- 172.21.15.150:0/4228225587 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+6533 (secure 0 0 0) 0x7fd91405f510 con 0x7fd924074db0 2024-09-17T05:07:16.703 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:16.703 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":131,"btime":"2024-09-17T05:06:58:539213+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5945,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:06:54.320176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6912},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6912":{"gid":6912,"name":"a","rank":0,"incarnation":127,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6912,"qdb_cluster":[6912]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:16.704 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 131 2024-09-17T05:07:16.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.707+0000 7fd92964f640 1 -- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd904073200 msgr2=0x7fd904075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:16.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.707+0000 7fd92964f640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd904073200 0x7fd904075710 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd92414ae30 tx=0x7fd90c009040 comp rx=0 tx=0).stop 2024-09-17T05:07:16.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.707+0000 7fd92964f640 1 -- 172.21.15.150:0/4228225587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd924074db0 msgr2=0x7fd9241f0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:16.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.707+0000 7fd92964f640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd924074db0 0x7fd9241f0fe0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd91400c490 tx=0x7fd91400c950 comp rx=0 tx=0).stop 2024-09-17T05:07:16.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.707+0000 7fd92964f640 1 -- 172.21.15.150:0/4228225587 shutdown_connections 2024-09-17T05:07:16.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.707+0000 7fd92964f640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd904073200 0x7fd904075710 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:16.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.707+0000 7fd92964f640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd92414d430 0x7fd9241425d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:16.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.707+0000 7fd92964f640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd924153bd0 0x7fd924144ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:16.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.707+0000 7fd92964f640 1 --2- 172.21.15.150:0/4228225587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd924074db0 0x7fd9241f0fe0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:16.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.707+0000 7fd92964f640 1 -- 172.21.15.150:0/4228225587 >> 172.21.15.150:0/4228225587 conn(0x7fd9240c5970 msgr2=0x7fd92406ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:16.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.707+0000 7fd92964f640 1 -- 172.21.15.150:0/4228225587 shutdown_connections 2024-09-17T05:07:16.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:16.707+0000 7fd92964f640 1 -- 172.21.15.150:0/4228225587 wait complete. 2024-09-17T05:07:16.724 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:07:18.725 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:18.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.890+0000 7f5277492640 1 -- 172.21.15.150:0/3805687267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52700c87e0 msgr2=0x7f5270155150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:18.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.890+0000 7f5277492640 1 --2- 172.21.15.150:0/3805687267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52700c87e0 0x7f5270155150 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f526c009c80 tx=0x7f526c01cd10 comp rx=0 tx=0).stop 2024-09-17T05:07:18.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.890+0000 7f5277492640 1 -- 172.21.15.150:0/3805687267 shutdown_connections 2024-09-17T05:07:18.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.890+0000 7f5277492640 1 --2- 172.21.15.150:0/3805687267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52700c87e0 0x7f5270155150 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:18.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.890+0000 7f5277492640 1 --2- 172.21.15.150:0/3805687267 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f52700c3740 0x7f5270074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:18.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.890+0000 7f5277492640 1 -- 172.21.15.150:0/3805687267 >> 172.21.15.150:0/3805687267 conn(0x7f527006e360 msgr2=0x7f527006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:18.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.890+0000 7f5277492640 1 -- 172.21.15.150:0/3805687267 shutdown_connections 2024-09-17T05:07:18.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.890+0000 7f5277492640 1 -- 172.21.15.150:0/3805687267 wait complete. 2024-09-17T05:07:18.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5277492640 1 Processor -- start 2024-09-17T05:07:18.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5277492640 1 -- start start 2024-09-17T05:07:18.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5277492640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52700c87e0 0x7f527014a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:18.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5277492640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5270155690 0x7f527014bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:18.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5277492640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52701f1850 0x7f52701508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:18.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5277492640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5270161be0 con 0x7f52700c87e0 2024-09-17T05:07:18.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5277492640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5270161a60 con 0x7f52701f1850 2024-09-17T05:07:18.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5277492640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5270161d60 con 0x7f5270155690 2024-09-17T05:07:18.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f527522e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52700c87e0 0x7f527014a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:18.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f527522e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52700c87e0 0x7f527014a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56930/0 (socket says 172.21.15.150:56930) 2024-09-17T05:07:18.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f527522e640 1 -- 172.21.15.150:0/522723086 learned_addr learned my addr 172.21.15.150:0/522723086 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:18.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5274a2d640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5270155690 0x7f527014bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:18.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5275a2f640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52701f1850 0x7f52701508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:18.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f527522e640 1 -- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5270155690 msgr2=0x7f527014bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:18.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f527522e640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5270155690 0x7f527014bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:18.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f527522e640 1 -- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52701f1850 msgr2=0x7f52701508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:18.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f527522e640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52701f1850 0x7f52701508a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:18.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f527522e640 1 -- 172.21.15.150:0/522723086 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52700abdd0 con 0x7f52700c87e0 2024-09-17T05:07:18.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5274a2d640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5270155690 0x7f527014bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:07:18.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f527522e640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52700c87e0 0x7f527014a980 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f526400e800 tx=0x7f526400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:18.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5261ffb640 1 -- 172.21.15.150:0/522723086 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526400cc60 con 0x7f52700c87e0 2024-09-17T05:07:18.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5261ffb640 1 -- 172.21.15.150:0/522723086 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5264010430 con 0x7f52700c87e0 2024-09-17T05:07:18.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5277492640 1 -- 172.21.15.150:0/522723086 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5270157390 con 0x7f52700c87e0 2024-09-17T05:07:18.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5261ffb640 1 -- 172.21.15.150:0/522723086 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5264010860 con 0x7f52700c87e0 2024-09-17T05:07:18.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.894+0000 7f5277492640 1 -- 172.21.15.150:0/522723086 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52701f0d70 con 0x7f52700c87e0 2024-09-17T05:07:18.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.898+0000 7f5277492640 1 -- 172.21.15.150:0/522723086 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52701f0b60 con 0x7f52700c87e0 2024-09-17T05:07:18.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.898+0000 7f5261ffb640 1 -- 172.21.15.150:0/522723086 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5264010a00 con 0x7f52700c87e0 2024-09-17T05:07:18.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.898+0000 7f5261ffb640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5254073230 0x7f5254075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:18.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.898+0000 7f5261ffb640 1 -- 172.21.15.150:0/522723086 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7f5264095620 con 0x7f52700c87e0 2024-09-17T05:07:18.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.898+0000 7f5274a2d640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5254073230 0x7f5254075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:18.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.898+0000 7f5274a2d640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5254073230 0x7f5254075740 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f526c0af000 tx=0x7f526c01b550 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:18.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:18.902+0000 7f5261ffb640 1 -- 172.21.15.150:0/522723086 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f526405fb90 con 0x7f52700c87e0 2024-09-17T05:07:19.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.106+0000 7f5277492640 1 -- 172.21.15.150:0/522723086 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52700c3740 con 0x7f52700c87e0 2024-09-17T05:07:19.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.106+0000 7f5261ffb640 1 -- 172.21.15.150:0/522723086 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+6533 (secure 0 0 0) 0x7f526405f490 con 0x7f52700c87e0 2024-09-17T05:07:19.109 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:19.109 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":131,"btime":"2024-09-17T05:06:58:539213+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5945,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:06:54.320176+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6912},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6912":{"gid":6912,"name":"a","rank":0,"incarnation":127,"state":"up:active","state_seq":7,"addr":"172.21.15.150:6833/523364553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":523364553},{"type":"v1","addr":"172.21.15.150:6833","nonce":523364553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6912,"qdb_cluster":[6912]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:19.109 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 131 2024-09-17T05:07:19.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.110+0000 7f5277492640 1 -- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5254073230 msgr2=0x7f5254075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:19.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.110+0000 7f5277492640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5254073230 0x7f5254075740 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f526c0af000 tx=0x7f526c01b550 comp rx=0 tx=0).stop 2024-09-17T05:07:19.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.110+0000 7f5277492640 1 -- 172.21.15.150:0/522723086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52700c87e0 msgr2=0x7f527014a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:19.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.110+0000 7f5277492640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52700c87e0 0x7f527014a980 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f526400e800 tx=0x7f526400ecc0 comp rx=0 tx=0).stop 2024-09-17T05:07:19.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.110+0000 7f5277492640 1 -- 172.21.15.150:0/522723086 shutdown_connections 2024-09-17T05:07:19.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.110+0000 7f5277492640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5254073230 0x7f5254075740 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:19.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.110+0000 7f5277492640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f52701f1850 0x7f52701508a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:19.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.110+0000 7f5277492640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5270155690 0x7f527014bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:19.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.110+0000 7f5277492640 1 --2- 172.21.15.150:0/522723086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52700c87e0 0x7f527014a980 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:19.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.110+0000 7f5277492640 1 -- 172.21.15.150:0/522723086 >> 172.21.15.150:0/522723086 conn(0x7f527006e360 msgr2=0x7f5270151880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:19.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.110+0000 7f5277492640 1 -- 172.21.15.150:0/522723086 shutdown_connections 2024-09-17T05:07:19.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:19.110+0000 7f5277492640 1 -- 172.21.15.150:0/522723086 wait complete. 2024-09-17T05:07:19.129 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:07:21.130 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:21.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.286+0000 7f9e681d9640 1 -- 172.21.15.150:0/1348769479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e601540b0 msgr2=0x7f9e600c8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.286+0000 7f9e681d9640 1 --2- 172.21.15.150:0/1348769479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e601540b0 0x7f9e600c8330 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009930 tx=0x7f9e5401c8e0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.286+0000 7f9e681d9640 1 -- 172.21.15.150:0/1348769479 shutdown_connections 2024-09-17T05:07:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.286+0000 7f9e681d9640 1 --2- 172.21.15.150:0/1348769479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e601540b0 0x7f9e600c8330 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.286+0000 7f9e681d9640 1 -- 172.21.15.150:0/1348769479 >> 172.21.15.150:0/1348769479 conn(0x7f9e600aab60 msgr2=0x7f9e600aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.286+0000 7f9e681d9640 1 -- 172.21.15.150:0/1348769479 shutdown_connections 2024-09-17T05:07:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.286+0000 7f9e681d9640 1 -- 172.21.15.150:0/1348769479 wait complete. 2024-09-17T05:07:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e681d9640 1 Processor -- start 2024-09-17T05:07:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e681d9640 1 -- start start 2024-09-17T05:07:21.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e681d9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e600c7eb0 0x7f9e6014d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:21.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e681d9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600c8870 0x7f9e600adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:21.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e681d9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e601540b0 0x7f9e601edf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e681d9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9e60156540 con 0x7f9e600c8870 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e681d9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9e601544c0 con 0x7f9e600c7eb0 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e681d9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9e601566c0 con 0x7f9e601540b0 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e65774640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600c8870 0x7f9e600adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e65774640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600c8870 0x7f9e600adfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56940/0 (socket says 172.21.15.150:56940) 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e65774640 1 -- 172.21.15.150:0/1983744291 learned_addr learned my addr 172.21.15.150:0/1983744291 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e66776640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e601540b0 0x7f9e601edf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e65f75640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e600c7eb0 0x7f9e6014d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e65774640 1 -- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e601540b0 msgr2=0x7f9e601edf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e65774640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e601540b0 0x7f9e601edf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e65774640 1 -- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e600c7eb0 msgr2=0x7f9e6014d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e65774640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e600c7eb0 0x7f9e6014d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e65774640 1 -- 172.21.15.150:0/1983744291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6014d680 con 0x7f9e600c8870 2024-09-17T05:07:21.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e65774640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600c8870 0x7f9e600adfc0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9e540b0040 tx=0x7f9e540a56b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:21.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e527fc640 1 -- 172.21.15.150:0/1983744291 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e540b2040 con 0x7f9e600c8870 2024-09-17T05:07:21.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e527fc640 1 -- 172.21.15.150:0/1983744291 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e54004430 con 0x7f9e600c8870 2024-09-17T05:07:21.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e681d9640 1 -- 172.21.15.150:0/1983744291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e60149670 con 0x7f9e600c8870 2024-09-17T05:07:21.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e527fc640 1 -- 172.21.15.150:0/1983744291 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e54003860 con 0x7f9e600c8870 2024-09-17T05:07:21.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e681d9640 1 -- 172.21.15.150:0/1983744291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e601f23a0 con 0x7f9e600c8870 2024-09-17T05:07:21.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.290+0000 7f9e681d9640 1 -- 172.21.15.150:0/1983744291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e601f2080 con 0x7f9e600c8870 2024-09-17T05:07:21.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.294+0000 7f9e527fc640 1 -- 172.21.15.150:0/1983744291 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9e54003a00 con 0x7f9e600c8870 2024-09-17T05:07:21.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.294+0000 7f9e527fc640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9e44073230 0x7f9e44075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:21.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.294+0000 7f9e527fc640 1 -- 172.21.15.150:0/1983744291 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f9e5412f030 con 0x7f9e600c8870 2024-09-17T05:07:21.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.294+0000 7f9e65f75640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9e44073230 0x7f9e44075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:21.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.294+0000 7f9e65f75640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9e44073230 0x7f9e44075740 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c000f70 tx=0x7f9e5c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:21.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.298+0000 7f9e527fc640 1 -- 172.21.15.150:0/1983744291 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9e601f2080 con 0x7f9e600c8870 2024-09-17T05:07:21.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.514+0000 7f9e681d9640 1 -- 172.21.15.150:0/1983744291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e600c8330 con 0x7f9e600c8870 2024-09-17T05:07:21.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.514+0000 7f9e527fc640 1 -- 172.21.15.150:0/1983744291 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+5700 (secure 0 0 0) 0x7f9e540164d0 con 0x7f9e600c8870 2024-09-17T05:07:21.515 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:21.515 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":132,"btime":"2024-09-17T05:07:21:092573+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:21.092562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:replay","state_seq":1,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:21.515 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 132 2024-09-17T05:07:21.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.514+0000 7f9e681d9640 1 -- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9e44073230 msgr2=0x7f9e44075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:21.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.514+0000 7f9e681d9640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9e44073230 0x7f9e44075740 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c000f70 tx=0x7f9e5c009040 comp rx=0 tx=0).stop 2024-09-17T05:07:21.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.514+0000 7f9e681d9640 1 -- 172.21.15.150:0/1983744291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600c8870 msgr2=0x7f9e600adfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:21.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.514+0000 7f9e681d9640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600c8870 0x7f9e600adfc0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9e540b0040 tx=0x7f9e540a56b0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.518+0000 7f9e681d9640 1 -- 172.21.15.150:0/1983744291 shutdown_connections 2024-09-17T05:07:21.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.518+0000 7f9e681d9640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9e44073230 0x7f9e44075740 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.518+0000 7f9e681d9640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e601540b0 0x7f9e601edf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.518+0000 7f9e681d9640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600c8870 0x7f9e600adfc0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.518+0000 7f9e681d9640 1 --2- 172.21.15.150:0/1983744291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e600c7eb0 0x7f9e6014d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.518+0000 7f9e681d9640 1 -- 172.21.15.150:0/1983744291 >> 172.21.15.150:0/1983744291 conn(0x7f9e600aab60 msgr2=0x7f9e6015f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:21.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.518+0000 7f9e681d9640 1 -- 172.21.15.150:0/1983744291 shutdown_connections 2024-09-17T05:07:21.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.518+0000 7f9e681d9640 1 -- 172.21.15.150:0/1983744291 wait complete. 2024-09-17T05:07:21.540 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-17T05:07:21.540 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:07:21.540 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:21.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.698+0000 7fd982e8d640 1 -- 172.21.15.150:0/1560648537 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd97c0c35b0 msgr2=0x7fd97c0c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:21.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.698+0000 7fd982e8d640 1 --2- 172.21.15.150:0/1560648537 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd97c0c35b0 0x7fd97c0c8a80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd970009900 tx=0x7fd97001c8b0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.698+0000 7fd982e8d640 1 -- 172.21.15.150:0/1560648537 shutdown_connections 2024-09-17T05:07:21.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.698+0000 7fd982e8d640 1 --2- 172.21.15.150:0/1560648537 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd97c0c35b0 0x7fd97c0c8a80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.698+0000 7fd982e8d640 1 --2- 172.21.15.150:0/1560648537 >> v2:172.21.15.176:3300/0 conn(0x7fd97c1568a0 0x7fd97c0c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.698+0000 7fd982e8d640 1 -- 172.21.15.150:0/1560648537 >> 172.21.15.150:0/1560648537 conn(0x7fd97c074b80 msgr2=0x7fd97c06f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:21.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.698+0000 7fd982e8d640 1 -- 172.21.15.150:0/1560648537 shutdown_connections 2024-09-17T05:07:21.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.698+0000 7fd982e8d640 1 -- 172.21.15.150:0/1560648537 wait complete. 2024-09-17T05:07:21.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.698+0000 7fd982e8d640 1 Processor -- start 2024-09-17T05:07:21.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.698+0000 7fd982e8d640 1 -- start start 2024-09-17T05:07:21.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd982e8d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd97c0c35b0 0x7fd97c1f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:21.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd982e8d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97c0c8fc0 0x7fd97c152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd982e8d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97c1568a0 0x7fd97c159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd982e8d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd97c0c6b70 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd982e8d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd97c0c69f0 con 0x7fd97c1568a0 2024-09-17T05:07:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd982e8d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd97c0c27b0 con 0x7fd97c0c35b0 2024-09-17T05:07:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd97bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97c0c8fc0 0x7fd97c152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd97bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97c0c8fc0 0x7fd97c152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56946/0 (socket says 172.21.15.150:56946) 2024-09-17T05:07:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd97bfff640 1 -- 172.21.15.150:0/165440029 learned_addr learned my addr 172.21.15.150:0/165440029 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd98142a640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97c1568a0 0x7fd97c159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd980c29640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd97c0c35b0 0x7fd97c1f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd97bfff640 1 -- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd97c0c35b0 msgr2=0x7fd97c1f0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:21.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd97bfff640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd97c0c35b0 0x7fd97c1f0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd97bfff640 1 -- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97c1568a0 msgr2=0x7fd97c159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd97bfff640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97c1568a0 0x7fd97c159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd97bfff640 1 -- 172.21.15.150:0/165440029 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd97c14d400 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd97bfff640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97c0c8fc0 0x7fd97c152a70 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fd970009850 tx=0x7fd970005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd9797fa640 1 -- 172.21.15.150:0/165440029 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9700b2040 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd9797fa640 1 -- 172.21.15.150:0/165440029 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd970005320 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd982e8d640 1 -- 172.21.15.150:0/165440029 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd97c0c9f90 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd9797fa640 1 -- 172.21.15.150:0/165440029 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9700046d0 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd982e8d640 1 -- 172.21.15.150:0/165440029 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd97c1efa60 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd982e8d640 1 -- 172.21.15.150:0/165440029 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd97c0c8a80 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.702+0000 7fd9797fa640 1 -- 172.21.15.150:0/165440029 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd9700bd060 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.706+0000 7fd9797fa640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd968073200 0x7fd968075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:21.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.706+0000 7fd9797fa640 1 -- 172.21.15.150:0/165440029 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7fd97012ea90 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.706+0000 7fd980c29640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd968073200 0x7fd968075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:21.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.706+0000 7fd980c29640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd968073200 0x7fd968075710 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009600 tx=0x7fd96c0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:21.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.710+0000 7fd9797fa640 1 -- 172.21.15.150:0/165440029 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd9700164d0 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.906+0000 7fd982e8d640 1 -- 172.21.15.150:0/165440029 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd97c0691c0 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.906+0000 7fd9797fa640 1 -- 172.21.15.150:0/165440029 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+5700 (secure 0 0 0) 0x7fd9700f88d0 con 0x7fd97c0c8fc0 2024-09-17T05:07:21.909 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:21.910 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":132,"btime":"2024-09-17T05:07:21:092573+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:21.092562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:replay","state_seq":1,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:21.910 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 132 2024-09-17T05:07:21.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.910+0000 7fd982e8d640 1 -- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd968073200 msgr2=0x7fd968075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:21.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.910+0000 7fd982e8d640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd968073200 0x7fd968075710 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009600 tx=0x7fd96c0091e0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.910+0000 7fd982e8d640 1 -- 172.21.15.150:0/165440029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97c0c8fc0 msgr2=0x7fd97c152a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:21.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.910+0000 7fd982e8d640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97c0c8fc0 0x7fd97c152a70 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fd970009850 tx=0x7fd970005de0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.910+0000 7fd982e8d640 1 -- 172.21.15.150:0/165440029 shutdown_connections 2024-09-17T05:07:21.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.910+0000 7fd982e8d640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd968073200 0x7fd968075710 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.910+0000 7fd982e8d640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97c1568a0 0x7fd97c159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.910+0000 7fd982e8d640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97c0c8fc0 0x7fd97c152a70 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.910+0000 7fd982e8d640 1 --2- 172.21.15.150:0/165440029 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd97c0c35b0 0x7fd97c1f0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:21.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.910+0000 7fd982e8d640 1 -- 172.21.15.150:0/165440029 >> 172.21.15.150:0/165440029 conn(0x7fd97c074b80 msgr2=0x7fd97c154aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:21.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.910+0000 7fd982e8d640 1 -- 172.21.15.150:0/165440029 shutdown_connections 2024-09-17T05:07:21.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:21.910+0000 7fd982e8d640 1 -- 172.21.15.150:0/165440029 wait complete. 2024-09-17T05:07:21.932 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:07:21:092573+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 132, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 132, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5945": { "addr": "172.21.15.176:6837/792092400", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 792092400, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 792092400, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5945, "incarnation": 132, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:07:21.092562+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5945 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 125, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6831": { "addr": "172.21.15.150:6835/92695211", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 92695211, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 92695211, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6831, "incarnation": 122, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:06:34.202583+0000", "qdb_cluster": [ 6831 ], "qdb_leader": 6831, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6831 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:07:23.933 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:24.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.098+0000 7f0d1712c640 1 -- 172.21.15.150:0/1187302467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d101535b0 msgr2=0x7f0d10157440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:24.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.098+0000 7f0d1712c640 1 --2- 172.21.15.150:0/1187302467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d101535b0 0x7f0d10157440 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f0d00009900 tx=0x7f0d0001c850 comp rx=0 tx=0).stop 2024-09-17T05:07:24.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.098+0000 7f0d1712c640 1 -- 172.21.15.150:0/1187302467 shutdown_connections 2024-09-17T05:07:24.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.098+0000 7f0d1712c640 1 --2- 172.21.15.150:0/1187302467 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d100c5760 0x7f0d100c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.098+0000 7f0d1712c640 1 --2- 172.21.15.150:0/1187302467 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d101535b0 0x7f0d10157440 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.098+0000 7f0d1712c640 1 -- 172.21.15.150:0/1187302467 >> 172.21.15.150:0/1187302467 conn(0x7f0d100aaa80 msgr2=0x7f0d100aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.098+0000 7f0d1712c640 1 -- 172.21.15.150:0/1187302467 shutdown_connections 2024-09-17T05:07:24.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.098+0000 7f0d1712c640 1 -- 172.21.15.150:0/1187302467 wait complete. 2024-09-17T05:07:24.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d1712c640 1 Processor -- start 2024-09-17T05:07:24.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d1712c640 1 -- start start 2024-09-17T05:07:24.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d1712c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d100a7f10 0x7f0d100c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:24.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d1712c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d100c5760 0x7f0d101f0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:24.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d1712c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d101535b0 0x7f0d10149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:24.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d1712c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0d100718d0 con 0x7f0d101535b0 2024-09-17T05:07:24.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d1712c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0d100a8730 con 0x7f0d100c5760 2024-09-17T05:07:24.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d1712c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0d100c5370 con 0x7f0d100a7f10 2024-09-17T05:07:24.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d156c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d101535b0 0x7f0d10149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:24.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d156c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d101535b0 0x7f0d10149290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56964/0 (socket says 172.21.15.150:56964) 2024-09-17T05:07:24.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d156c9640 1 -- 172.21.15.150:0/1260274142 learned_addr learned my addr 172.21.15.150:0/1260274142 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:24.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d14ec8640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d100a7f10 0x7f0d100c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:24.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d0ffff640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d100c5760 0x7f0d101f0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:24.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d156c9640 1 -- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d100a7f10 msgr2=0x7f0d100c22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:24.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d156c9640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d100a7f10 0x7f0d100c22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:24.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d156c9640 1 -- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d100c5760 msgr2=0x7f0d101f0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:24.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d156c9640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d100c5760 0x7f0d101f0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:24.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d156c9640 1 -- 172.21.15.150:0/1260274142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d10149e30 con 0x7f0d101535b0 2024-09-17T05:07:24.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d0ffff640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d100c5760 0x7f0d101f0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:07:24.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d156c9640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d101535b0 0x7f0d10149290 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0d0800c6e0 tx=0x7f0d0800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:24.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d0d7fa640 1 -- 172.21.15.150:0/1260274142 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08003b80 con 0x7f0d101535b0 2024-09-17T05:07:24.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d0d7fa640 1 -- 172.21.15.150:0/1260274142 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d08003d20 con 0x7f0d101535b0 2024-09-17T05:07:24.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d1712c640 1 -- 172.21.15.150:0/1260274142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d10150790 con 0x7f0d101535b0 2024-09-17T05:07:24.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d0d7fa640 1 -- 172.21.15.150:0/1260274142 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08005330 con 0x7f0d101535b0 2024-09-17T05:07:24.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d1712c640 1 -- 172.21.15.150:0/1260274142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d10150a30 con 0x7f0d101535b0 2024-09-17T05:07:24.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.102+0000 7f0d1712c640 1 -- 172.21.15.150:0/1260274142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d101539c0 con 0x7f0d101535b0 2024-09-17T05:07:24.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.106+0000 7f0d0d7fa640 1 -- 172.21.15.150:0/1260274142 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0d080054d0 con 0x7f0d101535b0 2024-09-17T05:07:24.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.106+0000 7f0d0d7fa640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0cfc073230 0x7f0cfc075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:24.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.106+0000 7f0d0d7fa640 1 -- 172.21.15.150:0/1260274142 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f0d08094ab0 con 0x7f0d101535b0 2024-09-17T05:07:24.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.106+0000 7f0d14ec8640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0cfc073230 0x7f0cfc075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:24.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.106+0000 7f0d14ec8640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0cfc073230 0x7f0cfc075740 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0d000af000 tx=0x7f0d0001b090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:24.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.110+0000 7f0d0d7fa640 1 -- 172.21.15.150:0/1260274142 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0d0805efd0 con 0x7f0d101535b0 2024-09-17T05:07:24.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.314+0000 7f0d1712c640 1 -- 172.21.15.150:0/1260274142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d100c3070 con 0x7f0d101535b0 2024-09-17T05:07:24.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d0d7fa640 1 -- 172.21.15.150:0/1260274142 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) ==== 95+0+5700 (secure 0 0 0) 0x7f0d0805e8d0 con 0x7f0d101535b0 2024-09-17T05:07:24.318 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:24.319 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":134,"btime":"2024-09-17T05:07:23:634550+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:22.642730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 134 2024-09-17T05:07:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d1712c640 1 -- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0cfc073230 msgr2=0x7f0cfc075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d1712c640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0cfc073230 0x7f0cfc075740 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0d000af000 tx=0x7f0d0001b090 comp rx=0 tx=0).stop 2024-09-17T05:07:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d1712c640 1 -- 172.21.15.150:0/1260274142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d101535b0 msgr2=0x7f0d10149290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d1712c640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d101535b0 0x7f0d10149290 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f0d0800c6e0 tx=0x7f0d0800cba0 comp rx=0 tx=0).stop 2024-09-17T05:07:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d1712c640 1 -- 172.21.15.150:0/1260274142 shutdown_connections 2024-09-17T05:07:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d1712c640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0cfc073230 0x7f0cfc075740 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d1712c640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d100c5760 0x7f0d101f0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d1712c640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0d101535b0 0x7f0d10149290 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d1712c640 1 --2- 172.21.15.150:0/1260274142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0d100a7f10 0x7f0d100c22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:24.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d1712c640 1 -- 172.21.15.150:0/1260274142 >> 172.21.15.150:0/1260274142 conn(0x7f0d100aaa80 msgr2=0x7f0d1015e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:24.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d1712c640 1 -- 172.21.15.150:0/1260274142 shutdown_connections 2024-09-17T05:07:24.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:24.318+0000 7f0d1712c640 1 -- 172.21.15.150:0/1260274142 wait complete. 2024-09-17T05:07:24.342 INFO:tasks.mds_thrash.fs.[a]:no change 2024-09-17T05:07:26.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:26.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.518+0000 7f9b98eac640 1 -- 172.21.15.150:0/2884779564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b94074570 msgr2=0x7f9b940aaa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.518+0000 7f9b98eac640 1 --2- 172.21.15.150:0/2884779564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b94074570 0x7f9b940aaa40 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009840 tx=0x7f9b7c01c7b0 comp rx=0 tx=0).stop 2024-09-17T05:07:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.518+0000 7f9b98eac640 1 -- 172.21.15.150:0/2884779564 shutdown_connections 2024-09-17T05:07:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.518+0000 7f9b98eac640 1 --2- 172.21.15.150:0/2884779564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b94074570 0x7f9b940aaa40 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.518+0000 7f9b98eac640 1 --2- 172.21.15.150:0/2884779564 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b9406fca0 0x7f9b94153790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.518+0000 7f9b98eac640 1 -- 172.21.15.150:0/2884779564 >> 172.21.15.150:0/2884779564 conn(0x7f9b940c5ae0 msgr2=0x7f9b94073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.522+0000 7f9b98eac640 1 -- 172.21.15.150:0/2884779564 shutdown_connections 2024-09-17T05:07:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.522+0000 7f9b98eac640 1 -- 172.21.15.150:0/2884779564 wait complete. 2024-09-17T05:07:26.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.522+0000 7f9b98eac640 1 Processor -- start 2024-09-17T05:07:26.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.522+0000 7f9b98eac640 1 -- start start 2024-09-17T05:07:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b98eac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b9406fca0 0x7f9b941ed2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b98eac640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b94074570 0x7f9b94154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b98eac640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b94074980 0x7f9b941542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b98eac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9b940c4ce0 con 0x7f9b9406fca0 2024-09-17T05:07:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b98eac640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9415d690 con 0x7f9b94074570 2024-09-17T05:07:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b98eac640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9b94155800 con 0x7f9b94074980 2024-09-17T05:07:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b9259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b9406fca0 0x7f9b941ed2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b9259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b9406fca0 0x7f9b941ed2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56996/0 (socket says 172.21.15.150:56996) 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b9259c640 1 -- 172.21.15.150:0/1778508005 learned_addr learned my addr 172.21.15.150:0/1778508005 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b91d9b640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b94074570 0x7f9b94154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b92d9d640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b94074980 0x7f9b941542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b9259c640 1 -- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b94074980 msgr2=0x7f9b941542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b9259c640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b94074980 0x7f9b941542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b9259c640 1 -- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b94074570 msgr2=0x7f9b94154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b9259c640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b94074570 0x7f9b94154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b9259c640 1 -- 172.21.15.150:0/1778508005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b940ad570 con 0x7f9b9406fca0 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b9259c640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b9406fca0 0x7f9b941ed2f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f9b8400e9a0 tx=0x7f9b8400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b8affd640 1 -- 172.21.15.150:0/1778508005 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8400cd80 con 0x7f9b9406fca0 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b8affd640 1 -- 172.21.15.150:0/1778508005 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b84004020 con 0x7f9b9406fca0 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b98eac640 1 -- 172.21.15.150:0/1778508005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b940ad850 con 0x7f9b9406fca0 2024-09-17T05:07:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b8affd640 1 -- 172.21.15.150:0/1778508005 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b84004450 con 0x7f9b9406fca0 2024-09-17T05:07:26.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b98eac640 1 -- 172.21.15.150:0/1778508005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b94155290 con 0x7f9b9406fca0 2024-09-17T05:07:26.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.526+0000 7f9b98eac640 1 -- 172.21.15.150:0/1778508005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b94153790 con 0x7f9b9406fca0 2024-09-17T05:07:26.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.530+0000 7f9b8affd640 1 -- 172.21.15.150:0/1778508005 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9b84003600 con 0x7f9b9406fca0 2024-09-17T05:07:26.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.530+0000 7f9b8affd640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b74073230 0x7f9b74075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:26.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.530+0000 7f9b8affd640 1 -- 172.21.15.150:0/1778508005 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f9b840955c0 con 0x7f9b9406fca0 2024-09-17T05:07:26.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.530+0000 7f9b91d9b640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b74073230 0x7f9b74075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:26.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.530+0000 7f9b91d9b640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b74073230 0x7f9b74075740 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c0af000 tx=0x7f9b7c01aff0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:26.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.534+0000 7f9b8affd640 1 -- 172.21.15.150:0/1778508005 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9b8405f400 con 0x7f9b9406fca0 2024-09-17T05:07:26.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.818+0000 7f9b98eac640 1 -- 172.21.15.150:0/1778508005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b94073dc0 con 0x7f9b9406fca0 2024-09-17T05:07:26.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.818+0000 7f9b8affd640 1 -- 172.21.15.150:0/1778508005 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5707 (secure 0 0 0) 0x7f9b84063e30 con 0x7f9b9406fca0 2024-09-17T05:07:26.821 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:26.821 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":135,"btime":"2024-09-17T05:07:24:656384+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:26.821 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 135 2024-09-17T05:07:26.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.822+0000 7f9b98eac640 1 -- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b74073230 msgr2=0x7f9b74075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:26.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.822+0000 7f9b98eac640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b74073230 0x7f9b74075740 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c0af000 tx=0x7f9b7c01aff0 comp rx=0 tx=0).stop 2024-09-17T05:07:26.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.822+0000 7f9b98eac640 1 -- 172.21.15.150:0/1778508005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b9406fca0 msgr2=0x7f9b941ed2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:26.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.822+0000 7f9b98eac640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b9406fca0 0x7f9b941ed2f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f9b8400e9a0 tx=0x7f9b8400ee60 comp rx=0 tx=0).stop 2024-09-17T05:07:26.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.822+0000 7f9b98eac640 1 -- 172.21.15.150:0/1778508005 shutdown_connections 2024-09-17T05:07:26.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.822+0000 7f9b98eac640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9b74073230 0x7f9b74075740 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:26.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.822+0000 7f9b98eac640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b94074570 0x7f9b94154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:26.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.822+0000 7f9b98eac640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9b94074980 0x7f9b941542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:26.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.822+0000 7f9b98eac640 1 --2- 172.21.15.150:0/1778508005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b9406fca0 0x7f9b941ed2f0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:26.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.822+0000 7f9b98eac640 1 -- 172.21.15.150:0/1778508005 >> 172.21.15.150:0/1778508005 conn(0x7f9b940c5ae0 msgr2=0x7f9b940c7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:26.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.822+0000 7f9b98eac640 1 -- 172.21.15.150:0/1778508005 shutdown_connections 2024-09-17T05:07:26.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:26.822+0000 7f9b98eac640 1 -- 172.21.15.150:0/1778508005 wait complete. 2024-09-17T05:07:26.847 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=6912 2024-09-17T05:07:26.847 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.a 2024-09-17T05:07:27.848 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-17T05:07:27.848 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T05:07:27.848 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T05:07:27.851 INFO:tasks.ceph.mds.a:Started 2024-09-17T05:07:27.851 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:27.949 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T05:07:28.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.014+0000 7f610374e640 1 -- 172.21.15.150:0/3910150505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e8004010 msgr2=0x7f60e8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:28.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.014+0000 7f610374e640 1 --2- 172.21.15.150:0/3910150505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e8004010 0x7f60e8004810 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f60fc152660 tx=0x7f60ec01ced0 comp rx=0 tx=0).stop 2024-09-17T05:07:28.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.014+0000 7f6101ceb640 1 -- 172.21.15.150:0/3910150505 reap_dead start 2024-09-17T05:07:28.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.014+0000 7f610374e640 1 -- 172.21.15.150:0/3910150505 shutdown_connections 2024-09-17T05:07:28.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.014+0000 7f610374e640 1 -- 172.21.15.150:0/3910150505 >> 172.21.15.150:0/3910150505 conn(0x7f60fc0c5ae0 msgr2=0x7f60fc073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:28.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.014+0000 7f610374e640 1 -- 172.21.15.150:0/3910150505 shutdown_connections 2024-09-17T05:07:28.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.014+0000 7f610374e640 1 -- 172.21.15.150:0/3910150505 wait complete. 2024-09-17T05:07:28.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.014+0000 7f610374e640 1 Processor -- start 2024-09-17T05:07:28.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.014+0000 7f610374e640 1 -- start start 2024-09-17T05:07:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f610374e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e8004010 0x7f60fc156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f610374e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60fc14a300 0x7f60fc1eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f610374e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60fc1571a0 0x7f60fc1ebca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f610374e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f60fc160330 con 0x7f60e8004010 2024-09-17T05:07:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f610374e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f60fc1601b0 con 0x7f60fc1571a0 2024-09-17T05:07:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f610374e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f60fc1604b0 con 0x7f60fc14a300 2024-09-17T05:07:28.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f61014ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e8004010 0x7f60fc156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:28.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f61014ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e8004010 0x7f60fc156c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57032/0 (socket says 172.21.15.150:57032) 2024-09-17T05:07:28.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f61014ea640 1 -- 172.21.15.150:0/608982171 learned_addr learned my addr 172.21.15.150:0/608982171 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:28.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f6100ce9640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60fc14a300 0x7f60fc1eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:28.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f6101ceb640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60fc1571a0 0x7f60fc1ebca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:28.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f61014ea640 1 -- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60fc14a300 msgr2=0x7f60fc1eee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:28.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f61014ea640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60fc14a300 0x7f60fc1eee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:28.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f61014ea640 1 -- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60fc1571a0 msgr2=0x7f60fc1ebca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:28.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f61014ea640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60fc1571a0 0x7f60fc1ebca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:28.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f61014ea640 1 -- 172.21.15.150:0/608982171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60fc1eb5b0 con 0x7f60e8004010 2024-09-17T05:07:28.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f61014ea640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e8004010 0x7f60fc156c60 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f60f8011ae0 tx=0x7f60f8011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:28.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f60f1ffb640 1 -- 172.21.15.150:0/608982171 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8004460 con 0x7f60e8004010 2024-09-17T05:07:28.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f60f1ffb640 1 -- 172.21.15.150:0/608982171 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60f800fa00 con 0x7f60e8004010 2024-09-17T05:07:28.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f610374e640 1 -- 172.21.15.150:0/608982171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60fc0acc50 con 0x7f60e8004010 2024-09-17T05:07:28.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f60f1ffb640 1 -- 172.21.15.150:0/608982171 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f800fe50 con 0x7f60e8004010 2024-09-17T05:07:28.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.018+0000 7f610374e640 1 -- 172.21.15.150:0/608982171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60fc0ad210 con 0x7f60e8004010 2024-09-17T05:07:28.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.022+0000 7f610374e640 1 -- 172.21.15.150:0/608982171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60fc151f50 con 0x7f60e8004010 2024-09-17T05:07:28.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.022+0000 7f60f1ffb640 1 -- 172.21.15.150:0/608982171 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f60f8003600 con 0x7f60e8004010 2024-09-17T05:07:28.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.022+0000 7f60f1ffb640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60e0073230 0x7f60e0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:28.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.022+0000 7f60f1ffb640 1 -- 172.21.15.150:0/608982171 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f60f8095dc0 con 0x7f60e8004010 2024-09-17T05:07:28.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.022+0000 7f6100ce9640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60e0073230 0x7f60e0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:28.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.022+0000 7f6100ce9640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60e0073230 0x7f60e0075740 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0ab000 tx=0x7f60ec019d10 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:28.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.026+0000 7f60f1ffb640 1 -- 172.21.15.150:0/608982171 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f60f805fc40 con 0x7f60e8004010 2024-09-17T05:07:28.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 -- 172.21.15.150:0/608982171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60fc069930 con 0x7f60e8004010 2024-09-17T05:07:28.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f60f1ffb640 1 -- 172.21.15.150:0/608982171 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+6536 (secure 0 0 0) 0x7f60f8064670 con 0x7f60e8004010 2024-09-17T05:07:28.274 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:28.274 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":136,"btime":"2024-09-17T05:07:28:102129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:28.275 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 136 2024-09-17T05:07:28.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 -- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60e0073230 msgr2=0x7f60e0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:28.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60e0073230 0x7f60e0075740 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0ab000 tx=0x7f60ec019d10 comp rx=0 tx=0).stop 2024-09-17T05:07:28.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 -- 172.21.15.150:0/608982171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e8004010 msgr2=0x7f60fc156c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:28.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e8004010 0x7f60fc156c60 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f60f8011ae0 tx=0x7f60f8011ee0 comp rx=0 tx=0).stop 2024-09-17T05:07:28.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 -- 172.21.15.150:0/608982171 shutdown_connections 2024-09-17T05:07:28.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f60e0073230 0x7f60e0075740 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:28.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f60fc1571a0 0x7f60fc1ebca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:28.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f60fc14a300 0x7f60fc1eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:28.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 --2- 172.21.15.150:0/608982171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60e8004010 0x7f60fc156c60 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:28.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 -- 172.21.15.150:0/608982171 >> 172.21.15.150:0/608982171 conn(0x7f60fc0c5ae0 msgr2=0x7f60fc070bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:28.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 -- 172.21.15.150:0/608982171 shutdown_connections 2024-09-17T05:07:28.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:28.274+0000 7f610374e640 1 -- 172.21.15.150:0/608982171 wait complete. 2024-09-17T05:07:28.297 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-17T05:07:28.297 INFO:tasks.mds_thrash.fs.[a]:waiting for 119 secs before thrashing 2024-09-17T05:07:28.553 INFO:tasks.ceph.mds.d.smithi176.stderr:2024-09-17T05:07:28.548+0000 7fcd4e581640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:07:32.831 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:07:32.834 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:07:57.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:57.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.761+0000 7ff88c441640 1 -- 172.21.15.150:0/2975003161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff870005260 msgr2=0x7ff87000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:57.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.761+0000 7ff88c441640 1 --2- 172.21.15.150:0/2975003161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff870005260 0x7ff87000b9d0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00f490 tx=0x7ff87c01e450 comp rx=0 tx=0).stop 2024-09-17T05:07:57.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 -- 172.21.15.150:0/2975003161 shutdown_connections 2024-09-17T05:07:57.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 --2- 172.21.15.150:0/2975003161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff870005260 0x7ff87000b9d0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:57.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 -- 172.21.15.150:0/2975003161 >> 172.21.15.150:0/2975003161 conn(0x7ff8840c4a00 msgr2=0x7ff8840c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:57.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 -- 172.21.15.150:0/2975003161 shutdown_connections 2024-09-17T05:07:57.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 -- 172.21.15.150:0/2975003161 wait complete. 2024-09-17T05:07:57.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 Processor -- start 2024-09-17T05:07:57.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 -- start start 2024-09-17T05:07:57.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff870005260 0x7ff8840ad780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:57.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8841540e0 0x7ff8841edaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:57.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88414acf0 0x7ff88414b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:57.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff884070da0 con 0x7ff88414acf0 2024-09-17T05:07:57.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff884070840 con 0x7ff870005260 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88c441640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff8840c1fb0 con 0x7ff8841540e0 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88a9de640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88414acf0 0x7ff88414b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88a9de640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88414acf0 0x7ff88414b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58240/0 (socket says 172.21.15.150:58240) 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88a9de640 1 -- 172.21.15.150:0/1523446495 learned_addr learned my addr 172.21.15.150:0/1523446495 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88a1dd640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff870005260 0x7ff8840ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff8899dc640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8841540e0 0x7ff8841edaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88a9de640 1 -- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8841540e0 msgr2=0x7ff8841edaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88a9de640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8841540e0 0x7ff8841edaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88a9de640 1 -- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff870005260 msgr2=0x7ff8840ad780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88a9de640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff870005260 0x7ff8840ad780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88a9de640 1 -- 172.21.15.150:0/1523446495 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88414b640 con 0x7ff88414acf0 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.765+0000 7ff88a9de640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88414acf0 0x7ff88414b100 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff87c01ecf0 tx=0x7ff87c0accb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.769+0000 7ff87affd640 1 -- 172.21.15.150:0/1523446495 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0bc070 con 0x7ff88414acf0 2024-09-17T05:07:57.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.769+0000 7ff87affd640 1 -- 172.21.15.150:0/1523446495 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff87c00f240 con 0x7ff88414acf0 2024-09-17T05:07:57.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.769+0000 7ff88c441640 1 -- 172.21.15.150:0/1523446495 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8841ee610 con 0x7ff88414acf0 2024-09-17T05:07:57.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.769+0000 7ff87affd640 1 -- 172.21.15.150:0/1523446495 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c004cb0 con 0x7ff88414acf0 2024-09-17T05:07:57.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.769+0000 7ff88c441640 1 -- 172.21.15.150:0/1523446495 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff884155680 con 0x7ff88414acf0 2024-09-17T05:07:57.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.769+0000 7ff88c441640 1 -- 172.21.15.150:0/1523446495 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff88414f550 con 0x7ff88414acf0 2024-09-17T05:07:57.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.769+0000 7ff87affd640 1 -- 172.21.15.150:0/1523446495 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff87c0bf070 con 0x7ff88414acf0 2024-09-17T05:07:57.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.769+0000 7ff87affd640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff868073200 0x7ff868075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:57.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.769+0000 7ff87affd640 1 -- 172.21.15.150:0/1523446495 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7ff87c12fb70 con 0x7ff88414acf0 2024-09-17T05:07:57.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.769+0000 7ff88a1dd640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff868073200 0x7ff868075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:57.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.773+0000 7ff88a1dd640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff868073200 0x7ff868075710 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff8800065d0 tx=0x7ff880006d20 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:57.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:57.777+0000 7ff87affd640 1 -- 172.21.15.150:0/1523446495 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff87c0181a0 con 0x7ff88414acf0 2024-09-17T05:07:58.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.097+0000 7ff88c441640 1 -- 172.21.15.150:0/1523446495 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8840c68c0 con 0x7ff88414acf0 2024-09-17T05:07:58.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff87affd640 1 -- 172.21.15.150:0/1523446495 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+6536 (secure 0 0 0) 0x7ff87c0f99b0 con 0x7ff88414acf0 2024-09-17T05:07:58.098 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:58.098 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":136,"btime":"2024-09-17T05:07:28:102129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:58.099 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 136 2024-09-17T05:07:58.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff88c441640 1 -- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff868073200 msgr2=0x7ff868075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:58.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff88c441640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff868073200 0x7ff868075710 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff8800065d0 tx=0x7ff880006d20 comp rx=0 tx=0).stop 2024-09-17T05:07:58.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff88c441640 1 -- 172.21.15.150:0/1523446495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88414acf0 msgr2=0x7ff88414b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:58.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff88c441640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88414acf0 0x7ff88414b100 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff87c01ecf0 tx=0x7ff87c0accb0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff88c441640 1 -- 172.21.15.150:0/1523446495 shutdown_connections 2024-09-17T05:07:58.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff88c441640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff868073200 0x7ff868075710 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff88c441640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff88414acf0 0x7ff88414b100 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff88c441640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff8841540e0 0x7ff8841edaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff88c441640 1 --2- 172.21.15.150:0/1523446495 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff870005260 0x7ff8840ad780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff88c441640 1 -- 172.21.15.150:0/1523446495 >> 172.21.15.150:0/1523446495 conn(0x7ff8840c4a00 msgr2=0x7ff8840c2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:58.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff88c441640 1 -- 172.21.15.150:0/1523446495 shutdown_connections 2024-09-17T05:07:58.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.101+0000 7ff88c441640 1 -- 172.21.15.150:0/1523446495 wait complete. 2024-09-17T05:07:58.121 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-17T05:07:58.121 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T05:07:58.122 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:07:58.133 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T05:07:58.133 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:07:58.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.357+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3597514197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e340c6cb0 msgr2=0x7f6e340ae340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:58.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.357+0000 7f6e3bbd5640 1 --2- 172.21.15.150:0/3597514197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e340c6cb0 0x7f6e340ae340 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f6e340aefc0 tx=0x7f6e3001d290 comp rx=0 tx=0).stop 2024-09-17T05:07:58.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.357+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3597514197 shutdown_connections 2024-09-17T05:07:58.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.357+0000 7f6e3bbd5640 1 --2- 172.21.15.150:0/3597514197 >> v2:172.21.15.176:3300/0 conn(0x7f6e34151980 0x7f6e34151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.357+0000 7f6e3bbd5640 1 --2- 172.21.15.150:0/3597514197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e340c6cb0 0x7f6e340ae340 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.357+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3597514197 >> 172.21.15.150:0/3597514197 conn(0x7f6e340aab60 msgr2=0x7f6e340aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:58.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.357+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3597514197 shutdown_connections 2024-09-17T05:07:58.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.357+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3597514197 wait complete. 2024-09-17T05:07:58.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e3bbd5640 1 Processor -- start 2024-09-17T05:07:58.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e3bbd5640 1 -- start start 2024-09-17T05:07:58.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e3bbd5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e340a85b0 0x7f6e34144f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e3bbd5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340c6cb0 0x7f6e34152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e3bbd5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e34151980 0x7f6e341f0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e3bbd5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6e340aea00 con 0x7f6e340a85b0 2024-09-17T05:07:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e3bbd5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6e340ae880 con 0x7f6e340c6cb0 2024-09-17T05:07:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e3bbd5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6e34155170 con 0x7f6e34151980 2024-09-17T05:07:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e39971640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e340a85b0 0x7f6e34144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e39971640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e340a85b0 0x7f6e34144f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58250/0 (socket says 172.21.15.150:58250) 2024-09-17T05:07:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e39971640 1 -- 172.21.15.150:0/3465464574 learned_addr learned my addr 172.21.15.150:0/3465464574 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:07:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e3a172640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e34151980 0x7f6e341f0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e39170640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340c6cb0 0x7f6e34152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e39971640 1 -- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e34151980 msgr2=0x7f6e341f0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e39971640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e34151980 0x7f6e341f0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e39971640 1 -- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340c6cb0 msgr2=0x7f6e34152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e39971640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340c6cb0 0x7f6e34152db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e39971640 1 -- 172.21.15.150:0/3465464574 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e34143680 con 0x7f6e340a85b0 2024-09-17T05:07:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e39971640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e340a85b0 0x7f6e34144f00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6e300b0040 tx=0x7f6e300a5600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e2a7fc640 1 -- 172.21.15.150:0/3465464574 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e300b2040 con 0x7f6e340a85b0 2024-09-17T05:07:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e2a7fc640 1 -- 172.21.15.150:0/3465464574 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e300043b0 con 0x7f6e340a85b0 2024-09-17T05:07:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3465464574 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e34145440 con 0x7f6e340a85b0 2024-09-17T05:07:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e2a7fc640 1 -- 172.21.15.150:0/3465464574 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30005280 con 0x7f6e340a85b0 2024-09-17T05:07:58.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3465464574 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e341433e0 con 0x7f6e340a85b0 2024-09-17T05:07:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.361+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3465464574 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e340ae340 con 0x7f6e340a85b0 2024-09-17T05:07:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.365+0000 7f6e2a7fc640 1 -- 172.21.15.150:0/3465464574 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6e30005420 con 0x7f6e340a85b0 2024-09-17T05:07:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.365+0000 7f6e2a7fc640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6e18073230 0x7f6e18075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:07:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.365+0000 7f6e2a7fc640 1 -- 172.21.15.150:0/3465464574 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f6e3012eeb0 con 0x7f6e340a85b0 2024-09-17T05:07:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.365+0000 7f6e39170640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6e18073230 0x7f6e18075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:07:58.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.365+0000 7f6e39170640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6e18073230 0x7f6e18075740 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6e3415ed60 tx=0x7f6e24009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:07:58.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.369+0000 7f6e2a7fc640 1 -- 172.21.15.150:0/3465464574 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6e3015ddb0 con 0x7f6e340a85b0 2024-09-17T05:07:58.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.581+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3465464574 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e3415eab0 con 0x7f6e340a85b0 2024-09-17T05:07:58.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.581+0000 7f6e2a7fc640 1 -- 172.21.15.150:0/3465464574 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+6536 (secure 0 0 0) 0x7f6e34145440 con 0x7f6e340a85b0 2024-09-17T05:07:58.580 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:07:58.581 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":136,"btime":"2024-09-17T05:07:28:102129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:07:58.581 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 136 2024-09-17T05:07:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.585+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6e18073230 msgr2=0x7f6e18075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.585+0000 7f6e3bbd5640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6e18073230 0x7f6e18075740 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6e3415ed60 tx=0x7f6e24009040 comp rx=0 tx=0).stop 2024-09-17T05:07:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.585+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3465464574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e340a85b0 msgr2=0x7f6e34144f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:07:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.585+0000 7f6e3bbd5640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e340a85b0 0x7f6e34144f00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6e300b0040 tx=0x7f6e300a5600 comp rx=0 tx=0).stop 2024-09-17T05:07:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.585+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3465464574 shutdown_connections 2024-09-17T05:07:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.585+0000 7f6e3bbd5640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6e18073230 0x7f6e18075740 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.585+0000 7f6e3bbd5640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340c6cb0 0x7f6e34152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.585+0000 7f6e3bbd5640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e34151980 0x7f6e341f0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.585+0000 7f6e3bbd5640 1 --2- 172.21.15.150:0/3465464574 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e340a85b0 0x7f6e34144f00 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:07:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.585+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3465464574 >> 172.21.15.150:0/3465464574 conn(0x7f6e340aab60 msgr2=0x7f6e340c7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:07:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.585+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3465464574 shutdown_connections 2024-09-17T05:07:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:07:58.585+0000 7f6e3bbd5640 1 -- 172.21.15.150:0/3465464574 wait complete. 2024-09-17T05:07:58.602 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:08:00.603 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:00.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.777+0000 7f4d87788640 1 -- 172.21.15.150:0/3489591097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d800c7eb0 msgr2=0x7f4d801540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:00.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.777+0000 7f4d87788640 1 --2- 172.21.15.150:0/3489591097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d800c7eb0 0x7f4d801540b0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f4d800755f0 tx=0x7f4d7c01c8e0 comp rx=0 tx=0).stop 2024-09-17T05:08:00.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.777+0000 7f4d87788640 1 -- 172.21.15.150:0/3489591097 shutdown_connections 2024-09-17T05:08:00.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.777+0000 7f4d87788640 1 --2- 172.21.15.150:0/3489591097 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d80149d70 0x7f4d8014a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:00.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.777+0000 7f4d87788640 1 --2- 172.21.15.150:0/3489591097 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d800c7eb0 0x7f4d801540b0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:00.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.777+0000 7f4d87788640 1 -- 172.21.15.150:0/3489591097 >> 172.21.15.150:0/3489591097 conn(0x7f4d800aab60 msgr2=0x7f4d800aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:00.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.777+0000 7f4d87788640 1 -- 172.21.15.150:0/3489591097 shutdown_connections 2024-09-17T05:08:00.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.777+0000 7f4d87788640 1 -- 172.21.15.150:0/3489591097 wait complete. 2024-09-17T05:08:00.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d87788640 1 Processor -- start 2024-09-17T05:08:00.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d87788640 1 -- start start 2024-09-17T05:08:00.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d87788640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d800c7eb0 0x7f4d80150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:00.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d87788640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d800c8330 0x7f4d801f1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:00.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d87788640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d80149d70 0x7f4d80147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:00.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d87788640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4d801568d0 con 0x7f4d800c8330 2024-09-17T05:08:00.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d87788640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4d80156750 con 0x7f4d800c7eb0 2024-09-17T05:08:00.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d87788640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4d800c8b50 con 0x7f4d80149d70 2024-09-17T05:08:00.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d84d23640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d800c8330 0x7f4d801f1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:00.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d84d23640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d800c8330 0x7f4d801f1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36660/0 (socket says 172.21.15.150:36660) 2024-09-17T05:08:00.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d84d23640 1 -- 172.21.15.150:0/3434105404 learned_addr learned my addr 172.21.15.150:0/3434105404 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:00.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d85524640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d800c7eb0 0x7f4d80150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:00.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d85d25640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d80149d70 0x7f4d80147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:00.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d84d23640 1 -- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d80149d70 msgr2=0x7f4d80147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:00.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d84d23640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d80149d70 0x7f4d80147fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:00.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d84d23640 1 -- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d800c7eb0 msgr2=0x7f4d80150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:00.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d84d23640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d800c7eb0 0x7f4d80150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:00.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d84d23640 1 -- 172.21.15.150:0/3434105404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d80149200 con 0x7f4d800c8330 2024-09-17T05:08:00.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d85524640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d800c7eb0 0x7f4d80150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:08:00.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d84d23640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d800c8330 0x7f4d801f1490 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000d950 tx=0x7f4d7000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:00.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d75ffb640 1 -- 172.21.15.150:0/3434105404 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70017070 con 0x7f4d800c8330 2024-09-17T05:08:00.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d75ffb640 1 -- 172.21.15.150:0/3434105404 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d70009d20 con 0x7f4d800c8330 2024-09-17T05:08:00.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.781+0000 7f4d87788640 1 -- 172.21.15.150:0/3434105404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d80148d20 con 0x7f4d800c8330 2024-09-17T05:08:00.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.785+0000 7f4d75ffb640 1 -- 172.21.15.150:0/3434105404 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d700106c0 con 0x7f4d800c8330 2024-09-17T05:08:00.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.785+0000 7f4d87788640 1 -- 172.21.15.150:0/3434105404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d80149b00 con 0x7f4d800c8330 2024-09-17T05:08:00.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.785+0000 7f4d87788640 1 -- 172.21.15.150:0/3434105404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d8014e270 con 0x7f4d800c8330 2024-09-17T05:08:00.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.785+0000 7f4d75ffb640 1 -- 172.21.15.150:0/3434105404 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4d7001a070 con 0x7f4d800c8330 2024-09-17T05:08:00.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.785+0000 7f4d75ffb640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d64073230 0x7f4d64075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:00.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.785+0000 7f4d75ffb640 1 -- 172.21.15.150:0/3434105404 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f4d70094fc0 con 0x7f4d800c8330 2024-09-17T05:08:00.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.785+0000 7f4d85524640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d64073230 0x7f4d64075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:00.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.785+0000 7f4d85524640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d64073230 0x7f4d64075740 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0af000 tx=0x7f4d7c01b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:00.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:00.793+0000 7f4d75ffb640 1 -- 172.21.15.150:0/3434105404 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4d7005f3c0 con 0x7f4d800c8330 2024-09-17T05:08:01.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.037+0000 7f4d87788640 1 -- 172.21.15.150:0/3434105404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d801540b0 con 0x7f4d800c8330 2024-09-17T05:08:01.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.037+0000 7f4d75ffb640 1 -- 172.21.15.150:0/3434105404 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+6536 (secure 0 0 0) 0x7f4d7005ecc0 con 0x7f4d800c8330 2024-09-17T05:08:01.035 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:01.035 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":136,"btime":"2024-09-17T05:07:28:102129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:08:01.036 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 136 2024-09-17T05:08:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.037+0000 7f4d87788640 1 -- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d64073230 msgr2=0x7f4d64075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.037+0000 7f4d87788640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d64073230 0x7f4d64075740 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0af000 tx=0x7f4d7c01b120 comp rx=0 tx=0).stop 2024-09-17T05:08:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.037+0000 7f4d87788640 1 -- 172.21.15.150:0/3434105404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d800c8330 msgr2=0x7f4d801f1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.037+0000 7f4d87788640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d800c8330 0x7f4d801f1490 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000d950 tx=0x7f4d7000de10 comp rx=0 tx=0).stop 2024-09-17T05:08:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.037+0000 7f4d87788640 1 -- 172.21.15.150:0/3434105404 shutdown_connections 2024-09-17T05:08:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.037+0000 7f4d87788640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4d64073230 0x7f4d64075740 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.037+0000 7f4d87788640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4d80149d70 0x7f4d80147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.037+0000 7f4d87788640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4d800c8330 0x7f4d801f1490 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.037+0000 7f4d87788640 1 --2- 172.21.15.150:0/3434105404 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4d800c7eb0 0x7f4d80150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.037+0000 7f4d87788640 1 -- 172.21.15.150:0/3434105404 >> 172.21.15.150:0/3434105404 conn(0x7f4d800aab60 msgr2=0x7f4d8006e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.041+0000 7f4d87788640 1 -- 172.21.15.150:0/3434105404 shutdown_connections 2024-09-17T05:08:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:01.041+0000 7f4d87788640 1 -- 172.21.15.150:0/3434105404 wait complete. 2024-09-17T05:08:01.056 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:08:02.891 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:08:02.894 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:08:03.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:03.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.225+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3836133184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fdc0c7e20 msgr2=0x7f3fdc071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:03.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.225+0000 7f3fe20bf640 1 --2- 172.21.15.150:0/3836133184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fdc0c7e20 0x7f3fdc071b10 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc073350 tx=0x7f3fd401caf0 comp rx=0 tx=0).stop 2024-09-17T05:08:03.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.225+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3836133184 shutdown_connections 2024-09-17T05:08:03.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.225+0000 7f3fe20bf640 1 --2- 172.21.15.150:0/3836133184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fdc0c7e20 0x7f3fdc071b10 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:03.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.225+0000 7f3fe20bf640 1 --2- 172.21.15.150:0/3836133184 >> v2:172.21.15.176:3300/0 conn(0x7f3fdc1557b0 0x7f3fdc0c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:03.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.225+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3836133184 >> 172.21.15.150:0/3836133184 conn(0x7f3fdc0aaa80 msgr2=0x7f3fdc0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:03.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.225+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3836133184 shutdown_connections 2024-09-17T05:08:03.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.225+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3836133184 wait complete. 2024-09-17T05:08:03.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.225+0000 7f3fe20bf640 1 Processor -- start 2024-09-17T05:08:03.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fe20bf640 1 -- start start 2024-09-17T05:08:03.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fe20bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fdc0c7580 0x7f3fdc0be090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fe20bf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3fdc0c7e20 0x7f3fdc152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fe20bf640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3fdc1557b0 0x7f3fdc0c0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fe20bf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3fdc161a40 con 0x7f3fdc0c7580 2024-09-17T05:08:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fe20bf640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3fdc1618c0 con 0x7f3fdc1557b0 2024-09-17T05:08:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fe20bf640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3fdc161bc0 con 0x7f3fdc0c7e20 2024-09-17T05:08:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fdb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fdc0c7580 0x7f3fdc0be090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fdb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fdc0c7580 0x7f3fdc0be090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36670/0 (socket says 172.21.15.150:36670) 2024-09-17T05:08:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fdb7fe640 1 -- 172.21.15.150:0/3128792181 learned_addr learned my addr 172.21.15.150:0/3128792181 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fdbfff640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3fdc1557b0 0x7f3fdc0c0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fd3fff640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3fdc0c7e20 0x7f3fdc152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:03.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fdb7fe640 1 -- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3fdc0c7e20 msgr2=0x7f3fdc152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fdb7fe640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3fdc0c7e20 0x7f3fdc152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fdb7fe640 1 -- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3fdc1557b0 msgr2=0x7f3fdc0c0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fdb7fe640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3fdc1557b0 0x7f3fdc0c0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fdb7fe640 1 -- 172.21.15.150:0/3128792181 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fdc14c280 con 0x7f3fdc0c7580 2024-09-17T05:08:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fdb7fe640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fdc0c7580 0x7f3fdc0be090 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00ca20 tx=0x7f3fcc00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fd8ff9640 1 -- 172.21.15.150:0/3128792181 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc007c70 con 0x7f3fdc0c7580 2024-09-17T05:08:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fd8ff9640 1 -- 172.21.15.150:0/3128792181 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3fcc00f450 con 0x7f3fdc0c7580 2024-09-17T05:08:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3128792181 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fdc15e2d0 con 0x7f3fdc0c7580 2024-09-17T05:08:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fd8ff9640 1 -- 172.21.15.150:0/3128792181 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc00f8a0 con 0x7f3fdc0c7580 2024-09-17T05:08:03.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3128792181 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fdc0be740 con 0x7f3fdc0c7580 2024-09-17T05:08:03.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.229+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3128792181 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fdc0be9a0 con 0x7f3fdc0c7580 2024-09-17T05:08:03.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.233+0000 7f3fd8ff9640 1 -- 172.21.15.150:0/3128792181 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3fcc018400 con 0x7f3fdc0c7580 2024-09-17T05:08:03.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.233+0000 7f3fd8ff9640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3fbc073230 0x7f3fbc075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:03.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.233+0000 7f3fd8ff9640 1 -- 172.21.15.150:0/3128792181 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f3fcc095890 con 0x7f3fdc0c7580 2024-09-17T05:08:03.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.233+0000 7f3fd3fff640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3fbc073230 0x7f3fbc075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:03.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.233+0000 7f3fd3fff640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3fbc073230 0x7f3fbc075740 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc06dbb0 tx=0x7f3fc4009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:03.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.237+0000 7f3fd8ff9640 1 -- 172.21.15.150:0/3128792181 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3fcc014030 con 0x7f3fdc0c7580 2024-09-17T05:08:03.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.469+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3128792181 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fdc071b10 con 0x7f3fdc0c7580 2024-09-17T05:08:03.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.469+0000 7f3fd8ff9640 1 -- 172.21.15.150:0/3128792181 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+6536 (secure 0 0 0) 0x7f3fcc05f7e0 con 0x7f3fdc0c7580 2024-09-17T05:08:03.468 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:03.468 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":136,"btime":"2024-09-17T05:07:28:102129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:08:03.468 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 136 2024-09-17T05:08:03.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.473+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3fbc073230 msgr2=0x7f3fbc075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:03.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.473+0000 7f3fe20bf640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3fbc073230 0x7f3fbc075740 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc06dbb0 tx=0x7f3fc4009040 comp rx=0 tx=0).stop 2024-09-17T05:08:03.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.473+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3128792181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fdc0c7580 msgr2=0x7f3fdc0be090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:03.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.473+0000 7f3fe20bf640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fdc0c7580 0x7f3fdc0be090 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00ca20 tx=0x7f3fcc00cee0 comp rx=0 tx=0).stop 2024-09-17T05:08:03.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.473+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3128792181 shutdown_connections 2024-09-17T05:08:03.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.473+0000 7f3fe20bf640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3fbc073230 0x7f3fbc075740 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:03.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.473+0000 7f3fe20bf640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3fdc1557b0 0x7f3fdc0c0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:03.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.473+0000 7f3fe20bf640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3fdc0c7e20 0x7f3fdc152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:03.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.473+0000 7f3fe20bf640 1 --2- 172.21.15.150:0/3128792181 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fdc0c7580 0x7f3fdc0be090 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:03.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.473+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3128792181 >> 172.21.15.150:0/3128792181 conn(0x7f3fdc0aaa80 msgr2=0x7f3fdc154980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:03.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.473+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3128792181 shutdown_connections 2024-09-17T05:08:03.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:03.473+0000 7f3fe20bf640 1 -- 172.21.15.150:0/3128792181 wait complete. 2024-09-17T05:08:03.490 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:08:05.491 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:05.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.669+0000 7fa2df7ed640 1 -- 172.21.15.150:0/1996786382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2d81546f0 msgr2=0x7fa2d8154b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:05.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.669+0000 7fa2df7ed640 1 --2- 172.21.15.150:0/1996786382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2d81546f0 0x7fa2d8154b00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009bc0 tx=0x7fa2d001ca80 comp rx=0 tx=0).stop 2024-09-17T05:08:05.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.669+0000 7fa2df7ed640 1 -- 172.21.15.150:0/1996786382 shutdown_connections 2024-09-17T05:08:05.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.669+0000 7fa2df7ed640 1 --2- 172.21.15.150:0/1996786382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2d81546f0 0x7fa2d8154b00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:05.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.669+0000 7fa2df7ed640 1 --2- 172.21.15.150:0/1996786382 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2d806d0d0 0x7fa2d80a8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:05.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.669+0000 7fa2df7ed640 1 -- 172.21.15.150:0/1996786382 >> 172.21.15.150:0/1996786382 conn(0x7fa2d80aab60 msgr2=0x7fa2d80aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:05.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.669+0000 7fa2df7ed640 1 -- 172.21.15.150:0/1996786382 shutdown_connections 2024-09-17T05:08:05.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.669+0000 7fa2df7ed640 1 -- 172.21.15.150:0/1996786382 wait complete. 2024-09-17T05:08:05.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.669+0000 7fa2df7ed640 1 Processor -- start 2024-09-17T05:08:05.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.669+0000 7fa2df7ed640 1 -- start start 2024-09-17T05:08:05.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2df7ed640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2d806d0d0 0x7fa2d814d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:05.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2df7ed640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2d80c6cb0 0x7fa2d8146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:05.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2df7ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d81546f0 0x7fa2d81453b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:05.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2df7ed640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8161c00 con 0x7fa2d81546f0 2024-09-17T05:08:05.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2df7ed640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8161a80 con 0x7fa2d806d0d0 2024-09-17T05:08:05.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2df7ed640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa2d8161d80 con 0x7fa2d80c6cb0 2024-09-17T05:08:05.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2ddd8a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d81546f0 0x7fa2d81453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:05.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2ddd8a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d81546f0 0x7fa2d81453b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36672/0 (socket says 172.21.15.150:36672) 2024-09-17T05:08:05.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2ddd8a640 1 -- 172.21.15.150:0/556088163 learned_addr learned my addr 172.21.15.150:0/556088163 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:05.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2dd589640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2d806d0d0 0x7fa2d814d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:05.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2dcd88640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2d80c6cb0 0x7fa2d8146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:05.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2ddd8a640 1 -- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2d80c6cb0 msgr2=0x7fa2d8146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:05.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2ddd8a640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2d80c6cb0 0x7fa2d8146a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:05.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2ddd8a640 1 -- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2d806d0d0 msgr2=0x7fa2d814d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:05.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2ddd8a640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2d806d0d0 0x7fa2d814d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:05.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2ddd8a640 1 -- 172.21.15.150:0/556088163 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d814cae0 con 0x7fa2d81546f0 2024-09-17T05:08:05.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2dcd88640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2d80c6cb0 0x7fa2d8146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:08:05.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2ddd8a640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d81546f0 0x7fa2d81453b0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009b10 tx=0x7fa2d0003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:05.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2cdffb640 1 -- 172.21.15.150:0/556088163 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d00bb070 con 0x7fa2d81546f0 2024-09-17T05:08:05.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2cdffb640 1 -- 172.21.15.150:0/556088163 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2d0004d40 con 0x7fa2d81546f0 2024-09-17T05:08:05.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2cdffb640 1 -- 172.21.15.150:0/556088163 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d00b2040 con 0x7fa2d81546f0 2024-09-17T05:08:05.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2df7ed640 1 -- 172.21.15.150:0/556088163 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2d8146fc0 con 0x7fa2d81546f0 2024-09-17T05:08:05.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.673+0000 7fa2df7ed640 1 -- 172.21.15.150:0/556088163 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2d814c530 con 0x7fa2d81546f0 2024-09-17T05:08:05.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.677+0000 7fa2df7ed640 1 -- 172.21.15.150:0/556088163 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2d814c7c0 con 0x7fa2d81546f0 2024-09-17T05:08:05.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.677+0000 7fa2cdffb640 1 -- 172.21.15.150:0/556088163 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa2d0005290 con 0x7fa2d81546f0 2024-09-17T05:08:05.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.677+0000 7fa2cdffb640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa2bc073230 0x7fa2bc075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:05.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.677+0000 7fa2cdffb640 1 -- 172.21.15.150:0/556088163 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7fa2d012eb50 con 0x7fa2d81546f0 2024-09-17T05:08:05.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.677+0000 7fa2dd589640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa2bc073230 0x7fa2bc075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:05.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.677+0000 7fa2dd589640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa2bc073230 0x7fa2bc075740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4007fd0 tx=0x7fa2d4009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:05.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.681+0000 7fa2cdffb640 1 -- 172.21.15.150:0/556088163 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa2d0016790 con 0x7fa2d81546f0 2024-09-17T05:08:05.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 -- 172.21.15.150:0/556088163 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2d814c170 con 0x7fa2d81546f0 2024-09-17T05:08:05.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2cdffb640 1 -- 172.21.15.150:0/556088163 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+6536 (secure 0 0 0) 0x7fa2d00f8990 con 0x7fa2d81546f0 2024-09-17T05:08:05.923 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:05.923 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":136,"btime":"2024-09-17T05:07:28:102129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:08:05.923 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 136 2024-09-17T05:08:05.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 -- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa2bc073230 msgr2=0x7fa2bc075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:05.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa2bc073230 0x7fa2bc075740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4007fd0 tx=0x7fa2d4009040 comp rx=0 tx=0).stop 2024-09-17T05:08:05.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 -- 172.21.15.150:0/556088163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d81546f0 msgr2=0x7fa2d81453b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:05.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d81546f0 0x7fa2d81453b0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009b10 tx=0x7fa2d0003c30 comp rx=0 tx=0).stop 2024-09-17T05:08:05.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 -- 172.21.15.150:0/556088163 shutdown_connections 2024-09-17T05:08:05.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa2bc073230 0x7fa2bc075740 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:05.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d81546f0 0x7fa2d81453b0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:05.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2d80c6cb0 0x7fa2d8146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:05.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 --2- 172.21.15.150:0/556088163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2d806d0d0 0x7fa2d814d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:05.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 -- 172.21.15.150:0/556088163 >> 172.21.15.150:0/556088163 conn(0x7fa2d80aab60 msgr2=0x7fa2d814ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:05.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 -- 172.21.15.150:0/556088163 shutdown_connections 2024-09-17T05:08:05.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:05.925+0000 7fa2df7ed640 1 -- 172.21.15.150:0/556088163 wait complete. 2024-09-17T05:08:05.946 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:08:07.947 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:08.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.145+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1759771917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ad00c4180 msgr2=0x7f4ad01542c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:08.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.145+0000 7f4ad7fa8640 1 --2- 172.21.15.150:0/1759771917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ad00c4180 0x7f4ad01542c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009900 tx=0x7f4acc01c7c0 comp rx=0 tx=0).stop 2024-09-17T05:08:08.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.145+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1759771917 shutdown_connections 2024-09-17T05:08:08.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.145+0000 7f4ad7fa8640 1 --2- 172.21.15.150:0/1759771917 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ad0150e90 0x7f4ad0156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:08.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.145+0000 7f4ad7fa8640 1 --2- 172.21.15.150:0/1759771917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ad00c4180 0x7f4ad01542c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:08.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.145+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1759771917 >> 172.21.15.150:0/1759771917 conn(0x7f4ad00c5ae0 msgr2=0x7f4ad0073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:08.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.145+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1759771917 shutdown_connections 2024-09-17T05:08:08.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.145+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1759771917 wait complete. 2024-09-17T05:08:08.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.145+0000 7f4ad7fa8640 1 Processor -- start 2024-09-17T05:08:08.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.145+0000 7f4ad7fa8640 1 -- start start 2024-09-17T05:08:08.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad7fa8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ad00aa9b0 0x7f4ad00be240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:08.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad7fa8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ad00c4180 0x7f4ad01edcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:08.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad7fa8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ad0150e90 0x7f4ad0149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:08.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad7fa8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad0157420 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad7fa8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad00adbc0 con 0x7f4ad00c4180 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad7fa8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4ad006e860 con 0x7f4ad0150e90 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad5d44640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ad00aa9b0 0x7f4ad00be240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad5d44640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ad00aa9b0 0x7f4ad00be240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36686/0 (socket says 172.21.15.150:36686) 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad5d44640 1 -- 172.21.15.150:0/1353070774 learned_addr learned my addr 172.21.15.150:0/1353070774 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad5543640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ad00c4180 0x7f4ad01edcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad6545640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ad0150e90 0x7f4ad0149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad5d44640 1 -- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ad0150e90 msgr2=0x7f4ad0149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad5d44640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ad0150e90 0x7f4ad0149e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad5d44640 1 -- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ad00c4180 msgr2=0x7f4ad01edcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad5d44640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ad00c4180 0x7f4ad01edcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad5d44640 1 -- 172.21.15.150:0/1353070774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad014ac60 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad5d44640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ad00aa9b0 0x7f4ad00be240 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f4acc0b0040 tx=0x7f4acc005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ac67fc640 1 -- 172.21.15.150:0/1353070774 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4acc0b2040 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1353070774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ad00c2640 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ac67fc640 1 -- 172.21.15.150:0/1353070774 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4acc0a5c30 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ac67fc640 1 -- 172.21.15.150:0/1353070774 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4acc005020 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1353070774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ad01ed4b0 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1353070774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ad01542c0 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ac67fc640 1 -- 172.21.15.150:0/1353070774 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4acc0bd060 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ac67fc640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4ab4073200 0x7f4ab4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:08.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.149+0000 7f4ac67fc640 1 -- 172.21.15.150:0/1353070774 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f4acc12e520 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.153+0000 7f4ad5543640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4ab4073200 0x7f4ab4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:08.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.153+0000 7f4ad5543640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4ab4073200 0x7f4ab4075710 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4ad014d9e0 tx=0x7f4ac00091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:08.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.157+0000 7f4ac67fc640 1 -- 172.21.15.150:0/1353070774 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4acc0164d0 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.425+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1353070774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ad0151d10 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.425+0000 7f4ac67fc640 1 -- 172.21.15.150:0/1353070774 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+6536 (secure 0 0 0) 0x7f4acc0f88a0 con 0x7f4ad00aa9b0 2024-09-17T05:08:08.425 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:08.425 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":136,"btime":"2024-09-17T05:07:28:102129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:08:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 136 2024-09-17T05:08:08.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.429+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4ab4073200 msgr2=0x7f4ab4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:08.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.429+0000 7f4ad7fa8640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4ab4073200 0x7f4ab4075710 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4ad014d9e0 tx=0x7f4ac00091e0 comp rx=0 tx=0).stop 2024-09-17T05:08:08.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.429+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1353070774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ad00aa9b0 msgr2=0x7f4ad00be240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:08.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.429+0000 7f4ad7fa8640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ad00aa9b0 0x7f4ad00be240 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f4acc0b0040 tx=0x7f4acc005de0 comp rx=0 tx=0).stop 2024-09-17T05:08:08.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.429+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1353070774 shutdown_connections 2024-09-17T05:08:08.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.429+0000 7f4ad7fa8640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4ab4073200 0x7f4ab4075710 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:08.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.429+0000 7f4ad7fa8640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4ad00c4180 0x7f4ad01edcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:08.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.429+0000 7f4ad7fa8640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4ad0150e90 0x7f4ad0149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:08.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.429+0000 7f4ad7fa8640 1 --2- 172.21.15.150:0/1353070774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ad00aa9b0 0x7f4ad00be240 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:08.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.429+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1353070774 >> 172.21.15.150:0/1353070774 conn(0x7f4ad00c5ae0 msgr2=0x7f4ad01515d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:08.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.429+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1353070774 shutdown_connections 2024-09-17T05:08:08.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:08.429+0000 7f4ad7fa8640 1 -- 172.21.15.150:0/1353070774 wait complete. 2024-09-17T05:08:08.452 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:08:10.453 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:10.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.649+0000 7f2d108e7640 1 -- 172.21.15.150:0/2553908620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d0c0c7cd0 msgr2=0x7f2d0c0c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:10.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.649+0000 7f2d108e7640 1 --2- 172.21.15.150:0/2553908620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d0c0c7cd0 0x7f2d0c0c80e0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f2d04009c80 tx=0x7f2d0401cda0 comp rx=0 tx=0).stop 2024-09-17T05:08:10.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.649+0000 7f2d108e7640 1 -- 172.21.15.150:0/2553908620 shutdown_connections 2024-09-17T05:08:10.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.649+0000 7f2d108e7640 1 --2- 172.21.15.150:0/2553908620 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d0c0c7cd0 0x7f2d0c0c80e0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:10.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.649+0000 7f2d108e7640 1 --2- 172.21.15.150:0/2553908620 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0c072ea0 0x7f2d0c0732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:10.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.649+0000 7f2d108e7640 1 -- 172.21.15.150:0/2553908620 >> 172.21.15.150:0/2553908620 conn(0x7f2d0c0abb00 msgr2=0x7f2d0c0abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:10.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.649+0000 7f2d108e7640 1 -- 172.21.15.150:0/2553908620 shutdown_connections 2024-09-17T05:08:10.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.649+0000 7f2d108e7640 1 -- 172.21.15.150:0/2553908620 wait complete. 2024-09-17T05:08:10.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d108e7640 1 Processor -- start 2024-09-17T05:08:10.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d108e7640 1 -- start start 2024-09-17T05:08:10.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d108e7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d0c072ea0 0x7f2d0c156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:10.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d108e7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0c0c7cd0 0x7f2d0c14e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:10.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d108e7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0c0c97a0 0x7f2d0c14c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:10.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d108e7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0c161910 con 0x7f2d0c072ea0 2024-09-17T05:08:10.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d108e7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0c161790 con 0x7f2d0c0c7cd0 2024-09-17T05:08:10.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d108e7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2d0c161a90 con 0x7f2d0c0c97a0 2024-09-17T05:08:10.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d0a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d0c072ea0 0x7f2d0c156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:10.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d0a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d0c072ea0 0x7f2d0c156030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45698/0 (socket says 172.21.15.150:45698) 2024-09-17T05:08:10.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d0a59c640 1 -- 172.21.15.150:0/2338646007 learned_addr learned my addr 172.21.15.150:0/2338646007 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:10.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d09d9b640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0c0c7cd0 0x7f2d0c14e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:10.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d0ad9d640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0c0c97a0 0x7f2d0c14c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:10.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d0a59c640 1 -- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0c0c97a0 msgr2=0x7f2d0c14c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:10.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d0a59c640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0c0c97a0 0x7f2d0c14c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:10.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d0a59c640 1 -- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0c0c7cd0 msgr2=0x7f2d0c14e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:10.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d0a59c640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0c0c7cd0 0x7f2d0c14e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:10.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d0a59c640 1 -- 172.21.15.150:0/2338646007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0c1f19d0 con 0x7f2d0c072ea0 2024-09-17T05:08:10.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d0ad9d640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0c0c97a0 0x7f2d0c14c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:08:10.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d0a59c640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d0c072ea0 0x7f2d0c156030 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00c3f0 tx=0x7f2cfc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:10.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2cfaffd640 1 -- 172.21.15.150:0/2338646007 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc017070 con 0x7f2d0c072ea0 2024-09-17T05:08:10.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2cfaffd640 1 -- 172.21.15.150:0/2338646007 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cfc0053c0 con 0x7f2d0c072ea0 2024-09-17T05:08:10.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2cfaffd640 1 -- 172.21.15.150:0/2338646007 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc003860 con 0x7f2d0c072ea0 2024-09-17T05:08:10.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.653+0000 7f2d108e7640 1 -- 172.21.15.150:0/2338646007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d0c1f1c70 con 0x7f2d0c072ea0 2024-09-17T05:08:10.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.657+0000 7f2d108e7640 1 -- 172.21.15.150:0/2338646007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d0c0c0f10 con 0x7f2d0c072ea0 2024-09-17T05:08:10.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.657+0000 7f2d108e7640 1 -- 172.21.15.150:0/2338646007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d0c0732b0 con 0x7f2d0c072ea0 2024-09-17T05:08:10.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.657+0000 7f2cfaffd640 1 -- 172.21.15.150:0/2338646007 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2cfc01a070 con 0x7f2d0c072ea0 2024-09-17T05:08:10.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.657+0000 7f2cfaffd640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cec073200 0x7f2cec075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:10.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.657+0000 7f2cfaffd640 1 -- 172.21.15.150:0/2338646007 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f2cfc094060 con 0x7f2d0c072ea0 2024-09-17T05:08:10.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.657+0000 7f2d09d9b640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cec073200 0x7f2cec075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:10.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.661+0000 7f2d09d9b640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cec073200 0x7f2cec075710 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c0c6380 tx=0x7f2d0000b2e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:10.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.661+0000 7f2cfaffd640 1 -- 172.21.15.150:0/2338646007 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2cfc05dea0 con 0x7f2d0c072ea0 2024-09-17T05:08:10.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.873+0000 7f2d108e7640 1 -- 172.21.15.150:0/2338646007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d0c0c40f0 con 0x7f2d0c072ea0 2024-09-17T05:08:10.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2cfaffd640 1 -- 172.21.15.150:0/2338646007 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+6536 (secure 0 0 0) 0x7f2cfc0628d0 con 0x7f2d0c072ea0 2024-09-17T05:08:10.874 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:10.874 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":136,"btime":"2024-09-17T05:07:28:102129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:08:10.875 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 136 2024-09-17T05:08:10.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2d108e7640 1 -- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cec073200 msgr2=0x7f2cec075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2d108e7640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cec073200 0x7f2cec075710 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c0c6380 tx=0x7f2d0000b2e0 comp rx=0 tx=0).stop 2024-09-17T05:08:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2d108e7640 1 -- 172.21.15.150:0/2338646007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d0c072ea0 msgr2=0x7f2d0c156030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2d108e7640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d0c072ea0 0x7f2d0c156030 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00c3f0 tx=0x7f2cfc00c8b0 comp rx=0 tx=0).stop 2024-09-17T05:08:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2d108e7640 1 -- 172.21.15.150:0/2338646007 shutdown_connections 2024-09-17T05:08:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2d108e7640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2cec073200 0x7f2cec075710 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2d108e7640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d0c0c7cd0 0x7f2d0c14e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2d108e7640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d0c0c97a0 0x7f2d0c14c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2d108e7640 1 --2- 172.21.15.150:0/2338646007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d0c072ea0 0x7f2d0c156030 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2d108e7640 1 -- 172.21.15.150:0/2338646007 >> 172.21.15.150:0/2338646007 conn(0x7f2d0c0abb00 msgr2=0x7f2d0c15e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2d108e7640 1 -- 172.21.15.150:0/2338646007 shutdown_connections 2024-09-17T05:08:10.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:10.877+0000 7f2d108e7640 1 -- 172.21.15.150:0/2338646007 wait complete. 2024-09-17T05:08:10.902 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:08:12.903 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:13.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.085+0000 7f43e9649640 1 -- 172.21.15.150:0/721683988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43d0004d50 msgr2=0x7f43d0005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:13.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.085+0000 7f43e9649640 1 --2- 172.21.15.150:0/721683988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43d0004d50 0x7f43d0005160 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f43e40735c0 tx=0x7f43dc01d2a0 comp rx=0 tx=0).stop 2024-09-17T05:08:13.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.085+0000 7f43e37fe640 1 -- 172.21.15.150:0/721683988 reap_dead start 2024-09-17T05:08:13.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.085+0000 7f43e9649640 1 -- 172.21.15.150:0/721683988 shutdown_connections 2024-09-17T05:08:13.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.085+0000 7f43e9649640 1 -- 172.21.15.150:0/721683988 >> 172.21.15.150:0/721683988 conn(0x7f43e40ca460 msgr2=0x7f43e40c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:13.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.085+0000 7f43e9649640 1 -- 172.21.15.150:0/721683988 shutdown_connections 2024-09-17T05:08:13.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.085+0000 7f43e9649640 1 -- 172.21.15.150:0/721683988 wait complete. 2024-09-17T05:08:13.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e9649640 1 Processor -- start 2024-09-17T05:08:13.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e9649640 1 -- start start 2024-09-17T05:08:13.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e9649640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e40aec90 0x7f43e41e2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:13.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e9649640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43e41e2570 0x7f43e414c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:13.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e9649640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f43e414cd10 0x7f43e41ec790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:13.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e9649640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f43e4157650 con 0x7f43e40aec90 2024-09-17T05:08:13.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e9649640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f43e4072070 con 0x7f43e41e2570 2024-09-17T05:08:13.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e9649640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f43e4072620 con 0x7f43e414cd10 2024-09-17T05:08:13.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e40aec90 0x7f43e41e2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:13.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e40aec90 0x7f43e41e2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45704/0 (socket says 172.21.15.150:45704) 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e2ffd640 1 -- 172.21.15.150:0/1593657418 learned_addr learned my addr 172.21.15.150:0/1593657418 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43dbfff640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43e41e2570 0x7f43e414c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e37fe640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f43e414cd10 0x7f43e41ec790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e2ffd640 1 -- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f43e414cd10 msgr2=0x7f43e41ec790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e2ffd640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f43e414cd10 0x7f43e41ec790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e2ffd640 1 -- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43e41e2570 msgr2=0x7f43e414c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e2ffd640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43e41e2570 0x7f43e414c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e2ffd640 1 -- 172.21.15.150:0/1593657418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e40b0150 con 0x7f43e40aec90 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e2ffd640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e40aec90 0x7f43e41e2030 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f43d4011a50 tx=0x7f43d4011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43db7fe640 1 -- 172.21.15.150:0/1593657418 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d401f070 con 0x7f43e40aec90 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43db7fe640 1 -- 172.21.15.150:0/1593657418 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43d400fe80 con 0x7f43e40aec90 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e9649640 1 -- 172.21.15.150:0/1593657418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e4154c60 con 0x7f43e40aec90 2024-09-17T05:08:13.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43db7fe640 1 -- 172.21.15.150:0/1593657418 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d40126b0 con 0x7f43e40aec90 2024-09-17T05:08:13.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.089+0000 7f43e9649640 1 -- 172.21.15.150:0/1593657418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e415ac10 con 0x7f43e40aec90 2024-09-17T05:08:13.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.093+0000 7f43e9649640 1 -- 172.21.15.150:0/1593657418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43e40c8a80 con 0x7f43e40aec90 2024-09-17T05:08:13.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.093+0000 7f43db7fe640 1 -- 172.21.15.150:0/1593657418 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f43d4022070 con 0x7f43e40aec90 2024-09-17T05:08:13.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.093+0000 7f43db7fe640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f43c4073200 0x7f43c4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:13.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.093+0000 7f43db7fe640 1 -- 172.21.15.150:0/1593657418 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f43d40960b0 con 0x7f43e40aec90 2024-09-17T05:08:13.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.093+0000 7f43dbfff640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f43c4073200 0x7f43c4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:13.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.093+0000 7f43dbfff640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f43c4073200 0x7f43c4075710 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f43e4158f20 tx=0x7f43cc0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:13.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.097+0000 7f43db7fe640 1 -- 172.21.15.150:0/1593657418 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f43e40b0150 con 0x7f43e40aec90 2024-09-17T05:08:13.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.337+0000 7f43e9649640 1 -- 172.21.15.150:0/1593657418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43e40c39d0 con 0x7f43e40aec90 2024-09-17T05:08:13.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.337+0000 7f43db7fe640 1 -- 172.21.15.150:0/1593657418 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+6536 (secure 0 0 0) 0x7f43d405ff70 con 0x7f43e40aec90 2024-09-17T05:08:13.338 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:13.338 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":136,"btime":"2024-09-17T05:07:28:102129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:08:13.338 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 136 2024-09-17T05:08:13.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.341+0000 7f43e9649640 1 -- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f43c4073200 msgr2=0x7f43c4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:13.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.341+0000 7f43e9649640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f43c4073200 0x7f43c4075710 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f43e4158f20 tx=0x7f43cc0091e0 comp rx=0 tx=0).stop 2024-09-17T05:08:13.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.341+0000 7f43e9649640 1 -- 172.21.15.150:0/1593657418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e40aec90 msgr2=0x7f43e41e2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:13.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.341+0000 7f43e9649640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e40aec90 0x7f43e41e2030 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f43d4011a50 tx=0x7f43d4011f10 comp rx=0 tx=0).stop 2024-09-17T05:08:13.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.341+0000 7f43e9649640 1 -- 172.21.15.150:0/1593657418 shutdown_connections 2024-09-17T05:08:13.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.341+0000 7f43e9649640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f43c4073200 0x7f43c4075710 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:13.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.341+0000 7f43e9649640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43e41e2570 0x7f43e414c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:13.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.341+0000 7f43e9649640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f43e414cd10 0x7f43e41ec790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:13.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.341+0000 7f43e9649640 1 --2- 172.21.15.150:0/1593657418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e40aec90 0x7f43e41e2030 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:13.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.341+0000 7f43e9649640 1 -- 172.21.15.150:0/1593657418 >> 172.21.15.150:0/1593657418 conn(0x7f43e40ca460 msgr2=0x7f43e4156000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:13.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.341+0000 7f43e9649640 1 -- 172.21.15.150:0/1593657418 shutdown_connections 2024-09-17T05:08:13.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:13.341+0000 7f43e9649640 1 -- 172.21.15.150:0/1593657418 wait complete. 2024-09-17T05:08:13.360 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:08:15.361 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:15.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.553+0000 7fa6b5705640 1 -- 172.21.15.150:0/3454210081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa698004d50 msgr2=0x7fa698005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.553+0000 7fa6b5705640 1 --2- 172.21.15.150:0/3454210081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa698004d50 0x7fa698005200 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fa6a800c660 tx=0x7fa6a801fa40 comp rx=0 tx=0).stop 2024-09-17T05:08:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.553+0000 7fa6af7fe640 1 -- 172.21.15.150:0/3454210081 reap_dead start 2024-09-17T05:08:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.553+0000 7fa6b5705640 1 -- 172.21.15.150:0/3454210081 shutdown_connections 2024-09-17T05:08:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.553+0000 7fa6b5705640 1 -- 172.21.15.150:0/3454210081 >> 172.21.15.150:0/3454210081 conn(0x7fa6b00c5ae0 msgr2=0x7fa6b0073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.553+0000 7fa6b5705640 1 -- 172.21.15.150:0/3454210081 shutdown_connections 2024-09-17T05:08:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.553+0000 7fa6b5705640 1 -- 172.21.15.150:0/3454210081 wait complete. 2024-09-17T05:08:15.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.553+0000 7fa6b5705640 1 Processor -- start 2024-09-17T05:08:15.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.553+0000 7fa6b5705640 1 -- start start 2024-09-17T05:08:15.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6b5705640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6b0156a00 0x7fa6b01ea980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:15.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6b5705640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6b01eb400 0x7fa6b0154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6b5705640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6b0155050 0x7fa6b014ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6b5705640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b00c4840 con 0x7fa6b0156a00 2024-09-17T05:08:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6b5705640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b00c46c0 con 0x7fa6b01eb400 2024-09-17T05:08:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6b5705640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa6b00c49c0 con 0x7fa6b0155050 2024-09-17T05:08:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6aeffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6b0156a00 0x7fa6b01ea980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6aeffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6b0156a00 0x7fa6b01ea980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45718/0 (socket says 172.21.15.150:45718) 2024-09-17T05:08:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6aeffd640 1 -- 172.21.15.150:0/3971249154 learned_addr learned my addr 172.21.15.150:0/3971249154 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6af7fe640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6b0155050 0x7fa6b014ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6ae7fc640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6b01eb400 0x7fa6b0154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6aeffd640 1 -- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6b0155050 msgr2=0x7fa6b014ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6aeffd640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6b0155050 0x7fa6b014ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6aeffd640 1 -- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6b01eb400 msgr2=0x7fa6b0154b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6aeffd640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6b01eb400 0x7fa6b0154b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6aeffd640 1 -- 172.21.15.150:0/3971249154 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b0143c40 con 0x7fa6b0156a00 2024-09-17T05:08:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6ae7fc640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6b01eb400 0x7fa6b0154b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:08:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6af7fe640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6b0155050 0x7fa6b014ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:08:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6aeffd640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6b0156a00 0x7fa6b01ea980 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000d580 tx=0x7fa6a000da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6977fe640 1 -- 172.21.15.150:0/3971249154 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0012630 con 0x7fa6b0156a00 2024-09-17T05:08:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6977fe640 1 -- 172.21.15.150:0/3971249154 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa6a0017070 con 0x7fa6b0156a00 2024-09-17T05:08:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6b5705640 1 -- 172.21.15.150:0/3971249154 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6b014d470 con 0x7fa6b0156a00 2024-09-17T05:08:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6977fe640 1 -- 172.21.15.150:0/3971249154 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0012a70 con 0x7fa6b0156a00 2024-09-17T05:08:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6b5705640 1 -- 172.21.15.150:0/3971249154 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6b0155e00 con 0x7fa6b0156a00 2024-09-17T05:08:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.557+0000 7fa6b5705640 1 -- 172.21.15.150:0/3971249154 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6b00c3fb0 con 0x7fa6b0156a00 2024-09-17T05:08:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.561+0000 7fa6977fe640 1 -- 172.21.15.150:0/3971249154 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa6a0012c10 con 0x7fa6b0156a00 2024-09-17T05:08:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.561+0000 7fa6977fe640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa69c073230 0x7fa69c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.561+0000 7fa6977fe640 1 -- 172.21.15.150:0/3971249154 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7fa6a0094960 con 0x7fa6b0156a00 2024-09-17T05:08:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.561+0000 7fa6ae7fc640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa69c073230 0x7fa69c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:15.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.561+0000 7fa6ae7fc640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa69c073230 0x7fa69c075740 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400abf0 tx=0x7fa6a400a2f0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:15.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.565+0000 7fa6977fe640 1 -- 172.21.15.150:0/3971249154 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa6a005ee00 con 0x7fa6b0156a00 2024-09-17T05:08:15.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.793+0000 7fa6b5705640 1 -- 172.21.15.150:0/3971249154 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6b0069520 con 0x7fa6b0156a00 2024-09-17T05:08:15.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.793+0000 7fa6977fe640 1 -- 172.21.15.150:0/3971249154 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+6536 (secure 0 0 0) 0x7fa6a005e700 con 0x7fa6b0156a00 2024-09-17T05:08:15.793 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:15.794 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":136,"btime":"2024-09-17T05:07:28:102129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7074,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":136}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:06:34.202583+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6831},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6831":{"gid":6831,"name":"c","rank":0,"incarnation":122,"state":"up:active","state_seq":9,"addr":"172.21.15.150:6835/92695211","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":92695211},{"type":"v1","addr":"172.21.15.150:6835","nonce":92695211}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6831,"qdb_cluster":[6831]},"id":2}]} 2024-09-17T05:08:15.794 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 136 2024-09-17T05:08:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.797+0000 7fa6b5705640 1 -- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa69c073230 msgr2=0x7fa69c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.797+0000 7fa6b5705640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa69c073230 0x7fa69c075740 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400abf0 tx=0x7fa6a400a2f0 comp rx=0 tx=0).stop 2024-09-17T05:08:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.797+0000 7fa6b5705640 1 -- 172.21.15.150:0/3971249154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6b0156a00 msgr2=0x7fa6b01ea980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.797+0000 7fa6b5705640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6b0156a00 0x7fa6b01ea980 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000d580 tx=0x7fa6a000da40 comp rx=0 tx=0).stop 2024-09-17T05:08:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.797+0000 7fa6b5705640 1 -- 172.21.15.150:0/3971249154 shutdown_connections 2024-09-17T05:08:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.797+0000 7fa6b5705640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa69c073230 0x7fa69c075740 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:15.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.797+0000 7fa6b5705640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6b01eb400 0x7fa6b0154b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:15.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.797+0000 7fa6b5705640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa6b0155050 0x7fa6b014ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:15.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.797+0000 7fa6b5705640 1 --2- 172.21.15.150:0/3971249154 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6b0156a00 0x7fa6b01ea980 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:15.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.797+0000 7fa6b5705640 1 -- 172.21.15.150:0/3971249154 >> 172.21.15.150:0/3971249154 conn(0x7fa6b00c5ae0 msgr2=0x7fa6b0150e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:15.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.797+0000 7fa6b5705640 1 -- 172.21.15.150:0/3971249154 shutdown_connections 2024-09-17T05:08:15.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:15.797+0000 7fa6b5705640 1 -- 172.21.15.150:0/3971249154 wait complete. 2024-09-17T05:08:15.818 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:08:17.819 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:18.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.021+0000 7f1b91fa9640 1 -- 172.21.15.150:0/791150012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b74004b50 msgr2=0x7f1b74004fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:18.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.021+0000 7f1b91fa9640 1 --2- 172.21.15.150:0/791150012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b74004b50 0x7f1b74004fc0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c0012d0 tx=0x7f1b7c01e9e0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.021+0000 7f1b8bfff640 1 -- 172.21.15.150:0/791150012 reap_dead start 2024-09-17T05:08:18.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.021+0000 7f1b91fa9640 1 -- 172.21.15.150:0/791150012 shutdown_connections 2024-09-17T05:08:18.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.021+0000 7f1b91fa9640 1 -- 172.21.15.150:0/791150012 >> 172.21.15.150:0/791150012 conn(0x7f1b8c0c92d0 msgr2=0x7f1b8c0c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:18.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.021+0000 7f1b91fa9640 1 -- 172.21.15.150:0/791150012 shutdown_connections 2024-09-17T05:08:18.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.021+0000 7f1b91fa9640 1 -- 172.21.15.150:0/791150012 wait complete. 2024-09-17T05:08:18.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b91fa9640 1 Processor -- start 2024-09-17T05:08:18.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b91fa9640 1 -- start start 2024-09-17T05:08:18.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b91fa9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b74004b50 0x7f1b8c1585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:18.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b91fa9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b8c1efc50 0x7f1b8c0c6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b91fa9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b8c141560 0x7f1b8c141970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b91fa9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8c0c4bc0 con 0x7f1b74004b50 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b91fa9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8c157160 con 0x7f1b8c1efc50 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b91fa9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1b8c15e7d0 con 0x7f1b8c141560 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b8b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b74004b50 0x7f1b8c1585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b8b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b74004b50 0x7f1b8c1585c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45744/0 (socket says 172.21.15.150:45744) 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b8b7fe640 1 -- 172.21.15.150:0/742751131 learned_addr learned my addr 172.21.15.150:0/742751131 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b8affd640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b8c1efc50 0x7f1b8c0c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b8bfff640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b8c141560 0x7f1b8c141970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b8b7fe640 1 -- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b8c141560 msgr2=0x7f1b8c141970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b8b7fe640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b8c141560 0x7f1b8c141970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b8b7fe640 1 -- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b8c1efc50 msgr2=0x7f1b8c0c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:18.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b8b7fe640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b8c1efc50 0x7f1b8c0c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b8b7fe640 1 -- 172.21.15.150:0/742751131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b8c142690 con 0x7f1b74004b50 2024-09-17T05:08:18.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b8b7fe640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b74004b50 0x7f1b8c1585c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c024f70 tx=0x7f1b7c001130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:18.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b73fff640 1 -- 172.21.15.150:0/742751131 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c003600 con 0x7f1b74004b50 2024-09-17T05:08:18.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b73fff640 1 -- 172.21.15.150:0/742751131 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b7c003890 con 0x7f1b74004b50 2024-09-17T05:08:18.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b91fa9640 1 -- 172.21.15.150:0/742751131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b8c1421f0 con 0x7f1b74004b50 2024-09-17T05:08:18.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b73fff640 1 -- 172.21.15.150:0/742751131 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c003c80 con 0x7f1b74004b50 2024-09-17T05:08:18.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b91fa9640 1 -- 172.21.15.150:0/742751131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b8c0c6e60 con 0x7f1b74004b50 2024-09-17T05:08:18.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.025+0000 7f1b91fa9640 1 -- 172.21.15.150:0/742751131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b8c142010 con 0x7f1b74004b50 2024-09-17T05:08:18.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.029+0000 7f1b73fff640 1 -- 172.21.15.150:0/742751131 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1b7c0a75b0 con 0x7f1b74004b50 2024-09-17T05:08:18.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.029+0000 7f1b73fff640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b78073200 0x7f1b78075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:18.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.029+0000 7f1b73fff640 1 -- 172.21.15.150:0/742751131 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f1b7c130a70 con 0x7f1b74004b50 2024-09-17T05:08:18.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.029+0000 7f1b8affd640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b78073200 0x7f1b78075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:18.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.029+0000 7f1b8affd640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b78073200 0x7f1b78075710 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0c28a0 tx=0x7f1b800091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:18.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.033+0000 7f1b73fff640 1 -- 172.21.15.150:0/742751131 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1b8c142010 con 0x7f1b74004b50 2024-09-17T05:08:18.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.277+0000 7f1b91fa9640 1 -- 172.21.15.150:0/742751131 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b8c073d40 con 0x7f1b74004b50 2024-09-17T05:08:18.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.277+0000 7f1b73fff640 1 -- 172.21.15.150:0/742751131 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) ==== 95+0+5710 (secure 0 0 0) 0x7f1b7c018740 con 0x7f1b74004b50 2024-09-17T05:08:18.278 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:18.278 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":138,"btime":"2024-09-17T05:08:17:259553+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:16.559428+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:08:18.278 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 138 2024-09-17T05:08:18.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.281+0000 7f1b91fa9640 1 -- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b78073200 msgr2=0x7f1b78075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:18.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.281+0000 7f1b91fa9640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b78073200 0x7f1b78075710 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c0c28a0 tx=0x7f1b800091e0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.281+0000 7f1b91fa9640 1 -- 172.21.15.150:0/742751131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b74004b50 msgr2=0x7f1b8c1585c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:18.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.281+0000 7f1b91fa9640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b74004b50 0x7f1b8c1585c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c024f70 tx=0x7f1b7c001130 comp rx=0 tx=0).stop 2024-09-17T05:08:18.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.281+0000 7f1b91fa9640 1 -- 172.21.15.150:0/742751131 shutdown_connections 2024-09-17T05:08:18.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.281+0000 7f1b91fa9640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1b78073200 0x7f1b78075710 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.281+0000 7f1b91fa9640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b8c1efc50 0x7f1b8c0c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.281+0000 7f1b91fa9640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1b8c141560 0x7f1b8c141970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.281+0000 7f1b91fa9640 1 --2- 172.21.15.150:0/742751131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b74004b50 0x7f1b8c1585c0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.281+0000 7f1b91fa9640 1 -- 172.21.15.150:0/742751131 >> 172.21.15.150:0/742751131 conn(0x7f1b8c0c92d0 msgr2=0x7f1b8c154ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:18.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.281+0000 7f1b91fa9640 1 -- 172.21.15.150:0/742751131 shutdown_connections 2024-09-17T05:08:18.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.281+0000 7f1b91fa9640 1 -- 172.21.15.150:0/742751131 wait complete. 2024-09-17T05:08:18.306 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-17T05:08:18.306 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:08:18.306 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:18.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.469+0000 7fa9873dc640 1 -- 172.21.15.150:0/2145699198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9800c4b20 msgr2=0x7fa980153bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:18.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.469+0000 7fa9873dc640 1 --2- 172.21.15.150:0/2145699198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9800c4b20 0x7fa980153bc0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa98008db40 tx=0x7fa97801cd80 comp rx=0 tx=0).stop 2024-09-17T05:08:18.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.469+0000 7fa9873dc640 1 -- 172.21.15.150:0/2145699198 shutdown_connections 2024-09-17T05:08:18.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.469+0000 7fa9873dc640 1 --2- 172.21.15.150:0/2145699198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9800c4b20 0x7fa980153bc0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.469+0000 7fa9873dc640 1 --2- 172.21.15.150:0/2145699198 >> v2:172.21.15.176:3300/0 conn(0x7fa9800c6340 0x7fa9800c1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.469+0000 7fa9873dc640 1 --2- 172.21.15.150:0/2145699198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9800c5df0 0x7fa980157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.469+0000 7fa9873dc640 1 -- 172.21.15.150:0/2145699198 >> 172.21.15.150:0/2145699198 conn(0x7fa980074750 msgr2=0x7fa980074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:18.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.469+0000 7fa9873dc640 1 -- 172.21.15.150:0/2145699198 shutdown_connections 2024-09-17T05:08:18.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.469+0000 7fa9873dc640 1 -- 172.21.15.150:0/2145699198 wait complete. 2024-09-17T05:08:18.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa9873dc640 1 Processor -- start 2024-09-17T05:08:18.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa9873dc640 1 -- start start 2024-09-17T05:08:18.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa9873dc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9800c4b20 0x7fa9800ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:18.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa9873dc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9800c5df0 0x7fa9800cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:18.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa9873dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9800c6340 0x7fa9801590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:18.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa9873dc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa98015a760 con 0x7fa9800c6340 2024-09-17T05:08:18.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa9873dc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa98015a5e0 con 0x7fa9800c4b20 2024-09-17T05:08:18.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa9873dc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa98015a8e0 con 0x7fa9800c5df0 2024-09-17T05:08:18.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa985979640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9800c6340 0x7fa9801590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:18.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa985979640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9800c6340 0x7fa9801590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45768/0 (socket says 172.21.15.150:45768) 2024-09-17T05:08:18.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa985979640 1 -- 172.21.15.150:0/1083050238 learned_addr learned my addr 172.21.15.150:0/1083050238 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa985178640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9800c4b20 0x7fa9800ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa984977640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9800c5df0 0x7fa9800cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa985979640 1 -- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9800c5df0 msgr2=0x7fa9800cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa985979640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9800c5df0 0x7fa9800cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa985979640 1 -- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9800c4b20 msgr2=0x7fa9800ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa985979640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9800c4b20 0x7fa9800ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa985979640 1 -- 172.21.15.150:0/1083050238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98014d670 con 0x7fa9800c6340 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa985178640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9800c4b20 0x7fa9800ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa984977640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9800c5df0 0x7fa9800cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa985979640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9800c6340 0x7fa9801590a0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa978009bd0 tx=0x7fa978005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa975ffb640 1 -- 172.21.15.150:0/1083050238 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9780bb070 con 0x7fa9800c6340 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa975ffb640 1 -- 172.21.15.150:0/1083050238 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa978004d40 con 0x7fa9800c6340 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa9873dc640 1 -- 172.21.15.150:0/1083050238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa98014d930 con 0x7fa9800c6340 2024-09-17T05:08:18.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa975ffb640 1 -- 172.21.15.150:0/1083050238 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9780b2040 con 0x7fa9800c6340 2024-09-17T05:08:18.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa9873dc640 1 -- 172.21.15.150:0/1083050238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa98014b0c0 con 0x7fa9800c6340 2024-09-17T05:08:18.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.473+0000 7fa9873dc640 1 -- 172.21.15.150:0/1083050238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa98014a140 con 0x7fa9800c6340 2024-09-17T05:08:18.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.477+0000 7fa975ffb640 1 -- 172.21.15.150:0/1083050238 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa9780be070 con 0x7fa9800c6340 2024-09-17T05:08:18.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.477+0000 7fa975ffb640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa964073230 0x7fa964075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:18.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.477+0000 7fa975ffb640 1 -- 172.21.15.150:0/1083050238 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7fa97812ec30 con 0x7fa9800c6340 2024-09-17T05:08:18.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.477+0000 7fa985178640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa964073230 0x7fa964075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:18.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.477+0000 7fa985178640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa964073230 0x7fa964075740 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa970005fd0 tx=0x7fa970005dc0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:18.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.481+0000 7fa975ffb640 1 -- 172.21.15.150:0/1083050238 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa9780f2db0 con 0x7fa9800c6340 2024-09-17T05:08:18.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.761+0000 7fa9873dc640 1 -- 172.21.15.150:0/1083050238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa98014b320 con 0x7fa9800c6340 2024-09-17T05:08:18.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.761+0000 7fa975ffb640 1 -- 172.21.15.150:0/1083050238 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+5707 (secure 0 0 0) 0x7fa978016970 con 0x7fa9800c6340 2024-09-17T05:08:18.760 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:18.760 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":139,"btime":"2024-09-17T05:08:18:272875+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:17.277405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:08:18.761 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 139 2024-09-17T05:08:18.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.765+0000 7fa9873dc640 1 -- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa964073230 msgr2=0x7fa964075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:18.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.765+0000 7fa9873dc640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa964073230 0x7fa964075740 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa970005fd0 tx=0x7fa970005dc0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.765+0000 7fa9873dc640 1 -- 172.21.15.150:0/1083050238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9800c6340 msgr2=0x7fa9801590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:18.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.765+0000 7fa9873dc640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9800c6340 0x7fa9801590a0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa978009bd0 tx=0x7fa978005b00 comp rx=0 tx=0).stop 2024-09-17T05:08:18.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.765+0000 7fa9873dc640 1 -- 172.21.15.150:0/1083050238 shutdown_connections 2024-09-17T05:08:18.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.765+0000 7fa9873dc640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa964073230 0x7fa964075740 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.765+0000 7fa9873dc640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9800c6340 0x7fa9801590a0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.765+0000 7fa9873dc640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa9800c5df0 0x7fa9800cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.765+0000 7fa9873dc640 1 --2- 172.21.15.150:0/1083050238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9800c4b20 0x7fa9800ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:18.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.765+0000 7fa9873dc640 1 -- 172.21.15.150:0/1083050238 >> 172.21.15.150:0/1083050238 conn(0x7fa980074750 msgr2=0x7fa9800c7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:18.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.769+0000 7fa9873dc640 1 -- 172.21.15.150:0/1083050238 shutdown_connections 2024-09-17T05:08:18.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:18.769+0000 7fa9873dc640 1 -- 172.21.15.150:0/1083050238 wait complete. 2024-09-17T05:08:18.792 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:08:18:272875+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 139, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 135, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5945": { "addr": "172.21.15.176:6837/792092400", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 792092400, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 792092400, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5945, "incarnation": 132, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:07:24.656365+0000", "qdb_cluster": [ 5945 ], "qdb_leader": 5945, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5945 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 139, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7074": { "addr": "172.21.15.150:6833/3622725045", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 3622725045, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 3622725045, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7074, "incarnation": 137, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:08:17.277405+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7074 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:08:20.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:20.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 -- 172.21.15.150:0/1897681534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0080c35b0 msgr2=0x7fe0080c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:20.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 --2- 172.21.15.150:0/1897681534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0080c35b0 0x7fe0080c8a80 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fdff0009840 tx=0x7fdff001c7f0 comp rx=0 tx=0).stop 2024-09-17T05:08:20.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 -- 172.21.15.150:0/1897681534 shutdown_connections 2024-09-17T05:08:20.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 --2- 172.21.15.150:0/1897681534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0080c35b0 0x7fe0080c8a80 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:20.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 -- 172.21.15.150:0/1897681534 >> 172.21.15.150:0/1897681534 conn(0x7fe008074b80 msgr2=0x7fe00806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:20.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 -- 172.21.15.150:0/1897681534 shutdown_connections 2024-09-17T05:08:20.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 -- 172.21.15.150:0/1897681534 wait complete. 2024-09-17T05:08:20.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 Processor -- start 2024-09-17T05:08:20.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 -- start start 2024-09-17T05:08:20.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0080c35b0 0x7fe0081f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:20.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0080c8fc0 0x7fe00814ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:20.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0081568a0 0x7fe008143430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:20.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe0080c6b70 con 0x7fe0080c35b0 2024-09-17T05:08:20.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe0080c69f0 con 0x7fe0081568a0 2024-09-17T05:08:20.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe00d7a1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe0080c27b0 con 0x7fe0080c8fc0 2024-09-17T05:08:20.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe006ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0080c35b0 0x7fe0081f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:20.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe006ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0080c35b0 0x7fe0081f2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42404/0 (socket says 172.21.15.150:42404) 2024-09-17T05:08:20.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe006ffd640 1 -- 172.21.15.150:0/3457397903 learned_addr learned my addr 172.21.15.150:0/3457397903 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:20.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fdffffff640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0080c8fc0 0x7fe00814ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:20.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.953+0000 7fe0077fe640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0081568a0 0x7fe008143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:20.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fe006ffd640 1 -- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0080c8fc0 msgr2=0x7fe00814ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:20.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fe006ffd640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0080c8fc0 0x7fe00814ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:20.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fe006ffd640 1 -- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0081568a0 msgr2=0x7fe008143430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:20.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fe006ffd640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0081568a0 0x7fe008143430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:20.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fe006ffd640 1 -- 172.21.15.150:0/3457397903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe00814e160 con 0x7fe0080c35b0 2024-09-17T05:08:20.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fdffffff640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0080c8fc0 0x7fe00814ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:08:20.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fe006ffd640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0080c35b0 0x7fe0081f2f40 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fdff800b410 tx=0x7fdff800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:20.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fdfff7fe640 1 -- 172.21.15.150:0/3457397903 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff8015030 con 0x7fe0080c35b0 2024-09-17T05:08:20.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fdfff7fe640 1 -- 172.21.15.150:0/3457397903 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdff8004020 con 0x7fe0080c35b0 2024-09-17T05:08:20.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fdfff7fe640 1 -- 172.21.15.150:0/3457397903 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff8004470 con 0x7fe0080c35b0 2024-09-17T05:08:20.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fe00d7a1640 1 -- 172.21.15.150:0/3457397903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe00814b5b0 con 0x7fe0080c35b0 2024-09-17T05:08:20.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fe00d7a1640 1 -- 172.21.15.150:0/3457397903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0081f3920 con 0x7fe0080c35b0 2024-09-17T05:08:20.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fe00d7a1640 1 -- 172.21.15.150:0/3457397903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe00814e330 con 0x7fe0080c35b0 2024-09-17T05:08:20.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fdfff7fe640 1 -- 172.21.15.150:0/3457397903 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fdff8018070 con 0x7fe0080c35b0 2024-09-17T05:08:20.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fdfff7fe640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfe8073230 0x7fdfe8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:20.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fdfff7fe640 1 -- 172.21.15.150:0/3457397903 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7fdff8094ab0 con 0x7fe0080c35b0 2024-09-17T05:08:20.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.957+0000 7fdffffff640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfe8073230 0x7fdfe8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:20.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.961+0000 7fdffffff640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfe8073230 0x7fdfe8075740 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdff00af000 tx=0x7fdff001b030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:20.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:20.965+0000 7fdfff7fe640 1 -- 172.21.15.150:0/3457397903 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdff805ee70 con 0x7fe0080c35b0 2024-09-17T05:08:21.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.201+0000 7fe00d7a1640 1 -- 172.21.15.150:0/3457397903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0080c8a80 con 0x7fe0080c35b0 2024-09-17T05:08:21.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fdfff7fe640 1 -- 172.21.15.150:0/3457397903 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+5714 (secure 0 0 0) 0x7fdff805e770 con 0x7fe0080c35b0 2024-09-17T05:08:21.203 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:21.203 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":140,"btime":"2024-09-17T05:08:19:286845+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:19.286834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7074,"qdb_cluster":[7074]},"id":2}]} 2024-09-17T05:08:21.203 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 140 2024-09-17T05:08:21.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fe00d7a1640 1 -- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfe8073230 msgr2=0x7fdfe8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:21.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fe00d7a1640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfe8073230 0x7fdfe8075740 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdff00af000 tx=0x7fdff001b030 comp rx=0 tx=0).stop 2024-09-17T05:08:21.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fe00d7a1640 1 -- 172.21.15.150:0/3457397903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0080c35b0 msgr2=0x7fe0081f2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:21.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fe00d7a1640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0080c35b0 0x7fe0081f2f40 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fdff800b410 tx=0x7fdff800b8d0 comp rx=0 tx=0).stop 2024-09-17T05:08:21.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fe00d7a1640 1 -- 172.21.15.150:0/3457397903 shutdown_connections 2024-09-17T05:08:21.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fe00d7a1640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfe8073230 0x7fdfe8075740 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:21.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fe00d7a1640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0081568a0 0x7fe008143430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:21.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fe00d7a1640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe0080c8fc0 0x7fe00814ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:21.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fe00d7a1640 1 --2- 172.21.15.150:0/3457397903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0080c35b0 0x7fe0081f2f40 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:21.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fe00d7a1640 1 -- 172.21.15.150:0/3457397903 >> 172.21.15.150:0/3457397903 conn(0x7fe008074b80 msgr2=0x7fe008155f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:21.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fe00d7a1640 1 -- 172.21.15.150:0/3457397903 shutdown_connections 2024-09-17T05:08:21.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:21.205+0000 7fe00d7a1640 1 -- 172.21.15.150:0/3457397903 wait complete. 2024-09-17T05:08:21.227 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=6831 2024-09-17T05:08:21.228 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.c 2024-09-17T05:08:23.116 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T05:08:23.117+0000 7f5aab4c1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:08:23.229 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-17T05:08:23.229 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T05:08:23.229 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T05:08:23.231 INFO:tasks.ceph.mds.c:Started 2024-09-17T05:08:23.232 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:23.340 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T05:08:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.409+0000 7f0f0667e640 1 -- 172.21.15.150:0/1535558598 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f000c3fa0 msgr2=0x7f0f001547f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.409+0000 7f0f0667e640 1 --2- 172.21.15.150:0/1535558598 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f000c3fa0 0x7f0f001547f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4009840 tx=0x7f0ef401c790 comp rx=0 tx=0).stop 2024-09-17T05:08:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.409+0000 7f0f0667e640 1 -- 172.21.15.150:0/1535558598 shutdown_connections 2024-09-17T05:08:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.409+0000 7f0f0667e640 1 --2- 172.21.15.150:0/1535558598 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f000c43b0 0x7f0f000743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.409+0000 7f0f0667e640 1 --2- 172.21.15.150:0/1535558598 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f000c3fa0 0x7f0f001547f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.409+0000 7f0f0667e640 1 -- 172.21.15.150:0/1535558598 >> 172.21.15.150:0/1535558598 conn(0x7f0f000c5ae0 msgr2=0x7f0f00073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.409+0000 7f0f0667e640 1 -- 172.21.15.150:0/1535558598 shutdown_connections 2024-09-17T05:08:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.409+0000 7f0f0667e640 1 -- 172.21.15.150:0/1535558598 wait complete. 2024-09-17T05:08:23.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.409+0000 7f0f0667e640 1 Processor -- start 2024-09-17T05:08:23.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.409+0000 7f0f0667e640 1 -- start start 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f0667e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f000c38c0 0x7f0f00148e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f0667e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f000c3fa0 0x7f0f00147b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f0667e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f000c43b0 0x7f0f0014fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f0667e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0f000c4d50 con 0x7f0f000c3fa0 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f0667e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0f00154f40 con 0x7f0f000c38c0 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f0667e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0f000c4ed0 con 0x7f0f000c43b0 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0eff7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f000c3fa0 0x7f0f00147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0eff7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f000c3fa0 0x7f0f00147b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42426/0 (socket says 172.21.15.150:42426) 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f04c1b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f000c43b0 0x7f0f0014fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0eff7fe640 1 -- 172.21.15.150:0/1984978288 learned_addr learned my addr 172.21.15.150:0/1984978288 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0efffff640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f000c38c0 0x7f0f00148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f04c1b640 1 -- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f000c38c0 msgr2=0x7f0f00148e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f04c1b640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f000c38c0 0x7f0f00148e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:23.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f04c1b640 1 -- 172.21.15.150:0/1984978288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f000c3fa0 msgr2=0x7f0f00147b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:23.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f04c1b640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f000c3fa0 0x7f0f00147b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:23.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f04c1b640 1 -- 172.21.15.150:0/1984978288 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0014f5d0 con 0x7f0f000c43b0 2024-09-17T05:08:23.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0eff7fe640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f000c3fa0 0x7f0f00147b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:08:23.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f04c1b640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f000c43b0 0x7f0f0014fe60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800ebf0 tx=0x7f0ef800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0efcff9640 1 -- 172.21.15.150:0/1984978288 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8010630 con 0x7f0f000c43b0 2024-09-17T05:08:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0efcff9640 1 -- 172.21.15.150:0/1984978288 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ef80107d0 con 0x7f0f000c43b0 2024-09-17T05:08:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f0667e640 1 -- 172.21.15.150:0/1984978288 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f00150d60 con 0x7f0f000c43b0 2024-09-17T05:08:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0efcff9640 1 -- 172.21.15.150:0/1984978288 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8010c00 con 0x7f0f000c43b0 2024-09-17T05:08:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f0667e640 1 -- 172.21.15.150:0/1984978288 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f00151320 con 0x7f0f000c43b0 2024-09-17T05:08:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.413+0000 7f0f0667e640 1 -- 172.21.15.150:0/1984978288 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f001547f0 con 0x7f0f000c43b0 2024-09-17T05:08:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.417+0000 7f0efcff9640 1 -- 172.21.15.150:0/1984978288 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0ef8021440 con 0x7f0f000c43b0 2024-09-17T05:08:23.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.417+0000 7f0efcff9640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0eec073200 0x7f0eec075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:23.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.417+0000 7f0efcff9640 1 -- 172.21.15.150:0/1984978288 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f0ef8095ae0 con 0x7f0f000c43b0 2024-09-17T05:08:23.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.417+0000 7f0efffff640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0eec073200 0x7f0eec075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:23.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.417+0000 7f0efffff640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0eec073200 0x7f0eec075710 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0007fd0 tx=0x7f0ef0007ea0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:23.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.421+0000 7f0efcff9640 1 -- 172.21.15.150:0/1984978288 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0ef805f960 con 0x7f0f000c43b0 2024-09-17T05:08:23.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.645+0000 7f0f0667e640 1 -- 172.21.15.150:0/1984978288 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f000aa500 con 0x7f0f000c43b0 2024-09-17T05:08:23.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0efcff9640 1 -- 172.21.15.150:0/1984978288 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) ==== 95+0+5714 (secure 0 0 0) 0x7f0ef8064390 con 0x7f0f000c43b0 2024-09-17T05:08:23.646 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:23.647 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":140,"btime":"2024-09-17T05:08:19:286845+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:19.286834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7074,"qdb_cluster":[7074]},"id":2}]} 2024-09-17T05:08:23.647 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 140 2024-09-17T05:08:23.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0f0667e640 1 -- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0eec073200 msgr2=0x7f0eec075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:23.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0f0667e640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0eec073200 0x7f0eec075710 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0007fd0 tx=0x7f0ef0007ea0 comp rx=0 tx=0).stop 2024-09-17T05:08:23.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0f0667e640 1 -- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f000c43b0 msgr2=0x7f0f0014fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:23.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0f0667e640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f000c43b0 0x7f0f0014fe60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800ebf0 tx=0x7f0ef800c6a0 comp rx=0 tx=0).stop 2024-09-17T05:08:23.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0f0667e640 1 -- 172.21.15.150:0/1984978288 shutdown_connections 2024-09-17T05:08:23.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0f0667e640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0eec073200 0x7f0eec075710 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:23.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0f0667e640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0f000c43b0 0x7f0f0014fe60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:23.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0f0667e640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f000c3fa0 0x7f0f00147b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:23.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0f0667e640 1 --2- 172.21.15.150:0/1984978288 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0f000c38c0 0x7f0f00148e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:23.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0f0667e640 1 -- 172.21.15.150:0/1984978288 >> 172.21.15.150:0/1984978288 conn(0x7f0f000c5ae0 msgr2=0x7f0f00070920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:23.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0f0667e640 1 -- 172.21.15.150:0/1984978288 shutdown_connections 2024-09-17T05:08:23.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:23.649+0000 7f0f0667e640 1 -- 172.21.15.150:0/1984978288 wait complete. 2024-09-17T05:08:23.667 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-17T05:08:25.669 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:25.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.869+0000 7f6e7359e640 1 -- 172.21.15.150:0/537592022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e60004d50 msgr2=0x7f6e60005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:25.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.869+0000 7f6e7359e640 1 --2- 172.21.15.150:0/537592022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e60004d50 0x7f6e60005160 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f6e74073920 tx=0x7f6e6c01d5b0 comp rx=0 tx=0).stop 2024-09-17T05:08:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.869+0000 7f6e72d9d640 1 -- 172.21.15.150:0/537592022 reap_dead start 2024-09-17T05:08:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.869+0000 7f6e7359e640 1 -- 172.21.15.150:0/537592022 shutdown_connections 2024-09-17T05:08:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.869+0000 7f6e7359e640 1 -- 172.21.15.150:0/537592022 >> 172.21.15.150:0/537592022 conn(0x7f6e74073970 msgr2=0x7f6e74073d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.869+0000 7f6e7359e640 1 -- 172.21.15.150:0/537592022 shutdown_connections 2024-09-17T05:08:25.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.869+0000 7f6e7359e640 1 -- 172.21.15.150:0/537592022 wait complete. 2024-09-17T05:08:25.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7359e640 1 Processor -- start 2024-09-17T05:08:25.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7359e640 1 -- start start 2024-09-17T05:08:25.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e60004d50 0x7f6e74150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:25.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7359e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e7414b100 0x7f6e741ed980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:25.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7359e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e74150fc0 0x7f6e741edec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:25.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7359e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6e740ad5a0 con 0x7f6e60004d50 2024-09-17T05:08:25.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7359e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6e740ac450 con 0x7f6e7414b100 2024-09-17T05:08:25.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7359e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6e740ad720 con 0x7f6e74150fc0 2024-09-17T05:08:25.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e60004d50 0x7f6e74150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e60004d50 0x7f6e74150a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42454/0 (socket says 172.21.15.150:42454) 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7259c640 1 -- 172.21.15.150:0/3562051665 learned_addr learned my addr 172.21.15.150:0/3562051665 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e6bfff640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e7414b100 0x7f6e741ed980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e72d9d640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e74150fc0 0x7f6e741edec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7259c640 1 -- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e74150fc0 msgr2=0x7f6e741edec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7259c640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e74150fc0 0x7f6e741edec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7259c640 1 -- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e7414b100 msgr2=0x7f6e741ed980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7259c640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e7414b100 0x7f6e741ed980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7259c640 1 -- 172.21.15.150:0/3562051665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e741505a0 con 0x7f6e60004d50 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e72d9d640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e74150fc0 0x7f6e741edec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e7259c640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e60004d50 0x7f6e74150a80 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f6e6400f3f0 tx=0x7f6e6400f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e6affd640 1 -- 172.21.15.150:0/3562051665 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6401a070 con 0x7f6e60004d50 2024-09-17T05:08:25.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e6affd640 1 -- 172.21.15.150:0/3562051665 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e640084e0 con 0x7f6e60004d50 2024-09-17T05:08:25.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.873+0000 7f6e6affd640 1 -- 172.21.15.150:0/3562051665 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e64006980 con 0x7f6e60004d50 2024-09-17T05:08:25.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.877+0000 7f6e7359e640 1 -- 172.21.15.150:0/3562051665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e741ee4a0 con 0x7f6e60004d50 2024-09-17T05:08:25.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.877+0000 7f6e7359e640 1 -- 172.21.15.150:0/3562051665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e7414ef70 con 0x7f6e60004d50 2024-09-17T05:08:25.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.877+0000 7f6e7359e640 1 -- 172.21.15.150:0/3562051665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e740c48b0 con 0x7f6e60004d50 2024-09-17T05:08:25.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.877+0000 7f6e6affd640 1 -- 172.21.15.150:0/3562051665 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6e6401d070 con 0x7f6e60004d50 2024-09-17T05:08:25.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.877+0000 7f6e6affd640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6e54073230 0x7f6e54075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:25.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.877+0000 7f6e6affd640 1 -- 172.21.15.150:0/3562051665 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f6e64097410 con 0x7f6e60004d50 2024-09-17T05:08:25.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.877+0000 7f6e6bfff640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6e54073230 0x7f6e54075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:25.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.881+0000 7f6e6bfff640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6e54073230 0x7f6e54075740 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6e74152d40 tx=0x7f6e5c009540 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:25.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:25.885+0000 7f6e6affd640 1 -- 172.21.15.150:0/3562051665 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6e64024020 con 0x7f6e60004d50 2024-09-17T05:08:26.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.125+0000 7f6e7359e640 1 -- 172.21.15.150:0/3562051665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e74074100 con 0x7f6e60004d50 2024-09-17T05:08:26.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.125+0000 7f6e6affd640 1 -- 172.21.15.150:0/3562051665 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+6540 (secure 0 0 0) 0x7f6e640618d0 con 0x7f6e60004d50 2024-09-17T05:08:26.126 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:26.126 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":141,"btime":"2024-09-17T05:08:24:299799+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:19.286834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7074,"qdb_cluster":[7074]},"id":2}]} 2024-09-17T05:08:26.126 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 141 2024-09-17T05:08:26.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.129+0000 7f6e7359e640 1 -- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6e54073230 msgr2=0x7f6e54075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:26.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.129+0000 7f6e7359e640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6e54073230 0x7f6e54075740 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6e74152d40 tx=0x7f6e5c009540 comp rx=0 tx=0).stop 2024-09-17T05:08:26.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.129+0000 7f6e7359e640 1 -- 172.21.15.150:0/3562051665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e60004d50 msgr2=0x7f6e74150a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:26.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.129+0000 7f6e7359e640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e60004d50 0x7f6e74150a80 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f6e6400f3f0 tx=0x7f6e6400f7f0 comp rx=0 tx=0).stop 2024-09-17T05:08:26.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.129+0000 7f6e7359e640 1 -- 172.21.15.150:0/3562051665 shutdown_connections 2024-09-17T05:08:26.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.129+0000 7f6e7359e640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6e54073230 0x7f6e54075740 secure :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6e74152d40 tx=0x7f6e5c009540 comp rx=0 tx=0).stop 2024-09-17T05:08:26.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.129+0000 7f6e7359e640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e7414b100 0x7f6e741ed980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:26.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.129+0000 7f6e7359e640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6e74150fc0 0x7f6e741edec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:26.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.129+0000 7f6e7359e640 1 --2- 172.21.15.150:0/3562051665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e60004d50 0x7f6e74150a80 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:26.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.129+0000 7f6e7359e640 1 -- 172.21.15.150:0/3562051665 >> 172.21.15.150:0/3562051665 conn(0x7f6e74073970 msgr2=0x7f6e740a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:26.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.129+0000 7f6e7359e640 1 -- 172.21.15.150:0/3562051665 shutdown_connections 2024-09-17T05:08:26.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:26.129+0000 7f6e7359e640 1 -- 172.21.15.150:0/3562051665 wait complete. 2024-09-17T05:08:26.153 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-17T05:08:26.153 INFO:tasks.mds_thrash.fs.[b]:waiting for 24 secs before thrashing 2024-09-17T05:08:32.972 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:08:32.975 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:08:50.154 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.324+0000 7f336c1c3640 1 -- 172.21.15.150:0/84641639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33640c7eb0 msgr2=0x7f33641540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.324+0000 7f336c1c3640 1 --2- 172.21.15.150:0/84641639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33640c7eb0 0x7f33641540b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3360009840 tx=0x7f336001c820 comp rx=0 tx=0).stop 2024-09-17T05:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.324+0000 7f336c1c3640 1 -- 172.21.15.150:0/84641639 shutdown_connections 2024-09-17T05:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.324+0000 7f336c1c3640 1 --2- 172.21.15.150:0/84641639 >> v2:172.21.15.176:3300/0 conn(0x7f336414a180 0x7f33640c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.324+0000 7f336c1c3640 1 --2- 172.21.15.150:0/84641639 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33640c7eb0 0x7f33641540b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.324+0000 7f336c1c3640 1 -- 172.21.15.150:0/84641639 >> 172.21.15.150:0/84641639 conn(0x7f33640aab60 msgr2=0x7f33640aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.324+0000 7f336c1c3640 1 -- 172.21.15.150:0/84641639 shutdown_connections 2024-09-17T05:08:50.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.324+0000 7f336c1c3640 1 -- 172.21.15.150:0/84641639 wait complete. 2024-09-17T05:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f336c1c3640 1 Processor -- start 2024-09-17T05:08:50.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f336c1c3640 1 -- start start 2024-09-17T05:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f336c1c3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33640c7eb0 0x7f33640ac280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f336c1c3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3364149d70 0x7f3364148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f336c1c3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336414a180 0x7f3364148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f336c1c3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f33640c7c60 con 0x7f3364149d70 2024-09-17T05:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f336c1c3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f33640c7ae0 con 0x7f336414a180 2024-09-17T05:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f336c1c3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3364156540 con 0x7f33640c7eb0 2024-09-17T05:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f336975e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3364149d70 0x7f3364148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:50.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f336975e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3364149d70 0x7f3364148450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52978/0 (socket says 172.21.15.150:52978) 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f336975e640 1 -- 172.21.15.150:0/2902662311 learned_addr learned my addr 172.21.15.150:0/2902662311 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f3369f5f640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33640c7eb0 0x7f33640ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.328+0000 7f336a760640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336414a180 0x7f3364148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f336975e640 1 -- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33640c7eb0 msgr2=0x7f33640ac280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f336975e640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33640c7eb0 0x7f33640ac280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f336975e640 1 -- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336414a180 msgr2=0x7f3364148990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f336975e640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336414a180 0x7f3364148990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f336975e640 1 -- 172.21.15.150:0/2902662311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33641508f0 con 0x7f3364149d70 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f336975e640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3364149d70 0x7f3364148450 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f335400c410 tx=0x7f335400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f335a7fc640 1 -- 172.21.15.150:0/2902662311 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354011790 con 0x7f3364149d70 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f335a7fc640 1 -- 172.21.15.150:0/2902662311 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3354017070 con 0x7f3364149d70 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f336c1c3640 1 -- 172.21.15.150:0/2902662311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33640be1e0 con 0x7f3364149d70 2024-09-17T05:08:50.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f335a7fc640 1 -- 172.21.15.150:0/2902662311 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354011ce0 con 0x7f3364149d70 2024-09-17T05:08:50.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f336c1c3640 1 -- 172.21.15.150:0/2902662311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33641f11b0 con 0x7f3364149d70 2024-09-17T05:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f336c1c3640 1 -- 172.21.15.150:0/2902662311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33641540b0 con 0x7f3364149d70 2024-09-17T05:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f335a7fc640 1 -- 172.21.15.150:0/2902662311 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f335401c460 con 0x7f3364149d70 2024-09-17T05:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f335a7fc640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3348073200 0x7f3348075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f335a7fc640 1 -- 172.21.15.150:0/2902662311 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f3354094b30 con 0x7f3364149d70 2024-09-17T05:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.332+0000 7f3369f5f640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3348073200 0x7f3348075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:50.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.336+0000 7f3369f5f640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3348073200 0x7f3348075710 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f33600af000 tx=0x7f336001b060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:50.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.340+0000 7f335a7fc640 1 -- 172.21.15.150:0/2902662311 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f335405e8a0 con 0x7f3364149d70 2024-09-17T05:08:50.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.576+0000 7f336c1c3640 1 -- 172.21.15.150:0/2902662311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f336414d880 con 0x7f3364149d70 2024-09-17T05:08:50.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.576+0000 7f335a7fc640 1 -- 172.21.15.150:0/2902662311 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+6540 (secure 0 0 0) 0x7f33540632d0 con 0x7f3364149d70 2024-09-17T05:08:50.576 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:50.576 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":141,"btime":"2024-09-17T05:08:24:299799+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:19.286834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7074,"qdb_cluster":[7074]},"id":2}]} 2024-09-17T05:08:50.576 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 141 2024-09-17T05:08:50.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.580+0000 7f336c1c3640 1 -- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3348073200 msgr2=0x7f3348075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:50.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.580+0000 7f336c1c3640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3348073200 0x7f3348075710 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f33600af000 tx=0x7f336001b060 comp rx=0 tx=0).stop 2024-09-17T05:08:50.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.580+0000 7f336c1c3640 1 -- 172.21.15.150:0/2902662311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3364149d70 msgr2=0x7f3364148450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:50.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.580+0000 7f336c1c3640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3364149d70 0x7f3364148450 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f335400c410 tx=0x7f335400c8d0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.580+0000 7f336c1c3640 1 -- 172.21.15.150:0/2902662311 shutdown_connections 2024-09-17T05:08:50.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.580+0000 7f336c1c3640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3348073200 0x7f3348075710 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.580+0000 7f336c1c3640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336414a180 0x7f3364148990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.580+0000 7f336c1c3640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3364149d70 0x7f3364148450 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.580+0000 7f336c1c3640 1 --2- 172.21.15.150:0/2902662311 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33640c7eb0 0x7f33640ac280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.580+0000 7f336c1c3640 1 -- 172.21.15.150:0/2902662311 >> 172.21.15.150:0/2902662311 conn(0x7f33640aab60 msgr2=0x7f33640c5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:50.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.580+0000 7f336c1c3640 1 -- 172.21.15.150:0/2902662311 shutdown_connections 2024-09-17T05:08:50.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.580+0000 7f336c1c3640 1 -- 172.21.15.150:0/2902662311 wait complete. 2024-09-17T05:08:50.598 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-17T05:08:50.598 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T05:08:50.598 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:08:50.613 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T05:08:50.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:50.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.812+0000 7f0a9ba95640 1 -- 172.21.15.150:0/517992033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a9414a180 msgr2=0x7f0a940c7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:50.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.812+0000 7f0a9ba95640 1 --2- 172.21.15.150:0/517992033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a9414a180 0x7f0a940c7190 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009c80 tx=0x7f0a8c01cda0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.812+0000 7f0a9ba95640 1 -- 172.21.15.150:0/517992033 shutdown_connections 2024-09-17T05:08:50.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.812+0000 7f0a9ba95640 1 --2- 172.21.15.150:0/517992033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a9414a180 0x7f0a940c7190 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.812+0000 7f0a9ba95640 1 --2- 172.21.15.150:0/517992033 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a940c7eb0 0x7f0a941540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.812+0000 7f0a9ba95640 1 -- 172.21.15.150:0/517992033 >> 172.21.15.150:0/517992033 conn(0x7f0a940aab60 msgr2=0x7f0a940aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:50.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.812+0000 7f0a9ba95640 1 -- 172.21.15.150:0/517992033 shutdown_connections 2024-09-17T05:08:50.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.812+0000 7f0a9ba95640 1 -- 172.21.15.150:0/517992033 wait complete. 2024-09-17T05:08:50.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.812+0000 7f0a9ba95640 1 Processor -- start 2024-09-17T05:08:50.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.812+0000 7f0a9ba95640 1 -- start start 2024-09-17T05:08:50.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a9ba95640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a940c7eb0 0x7f0a94151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:50.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a9ba95640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a94149d70 0x7f0a94147480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:50.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a9ba95640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9414a180 0x7f0a94141f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:50.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a9ba95640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0a940c7c60 con 0x7f0a940c7eb0 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a9ba95640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0a940c7ae0 con 0x7f0a94149d70 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a9ba95640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0a94156540 con 0x7f0a9414a180 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a99831640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a940c7eb0 0x7f0a94151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a99831640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a940c7eb0 0x7f0a94151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52990/0 (socket says 172.21.15.150:52990) 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a99831640 1 -- 172.21.15.150:0/405825843 learned_addr learned my addr 172.21.15.150:0/405825843 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a99030640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a94149d70 0x7f0a94147480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a9a032640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9414a180 0x7f0a94141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a99831640 1 -- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9414a180 msgr2=0x7f0a94141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a99831640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9414a180 0x7f0a94141f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a99831640 1 -- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a94149d70 msgr2=0x7f0a94147480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a99831640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a94149d70 0x7f0a94147480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a99831640 1 -- 172.21.15.150:0/405825843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a941530e0 con 0x7f0a940c7eb0 2024-09-17T05:08:50.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a99831640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a940c7eb0 0x7f0a94151ef0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000d750 tx=0x7f0a9000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:50.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a8a7fc640 1 -- 172.21.15.150:0/405825843 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a90017070 con 0x7f0a940c7eb0 2024-09-17T05:08:50.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a8a7fc640 1 -- 172.21.15.150:0/405825843 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a90009d20 con 0x7f0a940c7eb0 2024-09-17T05:08:50.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a8a7fc640 1 -- 172.21.15.150:0/405825843 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a9000de60 con 0x7f0a940c7eb0 2024-09-17T05:08:50.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a9ba95640 1 -- 172.21.15.150:0/405825843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a940c2d00 con 0x7f0a940c7eb0 2024-09-17T05:08:50.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a9ba95640 1 -- 172.21.15.150:0/405825843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a941519d0 con 0x7f0a940c7eb0 2024-09-17T05:08:50.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.816+0000 7f0a9ba95640 1 -- 172.21.15.150:0/405825843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a941540b0 con 0x7f0a940c7eb0 2024-09-17T05:08:50.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.820+0000 7f0a8a7fc640 1 -- 172.21.15.150:0/405825843 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0a90004280 con 0x7f0a940c7eb0 2024-09-17T05:08:50.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.820+0000 7f0a8a7fc640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a78073200 0x7f0a78075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:50.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.820+0000 7f0a8a7fc640 1 -- 172.21.15.150:0/405825843 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f0a90095220 con 0x7f0a940c7eb0 2024-09-17T05:08:50.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.820+0000 7f0a99030640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a78073200 0x7f0a78075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:50.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.820+0000 7f0a99030640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a78073200 0x7f0a78075710 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0a941edaa0 tx=0x7f0a84009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:50.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:50.824+0000 7f0a8a7fc640 1 -- 172.21.15.150:0/405825843 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0a90020020 con 0x7f0a940c7eb0 2024-09-17T05:08:51.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.060+0000 7f0a9ba95640 1 -- 172.21.15.150:0/405825843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a9406e8e0 con 0x7f0a940c7eb0 2024-09-17T05:08:51.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.060+0000 7f0a8a7fc640 1 -- 172.21.15.150:0/405825843 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+6540 (secure 0 0 0) 0x7f0a9005f6f0 con 0x7f0a940c7eb0 2024-09-17T05:08:51.061 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:51.061 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":141,"btime":"2024-09-17T05:08:24:299799+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:19.286834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7074,"qdb_cluster":[7074]},"id":2}]} 2024-09-17T05:08:51.062 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 141 2024-09-17T05:08:51.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.064+0000 7f0a9ba95640 1 -- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a78073200 msgr2=0x7f0a78075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:51.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.064+0000 7f0a9ba95640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a78073200 0x7f0a78075710 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0a941edaa0 tx=0x7f0a84009040 comp rx=0 tx=0).stop 2024-09-17T05:08:51.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.064+0000 7f0a9ba95640 1 -- 172.21.15.150:0/405825843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a940c7eb0 msgr2=0x7f0a94151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:51.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.064+0000 7f0a9ba95640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a940c7eb0 0x7f0a94151ef0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000d750 tx=0x7f0a9000dc10 comp rx=0 tx=0).stop 2024-09-17T05:08:51.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.064+0000 7f0a9ba95640 1 -- 172.21.15.150:0/405825843 shutdown_connections 2024-09-17T05:08:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.064+0000 7f0a9ba95640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a78073200 0x7f0a78075710 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.064+0000 7f0a9ba95640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a94149d70 0x7f0a94147480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.064+0000 7f0a9ba95640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a9414a180 0x7f0a94141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.064+0000 7f0a9ba95640 1 --2- 172.21.15.150:0/405825843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a940c7eb0 0x7f0a94151ef0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.064+0000 7f0a9ba95640 1 -- 172.21.15.150:0/405825843 >> 172.21.15.150:0/405825843 conn(0x7f0a940aab60 msgr2=0x7f0a9415e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.064+0000 7f0a9ba95640 1 -- 172.21.15.150:0/405825843 shutdown_connections 2024-09-17T05:08:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:51.064+0000 7f0a9ba95640 1 -- 172.21.15.150:0/405825843 wait complete. 2024-09-17T05:08:51.087 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:08:53.089 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:53.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.256+0000 7fc1cd155640 1 -- 172.21.15.150:0/1126237055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c806e370 msgr2=0x7fc1c8153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:53.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.256+0000 7fc1cd155640 1 --2- 172.21.15.150:0/1126237055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c806e370 0x7fc1c8153df0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009c80 tx=0x7fc1c001cc80 comp rx=0 tx=0).stop 2024-09-17T05:08:53.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.256+0000 7fc1cd155640 1 -- 172.21.15.150:0/1126237055 shutdown_connections 2024-09-17T05:08:53.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.256+0000 7fc1cd155640 1 --2- 172.21.15.150:0/1126237055 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c806e370 0x7fc1c8153df0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:53.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.256+0000 7fc1cd155640 1 --2- 172.21.15.150:0/1126237055 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1c80ab9d0 0x7fc1c814da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:53.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.256+0000 7fc1cd155640 1 -- 172.21.15.150:0/1126237055 >> 172.21.15.150:0/1126237055 conn(0x7fc1c8070b50 msgr2=0x7fc1c8070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:53.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.256+0000 7fc1cd155640 1 -- 172.21.15.150:0/1126237055 shutdown_connections 2024-09-17T05:08:53.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.256+0000 7fc1cd155640 1 -- 172.21.15.150:0/1126237055 wait complete. 2024-09-17T05:08:53.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.260+0000 7fc1cd155640 1 Processor -- start 2024-09-17T05:08:53.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.260+0000 7fc1cd155640 1 -- start start 2024-09-17T05:08:53.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.260+0000 7fc1cd155640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c806df60 0x7fc1c80c1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:53.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.260+0000 7fc1cd155640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1c806e370 0x7fc1c80adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:53.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.260+0000 7fc1cd155640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1c80ab9d0 0x7fc1c8147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:53.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.260+0000 7fc1cd155640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c8069bf0 con 0x7fc1c806df60 2024-09-17T05:08:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.260+0000 7fc1cd155640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c81543f0 con 0x7fc1c806e370 2024-09-17T05:08:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.260+0000 7fc1cd155640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c8069d70 con 0x7fc1c80ab9d0 2024-09-17T05:08:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1c6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c806df60 0x7fc1c80c1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1c6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c806df60 0x7fc1c80c1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53018/0 (socket says 172.21.15.150:53018) 2024-09-17T05:08:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1c6d9d640 1 -- 172.21.15.150:0/3220370387 learned_addr learned my addr 172.21.15.150:0/3220370387 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1c759e640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1c80ab9d0 0x7fc1c8147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1c659c640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1c806e370 0x7fc1c80adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1c6d9d640 1 -- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1c80ab9d0 msgr2=0x7fc1c8147000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1c6d9d640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1c80ab9d0 0x7fc1c8147000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1c6d9d640 1 -- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1c806e370 msgr2=0x7fc1c80adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1c6d9d640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1c806e370 0x7fc1c80adca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:53.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1c6d9d640 1 -- 172.21.15.150:0/3220370387 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c8150c20 con 0x7fc1c806df60 2024-09-17T05:08:53.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1c6d9d640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c806df60 0x7fc1c80c1cb0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800d950 tx=0x7fc1b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:53.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1af7fe640 1 -- 172.21.15.150:0/3220370387 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8017070 con 0x7fc1c806df60 2024-09-17T05:08:53.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1af7fe640 1 -- 172.21.15.150:0/3220370387 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1b8009d20 con 0x7fc1c806df60 2024-09-17T05:08:53.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1cd155640 1 -- 172.21.15.150:0/3220370387 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c8150980 con 0x7fc1c806df60 2024-09-17T05:08:53.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1af7fe640 1 -- 172.21.15.150:0/3220370387 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80106c0 con 0x7fc1c806df60 2024-09-17T05:08:53.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1cd155640 1 -- 172.21.15.150:0/3220370387 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c80ad860 con 0x7fc1c806df60 2024-09-17T05:08:53.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1cd155640 1 -- 172.21.15.150:0/3220370387 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c814da10 con 0x7fc1c806df60 2024-09-17T05:08:53.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.264+0000 7fc1af7fe640 1 -- 172.21.15.150:0/3220370387 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc1b801a070 con 0x7fc1c806df60 2024-09-17T05:08:53.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.268+0000 7fc1af7fe640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc1b4073200 0x7fc1b4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:53.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.268+0000 7fc1af7fe640 1 -- 172.21.15.150:0/3220370387 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7fc1b8095090 con 0x7fc1c806df60 2024-09-17T05:08:53.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.268+0000 7fc1c659c640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc1b4073200 0x7fc1b4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:53.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.268+0000 7fc1c659c640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc1b4073200 0x7fc1b4075710 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc1c80c4a70 tx=0x7fc1bc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:53.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.272+0000 7fc1af7fe640 1 -- 172.21.15.150:0/3220370387 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc1b805f4d0 con 0x7fc1c806df60 2024-09-17T05:08:53.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.536+0000 7fc1cd155640 1 -- 172.21.15.150:0/3220370387 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1c8075060 con 0x7fc1c806df60 2024-09-17T05:08:53.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.536+0000 7fc1af7fe640 1 -- 172.21.15.150:0/3220370387 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+6540 (secure 0 0 0) 0x7fc1b805edd0 con 0x7fc1c806df60 2024-09-17T05:08:53.537 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:53.538 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":141,"btime":"2024-09-17T05:08:24:299799+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:19.286834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7074,"qdb_cluster":[7074]},"id":2}]} 2024-09-17T05:08:53.538 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 141 2024-09-17T05:08:53.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.540+0000 7fc1cd155640 1 -- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc1b4073200 msgr2=0x7fc1b4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:53.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.540+0000 7fc1cd155640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc1b4073200 0x7fc1b4075710 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc1c80c4a70 tx=0x7fc1bc009040 comp rx=0 tx=0).stop 2024-09-17T05:08:53.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.540+0000 7fc1cd155640 1 -- 172.21.15.150:0/3220370387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c806df60 msgr2=0x7fc1c80c1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:53.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.540+0000 7fc1cd155640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c806df60 0x7fc1c80c1cb0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800d950 tx=0x7fc1b800de10 comp rx=0 tx=0).stop 2024-09-17T05:08:53.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.540+0000 7fc1cd155640 1 -- 172.21.15.150:0/3220370387 shutdown_connections 2024-09-17T05:08:53.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.540+0000 7fc1cd155640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc1b4073200 0x7fc1b4075710 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:53.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.540+0000 7fc1cd155640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc1c806e370 0x7fc1c80adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:53.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.540+0000 7fc1cd155640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc1c80ab9d0 0x7fc1c8147000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:53.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.540+0000 7fc1cd155640 1 --2- 172.21.15.150:0/3220370387 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc1c806df60 0x7fc1c80c1cb0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:53.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.540+0000 7fc1cd155640 1 -- 172.21.15.150:0/3220370387 >> 172.21.15.150:0/3220370387 conn(0x7fc1c8070b50 msgr2=0x7fc1c81481c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:53.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.540+0000 7fc1cd155640 1 -- 172.21.15.150:0/3220370387 shutdown_connections 2024-09-17T05:08:53.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:53.540+0000 7fc1cd155640 1 -- 172.21.15.150:0/3220370387 wait complete. 2024-09-17T05:08:53.561 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:08:55.562 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.744+0000 7f74c1990640 1 -- 172.21.15.150:0/2684407443 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74bc0c37b0 msgr2=0x7f74bc14d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.744+0000 7f74c1990640 1 --2- 172.21.15.150:0/2684407443 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74bc0c37b0 0x7f74bc14d530 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009900 tx=0x7f74ac01c850 comp rx=0 tx=0).stop 2024-09-17T05:08:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.744+0000 7f74c1990640 1 -- 172.21.15.150:0/2684407443 shutdown_connections 2024-09-17T05:08:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.744+0000 7f74c1990640 1 --2- 172.21.15.150:0/2684407443 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f74bc0c5c90 0x7f74bc0c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.744+0000 7f74c1990640 1 --2- 172.21.15.150:0/2684407443 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74bc0c37b0 0x7f74bc14d530 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.744+0000 7f74c1990640 1 -- 172.21.15.150:0/2684407443 >> 172.21.15.150:0/2684407443 conn(0x7f74bc0aaf60 msgr2=0x7f74bc0ab370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.744+0000 7f74c1990640 1 -- 172.21.15.150:0/2684407443 shutdown_connections 2024-09-17T05:08:55.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.744+0000 7f74c1990640 1 -- 172.21.15.150:0/2684407443 wait complete. 2024-09-17T05:08:55.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74c1990640 1 Processor -- start 2024-09-17T05:08:55.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74c1990640 1 -- start start 2024-09-17T05:08:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74c1990640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74bc0c37b0 0x7f74bc152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74c1990640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f74bc0c5c90 0x7f74bc1f09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74c1990640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74bc153570 0x7f74bc14c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74c1990640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f74bc072ee0 con 0x7f74bc0c37b0 2024-09-17T05:08:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74c1990640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f74bc072d60 con 0x7f74bc153570 2024-09-17T05:08:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74c1990640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f74bc160630 con 0x7f74bc0c5c90 2024-09-17T05:08:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74baffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74bc0c37b0 0x7f74bc152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74baffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74bc0c37b0 0x7f74bc152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53038/0 (socket says 172.21.15.150:53038) 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74baffd640 1 -- 172.21.15.150:0/2789949962 learned_addr learned my addr 172.21.15.150:0/2789949962 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74ba7fc640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f74bc0c5c90 0x7f74bc1f09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74bb7fe640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74bc153570 0x7f74bc14c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74baffd640 1 -- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f74bc0c5c90 msgr2=0x7f74bc1f09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74baffd640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f74bc0c5c90 0x7f74bc1f09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74baffd640 1 -- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74bc153570 msgr2=0x7f74bc14c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74baffd640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74bc153570 0x7f74bc14c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74baffd640 1 -- 172.21.15.150:0/2789949962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74bc1526a0 con 0x7f74bc0c37b0 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74baffd640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74bc0c37b0 0x7f74bc152dd0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009850 tx=0x7f74ac0a5700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74a3fff640 1 -- 172.21.15.150:0/2789949962 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac0b2040 con 0x7f74bc0c37b0 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74a3fff640 1 -- 172.21.15.150:0/2789949962 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74ac004340 con 0x7f74bc0c37b0 2024-09-17T05:08:55.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74c1990640 1 -- 172.21.15.150:0/2789949962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74bc151860 con 0x7f74bc0c37b0 2024-09-17T05:08:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.748+0000 7f74a3fff640 1 -- 172.21.15.150:0/2789949962 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac0044f0 con 0x7f74bc0c37b0 2024-09-17T05:08:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.752+0000 7f74c1990640 1 -- 172.21.15.150:0/2789949962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74bc1520a0 con 0x7f74bc0c37b0 2024-09-17T05:08:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.752+0000 7f74c1990640 1 -- 172.21.15.150:0/2789949962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74bc151680 con 0x7f74bc0c37b0 2024-09-17T05:08:55.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.752+0000 7f74a3fff640 1 -- 172.21.15.150:0/2789949962 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f74ac003860 con 0x7f74bc0c37b0 2024-09-17T05:08:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.752+0000 7f74a3fff640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f74a8073200 0x7f74a8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.752+0000 7f74a3fff640 1 -- 172.21.15.150:0/2789949962 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f74ac12ef40 con 0x7f74bc0c37b0 2024-09-17T05:08:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.752+0000 7f74ba7fc640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f74a8073200 0x7f74a8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:55.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.756+0000 7f74ba7fc640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f74a8073200 0x7f74a8075710 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f74b0005fd0 tx=0x7f74b0005d10 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:55.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:55.760+0000 7f74a3fff640 1 -- 172.21.15.150:0/2789949962 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f74bc151680 con 0x7f74bc0c37b0 2024-09-17T05:08:56.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.008+0000 7f74c1990640 1 -- 172.21.15.150:0/2789949962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74bc153310 con 0x7f74bc0c37b0 2024-09-17T05:08:56.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.008+0000 7f74a3fff640 1 -- 172.21.15.150:0/2789949962 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+6540 (secure 0 0 0) 0x7f74ac0164d0 con 0x7f74bc0c37b0 2024-09-17T05:08:56.010 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:56.010 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":141,"btime":"2024-09-17T05:08:24:299799+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:19.286834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7074,"qdb_cluster":[7074]},"id":2}]} 2024-09-17T05:08:56.010 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 141 2024-09-17T05:08:56.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.012+0000 7f74c1990640 1 -- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f74a8073200 msgr2=0x7f74a8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:56.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.012+0000 7f74c1990640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f74a8073200 0x7f74a8075710 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f74b0005fd0 tx=0x7f74b0005d10 comp rx=0 tx=0).stop 2024-09-17T05:08:56.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.012+0000 7f74c1990640 1 -- 172.21.15.150:0/2789949962 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74bc0c37b0 msgr2=0x7f74bc152dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:56.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.012+0000 7f74c1990640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74bc0c37b0 0x7f74bc152dd0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009850 tx=0x7f74ac0a5700 comp rx=0 tx=0).stop 2024-09-17T05:08:56.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.012+0000 7f74c1990640 1 -- 172.21.15.150:0/2789949962 shutdown_connections 2024-09-17T05:08:56.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.012+0000 7f74c1990640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f74a8073200 0x7f74a8075710 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:56.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.012+0000 7f74c1990640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f74bc153570 0x7f74bc14c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:56.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.012+0000 7f74c1990640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f74bc0c5c90 0x7f74bc1f09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:56.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.012+0000 7f74c1990640 1 --2- 172.21.15.150:0/2789949962 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f74bc0c37b0 0x7f74bc152dd0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:56.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.012+0000 7f74c1990640 1 -- 172.21.15.150:0/2789949962 >> 172.21.15.150:0/2789949962 conn(0x7f74bc0aaf60 msgr2=0x7f74bc145130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:56.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.012+0000 7f74c1990640 1 -- 172.21.15.150:0/2789949962 shutdown_connections 2024-09-17T05:08:56.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:56.012+0000 7f74c1990640 1 -- 172.21.15.150:0/2789949962 wait complete. 2024-09-17T05:08:56.031 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:08:58.032 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:08:58.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.200+0000 7f496a333640 1 -- 172.21.15.150:0/1343903833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4964157ca0 msgr2=0x7f49641580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:58.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.200+0000 7f496a333640 1 --2- 172.21.15.150:0/1343903833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4964157ca0 0x7f49641580b0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f49640c7870 tx=0x7f495c01cc60 comp rx=0 tx=0).stop 2024-09-17T05:08:58.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.200+0000 7f496a333640 1 -- 172.21.15.150:0/1343903833 shutdown_connections 2024-09-17T05:08:58.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.200+0000 7f496a333640 1 --2- 172.21.15.150:0/1343903833 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4964157ca0 0x7f49641580b0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:58.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.200+0000 7f496a333640 1 --2- 172.21.15.150:0/1343903833 >> v2:172.21.15.176:3300/0 conn(0x7f49640ad7f0 0x7f49640c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:58.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.200+0000 7f496a333640 1 -- 172.21.15.150:0/1343903833 >> 172.21.15.150:0/1343903833 conn(0x7f49640c9de0 msgr2=0x7f49640ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:58.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.200+0000 7f496a333640 1 -- 172.21.15.150:0/1343903833 shutdown_connections 2024-09-17T05:08:58.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.200+0000 7f496a333640 1 -- 172.21.15.150:0/1343903833 wait complete. 2024-09-17T05:08:58.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.204+0000 7f496a333640 1 Processor -- start 2024-09-17T05:08:58.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.204+0000 7f496a333640 1 -- start start 2024-09-17T05:08:58.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.204+0000 7f496a333640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49640ad7f0 0x7f496414d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:58.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.204+0000 7f496a333640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4964151e10 0x7f4964072780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:58.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.204+0000 7f496a333640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4964157ca0 0x7f4964156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:58.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.204+0000 7f496a333640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4964158770 con 0x7f4964151e10 2024-09-17T05:08:58.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.204+0000 7f496a333640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f49641585f0 con 0x7f4964157ca0 2024-09-17T05:08:58.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.204+0000 7f496a333640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f49640c7ce0 con 0x7f49640ad7f0 2024-09-17T05:08:58.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.204+0000 7f495bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4964151e10 0x7f4964072780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:58.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.204+0000 7f495bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4964151e10 0x7f4964072780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53062/0 (socket says 172.21.15.150:53062) 2024-09-17T05:08:58.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.204+0000 7f495bfff640 1 -- 172.21.15.150:0/3004383944 learned_addr learned my addr 172.21.15.150:0/3004383944 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:08:58.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.204+0000 7f49688d0640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4964157ca0 0x7f4964156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:58.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f4963fff640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49640ad7f0 0x7f496414d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:58.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f495bfff640 1 -- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49640ad7f0 msgr2=0x7f496414d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:58.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f495bfff640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49640ad7f0 0x7f496414d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:58.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f495bfff640 1 -- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4964157ca0 msgr2=0x7f4964156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:58.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f495bfff640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4964157ca0 0x7f4964156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:58.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f495bfff640 1 -- 172.21.15.150:0/3004383944 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49641561c0 con 0x7f4964151e10 2024-09-17T05:08:58.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f4963fff640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49640ad7f0 0x7f496414d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:08:58.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f495bfff640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4964151e10 0x7f4964072780 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f494c00c3f0 tx=0x7f494c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:58.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f49617fa640 1 -- 172.21.15.150:0/3004383944 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c017070 con 0x7f4964151e10 2024-09-17T05:08:58.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f496a333640 1 -- 172.21.15.150:0/3004383944 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49641564a0 con 0x7f4964151e10 2024-09-17T05:08:58.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f49617fa640 1 -- 172.21.15.150:0/3004383944 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f494c0053c0 con 0x7f4964151e10 2024-09-17T05:08:58.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f49617fa640 1 -- 172.21.15.150:0/3004383944 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c003860 con 0x7f4964151e10 2024-09-17T05:08:58.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f496a333640 1 -- 172.21.15.150:0/3004383944 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4964157000 con 0x7f4964151e10 2024-09-17T05:08:58.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f496a333640 1 -- 172.21.15.150:0/3004383944 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49640c6230 con 0x7f4964151e10 2024-09-17T05:08:58.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f49617fa640 1 -- 172.21.15.150:0/3004383944 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f494c01a070 con 0x7f4964151e10 2024-09-17T05:08:58.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f49617fa640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4944073230 0x7f4944075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:08:58.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.208+0000 7f49617fa640 1 -- 172.21.15.150:0/3004383944 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f494c094100 con 0x7f4964151e10 2024-09-17T05:08:58.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.212+0000 7f4963fff640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4944073230 0x7f4944075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:08:58.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.212+0000 7f4963fff640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4944073230 0x7f4944075740 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4954005fd0 tx=0x7f4954005e20 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:08:58.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.216+0000 7f49617fa640 1 -- 172.21.15.150:0/3004383944 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f494c05de70 con 0x7f4964151e10 2024-09-17T05:08:58.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.456+0000 7f496a333640 1 -- 172.21.15.150:0/3004383944 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4964156670 con 0x7f4964151e10 2024-09-17T05:08:58.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f49617fa640 1 -- 172.21.15.150:0/3004383944 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+6540 (secure 0 0 0) 0x7f494c0628a0 con 0x7f4964151e10 2024-09-17T05:08:58.458 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:08:58.459 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":141,"btime":"2024-09-17T05:08:24:299799+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:19.286834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7074,"qdb_cluster":[7074]},"id":2}]} 2024-09-17T05:08:58.459 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 141 2024-09-17T05:08:58.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f496a333640 1 -- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4944073230 msgr2=0x7f4944075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:58.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f496a333640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4944073230 0x7f4944075740 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4954005fd0 tx=0x7f4954005e20 comp rx=0 tx=0).stop 2024-09-17T05:08:58.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f496a333640 1 -- 172.21.15.150:0/3004383944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4964151e10 msgr2=0x7f4964072780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:08:58.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f496a333640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4964151e10 0x7f4964072780 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f494c00c3f0 tx=0x7f494c00c8b0 comp rx=0 tx=0).stop 2024-09-17T05:08:58.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f496a333640 1 -- 172.21.15.150:0/3004383944 shutdown_connections 2024-09-17T05:08:58.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f496a333640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4944073230 0x7f4944075740 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:58.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f496a333640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4964157ca0 0x7f4964156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:58.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f496a333640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4964151e10 0x7f4964072780 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:58.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f496a333640 1 --2- 172.21.15.150:0/3004383944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f49640ad7f0 0x7f496414d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:08:58.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f496a333640 1 -- 172.21.15.150:0/3004383944 >> 172.21.15.150:0/3004383944 conn(0x7f49640c9de0 msgr2=0x7f49640c8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:08:58.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f496a333640 1 -- 172.21.15.150:0/3004383944 shutdown_connections 2024-09-17T05:08:58.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:08:58.460+0000 7f496a333640 1 -- 172.21.15.150:0/3004383944 wait complete. 2024-09-17T05:08:58.480 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:09:00.480 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:00.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.648+0000 7f7fb3d68640 1 -- 172.21.15.150:0/4203831351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fac157af0 msgr2=0x7f7fac0a8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:00.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.648+0000 7f7fb3d68640 1 --2- 172.21.15.150:0/4203831351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fac157af0 0x7f7fac0a8120 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0c4070 tx=0x7f7fa801c830 comp rx=0 tx=0).stop 2024-09-17T05:09:00.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.648+0000 7f7fb3d68640 1 -- 172.21.15.150:0/4203831351 shutdown_connections 2024-09-17T05:09:00.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.648+0000 7f7fb3d68640 1 --2- 172.21.15.150:0/4203831351 >> v2:172.21.15.176:3300/0 conn(0x7f7fac074cf0 0x7f7fac0c7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:00.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.648+0000 7f7fb3d68640 1 --2- 172.21.15.150:0/4203831351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fac157af0 0x7f7fac0a8120 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:00.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.648+0000 7f7fb3d68640 1 -- 172.21.15.150:0/4203831351 >> 172.21.15.150:0/4203831351 conn(0x7f7fac0c9150 msgr2=0x7f7fac0c9560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:00.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.648+0000 7f7fb3d68640 1 -- 172.21.15.150:0/4203831351 shutdown_connections 2024-09-17T05:09:00.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.648+0000 7f7fb3d68640 1 -- 172.21.15.150:0/4203831351 wait complete. 2024-09-17T05:09:00.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb3d68640 1 Processor -- start 2024-09-17T05:09:00.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb3d68640 1 -- start start 2024-09-17T05:09:00.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb3d68640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7fac0748e0 0x7f7fac0c2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:00.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb3d68640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7fac074cf0 0x7f7fac14e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:00.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb3d68640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fac157af0 0x7f7fac15d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:00.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb3d68640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac154740 con 0x7f7fac157af0 2024-09-17T05:09:00.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb3d68640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7fac1545c0 con 0x7f7fac0748e0 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb3d68640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7fac1548c0 con 0x7f7fac074cf0 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb2305640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fac157af0 0x7f7fac15d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb2305640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fac157af0 0x7f7fac15d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60540/0 (socket says 172.21.15.150:60540) 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb2305640 1 -- 172.21.15.150:0/852735418 learned_addr learned my addr 172.21.15.150:0/852735418 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb1b04640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7fac0748e0 0x7f7fac0c2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb1303640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7fac074cf0 0x7f7fac14e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb2305640 1 -- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7fac074cf0 msgr2=0x7f7fac14e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb2305640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7fac074cf0 0x7f7fac14e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb2305640 1 -- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7fac0748e0 msgr2=0x7f7fac0c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb2305640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7fac0748e0 0x7f7fac0c2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb2305640 1 -- 172.21.15.150:0/852735418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fac0ade10 con 0x7f7fac157af0 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.652+0000 7f7fb2305640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fac157af0 0x7f7fac15d4c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400ba10 tx=0x7f7fa400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.656+0000 7f7fa27fc640 1 -- 172.21.15.150:0/852735418 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4019070 con 0x7f7fac157af0 2024-09-17T05:09:00.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.656+0000 7f7fa27fc640 1 -- 172.21.15.150:0/852735418 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fa400cc50 con 0x7f7fac157af0 2024-09-17T05:09:00.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.656+0000 7f7fb3d68640 1 -- 172.21.15.150:0/852735418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fac14fc00 con 0x7f7fac157af0 2024-09-17T05:09:00.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.656+0000 7f7fa27fc640 1 -- 172.21.15.150:0/852735418 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa40146a0 con 0x7f7fac157af0 2024-09-17T05:09:00.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.656+0000 7f7fb3d68640 1 -- 172.21.15.150:0/852735418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fac0acba0 con 0x7f7fac157af0 2024-09-17T05:09:00.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.656+0000 7f7fb3d68640 1 -- 172.21.15.150:0/852735418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fac14fe40 con 0x7f7fac157af0 2024-09-17T05:09:00.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.656+0000 7f7fa27fc640 1 -- 172.21.15.150:0/852735418 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7fa401c070 con 0x7f7fac157af0 2024-09-17T05:09:00.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.656+0000 7f7fa27fc640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7f90073230 0x7f7f90075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:00.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.656+0000 7f7fa27fc640 1 -- 172.21.15.150:0/852735418 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f7fa4094af0 con 0x7f7fac157af0 2024-09-17T05:09:00.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.656+0000 7f7fb1b04640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7f90073230 0x7f7f90075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:00.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.660+0000 7f7fb1b04640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7f90073230 0x7f7f90075740 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80af000 tx=0x7f7fa801b070 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:00.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.664+0000 7f7fa27fc640 1 -- 172.21.15.150:0/852735418 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7fac14fe40 con 0x7f7fac157af0 2024-09-17T05:09:00.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.868+0000 7f7fb3d68640 1 -- 172.21.15.150:0/852735418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7fac0a8120 con 0x7f7fac157af0 2024-09-17T05:09:00.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.868+0000 7f7fa27fc640 1 -- 172.21.15.150:0/852735418 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+6540 (secure 0 0 0) 0x7f7fa405ef30 con 0x7f7fac157af0 2024-09-17T05:09:00.868 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:00.869 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":141,"btime":"2024-09-17T05:08:24:299799+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:19.286834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7074,"qdb_cluster":[7074]},"id":2}]} 2024-09-17T05:09:00.869 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 141 2024-09-17T05:09:00.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.872+0000 7f7fb3d68640 1 -- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7f90073230 msgr2=0x7f7f90075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:00.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.872+0000 7f7fb3d68640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7f90073230 0x7f7f90075740 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80af000 tx=0x7f7fa801b070 comp rx=0 tx=0).stop 2024-09-17T05:09:00.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.872+0000 7f7fb3d68640 1 -- 172.21.15.150:0/852735418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fac157af0 msgr2=0x7f7fac15d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:00.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.872+0000 7f7fb3d68640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fac157af0 0x7f7fac15d4c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400ba10 tx=0x7f7fa400bed0 comp rx=0 tx=0).stop 2024-09-17T05:09:00.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.872+0000 7f7fb3d68640 1 -- 172.21.15.150:0/852735418 shutdown_connections 2024-09-17T05:09:00.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.872+0000 7f7fb3d68640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7f90073230 0x7f7f90075740 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:00.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.872+0000 7f7fb3d68640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7fac157af0 0x7f7fac15d4c0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:00.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.872+0000 7f7fb3d68640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7fac074cf0 0x7f7fac14e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:00.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.872+0000 7f7fb3d68640 1 --2- 172.21.15.150:0/852735418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7fac0748e0 0x7f7fac0c2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:00.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.872+0000 7f7fb3d68640 1 -- 172.21.15.150:0/852735418 >> 172.21.15.150:0/852735418 conn(0x7f7fac0c9150 msgr2=0x7f7fac0a89f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:00.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.872+0000 7f7fb3d68640 1 -- 172.21.15.150:0/852735418 shutdown_connections 2024-09-17T05:09:00.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:00.872+0000 7f7fb3d68640 1 -- 172.21.15.150:0/852735418 wait complete. 2024-09-17T05:09:00.893 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:09:02.894 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:03.012 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:09:03.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:09:03.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.072+0000 7f2306a9c640 1 -- 172.21.15.150:0/3301265142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23000745c0 msgr2=0x7f2300150d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:03.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.072+0000 7f2306a9c640 1 --2- 172.21.15.150:0/3301265142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23000745c0 0x7f2300150d00 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009900 tx=0x7f22f401c850 comp rx=0 tx=0).stop 2024-09-17T05:09:03.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.072+0000 7f2306a9c640 1 -- 172.21.15.150:0/3301265142 shutdown_connections 2024-09-17T05:09:03.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.072+0000 7f2306a9c640 1 --2- 172.21.15.150:0/3301265142 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23000700b0 0x7f230015dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:03.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.072+0000 7f2306a9c640 1 --2- 172.21.15.150:0/3301265142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23000745c0 0x7f2300150d00 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:03.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.072+0000 7f2306a9c640 1 -- 172.21.15.150:0/3301265142 >> 172.21.15.150:0/3301265142 conn(0x7f23000c5ae0 msgr2=0x7f2300073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:03.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.072+0000 7f2306a9c640 1 -- 172.21.15.150:0/3301265142 shutdown_connections 2024-09-17T05:09:03.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.072+0000 7f2306a9c640 1 -- 172.21.15.150:0/3301265142 wait complete. 2024-09-17T05:09:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.072+0000 7f2306a9c640 1 Processor -- start 2024-09-17T05:09:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.072+0000 7f2306a9c640 1 -- start start 2024-09-17T05:09:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2306a9c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f230006fca0 0x7f2300146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2306a9c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23000700b0 0x7f23001eda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2306a9c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23000745c0 0x7f2300142240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2306a9c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f23000c3020 con 0x7f23000745c0 2024-09-17T05:09:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2306a9c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2300151350 con 0x7f230006fca0 2024-09-17T05:09:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2306a9c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f23000c31a0 con 0x7f23000700b0 2024-09-17T05:09:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2305039640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23000745c0 0x7f2300142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2305039640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23000745c0 0x7f2300142240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60554/0 (socket says 172.21.15.150:60554) 2024-09-17T05:09:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2305039640 1 -- 172.21.15.150:0/4258255770 learned_addr learned my addr 172.21.15.150:0/4258255770 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f22fffff640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23000700b0 0x7f23001eda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2304838640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f230006fca0 0x7f2300146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2305039640 1 -- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23000700b0 msgr2=0x7f23001eda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2305039640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23000700b0 0x7f23001eda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2305039640 1 -- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f230006fca0 msgr2=0x7f2300146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2305039640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f230006fca0 0x7f2300146470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2305039640 1 -- 172.21.15.150:0/4258255770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2300145c50 con 0x7f23000745c0 2024-09-17T05:09:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2305039640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23000745c0 0x7f2300142240 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f22f800e9a0 tx=0x7f22f800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f22fd7fa640 1 -- 172.21.15.150:0/4258255770 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f8004d30 con 0x7f23000745c0 2024-09-17T05:09:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f22fd7fa640 1 -- 172.21.15.150:0/4258255770 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22f800cb20 con 0x7f23000745c0 2024-09-17T05:09:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f22fd7fa640 1 -- 172.21.15.150:0/4258255770 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f8005020 con 0x7f23000745c0 2024-09-17T05:09:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2306a9c640 1 -- 172.21.15.150:0/4258255770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2300154ab0 con 0x7f23000745c0 2024-09-17T05:09:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2306a9c640 1 -- 172.21.15.150:0/4258255770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2300142f50 con 0x7f23000745c0 2024-09-17T05:09:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f2306a9c640 1 -- 172.21.15.150:0/4258255770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23000ab140 con 0x7f23000745c0 2024-09-17T05:09:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f22fd7fa640 1 -- 172.21.15.150:0/4258255770 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f22f80051c0 con 0x7f23000745c0 2024-09-17T05:09:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.076+0000 7f22fd7fa640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f22e4073230 0x7f22e4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:03.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.080+0000 7f22fd7fa640 1 -- 172.21.15.150:0/4258255770 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f22f80955a0 con 0x7f23000745c0 2024-09-17T05:09:03.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.080+0000 7f2304838640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f22e4073230 0x7f22e4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:03.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.080+0000 7f2304838640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f22e4073230 0x7f22e4075740 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f22f40af000 tx=0x7f22f401b090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:03.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.084+0000 7f22fd7fa640 1 -- 172.21.15.150:0/4258255770 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f22f8016030 con 0x7f23000745c0 2024-09-17T05:09:03.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.304+0000 7f2306a9c640 1 -- 172.21.15.150:0/4258255770 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23000c36c0 con 0x7f23000745c0 2024-09-17T05:09:03.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f22fd7fa640 1 -- 172.21.15.150:0/4258255770 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+6540 (secure 0 0 0) 0x7f22f805f420 con 0x7f23000745c0 2024-09-17T05:09:03.307 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:03.307 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":141,"btime":"2024-09-17T05:08:24:299799+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:19.286834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7074,"qdb_cluster":[7074]},"id":2}]} 2024-09-17T05:09:03.307 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 141 2024-09-17T05:09:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f2306a9c640 1 -- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f22e4073230 msgr2=0x7f22e4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f2306a9c640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f22e4073230 0x7f22e4075740 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f22f40af000 tx=0x7f22f401b090 comp rx=0 tx=0).stop 2024-09-17T05:09:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f2306a9c640 1 -- 172.21.15.150:0/4258255770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23000745c0 msgr2=0x7f2300142240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f2306a9c640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23000745c0 0x7f2300142240 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f22f800e9a0 tx=0x7f22f800ee60 comp rx=0 tx=0).stop 2024-09-17T05:09:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f2306a9c640 1 -- 172.21.15.150:0/4258255770 shutdown_connections 2024-09-17T05:09:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f2306a9c640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f22e4073230 0x7f22e4075740 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f2306a9c640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23000745c0 0x7f2300142240 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f2306a9c640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f23000700b0 0x7f23001eda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f2306a9c640 1 --2- 172.21.15.150:0/4258255770 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f230006fca0 0x7f2300146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f2306a9c640 1 -- 172.21.15.150:0/4258255770 >> 172.21.15.150:0/4258255770 conn(0x7f23000c5ae0 msgr2=0x7f230014a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f2306a9c640 1 -- 172.21.15.150:0/4258255770 shutdown_connections 2024-09-17T05:09:03.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:03.308+0000 7f2306a9c640 1 -- 172.21.15.150:0/4258255770 wait complete. 2024-09-17T05:09:03.332 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:09:05.333 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:05.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 -- 172.21.15.150:0/3041820419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbd814e490 msgr2=0x7fcbd80c7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:05.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 --2- 172.21.15.150:0/3041820419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbd814e490 0x7fcbd80c7d60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4009840 tx=0x7fcbc401c820 comp rx=0 tx=0).stop 2024-09-17T05:09:05.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 -- 172.21.15.150:0/3041820419 shutdown_connections 2024-09-17T05:09:05.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 --2- 172.21.15.150:0/3041820419 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbd814e490 0x7fcbd80c7d60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:05.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 --2- 172.21.15.150:0/3041820419 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbd8071670 0x7fcbd80c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:05.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 -- 172.21.15.150:0/3041820419 >> 172.21.15.150:0/3041820419 conn(0x7fcbd80ca460 msgr2=0x7fcbd80c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:05.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 -- 172.21.15.150:0/3041820419 shutdown_connections 2024-09-17T05:09:05.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 -- 172.21.15.150:0/3041820419 wait complete. 2024-09-17T05:09:05.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 Processor -- start 2024-09-17T05:09:05.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 -- start start 2024-09-17T05:09:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbd8071670 0x7fcbd81eea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbd814e490 0x7fcbd80c3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbd8157c00 0x7fcbd8150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd80c6740 con 0x7fcbd8157c00 2024-09-17T05:09:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd80c65c0 con 0x7fcbd8071670 2024-09-17T05:09:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.512+0000 7fcbdefab640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcbd814e8a0 con 0x7fcbd814e490 2024-09-17T05:09:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdd548640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbd8157c00 0x7fcbd8150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdcd47640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbd8071670 0x7fcbd81eea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdd548640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbd8157c00 0x7fcbd8150fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60568/0 (socket says 172.21.15.150:60568) 2024-09-17T05:09:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdd548640 1 -- 172.21.15.150:0/3057686303 learned_addr learned my addr 172.21.15.150:0/3057686303 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbd7fff640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbd814e490 0x7fcbd80c3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:05.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdd548640 1 -- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbd814e490 msgr2=0x7fcbd80c3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:05.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdd548640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbd814e490 0x7fcbd80c3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:05.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdd548640 1 -- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbd8071670 msgr2=0x7fcbd81eea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:05.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdd548640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbd8071670 0x7fcbd81eea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:05.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdd548640 1 -- 172.21.15.150:0/3057686303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd80bf3d0 con 0x7fcbd8157c00 2024-09-17T05:09:05.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdd548640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbd8157c00 0x7fcbd8150fb0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000c6e0 tx=0x7fcbd000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:05.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbd57fa640 1 -- 172.21.15.150:0/3057686303 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0003b80 con 0x7fcbd8157c00 2024-09-17T05:09:05.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdefab640 1 -- 172.21.15.150:0/3057686303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd80c3ae0 con 0x7fcbd8157c00 2024-09-17T05:09:05.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbd57fa640 1 -- 172.21.15.150:0/3057686303 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbd0003d20 con 0x7fcbd8157c00 2024-09-17T05:09:05.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbd57fa640 1 -- 172.21.15.150:0/3057686303 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0005330 con 0x7fcbd8157c00 2024-09-17T05:09:05.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdefab640 1 -- 172.21.15.150:0/3057686303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd80c36b0 con 0x7fcbd8157c00 2024-09-17T05:09:05.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbdefab640 1 -- 172.21.15.150:0/3057686303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbd80722f0 con 0x7fcbd8157c00 2024-09-17T05:09:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.516+0000 7fcbd57fa640 1 -- 172.21.15.150:0/3057686303 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fcbd00054d0 con 0x7fcbd8157c00 2024-09-17T05:09:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.520+0000 7fcbd57fa640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbbc073200 0x7fcbbc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.520+0000 7fcbd57fa640 1 -- 172.21.15.150:0/3057686303 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7fcbd0095570 con 0x7fcbd8157c00 2024-09-17T05:09:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.520+0000 7fcbdcd47640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbbc073200 0x7fcbbc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.520+0000 7fcbdcd47640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbbc073200 0x7fcbbc075710 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0096c0 tx=0x7fcbcc009260 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:05.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.524+0000 7fcbd57fa640 1 -- 172.21.15.150:0/3057686303 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcbd005f3f0 con 0x7fcbd8157c00 2024-09-17T05:09:05.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.836+0000 7fcbdefab640 1 -- 172.21.15.150:0/3057686303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbd80c3ce0 con 0x7fcbd8157c00 2024-09-17T05:09:05.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.836+0000 7fcbd57fa640 1 -- 172.21.15.150:0/3057686303 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+6540 (secure 0 0 0) 0x7fcbd0063e20 con 0x7fcbd8157c00 2024-09-17T05:09:05.837 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:05.837 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":141,"btime":"2024-09-17T05:08:24:299799+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7158,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":141}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:08:19.286834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7074},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7074":{"gid":7074,"name":"a","rank":0,"incarnation":137,"state":"up:active","state_seq":16,"addr":"172.21.15.150:6833/3622725045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3622725045},{"type":"v1","addr":"172.21.15.150:6833","nonce":3622725045}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7074,"qdb_cluster":[7074]},"id":2}]} 2024-09-17T05:09:05.837 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 141 2024-09-17T05:09:05.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.840+0000 7fcbdefab640 1 -- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbbc073200 msgr2=0x7fcbbc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:05.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.840+0000 7fcbdefab640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbbc073200 0x7fcbbc075710 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0096c0 tx=0x7fcbcc009260 comp rx=0 tx=0).stop 2024-09-17T05:09:05.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.840+0000 7fcbdefab640 1 -- 172.21.15.150:0/3057686303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbd8157c00 msgr2=0x7fcbd8150fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:05.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.840+0000 7fcbdefab640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbd8157c00 0x7fcbd8150fb0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000c6e0 tx=0x7fcbd000cba0 comp rx=0 tx=0).stop 2024-09-17T05:09:05.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.840+0000 7fcbdefab640 1 -- 172.21.15.150:0/3057686303 shutdown_connections 2024-09-17T05:09:05.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.840+0000 7fcbdefab640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbbc073200 0x7fcbbc075710 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:05.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.840+0000 7fcbdefab640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbd8157c00 0x7fcbd8150fb0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:05.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.840+0000 7fcbdefab640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbd814e490 0x7fcbd80c3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:05.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.840+0000 7fcbdefab640 1 --2- 172.21.15.150:0/3057686303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbd8071670 0x7fcbd81eea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:05.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.840+0000 7fcbdefab640 1 -- 172.21.15.150:0/3057686303 >> 172.21.15.150:0/3057686303 conn(0x7fcbd80ca460 msgr2=0x7fcbd815f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:05.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.840+0000 7fcbdefab640 1 -- 172.21.15.150:0/3057686303 shutdown_connections 2024-09-17T05:09:05.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:05.840+0000 7fcbdefab640 1 -- 172.21.15.150:0/3057686303 wait complete. 2024-09-17T05:09:05.860 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:09:07.861 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:08.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.036+0000 7fcbcadcf640 1 -- 172.21.15.150:0/2431783036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc4074c20 msgr2=0x7fcbc40c2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:08.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.036+0000 7fcbcadcf640 1 --2- 172.21.15.150:0/2431783036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc4074c20 0x7fcbc40c2100 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4073350 tx=0x7fcbb401c830 comp rx=0 tx=0).stop 2024-09-17T05:09:08.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.036+0000 7fcbcadcf640 1 -- 172.21.15.150:0/2431783036 shutdown_connections 2024-09-17T05:09:08.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.036+0000 7fcbcadcf640 1 --2- 172.21.15.150:0/2431783036 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbc40a8500 0x7fcbc406ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.036+0000 7fcbcadcf640 1 --2- 172.21.15.150:0/2431783036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc4074c20 0x7fcbc40c2100 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.036+0000 7fcbcadcf640 1 --2- 172.21.15.150:0/2431783036 >> v2:172.21.15.176:3300/0 conn(0x7fcbc4071a10 0x7fcbc4154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.036+0000 7fcbcadcf640 1 -- 172.21.15.150:0/2431783036 >> 172.21.15.150:0/2431783036 conn(0x7fcbc40aaa80 msgr2=0x7fcbc40aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:08.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.036+0000 7fcbcadcf640 1 -- 172.21.15.150:0/2431783036 shutdown_connections 2024-09-17T05:09:08.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.036+0000 7fcbcadcf640 1 -- 172.21.15.150:0/2431783036 wait complete. 2024-09-17T05:09:08.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbcadcf640 1 Processor -- start 2024-09-17T05:09:08.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbcadcf640 1 -- start start 2024-09-17T05:09:08.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbcadcf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc4071a10 0x7fcbc41f0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:08.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbcadcf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbc4074c20 0x7fcbc4146780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:08.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbcadcf640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbc40a8500 0x7fcbc414b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:08.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbcadcf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc406d920 con 0x7fcbc4071a10 2024-09-17T05:09:08.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbcadcf640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc406d7a0 con 0x7fcbc40a8500 2024-09-17T05:09:08.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbcadcf640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc415a680 con 0x7fcbc4074c20 2024-09-17T05:09:08.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbc8b6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc4071a10 0x7fcbc41f0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:08.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbc8b6b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc4071a10 0x7fcbc41f0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60584/0 (socket says 172.21.15.150:60584) 2024-09-17T05:09:08.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbc8b6b640 1 -- 172.21.15.150:0/195670663 learned_addr learned my addr 172.21.15.150:0/195670663 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:08.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbbbfff640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbc4074c20 0x7fcbc4146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:08.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbc936c640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbc40a8500 0x7fcbc414b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:08.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbc8b6b640 1 -- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbc4074c20 msgr2=0x7fcbc4146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:08.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbc8b6b640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbc4074c20 0x7fcbc4146780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbc8b6b640 1 -- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbc40a8500 msgr2=0x7fcbc414b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:08.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbc8b6b640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbc40a8500 0x7fcbc414b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbc8b6b640 1 -- 172.21.15.150:0/195670663 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc41488b0 con 0x7fcbc4071a10 2024-09-17T05:09:08.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbc8b6b640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc4071a10 0x7fcbc41f0340 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00058d0 tx=0x7fcbc000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:08.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbb97fa640 1 -- 172.21.15.150:0/195670663 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0017070 con 0x7fcbc4071a10 2024-09-17T05:09:08.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbcadcf640 1 -- 172.21.15.150:0/195670663 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc4148b90 con 0x7fcbc4071a10 2024-09-17T05:09:08.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbb97fa640 1 -- 172.21.15.150:0/195670663 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbc0009cc0 con 0x7fcbc4071a10 2024-09-17T05:09:08.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbb97fa640 1 -- 172.21.15.150:0/195670663 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00106c0 con 0x7fcbc4071a10 2024-09-17T05:09:08.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.040+0000 7fcbcadcf640 1 -- 172.21.15.150:0/195670663 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc41efb40 con 0x7fcbc4071a10 2024-09-17T05:09:08.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.044+0000 7fcbcadcf640 1 -- 172.21.15.150:0/195670663 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbc41538d0 con 0x7fcbc4071a10 2024-09-17T05:09:08.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.044+0000 7fcbb97fa640 1 -- 172.21.15.150:0/195670663 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fcbc001a070 con 0x7fcbc4071a10 2024-09-17T05:09:08.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.044+0000 7fcbb97fa640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbb0073200 0x7fcbb0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:08.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.044+0000 7fcbb97fa640 1 -- 172.21.15.150:0/195670663 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7fcbc0095150 con 0x7fcbc4071a10 2024-09-17T05:09:08.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.044+0000 7fcbbbfff640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbb0073200 0x7fcbb0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:08.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.044+0000 7fcbbbfff640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbb0073200 0x7fcbb0075710 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40af000 tx=0x7fcbb401b070 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:08.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.052+0000 7fcbb97fa640 1 -- 172.21.15.150:0/195670663 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcbc005f420 con 0x7fcbc4071a10 2024-09-17T05:09:08.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.328+0000 7fcbcadcf640 1 -- 172.21.15.150:0/195670663 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbc415ac10 con 0x7fcbc4071a10 2024-09-17T05:09:08.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.328+0000 7fcbb97fa640 1 -- 172.21.15.150:0/195670663 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5704 (secure 0 0 0) 0x7fcbc005ed20 con 0x7fcbc4071a10 2024-09-17T05:09:08.330 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:08.330 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":144,"btime":"2024-09-17T05:09:08:126352+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:07.130828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:09:08.330 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 144 2024-09-17T05:09:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.332+0000 7fcbcadcf640 1 -- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbb0073200 msgr2=0x7fcbb0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.332+0000 7fcbcadcf640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbb0073200 0x7fcbb0075710 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40af000 tx=0x7fcbb401b070 comp rx=0 tx=0).stop 2024-09-17T05:09:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.332+0000 7fcbcadcf640 1 -- 172.21.15.150:0/195670663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc4071a10 msgr2=0x7fcbc41f0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.332+0000 7fcbcadcf640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc4071a10 0x7fcbc41f0340 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fcbc00058d0 tx=0x7fcbc000dde0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.332+0000 7fcbcadcf640 1 -- 172.21.15.150:0/195670663 shutdown_connections 2024-09-17T05:09:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.332+0000 7fcbcadcf640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcbb0073200 0x7fcbb0075710 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.332+0000 7fcbcadcf640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcbc40a8500 0x7fcbc414b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.332+0000 7fcbcadcf640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcbc4074c20 0x7fcbc4146780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.332+0000 7fcbcadcf640 1 --2- 172.21.15.150:0/195670663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbc4071a10 0x7fcbc41f0340 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.332+0000 7fcbcadcf640 1 -- 172.21.15.150:0/195670663 >> 172.21.15.150:0/195670663 conn(0x7fcbc40aaa80 msgr2=0x7fcbc4160a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.332+0000 7fcbcadcf640 1 -- 172.21.15.150:0/195670663 shutdown_connections 2024-09-17T05:09:08.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.332+0000 7fcbcadcf640 1 -- 172.21.15.150:0/195670663 wait complete. 2024-09-17T05:09:08.351 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-17T05:09:08.352 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:09:08.352 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:08.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.488+0000 7ff301e8d640 1 -- 172.21.15.150:0/832589800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2fc071670 msgr2=0x7ff2fc0c5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:08.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.488+0000 7ff301e8d640 1 --2- 172.21.15.150:0/832589800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2fc071670 0x7ff2fc0c5680 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc0a9a30 tx=0x7ff2ec01c8c0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.488+0000 7ff301e8d640 1 -- 172.21.15.150:0/832589800 shutdown_connections 2024-09-17T05:09:08.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.488+0000 7ff301e8d640 1 --2- 172.21.15.150:0/832589800 >> v2:172.21.15.176:3300/0 conn(0x7ff2fc157c00 0x7ff2fc158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.488+0000 7ff301e8d640 1 --2- 172.21.15.150:0/832589800 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2fc071670 0x7ff2fc0c5680 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.488+0000 7ff301e8d640 1 -- 172.21.15.150:0/832589800 >> 172.21.15.150:0/832589800 conn(0x7ff2fc0ca460 msgr2=0x7ff2fc0c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:08.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.488+0000 7ff301e8d640 1 -- 172.21.15.150:0/832589800 shutdown_connections 2024-09-17T05:09:08.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.488+0000 7ff301e8d640 1 -- 172.21.15.150:0/832589800 wait complete. 2024-09-17T05:09:08.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.488+0000 7ff301e8d640 1 Processor -- start 2024-09-17T05:09:08.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.488+0000 7ff301e8d640 1 -- start start 2024-09-17T05:09:08.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff301e8d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2fc071670 0x7ff2fc14d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:08.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff301e8d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2fc14e490 0x7ff2fc1f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:08.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff301e8d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2fc157c00 0x7ff2fc13e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:08.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff301e8d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc14ea20 con 0x7ff2fc071670 2024-09-17T05:09:08.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff301e8d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc14e8a0 con 0x7ff2fc14e490 2024-09-17T05:09:08.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff301e8d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff2fc072180 con 0x7ff2fc157c00 2024-09-17T05:09:08.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2fb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2fc071670 0x7ff2fc14d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:08.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2fb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2fc071670 0x7ff2fc14d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60604/0 (socket says 172.21.15.150:60604) 2024-09-17T05:09:08.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2fb7fe640 1 -- 172.21.15.150:0/1757677384 learned_addr learned my addr 172.21.15.150:0/1757677384 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:08.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2faffd640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2fc14e490 0x7ff2fc1f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:08.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2fbfff640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2fc157c00 0x7ff2fc13e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:08.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2fb7fe640 1 -- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2fc157c00 msgr2=0x7ff2fc13e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:08.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2fb7fe640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2fc157c00 0x7ff2fc13e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2fb7fe640 1 -- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2fc14e490 msgr2=0x7ff2fc1f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:08.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2fb7fe640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2fc14e490 0x7ff2fc1f3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2fb7fe640 1 -- 172.21.15.150:0/1757677384 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2fc1ef8d0 con 0x7ff2fc071670 2024-09-17T05:09:08.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2fbfff640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2fc157c00 0x7ff2fc13e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:09:08.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2fb7fe640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2fc071670 0x7ff2fc14d460 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec01af70 tx=0x7ff2ec0a56a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:08.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2e3fff640 1 -- 172.21.15.150:0/1757677384 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec0b2040 con 0x7ff2fc071670 2024-09-17T05:09:08.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2e3fff640 1 -- 172.21.15.150:0/1757677384 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff2ec004430 con 0x7ff2fc071670 2024-09-17T05:09:08.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2e3fff640 1 -- 172.21.15.150:0/1757677384 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec003860 con 0x7ff2fc071670 2024-09-17T05:09:08.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff301e8d640 1 -- 172.21.15.150:0/1757677384 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2fc144160 con 0x7ff2fc071670 2024-09-17T05:09:08.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff301e8d640 1 -- 172.21.15.150:0/1757677384 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2fc144720 con 0x7ff2fc071670 2024-09-17T05:09:08.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff301e8d640 1 -- 172.21.15.150:0/1757677384 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2fc144960 con 0x7ff2fc071670 2024-09-17T05:09:08.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.492+0000 7ff2e3fff640 1 -- 172.21.15.150:0/1757677384 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff2ec0bf060 con 0x7ff2fc071670 2024-09-17T05:09:08.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.496+0000 7ff2e3fff640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff2e8073200 0x7ff2e8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:08.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.496+0000 7ff2e3fff640 1 -- 172.21.15.150:0/1757677384 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7ff2ec12eab0 con 0x7ff2fc071670 2024-09-17T05:09:08.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.496+0000 7ff2faffd640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff2e8073200 0x7ff2e8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:08.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.496+0000 7ff2faffd640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff2e8073200 0x7ff2e8075710 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc143c50 tx=0x7ff2f0008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:08.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.500+0000 7ff2e3fff640 1 -- 172.21.15.150:0/1757677384 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff2fc144960 con 0x7ff2fc071670 2024-09-17T05:09:08.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.740+0000 7ff301e8d640 1 -- 172.21.15.150:0/1757677384 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2fc0c5680 con 0x7ff2fc071670 2024-09-17T05:09:08.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.740+0000 7ff2e3fff640 1 -- 172.21.15.150:0/1757677384 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5704 (secure 0 0 0) 0x7ff2ec0164b0 con 0x7ff2fc071670 2024-09-17T05:09:08.741 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:08.741 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":144,"btime":"2024-09-17T05:09:08:126352+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:07.130828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:09:08.741 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 144 2024-09-17T05:09:08.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.744+0000 7ff301e8d640 1 -- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff2e8073200 msgr2=0x7ff2e8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:08.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.744+0000 7ff301e8d640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff2e8073200 0x7ff2e8075710 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc143c50 tx=0x7ff2f0008040 comp rx=0 tx=0).stop 2024-09-17T05:09:08.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.744+0000 7ff301e8d640 1 -- 172.21.15.150:0/1757677384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2fc071670 msgr2=0x7ff2fc14d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:08.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.744+0000 7ff301e8d640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2fc071670 0x7ff2fc14d460 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec01af70 tx=0x7ff2ec0a56a0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.744+0000 7ff301e8d640 1 -- 172.21.15.150:0/1757677384 shutdown_connections 2024-09-17T05:09:08.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.744+0000 7ff301e8d640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff2e8073200 0x7ff2e8075710 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.744+0000 7ff301e8d640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2fc14e490 0x7ff2fc1f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.744+0000 7ff301e8d640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2fc157c00 0x7ff2fc13e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.744+0000 7ff301e8d640 1 --2- 172.21.15.150:0/1757677384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2fc071670 0x7ff2fc14d460 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:08.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.744+0000 7ff301e8d640 1 -- 172.21.15.150:0/1757677384 >> 172.21.15.150:0/1757677384 conn(0x7ff2fc0ca460 msgr2=0x7ff2fc149b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:08.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.744+0000 7ff301e8d640 1 -- 172.21.15.150:0/1757677384 shutdown_connections 2024-09-17T05:09:08.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:08.744+0000 7ff301e8d640 1 -- 172.21.15.150:0/1757677384 wait complete. 2024-09-17T05:09:08.763 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:09:08:126352+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 144, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 135, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5945": { "addr": "172.21.15.176:6837/792092400", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 792092400, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 792092400, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 5945, "incarnation": 132, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:07:24.656365+0000", "qdb_cluster": [ 5945 ], "qdb_leader": 5945, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5945 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 144, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7158": { "addr": "172.21.15.150:6835/285376980", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 285376980, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 285376980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7158, "incarnation": 142, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:09:07.130828+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7158 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:09:09.324 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T05:09:09.324+0000 7f9c548c9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:09:10.765 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:10.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.955+0000 7fa4f59bb640 1 -- 172.21.15.150:0/1388443198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4f00c97a0 msgr2=0x7fa4f0151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:10.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.955+0000 7fa4f59bb640 1 --2- 172.21.15.150:0/1388443198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4f00c97a0 0x7fa4f0151c70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009840 tx=0x7fa4e001c820 comp rx=0 tx=0).stop 2024-09-17T05:09:10.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.955+0000 7fa4f59bb640 1 -- 172.21.15.150:0/1388443198 shutdown_connections 2024-09-17T05:09:10.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.955+0000 7fa4f59bb640 1 --2- 172.21.15.150:0/1388443198 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4f00732b0 0x7fa4f01582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:10.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.955+0000 7fa4f59bb640 1 --2- 172.21.15.150:0/1388443198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4f00c97a0 0x7fa4f0151c70 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:10.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.955+0000 7fa4f59bb640 1 -- 172.21.15.150:0/1388443198 >> 172.21.15.150:0/1388443198 conn(0x7fa4f00abb00 msgr2=0x7fa4f00abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:10.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.955+0000 7fa4f59bb640 1 -- 172.21.15.150:0/1388443198 shutdown_connections 2024-09-17T05:09:10.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.955+0000 7fa4f59bb640 1 -- 172.21.15.150:0/1388443198 wait complete. 2024-09-17T05:09:10.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4f59bb640 1 Processor -- start 2024-09-17T05:09:10.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4f59bb640 1 -- start start 2024-09-17T05:09:10.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4f59bb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4f0072ea0 0x7fa4f0144050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:10.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4f59bb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4f00732b0 0x7fa4f01efb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:10.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4f59bb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f00c97a0 0x7fa4f00c3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:10.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4f59bb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f0152270 con 0x7fa4f00c97a0 2024-09-17T05:09:10.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4f59bb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f00c4df0 con 0x7fa4f00732b0 2024-09-17T05:09:10.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4f59bb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa4f0158830 con 0x7fa4f0072ea0 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4ef7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f00c97a0 0x7fa4f00c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4ef7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f00c97a0 0x7fa4f00c3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43742/0 (socket says 172.21.15.150:43742) 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4ef7fe640 1 -- 172.21.15.150:0/850012266 learned_addr learned my addr 172.21.15.150:0/850012266 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4eeffd640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4f0072ea0 0x7fa4f0144050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4ee7fc640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4f00732b0 0x7fa4f01efb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4ef7fe640 1 -- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4f0072ea0 msgr2=0x7fa4f0144050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4ef7fe640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4f0072ea0 0x7fa4f0144050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4ef7fe640 1 -- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4f00732b0 msgr2=0x7fa4f01efb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4ef7fe640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4f00732b0 0x7fa4f01efb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4ef7fe640 1 -- 172.21.15.150:0/850012266 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f0153600 con 0x7fa4f00c97a0 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4ef7fe640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f00c97a0 0x7fa4f00c3180 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800ec30 tx=0x7fa4e800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4d7fff640 1 -- 172.21.15.150:0/850012266 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8003b80 con 0x7fa4f00c97a0 2024-09-17T05:09:10.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4f59bb640 1 -- 172.21.15.150:0/850012266 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4f0147490 con 0x7fa4f00c97a0 2024-09-17T05:09:10.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4d7fff640 1 -- 172.21.15.150:0/850012266 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4e8003e10 con 0x7fa4f00c97a0 2024-09-17T05:09:10.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4d7fff640 1 -- 172.21.15.150:0/850012266 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e8004280 con 0x7fa4f00c97a0 2024-09-17T05:09:10.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.959+0000 7fa4f59bb640 1 -- 172.21.15.150:0/850012266 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4f0153300 con 0x7fa4f00c97a0 2024-09-17T05:09:10.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.963+0000 7fa4f59bb640 1 -- 172.21.15.150:0/850012266 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4f0151c70 con 0x7fa4f00c97a0 2024-09-17T05:09:10.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.963+0000 7fa4d7fff640 1 -- 172.21.15.150:0/850012266 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa4e8004420 con 0x7fa4f00c97a0 2024-09-17T05:09:10.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.963+0000 7fa4d7fff640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa4dc073230 0x7fa4dc075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:10.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.963+0000 7fa4d7fff640 1 -- 172.21.15.150:0/850012266 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7fa4e8095b70 con 0x7fa4f00c97a0 2024-09-17T05:09:10.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.963+0000 7fa4eeffd640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa4dc073230 0x7fa4dc075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:10.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.963+0000 7fa4eeffd640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa4dc073230 0x7fa4dc075740 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00af000 tx=0x7fa4e001b060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:10.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:10.967+0000 7fa4d7fff640 1 -- 172.21.15.150:0/850012266 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa4f0153600 con 0x7fa4f00c97a0 2024-09-17T05:09:11.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.195+0000 7fa4f59bb640 1 -- 172.21.15.150:0/850012266 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4f006f860 con 0x7fa4f00c97a0 2024-09-17T05:09:11.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.195+0000 7fa4d7fff640 1 -- 172.21.15.150:0/850012266 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+5711 (secure 0 0 0) 0x7fa4e805f910 con 0x7fa4f00c97a0 2024-09-17T05:09:11.197 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:11.197 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":145,"btime":"2024-09-17T05:09:09:137164+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:11.197 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 145 2024-09-17T05:09:11.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.199+0000 7fa4f59bb640 1 -- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa4dc073230 msgr2=0x7fa4dc075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:11.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.199+0000 7fa4f59bb640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa4dc073230 0x7fa4dc075740 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00af000 tx=0x7fa4e001b060 comp rx=0 tx=0).stop 2024-09-17T05:09:11.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.199+0000 7fa4f59bb640 1 -- 172.21.15.150:0/850012266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f00c97a0 msgr2=0x7fa4f00c3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:11.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.199+0000 7fa4f59bb640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f00c97a0 0x7fa4f00c3180 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800ec30 tx=0x7fa4e800c6a0 comp rx=0 tx=0).stop 2024-09-17T05:09:11.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.199+0000 7fa4f59bb640 1 -- 172.21.15.150:0/850012266 shutdown_connections 2024-09-17T05:09:11.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.199+0000 7fa4f59bb640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa4dc073230 0x7fa4dc075740 secure :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00af000 tx=0x7fa4e001b060 comp rx=0 tx=0).stop 2024-09-17T05:09:11.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.199+0000 7fa4f59bb640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa4f00732b0 0x7fa4f01efb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:11.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.199+0000 7fa4f59bb640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4f00c97a0 0x7fa4f00c3180 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:11.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.199+0000 7fa4f59bb640 1 --2- 172.21.15.150:0/850012266 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa4f0072ea0 0x7fa4f0144050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:11.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.199+0000 7fa4f59bb640 1 -- 172.21.15.150:0/850012266 >> 172.21.15.150:0/850012266 conn(0x7fa4f00abb00 msgr2=0x7fa4f01580f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:11.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.203+0000 7fa4f59bb640 1 -- 172.21.15.150:0/850012266 shutdown_connections 2024-09-17T05:09:11.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:11.203+0000 7fa4f59bb640 1 -- 172.21.15.150:0/850012266 wait complete. 2024-09-17T05:09:11.224 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=7074 2024-09-17T05:09:11.225 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.a 2024-09-17T05:09:18.225 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-17T05:09:18.226 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T05:09:18.226 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T05:09:18.229 INFO:tasks.ceph.mds.a:Started 2024-09-17T05:09:18.229 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:18.343 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T05:09:18.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.395+0000 7fe631f67640 1 -- 172.21.15.150:0/2418292697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe618003a10 msgr2=0x7fe618003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:18.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.395+0000 7fe631f67640 1 --2- 172.21.15.150:0/2418292697 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe618003a10 0x7fe618003e80 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009b50 tx=0x7fe61c01d290 comp rx=0 tx=0).stop 2024-09-17T05:09:18.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.395+0000 7fe62bfff640 1 -- 172.21.15.150:0/2418292697 reap_dead start 2024-09-17T05:09:18.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.395+0000 7fe631f67640 1 -- 172.21.15.150:0/2418292697 shutdown_connections 2024-09-17T05:09:18.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.395+0000 7fe631f67640 1 -- 172.21.15.150:0/2418292697 >> 172.21.15.150:0/2418292697 conn(0x7fe62c0c9de0 msgr2=0x7fe62c0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:18.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.395+0000 7fe631f67640 1 -- 172.21.15.150:0/2418292697 shutdown_connections 2024-09-17T05:09:18.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe631f67640 1 -- 172.21.15.150:0/2418292697 wait complete. 2024-09-17T05:09:18.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe631f67640 1 Processor -- start 2024-09-17T05:09:18.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe631f67640 1 -- start start 2024-09-17T05:09:18.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe631f67640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe618003a10 0x7fe62c148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:18.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe631f67640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe62c1f2c90 0x7fe62c0c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:18.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe631f67640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe62c150b40 0x7fe62c150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:18.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe631f67640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c0c7e60 con 0x7fe618003a10 2024-09-17T05:09:18.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe631f67640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c0c7ce0 con 0x7fe62c150b40 2024-09-17T05:09:18.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe631f67640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe62c15f030 con 0x7fe62c1f2c90 2024-09-17T05:09:18.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe62b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe618003a10 0x7fe62c148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:18.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe62b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe618003a10 0x7fe62c148fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43754/0 (socket says 172.21.15.150:43754) 2024-09-17T05:09:18.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe62b7fe640 1 -- 172.21.15.150:0/454761658 learned_addr learned my addr 172.21.15.150:0/454761658 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:18.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe62bfff640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe62c150b40 0x7fe62c150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:18.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.399+0000 7fe623fff640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe62c1f2c90 0x7fe62c0c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:18.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe62b7fe640 1 -- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe62c1f2c90 msgr2=0x7fe62c0c32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:18.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe62b7fe640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe62c1f2c90 0x7fe62c0c32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:18.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe62b7fe640 1 -- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe62c150b40 msgr2=0x7fe62c150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:18.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe62b7fe640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe62c150b40 0x7fe62c150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:18.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe62b7fe640 1 -- 172.21.15.150:0/454761658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62c147d20 con 0x7fe618003a10 2024-09-17T05:09:18.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe62bfff640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe62c150b40 0x7fe62c150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:09:18.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe62b7fe640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe618003a10 0x7fe62c148fb0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009aa0 tx=0x7fe61c004890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:18.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe628ff9640 1 -- 172.21.15.150:0/454761658 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c004d80 con 0x7fe618003a10 2024-09-17T05:09:18.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe628ff9640 1 -- 172.21.15.150:0/454761658 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe61c005030 con 0x7fe618003a10 2024-09-17T05:09:18.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe631f67640 1 -- 172.21.15.150:0/454761658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe62c1476b0 con 0x7fe618003a10 2024-09-17T05:09:18.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe628ff9640 1 -- 172.21.15.150:0/454761658 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe61c005460 con 0x7fe618003a10 2024-09-17T05:09:18.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe631f67640 1 -- 172.21.15.150:0/454761658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe62c14c8e0 con 0x7fe618003a10 2024-09-17T05:09:18.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe631f67640 1 -- 172.21.15.150:0/454761658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe62c0c6230 con 0x7fe618003a10 2024-09-17T05:09:18.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe628ff9640 1 -- 172.21.15.150:0/454761658 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe61c0ad430 con 0x7fe618003a10 2024-09-17T05:09:18.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe628ff9640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe60c073230 0x7fe60c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:18.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe628ff9640 1 -- 172.21.15.150:0/454761658 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7fe61c12fb30 con 0x7fe618003a10 2024-09-17T05:09:18.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.403+0000 7fe623fff640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe60c073230 0x7fe60c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:18.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.407+0000 7fe623fff640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe60c073230 0x7fe60c075740 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe6140058d0 tx=0x7fe614005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:18.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.411+0000 7fe628ff9640 1 -- 172.21.15.150:0/454761658 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe61c016fa0 con 0x7fe618003a10 2024-09-17T05:09:18.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.639+0000 7fe631f67640 1 -- 172.21.15.150:0/454761658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe62c147ef0 con 0x7fe618003a10 2024-09-17T05:09:18.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.639+0000 7fe628ff9640 1 -- 172.21.15.150:0/454761658 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+5711 (secure 0 0 0) 0x7fe61c0f9fa0 con 0x7fe618003a10 2024-09-17T05:09:18.640 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:18.640 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":145,"btime":"2024-09-17T05:09:09:137164+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:18.640 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 145 2024-09-17T05:09:18.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.643+0000 7fe631f67640 1 -- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe60c073230 msgr2=0x7fe60c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:18.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.643+0000 7fe631f67640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe60c073230 0x7fe60c075740 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fe6140058d0 tx=0x7fe614005860 comp rx=0 tx=0).stop 2024-09-17T05:09:18.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.643+0000 7fe631f67640 1 -- 172.21.15.150:0/454761658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe618003a10 msgr2=0x7fe62c148fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:18.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.643+0000 7fe631f67640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe618003a10 0x7fe62c148fb0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009aa0 tx=0x7fe61c004890 comp rx=0 tx=0).stop 2024-09-17T05:09:18.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.643+0000 7fe631f67640 1 -- 172.21.15.150:0/454761658 shutdown_connections 2024-09-17T05:09:18.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.643+0000 7fe631f67640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe60c073230 0x7fe60c075740 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:18.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.643+0000 7fe631f67640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe62c150b40 0x7fe62c150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:18.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.643+0000 7fe631f67640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe62c1f2c90 0x7fe62c0c32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:18.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.643+0000 7fe631f67640 1 --2- 172.21.15.150:0/454761658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe618003a10 0x7fe62c148fb0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:18.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.643+0000 7fe631f67640 1 -- 172.21.15.150:0/454761658 >> 172.21.15.150:0/454761658 conn(0x7fe62c0c9de0 msgr2=0x7fe62c14e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:18.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.643+0000 7fe631f67640 1 -- 172.21.15.150:0/454761658 shutdown_connections 2024-09-17T05:09:18.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:18.643+0000 7fe631f67640 1 -- 172.21.15.150:0/454761658 wait complete. 2024-09-17T05:09:18.664 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-17T05:09:20.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:20.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 -- 172.21.15.150:0/3492211860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51181568a0 msgr2=0x7f51180c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:20.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 --2- 172.21.15.150:0/3492211860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51181568a0 0x7f51180c5810 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f511808d8b0 tx=0x7f511401c800 comp rx=0 tx=0).stop 2024-09-17T05:09:20.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 -- 172.21.15.150:0/3492211860 shutdown_connections 2024-09-17T05:09:20.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 --2- 172.21.15.150:0/3492211860 >> v2:172.21.15.176:3300/0 conn(0x7f51180c35b0 0x7f51180c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:20.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 --2- 172.21.15.150:0/3492211860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51181568a0 0x7f51180c5810 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:20.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 -- 172.21.15.150:0/3492211860 >> 172.21.15.150:0/3492211860 conn(0x7f5118074b80 msgr2=0x7f511806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:20.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 -- 172.21.15.150:0/3492211860 shutdown_connections 2024-09-17T05:09:20.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 -- 172.21.15.150:0/3492211860 wait complete. 2024-09-17T05:09:20.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 Processor -- start 2024-09-17T05:09:20.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 -- start start 2024-09-17T05:09:20.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51180c35b0 0x7f51181f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:20.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51180c8fc0 0x7f511814f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:20.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181568a0 0x7f5118147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:20.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f51180c69f0 con 0x7f51180c35b0 2024-09-17T05:09:20.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5118154540 con 0x7f51181568a0 2024-09-17T05:09:20.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.851+0000 7f511f41a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f51180c6b70 con 0x7f51180c8fc0 2024-09-17T05:09:20.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511d1b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51180c35b0 0x7f51181f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511d1b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51180c35b0 0x7f51181f3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37224/0 (socket says 172.21.15.150:37224) 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511d1b6640 1 -- 172.21.15.150:0/723611218 learned_addr learned my addr 172.21.15.150:0/723611218 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511c9b5640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51180c8fc0 0x7f511814f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511d9b7640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181568a0 0x7f5118147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511d1b6640 1 -- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51180c8fc0 msgr2=0x7f511814f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511d1b6640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51180c8fc0 0x7f511814f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511d1b6640 1 -- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181568a0 msgr2=0x7f5118147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511d1b6640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181568a0 0x7f5118147470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511d1b6640 1 -- 172.21.15.150:0/723611218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51180cceb0 con 0x7f51180c35b0 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511d1b6640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51180c35b0 0x7f51181f3130 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f51140b0040 tx=0x7f5114004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f510dffb640 1 -- 172.21.15.150:0/723611218 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51140b2040 con 0x7f51180c35b0 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f510dffb640 1 -- 172.21.15.150:0/723611218 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5114004320 con 0x7f51180c35b0 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f510dffb640 1 -- 172.21.15.150:0/723611218 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5114003860 con 0x7f51180c35b0 2024-09-17T05:09:20.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511f41a640 1 -- 172.21.15.150:0/723611218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51181f3ce0 con 0x7f51180c35b0 2024-09-17T05:09:20.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511f41a640 1 -- 172.21.15.150:0/723611218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51180ccc90 con 0x7f51180c35b0 2024-09-17T05:09:20.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f50fb7fe640 1 -- 172.21.15.150:0/723611218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50f40052a0 con 0x7f51180c35b0 2024-09-17T05:09:20.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f510dffb640 1 -- 172.21.15.150:0/723611218 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f51140bf020 con 0x7f51180c35b0 2024-09-17T05:09:20.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f510dffb640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f50fc073200 0x7f50fc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:20.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f510dffb640 1 -- 172.21.15.150:0/723611218 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f511412f260 con 0x7f51180c35b0 2024-09-17T05:09:20.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.855+0000 7f511c9b5640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f50fc073200 0x7f50fc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:20.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.859+0000 7f511c9b5640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f50fc073200 0x7f50fc075710 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f51080058d0 tx=0x7f5108005db0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:20.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:20.863+0000 7f510dffb640 1 -- 172.21.15.150:0/723611218 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f51140c6020 con 0x7f51180c35b0 2024-09-17T05:09:21.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.103+0000 7f50fb7fe640 1 -- 172.21.15.150:0/723611218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50f4005860 con 0x7f51180c35b0 2024-09-17T05:09:21.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.103+0000 7f510dffb640 1 -- 172.21.15.150:0/723611218 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+6540 (secure 0 0 0) 0x7f51140164b0 con 0x7f51180c35b0 2024-09-17T05:09:21.106 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:21.106 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":146,"btime":"2024-09-17T05:09:19:171770+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:21.106 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 146 2024-09-17T05:09:21.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.107+0000 7f50fb7fe640 1 -- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f50fc073200 msgr2=0x7f50fc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:21.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.107+0000 7f50fb7fe640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f50fc073200 0x7f50fc075710 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f51080058d0 tx=0x7f5108005db0 comp rx=0 tx=0).stop 2024-09-17T05:09:21.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.107+0000 7f50fb7fe640 1 -- 172.21.15.150:0/723611218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51180c35b0 msgr2=0x7f51181f3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:21.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.107+0000 7f50fb7fe640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51180c35b0 0x7f51181f3130 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f51140b0040 tx=0x7f5114004040 comp rx=0 tx=0).stop 2024-09-17T05:09:21.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.107+0000 7f50fb7fe640 1 -- 172.21.15.150:0/723611218 shutdown_connections 2024-09-17T05:09:21.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.107+0000 7f50fb7fe640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f50fc073200 0x7f50fc075710 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:21.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.107+0000 7f50fb7fe640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51181568a0 0x7f5118147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:21.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.107+0000 7f50fb7fe640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51180c8fc0 0x7f511814f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:21.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.107+0000 7f50fb7fe640 1 --2- 172.21.15.150:0/723611218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51180c35b0 0x7f51181f3130 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:21.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.107+0000 7f50fb7fe640 1 -- 172.21.15.150:0/723611218 >> 172.21.15.150:0/723611218 conn(0x7f5118074b80 msgr2=0x7f5118155080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:21.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.107+0000 7f50fb7fe640 1 -- 172.21.15.150:0/723611218 shutdown_connections 2024-09-17T05:09:21.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:21.107+0000 7f50fb7fe640 1 -- 172.21.15.150:0/723611218 wait complete. 2024-09-17T05:09:21.132 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-17T05:09:21.133 INFO:tasks.mds_thrash.fs.[b]:waiting for 74 secs before thrashing 2024-09-17T05:09:27.298 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:27.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.471+0000 7fa92de2b640 1 -- 172.21.15.150:0/1805284321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa910005360 msgr2=0x7fa91000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:27.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.471+0000 7fa92de2b640 1 --2- 172.21.15.150:0/1805284321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa910005360 0x7fa91000b9d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fa9280c4a80 tx=0x7fa92001caf0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.471+0000 7fa92de2b640 1 -- 172.21.15.150:0/1805284321 shutdown_connections 2024-09-17T05:09:27.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.471+0000 7fa92de2b640 1 --2- 172.21.15.150:0/1805284321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa910005360 0x7fa91000b9d0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.471+0000 7fa92de2b640 1 -- 172.21.15.150:0/1805284321 >> 172.21.15.150:0/1805284321 conn(0x7fa9280c5ae0 msgr2=0x7fa928073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:27.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.471+0000 7fa92de2b640 1 -- 172.21.15.150:0/1805284321 shutdown_connections 2024-09-17T05:09:27.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.471+0000 7fa92de2b640 1 -- 172.21.15.150:0/1805284321 wait complete. 2024-09-17T05:09:27.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa92de2b640 1 Processor -- start 2024-09-17T05:09:27.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa92de2b640 1 -- start start 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa92de2b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa910005360 0x7fa9281de140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa92de2b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9280abf50 0x7fa928148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa92de2b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9280ac360 0x7fa928149180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa92de2b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa9280c3020 con 0x7fa9280abf50 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa92de2b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa928157620 con 0x7fa9280ac360 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa92de2b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa9280c31a0 con 0x7fa910005360 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa926ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9280abf50 0x7fa928148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa926ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9280abf50 0x7fa928148c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37246/0 (socket says 172.21.15.150:37246) 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa926ffd640 1 -- 172.21.15.150:0/528431918 learned_addr learned my addr 172.21.15.150:0/528431918 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa927fff640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9280ac360 0x7fa928149180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa9277fe640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa910005360 0x7fa9281de140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa926ffd640 1 -- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa910005360 msgr2=0x7fa9281de140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:27.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa926ffd640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa910005360 0x7fa9281de140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa926ffd640 1 -- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9280ac360 msgr2=0x7fa928149180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:27.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa926ffd640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9280ac360 0x7fa928149180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa926ffd640 1 -- 172.21.15.150:0/528431918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9281df8d0 con 0x7fa9280abf50 2024-09-17T05:09:27.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa9277fe640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa910005360 0x7fa9281de140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:09:27.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa926ffd640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9280abf50 0x7fa928148c40 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fa91c011850 tx=0x7fa91c011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:27.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa90ffff640 1 -- 172.21.15.150:0/528431918 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91c01f070 con 0x7fa9280abf50 2024-09-17T05:09:27.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa90ffff640 1 -- 172.21.15.150:0/528431918 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa91c00fe40 con 0x7fa9280abf50 2024-09-17T05:09:27.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa92de2b640 1 -- 172.21.15.150:0/528431918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9281496c0 con 0x7fa9280abf50 2024-09-17T05:09:27.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa90ffff640 1 -- 172.21.15.150:0/528431918 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91c0126b0 con 0x7fa9280abf50 2024-09-17T05:09:27.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.475+0000 7fa92de2b640 1 -- 172.21.15.150:0/528431918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa928156d60 con 0x7fa9280abf50 2024-09-17T05:09:27.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.479+0000 7fa92de2b640 1 -- 172.21.15.150:0/528431918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9280706f0 con 0x7fa9280abf50 2024-09-17T05:09:27.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.479+0000 7fa90ffff640 1 -- 172.21.15.150:0/528431918 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa91c022070 con 0x7fa9280abf50 2024-09-17T05:09:27.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.479+0000 7fa90ffff640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa914073230 0x7fa914075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:27.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.479+0000 7fa90ffff640 1 -- 172.21.15.150:0/528431918 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7fa91c0959e0 con 0x7fa9280abf50 2024-09-17T05:09:27.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.479+0000 7fa9277fe640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa914073230 0x7fa914075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:27.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.479+0000 7fa9277fe640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa914073230 0x7fa914075740 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa918001060 tx=0x7fa918000ee0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:27.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.487+0000 7fa90ffff640 1 -- 172.21.15.150:0/528431918 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa91c05f660 con 0x7fa9280abf50 2024-09-17T05:09:27.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.731+0000 7fa92de2b640 1 -- 172.21.15.150:0/528431918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa928149fe0 con 0x7fa9280abf50 2024-09-17T05:09:27.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa90ffff640 1 -- 172.21.15.150:0/528431918 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+6540 (secure 0 0 0) 0x7fa91c064090 con 0x7fa9280abf50 2024-09-17T05:09:27.734 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:27.735 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":146,"btime":"2024-09-17T05:09:19:171770+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:27.735 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 146 2024-09-17T05:09:27.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa92de2b640 1 -- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa914073230 msgr2=0x7fa914075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:27.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa92de2b640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa914073230 0x7fa914075740 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa918001060 tx=0x7fa918000ee0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa92de2b640 1 -- 172.21.15.150:0/528431918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9280abf50 msgr2=0x7fa928148c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:27.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa92de2b640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9280abf50 0x7fa928148c40 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fa91c011850 tx=0x7fa91c011d10 comp rx=0 tx=0).stop 2024-09-17T05:09:27.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa92de2b640 1 -- 172.21.15.150:0/528431918 shutdown_connections 2024-09-17T05:09:27.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa92de2b640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa914073230 0x7fa914075740 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa92de2b640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9280ac360 0x7fa928149180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa92de2b640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9280abf50 0x7fa928148c40 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa92de2b640 1 --2- 172.21.15.150:0/528431918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa910005360 0x7fa9281de140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa92de2b640 1 -- 172.21.15.150:0/528431918 >> 172.21.15.150:0/528431918 conn(0x7fa9280c5ae0 msgr2=0x7fa92814d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:27.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa92de2b640 1 -- 172.21.15.150:0/528431918 shutdown_connections 2024-09-17T05:09:27.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.735+0000 7fa92de2b640 1 -- 172.21.15.150:0/528431918 wait complete. 2024-09-17T05:09:27.757 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-09-17T05:09:27.757 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T05:09:27.757 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:09:27.774 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T05:09:27.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:27.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.943+0000 7f6db8115640 1 -- 172.21.15.150:0/267490949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db00ae320 msgr2=0x7f6db0157e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:27.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.943+0000 7f6db8115640 1 --2- 172.21.15.150:0/267490949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db00ae320 0x7f6db0157e00 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6db0073f60 tx=0x7f6da801ccf0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.943+0000 7f6db8115640 1 -- 172.21.15.150:0/267490949 shutdown_connections 2024-09-17T05:09:27.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.943+0000 7f6db8115640 1 --2- 172.21.15.150:0/267490949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db00ae320 0x7f6db0157e00 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.943+0000 7f6db8115640 1 --2- 172.21.15.150:0/267490949 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6db00c5d80 0x7f6db00adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.943+0000 7f6db8115640 1 --2- 172.21.15.150:0/267490949 >> v2:172.21.15.176:3300/0 conn(0x7f6db00c6ea0 0x7f6db00c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.943+0000 7f6db8115640 1 -- 172.21.15.150:0/267490949 >> 172.21.15.150:0/267490949 conn(0x7f6db00c8db0 msgr2=0x7f6db00c91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:27.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.943+0000 7f6db8115640 1 -- 172.21.15.150:0/267490949 shutdown_connections 2024-09-17T05:09:27.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.943+0000 7f6db8115640 1 -- 172.21.15.150:0/267490949 wait complete. 2024-09-17T05:09:27.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.943+0000 7f6db8115640 1 Processor -- start 2024-09-17T05:09:27.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.943+0000 7f6db8115640 1 -- start start 2024-09-17T05:09:27.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db8115640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6db00ae320 0x7f6db015d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:27.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db8115640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6db00c5d80 0x7f6db0070c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:27.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db8115640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db00c6ea0 0x7f6db01efc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:27.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db8115640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6db00abfe0 con 0x7f6db00c6ea0 2024-09-17T05:09:27.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db8115640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6db00abe60 con 0x7f6db00c5d80 2024-09-17T05:09:27.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db8115640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6db006d660 con 0x7f6db00ae320 2024-09-17T05:09:27.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db66b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db00c6ea0 0x7f6db01efc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:27.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db66b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db00c6ea0 0x7f6db01efc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37264/0 (socket says 172.21.15.150:37264) 2024-09-17T05:09:27.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db66b2640 1 -- 172.21.15.150:0/1890925952 learned_addr learned my addr 172.21.15.150:0/1890925952 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:27.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db56b0640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6db00c5d80 0x7f6db0070c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:27.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db5eb1640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6db00ae320 0x7f6db015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:27.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db66b2640 1 -- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6db00ae320 msgr2=0x7f6db015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:27.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db66b2640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6db00ae320 0x7f6db015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db66b2640 1 -- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6db00c5d80 msgr2=0x7f6db0070c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:27.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db66b2640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6db00c5d80 0x7f6db0070c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:27.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db66b2640 1 -- 172.21.15.150:0/1890925952 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db0071ad0 con 0x7f6db00c6ea0 2024-09-17T05:09:27.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db66b2640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db00c6ea0 0x7f6db01efc70 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f6da8009bd0 tx=0x7f6da8005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:27.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6da67fc640 1 -- 172.21.15.150:0/1890925952 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da80bb070 con 0x7f6db00c6ea0 2024-09-17T05:09:27.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6da67fc640 1 -- 172.21.15.150:0/1890925952 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6da8004d40 con 0x7f6db00c6ea0 2024-09-17T05:09:27.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db8115640 1 -- 172.21.15.150:0/1890925952 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6db0071db0 con 0x7f6db00c6ea0 2024-09-17T05:09:27.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6da67fc640 1 -- 172.21.15.150:0/1890925952 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da80b2040 con 0x7f6db00c6ea0 2024-09-17T05:09:27.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db8115640 1 -- 172.21.15.150:0/1890925952 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6db0070460 con 0x7f6db00c6ea0 2024-09-17T05:09:27.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.947+0000 7f6db8115640 1 -- 172.21.15.150:0/1890925952 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6db006da80 con 0x7f6db00c6ea0 2024-09-17T05:09:27.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.951+0000 7f6da67fc640 1 -- 172.21.15.150:0/1890925952 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6da80be070 con 0x7f6db00c6ea0 2024-09-17T05:09:27.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.951+0000 7f6da67fc640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6d94073200 0x7f6d94075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:27.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.951+0000 7f6da67fc640 1 -- 172.21.15.150:0/1890925952 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f6da812ed90 con 0x7f6db00c6ea0 2024-09-17T05:09:27.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.951+0000 7f6db5eb1640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6d94073200 0x7f6d94075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:27.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.951+0000 7f6db5eb1640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6d94073200 0x7f6d94075710 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6dac00bad0 tx=0x7f6dac00b2e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:27.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:27.955+0000 7f6da67fc640 1 -- 172.21.15.150:0/1890925952 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6da80a5e20 con 0x7f6db00c6ea0 2024-09-17T05:09:28.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.195+0000 7f6db8115640 1 -- 172.21.15.150:0/1890925952 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6db006f970 con 0x7f6db00c6ea0 2024-09-17T05:09:28.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.195+0000 7f6da67fc640 1 -- 172.21.15.150:0/1890925952 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+6540 (secure 0 0 0) 0x7f6da8016970 con 0x7f6db00c6ea0 2024-09-17T05:09:28.198 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:28.198 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":146,"btime":"2024-09-17T05:09:19:171770+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:28.198 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 146 2024-09-17T05:09:28.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.199+0000 7f6db8115640 1 -- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6d94073200 msgr2=0x7f6d94075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:28.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.199+0000 7f6db8115640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6d94073200 0x7f6d94075710 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6dac00bad0 tx=0x7f6dac00b2e0 comp rx=0 tx=0).stop 2024-09-17T05:09:28.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.199+0000 7f6db8115640 1 -- 172.21.15.150:0/1890925952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db00c6ea0 msgr2=0x7f6db01efc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:28.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.199+0000 7f6db8115640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db00c6ea0 0x7f6db01efc70 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f6da8009bd0 tx=0x7f6da8005b00 comp rx=0 tx=0).stop 2024-09-17T05:09:28.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.199+0000 7f6db8115640 1 -- 172.21.15.150:0/1890925952 shutdown_connections 2024-09-17T05:09:28.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.199+0000 7f6db8115640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6d94073200 0x7f6d94075710 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:28.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.199+0000 7f6db8115640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6db00c5d80 0x7f6db0070c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:28.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.199+0000 7f6db8115640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6db00c6ea0 0x7f6db01efc70 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:28.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.199+0000 7f6db8115640 1 --2- 172.21.15.150:0/1890925952 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6db00ae320 0x7f6db015d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:28.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.199+0000 7f6db8115640 1 -- 172.21.15.150:0/1890925952 >> 172.21.15.150:0/1890925952 conn(0x7f6db00c8db0 msgr2=0x7f6db015e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:28.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.199+0000 7f6db8115640 1 -- 172.21.15.150:0/1890925952 shutdown_connections 2024-09-17T05:09:28.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:28.199+0000 7f6db8115640 1 -- 172.21.15.150:0/1890925952 wait complete. 2024-09-17T05:09:28.220 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:09:30.221 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:30.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.387+0000 7fa2b529b640 1 -- 172.21.15.150:0/1147288371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b00a7f10 msgr2=0x7fa2b00a8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:30.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.387+0000 7fa2b529b640 1 --2- 172.21.15.150:0/1147288371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b00a7f10 0x7fa2b00a8320 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8009c80 tx=0x7fa2a801cc80 comp rx=0 tx=0).stop 2024-09-17T05:09:30.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.387+0000 7fa2b529b640 1 -- 172.21.15.150:0/1147288371 shutdown_connections 2024-09-17T05:09:30.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.387+0000 7fa2b529b640 1 --2- 172.21.15.150:0/1147288371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b00a7f10 0x7fa2b00a8320 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:30.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.387+0000 7fa2b529b640 1 --2- 172.21.15.150:0/1147288371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2b00c5760 0x7fa2b00c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:30.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.387+0000 7fa2b529b640 1 -- 172.21.15.150:0/1147288371 >> 172.21.15.150:0/1147288371 conn(0x7fa2b00aaa80 msgr2=0x7fa2b00aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:30.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.387+0000 7fa2b529b640 1 -- 172.21.15.150:0/1147288371 shutdown_connections 2024-09-17T05:09:30.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.387+0000 7fa2b529b640 1 -- 172.21.15.150:0/1147288371 wait complete. 2024-09-17T05:09:30.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.387+0000 7fa2b529b640 1 Processor -- start 2024-09-17T05:09:30.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.387+0000 7fa2b529b640 1 -- start start 2024-09-17T05:09:30.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2b529b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2b00a7f10 0x7fa2b0151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:30.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2b529b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b00c5760 0x7fa2b01f0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:30.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2b529b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2b01535b0 0x7fa2b0141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:30.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2b529b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b00c54f0 con 0x7fa2b00c5760 2024-09-17T05:09:30.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2b529b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b00c5370 con 0x7fa2b01535b0 2024-09-17T05:09:30.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2b529b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b01539c0 con 0x7fa2b00a7f10 2024-09-17T05:09:30.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a67fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b00c5760 0x7fa2b01f0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:30.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a67fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b00c5760 0x7fa2b01f0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32878/0 (socket says 172.21.15.150:32878) 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a67fc640 1 -- 172.21.15.150:0/4050132650 learned_addr learned my addr 172.21.15.150:0/4050132650 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2af7fe640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2b01535b0 0x7fa2b0141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2aeffd640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2b00a7f10 0x7fa2b0151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a67fc640 1 -- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2b00a7f10 msgr2=0x7fa2b0151a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a67fc640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2b00a7f10 0x7fa2b0151a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a67fc640 1 -- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2b01535b0 msgr2=0x7fa2b0141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a67fc640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2b01535b0 0x7fa2b0141080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a67fc640 1 -- 172.21.15.150:0/4050132650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b01f0b30 con 0x7fa2b00c5760 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2af7fe640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2b01535b0 0x7fa2b0141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a67fc640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b00c5760 0x7fa2b01f0250 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fa298009860 tx=0x7fa298009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a7fff640 1 -- 172.21.15.150:0/4050132650 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29801d070 con 0x7fa2b00c5760 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a7fff640 1 -- 172.21.15.150:0/4050132650 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa298007740 con 0x7fa2b00c5760 2024-09-17T05:09:30.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a7fff640 1 -- 172.21.15.150:0/4050132650 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa298007b70 con 0x7fa2b00c5760 2024-09-17T05:09:30.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2b529b640 1 -- 172.21.15.150:0/4050132650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b014cb10 con 0x7fa2b00c5760 2024-09-17T05:09:30.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2b529b640 1 -- 172.21.15.150:0/4050132650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b0147d00 con 0x7fa2b00c5760 2024-09-17T05:09:30.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2b529b640 1 -- 172.21.15.150:0/4050132650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2b00c6c50 con 0x7fa2b00c5760 2024-09-17T05:09:30.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a7fff640 1 -- 172.21.15.150:0/4050132650 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa298020070 con 0x7fa2b00c5760 2024-09-17T05:09:30.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a7fff640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa290073230 0x7fa290075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:30.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.391+0000 7fa2a7fff640 1 -- 172.21.15.150:0/4050132650 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7fa298096550 con 0x7fa2b00c5760 2024-09-17T05:09:30.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.395+0000 7fa2aeffd640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa290073230 0x7fa290075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:30.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.395+0000 7fa2aeffd640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa290073230 0x7fa290075740 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0007fd0 tx=0x7fa2a0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:30.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.399+0000 7fa2a7fff640 1 -- 172.21.15.150:0/4050132650 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa298060370 con 0x7fa2b00c5760 2024-09-17T05:09:30.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.611+0000 7fa2b529b640 1 -- 172.21.15.150:0/4050132650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2b00c3070 con 0x7fa2b00c5760 2024-09-17T05:09:30.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.611+0000 7fa2a7fff640 1 -- 172.21.15.150:0/4050132650 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+6540 (secure 0 0 0) 0x7fa298064da0 con 0x7fa2b00c5760 2024-09-17T05:09:30.614 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:30.614 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":146,"btime":"2024-09-17T05:09:19:171770+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:30.614 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 146 2024-09-17T05:09:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.615+0000 7fa2b529b640 1 -- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa290073230 msgr2=0x7fa290075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.615+0000 7fa2b529b640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa290073230 0x7fa290075740 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0007fd0 tx=0x7fa2a0009040 comp rx=0 tx=0).stop 2024-09-17T05:09:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.615+0000 7fa2b529b640 1 -- 172.21.15.150:0/4050132650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b00c5760 msgr2=0x7fa2b01f0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.615+0000 7fa2b529b640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b00c5760 0x7fa2b01f0250 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fa298009860 tx=0x7fa298009d20 comp rx=0 tx=0).stop 2024-09-17T05:09:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.615+0000 7fa2b529b640 1 -- 172.21.15.150:0/4050132650 shutdown_connections 2024-09-17T05:09:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.615+0000 7fa2b529b640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa290073230 0x7fa290075740 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.615+0000 7fa2b529b640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2b01535b0 0x7fa2b0141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.615+0000 7fa2b529b640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2b00c5760 0x7fa2b01f0250 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.615+0000 7fa2b529b640 1 --2- 172.21.15.150:0/4050132650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa2b00a7f10 0x7fa2b0151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.615+0000 7fa2b529b640 1 -- 172.21.15.150:0/4050132650 >> 172.21.15.150:0/4050132650 conn(0x7fa2b00aaa80 msgr2=0x7fa2b015e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.615+0000 7fa2b529b640 1 -- 172.21.15.150:0/4050132650 shutdown_connections 2024-09-17T05:09:30.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:30.615+0000 7fa2b529b640 1 -- 172.21.15.150:0/4050132650 wait complete. 2024-09-17T05:09:30.636 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:09:32.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.803+0000 7f3672130640 1 -- 172.21.15.150:0/2306582712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f366c0a7350 msgr2=0x7f366c150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.803+0000 7f3672130640 1 --2- 172.21.15.150:0/2306582712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f366c0a7350 0x7f366c150060 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f3654009900 tx=0x7f365401c8e0 comp rx=0 tx=0).stop 2024-09-17T05:09:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.803+0000 7f3672130640 1 -- 172.21.15.150:0/2306582712 shutdown_connections 2024-09-17T05:09:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.803+0000 7f3672130640 1 --2- 172.21.15.150:0/2306582712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f366c0a7350 0x7f366c150060 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.803+0000 7f3672130640 1 --2- 172.21.15.150:0/2306582712 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f366c0c42e0 0x7f366c0c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.803+0000 7f3672130640 1 -- 172.21.15.150:0/2306582712 >> 172.21.15.150:0/2306582712 conn(0x7f366c0c4a00 msgr2=0x7f366c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.803+0000 7f3672130640 1 -- 172.21.15.150:0/2306582712 shutdown_connections 2024-09-17T05:09:32.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.803+0000 7f3672130640 1 -- 172.21.15.150:0/2306582712 wait complete. 2024-09-17T05:09:32.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.803+0000 7f3672130640 1 Processor -- start 2024-09-17T05:09:32.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.803+0000 7f3672130640 1 -- start start 2024-09-17T05:09:32.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3672130640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f366c0a7350 0x7f366c1f0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:32.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3672130640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f366c0c42e0 0x7f366c14dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:32.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3672130640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f366c1505a0 0x7f366c155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:32.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3672130640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f366c0c6bd0 con 0x7f366c1505a0 2024-09-17T05:09:32.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3672130640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f366c070b40 con 0x7f366c0a7350 2024-09-17T05:09:32.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3672130640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f366c0c6d50 con 0x7f366c0c42e0 2024-09-17T05:09:32.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f366bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f366c1505a0 0x7f366c155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:32.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f366bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f366c1505a0 0x7f366c155730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32896/0 (socket says 172.21.15.150:32896) 2024-09-17T05:09:32.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f366bfff640 1 -- 172.21.15.150:0/4254200074 learned_addr learned my addr 172.21.15.150:0/4254200074 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:32.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3662ffd640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f366c0c42e0 0x7f366c14dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:32.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f366b7fe640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f366c0a7350 0x7f366c1f0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:32.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f366bfff640 1 -- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f366c0c42e0 msgr2=0x7f366c14dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:32.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f366bfff640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f366c0c42e0 0x7f366c14dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:32.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f366bfff640 1 -- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f366c0a7350 msgr2=0x7f366c1f0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:32.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f366bfff640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f366c0a7350 0x7f366c1f0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:32.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f366bfff640 1 -- 172.21.15.150:0/4254200074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f366c1531b0 con 0x7f366c1505a0 2024-09-17T05:09:32.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f366b7fe640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f366c0a7350 0x7f366c1f0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:09:32.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f366bfff640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f366c1505a0 0x7f366c155730 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f366400c760 tx=0x7f366400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:32.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3668ff9640 1 -- 172.21.15.150:0/4254200074 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3664003b80 con 0x7f366c1505a0 2024-09-17T05:09:32.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3668ff9640 1 -- 172.21.15.150:0/4254200074 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3664003d20 con 0x7f366c1505a0 2024-09-17T05:09:32.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3668ff9640 1 -- 172.21.15.150:0/4254200074 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366400aab0 con 0x7f366c1505a0 2024-09-17T05:09:32.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3672130640 1 -- 172.21.15.150:0/4254200074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f366c144fe0 con 0x7f366c1505a0 2024-09-17T05:09:32.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3672130640 1 -- 172.21.15.150:0/4254200074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f366c154820 con 0x7f366c1505a0 2024-09-17T05:09:32.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3672130640 1 -- 172.21.15.150:0/4254200074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f366c150060 con 0x7f366c1505a0 2024-09-17T05:09:32.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3668ff9640 1 -- 172.21.15.150:0/4254200074 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f366400ac50 con 0x7f366c1505a0 2024-09-17T05:09:32.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.807+0000 7f3668ff9640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f364c073230 0x7f364c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:32.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.811+0000 7f3668ff9640 1 -- 172.21.15.150:0/4254200074 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f3664094b20 con 0x7f366c1505a0 2024-09-17T05:09:32.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.811+0000 7f366b7fe640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f364c073230 0x7f364c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:32.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.811+0000 7f366b7fe640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f364c073230 0x7f364c075740 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f365c00bc70 tx=0x7f365c00b650 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:32.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:32.815+0000 7f3668ff9640 1 -- 172.21.15.150:0/4254200074 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f366405ef90 con 0x7f366c1505a0 2024-09-17T05:09:33.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.019+0000 7f3672130640 1 -- 172.21.15.150:0/4254200074 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f366c1542f0 con 0x7f366c1505a0 2024-09-17T05:09:33.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.019+0000 7f3668ff9640 1 -- 172.21.15.150:0/4254200074 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+6540 (secure 0 0 0) 0x7f366405e890 con 0x7f366c1505a0 2024-09-17T05:09:33.020 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:33.020 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":146,"btime":"2024-09-17T05:09:19:171770+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:33.020 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 146 2024-09-17T05:09:33.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.019+0000 7f3672130640 1 -- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f364c073230 msgr2=0x7f364c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:33.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.019+0000 7f3672130640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f364c073230 0x7f364c075740 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f365c00bc70 tx=0x7f365c00b650 comp rx=0 tx=0).stop 2024-09-17T05:09:33.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.019+0000 7f3672130640 1 -- 172.21.15.150:0/4254200074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f366c1505a0 msgr2=0x7f366c155730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:33.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.019+0000 7f3672130640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f366c1505a0 0x7f366c155730 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f366400c760 tx=0x7f366400cc20 comp rx=0 tx=0).stop 2024-09-17T05:09:33.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.019+0000 7f3672130640 1 -- 172.21.15.150:0/4254200074 shutdown_connections 2024-09-17T05:09:33.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.019+0000 7f3672130640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f364c073230 0x7f364c075740 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:33.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.019+0000 7f3672130640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f366c1505a0 0x7f366c155730 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:33.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.019+0000 7f3672130640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f366c0c42e0 0x7f366c14dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:33.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.019+0000 7f3672130640 1 --2- 172.21.15.150:0/4254200074 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f366c0a7350 0x7f366c1f0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:33.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.019+0000 7f3672130640 1 -- 172.21.15.150:0/4254200074 >> 172.21.15.150:0/4254200074 conn(0x7f366c0c4a00 msgr2=0x7f366c073440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:33.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.023+0000 7f3672130640 1 -- 172.21.15.150:0/4254200074 shutdown_connections 2024-09-17T05:09:33.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:33.023+0000 7f3672130640 1 -- 172.21.15.150:0/4254200074 wait complete. 2024-09-17T05:09:33.042 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:09:33.361 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:09:33.364 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:09:35.043 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:35.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.211+0000 7fbfd4f99640 1 -- 172.21.15.150:0/1872938481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfbc004810 msgr2=0x7fbfbc004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:35.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.211+0000 7fbfd4f99640 1 --2- 172.21.15.150:0/1872938481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfbc004810 0x7fbfbc004c20 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800c310 tx=0x7fbfb801cfc0 comp rx=0 tx=0).stop 2024-09-17T05:09:35.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.211+0000 7fbfced9d640 1 -- 172.21.15.150:0/1872938481 reap_dead start 2024-09-17T05:09:35.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.211+0000 7fbfd4f99640 1 -- 172.21.15.150:0/1872938481 shutdown_connections 2024-09-17T05:09:35.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.211+0000 7fbfd4f99640 1 -- 172.21.15.150:0/1872938481 >> 172.21.15.150:0/1872938481 conn(0x7fbfd00aaf60 msgr2=0x7fbfd00ab370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:35.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.211+0000 7fbfd4f99640 1 -- 172.21.15.150:0/1872938481 shutdown_connections 2024-09-17T05:09:35.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.211+0000 7fbfd4f99640 1 -- 172.21.15.150:0/1872938481 wait complete. 2024-09-17T05:09:35.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfd4f99640 1 Processor -- start 2024-09-17T05:09:35.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfd4f99640 1 -- start start 2024-09-17T05:09:35.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfd4f99640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfbc004810 0x7fbfd00c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:35.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfd4f99640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfd0145560 0x7fbfd0145970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:35.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfd4f99640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd01f3300 0x7fbfd01f3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:35.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfd4f99640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd00a8200 con 0x7fbfd0145560 2024-09-17T05:09:35.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfd4f99640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd01542a0 con 0x7fbfd01f3300 2024-09-17T05:09:35.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfd4f99640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd0072d60 con 0x7fbfbc004810 2024-09-17T05:09:35.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfc7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfd0145560 0x7fbfd0145970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:35.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfc7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfd0145560 0x7fbfd0145970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32918/0 (socket says 172.21.15.150:32918) 2024-09-17T05:09:35.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfc7fff640 1 -- 172.21.15.150:0/362304473 learned_addr learned my addr 172.21.15.150:0/362304473 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:35.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfce59c640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfbc004810 0x7fbfd00c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:35.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfced9d640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd01f3300 0x7fbfd01f3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:35.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfc7fff640 1 -- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfbc004810 msgr2=0x7fbfd00c23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:35.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfc7fff640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfbc004810 0x7fbfd00c23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:35.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfc7fff640 1 -- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd01f3300 msgr2=0x7fbfd01f3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:35.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfc7fff640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd01f3300 0x7fbfd01f3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:35.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfc7fff640 1 -- 172.21.15.150:0/362304473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd01f02e0 con 0x7fbfd0145560 2024-09-17T05:09:35.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfced9d640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd01f3300 0x7fbfd01f3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:09:35.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfc7fff640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfd0145560 0x7fbfd0145970 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800c310 tx=0x7fbfb8006c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:35.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfc6ffd640 1 -- 172.21.15.150:0/362304473 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb80060c0 con 0x7fbfd0145560 2024-09-17T05:09:35.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfc6ffd640 1 -- 172.21.15.150:0/362304473 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbfb8003600 con 0x7fbfd0145560 2024-09-17T05:09:35.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfd4f99640 1 -- 172.21.15.150:0/362304473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd014c750 con 0x7fbfd0145560 2024-09-17T05:09:35.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfc6ffd640 1 -- 172.21.15.150:0/362304473 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8003a50 con 0x7fbfd0145560 2024-09-17T05:09:35.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.215+0000 7fbfd4f99640 1 -- 172.21.15.150:0/362304473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd0146140 con 0x7fbfd0145560 2024-09-17T05:09:35.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.219+0000 7fbfd4f99640 1 -- 172.21.15.150:0/362304473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd014d530 con 0x7fbfd0145560 2024-09-17T05:09:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.219+0000 7fbfc6ffd640 1 -- 172.21.15.150:0/362304473 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbfb8003bf0 con 0x7fbfd0145560 2024-09-17T05:09:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.219+0000 7fbfc6ffd640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbfb0073200 0x7fbfb0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.219+0000 7fbfc6ffd640 1 -- 172.21.15.150:0/362304473 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7fbfb812f7c0 con 0x7fbfd0145560 2024-09-17T05:09:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.219+0000 7fbfce59c640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbfb0073200 0x7fbfb0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:35.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.219+0000 7fbfce59c640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbfb0073200 0x7fbfb0075710 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009d50 tx=0x7fbfc00091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:35.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.223+0000 7fbfc6ffd640 1 -- 172.21.15.150:0/362304473 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbfb8016d70 con 0x7fbfd0145560 2024-09-17T05:09:35.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.431+0000 7fbfd4f99640 1 -- 172.21.15.150:0/362304473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfd0154cf0 con 0x7fbfd0145560 2024-09-17T05:09:35.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.431+0000 7fbfc6ffd640 1 -- 172.21.15.150:0/362304473 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+6540 (secure 0 0 0) 0x7fbfb8006260 con 0x7fbfd0145560 2024-09-17T05:09:35.434 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:35.434 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":146,"btime":"2024-09-17T05:09:19:171770+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:35.435 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 146 2024-09-17T05:09:35.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.435+0000 7fbfd4f99640 1 -- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbfb0073200 msgr2=0x7fbfb0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:35.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.435+0000 7fbfd4f99640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbfb0073200 0x7fbfb0075710 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009d50 tx=0x7fbfc00091e0 comp rx=0 tx=0).stop 2024-09-17T05:09:35.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.435+0000 7fbfd4f99640 1 -- 172.21.15.150:0/362304473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfd0145560 msgr2=0x7fbfd0145970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:35.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.435+0000 7fbfd4f99640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfd0145560 0x7fbfd0145970 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fbfb800c310 tx=0x7fbfb8006c60 comp rx=0 tx=0).stop 2024-09-17T05:09:35.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.435+0000 7fbfd4f99640 1 -- 172.21.15.150:0/362304473 shutdown_connections 2024-09-17T05:09:35.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.435+0000 7fbfd4f99640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbfb0073200 0x7fbfb0075710 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:35.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.435+0000 7fbfd4f99640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd01f3300 0x7fbfd01f3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:35.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.435+0000 7fbfd4f99640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfd0145560 0x7fbfd0145970 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:35.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.435+0000 7fbfd4f99640 1 --2- 172.21.15.150:0/362304473 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbfbc004810 0x7fbfd00c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:35.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.435+0000 7fbfd4f99640 1 -- 172.21.15.150:0/362304473 >> 172.21.15.150:0/362304473 conn(0x7fbfd00aaf60 msgr2=0x7fbfd00c5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:35.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.435+0000 7fbfd4f99640 1 -- 172.21.15.150:0/362304473 shutdown_connections 2024-09-17T05:09:35.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:35.435+0000 7fbfd4f99640 1 -- 172.21.15.150:0/362304473 wait complete. 2024-09-17T05:09:35.456 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:09:37.457 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:37.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.623+0000 7f94f7def640 1 -- 172.21.15.150:0/2255311671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f00c7580 msgr2=0x7f94f00c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:37.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.623+0000 7f94f7def640 1 --2- 172.21.15.150:0/2255311671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f00c7580 0x7f94f00c7e20 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f94f0073350 tx=0x7f94e001c830 comp rx=0 tx=0).stop 2024-09-17T05:09:37.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.623+0000 7f94f7def640 1 -- 172.21.15.150:0/2255311671 shutdown_connections 2024-09-17T05:09:37.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.623+0000 7f94f7def640 1 --2- 172.21.15.150:0/2255311671 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f00c7580 0x7f94f00c7e20 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:37.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.623+0000 7f94f7def640 1 --2- 172.21.15.150:0/2255311671 >> v2:172.21.15.176:3300/0 conn(0x7f94f01557b0 0x7f94f00c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:37.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.623+0000 7f94f7def640 1 -- 172.21.15.150:0/2255311671 >> 172.21.15.150:0/2255311671 conn(0x7f94f00aaa80 msgr2=0x7f94f00aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:37.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.623+0000 7f94f7def640 1 -- 172.21.15.150:0/2255311671 shutdown_connections 2024-09-17T05:09:37.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.623+0000 7f94f7def640 1 -- 172.21.15.150:0/2255311671 wait complete. 2024-09-17T05:09:37.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f7def640 1 Processor -- start 2024-09-17T05:09:37.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f7def640 1 -- start start 2024-09-17T05:09:37.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f7def640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94f0074420 0x7f94f01488d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:37.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f7def640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f00c7580 0x7f94f01f10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:37.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f7def640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94f01557b0 0x7f94f0147300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:37.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f7def640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f94f00723a0 con 0x7f94f00c7580 2024-09-17T05:09:37.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f7def640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f94f0072220 con 0x7f94f01557b0 2024-09-17T05:09:37.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f7def640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f94f0074830 con 0x7f94f0074420 2024-09-17T05:09:37.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f538a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f00c7580 0x7f94f01f10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:37.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f538a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f00c7580 0x7f94f01f10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32930/0 (socket says 172.21.15.150:32930) 2024-09-17T05:09:37.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f5b8b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94f0074420 0x7f94f01488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:37.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f538a640 1 -- 172.21.15.150:0/654903351 learned_addr learned my addr 172.21.15.150:0/654903351 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:37.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f638c640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94f01557b0 0x7f94f0147300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:37.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f538a640 1 -- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94f0074420 msgr2=0x7f94f01488d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:37.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f538a640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94f0074420 0x7f94f01488d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:37.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f538a640 1 -- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94f01557b0 msgr2=0x7f94f0147300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:37.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f538a640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94f01557b0 0x7f94f0147300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:37.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f538a640 1 -- 172.21.15.150:0/654903351 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f0158990 con 0x7f94f00c7580 2024-09-17T05:09:37.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f538a640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f00c7580 0x7f94f01f10b0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f94e001aee0 tx=0x7f94e0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:37.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94e67fc640 1 -- 172.21.15.150:0/654903351 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e0004230 con 0x7f94f00c7580 2024-09-17T05:09:37.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94e67fc640 1 -- 172.21.15.150:0/654903351 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94e00044c0 con 0x7f94f00c7580 2024-09-17T05:09:37.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f7def640 1 -- 172.21.15.150:0/654903351 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94f01f1a90 con 0x7f94f00c7580 2024-09-17T05:09:37.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94e67fc640 1 -- 172.21.15.150:0/654903351 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94e0003860 con 0x7f94f00c7580 2024-09-17T05:09:37.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f7def640 1 -- 172.21.15.150:0/654903351 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94f0148e10 con 0x7f94f00c7580 2024-09-17T05:09:37.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.627+0000 7f94f7def640 1 -- 172.21.15.150:0/654903351 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94f00c7e20 con 0x7f94f00c7580 2024-09-17T05:09:37.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.631+0000 7f94e67fc640 1 -- 172.21.15.150:0/654903351 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f94e00bf020 con 0x7f94f00c7580 2024-09-17T05:09:37.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.631+0000 7f94e67fc640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f94d4073200 0x7f94d4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:37.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.631+0000 7f94e67fc640 1 -- 172.21.15.150:0/654903351 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f94e012f390 con 0x7f94f00c7580 2024-09-17T05:09:37.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.631+0000 7f94f5b8b640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f94d4073200 0x7f94d4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:37.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.631+0000 7f94f5b8b640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f94d4073200 0x7f94d4075710 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00ab40 tx=0x7f94ec00a250 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:37.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.635+0000 7f94e67fc640 1 -- 172.21.15.150:0/654903351 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f94e00c6020 con 0x7f94f00c7580 2024-09-17T05:09:37.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.831+0000 7f94f7def640 1 -- 172.21.15.150:0/654903351 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94f00c5eb0 con 0x7f94f00c7580 2024-09-17T05:09:37.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.831+0000 7f94e67fc640 1 -- 172.21.15.150:0/654903351 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+6540 (secure 0 0 0) 0x7f94e00164b0 con 0x7f94f00c7580 2024-09-17T05:09:37.833 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:37.833 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":146,"btime":"2024-09-17T05:09:19:171770+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:37.834 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 146 2024-09-17T05:09:37.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.835+0000 7f94f7def640 1 -- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f94d4073200 msgr2=0x7f94d4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:37.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.835+0000 7f94f7def640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f94d4073200 0x7f94d4075710 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00ab40 tx=0x7f94ec00a250 comp rx=0 tx=0).stop 2024-09-17T05:09:37.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.835+0000 7f94f7def640 1 -- 172.21.15.150:0/654903351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f00c7580 msgr2=0x7f94f01f10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:37.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.835+0000 7f94f7def640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f00c7580 0x7f94f01f10b0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f94e001aee0 tx=0x7f94e0004040 comp rx=0 tx=0).stop 2024-09-17T05:09:37.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.835+0000 7f94f7def640 1 -- 172.21.15.150:0/654903351 shutdown_connections 2024-09-17T05:09:37.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.835+0000 7f94f7def640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f94d4073200 0x7f94d4075710 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:37.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.835+0000 7f94f7def640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94f01557b0 0x7f94f0147300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:37.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.835+0000 7f94f7def640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94f00c7580 0x7f94f01f10b0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:37.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.835+0000 7f94f7def640 1 --2- 172.21.15.150:0/654903351 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f94f0074420 0x7f94f01488d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:37.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.835+0000 7f94f7def640 1 -- 172.21.15.150:0/654903351 >> 172.21.15.150:0/654903351 conn(0x7f94f00aaa80 msgr2=0x7f94f014be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:37.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.835+0000 7f94f7def640 1 -- 172.21.15.150:0/654903351 shutdown_connections 2024-09-17T05:09:37.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:37.835+0000 7f94f7def640 1 -- 172.21.15.150:0/654903351 wait complete. 2024-09-17T05:09:37.853 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:09:39.854 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:40.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.023+0000 7fe2a6b72640 1 -- 172.21.15.150:0/3731596773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe28c004d50 msgr2=0x7fe28c005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:40.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.023+0000 7fe2a6b72640 1 --2- 172.21.15.150:0/3731596773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe28c004d50 0x7fe28c005220 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fe2a014dc30 tx=0x7fe29001caf0 comp rx=0 tx=0).stop 2024-09-17T05:09:40.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.023+0000 7fe2a510f640 1 -- 172.21.15.150:0/3731596773 reap_dead start 2024-09-17T05:09:40.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.023+0000 7fe2a6b72640 1 -- 172.21.15.150:0/3731596773 shutdown_connections 2024-09-17T05:09:40.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.023+0000 7fe2a6b72640 1 -- 172.21.15.150:0/3731596773 >> 172.21.15.150:0/3731596773 conn(0x7fe2a00c4a00 msgr2=0x7fe2a00c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:40.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.023+0000 7fe2a6b72640 1 -- 172.21.15.150:0/3731596773 shutdown_connections 2024-09-17T05:09:40.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.023+0000 7fe2a6b72640 1 -- 172.21.15.150:0/3731596773 wait complete. 2024-09-17T05:09:40.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.023+0000 7fe2a6b72640 1 Processor -- start 2024-09-17T05:09:40.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.023+0000 7fe2a6b72640 1 -- start start 2024-09-17T05:09:40.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe2a6b72640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2a014dc30 0x7fe2a014d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:40.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe2a6b72640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a0144260 0x7fe2a0144670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe2a6b72640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2a014e730 0x7fe2a014eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe2a6b72640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a006f810 con 0x7fe2a0144260 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe2a6b72640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a00adef0 con 0x7fe2a014e730 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe2a6b72640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe2a0071260 con 0x7fe2a014dc30 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe29ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a0144260 0x7fe2a0144670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe29ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a0144260 0x7fe2a0144670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37450/0 (socket says 172.21.15.150:37450) 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe29ffff640 1 -- 172.21.15.150:0/1927038876 learned_addr learned my addr 172.21.15.150:0/1927038876 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe2a510f640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2a014e730 0x7fe2a014eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe2a490e640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2a014dc30 0x7fe2a014d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe29ffff640 1 -- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2a014dc30 msgr2=0x7fe2a014d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe29ffff640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2a014dc30 0x7fe2a014d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe29ffff640 1 -- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2a014e730 msgr2=0x7fe2a014eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:40.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe29ffff640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2a014e730 0x7fe2a014eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:40.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe29ffff640 1 -- 172.21.15.150:0/1927038876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a01f0c70 con 0x7fe2a0144260 2024-09-17T05:09:40.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe29ffff640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a0144260 0x7fe2a0144670 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fe288010a40 tx=0x7fe288010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:40.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe29d7fa640 1 -- 172.21.15.150:0/1927038876 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28800ece0 con 0x7fe2a0144260 2024-09-17T05:09:40.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe29d7fa640 1 -- 172.21.15.150:0/1927038876 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe28800ee80 con 0x7fe2a0144260 2024-09-17T05:09:40.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe2a6b72640 1 -- 172.21.15.150:0/1927038876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a01f0990 con 0x7fe2a0144260 2024-09-17T05:09:40.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe29d7fa640 1 -- 172.21.15.150:0/1927038876 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe288005260 con 0x7fe2a0144260 2024-09-17T05:09:40.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe2a6b72640 1 -- 172.21.15.150:0/1927038876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2a01500f0 con 0x7fe2a0144260 2024-09-17T05:09:40.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.027+0000 7fe2a6b72640 1 -- 172.21.15.150:0/1927038876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2a0154020 con 0x7fe2a0144260 2024-09-17T05:09:40.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.031+0000 7fe29d7fa640 1 -- 172.21.15.150:0/1927038876 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe288005400 con 0x7fe2a0144260 2024-09-17T05:09:40.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.031+0000 7fe29d7fa640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe280073230 0x7fe280075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:40.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.031+0000 7fe29d7fa640 1 -- 172.21.15.150:0/1927038876 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7fe288095b80 con 0x7fe2a0144260 2024-09-17T05:09:40.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.031+0000 7fe2a490e640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe280073230 0x7fe280075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:40.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.031+0000 7fe2a490e640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe280073230 0x7fe280075740 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe2a01513e0 tx=0x7fe29800c040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:40.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.035+0000 7fe29d7fa640 1 -- 172.21.15.150:0/1927038876 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe28805ff60 con 0x7fe2a0144260 2024-09-17T05:09:40.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.243+0000 7fe2a6b72640 1 -- 172.21.15.150:0/1927038876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2a0075490 con 0x7fe2a0144260 2024-09-17T05:09:40.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.243+0000 7fe29d7fa640 1 -- 172.21.15.150:0/1927038876 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+6540 (secure 0 0 0) 0x7fe28805f860 con 0x7fe2a0144260 2024-09-17T05:09:40.245 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:40.245 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":146,"btime":"2024-09-17T05:09:19:171770+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:40.246 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 146 2024-09-17T05:09:40.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.247+0000 7fe2a6b72640 1 -- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe280073230 msgr2=0x7fe280075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:40.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.247+0000 7fe2a6b72640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe280073230 0x7fe280075740 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe2a01513e0 tx=0x7fe29800c040 comp rx=0 tx=0).stop 2024-09-17T05:09:40.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.247+0000 7fe2a6b72640 1 -- 172.21.15.150:0/1927038876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a0144260 msgr2=0x7fe2a0144670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:40.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.247+0000 7fe2a6b72640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a0144260 0x7fe2a0144670 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fe288010a40 tx=0x7fe288010f00 comp rx=0 tx=0).stop 2024-09-17T05:09:40.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.247+0000 7fe2a6b72640 1 -- 172.21.15.150:0/1927038876 shutdown_connections 2024-09-17T05:09:40.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.247+0000 7fe2a6b72640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe280073230 0x7fe280075740 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:40.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.247+0000 7fe2a6b72640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2a014e730 0x7fe2a014eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:40.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.247+0000 7fe2a6b72640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2a0144260 0x7fe2a0144670 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:40.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.247+0000 7fe2a6b72640 1 --2- 172.21.15.150:0/1927038876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe2a014dc30 0x7fe2a014d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:40.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.247+0000 7fe2a6b72640 1 -- 172.21.15.150:0/1927038876 >> 172.21.15.150:0/1927038876 conn(0x7fe2a00c4a00 msgr2=0x7fe2a00a7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:40.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.247+0000 7fe2a6b72640 1 -- 172.21.15.150:0/1927038876 shutdown_connections 2024-09-17T05:09:40.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:40.247+0000 7fe2a6b72640 1 -- 172.21.15.150:0/1927038876 wait complete. 2024-09-17T05:09:40.269 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:09:42.270 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:42.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.435+0000 7f93e5cda640 1 -- 172.21.15.150:0/3956334646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93e0149d70 msgr2=0x7f93e014a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:42.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.435+0000 7f93e5cda640 1 --2- 172.21.15.150:0/3956334646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93e0149d70 0x7f93e014a180 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009c80 tx=0x7f93c801cd60 comp rx=0 tx=0).stop 2024-09-17T05:09:42.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.435+0000 7f93e5cda640 1 -- 172.21.15.150:0/3956334646 shutdown_connections 2024-09-17T05:09:42.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.435+0000 7f93e5cda640 1 --2- 172.21.15.150:0/3956334646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93e0149d70 0x7f93e014a180 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:42.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.435+0000 7f93e5cda640 1 --2- 172.21.15.150:0/3956334646 >> v2:172.21.15.176:3300/0 conn(0x7f93e00c7eb0 0x7f93e01540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:42.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.435+0000 7f93e5cda640 1 -- 172.21.15.150:0/3956334646 >> 172.21.15.150:0/3956334646 conn(0x7f93e00aab60 msgr2=0x7f93e00aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:42.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.435+0000 7f93e5cda640 1 -- 172.21.15.150:0/3956334646 shutdown_connections 2024-09-17T05:09:42.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.435+0000 7f93e5cda640 1 -- 172.21.15.150:0/3956334646 wait complete. 2024-09-17T05:09:42.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93e5cda640 1 Processor -- start 2024-09-17T05:09:42.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93e5cda640 1 -- start start 2024-09-17T05:09:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93e5cda640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93e00c7eb0 0x7f93e00c2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93e5cda640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93e00c8330 0x7f93e01efa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93e5cda640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93e0149d70 0x7f93e01eff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93e5cda640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f93e01568d0 con 0x7f93e00c7eb0 2024-09-17T05:09:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93e5cda640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f93e0156750 con 0x7f93e00c8330 2024-09-17T05:09:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93e5cda640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f93e01f0710 con 0x7f93e0149d70 2024-09-17T05:09:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93e00c7eb0 0x7f93e00c2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93e00c7eb0 0x7f93e00c2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37476/0 (socket says 172.21.15.150:37476) 2024-09-17T05:09:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93df7fe640 1 -- 172.21.15.150:0/743350409 learned_addr learned my addr 172.21.15.150:0/743350409 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93deffd640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93e00c8330 0x7f93e01efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:42.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93dffff640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93e0149d70 0x7f93e01eff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:42.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93df7fe640 1 -- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93e0149d70 msgr2=0x7f93e01eff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:42.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93df7fe640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93e0149d70 0x7f93e01eff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:42.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93df7fe640 1 -- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93e00c8330 msgr2=0x7f93e01efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:42.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93df7fe640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93e00c8330 0x7f93e01efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:42.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93df7fe640 1 -- 172.21.15.150:0/743350409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e00ae070 con 0x7f93e00c7eb0 2024-09-17T05:09:42.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93df7fe640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93e00c7eb0 0x7f93e00c2130 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f93d800d950 tx=0x7f93d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:42.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93d7fff640 1 -- 172.21.15.150:0/743350409 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d8017070 con 0x7f93e00c7eb0 2024-09-17T05:09:42.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93d7fff640 1 -- 172.21.15.150:0/743350409 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93d8009d20 con 0x7f93e00c7eb0 2024-09-17T05:09:42.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93d7fff640 1 -- 172.21.15.150:0/743350409 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d80106c0 con 0x7f93e00c7eb0 2024-09-17T05:09:42.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93e5cda640 1 -- 172.21.15.150:0/743350409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e00acc40 con 0x7f93e00c7eb0 2024-09-17T05:09:42.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.439+0000 7f93e5cda640 1 -- 172.21.15.150:0/743350409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e00ad200 con 0x7f93e00c7eb0 2024-09-17T05:09:42.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.443+0000 7f93e5cda640 1 -- 172.21.15.150:0/743350409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e01540b0 con 0x7f93e00c7eb0 2024-09-17T05:09:42.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.443+0000 7f93d7fff640 1 -- 172.21.15.150:0/743350409 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f93d801a070 con 0x7f93e00c7eb0 2024-09-17T05:09:42.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.443+0000 7f93d7fff640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f93c0073200 0x7f93c0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:42.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.443+0000 7f93d7fff640 1 -- 172.21.15.150:0/743350409 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f93d8095060 con 0x7f93e00c7eb0 2024-09-17T05:09:42.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.443+0000 7f93deffd640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f93c0073200 0x7f93c0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:42.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.443+0000 7f93deffd640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f93c0073200 0x7f93c0075710 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f93c80af000 tx=0x7f93c801b5a0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:42.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.447+0000 7f93d7fff640 1 -- 172.21.15.150:0/743350409 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f93e00ae070 con 0x7f93e00c7eb0 2024-09-17T05:09:42.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.663+0000 7f93e5cda640 1 -- 172.21.15.150:0/743350409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93e0153060 con 0x7f93e00c7eb0 2024-09-17T05:09:42.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.663+0000 7f93d7fff640 1 -- 172.21.15.150:0/743350409 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+6540 (secure 0 0 0) 0x7f93d805f470 con 0x7f93e00c7eb0 2024-09-17T05:09:42.666 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:42.666 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":146,"btime":"2024-09-17T05:09:19:171770+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:42.666 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 146 2024-09-17T05:09:42.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.667+0000 7f93e5cda640 1 -- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f93c0073200 msgr2=0x7f93c0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:42.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.667+0000 7f93e5cda640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f93c0073200 0x7f93c0075710 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f93c80af000 tx=0x7f93c801b5a0 comp rx=0 tx=0).stop 2024-09-17T05:09:42.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.667+0000 7f93e5cda640 1 -- 172.21.15.150:0/743350409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93e00c7eb0 msgr2=0x7f93e00c2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:42.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.667+0000 7f93e5cda640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93e00c7eb0 0x7f93e00c2130 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f93d800d950 tx=0x7f93d800de10 comp rx=0 tx=0).stop 2024-09-17T05:09:42.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.667+0000 7f93e5cda640 1 -- 172.21.15.150:0/743350409 shutdown_connections 2024-09-17T05:09:42.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.667+0000 7f93e5cda640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f93c0073200 0x7f93c0075710 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:42.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.667+0000 7f93e5cda640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93e00c8330 0x7f93e01efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:42.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.667+0000 7f93e5cda640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f93e0149d70 0x7f93e01eff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:42.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.667+0000 7f93e5cda640 1 --2- 172.21.15.150:0/743350409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93e00c7eb0 0x7f93e00c2130 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:42.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.667+0000 7f93e5cda640 1 -- 172.21.15.150:0/743350409 >> 172.21.15.150:0/743350409 conn(0x7f93e00aab60 msgr2=0x7f93e00a8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:42.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.667+0000 7f93e5cda640 1 -- 172.21.15.150:0/743350409 shutdown_connections 2024-09-17T05:09:42.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:42.667+0000 7f93e5cda640 1 -- 172.21.15.150:0/743350409 wait complete. 2024-09-17T05:09:42.688 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:09:44.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:44.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 -- 172.21.15.150:0/3161983802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44dc06fff0 msgr2=0x7f44dc070400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:44.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 --2- 172.21.15.150:0/3161983802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44dc06fff0 0x7f44dc070400 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f44dc071360 tx=0x7f44cc01c8c0 comp rx=0 tx=0).stop 2024-09-17T05:09:44.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 -- 172.21.15.150:0/3161983802 shutdown_connections 2024-09-17T05:09:44.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 --2- 172.21.15.150:0/3161983802 >> v2:172.21.15.176:3300/0 conn(0x7f44dc0c41c0 0x7f44dc06fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:44.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 --2- 172.21.15.150:0/3161983802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44dc06fff0 0x7f44dc070400 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:44.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 -- 172.21.15.150:0/3161983802 >> 172.21.15.150:0/3161983802 conn(0x7f44dc0c4a00 msgr2=0x7f44dc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:44.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 -- 172.21.15.150:0/3161983802 shutdown_connections 2024-09-17T05:09:44.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 -- 172.21.15.150:0/3161983802 wait complete. 2024-09-17T05:09:44.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 Processor -- start 2024-09-17T05:09:44.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 -- start start 2024-09-17T05:09:44.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44dc06fff0 0x7f44dc147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:44.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44dc0c3db0 0x7f44dc14ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:44.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44dc0c41c0 0x7f44dc0bd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:44.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc0adb30 con 0x7f44dc0c41c0 2024-09-17T05:09:44.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f44dc0ad9b0 con 0x7f44dc0c3db0 2024-09-17T05:09:44.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44e1d20640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f44dc0a75f0 con 0x7f44dc06fff0 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44dbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44dc0c41c0 0x7f44dc0bd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44dbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44dc0c41c0 0x7f44dc0bd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37484/0 (socket says 172.21.15.150:37484) 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.855+0000 7f44dbfff640 1 -- 172.21.15.150:0/373721437 learned_addr learned my addr 172.21.15.150:0/373721437 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44daffd640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44dc0c3db0 0x7f44dc14ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44dbfff640 1 -- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44dc06fff0 msgr2=0x7f44dc147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44db7fe640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44dc06fff0 0x7f44dc147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44dbfff640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44dc06fff0 0x7f44dc147ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44dbfff640 1 -- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44dc0c3db0 msgr2=0x7f44dc14ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44dbfff640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44dc0c3db0 0x7f44dc14ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44dbfff640 1 -- 172.21.15.150:0/373721437 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44dc14be20 con 0x7f44dc0c41c0 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44daffd640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44dc0c3db0 0x7f44dc14ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44dbfff640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44dc0c41c0 0x7f44dc0bd8b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f44cc01cda0 tx=0x7f44cc0a57a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:44.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44d3fff640 1 -- 172.21.15.150:0/373721437 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc0b2040 con 0x7f44dc0c41c0 2024-09-17T05:09:44.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44e1d20640 1 -- 172.21.15.150:0/373721437 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44dc0ab2e0 con 0x7f44dc0c41c0 2024-09-17T05:09:44.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44e1d20640 1 -- 172.21.15.150:0/373721437 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44dc14cf90 con 0x7f44dc0c41c0 2024-09-17T05:09:44.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44d3fff640 1 -- 172.21.15.150:0/373721437 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44cc0bd070 con 0x7f44dc0c41c0 2024-09-17T05:09:44.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44d3fff640 1 -- 172.21.15.150:0/373721437 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc003600 con 0x7f44dc0c41c0 2024-09-17T05:09:44.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44e1d20640 1 -- 172.21.15.150:0/373721437 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44dc070400 con 0x7f44dc0c41c0 2024-09-17T05:09:44.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44d3fff640 1 -- 172.21.15.150:0/373721437 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f44cc0037a0 con 0x7f44dc0c41c0 2024-09-17T05:09:44.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44d3fff640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f44bc073200 0x7f44bc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:44.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44d3fff640 1 -- 172.21.15.150:0/373721437 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f44cc12fb80 con 0x7f44dc0c41c0 2024-09-17T05:09:44.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.859+0000 7f44db7fe640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f44bc073200 0x7f44bc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:44.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.863+0000 7f44db7fe640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f44bc073200 0x7f44bc075710 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f44c40012f0 tx=0x7f44c4009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:44.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:44.863+0000 7f44d3fff640 1 -- 172.21.15.150:0/373721437 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f44cc0164b0 con 0x7f44dc0c41c0 2024-09-17T05:09:45.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.063+0000 7f44e1d20640 1 -- 172.21.15.150:0/373721437 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44dc0a7210 con 0x7f44dc0c41c0 2024-09-17T05:09:45.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44d3fff640 1 -- 172.21.15.150:0/373721437 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+6540 (secure 0 0 0) 0x7f44cc0f9ee0 con 0x7f44dc0c41c0 2024-09-17T05:09:45.067 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:45.068 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":146,"btime":"2024-09-17T05:09:19:171770+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:07:24.656365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5945},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5945":{"gid":5945,"name":"d","rank":0,"incarnation":132,"state":"up:active","state_seq":9,"addr":"172.21.15.176:6837/792092400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":792092400},{"type":"v1","addr":"172.21.15.176:6837","nonce":792092400}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5945,"qdb_cluster":[5945]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:45.068 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 146 2024-09-17T05:09:45.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44e1d20640 1 -- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f44bc073200 msgr2=0x7f44bc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:45.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44e1d20640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f44bc073200 0x7f44bc075710 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f44c40012f0 tx=0x7f44c4009040 comp rx=0 tx=0).stop 2024-09-17T05:09:45.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44e1d20640 1 -- 172.21.15.150:0/373721437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44dc0c41c0 msgr2=0x7f44dc0bd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:45.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44e1d20640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44dc0c41c0 0x7f44dc0bd8b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f44cc01cda0 tx=0x7f44cc0a57a0 comp rx=0 tx=0).stop 2024-09-17T05:09:45.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44e1d20640 1 -- 172.21.15.150:0/373721437 shutdown_connections 2024-09-17T05:09:45.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44e1d20640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f44bc073200 0x7f44bc075710 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:45.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44e1d20640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f44dc0c3db0 0x7f44dc14ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:45.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44e1d20640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44dc0c41c0 0x7f44dc0bd8b0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:45.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44e1d20640 1 --2- 172.21.15.150:0/373721437 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f44dc06fff0 0x7f44dc147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:45.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44e1d20640 1 -- 172.21.15.150:0/373721437 >> 172.21.15.150:0/373721437 conn(0x7f44dc0c4a00 msgr2=0x7f44dc0a7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:45.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44e1d20640 1 -- 172.21.15.150:0/373721437 shutdown_connections 2024-09-17T05:09:45.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:45.067+0000 7f44e1d20640 1 -- 172.21.15.150:0/373721437 wait complete. 2024-09-17T05:09:45.088 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:09:47.089 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:47.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.255+0000 7ff463e3f640 1 -- 172.21.15.150:0/3965024459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff444004810 msgr2=0x7ff444004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.255+0000 7ff463e3f640 1 --2- 172.21.15.150:0/3965024459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff444004810 0x7ff444004c20 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7ff4500016f0 tx=0x7ff45001db00 comp rx=0 tx=0).stop 2024-09-17T05:09:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.255+0000 7ff4623dc640 1 -- 172.21.15.150:0/3965024459 reap_dead start 2024-09-17T05:09:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.255+0000 7ff463e3f640 1 -- 172.21.15.150:0/3965024459 shutdown_connections 2024-09-17T05:09:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.255+0000 7ff463e3f640 1 -- 172.21.15.150:0/3965024459 >> 172.21.15.150:0/3965024459 conn(0x7ff45c0c4a00 msgr2=0x7ff45c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.255+0000 7ff463e3f640 1 -- 172.21.15.150:0/3965024459 shutdown_connections 2024-09-17T05:09:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.255+0000 7ff463e3f640 1 -- 172.21.15.150:0/3965024459 wait complete. 2024-09-17T05:09:47.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.255+0000 7ff463e3f640 1 Processor -- start 2024-09-17T05:09:47.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.255+0000 7ff463e3f640 1 -- start start 2024-09-17T05:09:47.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff463e3f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff45c156650 0x7ff45c0ac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:47.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff463e3f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff45c0ab1c0 0x7ff45c14b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:47.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff463e3f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff45c1491b0 0x7ff45c1495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:47.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff463e3f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c070da0 con 0x7ff45c1491b0 2024-09-17T05:09:47.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff463e3f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c070ac0 con 0x7ff45c156650 2024-09-17T05:09:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff463e3f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff45c150770 con 0x7ff45c0ab1c0 2024-09-17T05:09:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff4623dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff45c1491b0 0x7ff45c1495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff4623dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff45c1491b0 0x7ff45c1495c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37504/0 (socket says 172.21.15.150:37504) 2024-09-17T05:09:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff4623dc640 1 -- 172.21.15.150:0/1641100904 learned_addr learned my addr 172.21.15.150:0/1641100904 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff461bdb640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff45c156650 0x7ff45c0ac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff4613da640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff45c0ab1c0 0x7ff45c14b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff4623dc640 1 -- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff45c0ab1c0 msgr2=0x7ff45c14b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff4623dc640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff45c0ab1c0 0x7ff45c14b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff4623dc640 1 -- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff45c156650 msgr2=0x7ff45c0ac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff4623dc640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff45c156650 0x7ff45c0ac7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff4623dc640 1 -- 172.21.15.150:0/1641100904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45c149b00 con 0x7ff45c1491b0 2024-09-17T05:09:47.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff4623dc640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff45c1491b0 0x7ff45c1495c0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7ff45c0ae070 tx=0x7ff44c00d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff45a7fc640 1 -- 172.21.15.150:0/1641100904 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c012880 con 0x7ff45c1491b0 2024-09-17T05:09:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff45a7fc640 1 -- 172.21.15.150:0/1641100904 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff44c012b10 con 0x7ff45c1491b0 2024-09-17T05:09:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff45a7fc640 1 -- 172.21.15.150:0/1641100904 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c0113d0 con 0x7ff45c1491b0 2024-09-17T05:09:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff463e3f640 1 -- 172.21.15.150:0/1641100904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff45c0acd10 con 0x7ff45c1491b0 2024-09-17T05:09:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff463e3f640 1 -- 172.21.15.150:0/1641100904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff45c0c16f0 con 0x7ff45c1491b0 2024-09-17T05:09:47.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff463e3f640 1 -- 172.21.15.150:0/1641100904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff45c150060 con 0x7ff45c1491b0 2024-09-17T05:09:47.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff45a7fc640 1 -- 172.21.15.150:0/1641100904 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff44c011570 con 0x7ff45c1491b0 2024-09-17T05:09:47.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff45a7fc640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff448073200 0x7ff448075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:47.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.259+0000 7ff45a7fc640 1 -- 172.21.15.150:0/1641100904 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7ff44c098fb0 con 0x7ff45c1491b0 2024-09-17T05:09:47.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.263+0000 7ff461bdb640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff448073200 0x7ff448075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:47.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.263+0000 7ff461bdb640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff448073200 0x7ff448075710 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff45c153540 tx=0x7ff454009500 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:47.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.263+0000 7ff45a7fc640 1 -- 172.21.15.150:0/1641100904 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff45c149b00 con 0x7ff45c1491b0 2024-09-17T05:09:47.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.459+0000 7ff463e3f640 1 -- 172.21.15.150:0/1641100904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff45c069210 con 0x7ff45c1491b0 2024-09-17T05:09:47.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.459+0000 7ff45a7fc640 1 -- 172.21.15.150:0/1641100904 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+5710 (secure 0 0 0) 0x7ff44c063450 con 0x7ff45c1491b0 2024-09-17T05:09:47.460 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:47.460 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":148,"btime":"2024-09-17T05:09:47:285528+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:46.536543+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:47.461 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 148 2024-09-17T05:09:47.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.459+0000 7ff463e3f640 1 -- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff448073200 msgr2=0x7ff448075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:47.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.459+0000 7ff463e3f640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff448073200 0x7ff448075710 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff45c153540 tx=0x7ff454009500 comp rx=0 tx=0).stop 2024-09-17T05:09:47.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.459+0000 7ff463e3f640 1 -- 172.21.15.150:0/1641100904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff45c1491b0 msgr2=0x7ff45c1495c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:47.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.459+0000 7ff463e3f640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff45c1491b0 0x7ff45c1495c0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7ff45c0ae070 tx=0x7ff44c00d8f0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.463+0000 7ff463e3f640 1 -- 172.21.15.150:0/1641100904 shutdown_connections 2024-09-17T05:09:47.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.463+0000 7ff463e3f640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff448073200 0x7ff448075710 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.463+0000 7ff463e3f640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff45c1491b0 0x7ff45c1495c0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.463+0000 7ff463e3f640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff45c0ab1c0 0x7ff45c14b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.463+0000 7ff463e3f640 1 --2- 172.21.15.150:0/1641100904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff45c156650 0x7ff45c0ac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.463+0000 7ff463e3f640 1 -- 172.21.15.150:0/1641100904 >> 172.21.15.150:0/1641100904 conn(0x7ff45c0c4a00 msgr2=0x7ff45c1512b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:47.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.463+0000 7ff463e3f640 1 -- 172.21.15.150:0/1641100904 shutdown_connections 2024-09-17T05:09:47.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.463+0000 7ff463e3f640 1 -- 172.21.15.150:0/1641100904 wait complete. 2024-09-17T05:09:47.485 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-09-17T05:09:47.485 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:09:47.486 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:47.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.631+0000 7f579a151640 1 -- 172.21.15.150:0/791505117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794151d00 msgr2=0x7f57940c3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:47.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.631+0000 7f579a151640 1 --2- 172.21.15.150:0/791505117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794151d00 0x7f57940c3430 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f577c009840 tx=0x7f577c01c7c0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.631+0000 7f579a151640 1 -- 172.21.15.150:0/791505117 shutdown_connections 2024-09-17T05:09:47.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.631+0000 7f579a151640 1 --2- 172.21.15.150:0/791505117 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f57940c3970 0x7f57940c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.631+0000 7f579a151640 1 --2- 172.21.15.150:0/791505117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794151d00 0x7f57940c3430 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.631+0000 7f579a151640 1 -- 172.21.15.150:0/791505117 >> 172.21.15.150:0/791505117 conn(0x7f57940c4a00 msgr2=0x7f57940c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:47.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.631+0000 7f579a151640 1 -- 172.21.15.150:0/791505117 shutdown_connections 2024-09-17T05:09:47.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.631+0000 7f579a151640 1 -- 172.21.15.150:0/791505117 wait complete. 2024-09-17T05:09:47.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.631+0000 7f579a151640 1 Processor -- start 2024-09-17T05:09:47.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.631+0000 7f579a151640 1 -- start start 2024-09-17T05:09:47.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f579a151640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57940c3970 0x7f579414d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:47.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f579a151640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5794151d00 0x7f5794153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f579a151640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794157c30 0x7f579414af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f579a151640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f57941619b0 con 0x7f5794157c30 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f579a151640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5794161830 con 0x7f57940c3970 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f579a151640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5794161b30 con 0x7f5794151d00 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f5793fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794157c30 0x7f579414af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f5793fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794157c30 0x7f579414af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37522/0 (socket says 172.21.15.150:37522) 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f5793fff640 1 -- 172.21.15.150:0/3614186012 learned_addr learned my addr 172.21.15.150:0/3614186012 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f57937fe640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57940c3970 0x7f579414d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f5792ffd640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5794151d00 0x7f5794153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f5793fff640 1 -- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5794151d00 msgr2=0x7f5794153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f5793fff640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5794151d00 0x7f5794153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f5793fff640 1 -- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57940c3970 msgr2=0x7f579414d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:47.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f5793fff640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57940c3970 0x7f579414d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f5793fff640 1 -- 172.21.15.150:0/3614186012 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57941f12d0 con 0x7f5794157c30 2024-09-17T05:09:47.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f5793fff640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794157c30 0x7f579414af80 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f578400c3c0 tx=0x7f578400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:47.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f578bfff640 1 -- 172.21.15.150:0/3614186012 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784017070 con 0x7f5794157c30 2024-09-17T05:09:47.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f578bfff640 1 -- 172.21.15.150:0/3614186012 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5784003600 con 0x7f5794157c30 2024-09-17T05:09:47.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f578bfff640 1 -- 172.21.15.150:0/3614186012 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784003a50 con 0x7f5794157c30 2024-09-17T05:09:47.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f579a151640 1 -- 172.21.15.150:0/3614186012 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5794155480 con 0x7f5794157c30 2024-09-17T05:09:47.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f579a151640 1 -- 172.21.15.150:0/3614186012 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5794152e20 con 0x7f5794157c30 2024-09-17T05:09:47.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f579a151640 1 -- 172.21.15.150:0/3614186012 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57940c3d80 con 0x7f5794157c30 2024-09-17T05:09:47.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f578bfff640 1 -- 172.21.15.150:0/3614186012 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f578401a070 con 0x7f5794157c30 2024-09-17T05:09:47.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f578bfff640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5774073230 0x7f5774075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:47.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f578bfff640 1 -- 172.21.15.150:0/3614186012 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f5784094240 con 0x7f5794157c30 2024-09-17T05:09:47.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.635+0000 7f57937fe640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5774073230 0x7f5774075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:47.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.639+0000 7f57937fe640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5774073230 0x7f5774075740 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f577c0af000 tx=0x7f577c01b060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:47.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.639+0000 7f578bfff640 1 -- 172.21.15.150:0/3614186012 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f578405e5e0 con 0x7f5794157c30 2024-09-17T05:09:47.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.851+0000 7f579a151640 1 -- 172.21.15.150:0/3614186012 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57940a7730 con 0x7f5794157c30 2024-09-17T05:09:47.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.851+0000 7f578bfff640 1 -- 172.21.15.150:0/3614186012 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+5710 (secure 0 0 0) 0x7f578405dee0 con 0x7f5794157c30 2024-09-17T05:09:47.853 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:47.854 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":148,"btime":"2024-09-17T05:09:47:285528+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:46.536543+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:47.854 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 148 2024-09-17T05:09:47.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.855+0000 7f579a151640 1 -- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5774073230 msgr2=0x7f5774075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:47.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.855+0000 7f579a151640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5774073230 0x7f5774075740 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f577c0af000 tx=0x7f577c01b060 comp rx=0 tx=0).stop 2024-09-17T05:09:47.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.855+0000 7f579a151640 1 -- 172.21.15.150:0/3614186012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794157c30 msgr2=0x7f579414af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:47.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.855+0000 7f579a151640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794157c30 0x7f579414af80 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f578400c3c0 tx=0x7f578400c7c0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.855+0000 7f579a151640 1 -- 172.21.15.150:0/3614186012 shutdown_connections 2024-09-17T05:09:47.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.855+0000 7f579a151640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5774073230 0x7f5774075740 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.855+0000 7f579a151640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5794157c30 0x7f579414af80 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.855+0000 7f579a151640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5794151d00 0x7f5794153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.855+0000 7f579a151640 1 --2- 172.21.15.150:0/3614186012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f57940c3970 0x7f579414d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:47.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.855+0000 7f579a151640 1 -- 172.21.15.150:0/3614186012 >> 172.21.15.150:0/3614186012 conn(0x7f57940c4a00 msgr2=0x7f57940ab270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:47.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.855+0000 7f579a151640 1 -- 172.21.15.150:0/3614186012 shutdown_connections 2024-09-17T05:09:47.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:47.855+0000 7f579a151640 1 -- 172.21.15.150:0/3614186012 wait complete. 2024-09-17T05:09:47.879 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:09:47:285528+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 148, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 148, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7230": { "addr": "172.21.15.150:6833/2056467678", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2056467678, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 2056467678, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7230, "incarnation": 147, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:09:46.536543+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7230 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 145, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7158": { "addr": "172.21.15.150:6835/285376980", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 285376980, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 285376980, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7158, "incarnation": 142, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:09:09.137150+0000", "qdb_cluster": [ 7158 ], "qdb_leader": 7158, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7158 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:09:49.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:50.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.067+0000 7ffaeacae640 1 -- 172.21.15.150:0/72876131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40700b0 msgr2=0x7ffae415dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:50.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.067+0000 7ffaeacae640 1 --2- 172.21.15.150:0/72876131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40700b0 0x7ffae415dfa0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009bc0 tx=0x7ffadc01cb10 comp rx=0 tx=0).stop 2024-09-17T05:09:50.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.067+0000 7ffaeacae640 1 -- 172.21.15.150:0/72876131 shutdown_connections 2024-09-17T05:09:50.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.067+0000 7ffaeacae640 1 --2- 172.21.15.150:0/72876131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40700b0 0x7ffae415dfa0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:50.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.067+0000 7ffaeacae640 1 --2- 172.21.15.150:0/72876131 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffae40745c0 0x7ffae4150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:50.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.067+0000 7ffaeacae640 1 -- 172.21.15.150:0/72876131 >> 172.21.15.150:0/72876131 conn(0x7ffae40c5ae0 msgr2=0x7ffae4073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:50.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.067+0000 7ffaeacae640 1 -- 172.21.15.150:0/72876131 shutdown_connections 2024-09-17T05:09:50.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.067+0000 7ffaeacae640 1 -- 172.21.15.150:0/72876131 wait complete. 2024-09-17T05:09:50.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.067+0000 7ffaeacae640 1 Processor -- start 2024-09-17T05:09:50.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.067+0000 7ffaeacae640 1 -- start start 2024-09-17T05:09:50.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffaeacae640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffae406fca0 0x7ffae4148ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:50.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffaeacae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40700b0 0x7ffae40c1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:50.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffaeacae640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffae40745c0 0x7ffae4143c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:50.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffaeacae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffae40c3020 con 0x7ffae40700b0 2024-09-17T05:09:50.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffaeacae640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffae4151320 con 0x7ffae406fca0 2024-09-17T05:09:50.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffaeacae640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffae40c31a0 con 0x7ffae40745c0 2024-09-17T05:09:50.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40700b0 0x7ffae40c1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:50.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40700b0 0x7ffae40c1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39246/0 (socket says 172.21.15.150:39246) 2024-09-17T05:09:50.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae3fff640 1 -- 172.21.15.150:0/1956251346 learned_addr learned my addr 172.21.15.150:0/1956251346 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:50.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae8a4a640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffae406fca0 0x7ffae4148ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:50.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae924b640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffae40745c0 0x7ffae4143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:50.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae3fff640 1 -- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffae40745c0 msgr2=0x7ffae4143c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:50.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae3fff640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffae40745c0 0x7ffae4143c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:50.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae3fff640 1 -- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffae406fca0 msgr2=0x7ffae4148ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:50.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae3fff640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffae406fca0 0x7ffae4148ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:50.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae3fff640 1 -- 172.21.15.150:0/1956251346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae4144780 con 0x7ffae40700b0 2024-09-17T05:09:50.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae3fff640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40700b0 0x7ffae40c1c20 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ffad000c400 tx=0x7ffad000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:50.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae17fa640 1 -- 172.21.15.150:0/1956251346 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad0017070 con 0x7ffae40700b0 2024-09-17T05:09:50.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae17fa640 1 -- 172.21.15.150:0/1956251346 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffad0003600 con 0x7ffae40700b0 2024-09-17T05:09:50.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffaeacae640 1 -- 172.21.15.150:0/1956251346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae41f0590 con 0x7ffae40700b0 2024-09-17T05:09:50.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae17fa640 1 -- 172.21.15.150:0/1956251346 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad0003a50 con 0x7ffae40700b0 2024-09-17T05:09:50.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffaeacae640 1 -- 172.21.15.150:0/1956251346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae41efab0 con 0x7ffae40700b0 2024-09-17T05:09:50.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffaeacae640 1 -- 172.21.15.150:0/1956251346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffae40ab140 con 0x7ffae40700b0 2024-09-17T05:09:50.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.071+0000 7ffae17fa640 1 -- 172.21.15.150:0/1956251346 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ffad001a070 con 0x7ffae40700b0 2024-09-17T05:09:50.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.075+0000 7ffae17fa640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffac8073230 0x7ffac8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:50.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.075+0000 7ffae17fa640 1 -- 172.21.15.150:0/1956251346 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7ffad009b350 con 0x7ffae40700b0 2024-09-17T05:09:50.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.075+0000 7ffae8a4a640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffac8073230 0x7ffac8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:50.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.075+0000 7ffae8a4a640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffac8073230 0x7ffac8075740 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ffad800bb50 tx=0x7ffad800b2e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:50.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.079+0000 7ffae17fa640 1 -- 172.21.15.150:0/1956251346 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ffad005db90 con 0x7ffae40700b0 2024-09-17T05:09:50.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.355+0000 7ffaeacae640 1 -- 172.21.15.150:0/1956251346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffae4151000 con 0x7ffae40700b0 2024-09-17T05:09:50.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.355+0000 7ffae17fa640 1 -- 172.21.15.150:0/1956251346 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+5715 (secure 0 0 0) 0x7ffad00625c0 con 0x7ffae40700b0 2024-09-17T05:09:50.358 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:50.358 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":150,"btime":"2024-09-17T05:09:49:306545+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:50.358 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 150 2024-09-17T05:09:50.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.359+0000 7ffaeacae640 1 -- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffac8073230 msgr2=0x7ffac8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:50.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.359+0000 7ffaeacae640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffac8073230 0x7ffac8075740 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ffad800bb50 tx=0x7ffad800b2e0 comp rx=0 tx=0).stop 2024-09-17T05:09:50.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.359+0000 7ffaeacae640 1 -- 172.21.15.150:0/1956251346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40700b0 msgr2=0x7ffae40c1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:50.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.359+0000 7ffaeacae640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40700b0 0x7ffae40c1c20 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ffad000c400 tx=0x7ffad000c8c0 comp rx=0 tx=0).stop 2024-09-17T05:09:50.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.359+0000 7ffaeacae640 1 -- 172.21.15.150:0/1956251346 shutdown_connections 2024-09-17T05:09:50.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.359+0000 7ffaeacae640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffac8073230 0x7ffac8075740 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:50.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.359+0000 7ffaeacae640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffae40745c0 0x7ffae4143c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:50.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.359+0000 7ffaeacae640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffae40700b0 0x7ffae40c1c20 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:50.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.359+0000 7ffaeacae640 1 --2- 172.21.15.150:0/1956251346 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffae406fca0 0x7ffae4148ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:50.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.359+0000 7ffaeacae640 1 -- 172.21.15.150:0/1956251346 >> 172.21.15.150:0/1956251346 conn(0x7ffae40c5ae0 msgr2=0x7ffae4147ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:50.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.359+0000 7ffaeacae640 1 -- 172.21.15.150:0/1956251346 shutdown_connections 2024-09-17T05:09:50.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:50.359+0000 7ffaeacae640 1 -- 172.21.15.150:0/1956251346 wait complete. 2024-09-17T05:09:50.388 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=5945 2024-09-17T05:09:50.388 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.d 2024-09-17T05:09:54.181 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T05:09:54.179+0000 7f4f09bf9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:09:57.390 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-09-17T05:09:57.390 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T05:09:57.390 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T05:09:57.393 INFO:tasks.ceph.mds.d:Started 2024-09-17T05:09:57.393 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:09:57.509 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-09-17T05:09:57.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.586+0000 7f58ba42b640 1 -- 172.21.15.150:0/943363561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b40a8120 msgr2=0x7f58b4074900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:57.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.586+0000 7f58ba42b640 1 --2- 172.21.15.150:0/943363561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b40a8120 0x7f58b4074900 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f589c009840 tx=0x7f589c01c700 comp rx=0 tx=0).stop 2024-09-17T05:09:57.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.586+0000 7f58ba42b640 1 -- 172.21.15.150:0/943363561 shutdown_connections 2024-09-17T05:09:57.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.586+0000 7f58ba42b640 1 --2- 172.21.15.150:0/943363561 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b40a8120 0x7f58b4074900 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:57.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.586+0000 7f58ba42b640 1 --2- 172.21.15.150:0/943363561 >> v2:172.21.15.176:3300/0 conn(0x7f58b40c39d0 0x7f58b4150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.586+0000 7f58ba42b640 1 -- 172.21.15.150:0/943363561 >> 172.21.15.150:0/943363561 conn(0x7f58b406e360 msgr2=0x7f58b406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.586+0000 7f58ba42b640 1 -- 172.21.15.150:0/943363561 shutdown_connections 2024-09-17T05:09:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.586+0000 7f58ba42b640 1 -- 172.21.15.150:0/943363561 wait complete. 2024-09-17T05:09:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58ba42b640 1 Processor -- start 2024-09-17T05:09:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58ba42b640 1 -- start start 2024-09-17T05:09:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58ba42b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58b4074e40 0x7f58b41559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58ba42b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b40a8120 0x7f58b41562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58ba42b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58b40c39d0 0x7f58b4151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58ba42b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f58b40a86b0 con 0x7f58b4074e40 2024-09-17T05:09:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58ba42b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f58b40a8530 con 0x7f58b40c39d0 2024-09-17T05:09:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58ba42b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f58b4160780 con 0x7f58b40a8120 2024-09-17T05:09:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58b4074e40 0x7f58b41559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58b4074e40 0x7f58b41559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39258/0 (socket says 172.21.15.150:39258) 2024-09-17T05:09:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b3fff640 1 -- 172.21.15.150:0/407255065 learned_addr learned my addr 172.21.15.150:0/407255065 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:09:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58ab7fe640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b40a8120 0x7f58b41562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b89c8640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58b40c39d0 0x7f58b4151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b3fff640 1 -- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b40a8120 msgr2=0x7f58b41562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b3fff640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b40a8120 0x7f58b41562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b3fff640 1 -- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58b40c39d0 msgr2=0x7f58b4151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b3fff640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58b40c39d0 0x7f58b4151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b3fff640 1 -- 172.21.15.150:0/407255065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b41549d0 con 0x7f58b4074e40 2024-09-17T05:09:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b3fff640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58b4074e40 0x7f58b41559a0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f58a400ebf0 tx=0x7f58a400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b17fa640 1 -- 172.21.15.150:0/407255065 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a4010630 con 0x7f58b4074e40 2024-09-17T05:09:57.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b17fa640 1 -- 172.21.15.150:0/407255065 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58a40108c0 con 0x7f58b4074e40 2024-09-17T05:09:57.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58b17fa640 1 -- 172.21.15.150:0/407255065 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a4010cf0 con 0x7f58b4074e40 2024-09-17T05:09:57.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58ba42b640 1 -- 172.21.15.150:0/407255065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58b41e5e10 con 0x7f58b4074e40 2024-09-17T05:09:57.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58ba42b640 1 -- 172.21.15.150:0/407255065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58b4156830 con 0x7f58b4074e40 2024-09-17T05:09:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.590+0000 7f58ba42b640 1 -- 172.21.15.150:0/407255065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58b414e320 con 0x7f58b4074e40 2024-09-17T05:09:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.594+0000 7f58b17fa640 1 -- 172.21.15.150:0/407255065 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f58a4018420 con 0x7f58b4074e40 2024-09-17T05:09:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.594+0000 7f58b17fa640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5894073200 0x7f5894075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:09:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.594+0000 7f58b17fa640 1 -- 172.21.15.150:0/407255065 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f58a4095be0 con 0x7f58b4074e40 2024-09-17T05:09:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.594+0000 7f58ab7fe640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5894073200 0x7f5894075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:09:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.594+0000 7f58ab7fe640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5894073200 0x7f5894075710 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f589c0ae000 tx=0x7f589c01af40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:09:57.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.598+0000 7f58b17fa640 1 -- 172.21.15.150:0/407255065 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f58b414e320 con 0x7f58b4074e40 2024-09-17T05:09:57.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.858+0000 7f58ba42b640 1 -- 172.21.15.150:0/407255065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58b4074900 con 0x7f58b4074e40 2024-09-17T05:09:57.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58b17fa640 1 -- 172.21.15.150:0/407255065 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+6544 (secure 0 0 0) 0x7f58a405f920 con 0x7f58b4074e40 2024-09-17T05:09:57.863 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:09:57.863 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":151,"btime":"2024-09-17T05:09:57:585442+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:09:57.864 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 151 2024-09-17T05:09:57.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58ba42b640 1 -- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5894073200 msgr2=0x7f5894075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58ba42b640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5894073200 0x7f5894075710 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f589c0ae000 tx=0x7f589c01af40 comp rx=0 tx=0).stop 2024-09-17T05:09:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58ba42b640 1 -- 172.21.15.150:0/407255065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58b4074e40 msgr2=0x7f58b41559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:09:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58ba42b640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58b4074e40 0x7f58b41559a0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f58a400ebf0 tx=0x7f58a400c6a0 comp rx=0 tx=0).stop 2024-09-17T05:09:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58ba42b640 1 -- 172.21.15.150:0/407255065 shutdown_connections 2024-09-17T05:09:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58ba42b640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5894073200 0x7f5894075710 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58ba42b640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58b40c39d0 0x7f58b4151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58ba42b640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f58b40a8120 0x7f58b41562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58ba42b640 1 --2- 172.21.15.150:0/407255065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f58b4074e40 0x7f58b41559a0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:09:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58ba42b640 1 -- 172.21.15.150:0/407255065 >> 172.21.15.150:0/407255065 conn(0x7f58b406e360 msgr2=0x7f58b406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:09:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58ba42b640 1 -- 172.21.15.150:0/407255065 shutdown_connections 2024-09-17T05:09:57.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:09:57.862+0000 7f58ba42b640 1 -- 172.21.15.150:0/407255065 wait complete. 2024-09-17T05:09:57.888 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-09-17T05:09:57.889 INFO:tasks.mds_thrash.fs.[a]:waiting for 106 secs before thrashing 2024-09-17T05:10:03.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:10:03.406 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:10:33.677 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:10:33.681 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:10:35.133 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:10:35.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f381e7fc640 1 -- 172.21.15.150:0/2014655191 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810003800 con 0x7f38200ad7f0 2024-09-17T05:10:35.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 -- 172.21.15.150:0/2014655191 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38200ad7f0 msgr2=0x7f38200c6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:35.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 --2- 172.21.15.150:0/2014655191 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38200ad7f0 0x7f38200c6230 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3810009840 tx=0x7f381001c720 comp rx=0 tx=0).stop 2024-09-17T05:10:35.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 -- 172.21.15.150:0/2014655191 shutdown_connections 2024-09-17T05:10:35.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 --2- 172.21.15.150:0/2014655191 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38200ad7f0 0x7f38200c6230 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 -- 172.21.15.150:0/2014655191 >> 172.21.15.150:0/2014655191 conn(0x7f38200c9de0 msgr2=0x7f38200ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:35.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 -- 172.21.15.150:0/2014655191 shutdown_connections 2024-09-17T05:10:35.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 -- 172.21.15.150:0/2014655191 wait complete. 2024-09-17T05:10:35.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 Processor -- start 2024-09-17T05:10:35.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 -- start start 2024-09-17T05:10:35.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38200ad7f0 0x7f382014bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:35.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3820151e10 0x7f382014af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:35.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3820157ca0 0x7f3820152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:35.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.310+0000 7f3825a33640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f38200ad210 con 0x7f3820157ca0 2024-09-17T05:10:35.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f3825a33640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f38200c5a40 con 0x7f3820151e10 2024-09-17T05:10:35.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f3825a33640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3820162860 con 0x7f38200ad7f0 2024-09-17T05:10:35.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3820157ca0 0x7f3820152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:35.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3820157ca0 0x7f3820152800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39542/0 (socket says 172.21.15.150:39542) 2024-09-17T05:10:35.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381f7fe640 1 -- 172.21.15.150:0/276975354 learned_addr learned my addr 172.21.15.150:0/276975354 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:10:35.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381effd640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38200ad7f0 0x7f382014bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:35.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f3817fff640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3820151e10 0x7f382014af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:35.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381f7fe640 1 -- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38200ad7f0 msgr2=0x7f382014bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:35.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381f7fe640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38200ad7f0 0x7f382014bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381f7fe640 1 -- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3820151e10 msgr2=0x7f382014af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:35.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381f7fe640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3820151e10 0x7f382014af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381f7fe640 1 -- 172.21.15.150:0/276975354 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382014cc10 con 0x7f3820157ca0 2024-09-17T05:10:35.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381f7fe640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3820157ca0 0x7f3820152800 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f381800c790 tx=0x7f381800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:35.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381cff9640 1 -- 172.21.15.150:0/276975354 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3818003b80 con 0x7f3820157ca0 2024-09-17T05:10:35.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381cff9640 1 -- 172.21.15.150:0/276975354 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3818003d20 con 0x7f3820157ca0 2024-09-17T05:10:35.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f3825a33640 1 -- 172.21.15.150:0/276975354 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f382014ceb0 con 0x7f3820157ca0 2024-09-17T05:10:35.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381cff9640 1 -- 172.21.15.150:0/276975354 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381800aa50 con 0x7f3820157ca0 2024-09-17T05:10:35.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f3825a33640 1 -- 172.21.15.150:0/276975354 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3820150cb0 con 0x7f3820157ca0 2024-09-17T05:10:35.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f3825a33640 1 -- 172.21.15.150:0/276975354 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38200c6230 con 0x7f3820157ca0 2024-09-17T05:10:35.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381cff9640 1 -- 172.21.15.150:0/276975354 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f381800abf0 con 0x7f3820157ca0 2024-09-17T05:10:35.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381cff9640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3800073200 0x7f3800075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:35.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381cff9640 1 -- 172.21.15.150:0/276975354 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f3818094b90 con 0x7f3820157ca0 2024-09-17T05:10:35.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.314+0000 7f381effd640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3800073200 0x7f3800075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:35.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.318+0000 7f381effd640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3800073200 0x7f3800075710 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f38100ae000 tx=0x7f381001af60 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:35.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.322+0000 7f381cff9640 1 -- 172.21.15.150:0/276975354 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f381805e8e0 con 0x7f3820157ca0 2024-09-17T05:10:35.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.602+0000 7f3825a33640 1 -- 172.21.15.150:0/276975354 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38200c5330 con 0x7f3820157ca0 2024-09-17T05:10:35.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.602+0000 7f381cff9640 1 -- 172.21.15.150:0/276975354 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+6544 (secure 0 0 0) 0x7f3818063310 con 0x7f3820157ca0 2024-09-17T05:10:35.602 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:10:35.602 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":151,"btime":"2024-09-17T05:09:57:585442+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:10:35.603 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 151 2024-09-17T05:10:35.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.606+0000 7f3825a33640 1 -- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3800073200 msgr2=0x7f3800075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:35.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.606+0000 7f3825a33640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3800073200 0x7f3800075710 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f38100ae000 tx=0x7f381001af60 comp rx=0 tx=0).stop 2024-09-17T05:10:35.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.606+0000 7f3825a33640 1 -- 172.21.15.150:0/276975354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3820157ca0 msgr2=0x7f3820152800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:35.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.606+0000 7f3825a33640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3820157ca0 0x7f3820152800 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f381800c790 tx=0x7f381800cc50 comp rx=0 tx=0).stop 2024-09-17T05:10:35.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.606+0000 7f3825a33640 1 -- 172.21.15.150:0/276975354 shutdown_connections 2024-09-17T05:10:35.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.606+0000 7f3825a33640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3800073200 0x7f3800075710 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.606+0000 7f3825a33640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3820151e10 0x7f382014af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.606+0000 7f3825a33640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3820157ca0 0x7f3820152800 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.606+0000 7f3825a33640 1 --2- 172.21.15.150:0/276975354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f38200ad7f0 0x7f382014bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.606+0000 7f3825a33640 1 -- 172.21.15.150:0/276975354 >> 172.21.15.150:0/276975354 conn(0x7f38200c9de0 msgr2=0x7f38200c8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:35.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.606+0000 7f3825a33640 1 -- 172.21.15.150:0/276975354 shutdown_connections 2024-09-17T05:10:35.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.606+0000 7f3825a33640 1 -- 172.21.15.150:0/276975354 wait complete. 2024-09-17T05:10:35.631 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-09-17T05:10:35.632 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T05:10:35.632 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:10:35.645 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T05:10:35.645 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:10:35.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.850+0000 7ffadae94640 1 -- 172.21.15.150:0/883724890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad4151c50 msgr2=0x7ffad40a98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:35.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.850+0000 7ffadae94640 1 --2- 172.21.15.150:0/883724890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad4151c50 0x7ffad40a98a0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7ffad40c3140 tx=0x7ffac801c4b0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.850+0000 7ffadae94640 1 -- 172.21.15.150:0/883724890 shutdown_connections 2024-09-17T05:10:35.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.850+0000 7ffadae94640 1 --2- 172.21.15.150:0/883724890 >> v2:172.21.15.176:3300/0 conn(0x7ffad40746b0 0x7ffad4157cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.850+0000 7ffadae94640 1 --2- 172.21.15.150:0/883724890 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffad4073d60 0x7ffad4074170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.850+0000 7ffadae94640 1 --2- 172.21.15.150:0/883724890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad4151c50 0x7ffad40a98a0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.850+0000 7ffadae94640 1 -- 172.21.15.150:0/883724890 >> 172.21.15.150:0/883724890 conn(0x7ffad40c91b0 msgr2=0x7ffad40c51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:35.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.850+0000 7ffadae94640 1 -- 172.21.15.150:0/883724890 shutdown_connections 2024-09-17T05:10:35.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.850+0000 7ffadae94640 1 -- 172.21.15.150:0/883724890 wait complete. 2024-09-17T05:10:35.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffadae94640 1 Processor -- start 2024-09-17T05:10:35.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffadae94640 1 -- start start 2024-09-17T05:10:35.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffadae94640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffad4073d60 0x7ffad4148310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:35.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffadae94640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad40746b0 0x7ffad4150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:35.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffadae94640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffad4151c50 0x7ffad40c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:35.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffadae94640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffad40a9de0 con 0x7ffad40746b0 2024-09-17T05:10:35.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffadae94640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffad406e4f0 con 0x7ffad4073d60 2024-09-17T05:10:35.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffadae94640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffad4148ba0 con 0x7ffad4151c50 2024-09-17T05:10:35.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad40746b0 0x7ffad4150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:35.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad40746b0 0x7ffad4150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39574/0 (socket says 172.21.15.150:39574) 2024-09-17T05:10:35.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad3fff640 1 -- 172.21.15.150:0/4141554830 learned_addr learned my addr 172.21.15.150:0/4141554830 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:10:35.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad9431640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffad4151c50 0x7ffad40c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:35.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad8c30640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffad4073d60 0x7ffad4148310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:35.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad3fff640 1 -- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffad4151c50 msgr2=0x7ffad40c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:35.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad3fff640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffad4151c50 0x7ffad40c6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad3fff640 1 -- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffad4073d60 msgr2=0x7ffad4148310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:35.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad3fff640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffad4073d60 0x7ffad4148310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:35.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad3fff640 1 -- 172.21.15.150:0/4141554830 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad414f6e0 con 0x7ffad40746b0 2024-09-17T05:10:35.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad8c30640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffad4073d60 0x7ffad4148310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:10:35.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad3fff640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad40746b0 0x7ffad4150000 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7ffac000d720 tx=0x7ffac000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:35.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad17fa640 1 -- 172.21.15.150:0/4141554830 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac0017070 con 0x7ffad40746b0 2024-09-17T05:10:35.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad17fa640 1 -- 172.21.15.150:0/4141554830 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffac0005310 con 0x7ffad40746b0 2024-09-17T05:10:35.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffadae94640 1 -- 172.21.15.150:0/4141554830 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad414f9c0 con 0x7ffad40746b0 2024-09-17T05:10:35.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffad17fa640 1 -- 172.21.15.150:0/4141554830 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac0009820 con 0x7ffad40746b0 2024-09-17T05:10:35.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.854+0000 7ffadae94640 1 -- 172.21.15.150:0/4141554830 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad41488e0 con 0x7ffad40746b0 2024-09-17T05:10:35.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.858+0000 7ffadae94640 1 -- 172.21.15.150:0/4141554830 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad40aa210 con 0x7ffad40746b0 2024-09-17T05:10:35.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.858+0000 7ffad17fa640 1 -- 172.21.15.150:0/4141554830 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ffac00099c0 con 0x7ffad40746b0 2024-09-17T05:10:35.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.858+0000 7ffad17fa640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffab8073200 0x7ffab8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:35.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.858+0000 7ffad17fa640 1 -- 172.21.15.150:0/4141554830 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7ffac0094f90 con 0x7ffad40746b0 2024-09-17T05:10:35.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.858+0000 7ffad8c30640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffab8073200 0x7ffab8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:35.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.858+0000 7ffad8c30640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffab8073200 0x7ffab8075710 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ffac80af000 tx=0x7ffac801acf0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:35.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:35.862+0000 7ffad17fa640 1 -- 172.21.15.150:0/4141554830 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ffac005ec40 con 0x7ffad40746b0 2024-09-17T05:10:36.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.098+0000 7ffadae94640 1 -- 172.21.15.150:0/4141554830 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffad40c2b50 con 0x7ffad40746b0 2024-09-17T05:10:36.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.098+0000 7ffad17fa640 1 -- 172.21.15.150:0/4141554830 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+6544 (secure 0 0 0) 0x7ffac0063670 con 0x7ffad40746b0 2024-09-17T05:10:36.098 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:10:36.098 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":151,"btime":"2024-09-17T05:09:57:585442+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:10:36.098 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 151 2024-09-17T05:10:36.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.102+0000 7ffadae94640 1 -- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffab8073200 msgr2=0x7ffab8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:36.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.102+0000 7ffadae94640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffab8073200 0x7ffab8075710 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ffac80af000 tx=0x7ffac801acf0 comp rx=0 tx=0).stop 2024-09-17T05:10:36.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.102+0000 7ffadae94640 1 -- 172.21.15.150:0/4141554830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad40746b0 msgr2=0x7ffad4150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:36.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.102+0000 7ffadae94640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad40746b0 0x7ffad4150000 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7ffac000d720 tx=0x7ffac000dbe0 comp rx=0 tx=0).stop 2024-09-17T05:10:36.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.102+0000 7ffadae94640 1 -- 172.21.15.150:0/4141554830 shutdown_connections 2024-09-17T05:10:36.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.102+0000 7ffadae94640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffab8073200 0x7ffab8075710 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:36.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.102+0000 7ffadae94640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffad4151c50 0x7ffad40c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:36.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.102+0000 7ffadae94640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffad40746b0 0x7ffad4150000 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:36.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.102+0000 7ffadae94640 1 --2- 172.21.15.150:0/4141554830 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffad4073d60 0x7ffad4148310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:36.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.102+0000 7ffadae94640 1 -- 172.21.15.150:0/4141554830 >> 172.21.15.150:0/4141554830 conn(0x7ffad40c91b0 msgr2=0x7ffad414b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:36.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.102+0000 7ffadae94640 1 -- 172.21.15.150:0/4141554830 shutdown_connections 2024-09-17T05:10:36.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:36.102+0000 7ffadae94640 1 -- 172.21.15.150:0/4141554830 wait complete. 2024-09-17T05:10:36.124 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:10:38.125 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:10:38.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 -- 172.21.15.150:0/36089440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19980c7eb0 msgr2=0x7f19981540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:38.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 --2- 172.21.15.150:0/36089440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19980c7eb0 0x7f19981540b0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f1994009900 tx=0x7f199401c8a0 comp rx=0 tx=0).stop 2024-09-17T05:10:38.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 -- 172.21.15.150:0/36089440 shutdown_connections 2024-09-17T05:10:38.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 --2- 172.21.15.150:0/36089440 >> v2:172.21.15.176:3300/0 conn(0x7f199814a180 0x7f19980c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:38.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 --2- 172.21.15.150:0/36089440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19980c7eb0 0x7f19981540b0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:38.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 -- 172.21.15.150:0/36089440 >> 172.21.15.150:0/36089440 conn(0x7f19980aab60 msgr2=0x7f19980aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:38.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 -- 172.21.15.150:0/36089440 shutdown_connections 2024-09-17T05:10:38.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 -- 172.21.15.150:0/36089440 wait complete. 2024-09-17T05:10:38.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 Processor -- start 2024-09-17T05:10:38.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 -- start start 2024-09-17T05:10:38.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19980c7eb0 0x7f19981f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:38.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1998149d70 0x7f19980ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:38.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199814a180 0x7f19981481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:38.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f19980c7c60 con 0x7f199814a180 2024-09-17T05:10:38.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f19980c7ae0 con 0x7f19980c7eb0 2024-09-17T05:10:38.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199f9a3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1998156540 con 0x7f1998149d70 2024-09-17T05:10:38.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.306+0000 7f199df40640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199814a180 0x7f19981481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:38.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199df40640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199814a180 0x7f19981481a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39592/0 (socket says 172.21.15.150:39592) 2024-09-17T05:10:38.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199df40640 1 -- 172.21.15.150:0/822454241 learned_addr learned my addr 172.21.15.150:0/822454241 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:10:38.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199cf3e640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1998149d70 0x7f19980ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:38.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199d73f640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19980c7eb0 0x7f19981f1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:38.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199df40640 1 -- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1998149d70 msgr2=0x7f19980ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:38.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199df40640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1998149d70 0x7f19980ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:38.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199df40640 1 -- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19980c7eb0 msgr2=0x7f19981f1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:38.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199df40640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19980c7eb0 0x7f19981f1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:38.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199df40640 1 -- 172.21.15.150:0/822454241 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1998150e00 con 0x7f199814a180 2024-09-17T05:10:38.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199d73f640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19980c7eb0 0x7f19981f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:10:38.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199df40640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199814a180 0x7f19981481a0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f199000b970 tx=0x7f199000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:38.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f198dffb640 1 -- 172.21.15.150:0/822454241 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1990019070 con 0x7f199814a180 2024-09-17T05:10:38.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f198dffb640 1 -- 172.21.15.150:0/822454241 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f199000cc50 con 0x7f199814a180 2024-09-17T05:10:38.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f198dffb640 1 -- 172.21.15.150:0/822454241 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19900146a0 con 0x7f199814a180 2024-09-17T05:10:38.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199f9a3640 1 -- 172.21.15.150:0/822454241 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199814dc60 con 0x7f199814a180 2024-09-17T05:10:38.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199f9a3640 1 -- 172.21.15.150:0/822454241 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199814df40 con 0x7f199814a180 2024-09-17T05:10:38.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199f9a3640 1 -- 172.21.15.150:0/822454241 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19980c7190 con 0x7f199814a180 2024-09-17T05:10:38.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f198dffb640 1 -- 172.21.15.150:0/822454241 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f199001c070 con 0x7f199814a180 2024-09-17T05:10:38.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f198dffb640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f197c073200 0x7f197c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:38.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f198dffb640 1 -- 172.21.15.150:0/822454241 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f1990094c50 con 0x7f199814a180 2024-09-17T05:10:38.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.310+0000 7f199d73f640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f197c073200 0x7f197c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:38.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.314+0000 7f199d73f640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f197c073200 0x7f197c075710 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f19940af000 tx=0x7f199401b0e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:38.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.318+0000 7f198dffb640 1 -- 172.21.15.150:0/822454241 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f199009a020 con 0x7f199814a180 2024-09-17T05:10:38.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.570+0000 7f199f9a3640 1 -- 172.21.15.150:0/822454241 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19981540c0 con 0x7f199814a180 2024-09-17T05:10:38.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.570+0000 7f198dffb640 1 -- 172.21.15.150:0/822454241 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+6544 (secure 0 0 0) 0x7f199005f070 con 0x7f199814a180 2024-09-17T05:10:38.570 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:10:38.570 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":151,"btime":"2024-09-17T05:09:57:585442+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:10:38.570 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 151 2024-09-17T05:10:38.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.574+0000 7f199f9a3640 1 -- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f197c073200 msgr2=0x7f197c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:38.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.574+0000 7f199f9a3640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f197c073200 0x7f197c075710 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f19940af000 tx=0x7f199401b0e0 comp rx=0 tx=0).stop 2024-09-17T05:10:38.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.574+0000 7f199f9a3640 1 -- 172.21.15.150:0/822454241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199814a180 msgr2=0x7f19981481a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:38.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.574+0000 7f199f9a3640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199814a180 0x7f19981481a0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f199000b970 tx=0x7f199000be30 comp rx=0 tx=0).stop 2024-09-17T05:10:38.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.574+0000 7f199f9a3640 1 -- 172.21.15.150:0/822454241 shutdown_connections 2024-09-17T05:10:38.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.574+0000 7f199f9a3640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f197c073200 0x7f197c075710 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:38.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.574+0000 7f199f9a3640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199814a180 0x7f19981481a0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:38.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.574+0000 7f199f9a3640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1998149d70 0x7f19980ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:38.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.574+0000 7f199f9a3640 1 --2- 172.21.15.150:0/822454241 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f19980c7eb0 0x7f19981f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:38.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.574+0000 7f199f9a3640 1 -- 172.21.15.150:0/822454241 >> 172.21.15.150:0/822454241 conn(0x7f19980aab60 msgr2=0x7f19981534b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:38.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.574+0000 7f199f9a3640 1 -- 172.21.15.150:0/822454241 shutdown_connections 2024-09-17T05:10:38.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:38.574+0000 7f199f9a3640 1 -- 172.21.15.150:0/822454241 wait complete. 2024-09-17T05:10:38.593 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:10:40.595 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:10:40.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.742+0000 7fc814c82640 1 -- 172.21.15.150:0/493312873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7f8004e70 msgr2=0x7fc7f80052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:40.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.742+0000 7fc814c82640 1 --2- 172.21.15.150:0/493312873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc7f8004e70 0x7fc7f80052c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fc8100c9640 tx=0x7fc80801d280 comp rx=0 tx=0).stop 2024-09-17T05:10:40.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.742+0000 7fc80ed9d640 1 -- 172.21.15.150:0/493312873 reap_dead start 2024-09-17T05:10:40.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.742+0000 7fc814c82640 1 -- 172.21.15.150:0/493312873 shutdown_connections 2024-09-17T05:10:40.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.742+0000 7fc814c82640 1 -- 172.21.15.150:0/493312873 >> 172.21.15.150:0/493312873 conn(0x7fc8100c91b0 msgr2=0x7fc8100c51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:40.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.742+0000 7fc814c82640 1 -- 172.21.15.150:0/493312873 shutdown_connections 2024-09-17T05:10:40.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.742+0000 7fc814c82640 1 -- 172.21.15.150:0/493312873 wait complete. 2024-09-17T05:10:40.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc814c82640 1 Processor -- start 2024-09-17T05:10:40.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc814c82640 1 -- start start 2024-09-17T05:10:40.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc814c82640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8100c6ae0 0x7fc810145c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:40.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc814c82640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8101480f0 0x7fc81014f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:40.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc814c82640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc810148500 0x7fc81014fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:40.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc814c82640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc8100aa110 con 0x7fc8100c6ae0 2024-09-17T05:10:40.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc814c82640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc81006e060 con 0x7fc8101480f0 2024-09-17T05:10:40.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc814c82640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc8100aa290 con 0x7fc810148500 2024-09-17T05:10:40.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc80e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8100c6ae0 0x7fc810145c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:40.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc80e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8100c6ae0 0x7fc810145c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34232/0 (socket says 172.21.15.150:34232) 2024-09-17T05:10:40.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc80e59c640 1 -- 172.21.15.150:0/3415609661 learned_addr learned my addr 172.21.15.150:0/3415609661 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:10:40.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc80dd9b640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8101480f0 0x7fc81014f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:40.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc80ed9d640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc810148500 0x7fc81014fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:40.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc80e59c640 1 -- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc810148500 msgr2=0x7fc81014fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc80e59c640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc810148500 0x7fc81014fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc80e59c640 1 -- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8101480f0 msgr2=0x7fc81014f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc80e59c640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8101480f0 0x7fc81014f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc80e59c640 1 -- 172.21.15.150:0/3415609661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc810150b30 con 0x7fc8100c6ae0 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc80e59c640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8100c6ae0 0x7fc810145c90 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fc800004e30 tx=0x7fc800001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc7feffd640 1 -- 172.21.15.150:0/3415609661 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800015020 con 0x7fc8100c6ae0 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc7feffd640 1 -- 172.21.15.150:0/3415609661 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc800011960 con 0x7fc8100c6ae0 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.746+0000 7fc7feffd640 1 -- 172.21.15.150:0/3415609661 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800011d90 con 0x7fc8100c6ae0 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.750+0000 7fc814c82640 1 -- 172.21.15.150:0/3415609661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8101461d0 con 0x7fc8100c6ae0 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.750+0000 7fc814c82640 1 -- 172.21.15.150:0/3415609661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8100c8310 con 0x7fc8100c6ae0 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.750+0000 7fc814c82640 1 -- 172.21.15.150:0/3415609661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8100c62a0 con 0x7fc8100c6ae0 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.750+0000 7fc7feffd640 1 -- 172.21.15.150:0/3415609661 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc800018020 con 0x7fc8100c6ae0 2024-09-17T05:10:40.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.750+0000 7fc7feffd640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7f0073200 0x7fc7f0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.750+0000 7fc7feffd640 1 -- 172.21.15.150:0/3415609661 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7fc8000959e0 con 0x7fc8100c6ae0 2024-09-17T05:10:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.750+0000 7fc80dd9b640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7f0073200 0x7fc7f0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:40.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.750+0000 7fc80dd9b640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7f0073200 0x7fc7f0075710 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc8101f2090 tx=0x7fc8040091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:40.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.758+0000 7fc7feffd640 1 -- 172.21.15.150:0/3415609661 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc80001f020 con 0x7fc8100c6ae0 2024-09-17T05:10:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.990+0000 7fc814c82640 1 -- 172.21.15.150:0/3415609661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8100c5ef0 con 0x7fc8100c6ae0 2024-09-17T05:10:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.990+0000 7fc7feffd640 1 -- 172.21.15.150:0/3415609661 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+6544 (secure 0 0 0) 0x7fc8101461d0 con 0x7fc8100c6ae0 2024-09-17T05:10:40.988 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:10:40.988 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":151,"btime":"2024-09-17T05:09:57:585442+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:10:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 151 2024-09-17T05:10:40.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.994+0000 7fc814c82640 1 -- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7f0073200 msgr2=0x7fc7f0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:40.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.994+0000 7fc814c82640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7f0073200 0x7fc7f0075710 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc8101f2090 tx=0x7fc8040091e0 comp rx=0 tx=0).stop 2024-09-17T05:10:40.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.994+0000 7fc814c82640 1 -- 172.21.15.150:0/3415609661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8100c6ae0 msgr2=0x7fc810145c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:40.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.994+0000 7fc814c82640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8100c6ae0 0x7fc810145c90 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fc800004e30 tx=0x7fc800001640 comp rx=0 tx=0).stop 2024-09-17T05:10:40.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.994+0000 7fc814c82640 1 -- 172.21.15.150:0/3415609661 shutdown_connections 2024-09-17T05:10:40.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.994+0000 7fc814c82640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc7f0073200 0x7fc7f0075710 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:40.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.994+0000 7fc814c82640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8101480f0 0x7fc81014f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:40.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.994+0000 7fc814c82640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc810148500 0x7fc81014fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:40.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.994+0000 7fc814c82640 1 --2- 172.21.15.150:0/3415609661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8100c6ae0 0x7fc810145c90 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:40.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.994+0000 7fc814c82640 1 -- 172.21.15.150:0/3415609661 >> 172.21.15.150:0/3415609661 conn(0x7fc8100c91b0 msgr2=0x7fc810070fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:40.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.994+0000 7fc814c82640 1 -- 172.21.15.150:0/3415609661 shutdown_connections 2024-09-17T05:10:40.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:40.994+0000 7fc814c82640 1 -- 172.21.15.150:0/3415609661 wait complete. 2024-09-17T05:10:41.009 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:10:43.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:10:43.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.185+0000 7f598a747640 1 -- 172.21.15.150:0/627293410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59840c3d10 msgr2=0x7f598415df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:43.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.185+0000 7f598a747640 1 --2- 172.21.15.150:0/627293410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59840c3d10 0x7f598415df70 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f5974009900 tx=0x7f597401c8e0 comp rx=0 tx=0).stop 2024-09-17T05:10:43.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.185+0000 7f598a747640 1 -- 172.21.15.150:0/627293410 shutdown_connections 2024-09-17T05:10:43.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.185+0000 7f598a747640 1 --2- 172.21.15.150:0/627293410 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59840c3d10 0x7f598415df70 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:43.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.185+0000 7f598a747640 1 --2- 172.21.15.150:0/627293410 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59840c4250 0x7f5984153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:43.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.185+0000 7f598a747640 1 -- 172.21.15.150:0/627293410 >> 172.21.15.150:0/627293410 conn(0x7f598406e360 msgr2=0x7f598406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:43.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.185+0000 7f598a747640 1 -- 172.21.15.150:0/627293410 shutdown_connections 2024-09-17T05:10:43.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.185+0000 7f598a747640 1 -- 172.21.15.150:0/627293410 wait complete. 2024-09-17T05:10:43.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f598a747640 1 Processor -- start 2024-09-17T05:10:43.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f598a747640 1 -- start start 2024-09-17T05:10:43.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f598a747640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59840c3900 0x7f598415dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:43.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f598a747640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59840c3d10 0x7f59840bfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:43.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f598a747640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59840c4250 0x7f598406d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:43.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f598a747640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5984071120 con 0x7f59840c3d10 2024-09-17T05:10:43.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f598a747640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5984070fa0 con 0x7f59840c3900 2024-09-17T05:10:43.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f598a747640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5984072db0 con 0x7f59840c4250 2024-09-17T05:10:43.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f59837fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59840c3d10 0x7f59840bfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:43.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f59837fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59840c3d10 0x7f59840bfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34256/0 (socket says 172.21.15.150:34256) 2024-09-17T05:10:43.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f59837fe640 1 -- 172.21.15.150:0/3045702529 learned_addr learned my addr 172.21.15.150:0/3045702529 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:10:43.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f5983fff640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59840c3900 0x7f598415dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:43.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.189+0000 7f5988ce4640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59840c4250 0x7f598406d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:43.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f59837fe640 1 -- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59840c4250 msgr2=0x7f598406d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:43.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f59837fe640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59840c4250 0x7f598406d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:43.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f59837fe640 1 -- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59840c3900 msgr2=0x7f598415dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:43.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f59837fe640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59840c3900 0x7f598415dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:43.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f59837fe640 1 -- 172.21.15.150:0/3045702529 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59840ad7d0 con 0x7f59840c3d10 2024-09-17T05:10:43.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f5988ce4640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59840c4250 0x7f598406d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:10:43.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f5983fff640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59840c3900 0x7f598415dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:10:43.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f59837fe640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59840c3d10 0x7f59840bfbe0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f597800c3f0 tx=0x7f597800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:43.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f5980ff9640 1 -- 172.21.15.150:0/3045702529 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5978017070 con 0x7f59840c3d10 2024-09-17T05:10:43.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f5980ff9640 1 -- 172.21.15.150:0/3045702529 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59780053c0 con 0x7f59840c3d10 2024-09-17T05:10:43.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f598a747640 1 -- 172.21.15.150:0/3045702529 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59840bdd60 con 0x7f59840c3d10 2024-09-17T05:10:43.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f5980ff9640 1 -- 172.21.15.150:0/3045702529 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5978003860 con 0x7f59840c3d10 2024-09-17T05:10:43.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f598a747640 1 -- 172.21.15.150:0/3045702529 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59840ad4f0 con 0x7f59840c3d10 2024-09-17T05:10:43.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f598a747640 1 -- 172.21.15.150:0/3045702529 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5984153ee0 con 0x7f59840c3d10 2024-09-17T05:10:43.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f5980ff9640 1 -- 172.21.15.150:0/3045702529 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f597801a070 con 0x7f59840c3d10 2024-09-17T05:10:43.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f5980ff9640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5970073200 0x7f5970075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:43.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f5980ff9640 1 -- 172.21.15.150:0/3045702529 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f5978094290 con 0x7f59840c3d10 2024-09-17T05:10:43.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.193+0000 7f5983fff640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5970073200 0x7f5970075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:43.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.197+0000 7f5983fff640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5970073200 0x7f5970075710 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f597c009f70 tx=0x7f597c009500 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:43.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.201+0000 7f5980ff9640 1 -- 172.21.15.150:0/3045702529 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f597805e630 con 0x7f59840c3d10 2024-09-17T05:10:43.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.469+0000 7f598a747640 1 -- 172.21.15.150:0/3045702529 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59840bdf60 con 0x7f59840c3d10 2024-09-17T05:10:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.469+0000 7f5980ff9640 1 -- 172.21.15.150:0/3045702529 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+6544 (secure 0 0 0) 0x7f597805df30 con 0x7f59840c3d10 2024-09-17T05:10:43.468 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:10:43.468 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":151,"btime":"2024-09-17T05:09:57:585442+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:10:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 151 2024-09-17T05:10:43.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.469+0000 7f598a747640 1 -- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5970073200 msgr2=0x7f5970075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.469+0000 7f598a747640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5970073200 0x7f5970075710 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f597c009f70 tx=0x7f597c009500 comp rx=0 tx=0).stop 2024-09-17T05:10:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.469+0000 7f598a747640 1 -- 172.21.15.150:0/3045702529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59840c3d10 msgr2=0x7f59840bfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.469+0000 7f598a747640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59840c3d10 0x7f59840bfbe0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f597800c3f0 tx=0x7f597800c8b0 comp rx=0 tx=0).stop 2024-09-17T05:10:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.473+0000 7f598a747640 1 -- 172.21.15.150:0/3045702529 shutdown_connections 2024-09-17T05:10:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.473+0000 7f598a747640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5970073200 0x7f5970075710 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.473+0000 7f598a747640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f59840c4250 0x7f598406d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.473+0000 7f598a747640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59840c3d10 0x7f59840bfbe0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.473+0000 7f598a747640 1 --2- 172.21.15.150:0/3045702529 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59840c3900 0x7f598415dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.473+0000 7f598a747640 1 -- 172.21.15.150:0/3045702529 >> 172.21.15.150:0/3045702529 conn(0x7f598406e360 msgr2=0x7f5984152de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.473+0000 7f598a747640 1 -- 172.21.15.150:0/3045702529 shutdown_connections 2024-09-17T05:10:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:43.473+0000 7f598a747640 1 -- 172.21.15.150:0/3045702529 wait complete. 2024-09-17T05:10:43.495 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:10:45.495 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:10:45.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.673+0000 7f3c1df25640 1 -- 172.21.15.150:0/3364579481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c18149d70 msgr2=0x7f3c1814a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:45.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.673+0000 7f3c1df25640 1 --2- 172.21.15.150:0/3364579481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c18149d70 0x7f3c1814a180 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009840 tx=0x7f3c0c01c7e0 comp rx=0 tx=0).stop 2024-09-17T05:10:45.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.673+0000 7f3c1df25640 1 -- 172.21.15.150:0/3364579481 shutdown_connections 2024-09-17T05:10:45.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.673+0000 7f3c1df25640 1 --2- 172.21.15.150:0/3364579481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c18149d70 0x7f3c1814a180 secure :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009840 tx=0x7f3c0c01c7e0 comp rx=0 tx=0).stop 2024-09-17T05:10:45.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.673+0000 7f3c1df25640 1 --2- 172.21.15.150:0/3364579481 >> v2:172.21.15.176:3300/0 conn(0x7f3c180c7eb0 0x7f3c181540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:45.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.673+0000 7f3c1df25640 1 -- 172.21.15.150:0/3364579481 >> 172.21.15.150:0/3364579481 conn(0x7f3c180aab60 msgr2=0x7f3c180aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:45.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.673+0000 7f3c1df25640 1 -- 172.21.15.150:0/3364579481 shutdown_connections 2024-09-17T05:10:45.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.673+0000 7f3c1df25640 1 -- 172.21.15.150:0/3364579481 wait complete. 2024-09-17T05:10:45.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c1df25640 1 Processor -- start 2024-09-17T05:10:45.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c1df25640 1 -- start start 2024-09-17T05:10:45.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c1df25640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c180c7eb0 0x7f3c18152980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:45.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c1df25640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c180c8330 0x7f3c180c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:45.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c1df25640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c181410c0 0x7f3c181414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:45.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c1df25640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3c181568d0 con 0x7f3c181410c0 2024-09-17T05:10:45.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c1df25640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3c18156750 con 0x7f3c180c8330 2024-09-17T05:10:45.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c1df25640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3c181458d0 con 0x7f3c180c7eb0 2024-09-17T05:10:45.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c17fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c181410c0 0x7f3c181414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:45.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c17fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c181410c0 0x7f3c181414d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34270/0 (socket says 172.21.15.150:34270) 2024-09-17T05:10:45.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c17fff640 1 -- 172.21.15.150:0/2605602396 learned_addr learned my addr 172.21.15.150:0/2605602396 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:10:45.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c16ffd640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c180c8330 0x7f3c180c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:45.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c17fff640 1 -- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c180c7eb0 msgr2=0x7f3c18152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:45.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c17fff640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c180c7eb0 0x7f3c18152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:45.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c17fff640 1 -- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c180c8330 msgr2=0x7f3c180c3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:45.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c17fff640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c180c8330 0x7f3c180c3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:45.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c17fff640 1 -- 172.21.15.150:0/2605602396 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c18152ec0 con 0x7f3c181410c0 2024-09-17T05:10:45.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.677+0000 7f3c17fff640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c181410c0 0x7f3c181414d0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000cb40 tx=0x7f3c1000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:45.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.681+0000 7f3bfffff640 1 -- 172.21.15.150:0/2605602396 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c10016020 con 0x7f3c181410c0 2024-09-17T05:10:45.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.681+0000 7f3bfffff640 1 -- 172.21.15.150:0/2605602396 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c1000ab60 con 0x7f3c181410c0 2024-09-17T05:10:45.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.681+0000 7f3c1df25640 1 -- 172.21.15.150:0/2605602396 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c180ae550 con 0x7f3c181410c0 2024-09-17T05:10:45.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.681+0000 7f3bfffff640 1 -- 172.21.15.150:0/2605602396 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c10012420 con 0x7f3c181410c0 2024-09-17T05:10:45.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.681+0000 7f3c1df25640 1 -- 172.21.15.150:0/2605602396 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c180c4c90 con 0x7f3c181410c0 2024-09-17T05:10:45.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.681+0000 7f3c1df25640 1 -- 172.21.15.150:0/2605602396 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c180c49f0 con 0x7f3c181410c0 2024-09-17T05:10:45.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.681+0000 7f3bfffff640 1 -- 172.21.15.150:0/2605602396 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3c10019020 con 0x7f3c181410c0 2024-09-17T05:10:45.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.681+0000 7f3bfffff640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c04073200 0x7f3c04075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:45.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.681+0000 7f3bfffff640 1 -- 172.21.15.150:0/2605602396 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f3c100951a0 con 0x7f3c181410c0 2024-09-17T05:10:45.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.681+0000 7f3c177fe640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c04073200 0x7f3c04075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:45.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.685+0000 7f3c177fe640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c04073200 0x7f3c04075710 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3c080058d0 tx=0x7f3c08005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:45.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.689+0000 7f3bfffff640 1 -- 172.21.15.150:0/2605602396 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3c180c49f0 con 0x7f3c181410c0 2024-09-17T05:10:45.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.905+0000 7f3c1df25640 1 -- 172.21.15.150:0/2605602396 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c1814a180 con 0x7f3c181410c0 2024-09-17T05:10:45.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3bfffff640 1 -- 172.21.15.150:0/2605602396 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+6544 (secure 0 0 0) 0x7f3c10020020 con 0x7f3c181410c0 2024-09-17T05:10:45.907 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:10:45.907 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":151,"btime":"2024-09-17T05:09:57:585442+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:10:45.907 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 151 2024-09-17T05:10:45.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3c1df25640 1 -- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c04073200 msgr2=0x7f3c04075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:45.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3c1df25640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c04073200 0x7f3c04075710 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3c080058d0 tx=0x7f3c08005860 comp rx=0 tx=0).stop 2024-09-17T05:10:45.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3c1df25640 1 -- 172.21.15.150:0/2605602396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c181410c0 msgr2=0x7f3c181414d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:45.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3c1df25640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c181410c0 0x7f3c181414d0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000cb40 tx=0x7f3c1000cc30 comp rx=0 tx=0).stop 2024-09-17T05:10:45.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3c1df25640 1 -- 172.21.15.150:0/2605602396 shutdown_connections 2024-09-17T05:10:45.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3c1df25640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c04073200 0x7f3c04075710 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:45.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3c1df25640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c180c8330 0x7f3c180c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:45.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3c1df25640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c181410c0 0x7f3c181414d0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:45.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3c1df25640 1 --2- 172.21.15.150:0/2605602396 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c180c7eb0 0x7f3c18152980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:45.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3c1df25640 1 -- 172.21.15.150:0/2605602396 >> 172.21.15.150:0/2605602396 conn(0x7f3c180aab60 msgr2=0x7f3c180aa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:45.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3c1df25640 1 -- 172.21.15.150:0/2605602396 shutdown_connections 2024-09-17T05:10:45.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:45.909+0000 7f3c1df25640 1 -- 172.21.15.150:0/2605602396 wait complete. 2024-09-17T05:10:45.929 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:10:47.930 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:10:48.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.105+0000 7fa78daf3640 1 -- 172.21.15.150:0/2646576416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa770003a10 msgr2=0x7fa770003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:48.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.105+0000 7fa78daf3640 1 --2- 172.21.15.150:0/2646576416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa770003a10 0x7fa770003e80 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fa77800bb10 tx=0x7fa77801cdc0 comp rx=0 tx=0).stop 2024-09-17T05:10:48.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.105+0000 7fa787fff640 1 -- 172.21.15.150:0/2646576416 reap_dead start 2024-09-17T05:10:48.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.105+0000 7fa78daf3640 1 -- 172.21.15.150:0/2646576416 shutdown_connections 2024-09-17T05:10:48.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.105+0000 7fa78daf3640 1 -- 172.21.15.150:0/2646576416 >> 172.21.15.150:0/2646576416 conn(0x7fa788074750 msgr2=0x7fa788074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:48.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.105+0000 7fa78daf3640 1 -- 172.21.15.150:0/2646576416 shutdown_connections 2024-09-17T05:10:48.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.105+0000 7fa78daf3640 1 -- 172.21.15.150:0/2646576416 wait complete. 2024-09-17T05:10:48.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.109+0000 7fa78daf3640 1 Processor -- start 2024-09-17T05:10:48.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.109+0000 7fa78daf3640 1 -- start start 2024-09-17T05:10:48.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa78daf3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa770003a10 0x7fa7880cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:48.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa78daf3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7880cc850 0x7fa7881f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:48.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa78daf3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa78814b280 0x7fa78814b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:48.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa78daf3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa788154450 con 0x7fa770003a10 2024-09-17T05:10:48.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa78daf3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa7881542d0 con 0x7fa7880cc850 2024-09-17T05:10:48.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa78daf3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa788161970 con 0x7fa78814b280 2024-09-17T05:10:48.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa7877fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa770003a10 0x7fa7880cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:48.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa7877fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa770003a10 0x7fa7880cad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34282/0 (socket says 172.21.15.150:34282) 2024-09-17T05:10:48.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa7877fe640 1 -- 172.21.15.150:0/555425883 learned_addr learned my addr 172.21.15.150:0/555425883 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:10:48.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa786ffd640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7880cc850 0x7fa7881f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:48.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa7877fe640 1 -- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa78814b280 msgr2=0x7fa78814b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:48.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa7877fe640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa78814b280 0x7fa78814b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:48.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa7877fe640 1 -- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7880cc850 msgr2=0x7fa7881f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:48.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa7877fe640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7880cc850 0x7fa7881f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:48.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa7877fe640 1 -- 172.21.15.150:0/555425883 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7880cbb80 con 0x7fa770003a10 2024-09-17T05:10:48.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa7877fe640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa770003a10 0x7fa7880cad20 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fa77801cbf0 tx=0x7fa778005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:48.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa76ffff640 1 -- 172.21.15.150:0/555425883 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7780ab040 con 0x7fa770003a10 2024-09-17T05:10:48.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa76ffff640 1 -- 172.21.15.150:0/555425883 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7780a5ac0 con 0x7fa770003a10 2024-09-17T05:10:48.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa76ffff640 1 -- 172.21.15.150:0/555425883 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa778004280 con 0x7fa770003a10 2024-09-17T05:10:48.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa78daf3640 1 -- 172.21.15.150:0/555425883 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7880cb260 con 0x7fa770003a10 2024-09-17T05:10:48.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa78daf3640 1 -- 172.21.15.150:0/555425883 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa78815fba0 con 0x7fa770003a10 2024-09-17T05:10:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa78daf3640 1 -- 172.21.15.150:0/555425883 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7880c1300 con 0x7fa770003a10 2024-09-17T05:10:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa76ffff640 1 -- 172.21.15.150:0/555425883 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa7780c0020 con 0x7fa770003a10 2024-09-17T05:10:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa76ffff640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa774073250 0x7fa774075760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.113+0000 7fa76ffff640 1 -- 172.21.15.150:0/555425883 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7fa77812f6e0 con 0x7fa770003a10 2024-09-17T05:10:48.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.117+0000 7fa786ffd640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa774073250 0x7fa774075760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:48.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.117+0000 7fa786ffd640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa774073250 0x7fa774075760 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa77c005ec0 tx=0x7fa77c0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:48.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.121+0000 7fa76ffff640 1 -- 172.21.15.150:0/555425883 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa778016ad0 con 0x7fa770003a10 2024-09-17T05:10:48.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa78daf3640 1 -- 172.21.15.150:0/555425883 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7880b1b70 con 0x7fa770003a10 2024-09-17T05:10:48.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa76ffff640 1 -- 172.21.15.150:0/555425883 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+6544 (secure 0 0 0) 0x7fa7780a5c90 con 0x7fa770003a10 2024-09-17T05:10:48.360 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:10:48.360 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":151,"btime":"2024-09-17T05:09:57:585442+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:10:48.360 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 151 2024-09-17T05:10:48.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa78daf3640 1 -- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa774073250 msgr2=0x7fa774075760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:48.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa78daf3640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa774073250 0x7fa774075760 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa77c005ec0 tx=0x7fa77c0091e0 comp rx=0 tx=0).stop 2024-09-17T05:10:48.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa78daf3640 1 -- 172.21.15.150:0/555425883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa770003a10 msgr2=0x7fa7880cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:48.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa78daf3640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa770003a10 0x7fa7880cad20 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fa77801cbf0 tx=0x7fa778005640 comp rx=0 tx=0).stop 2024-09-17T05:10:48.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa78daf3640 1 -- 172.21.15.150:0/555425883 shutdown_connections 2024-09-17T05:10:48.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa78daf3640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa774073250 0x7fa774075760 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:48.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa78daf3640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa7880cc850 0x7fa7881f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:48.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa78daf3640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa78814b280 0x7fa78814b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:48.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa78daf3640 1 --2- 172.21.15.150:0/555425883 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa770003a10 0x7fa7880cad20 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:48.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa78daf3640 1 -- 172.21.15.150:0/555425883 >> 172.21.15.150:0/555425883 conn(0x7fa788074750 msgr2=0x7fa788074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:48.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.361+0000 7fa78daf3640 1 -- 172.21.15.150:0/555425883 shutdown_connections 2024-09-17T05:10:48.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:48.365+0000 7fa78daf3640 1 -- 172.21.15.150:0/555425883 wait complete. 2024-09-17T05:10:48.386 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:10:50.386 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:10:50.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.557+0000 7f5c0bedc640 1 -- 172.21.15.150:0/2234672191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0004810 msgr2=0x7f5bf0004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:50.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.557+0000 7f5c0bedc640 1 --2- 172.21.15.150:0/2234672191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0004810 0x7f5bf0004c20 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009b60 tx=0x7f5bf401d2b0 comp rx=0 tx=0).stop 2024-09-17T05:10:50.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.557+0000 7f5c0a479640 1 -- 172.21.15.150:0/2234672191 reap_dead start 2024-09-17T05:10:50.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.557+0000 7f5c0bedc640 1 -- 172.21.15.150:0/2234672191 shutdown_connections 2024-09-17T05:10:50.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.557+0000 7f5c0bedc640 1 -- 172.21.15.150:0/2234672191 >> 172.21.15.150:0/2234672191 conn(0x7f5c040aab60 msgr2=0x7f5c040aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:50.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.557+0000 7f5c0bedc640 1 -- 172.21.15.150:0/2234672191 shutdown_connections 2024-09-17T05:10:50.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.557+0000 7f5c0bedc640 1 -- 172.21.15.150:0/2234672191 wait complete. 2024-09-17T05:10:50.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c0bedc640 1 Processor -- start 2024-09-17T05:10:50.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c0bedc640 1 -- start start 2024-09-17T05:10:50.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c0bedc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0004810 0x7f5c0414dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:50.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c0bedc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c040ae450 0x7f5c0414eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:50.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c0bedc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c040ae860 0x7f5c04151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:50.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c0bedc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5c040c78a0 con 0x7f5bf0004810 2024-09-17T05:10:50.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c0bedc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5c04072500 con 0x7f5c040ae450 2024-09-17T05:10:50.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c0bedc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5c040c7a20 con 0x7f5c040ae860 2024-09-17T05:10:50.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c09c78640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0004810 0x7f5c0414dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:50.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c09c78640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0004810 0x7f5c0414dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49686/0 (socket says 172.21.15.150:49686) 2024-09-17T05:10:50.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c09c78640 1 -- 172.21.15.150:0/4048244176 learned_addr learned my addr 172.21.15.150:0/4048244176 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:10:50.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c0a479640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c040ae860 0x7f5c04151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:50.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c09477640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c040ae450 0x7f5c0414eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:50.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c09c78640 1 -- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c040ae860 msgr2=0x7f5c04151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:50.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c09c78640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c040ae860 0x7f5c04151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:50.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c09c78640 1 -- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c040ae450 msgr2=0x7f5c0414eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:50.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c09c78640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c040ae450 0x7f5c0414eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:50.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c09c78640 1 -- 172.21.15.150:0/4048244176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c04149820 con 0x7f5bf0004810 2024-09-17T05:10:50.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.561+0000 7f5c09c78640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0004810 0x7f5c0414dd60 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f5c00011a80 tx=0x7f5c00011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:50.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.565+0000 7f5bfa7fc640 1 -- 172.21.15.150:0/4048244176 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000fdb0 con 0x7f5bf0004810 2024-09-17T05:10:50.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.565+0000 7f5bfa7fc640 1 -- 172.21.15.150:0/4048244176 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c00016070 con 0x7f5bf0004810 2024-09-17T05:10:50.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.565+0000 7f5c0bedc640 1 -- 172.21.15.150:0/4048244176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c041493d0 con 0x7f5bf0004810 2024-09-17T05:10:50.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.565+0000 7f5bfa7fc640 1 -- 172.21.15.150:0/4048244176 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00012690 con 0x7f5bf0004810 2024-09-17T05:10:50.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.565+0000 7f5c0bedc640 1 -- 172.21.15.150:0/4048244176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c041f0010 con 0x7f5bf0004810 2024-09-17T05:10:50.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.565+0000 7f5c0bedc640 1 -- 172.21.15.150:0/4048244176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c040c2c70 con 0x7f5bf0004810 2024-09-17T05:10:50.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.565+0000 7f5bfa7fc640 1 -- 172.21.15.150:0/4048244176 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5c00018030 con 0x7f5bf0004810 2024-09-17T05:10:50.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.565+0000 7f5bfa7fc640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5be8073200 0x7f5be8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:50.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.565+0000 7f5bfa7fc640 1 -- 172.21.15.150:0/4048244176 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f5c00095f60 con 0x7f5bf0004810 2024-09-17T05:10:50.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.565+0000 7f5c09477640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5be8073200 0x7f5be8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:50.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.565+0000 7f5c09477640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5be8073200 0x7f5be8075710 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009b60 tx=0x7f5bf401a050 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:50.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.569+0000 7f5bfa7fc640 1 -- 172.21.15.150:0/4048244176 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5c00060370 con 0x7f5bf0004810 2024-09-17T05:10:50.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 -- 172.21.15.150:0/4048244176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c040c8820 con 0x7f5bf0004810 2024-09-17T05:10:50.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5bfa7fc640 1 -- 172.21.15.150:0/4048244176 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+6544 (secure 0 0 0) 0x7f5c0005fc70 con 0x7f5bf0004810 2024-09-17T05:10:50.827 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:10:50.828 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":151,"btime":"2024-09-17T05:09:57:585442+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":6799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":151}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:09:09.137150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7158},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7158":{"gid":7158,"name":"c","rank":0,"incarnation":142,"state":"up:active","state_seq":14,"addr":"172.21.15.150:6835/285376980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":285376980},{"type":"v1","addr":"172.21.15.150:6835","nonce":285376980}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7158,"qdb_cluster":[7158]},"id":2}]} 2024-09-17T05:10:50.828 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 151 2024-09-17T05:10:50.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 -- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5be8073200 msgr2=0x7f5be8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:50.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5be8073200 0x7f5be8075710 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009b60 tx=0x7f5bf401a050 comp rx=0 tx=0).stop 2024-09-17T05:10:50.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 -- 172.21.15.150:0/4048244176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0004810 msgr2=0x7f5c0414dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:50.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0004810 0x7f5c0414dd60 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f5c00011a80 tx=0x7f5c00011f40 comp rx=0 tx=0).stop 2024-09-17T05:10:50.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 -- 172.21.15.150:0/4048244176 shutdown_connections 2024-09-17T05:10:50.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5be8073200 0x7f5be8075710 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:50.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5c040ae450 0x7f5c0414eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:50.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5c040ae860 0x7f5c04151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:50.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 --2- 172.21.15.150:0/4048244176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bf0004810 0x7f5c0414dd60 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:50.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 -- 172.21.15.150:0/4048244176 >> 172.21.15.150:0/4048244176 conn(0x7f5c040aab60 msgr2=0x7f5c04153650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:50.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 -- 172.21.15.150:0/4048244176 shutdown_connections 2024-09-17T05:10:50.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:50.829+0000 7f5c0bedc640 1 -- 172.21.15.150:0/4048244176 wait complete. 2024-09-17T05:10:50.851 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:10:52.852 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:10:53.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 -- 172.21.15.150:0/2957268943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb738004810 msgr2=0x7fb738004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:53.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 --2- 172.21.15.150:0/2957268943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb738004810 0x7fb738004c20 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0016f0 tx=0x7fb73c01c7b0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb752a49640 1 -- 172.21.15.150:0/2957268943 reap_dead start 2024-09-17T05:10:53.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 -- 172.21.15.150:0/2957268943 shutdown_connections 2024-09-17T05:10:53.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 -- 172.21.15.150:0/2957268943 >> 172.21.15.150:0/2957268943 conn(0x7fb74c0ca460 msgr2=0x7fb74c0c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:53.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 -- 172.21.15.150:0/2957268943 shutdown_connections 2024-09-17T05:10:53.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 -- 172.21.15.150:0/2957268943 wait complete. 2024-09-17T05:10:53.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 Processor -- start 2024-09-17T05:10:53.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 -- start start 2024-09-17T05:10:53.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb738004810 0x7fb74c0aa600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:53.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb74c1de7d0 0x7fb74c0aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:53.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb74c1debe0 0x7fb74c1eb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:53.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c0c62d0 con 0x7fb74c1de7d0 2024-09-17T05:10:53.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c0c51f0 con 0x7fb738004810 2024-09-17T05:10:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb7544ac640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb74c0c65c0 con 0x7fb74c1debe0 2024-09-17T05:10:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb751a47640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb74c1de7d0 0x7fb74c0aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb751a47640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb74c1de7d0 0x7fb74c0aaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49696/0 (socket says 172.21.15.150:49696) 2024-09-17T05:10:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.033+0000 7fb751a47640 1 -- 172.21.15.150:0/2011820948 learned_addr learned my addr 172.21.15.150:0/2011820948 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:10:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb752248640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb738004810 0x7fb74c0aa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb752a49640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb74c1debe0 0x7fb74c1eb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb751a47640 1 -- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb74c1debe0 msgr2=0x7fb74c1eb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb751a47640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb74c1debe0 0x7fb74c1eb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb751a47640 1 -- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb738004810 msgr2=0x7fb74c0aa600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:53.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb751a47640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb738004810 0x7fb74c0aa600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb751a47640 1 -- 172.21.15.150:0/2011820948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74c155b30 con 0x7fb74c1de7d0 2024-09-17T05:10:53.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb751a47640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb74c1de7d0 0x7fb74c0aaea0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fb73c01aeb0 tx=0x7fb73c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:53.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb742ffd640 1 -- 172.21.15.150:0/2011820948 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c001310 con 0x7fb74c1de7d0 2024-09-17T05:10:53.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb742ffd640 1 -- 172.21.15.150:0/2011820948 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb73c006500 con 0x7fb74c1de7d0 2024-09-17T05:10:53.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb742ffd640 1 -- 172.21.15.150:0/2011820948 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c006930 con 0x7fb74c1de7d0 2024-09-17T05:10:53.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb7544ac640 1 -- 172.21.15.150:0/2011820948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb74c155dd0 con 0x7fb74c1de7d0 2024-09-17T05:10:53.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb7544ac640 1 -- 172.21.15.150:0/2011820948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb74c156390 con 0x7fb74c1de7d0 2024-09-17T05:10:53.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb7544ac640 1 -- 172.21.15.150:0/2011820948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb74c072210 con 0x7fb74c1de7d0 2024-09-17T05:10:53.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb742ffd640 1 -- 172.21.15.150:0/2011820948 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb73c0a6720 con 0x7fb74c1de7d0 2024-09-17T05:10:53.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb742ffd640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb730073200 0x7fb730075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:53.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb742ffd640 1 -- 172.21.15.150:0/2011820948 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7fb73c12f140 con 0x7fb74c1de7d0 2024-09-17T05:10:53.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.037+0000 7fb752248640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb730073200 0x7fb730075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:53.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.041+0000 7fb752248640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb730073200 0x7fb730075710 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb748001250 tx=0x7fb7480014c0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:53.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.045+0000 7fb742ffd640 1 -- 172.21.15.150:0/2011820948 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb73c0aa050 con 0x7fb74c1de7d0 2024-09-17T05:10:53.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.325+0000 7fb7544ac640 1 -- 172.21.15.150:0/2011820948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb74c1df520 con 0x7fb74c1de7d0 2024-09-17T05:10:53.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.325+0000 7fb742ffd640 1 -- 172.21.15.150:0/2011820948 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5711 (secure 0 0 0) 0x7fb73c016510 con 0x7fb74c1de7d0 2024-09-17T05:10:53.326 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:10:53.326 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":154,"btime":"2024-09-17T05:10:53:165908+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:52.171463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:10:53.327 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 154 2024-09-17T05:10:53.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.329+0000 7fb7544ac640 1 -- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb730073200 msgr2=0x7fb730075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.329+0000 7fb7544ac640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb730073200 0x7fb730075710 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb748001250 tx=0x7fb7480014c0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.329+0000 7fb7544ac640 1 -- 172.21.15.150:0/2011820948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb74c1de7d0 msgr2=0x7fb74c0aaea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.329+0000 7fb7544ac640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb74c1de7d0 0x7fb74c0aaea0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fb73c01aeb0 tx=0x7fb73c003b90 comp rx=0 tx=0).stop 2024-09-17T05:10:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.329+0000 7fb7544ac640 1 -- 172.21.15.150:0/2011820948 shutdown_connections 2024-09-17T05:10:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.329+0000 7fb7544ac640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb730073200 0x7fb730075710 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.329+0000 7fb7544ac640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb74c1debe0 0x7fb74c1eb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.329+0000 7fb7544ac640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb74c1de7d0 0x7fb74c0aaea0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.329+0000 7fb7544ac640 1 --2- 172.21.15.150:0/2011820948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb738004810 0x7fb74c0aa600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.329+0000 7fb7544ac640 1 -- 172.21.15.150:0/2011820948 >> 172.21.15.150:0/2011820948 conn(0x7fb74c0ca460 msgr2=0x7fb74c0c6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.329+0000 7fb7544ac640 1 -- 172.21.15.150:0/2011820948 shutdown_connections 2024-09-17T05:10:53.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.329+0000 7fb7544ac640 1 -- 172.21.15.150:0/2011820948 wait complete. 2024-09-17T05:10:53.350 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-09-17T05:10:53.350 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:10:53.350 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:10:53.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.509+0000 7f6c7e8de640 1 -- 172.21.15.150:0/1938134377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c780c6cb0 msgr2=0x7f6c780ae340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:53.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.509+0000 7f6c7e8de640 1 --2- 172.21.15.150:0/1938134377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c780c6cb0 0x7f6c780ae340 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009900 tx=0x7f6c6801d2b0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.509+0000 7f6c7e8de640 1 -- 172.21.15.150:0/1938134377 shutdown_connections 2024-09-17T05:10:53.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.509+0000 7f6c7e8de640 1 --2- 172.21.15.150:0/1938134377 >> v2:172.21.15.176:3300/0 conn(0x7f6c78151980 0x7f6c78151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.509+0000 7f6c7e8de640 1 --2- 172.21.15.150:0/1938134377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c780c6cb0 0x7f6c780ae340 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.509+0000 7f6c7e8de640 1 -- 172.21.15.150:0/1938134377 >> 172.21.15.150:0/1938134377 conn(0x7f6c780aab60 msgr2=0x7f6c780aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:53.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.509+0000 7f6c7e8de640 1 -- 172.21.15.150:0/1938134377 shutdown_connections 2024-09-17T05:10:53.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.509+0000 7f6c7e8de640 1 -- 172.21.15.150:0/1938134377 wait complete. 2024-09-17T05:10:53.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c7e8de640 1 Processor -- start 2024-09-17T05:10:53.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c7e8de640 1 -- start start 2024-09-17T05:10:53.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c7e8de640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c780a85b0 0x7f6c7814b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c7e8de640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c6cb0 0x7f6c781f13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c7e8de640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c78151980 0x7f6c78150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c7e8de640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6c78155170 con 0x7f6c780c6cb0 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c7e8de640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6c78152380 con 0x7f6c780a85b0 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c7e8de640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6c781552f0 con 0x7f6c78151980 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c777fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c6cb0 0x7f6c781f13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c777fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c6cb0 0x7f6c781f13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49698/0 (socket says 172.21.15.150:49698) 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c777fe640 1 -- 172.21.15.150:0/3890983721 learned_addr learned my addr 172.21.15.150:0/3890983721 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c77fff640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c780a85b0 0x7f6c7814b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c7ce7b640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c78151980 0x7f6c78150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c777fe640 1 -- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c78151980 msgr2=0x7f6c78150a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c777fe640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c78151980 0x7f6c78150a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c777fe640 1 -- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c780a85b0 msgr2=0x7f6c7814b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:53.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c777fe640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c780a85b0 0x7f6c7814b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c777fe640 1 -- 172.21.15.150:0/3890983721 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c78144c30 con 0x7f6c780c6cb0 2024-09-17T05:10:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c77fff640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c780a85b0 0x7f6c7814b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:10:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c7ce7b640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c78151980 0x7f6c78150a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:10:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.513+0000 7f6c777fe640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c6cb0 0x7f6c781f13e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00c390 tx=0x7f6c6c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.517+0000 7f6c74ff9640 1 -- 172.21.15.150:0/3890983721 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c017070 con 0x7f6c780c6cb0 2024-09-17T05:10:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.517+0000 7f6c74ff9640 1 -- 172.21.15.150:0/3890983721 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c6c0053c0 con 0x7f6c780c6cb0 2024-09-17T05:10:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.517+0000 7f6c74ff9640 1 -- 172.21.15.150:0/3890983721 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c003840 con 0x7f6c780c6cb0 2024-09-17T05:10:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.517+0000 7f6c7e8de640 1 -- 172.21.15.150:0/3890983721 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c78144f10 con 0x7f6c780c6cb0 2024-09-17T05:10:53.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.517+0000 7f6c7e8de640 1 -- 172.21.15.150:0/3890983721 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c78153860 con 0x7f6c780c6cb0 2024-09-17T05:10:53.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.517+0000 7f6c5e7fc640 1 -- 172.21.15.150:0/3890983721 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c500052a0 con 0x7f6c780c6cb0 2024-09-17T05:10:53.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.517+0000 7f6c74ff9640 1 -- 172.21.15.150:0/3890983721 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6c6c01a070 con 0x7f6c780c6cb0 2024-09-17T05:10:53.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.517+0000 7f6c74ff9640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6c64073230 0x7f6c64075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:53.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.517+0000 7f6c74ff9640 1 -- 172.21.15.150:0/3890983721 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f6c6c0944e0 con 0x7f6c780c6cb0 2024-09-17T05:10:53.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.517+0000 7f6c77fff640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6c64073230 0x7f6c64075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:53.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.521+0000 7f6c77fff640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6c64073230 0x7f6c64075740 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6c680af000 tx=0x7f6c6801b090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:53.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.525+0000 7f6c74ff9640 1 -- 172.21.15.150:0/3890983721 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6c6c021080 con 0x7f6c780c6cb0 2024-09-17T05:10:53.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.837+0000 7f6c5e7fc640 1 -- 172.21.15.150:0/3890983721 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c50005590 con 0x7f6c780c6cb0 2024-09-17T05:10:53.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c74ff9640 1 -- 172.21.15.150:0/3890983721 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5711 (secure 0 0 0) 0x7f6c6c05e780 con 0x7f6c780c6cb0 2024-09-17T05:10:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:10:53.839 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":154,"btime":"2024-09-17T05:10:53:165908+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:52.171463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:10:53.839 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 154 2024-09-17T05:10:53.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c5e7fc640 1 -- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6c64073230 msgr2=0x7f6c64075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:53.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c5e7fc640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6c64073230 0x7f6c64075740 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6c680af000 tx=0x7f6c6801b090 comp rx=0 tx=0).stop 2024-09-17T05:10:53.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c5e7fc640 1 -- 172.21.15.150:0/3890983721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c6cb0 msgr2=0x7f6c781f13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:53.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c5e7fc640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c6cb0 0x7f6c781f13e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00c390 tx=0x7f6c6c00c850 comp rx=0 tx=0).stop 2024-09-17T05:10:53.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c5e7fc640 1 -- 172.21.15.150:0/3890983721 shutdown_connections 2024-09-17T05:10:53.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c5e7fc640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6c64073230 0x7f6c64075740 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c5e7fc640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6c78151980 0x7f6c78150a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c5e7fc640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c780c6cb0 0x7f6c781f13e0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c5e7fc640 1 --2- 172.21.15.150:0/3890983721 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c780a85b0 0x7f6c7814b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:53.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c5e7fc640 1 -- 172.21.15.150:0/3890983721 >> 172.21.15.150:0/3890983721 conn(0x7f6c780aab60 msgr2=0x7f6c780c5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:53.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c5e7fc640 1 -- 172.21.15.150:0/3890983721 shutdown_connections 2024-09-17T05:10:53.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:53.841+0000 7f6c5e7fc640 1 -- 172.21.15.150:0/3890983721 wait complete. 2024-09-17T05:10:53.866 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:10:53:165908+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 154, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 150, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7230": { "addr": "172.21.15.150:6833/2056467678", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2056467678, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 2056467678, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7230, "incarnation": 147, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:09:49.306533+0000", "qdb_cluster": [ 7230 ], "qdb_leader": 7230, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7230 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6799": { "addr": "172.21.15.176:6837/2962113435", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 2962113435, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 2962113435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6799, "incarnation": 152, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:10:52.171463+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6799 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:10:55.867 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:10:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.057+0000 7fbde32c1640 1 -- 172.21.15.150:0/56540125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc158690 msgr2=0x7fbddc151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.057+0000 7fbde32c1640 1 --2- 172.21.15.150:0/56540125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc158690 0x7fbddc151ba0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009900 tx=0x7fbdcc01c8e0 comp rx=0 tx=0).stop 2024-09-17T05:10:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.057+0000 7fbde32c1640 1 -- 172.21.15.150:0/56540125 shutdown_connections 2024-09-17T05:10:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.057+0000 7fbde32c1640 1 --2- 172.21.15.150:0/56540125 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbddc1520e0 0x7fbddc154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.057+0000 7fbde32c1640 1 --2- 172.21.15.150:0/56540125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc158690 0x7fbddc151ba0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.057+0000 7fbde32c1640 1 -- 172.21.15.150:0/56540125 >> 172.21.15.150:0/56540125 conn(0x7fbddc06d670 msgr2=0x7fbddc0ab830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde32c1640 1 -- 172.21.15.150:0/56540125 shutdown_connections 2024-09-17T05:10:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde32c1640 1 -- 172.21.15.150:0/56540125 wait complete. 2024-09-17T05:10:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde32c1640 1 Processor -- start 2024-09-17T05:10:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde32c1640 1 -- start start 2024-09-17T05:10:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde32c1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbddc0a91c0 0x7fbddc0a9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde32c1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbddc1520e0 0x7fbddc0c3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde32c1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc158690 0x7fbddc155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde32c1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbddc161a30 con 0x7fbddc158690 2024-09-17T05:10:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde32c1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbddc0ac370 con 0x7fbddc0a91c0 2024-09-17T05:10:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde32c1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbddc161bb0 con 0x7fbddc1520e0 2024-09-17T05:10:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde2ac0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc158690 0x7fbddc155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde2ac0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc158690 0x7fbddc155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49714/0 (socket says 172.21.15.150:49714) 2024-09-17T05:10:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde2ac0640 1 -- 172.21.15.150:0/3842058585 learned_addr learned my addr 172.21.15.150:0/3842058585 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:10:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde22bf640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbddc0a91c0 0x7fbddc0a9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde1abe640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbddc1520e0 0x7fbddc0c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde2ac0640 1 -- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbddc1520e0 msgr2=0x7fbddc0c3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde2ac0640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbddc1520e0 0x7fbddc0c3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde2ac0640 1 -- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbddc0a91c0 msgr2=0x7fbddc0a9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde2ac0640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbddc0a91c0 0x7fbddc0a9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde2ac0640 1 -- 172.21.15.150:0/3842058585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbddc1f0a40 con 0x7fbddc158690 2024-09-17T05:10:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde22bf640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbddc0a91c0 0x7fbddc0a9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:10:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde1abe640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbddc1520e0 0x7fbddc0c3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:10:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.061+0000 7fbde2ac0640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc158690 0x7fbddc155b60 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fbdd400e9a0 tx=0x7fbdd400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.065+0000 7fbdd2ffd640 1 -- 172.21.15.150:0/3842058585 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd4004d30 con 0x7fbddc158690 2024-09-17T05:10:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.065+0000 7fbdd2ffd640 1 -- 172.21.15.150:0/3842058585 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbdd400cb20 con 0x7fbddc158690 2024-09-17T05:10:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.065+0000 7fbde32c1640 1 -- 172.21.15.150:0/3842058585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbddc156c80 con 0x7fbddc158690 2024-09-17T05:10:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.065+0000 7fbdd2ffd640 1 -- 172.21.15.150:0/3842058585 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd4005020 con 0x7fbddc158690 2024-09-17T05:10:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.065+0000 7fbde32c1640 1 -- 172.21.15.150:0/3842058585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbddc1565c0 con 0x7fbddc158690 2024-09-17T05:10:56.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.065+0000 7fbde32c1640 1 -- 172.21.15.150:0/3842058585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbddc1578d0 con 0x7fbddc158690 2024-09-17T05:10:56.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.065+0000 7fbdd2ffd640 1 -- 172.21.15.150:0/3842058585 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbdd40051c0 con 0x7fbddc158690 2024-09-17T05:10:56.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.065+0000 7fbdd2ffd640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbdc0073230 0x7fbdc0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:10:56.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.065+0000 7fbdd2ffd640 1 -- 172.21.15.150:0/3842058585 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7fbdd4095870 con 0x7fbddc158690 2024-09-17T05:10:56.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.065+0000 7fbde22bf640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbdc0073230 0x7fbdc0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:10:56.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.069+0000 7fbde22bf640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbdc0073230 0x7fbdc0075740 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80016b0 tx=0x7fbdd8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:10:56.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.073+0000 7fbdd2ffd640 1 -- 172.21.15.150:0/3842058585 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbdd4016030 con 0x7fbddc158690 2024-09-17T05:10:56.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.361+0000 7fbde32c1640 1 -- 172.21.15.150:0/3842058585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbddc156820 con 0x7fbddc158690 2024-09-17T05:10:56.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.361+0000 7fbdd2ffd640 1 -- 172.21.15.150:0/3842058585 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) ==== 95+0+5718 (secure 0 0 0) 0x7fbdd405fba0 con 0x7fbddc158690 2024-09-17T05:10:56.362 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:10:56.362 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":155,"btime":"2024-09-17T05:10:54:175445+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:10:56.362 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 155 2024-09-17T05:10:56.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.365+0000 7fbde32c1640 1 -- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbdc0073230 msgr2=0x7fbdc0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:56.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.365+0000 7fbde32c1640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbdc0073230 0x7fbdc0075740 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80016b0 tx=0x7fbdd8009040 comp rx=0 tx=0).stop 2024-09-17T05:10:56.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.365+0000 7fbde32c1640 1 -- 172.21.15.150:0/3842058585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc158690 msgr2=0x7fbddc155b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:10:56.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.365+0000 7fbde32c1640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc158690 0x7fbddc155b60 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fbdd400e9a0 tx=0x7fbdd400ee60 comp rx=0 tx=0).stop 2024-09-17T05:10:56.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.365+0000 7fbde32c1640 1 -- 172.21.15.150:0/3842058585 shutdown_connections 2024-09-17T05:10:56.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.365+0000 7fbde32c1640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbdc0073230 0x7fbdc0075740 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:56.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.365+0000 7fbde32c1640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbddc158690 0x7fbddc155b60 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:56.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.365+0000 7fbde32c1640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbddc1520e0 0x7fbddc0c3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:56.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.365+0000 7fbde32c1640 1 --2- 172.21.15.150:0/3842058585 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbddc0a91c0 0x7fbddc0a9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:10:56.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.365+0000 7fbde32c1640 1 -- 172.21.15.150:0/3842058585 >> 172.21.15.150:0/3842058585 conn(0x7fbddc06d670 msgr2=0x7fbddc0aa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:10:56.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.365+0000 7fbde32c1640 1 -- 172.21.15.150:0/3842058585 shutdown_connections 2024-09-17T05:10:56.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:10:56.365+0000 7fbde32c1640 1 -- 172.21.15.150:0/3842058585 wait complete. 2024-09-17T05:10:56.387 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=7158 2024-09-17T05:10:56.387 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before reviving mds.c 2024-09-17T05:10:57.594 INFO:tasks.ceph.mds.d.smithi176.stderr:2024-09-17T05:10:57.590+0000 7f8fe6ec0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:11:01.387 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-09-17T05:11:01.388 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T05:11:01.388 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T05:11:01.394 INFO:tasks.ceph.mds.c:Started 2024-09-17T05:11:01.394 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:11:01.507 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T05:11:01.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.561+0000 7f8c5ada4640 1 -- 172.21.15.150:0/3051274928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c54155730 msgr2=0x7f8c54158190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:01.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.561+0000 7f8c5ada4640 1 --2- 172.21.15.150:0/3051274928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c54155730 0x7f8c54158190 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c009bc0 tx=0x7f8c4c01cb10 comp rx=0 tx=0).stop 2024-09-17T05:11:01.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.561+0000 7f8c5ada4640 1 -- 172.21.15.150:0/3051274928 shutdown_connections 2024-09-17T05:11:01.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.561+0000 7f8c5ada4640 1 --2- 172.21.15.150:0/3051274928 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c54155730 0x7f8c54158190 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:01.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.561+0000 7f8c5ada4640 1 --2- 172.21.15.150:0/3051274928 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c540ae350 0x7f8c54074a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:01.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.561+0000 7f8c5ada4640 1 -- 172.21.15.150:0/3051274928 >> 172.21.15.150:0/3051274928 conn(0x7f8c5406e360 msgr2=0x7f8c5406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:01.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.561+0000 7f8c5ada4640 1 -- 172.21.15.150:0/3051274928 shutdown_connections 2024-09-17T05:11:01.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.561+0000 7f8c5ada4640 1 -- 172.21.15.150:0/3051274928 wait complete. 2024-09-17T05:11:01.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c5ada4640 1 Processor -- start 2024-09-17T05:11:01.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c5ada4640 1 -- start start 2024-09-17T05:11:01.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c5ada4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c540ae350 0x7f8c540a9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:01.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c5ada4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c540c3740 0x7f8c540a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:01.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c5ada4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c54155730 0x7f8c5415d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c5ada4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8c54158ae0 con 0x7f8c540ae350 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c5ada4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8c54161850 con 0x7f8c540c3740 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c5ada4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8c54151ea0 con 0x7f8c54155730 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c58b40640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c540ae350 0x7f8c540a9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c58b40640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c540ae350 0x7f8c540a9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59294/0 (socket says 172.21.15.150:59294) 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c58b40640 1 -- 172.21.15.150:0/2264834026 learned_addr learned my addr 172.21.15.150:0/2264834026 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c53fff640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c540c3740 0x7f8c540a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c59341640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c54155730 0x7f8c5415d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c58b40640 1 -- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c54155730 msgr2=0x7f8c5415d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c58b40640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c54155730 0x7f8c5415d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c58b40640 1 -- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c540c3740 msgr2=0x7f8c540a8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c58b40640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c540c3740 0x7f8c540a8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:01.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c58b40640 1 -- 172.21.15.150:0/2264834026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5414c3a0 con 0x7f8c540ae350 2024-09-17T05:11:01.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.565+0000 7f8c58b40640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c540ae350 0x7f8c540a9da0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f8c4800d950 tx=0x7f8c4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:01.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.569+0000 7f8c517fa640 1 -- 172.21.15.150:0/2264834026 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c48017070 con 0x7f8c540ae350 2024-09-17T05:11:01.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.569+0000 7f8c517fa640 1 -- 172.21.15.150:0/2264834026 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c48009d20 con 0x7f8c540ae350 2024-09-17T05:11:01.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.569+0000 7f8c5ada4640 1 -- 172.21.15.150:0/2264834026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c5414c680 con 0x7f8c540ae350 2024-09-17T05:11:01.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.569+0000 7f8c517fa640 1 -- 172.21.15.150:0/2264834026 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c480106c0 con 0x7f8c540ae350 2024-09-17T05:11:01.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.569+0000 7f8c5ada4640 1 -- 172.21.15.150:0/2264834026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c54072a70 con 0x7f8c540ae350 2024-09-17T05:11:01.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.569+0000 7f8c5ada4640 1 -- 172.21.15.150:0/2264834026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c54074a60 con 0x7f8c540ae350 2024-09-17T05:11:01.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.569+0000 7f8c517fa640 1 -- 172.21.15.150:0/2264834026 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8c4801a070 con 0x7f8c540ae350 2024-09-17T05:11:01.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.569+0000 7f8c517fa640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c38073200 0x7f8c38075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:01.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.569+0000 7f8c517fa640 1 -- 172.21.15.150:0/2264834026 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f8c48095480 con 0x7f8c540ae350 2024-09-17T05:11:01.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.569+0000 7f8c53fff640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c38073200 0x7f8c38075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:01.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.573+0000 7f8c53fff640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c38073200 0x7f8c38075710 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8c5406d810 tx=0x7f8c40009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:01.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.573+0000 7f8c517fa640 1 -- 172.21.15.150:0/2264834026 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8c4805f860 con 0x7f8c540ae350 2024-09-17T05:11:01.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.817+0000 7f8c5ada4640 1 -- 172.21.15.150:0/2264834026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c540c72c0 con 0x7f8c540ae350 2024-09-17T05:11:01.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.817+0000 7f8c517fa640 1 -- 172.21.15.150:0/2264834026 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) ==== 95+0+5718 (secure 0 0 0) 0x7f8c4805f160 con 0x7f8c540ae350 2024-09-17T05:11:01.816 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:11:01.816 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":155,"btime":"2024-09-17T05:10:54:175445+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:11:01.817 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 155 2024-09-17T05:11:01.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.821+0000 7f8c5ada4640 1 -- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c38073200 msgr2=0x7f8c38075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:01.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.821+0000 7f8c5ada4640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c38073200 0x7f8c38075710 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8c5406d810 tx=0x7f8c40009040 comp rx=0 tx=0).stop 2024-09-17T05:11:01.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.821+0000 7f8c5ada4640 1 -- 172.21.15.150:0/2264834026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c540ae350 msgr2=0x7f8c540a9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:01.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.821+0000 7f8c5ada4640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c540ae350 0x7f8c540a9da0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f8c4800d950 tx=0x7f8c4800de10 comp rx=0 tx=0).stop 2024-09-17T05:11:01.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.821+0000 7f8c5ada4640 1 -- 172.21.15.150:0/2264834026 shutdown_connections 2024-09-17T05:11:01.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.821+0000 7f8c5ada4640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c38073200 0x7f8c38075710 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:01.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.821+0000 7f8c5ada4640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c540c3740 0x7f8c540a8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:01.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.821+0000 7f8c5ada4640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8c54155730 0x7f8c5415d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:01.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.821+0000 7f8c5ada4640 1 --2- 172.21.15.150:0/2264834026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8c540ae350 0x7f8c540a9da0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:01.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.821+0000 7f8c5ada4640 1 -- 172.21.15.150:0/2264834026 >> 172.21.15.150:0/2264834026 conn(0x7f8c5406e360 msgr2=0x7f8c54156da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:01.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.821+0000 7f8c5ada4640 1 -- 172.21.15.150:0/2264834026 shutdown_connections 2024-09-17T05:11:01.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:01.821+0000 7f8c5ada4640 1 -- 172.21.15.150:0/2264834026 wait complete. 2024-09-17T05:11:01.838 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-17T05:11:03.735 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:11:03.738 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:11:03.839 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:11:04.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.021+0000 7f00e1c91640 1 -- 172.21.15.150:0/745369672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc149d70 msgr2=0x7f00dc14a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:04.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.021+0000 7f00e1c91640 1 --2- 172.21.15.150:0/745369672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc149d70 0x7f00dc14a180 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0755f0 tx=0x7f00d001cd80 comp rx=0 tx=0).stop 2024-09-17T05:11:04.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.021+0000 7f00e1c91640 1 -- 172.21.15.150:0/745369672 shutdown_connections 2024-09-17T05:11:04.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.021+0000 7f00e1c91640 1 --2- 172.21.15.150:0/745369672 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00dc0c8330 0x7f00dc0c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:04.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.021+0000 7f00e1c91640 1 --2- 172.21.15.150:0/745369672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc149d70 0x7f00dc14a180 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:04.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.021+0000 7f00e1c91640 1 --2- 172.21.15.150:0/745369672 >> v2:172.21.15.176:3300/0 conn(0x7f00dc0c7eb0 0x7f00dc1540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:04.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.021+0000 7f00e1c91640 1 -- 172.21.15.150:0/745369672 >> 172.21.15.150:0/745369672 conn(0x7f00dc0aab60 msgr2=0x7f00dc0aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:04.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.021+0000 7f00e1c91640 1 -- 172.21.15.150:0/745369672 shutdown_connections 2024-09-17T05:11:04.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.021+0000 7f00e1c91640 1 -- 172.21.15.150:0/745369672 wait complete. 2024-09-17T05:11:04.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00e1c91640 1 Processor -- start 2024-09-17T05:11:04.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00e1c91640 1 -- start start 2024-09-17T05:11:04.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00e1c91640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00dc0c7eb0 0x7f00dc15e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:04.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00e1c91640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc0c8330 0x7f00dc148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:04.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00e1c91640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00dc149d70 0x7f00dc149530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:04.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00e1c91640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc155d30 con 0x7f00dc0c8330 2024-09-17T05:11:04.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00e1c91640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc0c7bb0 con 0x7f00dc149d70 2024-09-17T05:11:04.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00e1c91640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f00dc155eb0 con 0x7f00dc0c7eb0 2024-09-17T05:11:04.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00daffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc0c8330 0x7f00dc148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:04.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00daffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc0c8330 0x7f00dc148ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59312/0 (socket says 172.21.15.150:59312) 2024-09-17T05:11:04.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00daffd640 1 -- 172.21.15.150:0/4030244812 learned_addr learned my addr 172.21.15.150:0/4030244812 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:11:04.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00db7fe640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00dc0c7eb0 0x7f00dc15e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:04.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00dbfff640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00dc149d70 0x7f00dc149530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:04.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00daffd640 1 -- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00dc0c7eb0 msgr2=0x7f00dc15e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:04.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00daffd640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00dc0c7eb0 0x7f00dc15e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:04.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00daffd640 1 -- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00dc149d70 msgr2=0x7f00dc149530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:04.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00daffd640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00dc149d70 0x7f00dc149530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:04.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00daffd640 1 -- 172.21.15.150:0/4030244812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00dc0c63a0 con 0x7f00dc0c8330 2024-09-17T05:11:04.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00daffd640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc0c8330 0x7f00dc148ff0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f00d0000f90 tx=0x7f00d0009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:04.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00c3fff640 1 -- 172.21.15.150:0/4030244812 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d00bb070 con 0x7f00dc0c8330 2024-09-17T05:11:04.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00c3fff640 1 -- 172.21.15.150:0/4030244812 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00d0004d40 con 0x7f00dc0c8330 2024-09-17T05:11:04.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00e1c91640 1 -- 172.21.15.150:0/4030244812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00dc1509b0 con 0x7f00dc0c8330 2024-09-17T05:11:04.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00c3fff640 1 -- 172.21.15.150:0/4030244812 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d00b2040 con 0x7f00dc0c8330 2024-09-17T05:11:04.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.025+0000 7f00e1c91640 1 -- 172.21.15.150:0/4030244812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00dc149ae0 con 0x7f00dc0c8330 2024-09-17T05:11:04.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.029+0000 7f00e1c91640 1 -- 172.21.15.150:0/4030244812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00dc0ad0e0 con 0x7f00dc0c8330 2024-09-17T05:11:04.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.029+0000 7f00c3fff640 1 -- 172.21.15.150:0/4030244812 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f00d00be070 con 0x7f00dc0c8330 2024-09-17T05:11:04.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.029+0000 7f00c3fff640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f00c8073200 0x7f00c8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:04.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.029+0000 7f00c3fff640 1 -- 172.21.15.150:0/4030244812 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f00d012ece0 con 0x7f00dc0c8330 2024-09-17T05:11:04.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.029+0000 7f00db7fe640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f00c8073200 0x7f00c8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:04.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.029+0000 7f00db7fe640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f00c8073200 0x7f00c8075710 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f00d400bad0 tx=0x7f00d400b260 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:04.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.033+0000 7f00c3fff640 1 -- 172.21.15.150:0/4030244812 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f00d0016970 con 0x7f00dc0c8330 2024-09-17T05:11:04.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.285+0000 7f00e1c91640 1 -- 172.21.15.150:0/4030244812 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00dc0c8740 con 0x7f00dc0c8330 2024-09-17T05:11:04.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.285+0000 7f00c3fff640 1 -- 172.21.15.150:0/4030244812 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+6547 (secure 0 0 0) 0x7f00d00f89d0 con 0x7f00dc0c8330 2024-09-17T05:11:04.285 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:11:04.285 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":156,"btime":"2024-09-17T05:11:02:196586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7407,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:11:04.285 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 156 2024-09-17T05:11:04.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.289+0000 7f00e1c91640 1 -- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f00c8073200 msgr2=0x7f00c8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:04.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.289+0000 7f00e1c91640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f00c8073200 0x7f00c8075710 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f00d400bad0 tx=0x7f00d400b260 comp rx=0 tx=0).stop 2024-09-17T05:11:04.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.289+0000 7f00e1c91640 1 -- 172.21.15.150:0/4030244812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc0c8330 msgr2=0x7f00dc148ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:04.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.289+0000 7f00e1c91640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc0c8330 0x7f00dc148ff0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f00d0000f90 tx=0x7f00d0009360 comp rx=0 tx=0).stop 2024-09-17T05:11:04.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.289+0000 7f00e1c91640 1 -- 172.21.15.150:0/4030244812 shutdown_connections 2024-09-17T05:11:04.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.289+0000 7f00e1c91640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f00c8073200 0x7f00c8075710 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:04.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.289+0000 7f00e1c91640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00dc149d70 0x7f00dc149530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:04.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.289+0000 7f00e1c91640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00dc0c8330 0x7f00dc148ff0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:04.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.289+0000 7f00e1c91640 1 --2- 172.21.15.150:0/4030244812 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00dc0c7eb0 0x7f00dc15e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:04.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.289+0000 7f00e1c91640 1 -- 172.21.15.150:0/4030244812 >> 172.21.15.150:0/4030244812 conn(0x7f00dc0aab60 msgr2=0x7f00dc0a8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:04.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.289+0000 7f00e1c91640 1 -- 172.21.15.150:0/4030244812 shutdown_connections 2024-09-17T05:11:04.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:04.289+0000 7f00e1c91640 1 -- 172.21.15.150:0/4030244812 wait complete. 2024-09-17T05:11:04.307 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-09-17T05:11:04.307 INFO:tasks.mds_thrash.fs.[b]:waiting for 58 secs before thrashing 2024-09-17T05:11:33.993 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:11:33.997 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:11:43.890 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:11:44.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.068+0000 7f9430b51640 1 -- 172.21.15.150:0/1179708504 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f942c0c5df0 msgr2=0x7f942c157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:44.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.068+0000 7f9430b51640 1 --2- 172.21.15.150:0/1179708504 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f942c0c5df0 0x7f942c157330 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f941c0097f0 tx=0x7f941c01c5b0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.068+0000 7f9430b51640 1 -- 172.21.15.150:0/1179708504 shutdown_connections 2024-09-17T05:11:44.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.068+0000 7f9430b51640 1 --2- 172.21.15.150:0/1179708504 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f942c0c5df0 0x7f942c157330 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.068+0000 7f9430b51640 1 -- 172.21.15.150:0/1179708504 >> 172.21.15.150:0/1179708504 conn(0x7f942c074750 msgr2=0x7f942c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:44.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.068+0000 7f9430b51640 1 -- 172.21.15.150:0/1179708504 shutdown_connections 2024-09-17T05:11:44.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.068+0000 7f9430b51640 1 -- 172.21.15.150:0/1179708504 wait complete. 2024-09-17T05:11:44.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.068+0000 7f9430b51640 1 Processor -- start 2024-09-17T05:11:44.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.068+0000 7f9430b51640 1 -- start start 2024-09-17T05:11:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9430b51640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f942c0c4b20 0x7f942c1e2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9430b51640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f942c0c5df0 0x7f942c149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9430b51640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f942c0c6340 0x7f942c1ed270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9430b51640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f942c1542d0 con 0x7f942c0c5df0 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9430b51640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f942c1567e0 con 0x7f942c0c6340 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9430b51640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f942c154450 con 0x7f942c0c4b20 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9423fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f942c0c5df0 0x7f942c149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9423fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f942c0c5df0 0x7f942c149380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52388/0 (socket says 172.21.15.150:52388) 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9423fff640 1 -- 172.21.15.150:0/3184963137 learned_addr learned my addr 172.21.15.150:0/3184963137 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f942a59c640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f942c0c4b20 0x7f942c1e2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f942ad9d640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f942c0c6340 0x7f942c1ed270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9423fff640 1 -- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f942c0c4b20 msgr2=0x7f942c1e2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9423fff640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f942c0c4b20 0x7f942c1e2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9423fff640 1 -- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f942c0c6340 msgr2=0x7f942c1ed270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9423fff640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f942c0c6340 0x7f942c1ed270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9423fff640 1 -- 172.21.15.150:0/3184963137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942c1e1290 con 0x7f942c0c5df0 2024-09-17T05:11:44.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9423fff640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f942c0c5df0 0x7f942c149380 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f941400c710 tx=0x7f941400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:44.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9422ffd640 1 -- 172.21.15.150:0/3184963137 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941400a840 con 0x7f942c0c5df0 2024-09-17T05:11:44.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9422ffd640 1 -- 172.21.15.150:0/3184963137 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9414017070 con 0x7f942c0c5df0 2024-09-17T05:11:44.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9430b51640 1 -- 172.21.15.150:0/3184963137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942c1f0fc0 con 0x7f942c0c5df0 2024-09-17T05:11:44.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9422ffd640 1 -- 172.21.15.150:0/3184963137 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941400ac80 con 0x7f942c0c5df0 2024-09-17T05:11:44.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9430b51640 1 -- 172.21.15.150:0/3184963137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f942c1e17e0 con 0x7f942c0c5df0 2024-09-17T05:11:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9430b51640 1 -- 172.21.15.150:0/3184963137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f942c1f0d30 con 0x7f942c0c5df0 2024-09-17T05:11:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.072+0000 7f9422ffd640 1 -- 172.21.15.150:0/3184963137 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f941401c460 con 0x7f942c0c5df0 2024-09-17T05:11:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.076+0000 7f9422ffd640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f940c073200 0x7f940c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.076+0000 7f9422ffd640 1 -- 172.21.15.150:0/3184963137 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f9414094db0 con 0x7f942c0c5df0 2024-09-17T05:11:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.076+0000 7f942a59c640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f940c073200 0x7f940c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:44.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.076+0000 7f942a59c640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f940c073200 0x7f940c075710 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f941c0ae000 tx=0x7f941c01adf0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:44.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.080+0000 7f9422ffd640 1 -- 172.21.15.150:0/3184963137 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f942c0b20f0 con 0x7f942c0c5df0 2024-09-17T05:11:44.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.312+0000 7f9430b51640 1 -- 172.21.15.150:0/3184963137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f942c0c50f0 con 0x7f942c0c5df0 2024-09-17T05:11:44.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.312+0000 7f9422ffd640 1 -- 172.21.15.150:0/3184963137 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+6547 (secure 0 0 0) 0x7f941400ae20 con 0x7f942c0c5df0 2024-09-17T05:11:44.314 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:11:44.314 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":156,"btime":"2024-09-17T05:11:02:196586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7407,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:11:44.314 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 156 2024-09-17T05:11:44.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.316+0000 7f9430b51640 1 -- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f940c073200 msgr2=0x7f940c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:44.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.316+0000 7f9430b51640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f940c073200 0x7f940c075710 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f941c0ae000 tx=0x7f941c01adf0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.316+0000 7f9430b51640 1 -- 172.21.15.150:0/3184963137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f942c0c5df0 msgr2=0x7f942c149380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:44.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.316+0000 7f9430b51640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f942c0c5df0 0x7f942c149380 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f941400c710 tx=0x7f941400cbd0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.316+0000 7f9430b51640 1 -- 172.21.15.150:0/3184963137 shutdown_connections 2024-09-17T05:11:44.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.316+0000 7f9430b51640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f940c073200 0x7f940c075710 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.316+0000 7f9430b51640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f942c0c6340 0x7f942c1ed270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.316+0000 7f9430b51640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f942c0c5df0 0x7f942c149380 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.316+0000 7f9430b51640 1 --2- 172.21.15.150:0/3184963137 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f942c0c4b20 0x7f942c1e2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.316+0000 7f9430b51640 1 -- 172.21.15.150:0/3184963137 >> 172.21.15.150:0/3184963137 conn(0x7f942c074750 msgr2=0x7f942c0c2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:44.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.316+0000 7f9430b51640 1 -- 172.21.15.150:0/3184963137 shutdown_connections 2024-09-17T05:11:44.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.316+0000 7f9430b51640 1 -- 172.21.15.150:0/3184963137 wait complete. 2024-09-17T05:11:44.340 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-17T05:11:44.341 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T05:11:44.341 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:11:44.359 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T05:11:44.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:11:44.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.560+0000 7f77c11e1640 1 -- 172.21.15.150:0/2253334849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc06df60 msgr2=0x7f77bc06e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:44.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.560+0000 7f77c11e1640 1 --2- 172.21.15.150:0/2253334849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc06df60 0x7f77bc06e370 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009840 tx=0x7f77a401c700 comp rx=0 tx=0).stop 2024-09-17T05:11:44.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.560+0000 7f77c11e1640 1 -- 172.21.15.150:0/2253334849 shutdown_connections 2024-09-17T05:11:44.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.560+0000 7f77c11e1640 1 --2- 172.21.15.150:0/2253334849 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc06df60 0x7f77bc06e370 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.560+0000 7f77c11e1640 1 --2- 172.21.15.150:0/2253334849 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77bc0ab9d0 0x7f77bc14da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.560+0000 7f77c11e1640 1 -- 172.21.15.150:0/2253334849 >> 172.21.15.150:0/2253334849 conn(0x7f77bc070b50 msgr2=0x7f77bc070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:44.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.560+0000 7f77c11e1640 1 -- 172.21.15.150:0/2253334849 shutdown_connections 2024-09-17T05:11:44.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.560+0000 7f77c11e1640 1 -- 172.21.15.150:0/2253334849 wait complete. 2024-09-17T05:11:44.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.560+0000 7f77c11e1640 1 Processor -- start 2024-09-17T05:11:44.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.560+0000 7f77c11e1640 1 -- start start 2024-09-17T05:11:44.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77c11e1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc06df60 0x7f77bc155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:44.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77c11e1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77bc0ab9d0 0x7f77bc156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:44.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77c11e1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc1536e0 0x7f77bc0adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:44.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77c11e1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f77bc154200 con 0x7f77bc1536e0 2024-09-17T05:11:44.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77c11e1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f77bc069e00 con 0x7f77bc06df60 2024-09-17T05:11:44.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77c11e1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f77bc154380 con 0x7f77bc0ab9d0 2024-09-17T05:11:44.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77bb59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc1536e0 0x7f77bc0adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:44.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77bb59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc1536e0 0x7f77bc0adbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52400/0 (socket says 172.21.15.150:52400) 2024-09-17T05:11:44.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77bb59e640 1 -- 172.21.15.150:0/125218945 learned_addr learned my addr 172.21.15.150:0/125218945 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:11:44.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77bad9d640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc06df60 0x7f77bc155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:44.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77ba59c640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77bc0ab9d0 0x7f77bc156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:44.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77bb59e640 1 -- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77bc0ab9d0 msgr2=0x7f77bc156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:44.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77bb59e640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77bc0ab9d0 0x7f77bc156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77bb59e640 1 -- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc06df60 msgr2=0x7f77bc155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:44.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77bb59e640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc06df60 0x7f77bc155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77bb59e640 1 -- 172.21.15.150:0/125218945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77bc1557f0 con 0x7f77bc1536e0 2024-09-17T05:11:44.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77bb59e640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc1536e0 0x7f77bc0adbf0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f77b400c3f0 tx=0x7f77b400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:44.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77b37fe640 1 -- 172.21.15.150:0/125218945 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b4017070 con 0x7f77bc1536e0 2024-09-17T05:11:44.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77c11e1640 1 -- 172.21.15.150:0/125218945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77bc156ec0 con 0x7f77bc1536e0 2024-09-17T05:11:44.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77b37fe640 1 -- 172.21.15.150:0/125218945 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77b40053c0 con 0x7f77bc1536e0 2024-09-17T05:11:44.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77b37fe640 1 -- 172.21.15.150:0/125218945 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b4003860 con 0x7f77bc1536e0 2024-09-17T05:11:44.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77c11e1640 1 -- 172.21.15.150:0/125218945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77bc155510 con 0x7f77bc1536e0 2024-09-17T05:11:44.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.564+0000 7f77c11e1640 1 -- 172.21.15.150:0/125218945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77bc06e370 con 0x7f77bc1536e0 2024-09-17T05:11:44.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.568+0000 7f77b37fe640 1 -- 172.21.15.150:0/125218945 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f77b401a070 con 0x7f77bc1536e0 2024-09-17T05:11:44.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.568+0000 7f77b37fe640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f779c073200 0x7f779c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:44.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.568+0000 7f77b37fe640 1 -- 172.21.15.150:0/125218945 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f77b4094600 con 0x7f77bc1536e0 2024-09-17T05:11:44.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.568+0000 7f77bad9d640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f779c073200 0x7f779c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:44.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.568+0000 7f77bad9d640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f779c073200 0x7f779c075710 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f77ac009e90 tx=0x7f77ac0095c0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:44.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.572+0000 7f77b37fe640 1 -- 172.21.15.150:0/125218945 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f77b4021020 con 0x7f77bc1536e0 2024-09-17T05:11:44.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 -- 172.21.15.150:0/125218945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77bc075060 con 0x7f77bc1536e0 2024-09-17T05:11:44.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77b37fe640 1 -- 172.21.15.150:0/125218945 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+6547 (secure 0 0 0) 0x7f77bc1557f0 con 0x7f77bc1536e0 2024-09-17T05:11:44.780 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:11:44.780 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":156,"btime":"2024-09-17T05:11:02:196586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7407,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:11:44.780 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 156 2024-09-17T05:11:44.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 -- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f779c073200 msgr2=0x7f779c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:44.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f779c073200 0x7f779c075710 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f77ac009e90 tx=0x7f77ac0095c0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 -- 172.21.15.150:0/125218945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc1536e0 msgr2=0x7f77bc0adbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:44.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc1536e0 0x7f77bc0adbf0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f77b400c3f0 tx=0x7f77b400c8b0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 -- 172.21.15.150:0/125218945 shutdown_connections 2024-09-17T05:11:44.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f779c073200 0x7f779c075710 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77bc1536e0 0x7f77bc0adbf0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77bc0ab9d0 0x7f77bc156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 --2- 172.21.15.150:0/125218945 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77bc06df60 0x7f77bc155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:44.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 -- 172.21.15.150:0/125218945 >> 172.21.15.150:0/125218945 conn(0x7f77bc070b50 msgr2=0x7f77bc151580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:44.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 -- 172.21.15.150:0/125218945 shutdown_connections 2024-09-17T05:11:44.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:44.780+0000 7f77c11e1640 1 -- 172.21.15.150:0/125218945 wait complete. 2024-09-17T05:11:44.800 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:11:46.801 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:11:46.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.968+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/3848353221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b080a98a0 msgr2=0x7f4b080a9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:46.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.968+0000 7f4b0f3c2640 1 --2- 172.21.15.150:0/3848353221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b080a98a0 0x7f4b080a9cb0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f4b08073720 tx=0x7f4af801c7a0 comp rx=0 tx=0).stop 2024-09-17T05:11:46.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.968+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/3848353221 shutdown_connections 2024-09-17T05:11:46.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.968+0000 7f4b0f3c2640 1 --2- 172.21.15.150:0/3848353221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b080a98a0 0x7f4b080a9cb0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:46.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.968+0000 7f4b0f3c2640 1 --2- 172.21.15.150:0/3848353221 >> v2:172.21.15.176:3300/0 conn(0x7f4b080699b0 0x7f4b08151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:46.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.968+0000 7f4b0f3c2640 1 --2- 172.21.15.150:0/3848353221 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b0806e590 0x7f4b080719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:46.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.968+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/3848353221 >> 172.21.15.150:0/3848353221 conn(0x7f4b08070b50 msgr2=0x7f4b08070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:46.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.968+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/3848353221 shutdown_connections 2024-09-17T05:11:46.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.968+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/3848353221 wait complete. 2024-09-17T05:11:46.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0f3c2640 1 Processor -- start 2024-09-17T05:11:46.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0f3c2640 1 -- start start 2024-09-17T05:11:46.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0f3c2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b080699b0 0x7f4b08150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:46.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0f3c2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b0806e590 0x7f4b0814b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:46.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0f3c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b080a98a0 0x7f4b081466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:46.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0f3c2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4b08158120 con 0x7f4b080a98a0 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0f3c2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4b08157fa0 con 0x7f4b0806e590 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0f3c2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4b081582a0 con 0x7f4b080699b0 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0d95f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b080a98a0 0x7f4b081466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0d95f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b080a98a0 0x7f4b081466b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52424/0 (socket says 172.21.15.150:52424) 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0d95f640 1 -- 172.21.15.150:0/300473935 learned_addr learned my addr 172.21.15.150:0/300473935 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0c95d640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b0806e590 0x7f4b0814b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0d15e640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b080699b0 0x7f4b08150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0d95f640 1 -- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b080699b0 msgr2=0x7f4b08150e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0d95f640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b080699b0 0x7f4b08150e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0d95f640 1 -- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b0806e590 msgr2=0x7f4b0814b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0d95f640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b0806e590 0x7f4b0814b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0d95f640 1 -- 172.21.15.150:0/300473935 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b08156c20 con 0x7f4b080a98a0 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0d15e640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b080699b0 0x7f4b08150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:11:46.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0d95f640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b080a98a0 0x7f4b081466b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f4af801ae50 tx=0x7f4af8005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:46.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4afdffb640 1 -- 172.21.15.150:0/300473935 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af80ac070 con 0x7f4b080a98a0 2024-09-17T05:11:46.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4afdffb640 1 -- 172.21.15.150:0/300473935 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4af8004320 con 0x7f4b080a98a0 2024-09-17T05:11:46.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/300473935 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b08157950 con 0x7f4b080a98a0 2024-09-17T05:11:46.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4afdffb640 1 -- 172.21.15.150:0/300473935 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8004a30 con 0x7f4b080a98a0 2024-09-17T05:11:46.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/300473935 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b081471d0 con 0x7f4b080a98a0 2024-09-17T05:11:46.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.972+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/300473935 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b08147400 con 0x7f4b080a98a0 2024-09-17T05:11:46.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.976+0000 7f4afdffb640 1 -- 172.21.15.150:0/300473935 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4af80af070 con 0x7f4b080a98a0 2024-09-17T05:11:46.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.976+0000 7f4afdffb640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4af4073230 0x7f4af4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:46.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.976+0000 7f4afdffb640 1 -- 172.21.15.150:0/300473935 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f4af812ee60 con 0x7f4b080a98a0 2024-09-17T05:11:46.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.976+0000 7f4b0d15e640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4af4073230 0x7f4af4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:46.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.976+0000 7f4b0d15e640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4af4073230 0x7f4af4075740 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009f70 tx=0x7f4b00009500 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:46.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:46.980+0000 7f4afdffb640 1 -- 172.21.15.150:0/300473935 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4b08147400 con 0x7f4b080a98a0 2024-09-17T05:11:47.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.180+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/300473935 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b081f1420 con 0x7f4b080a98a0 2024-09-17T05:11:47.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.180+0000 7f4afdffb640 1 -- 172.21.15.150:0/300473935 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+6547 (secure 0 0 0) 0x7f4af80164b0 con 0x7f4b080a98a0 2024-09-17T05:11:47.182 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:11:47.182 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":156,"btime":"2024-09-17T05:11:02:196586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7407,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:11:47.183 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 156 2024-09-17T05:11:47.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.184+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4af4073230 msgr2=0x7f4af4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:47.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.184+0000 7f4b0f3c2640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4af4073230 0x7f4af4075740 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009f70 tx=0x7f4b00009500 comp rx=0 tx=0).stop 2024-09-17T05:11:47.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.184+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/300473935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b080a98a0 msgr2=0x7f4b081466b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:47.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.184+0000 7f4b0f3c2640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b080a98a0 0x7f4b081466b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f4af801ae50 tx=0x7f4af8005da0 comp rx=0 tx=0).stop 2024-09-17T05:11:47.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.184+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/300473935 shutdown_connections 2024-09-17T05:11:47.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.184+0000 7f4b0f3c2640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4af4073230 0x7f4af4075740 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:47.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.184+0000 7f4b0f3c2640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b0806e590 0x7f4b0814b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:47.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.184+0000 7f4b0f3c2640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b080a98a0 0x7f4b081466b0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:47.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.184+0000 7f4b0f3c2640 1 --2- 172.21.15.150:0/300473935 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b080699b0 0x7f4b08150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:47.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.184+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/300473935 >> 172.21.15.150:0/300473935 conn(0x7f4b08070b50 msgr2=0x7f4b0814e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:47.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.184+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/300473935 shutdown_connections 2024-09-17T05:11:47.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:47.184+0000 7f4b0f3c2640 1 -- 172.21.15.150:0/300473935 wait complete. 2024-09-17T05:11:47.203 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:11:49.204 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:11:49.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.372+0000 7fe60220e640 1 -- 172.21.15.150:0/484552019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5e4004d50 msgr2=0x7fe5e4005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:49.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.372+0000 7fe60220e640 1 --2- 172.21.15.150:0/484552019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5e4004d50 0x7fe5e4005160 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009b30 tx=0x7fe5ec01d250 comp rx=0 tx=0).stop 2024-09-17T05:11:49.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.372+0000 7fe5fbfff640 1 -- 172.21.15.150:0/484552019 reap_dead start 2024-09-17T05:11:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.372+0000 7fe60220e640 1 -- 172.21.15.150:0/484552019 shutdown_connections 2024-09-17T05:11:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.372+0000 7fe60220e640 1 -- 172.21.15.150:0/484552019 >> 172.21.15.150:0/484552019 conn(0x7fe5fc0abb00 msgr2=0x7fe5fc0abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.372+0000 7fe60220e640 1 -- 172.21.15.150:0/484552019 shutdown_connections 2024-09-17T05:11:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.372+0000 7fe60220e640 1 -- 172.21.15.150:0/484552019 wait complete. 2024-09-17T05:11:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe60220e640 1 Processor -- start 2024-09-17T05:11:49.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe60220e640 1 -- start start 2024-09-17T05:11:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe60220e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5fc154440 0x7fe5fc1ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe60220e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5fc152e70 0x7fe5fc153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe60220e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5fc157800 0x7fe5fc157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe60220e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1521b0 con 0x7fe5fc157800 2024-09-17T05:11:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe60220e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc1553f0 con 0x7fe5fc152e70 2024-09-17T05:11:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe60220e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe5fc0c4be0 con 0x7fe5fc154440 2024-09-17T05:11:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5fbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5fc157800 0x7fe5fc157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5fbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5fc157800 0x7fe5fc157c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59742/0 (socket says 172.21.15.150:59742) 2024-09-17T05:11:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5fbfff640 1 -- 172.21.15.150:0/4086912486 learned_addr learned my addr 172.21.15.150:0/4086912486 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:11:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5faffd640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5fc152e70 0x7fe5fc153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5fb7fe640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5fc154440 0x7fe5fc1ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5fbfff640 1 -- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5fc154440 msgr2=0x7fe5fc1ee0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5fbfff640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5fc154440 0x7fe5fc1ee0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5fbfff640 1 -- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5fc152e70 msgr2=0x7fe5fc153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5fbfff640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5fc152e70 0x7fe5fc153280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5fbfff640 1 -- 172.21.15.150:0/4086912486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5fc158150 con 0x7fe5fc157800 2024-09-17T05:11:49.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5fbfff640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5fc157800 0x7fe5fc157c10 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec0af040 tx=0x7fe5ec0add30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:49.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5e3fff640 1 -- 172.21.15.150:0/4086912486 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec0b1040 con 0x7fe5fc157800 2024-09-17T05:11:49.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5e3fff640 1 -- 172.21.15.150:0/4086912486 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe5ec0bb070 con 0x7fe5fc157800 2024-09-17T05:11:49.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe60220e640 1 -- 172.21.15.150:0/4086912486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5fc1ddd80 con 0x7fe5fc157800 2024-09-17T05:11:49.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe5e3fff640 1 -- 172.21.15.150:0/4086912486 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec009730 con 0x7fe5fc157800 2024-09-17T05:11:49.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.376+0000 7fe60220e640 1 -- 172.21.15.150:0/4086912486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5fc151470 con 0x7fe5fc157800 2024-09-17T05:11:49.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.380+0000 7fe60220e640 1 -- 172.21.15.150:0/4086912486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5fc151c70 con 0x7fe5fc157800 2024-09-17T05:11:49.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.380+0000 7fe5e3fff640 1 -- 172.21.15.150:0/4086912486 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe5ec004030 con 0x7fe5fc157800 2024-09-17T05:11:49.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.380+0000 7fe5e3fff640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe5e8073200 0x7fe5e8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:49.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.380+0000 7fe5e3fff640 1 -- 172.21.15.150:0/4086912486 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7fe5ec130750 con 0x7fe5fc157800 2024-09-17T05:11:49.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.380+0000 7fe5fb7fe640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe5e8073200 0x7fe5e8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:49.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.380+0000 7fe5fb7fe640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe5e8073200 0x7fe5e8075710 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0ab7f0 tx=0x7fe5f4009500 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:49.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.384+0000 7fe5e3fff640 1 -- 172.21.15.150:0/4086912486 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe5ec016f60 con 0x7fe5fc157800 2024-09-17T05:11:49.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.592+0000 7fe60220e640 1 -- 172.21.15.150:0/4086912486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5fc0ab540 con 0x7fe5fc157800 2024-09-17T05:11:49.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe5e3fff640 1 -- 172.21.15.150:0/4086912486 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+6547 (secure 0 0 0) 0x7fe5ec0fa4e0 con 0x7fe5fc157800 2024-09-17T05:11:49.595 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:11:49.595 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":156,"btime":"2024-09-17T05:11:02:196586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7407,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:11:49.595 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 156 2024-09-17T05:11:49.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe60220e640 1 -- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe5e8073200 msgr2=0x7fe5e8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:49.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe60220e640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe5e8073200 0x7fe5e8075710 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0ab7f0 tx=0x7fe5f4009500 comp rx=0 tx=0).stop 2024-09-17T05:11:49.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe60220e640 1 -- 172.21.15.150:0/4086912486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5fc157800 msgr2=0x7fe5fc157c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:49.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe60220e640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5fc157800 0x7fe5fc157c10 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec0af040 tx=0x7fe5ec0add30 comp rx=0 tx=0).stop 2024-09-17T05:11:49.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe60220e640 1 -- 172.21.15.150:0/4086912486 shutdown_connections 2024-09-17T05:11:49.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe60220e640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe5e8073200 0x7fe5e8075710 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:49.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe60220e640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5fc152e70 0x7fe5fc153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:49.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe60220e640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5fc157800 0x7fe5fc157c10 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:49.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe60220e640 1 --2- 172.21.15.150:0/4086912486 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5fc154440 0x7fe5fc1ee0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:49.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe60220e640 1 -- 172.21.15.150:0/4086912486 >> 172.21.15.150:0/4086912486 conn(0x7fe5fc0abb00 msgr2=0x7fe5fc0c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:49.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe60220e640 1 -- 172.21.15.150:0/4086912486 shutdown_connections 2024-09-17T05:11:49.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:49.596+0000 7fe60220e640 1 -- 172.21.15.150:0/4086912486 wait complete. 2024-09-17T05:11:49.617 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:11:51.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:11:51.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.780+0000 7fb9417f7640 1 -- 172.21.15.150:0/3657371955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93c0701c0 msgr2=0x7fb93c160060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:51.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.780+0000 7fb9417f7640 1 --2- 172.21.15.150:0/3657371955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93c0701c0 0x7fb93c160060 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fb93c072820 tx=0x7fb93401cda0 comp rx=0 tx=0).stop 2024-09-17T05:11:51.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.780+0000 7fb9417f7640 1 -- 172.21.15.150:0/3657371955 shutdown_connections 2024-09-17T05:11:51.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.780+0000 7fb9417f7640 1 --2- 172.21.15.150:0/3657371955 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93c0701c0 0x7fb93c160060 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:51.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.780+0000 7fb9417f7640 1 -- 172.21.15.150:0/3657371955 >> 172.21.15.150:0/3657371955 conn(0x7fb93c0c4a00 msgr2=0x7fb93c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:51.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.780+0000 7fb9417f7640 1 -- 172.21.15.150:0/3657371955 shutdown_connections 2024-09-17T05:11:51.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.780+0000 7fb9417f7640 1 -- 172.21.15.150:0/3657371955 wait complete. 2024-09-17T05:11:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9417f7640 1 Processor -- start 2024-09-17T05:11:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9417f7640 1 -- start start 2024-09-17T05:11:51.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9417f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93c0701c0 0x7fb93c14f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:51.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9417f7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb93c0c3db0 0x7fb93c146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9417f7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb93c0c41c0 0x7fb93c146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9417f7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb93c070750 con 0x7fb93c0701c0 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9417f7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb93c0705d0 con 0x7fb93c0c3db0 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9417f7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb93c0ad9b0 con 0x7fb93c0c41c0 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb93affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93c0701c0 0x7fb93c14f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb93affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93c0701c0 0x7fb93c14f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59754/0 (socket says 172.21.15.150:59754) 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb93affd640 1 -- 172.21.15.150:0/2529816493 learned_addr learned my addr 172.21.15.150:0/2529816493 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb93a7fc640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb93c0c3db0 0x7fb93c146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb93b7fe640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb93c0c41c0 0x7fb93c146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb93affd640 1 -- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb93c0c41c0 msgr2=0x7fb93c146fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb93affd640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb93c0c41c0 0x7fb93c146fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb93affd640 1 -- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb93c0c3db0 msgr2=0x7fb93c146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb93affd640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb93c0c3db0 0x7fb93c146a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:51.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb93affd640 1 -- 172.21.15.150:0/2529816493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb93c14dbb0 con 0x7fb93c0701c0 2024-09-17T05:11:51.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb93affd640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93c0701c0 0x7fb93c14f410 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00c3f0 tx=0x7fb92c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:51.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9337fe640 1 -- 172.21.15.150:0/2529816493 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92c017070 con 0x7fb93c0701c0 2024-09-17T05:11:51.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9337fe640 1 -- 172.21.15.150:0/2529816493 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb92c0053c0 con 0x7fb93c0701c0 2024-09-17T05:11:51.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9417f7640 1 -- 172.21.15.150:0/2529816493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb93c155da0 con 0x7fb93c0701c0 2024-09-17T05:11:51.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9337fe640 1 -- 172.21.15.150:0/2529816493 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92c003860 con 0x7fb93c0701c0 2024-09-17T05:11:51.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.784+0000 7fb9417f7640 1 -- 172.21.15.150:0/2529816493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb93c156a80 con 0x7fb93c0701c0 2024-09-17T05:11:51.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.788+0000 7fb9417f7640 1 -- 172.21.15.150:0/2529816493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb93c0a76f0 con 0x7fb93c0701c0 2024-09-17T05:11:51.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.788+0000 7fb9337fe640 1 -- 172.21.15.150:0/2529816493 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb92c01a070 con 0x7fb93c0701c0 2024-09-17T05:11:51.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.788+0000 7fb9337fe640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb91c073200 0x7fb91c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:51.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.788+0000 7fb9337fe640 1 -- 172.21.15.150:0/2529816493 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7fb92c094220 con 0x7fb93c0701c0 2024-09-17T05:11:51.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.788+0000 7fb93a7fc640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb91c073200 0x7fb91c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:51.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.788+0000 7fb93a7fc640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb91c073200 0x7fb91c075710 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0c4e80 tx=0x7fb924008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:51.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:51.792+0000 7fb9337fe640 1 -- 172.21.15.150:0/2529816493 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb92c05e5e0 con 0x7fb93c0701c0 2024-09-17T05:11:51.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.000+0000 7fb9417f7640 1 -- 172.21.15.150:0/2529816493 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb93c1eaf10 con 0x7fb93c0701c0 2024-09-17T05:11:52.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.000+0000 7fb9337fe640 1 -- 172.21.15.150:0/2529816493 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+6547 (secure 0 0 0) 0x7fb92c001090 con 0x7fb93c0701c0 2024-09-17T05:11:52.000 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:11:52.000 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":156,"btime":"2024-09-17T05:11:02:196586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7407,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:11:52.001 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 156 2024-09-17T05:11:52.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.004+0000 7fb9417f7640 1 -- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb91c073200 msgr2=0x7fb91c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:52.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.004+0000 7fb9417f7640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb91c073200 0x7fb91c075710 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0c4e80 tx=0x7fb924008040 comp rx=0 tx=0).stop 2024-09-17T05:11:52.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.004+0000 7fb9417f7640 1 -- 172.21.15.150:0/2529816493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93c0701c0 msgr2=0x7fb93c14f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:52.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.004+0000 7fb9417f7640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93c0701c0 0x7fb93c14f410 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fb92c00c3f0 tx=0x7fb92c00c8b0 comp rx=0 tx=0).stop 2024-09-17T05:11:52.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.004+0000 7fb9417f7640 1 -- 172.21.15.150:0/2529816493 shutdown_connections 2024-09-17T05:11:52.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.004+0000 7fb9417f7640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb91c073200 0x7fb91c075710 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:52.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.004+0000 7fb9417f7640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb93c0c3db0 0x7fb93c146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:52.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.004+0000 7fb9417f7640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb93c0c41c0 0x7fb93c146fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:52.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.004+0000 7fb9417f7640 1 --2- 172.21.15.150:0/2529816493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb93c0701c0 0x7fb93c14f410 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:52.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.004+0000 7fb9417f7640 1 -- 172.21.15.150:0/2529816493 >> 172.21.15.150:0/2529816493 conn(0x7fb93c0c4a00 msgr2=0x7fb93c0a7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:52.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.004+0000 7fb9417f7640 1 -- 172.21.15.150:0/2529816493 shutdown_connections 2024-09-17T05:11:52.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:52.004+0000 7fb9417f7640 1 -- 172.21.15.150:0/2529816493 wait complete. 2024-09-17T05:11:52.022 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:11:54.023 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:11:54.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 -- 172.21.15.150:0/2627619963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17c8073440 msgr2=0x7f17c80a7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:54.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 --2- 172.21.15.150:0/2627619963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17c8073440 0x7f17c80a7300 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f17c0009c80 tx=0x7f17c001cd10 comp rx=0 tx=0).stop 2024-09-17T05:11:54.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 -- 172.21.15.150:0/2627619963 shutdown_connections 2024-09-17T05:11:54.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 --2- 172.21.15.150:0/2627619963 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17c8074450 0x7f17c8074860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:54.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 --2- 172.21.15.150:0/2627619963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17c8073440 0x7f17c80a7300 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:54.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 -- 172.21.15.150:0/2627619963 >> 172.21.15.150:0/2627619963 conn(0x7f17c80c4a00 msgr2=0x7f17c80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:54.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 -- 172.21.15.150:0/2627619963 shutdown_connections 2024-09-17T05:11:54.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 -- 172.21.15.150:0/2627619963 wait complete. 2024-09-17T05:11:54.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 Processor -- start 2024-09-17T05:11:54.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 -- start start 2024-09-17T05:11:54.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17c8070e50 0x7f17c81515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:54.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17c8073440 0x7f17c80ab390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:54.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17c8074450 0x7f17c814d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:54.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f17c80719f0 con 0x7f17c8074450 2024-09-17T05:11:54.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f17c80adef0 con 0x7f17c8070e50 2024-09-17T05:11:54.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.196+0000 7f17d04fc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f17c80ad610 con 0x7f17c8073440 2024-09-17T05:11:54.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17ce298640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17c8070e50 0x7f17c81515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:54.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17ce298640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17c8070e50 0x7f17c81515e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:45122/0 (socket says 172.21.15.150:45122) 2024-09-17T05:11:54.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17ce298640 1 -- 172.21.15.150:0/706315815 learned_addr learned my addr 172.21.15.150:0/706315815 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:11:54.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17cda97640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17c8073440 0x7f17c80ab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:54.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17cea99640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17c8074450 0x7f17c814d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:54.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17cea99640 1 -- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17c8073440 msgr2=0x7f17c80ab390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:54.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17cea99640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17c8073440 0x7f17c80ab390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:54.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17cea99640 1 -- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17c8070e50 msgr2=0x7f17c81515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:54.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17cea99640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17c8070e50 0x7f17c81515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:54.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17cea99640 1 -- 172.21.15.150:0/706315815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c80aabd0 con 0x7f17c8074450 2024-09-17T05:11:54.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17cea99640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17c8074450 0x7f17c814d010 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f17b800c660 tx=0x7f17b800cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:54.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17c6ffd640 1 -- 172.21.15.150:0/706315815 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8005220 con 0x7f17c8074450 2024-09-17T05:11:54.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17c6ffd640 1 -- 172.21.15.150:0/706315815 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17b80054d0 con 0x7f17c8074450 2024-09-17T05:11:54.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17d04fc640 1 -- 172.21.15.150:0/706315815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c81f0a10 con 0x7f17c8074450 2024-09-17T05:11:54.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17c6ffd640 1 -- 172.21.15.150:0/706315815 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8004a10 con 0x7f17c8074450 2024-09-17T05:11:54.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17d04fc640 1 -- 172.21.15.150:0/706315815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c81f0cf0 con 0x7f17c8074450 2024-09-17T05:11:54.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17d04fc640 1 -- 172.21.15.150:0/706315815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17c80ada60 con 0x7f17c8074450 2024-09-17T05:11:54.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17c6ffd640 1 -- 172.21.15.150:0/706315815 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f17b8004bb0 con 0x7f17c8074450 2024-09-17T05:11:54.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17c6ffd640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f17b4073230 0x7f17b4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:54.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.200+0000 7f17c6ffd640 1 -- 172.21.15.150:0/706315815 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f17b80951c0 con 0x7f17c8074450 2024-09-17T05:11:54.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.204+0000 7f17ce298640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f17b4073230 0x7f17b4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:54.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.204+0000 7f17ce298640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f17b4073230 0x7f17b4075740 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f17c001cb40 tx=0x7f17c001b550 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:54.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.208+0000 7f17c6ffd640 1 -- 172.21.15.150:0/706315815 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f17b8014030 con 0x7f17c8074450 2024-09-17T05:11:54.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.412+0000 7f17d04fc640 1 -- 172.21.15.150:0/706315815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17c80c60f0 con 0x7f17c8074450 2024-09-17T05:11:54.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.412+0000 7f17c6ffd640 1 -- 172.21.15.150:0/706315815 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+6547 (secure 0 0 0) 0x7f17b805ee10 con 0x7f17c8074450 2024-09-17T05:11:54.414 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:11:54.414 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":156,"btime":"2024-09-17T05:11:02:196586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7407,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:11:54.414 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 156 2024-09-17T05:11:54.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.416+0000 7f17d04fc640 1 -- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f17b4073230 msgr2=0x7f17b4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:54.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.416+0000 7f17d04fc640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f17b4073230 0x7f17b4075740 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f17c001cb40 tx=0x7f17c001b550 comp rx=0 tx=0).stop 2024-09-17T05:11:54.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.416+0000 7f17d04fc640 1 -- 172.21.15.150:0/706315815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17c8074450 msgr2=0x7f17c814d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:54.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.416+0000 7f17d04fc640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17c8074450 0x7f17c814d010 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f17b800c660 tx=0x7f17b800cb20 comp rx=0 tx=0).stop 2024-09-17T05:11:54.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.416+0000 7f17d04fc640 1 -- 172.21.15.150:0/706315815 shutdown_connections 2024-09-17T05:11:54.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.416+0000 7f17d04fc640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f17b4073230 0x7f17b4075740 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:54.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.416+0000 7f17d04fc640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17c8074450 0x7f17c814d010 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:54.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.416+0000 7f17d04fc640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f17c8073440 0x7f17c80ab390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:54.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.416+0000 7f17d04fc640 1 --2- 172.21.15.150:0/706315815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f17c8070e50 0x7f17c81515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:54.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.416+0000 7f17d04fc640 1 -- 172.21.15.150:0/706315815 >> 172.21.15.150:0/706315815 conn(0x7f17c80c4a00 msgr2=0x7f17c8152e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:54.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.416+0000 7f17d04fc640 1 -- 172.21.15.150:0/706315815 shutdown_connections 2024-09-17T05:11:54.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:54.416+0000 7f17d04fc640 1 -- 172.21.15.150:0/706315815 wait complete. 2024-09-17T05:11:54.436 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:11:56.437 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:11:56.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.600+0000 7f8a06e2b640 1 -- 172.21.15.150:0/2248302073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a000a7350 msgr2=0x7f8a00150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:56.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.600+0000 7f8a06e2b640 1 --2- 172.21.15.150:0/2248302073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a000a7350 0x7f8a00150060 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009900 tx=0x7f89f401c8e0 comp rx=0 tx=0).stop 2024-09-17T05:11:56.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.600+0000 7f8a06e2b640 1 -- 172.21.15.150:0/2248302073 shutdown_connections 2024-09-17T05:11:56.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.600+0000 7f8a06e2b640 1 --2- 172.21.15.150:0/2248302073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a000a7350 0x7f8a00150060 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:56.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.600+0000 7f8a06e2b640 1 -- 172.21.15.150:0/2248302073 >> 172.21.15.150:0/2248302073 conn(0x7f8a000c4a00 msgr2=0x7f8a000c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:56.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.600+0000 7f8a06e2b640 1 -- 172.21.15.150:0/2248302073 shutdown_connections 2024-09-17T05:11:56.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.600+0000 7f8a06e2b640 1 -- 172.21.15.150:0/2248302073 wait complete. 2024-09-17T05:11:56.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a06e2b640 1 Processor -- start 2024-09-17T05:11:56.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a06e2b640 1 -- start start 2024-09-17T05:11:56.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a06e2b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a000a7350 0x7f8a000c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:56.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a06e2b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a000c42e0 0x7f8a001f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:56.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a06e2b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a001505a0 0x7f8a00147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:56.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a06e2b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8a00160ea0 con 0x7f8a000a7350 2024-09-17T05:11:56.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a06e2b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a00153810 con 0x7f8a001505a0 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a06e2b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8a00161020 con 0x7f8a000c42e0 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a04bc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a000a7350 0x7f8a000c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a04bc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a000a7350 0x7f8a000c1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59778/0 (socket says 172.21.15.150:59778) 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a04bc7640 1 -- 172.21.15.150:0/1561852368 learned_addr learned my addr 172.21.15.150:0/1561852368 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f89fffff640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a000c42e0 0x7f8a001f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a053c8640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a001505a0 0x7f8a00147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a04bc7640 1 -- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a000c42e0 msgr2=0x7f8a001f2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a04bc7640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a000c42e0 0x7f8a001f2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a04bc7640 1 -- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a001505a0 msgr2=0x7f8a00147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a04bc7640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a001505a0 0x7f8a00147f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a04bc7640 1 -- 172.21.15.150:0/1561852368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a001540e0 con 0x7f8a000a7350 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a053c8640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a001505a0 0x7f8a00147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a04bc7640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a000a7350 0x7f8a000c1330 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f89f000c3f0 tx=0x7f89f000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:56.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f89fd7fa640 1 -- 172.21.15.150:0/1561852368 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f0017070 con 0x7f8a000a7350 2024-09-17T05:11:56.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f89fd7fa640 1 -- 172.21.15.150:0/1561852368 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89f00053c0 con 0x7f8a000a7350 2024-09-17T05:11:56.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a06e2b640 1 -- 172.21.15.150:0/1561852368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a00152490 con 0x7f8a000a7350 2024-09-17T05:11:56.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f89fd7fa640 1 -- 172.21.15.150:0/1561852368 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f0003860 con 0x7f8a000a7350 2024-09-17T05:11:56.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.604+0000 7f8a06e2b640 1 -- 172.21.15.150:0/1561852368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a001f0660 con 0x7f8a000a7350 2024-09-17T05:11:56.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.608+0000 7f8a06e2b640 1 -- 172.21.15.150:0/1561852368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a00150060 con 0x7f8a000a7350 2024-09-17T05:11:56.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.608+0000 7f89fd7fa640 1 -- 172.21.15.150:0/1561852368 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f89f001a070 con 0x7f8a000a7350 2024-09-17T05:11:56.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.608+0000 7f89fd7fa640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89ec073200 0x7f89ec075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:56.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.608+0000 7f89fd7fa640 1 -- 172.21.15.150:0/1561852368 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f89f0094210 con 0x7f8a000a7350 2024-09-17T05:11:56.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.608+0000 7f89fffff640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89ec073200 0x7f89ec075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:56.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.608+0000 7f89fffff640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89ec073200 0x7f89ec075710 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f89f40af000 tx=0x7f89f401b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:56.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.612+0000 7f89fd7fa640 1 -- 172.21.15.150:0/1561852368 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f89f005e540 con 0x7f8a000a7350 2024-09-17T05:11:56.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.816+0000 7f8a06e2b640 1 -- 172.21.15.150:0/1561852368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a000a7130 con 0x7f8a000a7350 2024-09-17T05:11:56.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.816+0000 7f89fd7fa640 1 -- 172.21.15.150:0/1561852368 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+6547 (secure 0 0 0) 0x7f89f005de40 con 0x7f8a000a7350 2024-09-17T05:11:56.818 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:11:56.819 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":156,"btime":"2024-09-17T05:11:02:196586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7407,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:11:56.819 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 156 2024-09-17T05:11:56.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.820+0000 7f8a06e2b640 1 -- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89ec073200 msgr2=0x7f89ec075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:56.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.820+0000 7f8a06e2b640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89ec073200 0x7f89ec075710 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f89f40af000 tx=0x7f89f401b120 comp rx=0 tx=0).stop 2024-09-17T05:11:56.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.820+0000 7f8a06e2b640 1 -- 172.21.15.150:0/1561852368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a000a7350 msgr2=0x7f8a000c1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:56.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.820+0000 7f8a06e2b640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a000a7350 0x7f8a000c1330 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f89f000c3f0 tx=0x7f89f000c8b0 comp rx=0 tx=0).stop 2024-09-17T05:11:56.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.820+0000 7f8a06e2b640 1 -- 172.21.15.150:0/1561852368 shutdown_connections 2024-09-17T05:11:56.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.820+0000 7f8a06e2b640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89ec073200 0x7f89ec075710 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:56.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.820+0000 7f8a06e2b640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a001505a0 0x7f8a00147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:56.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.820+0000 7f8a06e2b640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8a000c42e0 0x7f8a001f2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:56.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.820+0000 7f8a06e2b640 1 --2- 172.21.15.150:0/1561852368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a000a7350 0x7f8a000c1330 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:56.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.820+0000 7f8a06e2b640 1 -- 172.21.15.150:0/1561852368 >> 172.21.15.150:0/1561852368 conn(0x7f8a000c4a00 msgr2=0x7f8a000c6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:56.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.820+0000 7f8a06e2b640 1 -- 172.21.15.150:0/1561852368 shutdown_connections 2024-09-17T05:11:56.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:56.820+0000 7f8a06e2b640 1 -- 172.21.15.150:0/1561852368 wait complete. 2024-09-17T05:11:56.840 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:11:58.841 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:11:59.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.012+0000 7efd5c953640 1 -- 172.21.15.150:0/3413433260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd44004d50 msgr2=0x7efd440051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.012+0000 7efd5c953640 1 --2- 172.21.15.150:0/3413433260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd44004d50 0x7efd440051c0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7efd50009e00 tx=0x7efd5001d170 comp rx=0 tx=0).stop 2024-09-17T05:11:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.012+0000 7efd56d9d640 1 -- 172.21.15.150:0/3413433260 reap_dead start 2024-09-17T05:11:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.012+0000 7efd5c953640 1 -- 172.21.15.150:0/3413433260 shutdown_connections 2024-09-17T05:11:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.012+0000 7efd5c953640 1 -- 172.21.15.150:0/3413433260 >> 172.21.15.150:0/3413433260 conn(0x7efd580aaf60 msgr2=0x7efd580ab370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.012+0000 7efd5c953640 1 -- 172.21.15.150:0/3413433260 shutdown_connections 2024-09-17T05:11:59.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.012+0000 7efd5c953640 1 -- 172.21.15.150:0/3413433260 wait complete. 2024-09-17T05:11:59.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd5c953640 1 Processor -- start 2024-09-17T05:11:59.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd5c953640 1 -- start start 2024-09-17T05:11:59.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd5c953640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efd44004d50 0x7efd580ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:59.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd5c953640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd5806d020 0x7efd580ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:59.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd5c953640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd5806d430 0x7efd58147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:59.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd5c953640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efd580a8120 con 0x7efd5806d020 2024-09-17T05:11:59.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd5c953640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efd58070b30 con 0x7efd5806d430 2024-09-17T05:11:59.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd5c953640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efd580c63a0 con 0x7efd44004d50 2024-09-17T05:11:59.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd4dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd5806d020 0x7efd580ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:59.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd4dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd5806d020 0x7efd580ae0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59800/0 (socket says 172.21.15.150:59800) 2024-09-17T05:11:59.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd4dd9b640 1 -- 172.21.15.150:0/1629675626 learned_addr learned my addr 172.21.15.150:0/1629675626 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:11:59.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd56d9d640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd5806d430 0x7efd58147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:59.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd5659c640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efd44004d50 0x7efd580ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:59.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd4dd9b640 1 -- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efd44004d50 msgr2=0x7efd580ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:59.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd4dd9b640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efd44004d50 0x7efd580ad080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:59.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd4dd9b640 1 -- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd5806d430 msgr2=0x7efd58147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:59.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd4dd9b640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd5806d430 0x7efd58147f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:59.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd4dd9b640 1 -- 172.21.15.150:0/1629675626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd580ad5c0 con 0x7efd5806d020 2024-09-17T05:11:59.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd4dd9b640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd5806d020 0x7efd580ae0d0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7efd4000d780 tx=0x7efd4000dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:59.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd4f7fe640 1 -- 172.21.15.150:0/1629675626 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd40017070 con 0x7efd5806d020 2024-09-17T05:11:59.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd4f7fe640 1 -- 172.21.15.150:0/1629675626 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd4000de90 con 0x7efd5806d020 2024-09-17T05:11:59.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd4f7fe640 1 -- 172.21.15.150:0/1629675626 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd40005020 con 0x7efd5806d020 2024-09-17T05:11:59.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd5c953640 1 -- 172.21.15.150:0/1629675626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd581ea5d0 con 0x7efd5806d020 2024-09-17T05:11:59.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.016+0000 7efd5c953640 1 -- 172.21.15.150:0/1629675626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd580ac180 con 0x7efd5806d020 2024-09-17T05:11:59.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.020+0000 7efd5c953640 1 -- 172.21.15.150:0/1629675626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd581508a0 con 0x7efd5806d020 2024-09-17T05:11:59.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.020+0000 7efd4f7fe640 1 -- 172.21.15.150:0/1629675626 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7efd400051c0 con 0x7efd5806d020 2024-09-17T05:11:59.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.020+0000 7efd4f7fe640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efd38073230 0x7efd38075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:11:59.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.020+0000 7efd4f7fe640 1 -- 172.21.15.150:0/1629675626 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7efd400952f0 con 0x7efd5806d020 2024-09-17T05:11:59.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.020+0000 7efd5659c640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efd38073230 0x7efd38075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:11:59.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.020+0000 7efd5659c640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efd38073230 0x7efd38075740 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efd48001870 tx=0x7efd480019a0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:11:59.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.024+0000 7efd4f7fe640 1 -- 172.21.15.150:0/1629675626 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7efd580ad5c0 con 0x7efd5806d020 2024-09-17T05:11:59.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.232+0000 7efd5c953640 1 -- 172.21.15.150:0/1629675626 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd58072ed0 con 0x7efd5806d020 2024-09-17T05:11:59.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.232+0000 7efd4f7fe640 1 -- 172.21.15.150:0/1629675626 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+6547 (secure 0 0 0) 0x7efd4005f730 con 0x7efd5806d020 2024-09-17T05:11:59.232 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:11:59.232 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":156,"btime":"2024-09-17T05:11:02:196586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7407,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:09:49.306533+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7230":{"gid":7230,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":11,"addr":"172.21.15.150:6833/2056467678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2056467678},{"type":"v1","addr":"172.21.15.150:6833","nonce":2056467678}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7230,"qdb_cluster":[7230]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:11:59.232 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 156 2024-09-17T05:11:59.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.232+0000 7efd5c953640 1 -- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efd38073230 msgr2=0x7efd38075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:59.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.232+0000 7efd5c953640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efd38073230 0x7efd38075740 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efd48001870 tx=0x7efd480019a0 comp rx=0 tx=0).stop 2024-09-17T05:11:59.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.232+0000 7efd5c953640 1 -- 172.21.15.150:0/1629675626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd5806d020 msgr2=0x7efd580ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:11:59.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.232+0000 7efd5c953640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd5806d020 0x7efd580ae0d0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7efd4000d780 tx=0x7efd4000dc40 comp rx=0 tx=0).stop 2024-09-17T05:11:59.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.232+0000 7efd5c953640 1 -- 172.21.15.150:0/1629675626 shutdown_connections 2024-09-17T05:11:59.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.232+0000 7efd5c953640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efd38073230 0x7efd38075740 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:59.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.232+0000 7efd5c953640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd5806d430 0x7efd58147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:59.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.232+0000 7efd5c953640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd5806d020 0x7efd580ae0d0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:59.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.232+0000 7efd5c953640 1 --2- 172.21.15.150:0/1629675626 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efd44004d50 0x7efd580ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:11:59.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.232+0000 7efd5c953640 1 -- 172.21.15.150:0/1629675626 >> 172.21.15.150:0/1629675626 conn(0x7efd580aaf60 msgr2=0x7efd580c4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:11:59.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.236+0000 7efd5c953640 1 -- 172.21.15.150:0/1629675626 shutdown_connections 2024-09-17T05:11:59.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:11:59.236+0000 7efd5c953640 1 -- 172.21.15.150:0/1629675626 wait complete. 2024-09-17T05:11:59.254 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:12:01.256 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:01.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.400+0000 7f22e6eac640 1 -- 172.21.15.150:0/890862923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e01540b0 msgr2=0x7f22e00c8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:01.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.400+0000 7f22e6eac640 1 --2- 172.21.15.150:0/890862923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e01540b0 0x7f22e00c8330 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009900 tx=0x7f22cc01c8e0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.400+0000 7f22e6eac640 1 -- 172.21.15.150:0/890862923 shutdown_connections 2024-09-17T05:12:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.400+0000 7f22e6eac640 1 --2- 172.21.15.150:0/890862923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22e00c8870 0x7f22e00c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.400+0000 7f22e6eac640 1 --2- 172.21.15.150:0/890862923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e01540b0 0x7f22e00c8330 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.400+0000 7f22e6eac640 1 -- 172.21.15.150:0/890862923 >> 172.21.15.150:0/890862923 conn(0x7f22e00aab60 msgr2=0x7f22e00aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.400+0000 7f22e6eac640 1 -- 172.21.15.150:0/890862923 shutdown_connections 2024-09-17T05:12:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.400+0000 7f22e6eac640 1 -- 172.21.15.150:0/890862923 wait complete. 2024-09-17T05:12:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e6eac640 1 Processor -- start 2024-09-17T05:12:01.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e6eac640 1 -- start start 2024-09-17T05:12:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e6eac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e00c7eb0 0x7f22e0150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e6eac640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22e00c8870 0x7f22e00ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e6eac640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22e01540b0 0x7f22e014d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e6eac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f22e00c7ae0 con 0x7f22e00c7eb0 2024-09-17T05:12:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e6eac640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f22e00c7c60 con 0x7f22e00c8870 2024-09-17T05:12:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e6eac640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f22e0156540 con 0x7f22e01540b0 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e4c48640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e00c7eb0 0x7f22e0150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e4c48640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e00c7eb0 0x7f22e0150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50898/0 (socket says 172.21.15.150:50898) 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e4c48640 1 -- 172.21.15.150:0/926407579 learned_addr learned my addr 172.21.15.150:0/926407579 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e5449640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22e01540b0 0x7f22e014d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22dffff640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22e00c8870 0x7f22e00ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e4c48640 1 -- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22e01540b0 msgr2=0x7f22e014d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e4c48640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22e01540b0 0x7f22e014d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e4c48640 1 -- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22e00c8870 msgr2=0x7f22e00ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e4c48640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22e00c8870 0x7f22e00ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e4c48640 1 -- 172.21.15.150:0/926407579 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e01f1f10 con 0x7f22e00c7eb0 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e5449640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22e01540b0 0x7f22e014d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e4c48640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e00c7eb0 0x7f22e0150370 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f22d400c3f0 tx=0x7f22d400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22dd7fa640 1 -- 172.21.15.150:0/926407579 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d4017070 con 0x7f22e00c7eb0 2024-09-17T05:12:01.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22dd7fa640 1 -- 172.21.15.150:0/926407579 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22d40053c0 con 0x7f22e00c7eb0 2024-09-17T05:12:01.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e6eac640 1 -- 172.21.15.150:0/926407579 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22e01f21b0 con 0x7f22e00c7eb0 2024-09-17T05:12:01.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22dd7fa640 1 -- 172.21.15.150:0/926407579 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d4003860 con 0x7f22e00c7eb0 2024-09-17T05:12:01.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.404+0000 7f22e6eac640 1 -- 172.21.15.150:0/926407579 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22e014c3c0 con 0x7f22e00c7eb0 2024-09-17T05:12:01.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.408+0000 7f22e6eac640 1 -- 172.21.15.150:0/926407579 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22e0156900 con 0x7f22e00c7eb0 2024-09-17T05:12:01.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.408+0000 7f22dd7fa640 1 -- 172.21.15.150:0/926407579 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f22d401a070 con 0x7f22e00c7eb0 2024-09-17T05:12:01.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.408+0000 7f22dd7fa640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f22c4073200 0x7f22c4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:01.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.408+0000 7f22dd7fa640 1 -- 172.21.15.150:0/926407579 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f22d40947b0 con 0x7f22e00c7eb0 2024-09-17T05:12:01.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.408+0000 7f22dffff640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f22c4073200 0x7f22c4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:01.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.408+0000 7f22dffff640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f22c4073200 0x7f22c4075710 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f22cc0af000 tx=0x7f22cc01b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:01.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.412+0000 7f22dd7fa640 1 -- 172.21.15.150:0/926407579 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f22d4021020 con 0x7f22e00c7eb0 2024-09-17T05:12:01.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.612+0000 7f22e6eac640 1 -- 172.21.15.150:0/926407579 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22e00ae580 con 0x7f22e00c7eb0 2024-09-17T05:12:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.612+0000 7f22dd7fa640 1 -- 172.21.15.150:0/926407579 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+5710 (secure 0 0 0) 0x7f22d405ea90 con 0x7f22e00c7eb0 2024-09-17T05:12:01.614 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:01.614 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":157,"btime":"2024-09-17T05:12:01:174200+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:01.174189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 157 2024-09-17T05:12:01.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.616+0000 7f22e6eac640 1 -- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f22c4073200 msgr2=0x7f22c4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:01.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.616+0000 7f22e6eac640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f22c4073200 0x7f22c4075710 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f22cc0af000 tx=0x7f22cc01b120 comp rx=0 tx=0).stop 2024-09-17T05:12:01.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.616+0000 7f22e6eac640 1 -- 172.21.15.150:0/926407579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e00c7eb0 msgr2=0x7f22e0150370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:01.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.616+0000 7f22e6eac640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e00c7eb0 0x7f22e0150370 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f22d400c3f0 tx=0x7f22d400c8b0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.616+0000 7f22e6eac640 1 -- 172.21.15.150:0/926407579 shutdown_connections 2024-09-17T05:12:01.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.616+0000 7f22e6eac640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f22c4073200 0x7f22c4075710 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.616+0000 7f22e6eac640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22e00c8870 0x7f22e00ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.616+0000 7f22e6eac640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22e01540b0 0x7f22e014d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.616+0000 7f22e6eac640 1 --2- 172.21.15.150:0/926407579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22e00c7eb0 0x7f22e0150370 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.616+0000 7f22e6eac640 1 -- 172.21.15.150:0/926407579 >> 172.21.15.150:0/926407579 conn(0x7f22e00aab60 msgr2=0x7f22e00aa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:01.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.616+0000 7f22e6eac640 1 -- 172.21.15.150:0/926407579 shutdown_connections 2024-09-17T05:12:01.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.616+0000 7f22e6eac640 1 -- 172.21.15.150:0/926407579 wait complete. 2024-09-17T05:12:01.639 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-17T05:12:01.639 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:12:01.639 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:01.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.788+0000 7fb7e3733640 1 -- 172.21.15.150:0/2186448647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc0aa9b0 msgr2=0x7fb7dc0aadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:01.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.788+0000 7fb7e3733640 1 --2- 172.21.15.150:0/2186448647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc0aa9b0 0x7fb7dc0aadc0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc073740 tx=0x7fb7d401cc60 comp rx=0 tx=0).stop 2024-09-17T05:12:01.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.788+0000 7fb7e3733640 1 -- 172.21.15.150:0/2186448647 shutdown_connections 2024-09-17T05:12:01.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.788+0000 7fb7e3733640 1 --2- 172.21.15.150:0/2186448647 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc0aa9b0 0x7fb7dc0aadc0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.788+0000 7fb7e3733640 1 --2- 172.21.15.150:0/2186448647 >> v2:172.21.15.176:3300/0 conn(0x7fb7dc06e620 0x7fb7dc154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.788+0000 7fb7e3733640 1 -- 172.21.15.150:0/2186448647 >> 172.21.15.150:0/2186448647 conn(0x7fb7dc0c5ae0 msgr2=0x7fb7dc073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:01.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.788+0000 7fb7e3733640 1 -- 172.21.15.150:0/2186448647 shutdown_connections 2024-09-17T05:12:01.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.788+0000 7fb7e3733640 1 -- 172.21.15.150:0/2186448647 wait complete. 2024-09-17T05:12:01.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e3733640 1 Processor -- start 2024-09-17T05:12:01.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e3733640 1 -- start start 2024-09-17T05:12:01.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e3733640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7dc0aa9b0 0x7fb7dc0aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:01.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e3733640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc150de0 0x7fb7dc0a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:01.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e3733640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7dc1502a0 0x7fb7dc148cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:01.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e3733640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb7dc06fca0 con 0x7fb7dc150de0 2024-09-17T05:12:01.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e3733640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb7dc070820 con 0x7fb7dc0aa9b0 2024-09-17T05:12:01.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e3733640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb7dc06fe20 con 0x7fb7dc1502a0 2024-09-17T05:12:01.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e0cce640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc150de0 0x7fb7dc0a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:01.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e0cce640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc150de0 0x7fb7dc0a98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50918/0 (socket says 172.21.15.150:50918) 2024-09-17T05:12:01.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e0cce640 1 -- 172.21.15.150:0/3867306587 learned_addr learned my addr 172.21.15.150:0/3867306587 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:01.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e1cd0640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7dc1502a0 0x7fb7dc148cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:01.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e14cf640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7dc0aa9b0 0x7fb7dc0aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:01.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e0cce640 1 -- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7dc1502a0 msgr2=0x7fb7dc148cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:01.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e0cce640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7dc1502a0 0x7fb7dc148cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e0cce640 1 -- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7dc0aa9b0 msgr2=0x7fb7dc0aceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:01.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e0cce640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7dc0aa9b0 0x7fb7dc0aceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e0cce640 1 -- 172.21.15.150:0/3867306587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7dc0c0a30 con 0x7fb7dc150de0 2024-09-17T05:12:01.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e0cce640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc150de0 0x7fb7dc0a98a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800c3c0 tx=0x7fb7d800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:01.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7d1ffb640 1 -- 172.21.15.150:0/3867306587 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8017070 con 0x7fb7dc150de0 2024-09-17T05:12:01.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7d1ffb640 1 -- 172.21.15.150:0/3867306587 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7d80053c0 con 0x7fb7dc150de0 2024-09-17T05:12:01.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7d1ffb640 1 -- 172.21.15.150:0/3867306587 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8003840 con 0x7fb7dc150de0 2024-09-17T05:12:01.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e3733640 1 -- 172.21.15.150:0/3867306587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7dc0c0d10 con 0x7fb7dc150de0 2024-09-17T05:12:01.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e3733640 1 -- 172.21.15.150:0/3867306587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7dc0aa2c0 con 0x7fb7dc150de0 2024-09-17T05:12:01.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.792+0000 7fb7e3733640 1 -- 172.21.15.150:0/3867306587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7dc0aa580 con 0x7fb7dc150de0 2024-09-17T05:12:01.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.796+0000 7fb7d1ffb640 1 -- 172.21.15.150:0/3867306587 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb7d801a070 con 0x7fb7dc150de0 2024-09-17T05:12:01.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.796+0000 7fb7d1ffb640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb7c0073200 0x7fb7c0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:01.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.796+0000 7fb7d1ffb640 1 -- 172.21.15.150:0/3867306587 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7fb7d8094310 con 0x7fb7dc150de0 2024-09-17T05:12:01.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.796+0000 7fb7e14cf640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb7c0073200 0x7fb7c0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:01.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.796+0000 7fb7e14cf640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb7c0073200 0x7fb7c0075710 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0058d0 tx=0x7fb7cc005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:01.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.800+0000 7fb7d1ffb640 1 -- 172.21.15.150:0/3867306587 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb7d805e570 con 0x7fb7dc150de0 2024-09-17T05:12:01.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.988+0000 7fb7e3733640 1 -- 172.21.15.150:0/3867306587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7dc06e620 con 0x7fb7dc150de0 2024-09-17T05:12:01.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.988+0000 7fb7d1ffb640 1 -- 172.21.15.150:0/3867306587 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+5710 (secure 0 0 0) 0x7fb7d805de70 con 0x7fb7dc150de0 2024-09-17T05:12:01.989 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:01.989 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":157,"btime":"2024-09-17T05:12:01:174200+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:01.174189+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:01.989 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 157 2024-09-17T05:12:01.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.992+0000 7fb7e3733640 1 -- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb7c0073200 msgr2=0x7fb7c0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:01.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.992+0000 7fb7e3733640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb7c0073200 0x7fb7c0075710 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0058d0 tx=0x7fb7cc005860 comp rx=0 tx=0).stop 2024-09-17T05:12:01.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.992+0000 7fb7e3733640 1 -- 172.21.15.150:0/3867306587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc150de0 msgr2=0x7fb7dc0a98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:01.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.992+0000 7fb7e3733640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc150de0 0x7fb7dc0a98a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800c3c0 tx=0x7fb7d800c880 comp rx=0 tx=0).stop 2024-09-17T05:12:01.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.992+0000 7fb7e3733640 1 -- 172.21.15.150:0/3867306587 shutdown_connections 2024-09-17T05:12:01.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.992+0000 7fb7e3733640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb7c0073200 0x7fb7c0075710 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.992+0000 7fb7e3733640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb7dc1502a0 0x7fb7dc148cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.992+0000 7fb7e3733640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7dc150de0 0x7fb7dc0a98a0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.992+0000 7fb7e3733640 1 --2- 172.21.15.150:0/3867306587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7dc0aa9b0 0x7fb7dc0aceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:01.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.992+0000 7fb7e3733640 1 -- 172.21.15.150:0/3867306587 >> 172.21.15.150:0/3867306587 conn(0x7fb7dc0c5ae0 msgr2=0x7fb7dc14f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:01.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.992+0000 7fb7e3733640 1 -- 172.21.15.150:0/3867306587 shutdown_connections 2024-09-17T05:12:01.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:01.992+0000 7fb7e3733640 1 -- 172.21.15.150:0/3867306587 wait complete. 2024-09-17T05:12:02.012 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:12:01:174200+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 157, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 157, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7407": { "addr": "172.21.15.150:6835/3179072684", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 3179072684, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 3179072684, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7407, "incarnation": 157, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:12:01.174189+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7407 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 155, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6799": { "addr": "172.21.15.176:6837/2962113435", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 2962113435, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 2962113435, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6799, "incarnation": 152, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:10:54.175436+0000", "qdb_cluster": [ 6799 ], "qdb_leader": 6799, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6799 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:12:02.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:02.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 -- 172.21.15.150:0/2896175246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64a4073440 msgr2=0x7f64a40a7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:02.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 --2- 172.21.15.150:0/2896175246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64a4073440 0x7f64a40a7300 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009900 tx=0x7f64a001c850 comp rx=0 tx=0).stop 2024-09-17T05:12:02.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 -- 172.21.15.150:0/2896175246 shutdown_connections 2024-09-17T05:12:02.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 --2- 172.21.15.150:0/2896175246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64a4073440 0x7f64a40a7300 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:02.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 -- 172.21.15.150:0/2896175246 >> 172.21.15.150:0/2896175246 conn(0x7f64a40c4a00 msgr2=0x7f64a40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:02.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 -- 172.21.15.150:0/2896175246 shutdown_connections 2024-09-17T05:12:02.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 -- 172.21.15.150:0/2896175246 wait complete. 2024-09-17T05:12:02.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 Processor -- start 2024-09-17T05:12:02.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 -- start start 2024-09-17T05:12:02.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64a4070e50 0x7f64a4149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:02.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64a4073440 0x7f64a40abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:02.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64a4074450 0x7f64a41432e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:02.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f64a4074f70 con 0x7f64a4074450 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f64a40adef0 con 0x7f64a4070e50 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64ab65b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f64a40750f0 con 0x7f64a4073440 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64a9bf8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64a4074450 0x7f64a41432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64a9bf8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64a4074450 0x7f64a41432e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50932/0 (socket says 172.21.15.150:50932) 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.476+0000 7f64a9bf8640 1 -- 172.21.15.150:0/807236909 learned_addr learned my addr 172.21.15.150:0/807236909 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64a93f7640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64a4070e50 0x7f64a4149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64a8bf6640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64a4073440 0x7f64a40abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64a9bf8640 1 -- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64a4073440 msgr2=0x7f64a40abcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64a9bf8640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64a4073440 0x7f64a40abcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64a9bf8640 1 -- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64a4070e50 msgr2=0x7f64a4149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64a9bf8640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64a4070e50 0x7f64a4149810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64a9bf8640 1 -- 172.21.15.150:0/807236909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a40aacb0 con 0x7f64a4074450 2024-09-17T05:12:02.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64a9bf8640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64a4074450 0x7f64a41432e0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f649c00e9d0 tx=0x7f649c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:02.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f6499ffb640 1 -- 172.21.15.150:0/807236909 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c001f20 con 0x7f64a4074450 2024-09-17T05:12:02.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f6499ffb640 1 -- 172.21.15.150:0/807236909 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f649c003980 con 0x7f64a4074450 2024-09-17T05:12:02.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64ab65b640 1 -- 172.21.15.150:0/807236909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64a40ab7b0 con 0x7f64a4074450 2024-09-17T05:12:02.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f6499ffb640 1 -- 172.21.15.150:0/807236909 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c003d90 con 0x7f64a4074450 2024-09-17T05:12:02.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64ab65b640 1 -- 172.21.15.150:0/807236909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64a41effb0 con 0x7f64a4074450 2024-09-17T05:12:02.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64ab65b640 1 -- 172.21.15.150:0/807236909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64a40ad8c0 con 0x7f64a4074450 2024-09-17T05:12:02.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f6499ffb640 1 -- 172.21.15.150:0/807236909 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f649c0050d0 con 0x7f64a4074450 2024-09-17T05:12:02.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f6499ffb640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6488073230 0x7f6488075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:02.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f6499ffb640 1 -- 172.21.15.150:0/807236909 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f649c097540 con 0x7f64a4074450 2024-09-17T05:12:02.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64a93f7640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6488073230 0x7f6488075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:02.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.480+0000 7f64a93f7640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6488073230 0x7f6488075740 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f64a00af000 tx=0x7f64a001b090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:02.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.484+0000 7f6499ffb640 1 -- 172.21.15.150:0/807236909 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f649c016020 con 0x7f64a4074450 2024-09-17T05:12:02.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 -- 172.21.15.150:0/807236909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64a40c60f0 con 0x7f64a4074450 2024-09-17T05:12:02.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f6499ffb640 1 -- 172.21.15.150:0/807236909 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) ==== 95+0+5714 (secure 0 0 0) 0x7f649c066e20 con 0x7f64a4074450 2024-09-17T05:12:02.692 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:02.692 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":158,"btime":"2024-09-17T05:12:02:445124+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:01.628025+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:02.692 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 158 2024-09-17T05:12:02.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 -- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6488073230 msgr2=0x7f6488075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:02.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6488073230 0x7f6488075740 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f64a00af000 tx=0x7f64a001b090 comp rx=0 tx=0).stop 2024-09-17T05:12:02.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 -- 172.21.15.150:0/807236909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64a4074450 msgr2=0x7f64a41432e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:02.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64a4074450 0x7f64a41432e0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f649c00e9d0 tx=0x7f649c00ee90 comp rx=0 tx=0).stop 2024-09-17T05:12:02.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 -- 172.21.15.150:0/807236909 shutdown_connections 2024-09-17T05:12:02.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6488073230 0x7f6488075740 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:02.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64a4074450 0x7f64a41432e0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:02.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f64a4073440 0x7f64a40abcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:02.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 --2- 172.21.15.150:0/807236909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64a4070e50 0x7f64a4149810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:02.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 -- 172.21.15.150:0/807236909 >> 172.21.15.150:0/807236909 conn(0x7f64a40c4a00 msgr2=0x7f64a415f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:02.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 -- 172.21.15.150:0/807236909 shutdown_connections 2024-09-17T05:12:02.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.692+0000 7f64ab65b640 1 -- 172.21.15.150:0/807236909 wait complete. 2024-09-17T05:12:02.716 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-17T05:12:02.716 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T05:12:02.716 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:12:02.741 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T05:12:02.741 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:02.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.892+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/3918942053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6b80c42e0 msgr2=0x7ff6b80c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:02.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.892+0000 7ff6bfb1b640 1 --2- 172.21.15.150:0/3918942053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6b80c42e0 0x7ff6b80c6690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4009900 tx=0x7ff6b401c8e0 comp rx=0 tx=0).stop 2024-09-17T05:12:02.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.892+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/3918942053 shutdown_connections 2024-09-17T05:12:02.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.892+0000 7ff6bfb1b640 1 --2- 172.21.15.150:0/3918942053 >> v2:172.21.15.176:3300/0 conn(0x7ff6b80a7350 0x7ff6b8150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:02.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.892+0000 7ff6bfb1b640 1 --2- 172.21.15.150:0/3918942053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6b80c42e0 0x7ff6b80c6690 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:02.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.892+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/3918942053 >> 172.21.15.150:0/3918942053 conn(0x7ff6b80c4a00 msgr2=0x7ff6b80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:02.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.892+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/3918942053 shutdown_connections 2024-09-17T05:12:02.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.892+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/3918942053 wait complete. 2024-09-17T05:12:02.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bfb1b640 1 Processor -- start 2024-09-17T05:12:02.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bfb1b640 1 -- start start 2024-09-17T05:12:02.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bfb1b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6b80a7350 0x7ff6b80c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:02.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bfb1b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6b80c42e0 0x7ff6b81f02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:02.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bfb1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6b81505a0 0x7ff6b8147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:02.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bfb1b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b80c6bd0 con 0x7ff6b81505a0 2024-09-17T05:12:02.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bfb1b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b8070b80 con 0x7ff6b80c42e0 2024-09-17T05:12:02.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bfb1b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff6b80c6d50 con 0x7ff6b80a7350 2024-09-17T05:12:02.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6be0b8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6b81505a0 0x7ff6b8147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:02.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6be0b8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6b81505a0 0x7ff6b8147f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50942/0 (socket says 172.21.15.150:50942) 2024-09-17T05:12:02.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6be0b8640 1 -- 172.21.15.150:0/1236708793 learned_addr learned my addr 172.21.15.150:0/1236708793 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bd8b7640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6b80a7350 0x7ff6b80c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bd0b6640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6b80c42e0 0x7ff6b81f02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6be0b8640 1 -- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6b80a7350 msgr2=0x7ff6b80c1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6be0b8640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6b80a7350 0x7ff6b80c1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6be0b8640 1 -- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6b80c42e0 msgr2=0x7ff6b81f02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6be0b8640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6b80c42e0 0x7ff6b81f02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6be0b8640 1 -- 172.21.15.150:0/1236708793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b80abf50 con 0x7ff6b81505a0 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6be0b8640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6b81505a0 0x7ff6b8147f70 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000c730 tx=0x7ff6b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6ae7fc640 1 -- 172.21.15.150:0/1236708793 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0003b80 con 0x7ff6b81505a0 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/1236708793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b80a82f0 con 0x7ff6b81505a0 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6ae7fc640 1 -- 172.21.15.150:0/1236708793 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff6b0003d20 con 0x7ff6b81505a0 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6ae7fc640 1 -- 172.21.15.150:0/1236708793 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b000aab0 con 0x7ff6b81505a0 2024-09-17T05:12:02.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/1236708793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6b81f3c10 con 0x7ff6b81505a0 2024-09-17T05:12:02.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.896+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/1236708793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6b80ac120 con 0x7ff6b81505a0 2024-09-17T05:12:02.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.900+0000 7ff6ae7fc640 1 -- 172.21.15.150:0/1236708793 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff6b000ac50 con 0x7ff6b81505a0 2024-09-17T05:12:02.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.900+0000 7ff6ae7fc640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff69c073200 0x7ff69c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:02.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.900+0000 7ff6ae7fc640 1 -- 172.21.15.150:0/1236708793 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7ff6b0095820 con 0x7ff6b81505a0 2024-09-17T05:12:02.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.900+0000 7ff6bd8b7640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff69c073200 0x7ff69c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:02.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.900+0000 7ff6bd8b7640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff69c073200 0x7ff69c075710 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40af000 tx=0x7ff6b401b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:02.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:02.904+0000 7ff6ae7fc640 1 -- 172.21.15.150:0/1236708793 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff6b005fb00 con 0x7ff6b81505a0 2024-09-17T05:12:03.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.112+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/1236708793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6b8150060 con 0x7ff6b81505a0 2024-09-17T05:12:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.112+0000 7ff6ae7fc640 1 -- 172.21.15.150:0/1236708793 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) ==== 95+0+5714 (secure 0 0 0) 0x7ff6b005f400 con 0x7ff6b81505a0 2024-09-17T05:12:03.113 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:03.113 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":158,"btime":"2024-09-17T05:12:02:445124+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:01.628025+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:03.113 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 158 2024-09-17T05:12:03.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.116+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff69c073200 msgr2=0x7ff69c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:03.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.116+0000 7ff6bfb1b640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff69c073200 0x7ff69c075710 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40af000 tx=0x7ff6b401b120 comp rx=0 tx=0).stop 2024-09-17T05:12:03.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.116+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/1236708793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6b81505a0 msgr2=0x7ff6b8147f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:03.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.116+0000 7ff6bfb1b640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6b81505a0 0x7ff6b8147f70 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000c730 tx=0x7ff6b000cbf0 comp rx=0 tx=0).stop 2024-09-17T05:12:03.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.116+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/1236708793 shutdown_connections 2024-09-17T05:12:03.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.116+0000 7ff6bfb1b640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff69c073200 0x7ff69c075710 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:03.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.116+0000 7ff6bfb1b640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff6b80c42e0 0x7ff6b81f02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:03.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.116+0000 7ff6bfb1b640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6b81505a0 0x7ff6b8147f70 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:03.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.116+0000 7ff6bfb1b640 1 --2- 172.21.15.150:0/1236708793 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff6b80a7350 0x7ff6b80c1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:03.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.116+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/1236708793 >> 172.21.15.150:0/1236708793 conn(0x7ff6b80c4a00 msgr2=0x7ff6b80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:03.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.116+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/1236708793 shutdown_connections 2024-09-17T05:12:03.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:03.116+0000 7ff6bfb1b640 1 -- 172.21.15.150:0/1236708793 wait complete. 2024-09-17T05:12:03.135 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:12:04.014 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:04.041 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:12:04.060 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:12:04.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.180+0000 7f16a572f640 1 -- 172.21.15.150:0/4156694666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c004810 msgr2=0x7f168c004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:04.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.180+0000 7f16a572f640 1 --2- 172.21.15.150:0/4156694666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c004810 0x7f168c004c20 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f1688009b60 tx=0x7f168801d230 comp rx=0 tx=0).stop 2024-09-17T05:12:04.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.180+0000 7f169f7fe640 1 -- 172.21.15.150:0/4156694666 reap_dead start 2024-09-17T05:12:04.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.180+0000 7f16a572f640 1 -- 172.21.15.150:0/4156694666 shutdown_connections 2024-09-17T05:12:04.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.180+0000 7f16a572f640 1 -- 172.21.15.150:0/4156694666 >> 172.21.15.150:0/4156694666 conn(0x7f16a00c5ae0 msgr2=0x7f16a0073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:04.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.180+0000 7f16a572f640 1 -- 172.21.15.150:0/4156694666 shutdown_connections 2024-09-17T05:12:04.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.180+0000 7f16a572f640 1 -- 172.21.15.150:0/4156694666 wait complete. 2024-09-17T05:12:04.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16a572f640 1 Processor -- start 2024-09-17T05:12:04.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16a572f640 1 -- start start 2024-09-17T05:12:04.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16a572f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c004810 0x7f16a01489c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:04.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16a572f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16a014c990 0x7f16a014e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:04.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16a572f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16a01490f0 0x7f16a014e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:04.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16a572f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f16a00aadd0 con 0x7f168c004810 2024-09-17T05:12:04.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16a572f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f16a00aac50 con 0x7f16a01490f0 2024-09-17T05:12:04.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16a572f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f16a00aaf50 con 0x7f16a014c990 2024-09-17T05:12:04.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f169effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c004810 0x7f16a01489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:04.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f169effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c004810 0x7f16a01489c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50950/0 (socket says 172.21.15.150:50950) 2024-09-17T05:12:04.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f169effd640 1 -- 172.21.15.150:0/75647690 learned_addr learned my addr 172.21.15.150:0/75647690 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:04.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f169e7fc640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16a014c990 0x7f16a014e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:04.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f169f7fe640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16a01490f0 0x7f16a014e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:04.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f169effd640 1 -- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16a014c990 msgr2=0x7f16a014e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:04.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f169effd640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16a014c990 0x7f16a014e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f169effd640 1 -- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16a01490f0 msgr2=0x7f16a014e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f169effd640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16a01490f0 0x7f16a014e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f169effd640 1 -- 172.21.15.150:0/75647690 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a0148f00 con 0x7f168c004810 2024-09-17T05:12:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f169effd640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c004810 0x7f16a01489c0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f169800e590 tx=0x7f169800ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16977fe640 1 -- 172.21.15.150:0/75647690 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1698015070 con 0x7f168c004810 2024-09-17T05:12:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16977fe640 1 -- 172.21.15.150:0/75647690 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f169800c9e0 con 0x7f168c004810 2024-09-17T05:12:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16a572f640 1 -- 172.21.15.150:0/75647690 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a014e5c0 con 0x7f168c004810 2024-09-17T05:12:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16977fe640 1 -- 172.21.15.150:0/75647690 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169800ce30 con 0x7f168c004810 2024-09-17T05:12:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.184+0000 7f16a572f640 1 -- 172.21.15.150:0/75647690 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a014bc20 con 0x7f168c004810 2024-09-17T05:12:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.188+0000 7f16a572f640 1 -- 172.21.15.150:0/75647690 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16a00c3fb0 con 0x7f168c004810 2024-09-17T05:12:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.188+0000 7f16977fe640 1 -- 172.21.15.150:0/75647690 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1698018070 con 0x7f168c004810 2024-09-17T05:12:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.188+0000 7f16977fe640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1680073200 0x7f1680075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:04.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.188+0000 7f16977fe640 1 -- 172.21.15.150:0/75647690 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f16980953a0 con 0x7f168c004810 2024-09-17T05:12:04.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.188+0000 7f169e7fc640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1680073200 0x7f1680075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:04.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.188+0000 7f169e7fc640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1680073200 0x7f1680075710 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f16880ae000 tx=0x7f168801a030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:04.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.192+0000 7f16977fe640 1 -- 172.21.15.150:0/75647690 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f169805f030 con 0x7f168c004810 2024-09-17T05:12:04.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.400+0000 7f16a572f640 1 -- 172.21.15.150:0/75647690 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16a00c2b30 con 0x7f168c004810 2024-09-17T05:12:04.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.400+0000 7f16977fe640 1 -- 172.21.15.150:0/75647690 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) ==== 95+0+5711 (secure 0 0 0) 0x7f1698063a60 con 0x7f168c004810 2024-09-17T05:12:04.400 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:04.400 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":159,"btime":"2024-09-17T05:12:03:457778+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:02.466051+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:04.401 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 159 2024-09-17T05:12:04.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.404+0000 7f16a572f640 1 -- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1680073200 msgr2=0x7f1680075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:04.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.404+0000 7f16a572f640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1680073200 0x7f1680075710 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f16880ae000 tx=0x7f168801a030 comp rx=0 tx=0).stop 2024-09-17T05:12:04.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.404+0000 7f16a572f640 1 -- 172.21.15.150:0/75647690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c004810 msgr2=0x7f16a01489c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:04.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.404+0000 7f16a572f640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c004810 0x7f16a01489c0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f169800e590 tx=0x7f169800ea50 comp rx=0 tx=0).stop 2024-09-17T05:12:04.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.404+0000 7f16a572f640 1 -- 172.21.15.150:0/75647690 shutdown_connections 2024-09-17T05:12:04.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.404+0000 7f16a572f640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1680073200 0x7f1680075710 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:04.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.404+0000 7f16a572f640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16a01490f0 0x7f16a014e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:04.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.404+0000 7f16a572f640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f16a014c990 0x7f16a014e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:04.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.404+0000 7f16a572f640 1 --2- 172.21.15.150:0/75647690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f168c004810 0x7f16a01489c0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:04.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.404+0000 7f16a572f640 1 -- 172.21.15.150:0/75647690 >> 172.21.15.150:0/75647690 conn(0x7f16a00c5ae0 msgr2=0x7f16a006ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:04.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.404+0000 7f16a572f640 1 -- 172.21.15.150:0/75647690 shutdown_connections 2024-09-17T05:12:04.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:04.404+0000 7f16a572f640 1 -- 172.21.15.150:0/75647690 wait complete. 2024-09-17T05:12:04.424 INFO:tasks.mds_thrash.fs.[a]:no change 2024-09-17T05:12:05.137 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:05.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.316+0000 7fcf2d73b640 1 -- 172.21.15.150:0/1389583795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf2806e5d0 msgr2=0x7fcf28153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:05.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.316+0000 7fcf2d73b640 1 --2- 172.21.15.150:0/1389583795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf2806e5d0 0x7fcf28153660 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fcf28069a20 tx=0x7fcf2001ccf0 comp rx=0 tx=0).stop 2024-09-17T05:12:05.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.316+0000 7fcf2d73b640 1 -- 172.21.15.150:0/1389583795 shutdown_connections 2024-09-17T05:12:05.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.316+0000 7fcf2d73b640 1 --2- 172.21.15.150:0/1389583795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf2806e5d0 0x7fcf28153660 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:05.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.316+0000 7fcf2d73b640 1 --2- 172.21.15.150:0/1389583795 >> v2:172.21.15.176:3300/0 conn(0x7fcf2806fca0 0x7fcf2806e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:05.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.316+0000 7fcf2d73b640 1 --2- 172.21.15.150:0/1389583795 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf280745c0 0x7fcf28150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:05.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.316+0000 7fcf2d73b640 1 -- 172.21.15.150:0/1389583795 >> 172.21.15.150:0/1389583795 conn(0x7fcf280c5ae0 msgr2=0x7fcf28073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:05.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.316+0000 7fcf2d73b640 1 -- 172.21.15.150:0/1389583795 shutdown_connections 2024-09-17T05:12:05.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.316+0000 7fcf2d73b640 1 -- 172.21.15.150:0/1389583795 wait complete. 2024-09-17T05:12:05.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf2d73b640 1 Processor -- start 2024-09-17T05:12:05.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf2d73b640 1 -- start start 2024-09-17T05:12:05.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf2d73b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf2806e5d0 0x7fcf28156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:05.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf2d73b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf2806fca0 0x7fcf280c14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:05.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf2d73b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf280745c0 0x7fcf28149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:05.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf2d73b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcf280c31a0 con 0x7fcf2806e5d0 2024-09-17T05:12:05.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf2d73b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcf280c3020 con 0x7fcf280745c0 2024-09-17T05:12:05.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf2d73b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcf28070610 con 0x7fcf2806fca0 2024-09-17T05:12:05.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf26ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf2806e5d0 0x7fcf28156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:05.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf26ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf2806e5d0 0x7fcf28156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50964/0 (socket says 172.21.15.150:50964) 2024-09-17T05:12:05.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf26ffd640 1 -- 172.21.15.150:0/3873828941 learned_addr learned my addr 172.21.15.150:0/3873828941 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:05.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf277fe640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf280745c0 0x7fcf28149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:05.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf267fc640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf2806fca0 0x7fcf280c14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:05.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf26ffd640 1 -- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf2806fca0 msgr2=0x7fcf280c14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:05.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf26ffd640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf2806fca0 0x7fcf280c14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:05.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf26ffd640 1 -- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf280745c0 msgr2=0x7fcf28149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:05.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf26ffd640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf280745c0 0x7fcf28149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:05.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf26ffd640 1 -- 172.21.15.150:0/3873828941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf280bf890 con 0x7fcf2806e5d0 2024-09-17T05:12:05.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf26ffd640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf2806e5d0 0x7fcf28156260 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800c970 tx=0x7fcf1800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:05.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf0f7fe640 1 -- 172.21.15.150:0/3873828941 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf18004220 con 0x7fcf2806e5d0 2024-09-17T05:12:05.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf0f7fe640 1 -- 172.21.15.150:0/3873828941 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf180079b0 con 0x7fcf2806e5d0 2024-09-17T05:12:05.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf0f7fe640 1 -- 172.21.15.150:0/3873828941 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf180044b0 con 0x7fcf2806e5d0 2024-09-17T05:12:05.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf2d73b640 1 -- 172.21.15.150:0/3873828941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf281f1e00 con 0x7fcf2806e5d0 2024-09-17T05:12:05.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf2d73b640 1 -- 172.21.15.150:0/3873828941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf281536e0 con 0x7fcf2806e5d0 2024-09-17T05:12:05.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.320+0000 7fcf2d73b640 1 -- 172.21.15.150:0/3873828941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf280c7310 con 0x7fcf2806e5d0 2024-09-17T05:12:05.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.324+0000 7fcf0f7fe640 1 -- 172.21.15.150:0/3873828941 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fcf18003600 con 0x7fcf2806e5d0 2024-09-17T05:12:05.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.324+0000 7fcf0f7fe640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf14073200 0x7fcf14075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:05.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.324+0000 7fcf0f7fe640 1 -- 172.21.15.150:0/3873828941 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7fcf180950a0 con 0x7fcf2806e5d0 2024-09-17T05:12:05.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.324+0000 7fcf267fc640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf14073200 0x7fcf14075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:05.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.324+0000 7fcf267fc640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf14073200 0x7fcf14075710 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c006fd0 tx=0x7fcf1c008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:05.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.332+0000 7fcf0f7fe640 1 -- 172.21.15.150:0/3873828941 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcf18014030 con 0x7fcf2806e5d0 2024-09-17T05:12:05.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.560+0000 7fcf2d73b640 1 -- 172.21.15.150:0/3873828941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf280ab2b0 con 0x7fcf2806e5d0 2024-09-17T05:12:05.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.560+0000 7fcf0f7fe640 1 -- 172.21.15.150:0/3873828941 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+5718 (secure 0 0 0) 0x7fcf1805f380 con 0x7fcf2806e5d0 2024-09-17T05:12:05.561 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:05.561 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":160,"btime":"2024-09-17T05:12:04:469094+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:05.561 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 160 2024-09-17T05:12:05.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.564+0000 7fcf2d73b640 1 -- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf14073200 msgr2=0x7fcf14075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:05.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.564+0000 7fcf2d73b640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf14073200 0x7fcf14075710 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c006fd0 tx=0x7fcf1c008040 comp rx=0 tx=0).stop 2024-09-17T05:12:05.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.564+0000 7fcf2d73b640 1 -- 172.21.15.150:0/3873828941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf2806e5d0 msgr2=0x7fcf28156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:05.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.564+0000 7fcf2d73b640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf2806e5d0 0x7fcf28156260 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800c970 tx=0x7fcf1800ce30 comp rx=0 tx=0).stop 2024-09-17T05:12:05.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.564+0000 7fcf2d73b640 1 -- 172.21.15.150:0/3873828941 shutdown_connections 2024-09-17T05:12:05.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.564+0000 7fcf2d73b640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf14073200 0x7fcf14075710 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:05.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.564+0000 7fcf2d73b640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf280745c0 0x7fcf28149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:05.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.564+0000 7fcf2d73b640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf2806fca0 0x7fcf280c14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:05.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.564+0000 7fcf2d73b640 1 --2- 172.21.15.150:0/3873828941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf2806e5d0 0x7fcf28156260 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:05.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.564+0000 7fcf2d73b640 1 -- 172.21.15.150:0/3873828941 >> 172.21.15.150:0/3873828941 conn(0x7fcf280c5ae0 msgr2=0x7fcf2814c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:05.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.564+0000 7fcf2d73b640 1 -- 172.21.15.150:0/3873828941 shutdown_connections 2024-09-17T05:12:05.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:05.564+0000 7fcf2d73b640 1 -- 172.21.15.150:0/3873828941 wait complete. 2024-09-17T05:12:05.585 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:12:06.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.608+0000 7f728d36e640 1 -- 172.21.15.150:0/3507255563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72880c7cd0 msgr2=0x7f72880c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:06.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.608+0000 7f728d36e640 1 --2- 172.21.15.150:0/3507255563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72880c7cd0 0x7f72880c80e0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f7280009c80 tx=0x7f728001cda0 comp rx=0 tx=0).stop 2024-09-17T05:12:06.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.608+0000 7f728d36e640 1 -- 172.21.15.150:0/3507255563 shutdown_connections 2024-09-17T05:12:06.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.608+0000 7f728d36e640 1 --2- 172.21.15.150:0/3507255563 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7288072ea0 0x7f72880732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:06.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.608+0000 7f728d36e640 1 --2- 172.21.15.150:0/3507255563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72880c7cd0 0x7f72880c80e0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:06.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.608+0000 7f728d36e640 1 -- 172.21.15.150:0/3507255563 >> 172.21.15.150:0/3507255563 conn(0x7f72880abb00 msgr2=0x7f72880abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:06.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.612+0000 7f728d36e640 1 -- 172.21.15.150:0/3507255563 shutdown_connections 2024-09-17T05:12:06.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.612+0000 7f728d36e640 1 -- 172.21.15.150:0/3507255563 wait complete. 2024-09-17T05:12:06.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.612+0000 7f728d36e640 1 Processor -- start 2024-09-17T05:12:06.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.612+0000 7f728d36e640 1 -- start start 2024-09-17T05:12:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.612+0000 7f728d36e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7288072ea0 0x7f72880addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.612+0000 7f728d36e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72880c7cd0 0x7f728814c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f728d36e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72880c97a0 0x7f728814b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f728d36e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7288161910 con 0x7f72880c7cd0 2024-09-17T05:12:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f728d36e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7288161790 con 0x7f72880c97a0 2024-09-17T05:12:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f728d36e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7288161a90 con 0x7f7288072ea0 2024-09-17T05:12:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72880c7cd0 0x7f728814c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72880c7cd0 0x7f728814c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50972/0 (socket says 172.21.15.150:50972) 2024-09-17T05:12:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727ffff640 1 -- 172.21.15.150:0/2506769969 learned_addr learned my addr 172.21.15.150:0/2506769969 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f72877fe640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72880c97a0 0x7f728814b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f7286ffd640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7288072ea0 0x7f72880addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727ffff640 1 -- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7288072ea0 msgr2=0x7f72880addf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727ffff640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7288072ea0 0x7f72880addf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727ffff640 1 -- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72880c97a0 msgr2=0x7f728814b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727ffff640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72880c97a0 0x7f728814b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727ffff640 1 -- 172.21.15.150:0/2506769969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728814dba0 con 0x7f72880c7cd0 2024-09-17T05:12:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727ffff640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72880c7cd0 0x7f728814c880 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f727000c9a0 tx=0x7f727000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727f7fe640 1 -- 172.21.15.150:0/2506769969 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270004220 con 0x7f72880c7cd0 2024-09-17T05:12:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727f7fe640 1 -- 172.21.15.150:0/2506769969 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72700044b0 con 0x7f72880c7cd0 2024-09-17T05:12:06.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f728d36e640 1 -- 172.21.15.150:0/2506769969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f728814d8c0 con 0x7f72880c7cd0 2024-09-17T05:12:06.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727f7fe640 1 -- 172.21.15.150:0/2506769969 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270007bb0 con 0x7f72880c7cd0 2024-09-17T05:12:06.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f728d36e640 1 -- 172.21.15.150:0/2506769969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72881f1cf0 con 0x7f72880c7cd0 2024-09-17T05:12:06.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f728d36e640 1 -- 172.21.15.150:0/2506769969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72880732b0 con 0x7f72880c7cd0 2024-09-17T05:12:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727f7fe640 1 -- 172.21.15.150:0/2506769969 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7270003620 con 0x7f72880c7cd0 2024-09-17T05:12:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727f7fe640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7268073230 0x7f7268075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.616+0000 7f727f7fe640 1 -- 172.21.15.150:0/2506769969 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f7270095360 con 0x7f72880c7cd0 2024-09-17T05:12:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.620+0000 7f7286ffd640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7268073230 0x7f7268075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.620+0000 7f7286ffd640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7268073230 0x7f7268075740 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7278001650 tx=0x7f7278009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:06.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.624+0000 7f727f7fe640 1 -- 172.21.15.150:0/2506769969 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7270014030 con 0x7f72880c7cd0 2024-09-17T05:12:06.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.892+0000 7f728d36e640 1 -- 172.21.15.150:0/2506769969 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f728814f760 con 0x7f72880c7cd0 2024-09-17T05:12:06.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.892+0000 7f727f7fe640 1 -- 172.21.15.150:0/2506769969 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+5718 (secure 0 0 0) 0x7f727005f5c0 con 0x7f72880c7cd0 2024-09-17T05:12:06.893 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:06.893 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":160,"btime":"2024-09-17T05:12:04:469094+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:06.894 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 160 2024-09-17T05:12:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.896+0000 7f728d36e640 1 -- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7268073230 msgr2=0x7f7268075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.896+0000 7f728d36e640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7268073230 0x7f7268075740 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7278001650 tx=0x7f7278009040 comp rx=0 tx=0).stop 2024-09-17T05:12:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.896+0000 7f728d36e640 1 -- 172.21.15.150:0/2506769969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72880c7cd0 msgr2=0x7f728814c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.896+0000 7f728d36e640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72880c7cd0 0x7f728814c880 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f727000c9a0 tx=0x7f727000ce60 comp rx=0 tx=0).stop 2024-09-17T05:12:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.896+0000 7f728d36e640 1 -- 172.21.15.150:0/2506769969 shutdown_connections 2024-09-17T05:12:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.896+0000 7f728d36e640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7268073230 0x7f7268075740 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.896+0000 7f728d36e640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72880c97a0 0x7f728814b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.896+0000 7f728d36e640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72880c7cd0 0x7f728814c880 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.896+0000 7f728d36e640 1 --2- 172.21.15.150:0/2506769969 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7288072ea0 0x7f72880addf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.896+0000 7f728d36e640 1 -- 172.21.15.150:0/2506769969 >> 172.21.15.150:0/2506769969 conn(0x7f72880abb00 msgr2=0x7f7288150e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.896+0000 7f728d36e640 1 -- 172.21.15.150:0/2506769969 shutdown_connections 2024-09-17T05:12:06.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:06.896+0000 7f728d36e640 1 -- 172.21.15.150:0/2506769969 wait complete. 2024-09-17T05:12:06.920 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=7230 2024-09-17T05:12:06.920 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.a 2024-09-17T05:12:07.216 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T05:12:07.216+0000 7f6b7b24b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:12:07.586 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:07.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 -- 172.21.15.150:0/2028794612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa328004810 msgr2=0x7fa328004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:07.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 --2- 172.21.15.150:0/2028794612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa328004810 0x7fa328004c20 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fa33c08e250 tx=0x7fa32401d210 comp rx=0 tx=0).stop 2024-09-17T05:12:07.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 -- 172.21.15.150:0/2028794612 shutdown_connections 2024-09-17T05:12:07.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 --2- 172.21.15.150:0/2028794612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa328004810 0x7fa328004c20 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:07.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 -- 172.21.15.150:0/2028794612 >> 172.21.15.150:0/2028794612 conn(0x7fa33c074b80 msgr2=0x7fa33c06f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:07.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 -- 172.21.15.150:0/2028794612 shutdown_connections 2024-09-17T05:12:07.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 -- 172.21.15.150:0/2028794612 wait complete. 2024-09-17T05:12:07.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 Processor -- start 2024-09-17T05:12:07.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 -- start start 2024-09-17T05:12:07.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa328004810 0x7fa33c0af530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:07.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa33c14c170 0x7fa33c159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa33c1ece60 0x7fa33c1ed270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa33c0c39c0 con 0x7fa33c14c170 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa33c0c1bd0 con 0x7fa328004810 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa340fc9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa33c0c3b40 con 0x7fa33c1ece60 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa333fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa33c14c170 0x7fa33c159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa333fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa33c14c170 0x7fa33c159440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50992/0 (socket says 172.21.15.150:50992) 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.764+0000 7fa333fff640 1 -- 172.21.15.150:0/543726571 learned_addr learned my addr 172.21.15.150:0/543726571 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa33a59c640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa328004810 0x7fa33c0af530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa333fff640 1 -- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa33c1ece60 msgr2=0x7fa33c1ed270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa333fff640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa33c1ece60 0x7fa33c1ed270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa333fff640 1 -- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa328004810 msgr2=0x7fa33c0af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa333fff640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa328004810 0x7fa33c0af530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:07.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa333fff640 1 -- 172.21.15.150:0/543726571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33c1ed8c0 con 0x7fa33c14c170 2024-09-17T05:12:07.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa33a59c640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa328004810 0x7fa33c0af530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:12:07.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa333fff640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa33c14c170 0x7fa33c159440 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fa32401d040 tx=0x7fa3240ace10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:07.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa332ffd640 1 -- 172.21.15.150:0/543726571 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3240b1040 con 0x7fa33c14c170 2024-09-17T05:12:07.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa332ffd640 1 -- 172.21.15.150:0/543726571 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3240ac400 con 0x7fa33c14c170 2024-09-17T05:12:07.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa340fc9640 1 -- 172.21.15.150:0/543726571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33c1ec960 con 0x7fa33c14c170 2024-09-17T05:12:07.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa332ffd640 1 -- 172.21.15.150:0/543726571 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa324005280 con 0x7fa33c14c170 2024-09-17T05:12:07.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa340fc9640 1 -- 172.21.15.150:0/543726571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa33c0ad190 con 0x7fa33c14c170 2024-09-17T05:12:07.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa340fc9640 1 -- 172.21.15.150:0/543726571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa33c0c8a80 con 0x7fa33c14c170 2024-09-17T05:12:07.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.768+0000 7fa332ffd640 1 -- 172.21.15.150:0/543726571 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa3240c0020 con 0x7fa33c14c170 2024-09-17T05:12:07.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.772+0000 7fa332ffd640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa31c073200 0x7fa31c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:07.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.772+0000 7fa332ffd640 1 -- 172.21.15.150:0/543726571 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7fa324130e90 con 0x7fa33c14c170 2024-09-17T05:12:07.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.772+0000 7fa33a59c640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa31c073200 0x7fa31c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:07.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.772+0000 7fa33a59c640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa31c073200 0x7fa31c075710 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0066a0 tx=0x7fa32c006c90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:07.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:07.776+0000 7fa332ffd640 1 -- 172.21.15.150:0/543726571 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa324016f20 con 0x7fa33c14c170 2024-09-17T05:12:08.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.016+0000 7fa340fc9640 1 -- 172.21.15.150:0/543726571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa33c157790 con 0x7fa33c14c170 2024-09-17T05:12:08.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa332ffd640 1 -- 172.21.15.150:0/543726571 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+5718 (secure 0 0 0) 0x7fa3240fb310 con 0x7fa33c14c170 2024-09-17T05:12:08.019 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:08.019 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":160,"btime":"2024-09-17T05:12:04:469094+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:08.019 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 160 2024-09-17T05:12:08.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa340fc9640 1 -- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa31c073200 msgr2=0x7fa31c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:08.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa340fc9640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa31c073200 0x7fa31c075710 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0066a0 tx=0x7fa32c006c90 comp rx=0 tx=0).stop 2024-09-17T05:12:08.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa340fc9640 1 -- 172.21.15.150:0/543726571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa33c14c170 msgr2=0x7fa33c159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:08.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa340fc9640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa33c14c170 0x7fa33c159440 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fa32401d040 tx=0x7fa3240ace10 comp rx=0 tx=0).stop 2024-09-17T05:12:08.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa340fc9640 1 -- 172.21.15.150:0/543726571 shutdown_connections 2024-09-17T05:12:08.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa340fc9640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa31c073200 0x7fa31c075710 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:08.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa340fc9640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa33c1ece60 0x7fa33c1ed270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:08.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa340fc9640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa33c14c170 0x7fa33c159440 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:08.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa340fc9640 1 --2- 172.21.15.150:0/543726571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa328004810 0x7fa33c0af530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:08.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa340fc9640 1 -- 172.21.15.150:0/543726571 >> 172.21.15.150:0/543726571 conn(0x7fa33c074b80 msgr2=0x7fa33c0c6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:08.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa340fc9640 1 -- 172.21.15.150:0/543726571 shutdown_connections 2024-09-17T05:12:08.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:08.020+0000 7fa340fc9640 1 -- 172.21.15.150:0/543726571 wait complete. 2024-09-17T05:12:08.041 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:12:08.617 INFO:tasks.workunit.client.0.smithi150.stdout:fs setup took 1217 secs 2024-09-17T05:12:09.498 INFO:tasks.workunit.client.0.smithi150.stdout:Syncing()...0 sec 2024-09-17T05:12:09.499 INFO:tasks.workunit.client.0.smithi150.stdout:Starting Actual Benchmark At: Tue Sep 17 05:12:09 2024 2024-09-17T05:12:09.500 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T05:12:10.042 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:10.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.264+0000 7f7d5183b640 1 -- 172.21.15.150:0/1598599838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4c071a10 msgr2=0x7f7d4c154010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:10.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.264+0000 7f7d5183b640 1 --2- 172.21.15.150:0/1598599838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4c071a10 0x7f7d4c154010 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c0704d0 tx=0x7f7d3c01c830 comp rx=0 tx=0).stop 2024-09-17T05:12:10.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.264+0000 7f7d5183b640 1 -- 172.21.15.150:0/1598599838 shutdown_connections 2024-09-17T05:12:10.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.264+0000 7f7d5183b640 1 --2- 172.21.15.150:0/1598599838 >> v2:172.21.15.176:3300/0 conn(0x7f7d4c0a8500 0x7f7d4c06ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:10.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.264+0000 7f7d5183b640 1 --2- 172.21.15.150:0/1598599838 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d4c074c20 0x7f7d4c0c2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:10.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.264+0000 7f7d5183b640 1 --2- 172.21.15.150:0/1598599838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4c071a10 0x7f7d4c154010 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:10.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.264+0000 7f7d5183b640 1 -- 172.21.15.150:0/1598599838 >> 172.21.15.150:0/1598599838 conn(0x7f7d4c0aaa80 msgr2=0x7f7d4c0aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:10.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.268+0000 7f7d5183b640 1 -- 172.21.15.150:0/1598599838 shutdown_connections 2024-09-17T05:12:10.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.268+0000 7f7d5183b640 1 -- 172.21.15.150:0/1598599838 wait complete. 2024-09-17T05:12:10.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.268+0000 7f7d5183b640 1 Processor -- start 2024-09-17T05:12:10.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.268+0000 7f7d5183b640 1 -- start start 2024-09-17T05:12:10.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d5183b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4c071a10 0x7f7d4c0c8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:10.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d5183b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d4c074c20 0x7f7d4c0c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:10.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d5183b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d4c0a8500 0x7f7d4c1fc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:10.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d4a7fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d4c074c20 0x7f7d4c0c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:10.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d4a7fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d4c074c20 0x7f7d4c0c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:56652/0 (socket says 172.21.15.150:56652) 2024-09-17T05:12:10.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d4a7fc640 1 -- 172.21.15.150:0/2403648371 learned_addr learned my addr 172.21.15.150:0/2403648371 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:10.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d4affd640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4c071a10 0x7f7d4c0c8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:10.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d4b7fe640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d4c0a8500 0x7f7d4c1fc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:10.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4c06d920 con 0x7f7d4c071a10 2024-09-17T05:12:10.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4c06d7a0 con 0x7f7d4c0a8500 2024-09-17T05:12:10.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7d4c15a680 con 0x7f7d4c074c20 2024-09-17T05:12:10.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d4affd640 1 -- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d4c074c20 msgr2=0x7f7d4c0c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:10.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d4affd640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d4c074c20 0x7f7d4c0c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:10.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d4affd640 1 -- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d4c0a8500 msgr2=0x7f7d4c1fc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:10.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d4affd640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d4c0a8500 0x7f7d4c1fc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:10.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d4affd640 1 -- 172.21.15.150:0/2403648371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d4c0ace70 con 0x7f7d4c071a10 2024-09-17T05:12:10.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d4affd640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4c071a10 0x7f7d4c0c8ea0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c0ac000 tx=0x7f7d3c005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:10.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d33fff640 1 -- 172.21.15.150:0/2403648371 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c0b2040 con 0x7f7d4c071a10 2024-09-17T05:12:10.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.272+0000 7f7d33fff640 1 -- 172.21.15.150:0/2403648371 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d3c004240 con 0x7f7d4c071a10 2024-09-17T05:12:10.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.276+0000 7f7d33fff640 1 -- 172.21.15.150:0/2403648371 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c003600 con 0x7f7d4c071a10 2024-09-17T05:12:10.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.276+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d4c1fd360 con 0x7f7d4c071a10 2024-09-17T05:12:10.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.276+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d4c0c86a0 con 0x7f7d4c071a10 2024-09-17T05:12:10.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.276+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d4c15abe0 con 0x7f7d4c071a10 2024-09-17T05:12:10.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.288+0000 7f7d33fff640 1 -- 172.21.15.150:0/2403648371 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7d3c0037c0 con 0x7f7d4c071a10 2024-09-17T05:12:10.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.288+0000 7f7d33fff640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d38073200 0x7f7d38075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:10.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.288+0000 7f7d33fff640 1 -- 172.21.15.150:0/2403648371 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f7d3c12efc0 con 0x7f7d4c071a10 2024-09-17T05:12:10.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.288+0000 7f7d4a7fc640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d38073200 0x7f7d38075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:10.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.288+0000 7f7d33fff640 1 -- 172.21.15.150:0/2403648371 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7d3c0164b0 con 0x7f7d4c071a10 2024-09-17T05:12:10.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.308+0000 7f7d4a7fc640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d38073200 0x7f7d38075710 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c153da0 tx=0x7f7d400090e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:10.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.792+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d4c074a00 con 0x7f7d4c071a10 2024-09-17T05:12:10.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.792+0000 7f7d33fff640 1 -- 172.21.15.150:0/2403648371 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+5718 (secure 0 0 0) 0x7f7d3c0f8c80 con 0x7f7d4c071a10 2024-09-17T05:12:10.793 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:10.793 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":160,"btime":"2024-09-17T05:12:04:469094+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:10.793 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 160 2024-09-17T05:12:10.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.800+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d38073200 msgr2=0x7f7d38075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:10.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.800+0000 7f7d5183b640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d38073200 0x7f7d38075710 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c153da0 tx=0x7f7d400090e0 comp rx=0 tx=0).stop 2024-09-17T05:12:10.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.800+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4c071a10 msgr2=0x7f7d4c0c8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:10.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.800+0000 7f7d5183b640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4c071a10 0x7f7d4c0c8ea0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c0ac000 tx=0x7f7d3c005e00 comp rx=0 tx=0).stop 2024-09-17T05:12:10.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.800+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 shutdown_connections 2024-09-17T05:12:10.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.800+0000 7f7d5183b640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d38073200 0x7f7d38075710 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:10.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.800+0000 7f7d5183b640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d4c0a8500 0x7f7d4c1fc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:10.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.800+0000 7f7d5183b640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d4c074c20 0x7f7d4c0c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:10.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.800+0000 7f7d5183b640 1 --2- 172.21.15.150:0/2403648371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d4c071a10 0x7f7d4c0c8ea0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:10.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.800+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 >> 172.21.15.150:0/2403648371 conn(0x7f7d4c0aaa80 msgr2=0x7f7d4c14e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:10.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.800+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 shutdown_connections 2024-09-17T05:12:10.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:10.800+0000 7f7d5183b640 1 -- 172.21.15.150:0/2403648371 wait complete. 2024-09-17T05:12:10.841 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:12:10.921 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-17T05:12:10.921 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T05:12:10.922 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T05:12:10.925 INFO:tasks.ceph.mds.a:Started 2024-09-17T05:12:10.925 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:11.126 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T05:12:11.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.196+0000 7f99aa104640 1 -- 172.21.15.150:0/934691009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f999c0d7a40 msgr2=0x7f9990004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:11.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.196+0000 7f99aa104640 1 --2- 172.21.15.150:0/934691009 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f999c0d7a40 0x7f9990004810 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9994009b10 tx=0x7f999401d290 comp rx=0 tx=0).stop 2024-09-17T05:12:11.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.196+0000 7f99a9903640 1 -- 172.21.15.150:0/934691009 reap_dead start 2024-09-17T05:12:11.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.196+0000 7f99aa104640 1 -- 172.21.15.150:0/934691009 shutdown_connections 2024-09-17T05:12:11.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.196+0000 7f99aa104640 1 -- 172.21.15.150:0/934691009 >> 172.21.15.150:0/934691009 conn(0x7f999c04c880 msgr2=0x7f999c0312f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:11.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.196+0000 7f99aa104640 1 -- 172.21.15.150:0/934691009 shutdown_connections 2024-09-17T05:12:11.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.196+0000 7f99aa104640 1 -- 172.21.15.150:0/934691009 wait complete. 2024-09-17T05:12:11.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99aa104640 1 Processor -- start 2024-09-17T05:12:11.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99aa104640 1 -- start start 2024-09-17T05:12:11.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99aa104640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f999c0d7a40 0x7f999c0333a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:11.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99aa104640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c0341f0 0x7f999c0dc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:11.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99aa104640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f999c17fb20 0x7f999c17ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:11.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99aa104640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f999c032110 con 0x7f999c0341f0 2024-09-17T05:12:11.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99aa104640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f999c00a2e0 con 0x7f999c17fb20 2024-09-17T05:12:11.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99aa104640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f999c1803e0 con 0x7f999c0d7a40 2024-09-17T05:12:11.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99a8901640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c0341f0 0x7f999c0dc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99a8901640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c0341f0 0x7f999c0dc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49562/0 (socket says 172.21.15.150:49562) 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99a9903640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f999c17fb20 0x7f999c17ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99a8901640 1 -- 172.21.15.150:0/797254649 learned_addr learned my addr 172.21.15.150:0/797254649 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99a9102640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f999c0d7a40 0x7f999c0333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99a8901640 1 -- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f999c0d7a40 msgr2=0x7f999c0333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99a8901640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f999c0d7a40 0x7f999c0333a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99a8901640 1 -- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f999c17fb20 msgr2=0x7f999c17ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99a8901640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f999c17fb20 0x7f999c17ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99a8901640 1 -- 172.21.15.150:0/797254649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f999c04e4c0 con 0x7f999c0341f0 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99a8901640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c0341f0 0x7f999c0dc7a0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f99940ae040 tx=0x7f9994005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f9999ffb640 1 -- 172.21.15.150:0/797254649 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99940bb030 con 0x7f999c0341f0 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f9999ffb640 1 -- 172.21.15.150:0/797254649 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9994004900 con 0x7f999c0341f0 2024-09-17T05:12:11.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99aa104640 1 -- 172.21.15.150:0/797254649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f999c045fe0 con 0x7f999c0341f0 2024-09-17T05:12:11.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f99aa104640 1 -- 172.21.15.150:0/797254649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f999c034c30 con 0x7f999c0341f0 2024-09-17T05:12:11.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.200+0000 7f9999ffb640 1 -- 172.21.15.150:0/797254649 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9994004d30 con 0x7f999c0341f0 2024-09-17T05:12:11.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.204+0000 7f99aa104640 1 -- 172.21.15.150:0/797254649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99840052a0 con 0x7f999c0341f0 2024-09-17T05:12:11.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.204+0000 7f9999ffb640 1 -- 172.21.15.150:0/797254649 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f99940be030 con 0x7f999c0341f0 2024-09-17T05:12:11.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.204+0000 7f9999ffb640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9988073200 0x7f9988075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:11.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.208+0000 7f99a9102640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9988073200 0x7f9988075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:11.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.208+0000 7f9999ffb640 1 -- 172.21.15.150:0/797254649 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f9994130840 con 0x7f999c0341f0 2024-09-17T05:12:11.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.208+0000 7f99a9102640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9988073200 0x7f9988075710 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f99a000b830 tx=0x7f99a0008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:11.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.212+0000 7f9999ffb640 1 -- 172.21.15.150:0/797254649 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f999c04e4c0 con 0x7f999c0341f0 2024-09-17T05:12:11.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.660+0000 7f99aa104640 1 -- 172.21.15.150:0/797254649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9984005590 con 0x7f999c0341f0 2024-09-17T05:12:11.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.660+0000 7f9999ffb640 1 -- 172.21.15.150:0/797254649 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) ==== 95+0+5718 (secure 0 0 0) 0x7f9994016fa0 con 0x7f999c0341f0 2024-09-17T05:12:11.663 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:11.663 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":160,"btime":"2024-09-17T05:12:04:469094+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:11.663 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 160 2024-09-17T05:12:11.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.664+0000 7f99aa104640 1 -- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9988073200 msgr2=0x7f9988075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:11.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.664+0000 7f99aa104640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9988073200 0x7f9988075710 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f99a000b830 tx=0x7f99a0008040 comp rx=0 tx=0).stop 2024-09-17T05:12:11.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.664+0000 7f99aa104640 1 -- 172.21.15.150:0/797254649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c0341f0 msgr2=0x7f999c0dc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:11.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.664+0000 7f99aa104640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c0341f0 0x7f999c0dc7a0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f99940ae040 tx=0x7f9994005020 comp rx=0 tx=0).stop 2024-09-17T05:12:11.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.668+0000 7f99aa104640 1 -- 172.21.15.150:0/797254649 shutdown_connections 2024-09-17T05:12:11.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.668+0000 7f99aa104640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9988073200 0x7f9988075710 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:11.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.668+0000 7f99aa104640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f999c17fb20 0x7f999c17ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:11.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.668+0000 7f99aa104640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f999c0341f0 0x7f999c0dc7a0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:11.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.668+0000 7f99aa104640 1 --2- 172.21.15.150:0/797254649 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f999c0d7a40 0x7f999c0333a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:11.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.668+0000 7f99aa104640 1 -- 172.21.15.150:0/797254649 >> 172.21.15.150:0/797254649 conn(0x7f999c04c880 msgr2=0x7f999c0dfd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:11.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.668+0000 7f99aa104640 1 -- 172.21.15.150:0/797254649 shutdown_connections 2024-09-17T05:12:11.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:11.668+0000 7f99aa104640 1 -- 172.21.15.150:0/797254649 wait complete. 2024-09-17T05:12:11.699 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-09-17T05:12:12.842 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:13.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.100+0000 7fc69ffff640 1 -- 172.21.15.150:0/2302162713 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc698030cc0 msgr2=0x7fc6980d5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:13.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.100+0000 7fc69ffff640 1 --2- 172.21.15.150:0/2302162713 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc698030cc0 0x7fc6980d5b30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0000b70 tx=0x7fc6a009bac0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.100+0000 7fc69ffff640 1 -- 172.21.15.150:0/2302162713 shutdown_connections 2024-09-17T05:12:13.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.100+0000 7fc69ffff640 1 --2- 172.21.15.150:0/2302162713 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc698030cc0 0x7fc6980d5b30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.100+0000 7fc69ffff640 1 -- 172.21.15.150:0/2302162713 >> 172.21.15.150:0/2302162713 conn(0x7fc698048630 msgr2=0x7fc69802cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:13.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.100+0000 7fc69ffff640 1 -- 172.21.15.150:0/2302162713 shutdown_connections 2024-09-17T05:12:13.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.100+0000 7fc69ffff640 1 -- 172.21.15.150:0/2302162713 wait complete. 2024-09-17T05:12:13.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69ffff640 1 Processor -- start 2024-09-17T05:12:13.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69ffff640 1 -- start start 2024-09-17T05:12:13.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69ffff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc69802d5f0 0x7fc6980d4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:13.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69ffff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698030cc0 0x7fc69803fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:13.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc698046480 0x7fc69803ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:13.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69ffff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc6980d6070 con 0x7fc698046480 2024-09-17T05:12:13.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69ffff640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc6980d6c80 con 0x7fc698030cc0 2024-09-17T05:12:13.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69ffff640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc6980da250 con 0x7fc69802d5f0 2024-09-17T05:12:13.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69effd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc69802d5f0 0x7fc6980d4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:13.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc698046480 0x7fc69803ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:13.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69e7fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698030cc0 0x7fc69803fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:13.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc698046480 0x7fc69803ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49570/0 (socket says 172.21.15.150:49570) 2024-09-17T05:12:13.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.104+0000 7fc69f7fe640 1 -- 172.21.15.150:0/2603865925 learned_addr learned my addr 172.21.15.150:0/2603865925 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:13.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.108+0000 7fc69f7fe640 1 -- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc69802d5f0 msgr2=0x7fc6980d4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:13.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.108+0000 7fc69f7fe640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc69802d5f0 0x7fc6980d4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.108+0000 7fc69f7fe640 1 -- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698030cc0 msgr2=0x7fc69803fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:13.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.108+0000 7fc69f7fe640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698030cc0 0x7fc69803fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.108+0000 7fc69f7fe640 1 -- 172.21.15.150:0/2603865925 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6980d9550 con 0x7fc698046480 2024-09-17T05:12:13.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.108+0000 7fc69e7fc640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698030cc0 0x7fc69803fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:12:13.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.108+0000 7fc69f7fe640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc698046480 0x7fc69803ff50 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fc69000c3f0 tx=0x7fc69000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:13.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.108+0000 7fc687fff640 1 -- 172.21.15.150:0/2603865925 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690017070 con 0x7fc698046480 2024-09-17T05:12:13.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.108+0000 7fc687fff640 1 -- 172.21.15.150:0/2603865925 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6900053c0 con 0x7fc698046480 2024-09-17T05:12:13.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.108+0000 7fc687fff640 1 -- 172.21.15.150:0/2603865925 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690017070 con 0x7fc698046480 2024-09-17T05:12:13.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.108+0000 7fc69ffff640 1 -- 172.21.15.150:0/2603865925 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6981736f0 con 0x7fc698046480 2024-09-17T05:12:13.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.108+0000 7fc69ffff640 1 -- 172.21.15.150:0/2603865925 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6980d1a90 con 0x7fc698046480 2024-09-17T05:12:13.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.112+0000 7fc687fff640 1 -- 172.21.15.150:0/2603865925 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc69001a070 con 0x7fc698046480 2024-09-17T05:12:13.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.112+0000 7fc687fff640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc68c073630 0x7fc68c075b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:13.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.112+0000 7fc687fff640 1 -- 172.21.15.150:0/2603865925 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7fc6900050d0 con 0x7fc698046480 2024-09-17T05:12:13.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.112+0000 7fc69effd640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc68c073630 0x7fc68c075b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:13.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.112+0000 7fc69effd640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc68c073630 0x7fc68c075b40 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc698047bf0 tx=0x7fc6a012e040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:13.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.112+0000 7fc685ffb640 1 -- 172.21.15.150:0/2603865925 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6980d5b30 con 0x7fc698046480 2024-09-17T05:12:13.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.120+0000 7fc687fff640 1 -- 172.21.15.150:0/2603865925 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc690062280 con 0x7fc698046480 2024-09-17T05:12:13.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.532+0000 7fc685ffb640 1 -- 172.21.15.150:0/2603865925 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6980cf090 con 0x7fc698046480 2024-09-17T05:12:13.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.532+0000 7fc687fff640 1 -- 172.21.15.150:0/2603865925 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) ==== 95+0+6541 (secure 0 0 0) 0x7fc6900671c0 con 0x7fc698046480 2024-09-17T05:12:13.534 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:13.534 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":161,"btime":"2024-09-17T05:12:11:699234+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7530,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:13.534 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 161 2024-09-17T05:12:13.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.536+0000 7fc685ffb640 1 -- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc68c073630 msgr2=0x7fc68c075b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:13.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.536+0000 7fc685ffb640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc68c073630 0x7fc68c075b40 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc698047bf0 tx=0x7fc6a012e040 comp rx=0 tx=0).stop 2024-09-17T05:12:13.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.536+0000 7fc685ffb640 1 -- 172.21.15.150:0/2603865925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc698046480 msgr2=0x7fc69803ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:13.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.536+0000 7fc685ffb640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc698046480 0x7fc69803ff50 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fc69000c3f0 tx=0x7fc69000c7f0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.536+0000 7fc685ffb640 1 -- 172.21.15.150:0/2603865925 shutdown_connections 2024-09-17T05:12:13.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.536+0000 7fc685ffb640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc68c073630 0x7fc68c075b40 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.536+0000 7fc685ffb640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc698030cc0 0x7fc69803fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.536+0000 7fc685ffb640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc698046480 0x7fc69803ff50 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.536+0000 7fc685ffb640 1 --2- 172.21.15.150:0/2603865925 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc69802d5f0 0x7fc6980d4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.536+0000 7fc685ffb640 1 -- 172.21.15.150:0/2603865925 >> 172.21.15.150:0/2603865925 conn(0x7fc698048630 msgr2=0x7fc69802f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.536+0000 7fc685ffb640 1 -- 172.21.15.150:0/2603865925 shutdown_connections 2024-09-17T05:12:13.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.536+0000 7fc685ffb640 1 -- 172.21.15.150:0/2603865925 wait complete. 2024-09-17T05:12:13.566 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:12:13.700 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:13.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.932+0000 7feb3b59e640 1 -- 172.21.15.150:0/1591305421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb340478a0 msgr2=0x7feb340d79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:13.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.932+0000 7feb3b59e640 1 --2- 172.21.15.150:0/1591305421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb340478a0 0x7feb340d79a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0098f0 tx=0x7feb2c01c8e0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.932+0000 7feb3b59e640 1 -- 172.21.15.150:0/1591305421 shutdown_connections 2024-09-17T05:12:13.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.932+0000 7feb3b59e640 1 --2- 172.21.15.150:0/1591305421 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb3402e8e0 0x7feb340d4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.932+0000 7feb3b59e640 1 --2- 172.21.15.150:0/1591305421 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb340478a0 0x7feb340d79a0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.932+0000 7feb3b59e640 1 -- 172.21.15.150:0/1591305421 >> 172.21.15.150:0/1591305421 conn(0x7feb34049590 msgr2=0x7feb3402e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:13.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.932+0000 7feb3b59e640 1 -- 172.21.15.150:0/1591305421 shutdown_connections 2024-09-17T05:12:13.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.932+0000 7feb3b59e640 1 -- 172.21.15.150:0/1591305421 wait complete. 2024-09-17T05:12:13.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.936+0000 7feb3b59e640 1 Processor -- start 2024-09-17T05:12:13.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.936+0000 7feb3b59e640 1 -- start start 2024-09-17T05:12:13.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb3b59e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb3402e8e0 0x7feb34177170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:13.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb3b59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3402ecf0 0x7feb340cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:13.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb3b59e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb340478a0 0x7feb34045cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:13.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb3a59c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb3402e8e0 0x7feb34177170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:13.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb3a59c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb3402e8e0 0x7feb34177170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:40508/0 (socket says 172.21.15.150:40508) 2024-09-17T05:12:13.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb3a59c640 1 -- 172.21.15.150:0/3794460767 learned_addr learned my addr 172.21.15.150:0/3794460767 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:13.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb39d9b640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3402ecf0 0x7feb340cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:13.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb3b59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feb34006bc0 con 0x7feb3402ecf0 2024-09-17T05:12:13.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb3b59e640 1 -- 172.21.15.150:0/3794460767 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7feb34006a40 con 0x7feb3402e8e0 2024-09-17T05:12:13.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb3b59e640 1 -- 172.21.15.150:0/3794460767 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feb34006d40 con 0x7feb340478a0 2024-09-17T05:12:13.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb3ad9d640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb340478a0 0x7feb34045cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:13.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb39d9b640 1 -- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb340478a0 msgr2=0x7feb34045cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:13.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb39d9b640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb340478a0 0x7feb34045cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb39d9b640 1 -- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb3402e8e0 msgr2=0x7feb34177170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:13.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb39d9b640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb3402e8e0 0x7feb34177170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:13.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb39d9b640 1 -- 172.21.15.150:0/3794460767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb34174b30 con 0x7feb3402ecf0 2024-09-17T05:12:13.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb39d9b640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3402ecf0 0x7feb340cc9d0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7feb3000b410 tx=0x7feb3000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:13.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb2affd640 1 -- 172.21.15.150:0/3794460767 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30015020 con 0x7feb3402ecf0 2024-09-17T05:12:13.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb2affd640 1 -- 172.21.15.150:0/3794460767 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb30011830 con 0x7feb3402ecf0 2024-09-17T05:12:13.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb2affd640 1 -- 172.21.15.150:0/3794460767 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30011c80 con 0x7feb3402ecf0 2024-09-17T05:12:13.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb3b59e640 1 -- 172.21.15.150:0/3794460767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb341776b0 con 0x7feb3402ecf0 2024-09-17T05:12:13.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.940+0000 7feb3b59e640 1 -- 172.21.15.150:0/3794460767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb340d2860 con 0x7feb3402ecf0 2024-09-17T05:12:13.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.944+0000 7feb3b59e640 1 -- 172.21.15.150:0/3794460767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb340d4330 con 0x7feb3402ecf0 2024-09-17T05:12:13.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.944+0000 7feb2affd640 1 -- 172.21.15.150:0/3794460767 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7feb30018020 con 0x7feb3402ecf0 2024-09-17T05:12:13.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.944+0000 7feb2affd640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feb1c073200 0x7feb1c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:13.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.948+0000 7feb3a59c640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feb1c073200 0x7feb1c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:13.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.952+0000 7feb2affd640 1 -- 172.21.15.150:0/3794460767 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7feb30091620 con 0x7feb3402ecf0 2024-09-17T05:12:13.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.952+0000 7feb3a59c640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feb1c073200 0x7feb1c075710 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7feb2c01c710 tx=0x7feb2c01b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:13.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:13.952+0000 7feb2affd640 1 -- 172.21.15.150:0/3794460767 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7feb3005db50 con 0x7feb3402ecf0 2024-09-17T05:12:14.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.360+0000 7feb3b59e640 1 -- 172.21.15.150:0/3794460767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb340e0090 con 0x7feb3402ecf0 2024-09-17T05:12:14.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.360+0000 7feb2affd640 1 -- 172.21.15.150:0/3794460767 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) ==== 95+0+6541 (secure 0 0 0) 0x7feb30011e20 con 0x7feb3402ecf0 2024-09-17T05:12:14.362 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:14.362 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":161,"btime":"2024-09-17T05:12:11:699234+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7530,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:14.362 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 161 2024-09-17T05:12:14.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.368+0000 7feb3b59e640 1 -- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feb1c073200 msgr2=0x7feb1c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:14.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.368+0000 7feb3b59e640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feb1c073200 0x7feb1c075710 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7feb2c01c710 tx=0x7feb2c01b120 comp rx=0 tx=0).stop 2024-09-17T05:12:14.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.368+0000 7feb3b59e640 1 -- 172.21.15.150:0/3794460767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3402ecf0 msgr2=0x7feb340cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:14.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.368+0000 7feb3b59e640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3402ecf0 0x7feb340cc9d0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7feb3000b410 tx=0x7feb3000b810 comp rx=0 tx=0).stop 2024-09-17T05:12:14.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.368+0000 7feb3b59e640 1 -- 172.21.15.150:0/3794460767 shutdown_connections 2024-09-17T05:12:14.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.368+0000 7feb3b59e640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feb1c073200 0x7feb1c075710 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:14.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.368+0000 7feb3b59e640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feb340478a0 0x7feb34045cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:14.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.368+0000 7feb3b59e640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb3402ecf0 0x7feb340cc9d0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:14.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.368+0000 7feb3b59e640 1 --2- 172.21.15.150:0/3794460767 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb3402e8e0 0x7feb34177170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:14.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.368+0000 7feb3b59e640 1 -- 172.21.15.150:0/3794460767 >> 172.21.15.150:0/3794460767 conn(0x7feb34049590 msgr2=0x7feb340cdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:14.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.368+0000 7feb3b59e640 1 -- 172.21.15.150:0/3794460767 shutdown_connections 2024-09-17T05:12:14.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:14.368+0000 7feb3b59e640 1 -- 172.21.15.150:0/3794460767 wait complete. 2024-09-17T05:12:14.404 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-17T05:12:14.404 INFO:tasks.mds_thrash.fs.[a]:waiting for 113 secs before thrashing 2024-09-17T05:12:15.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:15.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.820+0000 7fbe9723a640 1 -- 172.21.15.150:0/3016513778 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe90157ca0 msgr2=0x7fbe901580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.820+0000 7fbe9723a640 1 --2- 172.21.15.150:0/3016513778 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe90157ca0 0x7fbe901580b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009bc0 tx=0x7fbe8801c8e0 comp rx=0 tx=0).stop 2024-09-17T05:12:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.820+0000 7fbe9723a640 1 -- 172.21.15.150:0/3016513778 shutdown_connections 2024-09-17T05:12:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.820+0000 7fbe9723a640 1 --2- 172.21.15.150:0/3016513778 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe90157ca0 0x7fbe901580b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.820+0000 7fbe9723a640 1 --2- 172.21.15.150:0/3016513778 >> v2:172.21.15.150:3300/0 conn(0x7fbe900c6230 0x7fbe900c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.820+0000 7fbe9723a640 1 -- 172.21.15.150:0/3016513778 >> 172.21.15.150:0/3016513778 conn(0x7fbe900c9de0 msgr2=0x7fbe900ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.820+0000 7fbe9723a640 1 -- 172.21.15.150:0/3016513778 shutdown_connections 2024-09-17T05:12:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.820+0000 7fbe9723a640 1 -- 172.21.15.150:0/3016513778 wait complete. 2024-09-17T05:12:15.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.824+0000 7fbe9723a640 1 Processor -- start 2024-09-17T05:12:15.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.824+0000 7fbe9723a640 1 -- start start 2024-09-17T05:12:15.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.824+0000 7fbe9723a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe900ad7f0 0x7fbe90147990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:15.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.824+0000 7fbe9723a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe900c6230 0x7fbe900723c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:15.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.824+0000 7fbe9723a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe90157ca0 0x7fbe9014bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:15.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.824+0000 7fbe94fd6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe900ad7f0 0x7fbe90147990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:15.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.824+0000 7fbe94fd6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe900ad7f0 0x7fbe90147990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49594/0 (socket says 172.21.15.150:49594) 2024-09-17T05:12:15.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.824+0000 7fbe94fd6640 1 -- 172.21.15.150:0/2988083608 learned_addr learned my addr 172.21.15.150:0/2988083608 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:15.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.824+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbe900c59f0 con 0x7fbe900ad7f0 2024-09-17T05:12:15.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.824+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbe900c5870 con 0x7fbe90157ca0 2024-09-17T05:12:15.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.824+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbe900c7ce0 con 0x7fbe900c6230 2024-09-17T05:12:15.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.824+0000 7fbe957d7640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe90157ca0 0x7fbe9014bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:15.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.828+0000 7fbe8ffff640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe900c6230 0x7fbe900723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:15.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.828+0000 7fbe94fd6640 1 -- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe900c6230 msgr2=0x7fbe900723c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:15.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.828+0000 7fbe94fd6640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe900c6230 0x7fbe900723c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:15.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.828+0000 7fbe94fd6640 1 -- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe90157ca0 msgr2=0x7fbe9014bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:15.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.828+0000 7fbe94fd6640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe90157ca0 0x7fbe9014bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:15.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.828+0000 7fbe94fd6640 1 -- 172.21.15.150:0/2988083608 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9014d550 con 0x7fbe900ad7f0 2024-09-17T05:12:15.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.828+0000 7fbe94fd6640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe900ad7f0 0x7fbe90147990 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400c730 tx=0x7fbe8400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:15.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.828+0000 7fbe8d7fa640 1 -- 172.21.15.150:0/2988083608 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84017070 con 0x7fbe900ad7f0 2024-09-17T05:12:15.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.828+0000 7fbe8d7fa640 1 -- 172.21.15.150:0/2988083608 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe8400ce40 con 0x7fbe900ad7f0 2024-09-17T05:12:15.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.828+0000 7fbe8d7fa640 1 -- 172.21.15.150:0/2988083608 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8400a8a0 con 0x7fbe900ad7f0 2024-09-17T05:12:15.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.828+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe901f4520 con 0x7fbe900ad7f0 2024-09-17T05:12:15.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.832+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe9014cf90 con 0x7fbe900ad7f0 2024-09-17T05:12:15.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.832+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe9014cd80 con 0x7fbe900ad7f0 2024-09-17T05:12:15.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.836+0000 7fbe8d7fa640 1 -- 172.21.15.150:0/2988083608 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbe8401a070 con 0x7fbe900ad7f0 2024-09-17T05:12:15.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.836+0000 7fbe8d7fa640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe74073300 0x7fbe74075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:15.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.836+0000 7fbe8d7fa640 1 -- 172.21.15.150:0/2988083608 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7fbe84098060 con 0x7fbe900ad7f0 2024-09-17T05:12:15.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.840+0000 7fbe8ffff640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe74073300 0x7fbe74075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:15.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.840+0000 7fbe8d7fa640 1 -- 172.21.15.150:0/2988083608 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbe8405dfc0 con 0x7fbe900ad7f0 2024-09-17T05:12:15.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:15.840+0000 7fbe8ffff640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe74073300 0x7fbe74075810 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fbe90158570 tx=0x7fbe7c0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:16.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.244+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe900c5330 con 0x7fbe900ad7f0 2024-09-17T05:12:16.246 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:16.246 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":161,"btime":"2024-09-17T05:12:11:699234+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7530,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:16.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.244+0000 7fbe8d7fa640 1 -- 172.21.15.150:0/2988083608 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) ==== 95+0+6541 (secure 0 0 0) 0x7fbe840629f0 con 0x7fbe900ad7f0 2024-09-17T05:12:16.246 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 161 2024-09-17T05:12:16.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.256+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe74073300 msgr2=0x7fbe74075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:16.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.256+0000 7fbe9723a640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe74073300 0x7fbe74075810 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fbe90158570 tx=0x7fbe7c0091e0 comp rx=0 tx=0).stop 2024-09-17T05:12:16.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.256+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe900ad7f0 msgr2=0x7fbe90147990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:16.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.256+0000 7fbe9723a640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe900ad7f0 0x7fbe90147990 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400c730 tx=0x7fbe8400cbf0 comp rx=0 tx=0).stop 2024-09-17T05:12:16.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.256+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 shutdown_connections 2024-09-17T05:12:16.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.256+0000 7fbe9723a640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe74073300 0x7fbe74075810 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:16.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.256+0000 7fbe9723a640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe90157ca0 0x7fbe9014bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:16.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.256+0000 7fbe9723a640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe900c6230 0x7fbe900723c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:16.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.256+0000 7fbe9723a640 1 --2- 172.21.15.150:0/2988083608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe900ad7f0 0x7fbe90147990 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:16.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.256+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 >> 172.21.15.150:0/2988083608 conn(0x7fbe900c9de0 msgr2=0x7fbe900aa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:16.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.256+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 shutdown_connections 2024-09-17T05:12:16.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:16.256+0000 7fbe9723a640 1 -- 172.21.15.150:0/2988083608 wait complete. 2024-09-17T05:12:16.294 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:12:18.295 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:18.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.532+0000 7fc552ec7640 1 -- 172.21.15.150:0/2514793118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc54c0c4860 msgr2=0x7fc54c1576a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:18.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.532+0000 7fc552ec7640 1 --2- 172.21.15.150:0/2514793118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc54c0c4860 0x7fc54c1576a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0c9e10 tx=0x7fc54401cc80 comp rx=0 tx=0).stop 2024-09-17T05:12:18.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.532+0000 7fc552ec7640 1 -- 172.21.15.150:0/2514793118 shutdown_connections 2024-09-17T05:12:18.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.532+0000 7fc552ec7640 1 --2- 172.21.15.150:0/2514793118 >> v2:172.21.15.176:3300/0 conn(0x7fc54c0c80a0 0x7fc54c0c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:18.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.532+0000 7fc552ec7640 1 --2- 172.21.15.150:0/2514793118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc54c0c4860 0x7fc54c1576a0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:18.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.532+0000 7fc552ec7640 1 -- 172.21.15.150:0/2514793118 >> 172.21.15.150:0/2514793118 conn(0x7fc54c0aaaf0 msgr2=0x7fc54c0aaf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:18.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.532+0000 7fc552ec7640 1 -- 172.21.15.150:0/2514793118 shutdown_connections 2024-09-17T05:12:18.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.532+0000 7fc552ec7640 1 -- 172.21.15.150:0/2514793118 wait complete. 2024-09-17T05:12:18.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.536+0000 7fc552ec7640 1 Processor -- start 2024-09-17T05:12:18.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.536+0000 7fc552ec7640 1 -- start start 2024-09-17T05:12:18.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.536+0000 7fc552ec7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc54c0c4860 0x7fc54c0adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:18.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.536+0000 7fc552ec7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc54c0c80a0 0x7fc54c1516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:18.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.536+0000 7fc552ec7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc54c0c9210 0x7fc54c1f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:18.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.536+0000 7fc552ec7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc54c1606a0 con 0x7fc54c0c9210 2024-09-17T05:12:18.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.536+0000 7fc552ec7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc54c0c56c0 con 0x7fc54c0c80a0 2024-09-17T05:12:18.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.536+0000 7fc552ec7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc54c160820 con 0x7fc54c0c4860 2024-09-17T05:12:18.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.536+0000 7fc551464640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc54c0c9210 0x7fc54c1f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:18.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.536+0000 7fc551464640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc54c0c9210 0x7fc54c1f3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49602/0 (socket says 172.21.15.150:49602) 2024-09-17T05:12:18.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.536+0000 7fc551464640 1 -- 172.21.15.150:0/1388585006 learned_addr learned my addr 172.21.15.150:0/1388585006 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:18.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc54bfff640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc54c0c80a0 0x7fc54c1516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:18.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc550c63640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc54c0c4860 0x7fc54c0adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:18.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc551464640 1 -- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc54c0c4860 msgr2=0x7fc54c0adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:18.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc551464640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc54c0c4860 0x7fc54c0adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:18.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc551464640 1 -- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc54c0c80a0 msgr2=0x7fc54c1516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:18.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc551464640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc54c0c80a0 0x7fc54c1516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:18.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc551464640 1 -- 172.21.15.150:0/1388585006 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc54c14fcd0 con 0x7fc54c0c9210 2024-09-17T05:12:18.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc551464640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc54c0c9210 0x7fc54c1f3770 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00b440 tx=0x7fc53c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:18.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc5497fa640 1 -- 172.21.15.150:0/1388585006 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc53c015020 con 0x7fc54c0c9210 2024-09-17T05:12:18.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc5497fa640 1 -- 172.21.15.150:0/1388585006 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc53c011940 con 0x7fc54c0c9210 2024-09-17T05:12:18.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc5497fa640 1 -- 172.21.15.150:0/1388585006 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc53c011d90 con 0x7fc54c0c9210 2024-09-17T05:12:18.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc552ec7640 1 -- 172.21.15.150:0/1388585006 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc54c14fa30 con 0x7fc54c0c9210 2024-09-17T05:12:18.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc552ec7640 1 -- 172.21.15.150:0/1388585006 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc54c0aa8f0 con 0x7fc54c0c9210 2024-09-17T05:12:18.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.540+0000 7fc552ec7640 1 -- 172.21.15.150:0/1388585006 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc54c1576a0 con 0x7fc54c0c9210 2024-09-17T05:12:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.548+0000 7fc5497fa640 1 -- 172.21.15.150:0/1388585006 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc53c018020 con 0x7fc54c0c9210 2024-09-17T05:12:18.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.548+0000 7fc5497fa640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc538073200 0x7fc538075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:18.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.548+0000 7fc5497fa640 1 -- 172.21.15.150:0/1388585006 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7fc53c095350 con 0x7fc54c0c9210 2024-09-17T05:12:18.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.552+0000 7fc5497fa640 1 -- 172.21.15.150:0/1388585006 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc53c01f020 con 0x7fc54c0c9210 2024-09-17T05:12:18.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.552+0000 7fc550c63640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc538073200 0x7fc538075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:18.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.552+0000 7fc550c63640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc538073200 0x7fc538075710 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc5440af000 tx=0x7fc54401b4c0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:18.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.956+0000 7fc552ec7640 1 -- 172.21.15.150:0/1388585006 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc54c0a9e00 con 0x7fc54c0c9210 2024-09-17T05:12:18.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.960+0000 7fc5497fa640 1 -- 172.21.15.150:0/1388585006 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) ==== 95+0+6541 (secure 0 0 0) 0x7fc54c14fa30 con 0x7fc54c0c9210 2024-09-17T05:12:18.959 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:18.959 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":161,"btime":"2024-09-17T05:12:11:699234+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7530,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:10:54.175436+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6799":{"gid":6799,"name":"d","rank":0,"incarnation":152,"state":"up:active","state_seq":17,"addr":"172.21.15.176:6837/2962113435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2962113435},{"type":"v1","addr":"172.21.15.176:6837","nonce":2962113435}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6799,"qdb_cluster":[6799]},"id":2}]} 2024-09-17T05:12:18.959 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 161 2024-09-17T05:12:18.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.964+0000 7fc552ec7640 1 -- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc538073200 msgr2=0x7fc538075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:18.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.964+0000 7fc552ec7640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc538073200 0x7fc538075710 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc5440af000 tx=0x7fc54401b4c0 comp rx=0 tx=0).stop 2024-09-17T05:12:18.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.964+0000 7fc552ec7640 1 -- 172.21.15.150:0/1388585006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc54c0c9210 msgr2=0x7fc54c1f3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:18.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.964+0000 7fc552ec7640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc54c0c9210 0x7fc54c1f3770 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00b440 tx=0x7fc53c00b900 comp rx=0 tx=0).stop 2024-09-17T05:12:18.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.964+0000 7fc552ec7640 1 -- 172.21.15.150:0/1388585006 shutdown_connections 2024-09-17T05:12:18.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.964+0000 7fc552ec7640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc538073200 0x7fc538075710 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:18.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.964+0000 7fc552ec7640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc54c0c80a0 0x7fc54c1516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:18.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.964+0000 7fc552ec7640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc54c0c9210 0x7fc54c1f3770 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:18.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.964+0000 7fc552ec7640 1 --2- 172.21.15.150:0/1388585006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc54c0c4860 0x7fc54c0adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:18.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.964+0000 7fc552ec7640 1 -- 172.21.15.150:0/1388585006 >> 172.21.15.150:0/1388585006 conn(0x7fc54c0aaaf0 msgr2=0x7fc54c0c6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:18.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.964+0000 7fc552ec7640 1 -- 172.21.15.150:0/1388585006 shutdown_connections 2024-09-17T05:12:18.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:18.964+0000 7fc552ec7640 1 -- 172.21.15.150:0/1388585006 wait complete. 2024-09-17T05:12:18.993 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:12:20.994 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:21.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.252+0000 7fe3c5dc2640 1 -- 172.21.15.150:0/1724687039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c00c80b0 msgr2=0x7fe3c00c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:21.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.252+0000 7fe3c5dc2640 1 --2- 172.21.15.150:0/1724687039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c00c80b0 0x7fe3c00c84c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009840 tx=0x7fe3b001c820 comp rx=0 tx=0).stop 2024-09-17T05:12:21.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.252+0000 7fe3c5dc2640 1 -- 172.21.15.150:0/1724687039 shutdown_connections 2024-09-17T05:12:21.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.252+0000 7fe3c5dc2640 1 --2- 172.21.15.150:0/1724687039 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c00723e0 0x7fe3c00727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:21.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.252+0000 7fe3c5dc2640 1 --2- 172.21.15.150:0/1724687039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c00c80b0 0x7fe3c00c84c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:21.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.252+0000 7fe3c5dc2640 1 -- 172.21.15.150:0/1724687039 >> 172.21.15.150:0/1724687039 conn(0x7fe3c00c9180 msgr2=0x7fe3c00c9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:21.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.252+0000 7fe3c5dc2640 1 -- 172.21.15.150:0/1724687039 shutdown_connections 2024-09-17T05:12:21.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.252+0000 7fe3c5dc2640 1 -- 172.21.15.150:0/1724687039 wait complete. 2024-09-17T05:12:21.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.256+0000 7fe3c5dc2640 1 Processor -- start 2024-09-17T05:12:21.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.256+0000 7fe3c5dc2640 1 -- start start 2024-09-17T05:12:21.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.256+0000 7fe3c5dc2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c00723e0 0x7fe3c0163900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:21.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.256+0000 7fe3c5dc2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c00c4830 0x7fe3c00ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:21.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.256+0000 7fe3c5dc2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c00c80b0 0x7fe3c0159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:21.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.256+0000 7fe3c5dc2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c00c7410 con 0x7fe3c00c4830 2024-09-17T05:12:21.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.256+0000 7fe3bffff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c00c80b0 0x7fe3c0159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:21.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.256+0000 7fe3bffff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c00c80b0 0x7fe3c0159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:48662/0 (socket says 172.21.15.150:48662) 2024-09-17T05:12:21.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.256+0000 7fe3bffff640 1 -- 172.21.15.150:0/452514174 learned_addr learned my addr 172.21.15.150:0/452514174 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:21.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3bf7fe640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c00723e0 0x7fe3c0163900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:21.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3beffd640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c00c4830 0x7fe3c00ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:21.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.256+0000 7fe3c5dc2640 1 -- 172.21.15.150:0/452514174 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c00c7290 con 0x7fe3c00723e0 2024-09-17T05:12:21.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3c5dc2640 1 -- 172.21.15.150:0/452514174 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c00c7590 con 0x7fe3c00c80b0 2024-09-17T05:12:21.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3bffff640 1 -- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c00723e0 msgr2=0x7fe3c0163900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:21.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3bffff640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c00723e0 0x7fe3c0163900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:21.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3bffff640 1 -- 172.21.15.150:0/452514174 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c00c4830 msgr2=0x7fe3c00ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:21.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3bffff640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c00c4830 0x7fe3c00ae670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:21.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3bffff640 1 -- 172.21.15.150:0/452514174 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c01f4650 con 0x7fe3c00c80b0 2024-09-17T05:12:21.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3bf7fe640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c00723e0 0x7fe3c0163900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:12:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3bffff640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c00c80b0 0x7fe3c0159d70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800cae0 tx=0x7fe3b800cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3a7fff640 1 -- 172.21.15.150:0/452514174 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8016020 con 0x7fe3c00c80b0 2024-09-17T05:12:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3a7fff640 1 -- 172.21.15.150:0/452514174 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3b800ab40 con 0x7fe3c00c80b0 2024-09-17T05:12:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3a7fff640 1 -- 172.21.15.150:0/452514174 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b800ce20 con 0x7fe3c00c80b0 2024-09-17T05:12:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3c5dc2640 1 -- 172.21.15.150:0/452514174 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c015a2b0 con 0x7fe3c00c80b0 2024-09-17T05:12:21.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3c5dc2640 1 -- 172.21.15.150:0/452514174 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c0159570 con 0x7fe3c00c80b0 2024-09-17T05:12:21.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.260+0000 7fe3a5ffb640 1 -- 172.21.15.150:0/452514174 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3980052a0 con 0x7fe3c00c80b0 2024-09-17T05:12:21.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.264+0000 7fe3a7fff640 1 -- 172.21.15.150:0/452514174 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe3b8019020 con 0x7fe3c00c80b0 2024-09-17T05:12:21.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.264+0000 7fe3a7fff640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3ac073200 0x7fe3ac075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:21.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.264+0000 7fe3a7fff640 1 -- 172.21.15.150:0/452514174 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7fe3b80950e0 con 0x7fe3c00c80b0 2024-09-17T05:12:21.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.264+0000 7fe3bf7fe640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3ac073200 0x7fe3ac075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:21.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.264+0000 7fe3bf7fe640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3ac073200 0x7fe3ac075710 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00af000 tx=0x7fe3b001b060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:21.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.268+0000 7fe3a7fff640 1 -- 172.21.15.150:0/452514174 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe3b8020020 con 0x7fe3c00c80b0 2024-09-17T05:12:21.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.900+0000 7fe3a5ffb640 1 -- 172.21.15.150:0/452514174 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe398005590 con 0x7fe3c00c80b0 2024-09-17T05:12:21.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.904+0000 7fe3a7fff640 1 -- 172.21.15.150:0/452514174 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+5704 (secure 0 0 0) 0x7fe3b805edb0 con 0x7fe3c00c80b0 2024-09-17T05:12:21.903 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:21.903 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":162,"btime":"2024-09-17T05:12:21:180028+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:12:21.180022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7530":{"gid":7530,"name":"a","rank":0,"incarnation":162,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:12:21.903 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 162 2024-09-17T05:12:21.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.908+0000 7fe3a5ffb640 1 -- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3ac073200 msgr2=0x7fe3ac075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:21.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.908+0000 7fe3a5ffb640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3ac073200 0x7fe3ac075710 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe3b00af000 tx=0x7fe3b001b060 comp rx=0 tx=0).stop 2024-09-17T05:12:21.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.908+0000 7fe3a5ffb640 1 -- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c00c80b0 msgr2=0x7fe3c0159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:21.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.908+0000 7fe3a5ffb640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c00c80b0 0x7fe3c0159d70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800cae0 tx=0x7fe3b800cbd0 comp rx=0 tx=0).stop 2024-09-17T05:12:21.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.908+0000 7fe3a5ffb640 1 -- 172.21.15.150:0/452514174 shutdown_connections 2024-09-17T05:12:21.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.908+0000 7fe3a5ffb640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3ac073200 0x7fe3ac075710 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:21.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.908+0000 7fe3a5ffb640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3c00c80b0 0x7fe3c0159d70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:21.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.908+0000 7fe3a5ffb640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3c00c4830 0x7fe3c00ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:21.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.908+0000 7fe3a5ffb640 1 --2- 172.21.15.150:0/452514174 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3c00723e0 0x7fe3c0163900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:21.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.908+0000 7fe3a5ffb640 1 -- 172.21.15.150:0/452514174 >> 172.21.15.150:0/452514174 conn(0x7fe3c00c9180 msgr2=0x7fe3c00a8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:21.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.908+0000 7fe3a5ffb640 1 -- 172.21.15.150:0/452514174 shutdown_connections 2024-09-17T05:12:21.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:21.908+0000 7fe3a5ffb640 1 -- 172.21.15.150:0/452514174 wait complete. 2024-09-17T05:12:21.938 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-17T05:12:21.938 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:12:21.938 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:22.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.292+0000 7f489b88a640 1 -- 172.21.15.150:0/3065180559 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f489416ce80 msgr2=0x7f489416f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:22.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.292+0000 7f489b88a640 1 --2- 172.21.15.150:0/3065180559 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f489416ce80 0x7f489416f2a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4884005e80 tx=0x7f488401ced0 comp rx=0 tx=0).stop 2024-09-17T05:12:22.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.296+0000 7f489b88a640 1 -- 172.21.15.150:0/3065180559 shutdown_connections 2024-09-17T05:12:22.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.296+0000 7f489b88a640 1 --2- 172.21.15.150:0/3065180559 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f489416ce80 0x7f489416f2a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:22.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.296+0000 7f489b88a640 1 -- 172.21.15.150:0/3065180559 >> 172.21.15.150:0/3065180559 conn(0x7f48940c4e30 msgr2=0x7f489406ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:22.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.296+0000 7f489b88a640 1 -- 172.21.15.150:0/3065180559 shutdown_connections 2024-09-17T05:12:22.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.296+0000 7f489b88a640 1 -- 172.21.15.150:0/3065180559 wait complete. 2024-09-17T05:12:22.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.296+0000 7f489b88a640 1 Processor -- start 2024-09-17T05:12:22.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.296+0000 7f489b88a640 1 -- start start 2024-09-17T05:12:22.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f489b88a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4894074bd0 0x7f489408b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:22.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f489b88a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48940a74b0 0x7f48940ac610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:22.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f489b88a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f489416ce80 0x7f48940a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:22.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f489b88a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f489406dc70 con 0x7f48940a74b0 2024-09-17T05:12:22.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f489b88a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f48940c3b60 con 0x7f489416ce80 2024-09-17T05:12:22.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f489b88a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f489406ddf0 con 0x7f4894074bd0 2024-09-17T05:12:22.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4898e25640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48940a74b0 0x7f48940ac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:22.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4898e25640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48940a74b0 0x7f48940ac610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44916/0 (socket says 172.21.15.150:44916) 2024-09-17T05:12:22.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4898e25640 1 -- 172.21.15.150:0/699608661 learned_addr learned my addr 172.21.15.150:0/699608661 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:22.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4899e27640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f489416ce80 0x7f48940a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:22.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4899626640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4894074bd0 0x7f489408b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:22.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4898e25640 1 -- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4894074bd0 msgr2=0x7f489408b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:22.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4898e25640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4894074bd0 0x7f489408b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:22.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4898e25640 1 -- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f489416ce80 msgr2=0x7f48940a3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:22.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4898e25640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f489416ce80 0x7f48940a3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:22.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4898e25640 1 -- 172.21.15.150:0/699608661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48940aade0 con 0x7f48940a74b0 2024-09-17T05:12:22.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4898e25640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48940a74b0 0x7f48940ac610 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f488c00c8a0 tx=0x7f488c00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:22.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4889ffb640 1 -- 172.21.15.150:0/699608661 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c017070 con 0x7f48940a74b0 2024-09-17T05:12:22.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4889ffb640 1 -- 172.21.15.150:0/699608661 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f488c005070 con 0x7f48940a74b0 2024-09-17T05:12:22.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f489b88a640 1 -- 172.21.15.150:0/699608661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4894205760 con 0x7f48940a74b0 2024-09-17T05:12:22.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f4889ffb640 1 -- 172.21.15.150:0/699608661 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c0054a0 con 0x7f48940a74b0 2024-09-17T05:12:22.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.300+0000 7f489b88a640 1 -- 172.21.15.150:0/699608661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48940acb50 con 0x7f48940a74b0 2024-09-17T05:12:22.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.304+0000 7f489b88a640 1 -- 172.21.15.150:0/699608661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48940ace40 con 0x7f48940a74b0 2024-09-17T05:12:22.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.304+0000 7f4889ffb640 1 -- 172.21.15.150:0/699608661 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f488c01a070 con 0x7f48940a74b0 2024-09-17T05:12:22.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.304+0000 7f4889ffb640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4878073200 0x7f4878075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:22.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.304+0000 7f4899626640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4878073200 0x7f4878075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:22.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.304+0000 7f4899626640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4878073200 0x7f4878075710 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4894072650 tx=0x7f4890009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:22.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.312+0000 7f4889ffb640 1 -- 172.21.15.150:0/699608661 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f488c098060 con 0x7f48940a74b0 2024-09-17T05:12:22.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.320+0000 7f4889ffb640 1 -- 172.21.15.150:0/699608661 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f488c05e0a0 con 0x7f48940a74b0 2024-09-17T05:12:22.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.804+0000 7f489b88a640 1 -- 172.21.15.150:0/699608661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4894205550 con 0x7f48940a74b0 2024-09-17T05:12:22.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.808+0000 7f4889ffb640 1 -- 172.21.15.150:0/699608661 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+5704 (secure 0 0 0) 0x7f488c062ad0 con 0x7f48940a74b0 2024-09-17T05:12:22.807 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:22.807 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":162,"btime":"2024-09-17T05:12:21:180028+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:12:21.180022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7530":{"gid":7530,"name":"a","rank":0,"incarnation":162,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:12:22.807 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 162 2024-09-17T05:12:22.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.812+0000 7f489b88a640 1 -- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4878073200 msgr2=0x7f4878075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:22.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.812+0000 7f489b88a640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4878073200 0x7f4878075710 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4894072650 tx=0x7f4890009040 comp rx=0 tx=0).stop 2024-09-17T05:12:22.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.812+0000 7f489b88a640 1 -- 172.21.15.150:0/699608661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48940a74b0 msgr2=0x7f48940ac610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:22.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.812+0000 7f489b88a640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48940a74b0 0x7f48940ac610 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f488c00c8a0 tx=0x7f488c00cd60 comp rx=0 tx=0).stop 2024-09-17T05:12:22.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.812+0000 7f489b88a640 1 -- 172.21.15.150:0/699608661 shutdown_connections 2024-09-17T05:12:22.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.812+0000 7f489b88a640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4878073200 0x7f4878075710 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:22.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.812+0000 7f489b88a640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f489416ce80 0x7f48940a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:22.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.812+0000 7f489b88a640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48940a74b0 0x7f48940ac610 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:22.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.812+0000 7f489b88a640 1 --2- 172.21.15.150:0/699608661 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4894074bd0 0x7f489408b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:22.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.812+0000 7f489b88a640 1 -- 172.21.15.150:0/699608661 >> 172.21.15.150:0/699608661 conn(0x7f48940c4e30 msgr2=0x7f4894073c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:22.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.812+0000 7f489b88a640 1 -- 172.21.15.150:0/699608661 shutdown_connections 2024-09-17T05:12:22.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:22.812+0000 7f489b88a640 1 -- 172.21.15.150:0/699608661 wait complete. 2024-09-17T05:12:22.856 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:12:21:180028+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 162, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 160, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7407": { "addr": "172.21.15.150:6835/3179072684", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 3179072684, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 3179072684, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7407, "incarnation": 157, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:12:04.469086+0000", "qdb_cluster": [ 7407 ], "qdb_leader": 7407, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7407 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 162, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7530": { "addr": "172.21.15.150:6833/12982302", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 12982302, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 12982302, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7530, "incarnation": 162, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:12:21.180022+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7530 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:12:24.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:25.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.096+0000 7f2d796cd640 1 -- 172.21.15.150:0/1528941717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d5c004810 msgr2=0x7f2d5c004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:25.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.096+0000 7f2d796cd640 1 --2- 172.21.15.150:0/1528941717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d5c004810 0x7f2d5c004c20 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800a980 tx=0x7f2d6801c810 comp rx=0 tx=0).stop 2024-09-17T05:12:25.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.096+0000 7f2d737fe640 1 -- 172.21.15.150:0/1528941717 reap_dead start 2024-09-17T05:12:25.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.096+0000 7f2d796cd640 1 -- 172.21.15.150:0/1528941717 shutdown_connections 2024-09-17T05:12:25.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.096+0000 7f2d796cd640 1 -- 172.21.15.150:0/1528941717 >> 172.21.15.150:0/1528941717 conn(0x7f2d740ca460 msgr2=0x7f2d740c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:25.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.100+0000 7f2d796cd640 1 -- 172.21.15.150:0/1528941717 shutdown_connections 2024-09-17T05:12:25.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.100+0000 7f2d796cd640 1 -- 172.21.15.150:0/1528941717 wait complete. 2024-09-17T05:12:25.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.100+0000 7f2d796cd640 1 Processor -- start 2024-09-17T05:12:25.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.100+0000 7f2d796cd640 1 -- start start 2024-09-17T05:12:25.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d796cd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d74152200 0x7f2d74157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:25.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d796cd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d74157f90 0x7f2d74150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:25.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d796cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d74151490 0x7f2d741518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:25.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d796cd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2d740c9bd0 con 0x7f2d74151490 2024-09-17T05:12:25.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d796cd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2d740c9a50 con 0x7f2d74152200 2024-09-17T05:12:25.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d727fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d74157f90 0x7f2d74150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:25.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d727fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d74157f90 0x7f2d74150f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:48700/0 (socket says 172.21.15.150:48700) 2024-09-17T05:12:25.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d727fc640 1 -- 172.21.15.150:0/386550638 learned_addr learned my addr 172.21.15.150:0/386550638 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:25.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d737fe640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d74151490 0x7f2d741518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:25.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d796cd640 1 -- 172.21.15.150:0/386550638 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2d740c31b0 con 0x7f2d74157f90 2024-09-17T05:12:25.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d72ffd640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d74152200 0x7f2d74157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:25.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d727fc640 1 -- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d74152200 msgr2=0x7f2d74157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:25.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d727fc640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d74152200 0x7f2d74157a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:25.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d727fc640 1 -- 172.21.15.150:0/386550638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d74151490 msgr2=0x7f2d741518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:25.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d727fc640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d74151490 0x7f2d741518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:25.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d727fc640 1 -- 172.21.15.150:0/386550638 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d7414cde0 con 0x7f2d74157f90 2024-09-17T05:12:25.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d727fc640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d74157f90 0x7f2d74150f50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2d680af040 tx=0x7f2d680a5d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:25.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d5b7fe640 1 -- 172.21.15.150:0/386550638 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6800a580 con 0x7f2d74157f90 2024-09-17T05:12:25.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d5b7fe640 1 -- 172.21.15.150:0/386550638 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d680b7550 con 0x7f2d74157f90 2024-09-17T05:12:25.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.104+0000 7f2d5b7fe640 1 -- 172.21.15.150:0/386550638 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68005c80 con 0x7f2d74157f90 2024-09-17T05:12:25.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.108+0000 7f2d796cd640 1 -- 172.21.15.150:0/386550638 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d741f9f20 con 0x7f2d74157f90 2024-09-17T05:12:25.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.108+0000 7f2d796cd640 1 -- 172.21.15.150:0/386550638 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d74153940 con 0x7f2d74157f90 2024-09-17T05:12:25.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.112+0000 7f2d5b7fe640 1 -- 172.21.15.150:0/386550638 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2d680bf020 con 0x7f2d74157f90 2024-09-17T05:12:25.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.112+0000 7f2d796cd640 1 -- 172.21.15.150:0/386550638 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d740c8f00 con 0x7f2d74157f90 2024-09-17T05:12:25.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.112+0000 7f2d5b7fe640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d60073280 0x7f2d60075790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:25.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.112+0000 7f2d5b7fe640 1 -- 172.21.15.150:0/386550638 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f2d6812f410 con 0x7f2d74157f90 2024-09-17T05:12:25.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.112+0000 7f2d72ffd640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d60073280 0x7f2d60075790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:25.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.116+0000 7f2d72ffd640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d60073280 0x7f2d60075790 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2d64000c10 tx=0x7f2d64001320 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:25.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.124+0000 7f2d5b7fe640 1 -- 172.21.15.150:0/386550638 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2d680c6020 con 0x7f2d74157f90 2024-09-17T05:12:25.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.544+0000 7f2d796cd640 1 -- 172.21.15.150:0/386550638 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d740ac920 con 0x7f2d74157f90 2024-09-17T05:12:25.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.548+0000 7f2d5b7fe640 1 -- 172.21.15.150:0/386550638 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) ==== 95+0+5711 (secure 0 0 0) 0x7f2d68016580 con 0x7f2d74157f90 2024-09-17T05:12:25.550 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:25.550 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":165,"btime":"2024-09-17T05:12:25:196444+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:12:25.196435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7530":{"gid":7530,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7530,"qdb_cluster":[7530]},"id":2}]} 2024-09-17T05:12:25.550 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 165 2024-09-17T05:12:25.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.552+0000 7f2d796cd640 1 -- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d60073280 msgr2=0x7f2d60075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:25.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.552+0000 7f2d796cd640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d60073280 0x7f2d60075790 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2d64000c10 tx=0x7f2d64001320 comp rx=0 tx=0).stop 2024-09-17T05:12:25.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.552+0000 7f2d796cd640 1 -- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d74157f90 msgr2=0x7f2d74150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.552+0000 7f2d796cd640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d74157f90 0x7f2d74150f50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2d680af040 tx=0x7f2d680a5d10 comp rx=0 tx=0).stop 2024-09-17T05:12:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.552+0000 7f2d796cd640 1 -- 172.21.15.150:0/386550638 shutdown_connections 2024-09-17T05:12:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.552+0000 7f2d796cd640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d60073280 0x7f2d60075790 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.552+0000 7f2d796cd640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d74151490 0x7f2d741518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.552+0000 7f2d796cd640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d74157f90 0x7f2d74150f50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.552+0000 7f2d796cd640 1 --2- 172.21.15.150:0/386550638 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d74152200 0x7f2d74157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.552+0000 7f2d796cd640 1 -- 172.21.15.150:0/386550638 >> 172.21.15.150:0/386550638 conn(0x7f2d740ca460 msgr2=0x7f2d740c66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.552+0000 7f2d796cd640 1 -- 172.21.15.150:0/386550638 shutdown_connections 2024-09-17T05:12:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:25.552+0000 7f2d796cd640 1 -- 172.21.15.150:0/386550638 wait complete. 2024-09-17T05:12:25.583 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=6799 2024-09-17T05:12:25.583 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.d 2024-09-17T05:12:26.584 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-17T05:12:26.585 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T05:12:26.585 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T05:12:26.587 INFO:tasks.ceph.mds.d:Started 2024-09-17T05:12:26.587 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:12:26.690 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-09-17T05:12:26.722 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T05:12:26.724+0000 7f86741db640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:12:26.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.840+0000 7f3a89ba9640 1 -- 172.21.15.150:0/3361119682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a84153430 msgr2=0x7f3a840c2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:26.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.840+0000 7f3a89ba9640 1 --2- 172.21.15.150:0/3361119682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a84153430 0x7f3a840c2c50 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f3a84153120 tx=0x7f3a7801d480 comp rx=0 tx=0).stop 2024-09-17T05:12:26.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.840+0000 7f3a89ba9640 1 -- 172.21.15.150:0/3361119682 shutdown_connections 2024-09-17T05:12:26.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.840+0000 7f3a89ba9640 1 --2- 172.21.15.150:0/3361119682 >> v2:172.21.15.176:3300/0 conn(0x7f3a840738d0 0x7f3a84073ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:26.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.840+0000 7f3a89ba9640 1 --2- 172.21.15.150:0/3361119682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a84153430 0x7f3a840c2c50 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:26.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.840+0000 7f3a89ba9640 1 -- 172.21.15.150:0/3361119682 >> 172.21.15.150:0/3361119682 conn(0x7f3a840c8250 msgr2=0x7f3a840c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:26.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.840+0000 7f3a89ba9640 1 -- 172.21.15.150:0/3361119682 shutdown_connections 2024-09-17T05:12:26.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.844+0000 7f3a89ba9640 1 -- 172.21.15.150:0/3361119682 wait complete. 2024-09-17T05:12:26.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.844+0000 7f3a89ba9640 1 Processor -- start 2024-09-17T05:12:26.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.844+0000 7f3a89ba9640 1 -- start start 2024-09-17T05:12:26.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a89ba9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a840738d0 0x7f3a840c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:26.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a837fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a840738d0 0x7f3a840c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:26.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a837fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a840738d0 0x7f3a840c6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44944/0 (socket says 172.21.15.150:44944) 2024-09-17T05:12:26.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a89ba9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a84074500 0x7f3a841491f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:26.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a837fe640 1 -- 172.21.15.150:0/448469857 learned_addr learned my addr 172.21.15.150:0/448469857 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:12:26.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a89ba9640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a84149730 0x7f3a841f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:26.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3a840c4af0 con 0x7f3a840738d0 2024-09-17T05:12:26.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3a840c4970 con 0x7f3a84074500 2024-09-17T05:12:26.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3a84072000 con 0x7f3a84149730 2024-09-17T05:12:26.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a83fff640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a84149730 0x7f3a841f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:26.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a83fff640 1 -- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a84074500 msgr2=0x7f3a841491f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:12:26.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a83fff640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a84074500 0x7f3a841491f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:26.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a83fff640 1 -- 172.21.15.150:0/448469857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a840738d0 msgr2=0x7f3a840c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:26.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a83fff640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a840738d0 0x7f3a840c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:26.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a83fff640 1 -- 172.21.15.150:0/448469857 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a840c7a10 con 0x7f3a84149730 2024-09-17T05:12:26.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a83fff640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a84149730 0x7f3a841f4cb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009fd0 tx=0x7f3a7400ead0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:26.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a6bfff640 1 -- 172.21.15.150:0/448469857 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74004220 con 0x7f3a84149730 2024-09-17T05:12:26.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a840c7450 con 0x7f3a84149730 2024-09-17T05:12:26.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.848+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a840c6210 con 0x7f3a84149730 2024-09-17T05:12:26.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.852+0000 7f3a6bfff640 1 -- 172.21.15.150:0/448469857 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a74005020 con 0x7f3a84149730 2024-09-17T05:12:26.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.852+0000 7f3a6bfff640 1 -- 172.21.15.150:0/448469857 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74005470 con 0x7f3a84149730 2024-09-17T05:12:26.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.852+0000 7f3a6bfff640 1 -- 172.21.15.150:0/448469857 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3a740047b0 con 0x7f3a84149730 2024-09-17T05:12:26.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.852+0000 7f3a6bfff640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3a70073200 0x7f3a70075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:12:26.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.852+0000 7f3a837fe640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3a70073200 0x7f3a70075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:12:26.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.852+0000 7f3a6bfff640 1 -- 172.21.15.150:0/448469857 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f3a740954f0 con 0x7f3a84149730 2024-09-17T05:12:26.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.856+0000 7f3a837fe640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3a70073200 0x7f3a70075710 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0020a0 tx=0x7f3a7c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:12:26.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.856+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a640052a0 con 0x7f3a84149730 2024-09-17T05:12:26.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:26.864+0000 7f3a6bfff640 1 -- 172.21.15.150:0/448469857 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3a7405f6f0 con 0x7f3a84149730 2024-09-17T05:12:27.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.312+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a64005590 con 0x7f3a84149730 2024-09-17T05:12:27.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.316+0000 7f3a6bfff640 1 -- 172.21.15.150:0/448469857 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6540 (secure 0 0 0) 0x7f3a7405eff0 con 0x7f3a84149730 2024-09-17T05:12:27.316 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:12:27.316 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":166,"btime":"2024-09-17T05:12:27:201243+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:12:25.196435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7530":{"gid":7530,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7530,"qdb_cluster":[7530]},"id":2}]} 2024-09-17T05:12:27.316 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 166 2024-09-17T05:12:27.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.320+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3a70073200 msgr2=0x7f3a70075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.320+0000 7f3a89ba9640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3a70073200 0x7f3a70075710 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0020a0 tx=0x7f3a7c009040 comp rx=0 tx=0).stop 2024-09-17T05:12:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.320+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a84149730 msgr2=0x7f3a841f4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:12:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.320+0000 7f3a89ba9640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a84149730 0x7f3a841f4cb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009fd0 tx=0x7f3a7400ead0 comp rx=0 tx=0).stop 2024-09-17T05:12:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.320+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 shutdown_connections 2024-09-17T05:12:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.320+0000 7f3a89ba9640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3a70073200 0x7f3a70075710 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.320+0000 7f3a89ba9640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3a84074500 0x7f3a841491f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.320+0000 7f3a89ba9640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3a84149730 0x7f3a841f4cb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.320+0000 7f3a89ba9640 1 --2- 172.21.15.150:0/448469857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a840738d0 0x7f3a840c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:12:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.320+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 >> 172.21.15.150:0/448469857 conn(0x7f3a840c8250 msgr2=0x7f3a8415c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:12:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.320+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 shutdown_connections 2024-09-17T05:12:27.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:12:27.320+0000 7f3a89ba9640 1 -- 172.21.15.150:0/448469857 wait complete. 2024-09-17T05:12:27.352 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-17T05:12:27.352 INFO:tasks.mds_thrash.fs.[b]:waiting for 62 secs before thrashing 2024-09-17T05:12:34.167 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:12:34.169 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:13:04.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:13:04.495 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:13:29.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:13:29.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.726+0000 7eff349c2640 1 -- 172.21.15.150:0/1648001380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff280d79a0 msgr2=0x7eff280d4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:29.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.726+0000 7eff349c2640 1 --2- 172.21.15.150:0/1648001380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff280d79a0 0x7eff280d4330 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7eff1c009840 tx=0x7eff1c01c630 comp rx=0 tx=0).stop 2024-09-17T05:13:29.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.730+0000 7eff349c2640 1 -- 172.21.15.150:0/1648001380 shutdown_connections 2024-09-17T05:13:29.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.730+0000 7eff349c2640 1 --2- 172.21.15.150:0/1648001380 >> v2:172.21.15.176:3300/0 conn(0x7eff280d4870 0x7eff28007d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:29.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.730+0000 7eff349c2640 1 --2- 172.21.15.150:0/1648001380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff280d79a0 0x7eff280d4330 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:29.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.730+0000 7eff349c2640 1 -- 172.21.15.150:0/1648001380 >> 172.21.15.150:0/1648001380 conn(0x7eff28049590 msgr2=0x7eff2802e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:29.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.730+0000 7eff349c2640 1 -- 172.21.15.150:0/1648001380 shutdown_connections 2024-09-17T05:13:29.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.730+0000 7eff349c2640 1 -- 172.21.15.150:0/1648001380 wait complete. 2024-09-17T05:13:29.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.734+0000 7eff349c2640 1 Processor -- start 2024-09-17T05:13:29.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.734+0000 7eff349c2640 1 -- start start 2024-09-17T05:13:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff349c2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff280478a0 0x7eff280da880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff349c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff280d4870 0x7eff28043d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff349c2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff280d79a0 0x7eff280cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff349c2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7eff28006bc0 con 0x7eff280d4870 2024-09-17T05:13:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff2f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff280478a0 0x7eff280da880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff2f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff280478a0 0x7eff280da880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:39238/0 (socket says 172.21.15.150:39238) 2024-09-17T05:13:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff2f7fe640 1 -- 172.21.15.150:0/2484007589 learned_addr learned my addr 172.21.15.150:0/2484007589 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:13:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff2effd640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff280d4870 0x7eff28043d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff349c2640 1 -- 172.21.15.150:0/2484007589 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7eff28006a40 con 0x7eff280d79a0 2024-09-17T05:13:29.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff2ffff640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff280d79a0 0x7eff280cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:29.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff349c2640 1 -- 172.21.15.150:0/2484007589 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7eff28006d40 con 0x7eff280478a0 2024-09-17T05:13:29.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff2effd640 1 -- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff280478a0 msgr2=0x7eff280da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:29.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff2effd640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff280478a0 0x7eff280da880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:29.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff2effd640 1 -- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff280d79a0 msgr2=0x7eff280cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:29.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff2effd640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff280d79a0 0x7eff280cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:29.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff2effd640 1 -- 172.21.15.150:0/2484007589 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff280456b0 con 0x7eff280d4870 2024-09-17T05:13:29.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff2effd640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff280d4870 0x7eff28043d90 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0ac000 tx=0x7eff1c0ac4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:29.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff1bfff640 1 -- 172.21.15.150:0/2484007589 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0b1070 con 0x7eff280d4870 2024-09-17T05:13:29.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff1bfff640 1 -- 172.21.15.150:0/2484007589 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff1c0be070 con 0x7eff280d4870 2024-09-17T05:13:29.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.738+0000 7eff1bfff640 1 -- 172.21.15.150:0/2484007589 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1c005020 con 0x7eff280d4870 2024-09-17T05:13:29.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.742+0000 7eff349c2640 1 -- 172.21.15.150:0/2484007589 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff280450f0 con 0x7eff280d4870 2024-09-17T05:13:29.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.746+0000 7eff349c2640 1 -- 172.21.15.150:0/2484007589 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff2802e3d0 con 0x7eff280d4870 2024-09-17T05:13:29.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.746+0000 7eff19ffb640 1 -- 172.21.15.150:0/2484007589 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff080052a0 con 0x7eff280d4870 2024-09-17T05:13:29.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.746+0000 7eff1bfff640 1 -- 172.21.15.150:0/2484007589 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7eff1c0051c0 con 0x7eff280d4870 2024-09-17T05:13:29.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.746+0000 7eff1bfff640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff14073250 0x7eff14075760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:29.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.746+0000 7eff1bfff640 1 -- 172.21.15.150:0/2484007589 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7eff1c136d90 con 0x7eff280d4870 2024-09-17T05:13:29.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.746+0000 7eff2f7fe640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff14073250 0x7eff14075760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:29.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.750+0000 7eff2f7fe640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff14073250 0x7eff14075760 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7eff24009e60 tx=0x7eff24003720 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:29.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:29.754+0000 7eff1bfff640 1 -- 172.21.15.150:0/2484007589 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7eff280456b0 con 0x7eff280d4870 2024-09-17T05:13:30.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.338+0000 7eff19ffb640 1 -- 172.21.15.150:0/2484007589 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff08005590 con 0x7eff280d4870 2024-09-17T05:13:30.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.338+0000 7eff1bfff640 1 -- 172.21.15.150:0/2484007589 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6540 (secure 0 0 0) 0x7eff1c016330 con 0x7eff280d4870 2024-09-17T05:13:30.341 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:13:30.341 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":166,"btime":"2024-09-17T05:12:27:201243+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:12:25.196435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7530":{"gid":7530,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7530,"qdb_cluster":[7530]},"id":2}]} 2024-09-17T05:13:30.341 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 166 2024-09-17T05:13:30.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.342+0000 7eff19ffb640 1 -- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff14073250 msgr2=0x7eff14075760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.342+0000 7eff19ffb640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff14073250 0x7eff14075760 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7eff24009e60 tx=0x7eff24003720 comp rx=0 tx=0).stop 2024-09-17T05:13:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.342+0000 7eff19ffb640 1 -- 172.21.15.150:0/2484007589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff280d4870 msgr2=0x7eff28043d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.342+0000 7eff19ffb640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff280d4870 0x7eff28043d90 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7eff1c0ac000 tx=0x7eff1c0ac4c0 comp rx=0 tx=0).stop 2024-09-17T05:13:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.342+0000 7eff19ffb640 1 -- 172.21.15.150:0/2484007589 shutdown_connections 2024-09-17T05:13:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.342+0000 7eff19ffb640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7eff14073250 0x7eff14075760 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.342+0000 7eff19ffb640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff280d79a0 0x7eff280cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.342+0000 7eff19ffb640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff280d4870 0x7eff28043d90 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.342+0000 7eff19ffb640 1 --2- 172.21.15.150:0/2484007589 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7eff280478a0 0x7eff280da880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.342+0000 7eff19ffb640 1 -- 172.21.15.150:0/2484007589 >> 172.21.15.150:0/2484007589 conn(0x7eff28049590 msgr2=0x7eff2804af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.342+0000 7eff19ffb640 1 -- 172.21.15.150:0/2484007589 shutdown_connections 2024-09-17T05:13:30.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.342+0000 7eff19ffb640 1 -- 172.21.15.150:0/2484007589 wait complete. 2024-09-17T05:13:30.411 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-17T05:13:30.411 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T05:13:30.411 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:13:30.427 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T05:13:30.427 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:13:30.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.794+0000 7f4e5885e640 1 -- 172.21.15.150:0/1508923370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e4c00b8f0 msgr2=0x7f4e4c005c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:30.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.794+0000 7f4e5885e640 1 --2- 172.21.15.150:0/1508923370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e4c00b8f0 0x7f4e4c005c10 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00d000 tx=0x7f4e54099420 comp rx=0 tx=0).stop 2024-09-17T05:13:30.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.806+0000 7f4e5885e640 1 -- 172.21.15.150:0/1508923370 shutdown_connections 2024-09-17T05:13:30.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.806+0000 7f4e5885e640 1 --2- 172.21.15.150:0/1508923370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e4c00b8f0 0x7f4e4c005c10 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:30.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.806+0000 7f4e5885e640 1 --2- 172.21.15.150:0/1508923370 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e4c02d5f0 0x7f4e4c02da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:30.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.806+0000 7f4e5885e640 1 --2- 172.21.15.150:0/1508923370 >> v2:172.21.15.176:3300/0 conn(0x7f4e4c030cc0 0x7f4e4c0d5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:30.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.806+0000 7f4e5885e640 1 -- 172.21.15.150:0/1508923370 >> 172.21.15.150:0/1508923370 conn(0x7f4e4c048630 msgr2=0x7f4e4c02cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:30.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.806+0000 7f4e5885e640 1 -- 172.21.15.150:0/1508923370 shutdown_connections 2024-09-17T05:13:30.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.806+0000 7f4e5885e640 1 -- 172.21.15.150:0/1508923370 wait complete. 2024-09-17T05:13:30.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e5885e640 1 Processor -- start 2024-09-17T05:13:30.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e5885e640 1 -- start start 2024-09-17T05:13:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e5885e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e4c00b8f0 0x7f4e4c0ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e5885e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e4c02d5f0 0x7f4e4c0d88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e5885e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e4c030cc0 0x7f4e4c0cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e5885e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4c0d6070 con 0x7f4e4c00b8f0 2024-09-17T05:13:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e5885e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4c0d6c80 con 0x7f4e4c030cc0 2024-09-17T05:13:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e5885e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4e4c0da250 con 0x7f4e4c02d5f0 2024-09-17T05:13:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e5359e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e4c030cc0 0x7f4e4c0cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e52d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e4c00b8f0 0x7f4e4c0ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:30.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e52d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e4c00b8f0 0x7f4e4c0ceae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52612/0 (socket says 172.21.15.150:52612) 2024-09-17T05:13:30.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e52d9d640 1 -- 172.21.15.150:0/3485008203 learned_addr learned my addr 172.21.15.150:0/3485008203 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:13:30.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.810+0000 7f4e5259c640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e4c02d5f0 0x7f4e4c0d88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:30.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.814+0000 7f4e5259c640 1 -- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e4c030cc0 msgr2=0x7f4e4c0cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:30.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.814+0000 7f4e5259c640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e4c030cc0 0x7f4e4c0cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:30.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.814+0000 7f4e5259c640 1 -- 172.21.15.150:0/3485008203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e4c00b8f0 msgr2=0x7f4e4c0ceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:30.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.814+0000 7f4e5259c640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e4c00b8f0 0x7f4e4c0ceae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:30.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.814+0000 7f4e5259c640 1 -- 172.21.15.150:0/3485008203 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e4c0cf020 con 0x7f4e4c02d5f0 2024-09-17T05:13:30.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.814+0000 7f4e5259c640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e4c02d5f0 0x7f4e4c0d88e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800b470 tx=0x7f4e4800b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:30.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.814+0000 7f4e3b7fe640 1 -- 172.21.15.150:0/3485008203 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e48015020 con 0x7f4e4c02d5f0 2024-09-17T05:13:30.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.814+0000 7f4e3b7fe640 1 -- 172.21.15.150:0/3485008203 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4e48011920 con 0x7f4e4c02d5f0 2024-09-17T05:13:30.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.814+0000 7f4e3b7fe640 1 -- 172.21.15.150:0/3485008203 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e48011d70 con 0x7f4e4c02d5f0 2024-09-17T05:13:30.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.814+0000 7f4e5885e640 1 -- 172.21.15.150:0/3485008203 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e4c0cb4f0 con 0x7f4e4c02d5f0 2024-09-17T05:13:30.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.814+0000 7f4e5885e640 1 -- 172.21.15.150:0/3485008203 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e4c041e70 con 0x7f4e4c02d5f0 2024-09-17T05:13:30.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.814+0000 7f4e5885e640 1 -- 172.21.15.150:0/3485008203 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e4c02da00 con 0x7f4e4c02d5f0 2024-09-17T05:13:30.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.818+0000 7f4e3b7fe640 1 -- 172.21.15.150:0/3485008203 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4e48018020 con 0x7f4e4c02d5f0 2024-09-17T05:13:30.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.818+0000 7f4e3b7fe640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4e40073230 0x7f4e40075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:30.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.818+0000 7f4e3b7fe640 1 -- 172.21.15.150:0/3485008203 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f4e48095370 con 0x7f4e4c02d5f0 2024-09-17T05:13:30.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.818+0000 7f4e52d9d640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4e40073230 0x7f4e40075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:30.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.818+0000 7f4e52d9d640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4e40073230 0x7f4e40075740 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4e44009b90 tx=0x7f4e440091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:30.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:30.822+0000 7f4e3b7fe640 1 -- 172.21.15.150:0/3485008203 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4e4c0cf020 con 0x7f4e4c02d5f0 2024-09-17T05:13:31.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.546+0000 7f4e5885e640 1 -- 172.21.15.150:0/3485008203 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e4c02f840 con 0x7f4e4c02d5f0 2024-09-17T05:13:31.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.546+0000 7f4e3b7fe640 1 -- 172.21.15.150:0/3485008203 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6540 (secure 0 0 0) 0x7f4e4805f680 con 0x7f4e4c02d5f0 2024-09-17T05:13:31.550 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:13:31.550 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":166,"btime":"2024-09-17T05:12:27:201243+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:12:25.196435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7530":{"gid":7530,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7530,"qdb_cluster":[7530]},"id":2}]} 2024-09-17T05:13:31.550 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 166 2024-09-17T05:13:31.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.550+0000 7f4e5885e640 1 -- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4e40073230 msgr2=0x7f4e40075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:31.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.550+0000 7f4e5885e640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4e40073230 0x7f4e40075740 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4e44009b90 tx=0x7f4e440091e0 comp rx=0 tx=0).stop 2024-09-17T05:13:31.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.550+0000 7f4e5885e640 1 -- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e4c02d5f0 msgr2=0x7f4e4c0d88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:31.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.550+0000 7f4e5885e640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e4c02d5f0 0x7f4e4c0d88e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800b470 tx=0x7f4e4800b870 comp rx=0 tx=0).stop 2024-09-17T05:13:31.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.550+0000 7f4e5885e640 1 -- 172.21.15.150:0/3485008203 shutdown_connections 2024-09-17T05:13:31.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.550+0000 7f4e5885e640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4e40073230 0x7f4e40075740 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:31.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.550+0000 7f4e5885e640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4e4c030cc0 0x7f4e4c0cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:31.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.550+0000 7f4e5885e640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4e4c02d5f0 0x7f4e4c0d88e0 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4e4800b470 tx=0x7f4e4800b870 comp rx=0 tx=0).stop 2024-09-17T05:13:31.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.550+0000 7f4e5885e640 1 --2- 172.21.15.150:0/3485008203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e4c00b8f0 0x7f4e4c0ceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:31.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.550+0000 7f4e5885e640 1 -- 172.21.15.150:0/3485008203 >> 172.21.15.150:0/3485008203 conn(0x7f4e4c048630 msgr2=0x7f4e4c030040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:31.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.550+0000 7f4e5885e640 1 -- 172.21.15.150:0/3485008203 shutdown_connections 2024-09-17T05:13:31.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:31.550+0000 7f4e5885e640 1 -- 172.21.15.150:0/3485008203 wait complete. 2024-09-17T05:13:31.612 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:13:33.614 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:13:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.922+0000 7f0e72425640 1 -- 172.21.15.150:0/3038965377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e6c157c00 msgr2=0x7f0e6c158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.922+0000 7f0e72425640 1 --2- 172.21.15.150:0/3038965377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e6c157c00 0x7f0e6c158010 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0e60009840 tx=0x7f0e6001c860 comp rx=0 tx=0).stop 2024-09-17T05:13:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.926+0000 7f0e72425640 1 -- 172.21.15.150:0/3038965377 shutdown_connections 2024-09-17T05:13:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.926+0000 7f0e72425640 1 --2- 172.21.15.150:0/3038965377 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e6c157c00 0x7f0e6c158010 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.926+0000 7f0e72425640 1 --2- 172.21.15.150:0/3038965377 >> v2:172.21.15.176:3300/0 conn(0x7f0e6c071670 0x7f0e6c0c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.926+0000 7f0e72425640 1 -- 172.21.15.150:0/3038965377 >> 172.21.15.150:0/3038965377 conn(0x7f0e6c0ca460 msgr2=0x7f0e6c0c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.926+0000 7f0e72425640 1 -- 172.21.15.150:0/3038965377 shutdown_connections 2024-09-17T05:13:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.926+0000 7f0e72425640 1 -- 172.21.15.150:0/3038965377 wait complete. 2024-09-17T05:13:33.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.926+0000 7f0e72425640 1 Processor -- start 2024-09-17T05:13:33.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.926+0000 7f0e72425640 1 -- start start 2024-09-17T05:13:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e72425640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e6c071670 0x7f0e6c151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e72425640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e6c14e490 0x7f0e6c14f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e72425640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e6c157c00 0x7f0e6c1f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e72425640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c14ea20 con 0x7f0e6c071670 2024-09-17T05:13:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e72425640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c14e8a0 con 0x7f0e6c157c00 2024-09-17T05:13:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e72425640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0e6c072180 con 0x7f0e6c14e490 2024-09-17T05:13:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e6bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e6c071670 0x7f0e6c151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e6b7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e6c14e490 0x7f0e6c14f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e709c2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e6c157c00 0x7f0e6c1f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e6b7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e6c14e490 0x7f0e6c14f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:39270/0 (socket says 172.21.15.150:39270) 2024-09-17T05:13:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e6b7fe640 1 -- 172.21.15.150:0/3207386641 learned_addr learned my addr 172.21.15.150:0/3207386641 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:13:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e709c2640 1 -- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e6c14e490 msgr2=0x7f0e6c14f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e709c2640 1 --2- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e6c14e490 0x7f0e6c14f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e709c2640 1 -- 172.21.15.150:0/3207386641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e6c071670 msgr2=0x7f0e6c151c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e709c2640 1 --2- 172.21.15.150:0/3207386641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e6c071670 0x7f0e6c151c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e709c2640 1 -- 172.21.15.150:0/3207386641 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6c147d30 con 0x7f0e6c157c00 2024-09-17T05:13:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e709c2640 1 --2- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e6c157c00 0x7f0e6c1f32d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0e600ac000 tx=0x7f0e600a5530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e68ff9640 1 -- 172.21.15.150:0/3207386641 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e600bb070 con 0x7f0e6c157c00 2024-09-17T05:13:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e68ff9640 1 -- 172.21.15.150:0/3207386641 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e60004470 con 0x7f0e6c157c00 2024-09-17T05:13:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e68ff9640 1 -- 172.21.15.150:0/3207386641 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e600b2040 con 0x7f0e6c157c00 2024-09-17T05:13:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e72425640 1 -- 172.21.15.150:0/3207386641 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e6c155100 con 0x7f0e6c157c00 2024-09-17T05:13:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e72425640 1 -- 172.21.15.150:0/3207386641 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e6c147fe0 con 0x7f0e6c157c00 2024-09-17T05:13:33.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.930+0000 7f0e72425640 1 -- 172.21.15.150:0/3207386641 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e6c14ee90 con 0x7f0e6c157c00 2024-09-17T05:13:33.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.934+0000 7f0e68ff9640 1 -- 172.21.15.150:0/3207386641 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0e600be070 con 0x7f0e6c157c00 2024-09-17T05:13:33.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.934+0000 7f0e68ff9640 1 --2- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0e58073200 0x7f0e58075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:33.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.934+0000 7f0e68ff9640 1 -- 172.21.15.150:0/3207386641 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f0e6012e650 con 0x7f0e6c157c00 2024-09-17T05:13:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.934+0000 7f0e6bfff640 1 --2- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0e58073200 0x7f0e58075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.934+0000 7f0e6bfff640 1 --2- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0e58073200 0x7f0e58075710 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009e50 tx=0x7f0e640095e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:33.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:33.938+0000 7f0e68ff9640 1 -- 172.21.15.150:0/3207386641 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0e60016450 con 0x7f0e6c157c00 2024-09-17T05:13:34.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.510+0000 7f0e72425640 1 -- 172.21.15.150:0/3207386641 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e6c0c5680 con 0x7f0e6c157c00 2024-09-17T05:13:34.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.510+0000 7f0e68ff9640 1 -- 172.21.15.150:0/3207386641 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6540 (secure 0 0 0) 0x7f0e600f8320 con 0x7f0e6c157c00 2024-09-17T05:13:34.516 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:13:34.516 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":166,"btime":"2024-09-17T05:12:27:201243+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:12:25.196435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7530":{"gid":7530,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7530,"qdb_cluster":[7530]},"id":2}]} 2024-09-17T05:13:34.516 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 166 2024-09-17T05:13:34.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.514+0000 7f0e527fc640 1 -- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0e58073200 msgr2=0x7f0e58075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:34.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.514+0000 7f0e527fc640 1 --2- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0e58073200 0x7f0e58075710 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009e50 tx=0x7f0e640095e0 comp rx=0 tx=0).stop 2024-09-17T05:13:34.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.514+0000 7f0e527fc640 1 -- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e6c157c00 msgr2=0x7f0e6c1f32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:34.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.514+0000 7f0e527fc640 1 --2- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e6c157c00 0x7f0e6c1f32d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0e600ac000 tx=0x7f0e600a5530 comp rx=0 tx=0).stop 2024-09-17T05:13:34.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.514+0000 7f0e527fc640 1 -- 172.21.15.150:0/3207386641 shutdown_connections 2024-09-17T05:13:34.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.514+0000 7f0e527fc640 1 --2- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0e58073200 0x7f0e58075710 secure :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009e50 tx=0x7f0e640095e0 comp rx=0 tx=0).stop 2024-09-17T05:13:34.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.514+0000 7f0e527fc640 1 --2- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0e6c157c00 0x7f0e6c1f32d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:34.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.514+0000 7f0e527fc640 1 --2- 172.21.15.150:0/3207386641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0e6c14e490 0x7f0e6c14f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:34.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.514+0000 7f0e527fc640 1 --2- 172.21.15.150:0/3207386641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e6c071670 0x7f0e6c151c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:34.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.514+0000 7f0e527fc640 1 -- 172.21.15.150:0/3207386641 >> 172.21.15.150:0/3207386641 conn(0x7f0e6c0ca460 msgr2=0x7f0e6c14d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:34.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.514+0000 7f0e527fc640 1 -- 172.21.15.150:0/3207386641 shutdown_connections 2024-09-17T05:13:34.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:34.522+0000 7f0e527fc640 1 -- 172.21.15.150:0/3207386641 wait complete. 2024-09-17T05:13:34.558 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:13:34.750 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:13:34.752 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:13:36.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:13:36.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.846+0000 7fd9259d8640 1 -- 172.21.15.150:0/752672060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9200738d0 msgr2=0x7fd9200c32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:36.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.846+0000 7fd9259d8640 1 --2- 172.21.15.150:0/752672060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9200738d0 0x7fd9200c32c0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fd918009bc0 tx=0x7fd91801caa0 comp rx=0 tx=0).stop 2024-09-17T05:13:36.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.846+0000 7fd9259d8640 1 -- 172.21.15.150:0/752672060 shutdown_connections 2024-09-17T05:13:36.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.846+0000 7fd9259d8640 1 --2- 172.21.15.150:0/752672060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9200738d0 0x7fd9200c32c0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:36.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.846+0000 7fd9259d8640 1 -- 172.21.15.150:0/752672060 >> 172.21.15.150:0/752672060 conn(0x7fd9200c8250 msgr2=0x7fd9200c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:36.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.846+0000 7fd9259d8640 1 -- 172.21.15.150:0/752672060 shutdown_connections 2024-09-17T05:13:36.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.850+0000 7fd9259d8640 1 -- 172.21.15.150:0/752672060 wait complete. 2024-09-17T05:13:36.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd9259d8640 1 Processor -- start 2024-09-17T05:13:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd9259d8640 1 -- start start 2024-09-17T05:13:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd9259d8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9200738d0 0x7fd92014cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd9259d8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9200c2c50 0x7fd920156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd9259d8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd920153a70 0x7fd92014c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd9259d8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd9201577d0 con 0x7fd9200c2c50 2024-09-17T05:13:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd9259d8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd9200c3a10 con 0x7fd9200738d0 2024-09-17T05:13:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd9259d8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd920157950 con 0x7fd920153a70 2024-09-17T05:13:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd917fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9200c2c50 0x7fd920156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd917fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9200c2c50 0x7fd920156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52640/0 (socket says 172.21.15.150:52640) 2024-09-17T05:13:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd917fff640 1 -- 172.21.15.150:0/1246893853 learned_addr learned my addr 172.21.15.150:0/1246893853 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:13:36.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd91effd640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9200738d0 0x7fd92014cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:36.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd91f7fe640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd920153a70 0x7fd92014c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:36.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd917fff640 1 -- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd920153a70 msgr2=0x7fd92014c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:36.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd917fff640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd920153a70 0x7fd92014c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:36.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd917fff640 1 -- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9200738d0 msgr2=0x7fd92014cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:36.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd917fff640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9200738d0 0x7fd92014cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:36.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd917fff640 1 -- 172.21.15.150:0/1246893853 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd920155440 con 0x7fd9200c2c50 2024-09-17T05:13:36.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd917fff640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9200c2c50 0x7fd920156ef0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fd90800b440 tx=0x7fd90800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:36.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd91cff9640 1 -- 172.21.15.150:0/1246893853 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd908015020 con 0x7fd9200c2c50 2024-09-17T05:13:36.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd91cff9640 1 -- 172.21.15.150:0/1246893853 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9080119a0 con 0x7fd9200c2c50 2024-09-17T05:13:36.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd9259d8640 1 -- 172.21.15.150:0/1246893853 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd92014c0e0 con 0x7fd9200c2c50 2024-09-17T05:13:36.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd91cff9640 1 -- 172.21.15.150:0/1246893853 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd908011df0 con 0x7fd9200c2c50 2024-09-17T05:13:36.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.854+0000 7fd9259d8640 1 -- 172.21.15.150:0/1246893853 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd920156430 con 0x7fd9200c2c50 2024-09-17T05:13:36.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.862+0000 7fd91cff9640 1 -- 172.21.15.150:0/1246893853 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd908018020 con 0x7fd9200c2c50 2024-09-17T05:13:36.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.862+0000 7fd91cff9640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd900073230 0x7fd900075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:36.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.862+0000 7fd91effd640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd900073230 0x7fd900075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:36.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.862+0000 7fd91cff9640 1 -- 172.21.15.150:0/1246893853 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7fd908095150 con 0x7fd9200c2c50 2024-09-17T05:13:36.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.862+0000 7fd91effd640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd900073230 0x7fd900075740 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd910001870 tx=0x7fd910001380 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:36.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.862+0000 7fd9259d8640 1 -- 172.21.15.150:0/1246893853 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9040052a0 con 0x7fd9200c2c50 2024-09-17T05:13:36.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:36.874+0000 7fd91cff9640 1 -- 172.21.15.150:0/1246893853 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd90801f020 con 0x7fd9200c2c50 2024-09-17T05:13:37.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.462+0000 7fd9259d8640 1 -- 172.21.15.150:0/1246893853 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd904005590 con 0x7fd9200c2c50 2024-09-17T05:13:37.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd91cff9640 1 -- 172.21.15.150:0/1246893853 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6540 (secure 0 0 0) 0x7fd920155440 con 0x7fd9200c2c50 2024-09-17T05:13:37.467 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:13:37.467 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":166,"btime":"2024-09-17T05:12:27:201243+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:12:25.196435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7530":{"gid":7530,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7530,"qdb_cluster":[7530]},"id":2}]} 2024-09-17T05:13:37.467 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 166 2024-09-17T05:13:37.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd9259d8640 1 -- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd900073230 msgr2=0x7fd900075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd9259d8640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd900073230 0x7fd900075740 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd910001870 tx=0x7fd910001380 comp rx=0 tx=0).stop 2024-09-17T05:13:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd9259d8640 1 -- 172.21.15.150:0/1246893853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9200c2c50 msgr2=0x7fd920156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd9259d8640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9200c2c50 0x7fd920156ef0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fd90800b440 tx=0x7fd90800b900 comp rx=0 tx=0).stop 2024-09-17T05:13:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd9259d8640 1 -- 172.21.15.150:0/1246893853 shutdown_connections 2024-09-17T05:13:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd9259d8640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd900073230 0x7fd900075740 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:37.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd9259d8640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd920153a70 0x7fd92014c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:37.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd9259d8640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9200c2c50 0x7fd920156ef0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:37.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd9259d8640 1 --2- 172.21.15.150:0/1246893853 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9200738d0 0x7fd92014cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:37.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd9259d8640 1 -- 172.21.15.150:0/1246893853 >> 172.21.15.150:0/1246893853 conn(0x7fd9200c8250 msgr2=0x7fd9200c4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:37.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd9259d8640 1 -- 172.21.15.150:0/1246893853 shutdown_connections 2024-09-17T05:13:37.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:37.466+0000 7fd9259d8640 1 -- 172.21.15.150:0/1246893853 wait complete. 2024-09-17T05:13:37.510 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:13:39.511 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:13:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.786+0000 7fbf7359e640 1 -- 172.21.15.150:0/2643475501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf6c007d20 msgr2=0x7fbf6c008130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.786+0000 7fbf7359e640 1 --2- 172.21.15.150:0/2643475501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf6c007d20 0x7fbf6c008130 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fbf74066410 tx=0x7fbf74099540 comp rx=0 tx=0).stop 2024-09-17T05:13:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.786+0000 7fbf7359e640 1 -- 172.21.15.150:0/2643475501 shutdown_connections 2024-09-17T05:13:39.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.786+0000 7fbf7359e640 1 --2- 172.21.15.150:0/2643475501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf6c007d20 0x7fbf6c008130 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:39.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.786+0000 7fbf7359e640 1 -- 172.21.15.150:0/2643475501 >> 172.21.15.150:0/2643475501 conn(0x7fbf6c049590 msgr2=0x7fbf6c02e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:39.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.790+0000 7fbf7359e640 1 -- 172.21.15.150:0/2643475501 shutdown_connections 2024-09-17T05:13:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.790+0000 7fbf7359e640 1 -- 172.21.15.150:0/2643475501 wait complete. 2024-09-17T05:13:39.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.794+0000 7fbf7359e640 1 Processor -- start 2024-09-17T05:13:39.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.794+0000 7fbf7359e640 1 -- start start 2024-09-17T05:13:39.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.794+0000 7fbf7359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c0478a0 0x7fbf6c0451a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:39.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.794+0000 7fbf7359e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6c0d79a0 0x7fbf6c02d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:39.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.794+0000 7fbf7359e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf6c041460 0x7fbf6c041870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:39.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.794+0000 7fbf7359e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c0d4a40 con 0x7fbf6c0478a0 2024-09-17T05:13:39.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.794+0000 7fbf7359e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c047560 con 0x7fbf6c041460 2024-09-17T05:13:39.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.794+0000 7fbf7359e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbf6c0d4bc0 con 0x7fbf6c0d79a0 2024-09-17T05:13:39.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.794+0000 7fbf71d9b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6c0d79a0 0x7fbf6c02d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:39.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.794+0000 7fbf71d9b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6c0d79a0 0x7fbf6c02d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:56002/0 (socket says 172.21.15.150:56002) 2024-09-17T05:13:39.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.794+0000 7fbf71d9b640 1 -- 172.21.15.150:0/721819381 learned_addr learned my addr 172.21.15.150:0/721819381 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:13:39.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.794+0000 7fbf72d9d640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf6c041460 0x7fbf6c041870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:39.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf7259c640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c0478a0 0x7fbf6c0451a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:39.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf71d9b640 1 -- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf6c041460 msgr2=0x7fbf6c041870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:39.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf71d9b640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf6c041460 0x7fbf6c041870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:39.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf71d9b640 1 -- 172.21.15.150:0/721819381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c0478a0 msgr2=0x7fbf6c0451a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:39.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf71d9b640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c0478a0 0x7fbf6c0451a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:39.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf71d9b640 1 -- 172.21.15.150:0/721819381 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf6c044000 con 0x7fbf6c0d79a0 2024-09-17T05:13:39.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf7259c640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c0478a0 0x7fbf6c0451a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:13:39.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf71d9b640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6c0d79a0 0x7fbf6c02d3b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbf6800fad0 tx=0x7fbf6800ff90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:39.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf62ffd640 1 -- 172.21.15.150:0/721819381 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6801a070 con 0x7fbf6c0d79a0 2024-09-17T05:13:39.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf7359e640 1 -- 172.21.15.150:0/721819381 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf6c046380 con 0x7fbf6c0d79a0 2024-09-17T05:13:39.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf7359e640 1 -- 172.21.15.150:0/721819381 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf6c044580 con 0x7fbf6c0d79a0 2024-09-17T05:13:39.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf62ffd640 1 -- 172.21.15.150:0/721819381 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf68008570 con 0x7fbf6c0d79a0 2024-09-17T05:13:39.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf62ffd640 1 -- 172.21.15.150:0/721819381 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf6800d9a0 con 0x7fbf6c0d79a0 2024-09-17T05:13:39.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf7359e640 1 -- 172.21.15.150:0/721819381 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf580052a0 con 0x7fbf6c0d79a0 2024-09-17T05:13:39.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf62ffd640 1 -- 172.21.15.150:0/721819381 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbf6801d070 con 0x7fbf6c0d79a0 2024-09-17T05:13:39.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.798+0000 7fbf62ffd640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf54073230 0x7fbf54075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:39.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.802+0000 7fbf62ffd640 1 -- 172.21.15.150:0/721819381 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7fbf680978c0 con 0x7fbf6c0d79a0 2024-09-17T05:13:39.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.802+0000 7fbf7259c640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf54073230 0x7fbf54075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:39.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.802+0000 7fbf7259c640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf54073230 0x7fbf54075740 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400a810 tx=0x7fbf64008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:39.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:39.806+0000 7fbf62ffd640 1 -- 172.21.15.150:0/721819381 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbf6c0d7db0 con 0x7fbf6c0d79a0 2024-09-17T05:13:40.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.150+0000 7fbf7359e640 1 -- 172.21.15.150:0/721819381 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf58005590 con 0x7fbf6c0d79a0 2024-09-17T05:13:40.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.154+0000 7fbf62ffd640 1 -- 172.21.15.150:0/721819381 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6540 (secure 0 0 0) 0x7fbf68061af0 con 0x7fbf6c0d79a0 2024-09-17T05:13:40.156 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:13:40.156 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":166,"btime":"2024-09-17T05:12:27:201243+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:12:25.196435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7530":{"gid":7530,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7530,"qdb_cluster":[7530]},"id":2}]} 2024-09-17T05:13:40.157 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 166 2024-09-17T05:13:40.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.162+0000 7fbf7359e640 1 -- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf54073230 msgr2=0x7fbf54075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.162+0000 7fbf7359e640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf54073230 0x7fbf54075740 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400a810 tx=0x7fbf64008040 comp rx=0 tx=0).stop 2024-09-17T05:13:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.162+0000 7fbf7359e640 1 -- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6c0d79a0 msgr2=0x7fbf6c02d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.162+0000 7fbf7359e640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6c0d79a0 0x7fbf6c02d3b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbf6800fad0 tx=0x7fbf6800ff90 comp rx=0 tx=0).stop 2024-09-17T05:13:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.162+0000 7fbf7359e640 1 -- 172.21.15.150:0/721819381 shutdown_connections 2024-09-17T05:13:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.162+0000 7fbf7359e640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf54073230 0x7fbf54075740 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.162+0000 7fbf7359e640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf6c041460 0x7fbf6c041870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.162+0000 7fbf7359e640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6c0d79a0 0x7fbf6c02d3b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.162+0000 7fbf7359e640 1 --2- 172.21.15.150:0/721819381 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c0478a0 0x7fbf6c0451a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.162+0000 7fbf7359e640 1 -- 172.21.15.150:0/721819381 >> 172.21.15.150:0/721819381 conn(0x7fbf6c049590 msgr2=0x7fbf6c00b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.162+0000 7fbf7359e640 1 -- 172.21.15.150:0/721819381 shutdown_connections 2024-09-17T05:13:40.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:40.162+0000 7fbf7359e640 1 -- 172.21.15.150:0/721819381 wait complete. 2024-09-17T05:13:40.204 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:13:42.205 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:13:42.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.822+0000 7ff56bfff640 1 -- 172.21.15.150:0/3652842584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5640463b0 msgr2=0x7ff5640f5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:42.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.822+0000 7ff56bfff640 1 --2- 172.21.15.150:0/3652842584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5640463b0 0x7ff5640f5240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff55c005fd0 tx=0x7ff55c01d030 comp rx=0 tx=0).stop 2024-09-17T05:13:42.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.822+0000 7ff56bfff640 1 -- 172.21.15.150:0/3652842584 shutdown_connections 2024-09-17T05:13:42.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.822+0000 7ff56bfff640 1 --2- 172.21.15.150:0/3652842584 >> v2:172.21.15.176:3300/0 conn(0x7ff5640fba80 0x7ff56400ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:42.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.822+0000 7ff56bfff640 1 --2- 172.21.15.150:0/3652842584 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5640463b0 0x7ff5640f5240 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:42.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.822+0000 7ff56bfff640 1 -- 172.21.15.150:0/3652842584 >> 172.21.15.150:0/3652842584 conn(0x7ff564048270 msgr2=0x7ff564048680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:42.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.822+0000 7ff56bfff640 1 -- 172.21.15.150:0/3652842584 shutdown_connections 2024-09-17T05:13:42.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.822+0000 7ff56bfff640 1 -- 172.21.15.150:0/3652842584 wait complete. 2024-09-17T05:13:42.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56bfff640 1 Processor -- start 2024-09-17T05:13:42.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56bfff640 1 -- start start 2024-09-17T05:13:42.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff56400a850 0x7ff5641019d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:42.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56bfff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5640463b0 0x7ff5640f9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:42.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56bfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5640fba80 0x7ff564196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:42.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff56400a850 0x7ff5641019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:42.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff56400a850 0x7ff5641019d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44396/0 (socket says 172.21.15.150:44396) 2024-09-17T05:13:42.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff56404a430 con 0x7ff56400a850 2024-09-17T05:13:42.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56affd640 1 -- 172.21.15.150:0/3411570785 learned_addr learned my addr 172.21.15.150:0/3411570785 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:13:42.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff56404a110 con 0x7ff5640fba80 2024-09-17T05:13:42.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff564046970 con 0x7ff5640463b0 2024-09-17T05:13:42.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56a7fc640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5640463b0 0x7ff5640f9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:42.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56b7fe640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5640fba80 0x7ff564196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:42.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56affd640 1 -- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5640463b0 msgr2=0x7ff5640f9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:42.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56affd640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5640463b0 0x7ff5640f9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:42.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56affd640 1 -- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5640fba80 msgr2=0x7ff564196ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:42.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56affd640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5640fba80 0x7ff564196ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:42.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.826+0000 7ff56affd640 1 -- 172.21.15.150:0/3411570785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5640eecb0 con 0x7ff56400a850 2024-09-17T05:13:42.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.842+0000 7ff56affd640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff56400a850 0x7ff5641019d0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff55c01b6e0 tx=0x7ff55c0058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:42.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.846+0000 7ff5537fe640 1 -- 172.21.15.150:0/3411570785 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c0bb020 con 0x7ff56400a850 2024-09-17T05:13:42.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.846+0000 7ff5537fe640 1 -- 172.21.15.150:0/3411570785 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff55c004280 con 0x7ff56400a850 2024-09-17T05:13:42.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.846+0000 7ff5537fe640 1 -- 172.21.15.150:0/3411570785 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c0a5e20 con 0x7ff56400a850 2024-09-17T05:13:42.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.846+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff564197590 con 0x7ff56400a850 2024-09-17T05:13:42.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.846+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff564042430 con 0x7ff56400a850 2024-09-17T05:13:42.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.846+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5640f5240 con 0x7ff56400a850 2024-09-17T05:13:42.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.874+0000 7ff5537fe640 1 -- 172.21.15.150:0/3411570785 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff55c0be020 con 0x7ff56400a850 2024-09-17T05:13:42.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.874+0000 7ff5537fe640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff558073200 0x7ff558075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:42.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.874+0000 7ff5537fe640 1 -- 172.21.15.150:0/3411570785 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7ff55c12fa80 con 0x7ff56400a850 2024-09-17T05:13:42.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.874+0000 7ff56a7fc640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff558073200 0x7ff558075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:42.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.874+0000 7ff56a7fc640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff558073200 0x7ff558075710 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff5640f2580 tx=0x7ff560009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:42.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:42.882+0000 7ff5537fe640 1 -- 172.21.15.150:0/3411570785 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff55c0c5030 con 0x7ff56400a850 2024-09-17T05:13:43.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.490+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5640f87c0 con 0x7ff56400a850 2024-09-17T05:13:43.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.490+0000 7ff5537fe640 1 -- 172.21.15.150:0/3411570785 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+6540 (secure 0 0 0) 0x7ff55c0c5030 con 0x7ff56400a850 2024-09-17T05:13:43.491 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:13:43.491 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":166,"btime":"2024-09-17T05:12:27:201243+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7015,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:12:25.196435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7530":{"gid":7530,"name":"a","rank":0,"incarnation":162,"state":"up:active","state_seq":6,"addr":"172.21.15.150:6833/12982302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":12982302},{"type":"v1","addr":"172.21.15.150:6833","nonce":12982302}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7530,"qdb_cluster":[7530]},"id":2}]} 2024-09-17T05:13:43.491 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 166 2024-09-17T05:13:43.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.490+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff558073200 msgr2=0x7ff558075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:43.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.490+0000 7ff56bfff640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff558073200 0x7ff558075710 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff5640f2580 tx=0x7ff560009040 comp rx=0 tx=0).stop 2024-09-17T05:13:43.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.490+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff56400a850 msgr2=0x7ff5641019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:43.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.490+0000 7ff56bfff640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff56400a850 0x7ff5641019d0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff55c01b6e0 tx=0x7ff55c0058e0 comp rx=0 tx=0).stop 2024-09-17T05:13:43.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.494+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 shutdown_connections 2024-09-17T05:13:43.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.494+0000 7ff56bfff640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff558073200 0x7ff558075710 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:43.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.494+0000 7ff56bfff640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5640fba80 0x7ff564196ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:43.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.494+0000 7ff56bfff640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff5640463b0 0x7ff5640f9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:43.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.494+0000 7ff56bfff640 1 --2- 172.21.15.150:0/3411570785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff56400a850 0x7ff5641019d0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:43.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.494+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 >> 172.21.15.150:0/3411570785 conn(0x7ff564048270 msgr2=0x7ff5640f7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:43.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.494+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 shutdown_connections 2024-09-17T05:13:43.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:43.494+0000 7ff56bfff640 1 -- 172.21.15.150:0/3411570785 wait complete. 2024-09-17T05:13:43.574 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:13:45.575 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:13:46.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.022+0000 7f2bfca19640 1 -- 172.21.15.150:0/1792010215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf80a83b0 msgr2=0x7f2bf80713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:46.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.022+0000 7f2bfca19640 1 --2- 172.21.15.150:0/1792010215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf80a83b0 0x7f2bf80713d0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f2be8009840 tx=0x7f2be801c790 comp rx=0 tx=0).stop 2024-09-17T05:13:46.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.026+0000 7f2bfca19640 1 -- 172.21.15.150:0/1792010215 shutdown_connections 2024-09-17T05:13:46.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.026+0000 7f2bfca19640 1 --2- 172.21.15.150:0/1792010215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf80a83b0 0x7f2bf80713d0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:46.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.026+0000 7f2bfca19640 1 -- 172.21.15.150:0/1792010215 >> 172.21.15.150:0/1792010215 conn(0x7f2bf80aa930 msgr2=0x7f2bf80aad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:46.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.026+0000 7f2bfca19640 1 -- 172.21.15.150:0/1792010215 shutdown_connections 2024-09-17T05:13:46.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.026+0000 7f2bfca19640 1 -- 172.21.15.150:0/1792010215 wait complete. 2024-09-17T05:13:46.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.026+0000 7f2bfca19640 1 Processor -- start 2024-09-17T05:13:46.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.026+0000 7f2bfca19640 1 -- start start 2024-09-17T05:13:46.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bfca19640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2bf80c2e00 0x7f2bf81fce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:46.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bfca19640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2bf8156ae0 0x7f2bf81528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:46.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bfca19640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf81ed4c0 0x7f2bf81ee5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:46.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bfca19640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf81617c0 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bfca19640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf8161640 con 0x7f2bf80c2e00 2024-09-17T05:13:46.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bfca19640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2bf8161940 con 0x7f2bf8156ae0 2024-09-17T05:13:46.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bf759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf81ed4c0 0x7f2bf81ee5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:46.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bf759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf81ed4c0 0x7f2bf81ee5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44418/0 (socket says 172.21.15.150:44418) 2024-09-17T05:13:46.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bf759e640 1 -- 172.21.15.150:0/1857642665 learned_addr learned my addr 172.21.15.150:0/1857642665 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:13:46.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bf6d9d640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2bf80c2e00 0x7f2bf81fce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:46.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bee59c640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2bf8156ae0 0x7f2bf81528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:46.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bf759e640 1 -- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2bf8156ae0 msgr2=0x7f2bf81528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:46.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bf759e640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2bf8156ae0 0x7f2bf81528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:46.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bf759e640 1 -- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2bf80c2e00 msgr2=0x7f2bf81fce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:46.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bf759e640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2bf80c2e00 0x7f2bf81fce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:46.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bf759e640 1 -- 172.21.15.150:0/1857642665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf8152e00 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bee59c640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2bf8156ae0 0x7f2bf81528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:13:46.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bf759e640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf81ed4c0 0x7f2bf81ee5e0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000cb10 tx=0x7f2bf000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:46.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2beffff640 1 -- 172.21.15.150:0/1857642665 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0016020 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2beffff640 1 -- 172.21.15.150:0/1857642665 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2bf000ce50 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bfca19640 1 -- 172.21.15.150:0/1857642665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf81530d0 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2beffff640 1 -- 172.21.15.150:0/1857642665 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf000ae00 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bfca19640 1 -- 172.21.15.150:0/1857642665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bf81ef300 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.030+0000 7f2bfca19640 1 -- 172.21.15.150:0/1857642665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bf8153310 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.038+0000 7f2beffff640 1 -- 172.21.15.150:0/1857642665 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2bf0019020 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.038+0000 7f2beffff640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2bd8073200 0x7f2bd8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:46.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.038+0000 7f2beffff640 1 -- 172.21.15.150:0/1857642665 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f2bf00961d0 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.038+0000 7f2bf6d9d640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2bd8073200 0x7f2bd8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:46.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.042+0000 7f2beffff640 1 -- 172.21.15.150:0/1857642665 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2bf8153310 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.042+0000 7f2bf6d9d640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2bd8073200 0x7f2bd8075710 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2be80ae000 tx=0x7f2be801afd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:46.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.546+0000 7f2bfca19640 1 -- 172.21.15.150:0/1857642665 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2bf80a83b0 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.550+0000 7f2beffff640 1 -- 172.21.15.150:0/1857642665 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+5710 (secure 0 0 0) 0x7f2bf0020020 con 0x7f2bf81ed4c0 2024-09-17T05:13:46.552 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:13:46.552 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":167,"btime":"2024-09-17T05:13:46:203835+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:46.203832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:replay","state_seq":1,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:13:46.552 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 167 2024-09-17T05:13:46.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.554+0000 7f2bfca19640 1 -- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2bd8073200 msgr2=0x7f2bd8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:46.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.554+0000 7f2bfca19640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2bd8073200 0x7f2bd8075710 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f2be80ae000 tx=0x7f2be801afd0 comp rx=0 tx=0).stop 2024-09-17T05:13:46.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.554+0000 7f2bfca19640 1 -- 172.21.15.150:0/1857642665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf81ed4c0 msgr2=0x7f2bf81ee5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:46.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.554+0000 7f2bfca19640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf81ed4c0 0x7f2bf81ee5e0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000cb10 tx=0x7f2bf000cc00 comp rx=0 tx=0).stop 2024-09-17T05:13:46.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.554+0000 7f2bfca19640 1 -- 172.21.15.150:0/1857642665 shutdown_connections 2024-09-17T05:13:46.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.554+0000 7f2bfca19640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2bd8073200 0x7f2bd8075710 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:46.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.554+0000 7f2bfca19640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2bf81ed4c0 0x7f2bf81ee5e0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:46.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.554+0000 7f2bfca19640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2bf8156ae0 0x7f2bf81528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:46.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.554+0000 7f2bfca19640 1 --2- 172.21.15.150:0/1857642665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2bf80c2e00 0x7f2bf81fce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:46.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.554+0000 7f2bfca19640 1 -- 172.21.15.150:0/1857642665 >> 172.21.15.150:0/1857642665 conn(0x7f2bf80aa930 msgr2=0x7f2bf80c7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:46.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.554+0000 7f2bfca19640 1 -- 172.21.15.150:0/1857642665 shutdown_connections 2024-09-17T05:13:46.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:46.554+0000 7f2bfca19640 1 -- 172.21.15.150:0/1857642665 wait complete. 2024-09-17T05:13:46.608 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-17T05:13:46.608 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:13:46.608 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:13:47.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.646+0000 7fd975474640 1 -- 172.21.15.150:0/2790132578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97006fca0 msgr2=0x7fd97006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:47.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.646+0000 7fd975474640 1 --2- 172.21.15.150:0/2790132578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97006fca0 0x7fd97006e090 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd964009900 tx=0x7fd96401c850 comp rx=0 tx=0).stop 2024-09-17T05:13:47.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.650+0000 7fd975474640 1 -- 172.21.15.150:0/2790132578 shutdown_connections 2024-09-17T05:13:47.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.650+0000 7fd975474640 1 --2- 172.21.15.150:0/2790132578 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd97006e5d0 0x7fd97015dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:47.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.650+0000 7fd975474640 1 --2- 172.21.15.150:0/2790132578 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97006fca0 0x7fd97006e090 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:47.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.650+0000 7fd975474640 1 -- 172.21.15.150:0/2790132578 >> 172.21.15.150:0/2790132578 conn(0x7fd9700c5ae0 msgr2=0x7fd970073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:47.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.650+0000 7fd975474640 1 -- 172.21.15.150:0/2790132578 shutdown_connections 2024-09-17T05:13:47.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.650+0000 7fd975474640 1 -- 172.21.15.150:0/2790132578 wait complete. 2024-09-17T05:13:47.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.650+0000 7fd975474640 1 Processor -- start 2024-09-17T05:13:47.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.650+0000 7fd975474640 1 -- start start 2024-09-17T05:13:47.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd975474640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97006e5d0 0x7fd970143020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:47.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd975474640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97006fca0 0x7fd970154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:47.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd975474640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd9700745c0 0x7fd9700acc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:47.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd975474640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd970070610 con 0x7fd97006e5d0 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd975474640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd970070790 con 0x7fd97006fca0 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd975474640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd9700ab140 con 0x7fd9700745c0 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd96effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97006e5d0 0x7fd970143020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd96effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97006e5d0 0x7fd970143020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44434/0 (socket says 172.21.15.150:44434) 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd96effd640 1 -- 172.21.15.150:0/4269356773 learned_addr learned my addr 172.21.15.150:0/4269356773 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd96e7fc640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97006fca0 0x7fd970154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd96e7fc640 1 -- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd9700745c0 msgr2=0x7fd9700acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd96e7fc640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd9700745c0 0x7fd9700acc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd96e7fc640 1 -- 172.21.15.150:0/4269356773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97006e5d0 msgr2=0x7fd970143020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd96e7fc640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97006e5d0 0x7fd970143020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd96e7fc640 1 -- 172.21.15.150:0/4269356773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9701f4000 con 0x7fd97006fca0 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd96e7fc640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97006fca0 0x7fd970154d80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd9640ac000 tx=0x7fd9640a5620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd9577fe640 1 -- 172.21.15.150:0/4269356773 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9640042c0 con 0x7fd97006fca0 2024-09-17T05:13:47.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd975474640 1 -- 172.21.15.150:0/4269356773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd970156040 con 0x7fd97006fca0 2024-09-17T05:13:47.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.654+0000 7fd975474640 1 -- 172.21.15.150:0/4269356773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9701557e0 con 0x7fd97006fca0 2024-09-17T05:13:47.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.658+0000 7fd9577fe640 1 -- 172.21.15.150:0/4269356773 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9640bd070 con 0x7fd97006fca0 2024-09-17T05:13:47.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.658+0000 7fd9577fe640 1 -- 172.21.15.150:0/4269356773 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9640042c0 con 0x7fd97006fca0 2024-09-17T05:13:47.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.658+0000 7fd975474640 1 -- 172.21.15.150:0/4269356773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd97006e090 con 0x7fd97006fca0 2024-09-17T05:13:47.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.658+0000 7fd9577fe640 1 -- 172.21.15.150:0/4269356773 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd9640bf030 con 0x7fd97006fca0 2024-09-17T05:13:47.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.658+0000 7fd9577fe640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd95c073430 0x7fd95c075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:47.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.658+0000 7fd9577fe640 1 -- 172.21.15.150:0/4269356773 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fd964136aa0 con 0x7fd97006fca0 2024-09-17T05:13:47.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.658+0000 7fd96effd640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd95c073430 0x7fd95c075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:47.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.658+0000 7fd96effd640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd95c073430 0x7fd95c075940 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd9600018f0 tx=0x7fd960009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:47.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:47.670+0000 7fd9577fe640 1 -- 172.21.15.150:0/4269356773 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd9640c6030 con 0x7fd97006fca0 2024-09-17T05:13:48.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.218+0000 7fd975474640 1 -- 172.21.15.150:0/4269356773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd970150400 con 0x7fd97006fca0 2024-09-17T05:13:48.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.222+0000 7fd9577fe640 1 -- 172.21.15.150:0/4269356773 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) ==== 95+0+5714 (secure 0 0 0) 0x7fd9640164d0 con 0x7fd97006fca0 2024-09-17T05:13:48.225 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:13:48.225 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":168,"btime":"2024-09-17T05:13:47:780794+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:46.829206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:13:48.225 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 168 2024-09-17T05:13:48.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.226+0000 7fd975474640 1 -- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd95c073430 msgr2=0x7fd95c075940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:48.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.226+0000 7fd975474640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd95c073430 0x7fd95c075940 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd9600018f0 tx=0x7fd960009040 comp rx=0 tx=0).stop 2024-09-17T05:13:48.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.226+0000 7fd975474640 1 -- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97006fca0 msgr2=0x7fd970154d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:48.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.226+0000 7fd975474640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97006fca0 0x7fd970154d80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd9640ac000 tx=0x7fd9640a5620 comp rx=0 tx=0).stop 2024-09-17T05:13:48.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.230+0000 7fd975474640 1 -- 172.21.15.150:0/4269356773 shutdown_connections 2024-09-17T05:13:48.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.230+0000 7fd975474640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd95c073430 0x7fd95c075940 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:48.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.230+0000 7fd975474640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd97006fca0 0x7fd970154d80 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:48.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.230+0000 7fd975474640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd9700745c0 0x7fd9700acc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:48.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.230+0000 7fd975474640 1 --2- 172.21.15.150:0/4269356773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd97006e5d0 0x7fd970143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:48.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.230+0000 7fd975474640 1 -- 172.21.15.150:0/4269356773 >> 172.21.15.150:0/4269356773 conn(0x7fd9700c5ae0 msgr2=0x7fd970150d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:48.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.230+0000 7fd975474640 1 -- 172.21.15.150:0/4269356773 shutdown_connections 2024-09-17T05:13:48.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:48.230+0000 7fd975474640 1 -- 172.21.15.150:0/4269356773 wait complete. 2024-09-17T05:13:48.359 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:13:47:780794+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 168, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 160, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7407": { "addr": "172.21.15.150:6835/3179072684", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 3179072684, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 3179072684, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7407, "incarnation": 157, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:12:04.469086+0000", "qdb_cluster": [ 7407 ], "qdb_leader": 7407, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7407 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 168, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7015": { "addr": "172.21.15.176:6837/2338457617", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 2338457617, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 2338457617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7015, "incarnation": 167, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:13:46.829206+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7015 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:13:50.360 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:13:50.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.654+0000 7f89f3d71640 1 -- 172.21.15.150:0/2423287416 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89d8003a10 msgr2=0x7f89d8003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:50.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.654+0000 7f89f3d71640 1 --2- 172.21.15.150:0/2423287416 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89d8003a10 0x7f89d8003e60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f89e800c310 tx=0x7f89e801d050 comp rx=0 tx=0).stop 2024-09-17T05:13:50.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.654+0000 7f89f3d71640 1 -- 172.21.15.150:0/2423287416 shutdown_connections 2024-09-17T05:13:50.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.654+0000 7f89f3d71640 1 --2- 172.21.15.150:0/2423287416 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89d8003a10 0x7f89d8003e60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:50.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.654+0000 7f89f3d71640 1 -- 172.21.15.150:0/2423287416 >> 172.21.15.150:0/2423287416 conn(0x7f89ec0ca460 msgr2=0x7f89ec0c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:50.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.654+0000 7f89f3d71640 1 -- 172.21.15.150:0/2423287416 shutdown_connections 2024-09-17T05:13:50.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.654+0000 7f89f3d71640 1 -- 172.21.15.150:0/2423287416 wait complete. 2024-09-17T05:13:50.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.658+0000 7f89f3d71640 1 Processor -- start 2024-09-17T05:13:50.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.658+0000 7f89f3d71640 1 -- start start 2024-09-17T05:13:50.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.658+0000 7f89f3d71640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89d8003a10 0x7f89ec1ec040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:50.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.658+0000 7f89f3d71640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89ec1450c0 0x7f89ec155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:50.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.658+0000 7f89f3d71640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89ec156190 0x7f89ec143950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:50.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.658+0000 7f89f3d71640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f89ec0c62d0 con 0x7f89ec1450c0 2024-09-17T05:13:50.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.658+0000 7f89f3d71640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f89ec0c5220 con 0x7f89ec156190 2024-09-17T05:13:50.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.658+0000 7f89f3d71640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f89ec0c65c0 con 0x7f89d8003a10 2024-09-17T05:13:50.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.662+0000 7f89f230e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89ec156190 0x7f89ec143950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.662+0000 7f89f230e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89ec156190 0x7f89ec143950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:54726/0 (socket says 172.21.15.150:54726) 2024-09-17T05:13:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.662+0000 7f89f230e640 1 -- 172.21.15.150:0/1011536861 learned_addr learned my addr 172.21.15.150:0/1011536861 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:13:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.662+0000 7f89f130c640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89ec1450c0 0x7f89ec155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:50.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.662+0000 7f89f130c640 1 -- 172.21.15.150:0/1011536861 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89d8003a10 msgr2=0x7f89ec1ec040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:50.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.662+0000 7f89f130c640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89d8003a10 0x7f89ec1ec040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:50.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.662+0000 7f89f130c640 1 -- 172.21.15.150:0/1011536861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89ec156190 msgr2=0x7f89ec143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:50.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.662+0000 7f89f130c640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89ec156190 0x7f89ec143950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:50.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.662+0000 7f89f130c640 1 -- 172.21.15.150:0/1011536861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89ec0ae050 con 0x7f89ec1450c0 2024-09-17T05:13:50.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.666+0000 7f89f130c640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89ec1450c0 0x7f89ec155c50 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f89ec06ab10 tx=0x7f89dc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.666+0000 7f89e27fc640 1 -- 172.21.15.150:0/1011536861 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc015020 con 0x7f89ec1450c0 2024-09-17T05:13:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.666+0000 7f89e27fc640 1 -- 172.21.15.150:0/1011536861 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89dc011920 con 0x7f89ec1450c0 2024-09-17T05:13:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.666+0000 7f89e27fc640 1 -- 172.21.15.150:0/1011536861 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc011db0 con 0x7f89ec1450c0 2024-09-17T05:13:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.666+0000 7f89f3d71640 1 -- 172.21.15.150:0/1011536861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89ec142f50 con 0x7f89ec1450c0 2024-09-17T05:13:50.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.666+0000 7f89f3d71640 1 -- 172.21.15.150:0/1011536861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89ec142b00 con 0x7f89ec1450c0 2024-09-17T05:13:50.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.670+0000 7f89e27fc640 1 -- 172.21.15.150:0/1011536861 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f89dc018020 con 0x7f89ec1450c0 2024-09-17T05:13:50.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.670+0000 7f89e27fc640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89d0073200 0x7f89d0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:13:50.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.670+0000 7f89f1b0d640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89d0073200 0x7f89d0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:13:50.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.670+0000 7f89e27fc640 1 -- 172.21.15.150:0/1011536861 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7f89dc0917a0 con 0x7f89ec1450c0 2024-09-17T05:13:50.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.670+0000 7f89cffff640 1 -- 172.21.15.150:0/1011536861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c80052a0 con 0x7f89ec1450c0 2024-09-17T05:13:50.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.674+0000 7f89f1b0d640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89d0073200 0x7f89d0075710 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f89e80ac000 tx=0x7f89e8000bd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:13:50.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:50.690+0000 7f89e27fc640 1 -- 172.21.15.150:0/1011536861 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f89dc05e2d0 con 0x7f89ec1450c0 2024-09-17T05:13:51.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.298+0000 7f89cffff640 1 -- 172.21.15.150:0/1011536861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89c8005860 con 0x7f89ec1450c0 2024-09-17T05:13:51.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.298+0000 7f89e27fc640 1 -- 172.21.15.150:0/1011536861 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) ==== 95+0+5718 (secure 0 0 0) 0x7f89dc05dbd0 con 0x7f89ec1450c0 2024-09-17T05:13:51.296 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:13:51.296 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":170,"btime":"2024-09-17T05:13:49:801015+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:13:51.297 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 170 2024-09-17T05:13:51.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.302+0000 7f89cffff640 1 -- 172.21.15.150:0/1011536861 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89d0073200 msgr2=0x7f89d0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:51.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.302+0000 7f89cffff640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89d0073200 0x7f89d0075710 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f89e80ac000 tx=0x7f89e8000bd0 comp rx=0 tx=0).stop 2024-09-17T05:13:51.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.302+0000 7f89cffff640 1 -- 172.21.15.150:0/1011536861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89ec1450c0 msgr2=0x7f89ec155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:13:51.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.302+0000 7f89cffff640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89ec1450c0 0x7f89ec155c50 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f89ec06ab10 tx=0x7f89dc00b870 comp rx=0 tx=0).stop 2024-09-17T05:13:51.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.306+0000 7f89cffff640 1 -- 172.21.15.150:0/1011536861 shutdown_connections 2024-09-17T05:13:51.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.306+0000 7f89cffff640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f89d0073200 0x7f89d0075710 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:51.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.306+0000 7f89cffff640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f89ec156190 0x7f89ec143950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:51.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.306+0000 7f89cffff640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89ec1450c0 0x7f89ec155c50 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:51.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.306+0000 7f89cffff640 1 --2- 172.21.15.150:0/1011536861 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f89d8003a10 0x7f89ec1ec040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:13:51.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.306+0000 7f89cffff640 1 -- 172.21.15.150:0/1011536861 >> 172.21.15.150:0/1011536861 conn(0x7f89ec0ca460 msgr2=0x7f89ec14d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:13:51.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.306+0000 7f89cffff640 1 -- 172.21.15.150:0/1011536861 shutdown_connections 2024-09-17T05:13:51.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:13:51.306+0000 7f89cffff640 1 -- 172.21.15.150:0/1011536861 wait complete. 2024-09-17T05:13:51.348 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=7530 2024-09-17T05:13:51.348 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before reviving mds.a 2024-09-17T05:14:00.349 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-09-17T05:14:00.349 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T05:14:00.350 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T05:14:00.352 INFO:tasks.ceph.mds.a:Started 2024-09-17T05:14:00.352 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:00.654 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T05:14:00.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.706+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2073588630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e60048980 msgr2=0x7f9e600330d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:00.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.706+0000 7f9e65d9b640 1 -- 172.21.15.150:0/2073588630 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58001830 con 0x7f9e60048980 2024-09-17T05:14:00.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.710+0000 7f9e6d1f6640 1 --2- 172.21.15.150:0/2073588630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e60048980 0x7f9e600330d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c009870 tx=0x7f9e5c01c760 comp rx=0 tx=0).stop 2024-09-17T05:14:00.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.718+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2073588630 shutdown_connections 2024-09-17T05:14:00.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.718+0000 7f9e6d1f6640 1 --2- 172.21.15.150:0/2073588630 >> v2:172.21.15.150:3300/0 conn(0x7f9e60049d50 0x7f9e6004a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:00.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.718+0000 7f9e6d1f6640 1 --2- 172.21.15.150:0/2073588630 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e60048980 0x7f9e600330d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:00.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.718+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2073588630 >> 172.21.15.150:0/2073588630 conn(0x7f9e6000a740 msgr2=0x7f9e6004d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:00.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.718+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2073588630 shutdown_connections 2024-09-17T05:14:00.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.718+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2073588630 wait complete. 2024-09-17T05:14:00.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.718+0000 7f9e6d1f6640 1 Processor -- start 2024-09-17T05:14:00.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.718+0000 7f9e6d1f6640 1 -- start start 2024-09-17T05:14:00.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6d1f6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e60048980 0x7f9e60170710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:00.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6d1f6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e60049d50 0x7f9e60007960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:00.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6d1f6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600d1920 0x7f9e600d1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:00.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6d1f6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9e600e50e0 con 0x7f9e600d1920 2024-09-17T05:14:00.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6d1f6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9e60048730 con 0x7f9e60049d50 2024-09-17T05:14:00.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6d1f6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9e600e5260 con 0x7f9e60048980 2024-09-17T05:14:00.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600d1920 0x7f9e600d1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:00.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600d1920 0x7f9e600d1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56026/0 (socket says 172.21.15.150:56026) 2024-09-17T05:14:00.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6759e640 1 -- 172.21.15.150:0/2872207006 learned_addr learned my addr 172.21.15.150:0/2872207006 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:00.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6659c640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e60049d50 0x7f9e60007960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:00.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e66d9d640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e60048980 0x7f9e60170710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:00.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6659c640 1 -- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e60048980 msgr2=0x7f9e60170710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:00.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6659c640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e60048980 0x7f9e60170710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:00.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.722+0000 7f9e6659c640 1 -- 172.21.15.150:0/2872207006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600d1920 msgr2=0x7f9e600d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:00.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.726+0000 7f9e6659c640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600d1920 0x7f9e600d1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:00.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.726+0000 7f9e6659c640 1 -- 172.21.15.150:0/2872207006 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e601730c0 con 0x7f9e60049d50 2024-09-17T05:14:00.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.726+0000 7f9e66d9d640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e60048980 0x7f9e60170710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:14:00.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.726+0000 7f9e6759e640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600d1920 0x7f9e600d1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:14:00.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.726+0000 7f9e6659c640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e60049d50 0x7f9e60007960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0ac000 tx=0x7f9e5c005ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:00.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.726+0000 7f9e4f7fe640 1 -- 172.21.15.150:0/2872207006 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c004750 con 0x7f9e60049d50 2024-09-17T05:14:00.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.726+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2872207006 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e601768c0 con 0x7f9e60049d50 2024-09-17T05:14:00.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.726+0000 7f9e4f7fe640 1 -- 172.21.15.150:0/2872207006 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e5c0048f0 con 0x7f9e60049d50 2024-09-17T05:14:00.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.726+0000 7f9e4f7fe640 1 -- 172.21.15.150:0/2872207006 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c004d00 con 0x7f9e60049d50 2024-09-17T05:14:00.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.726+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2872207006 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e60176e80 con 0x7f9e60049d50 2024-09-17T05:14:00.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.726+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2872207006 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e60173330 con 0x7f9e60049d50 2024-09-17T05:14:00.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.730+0000 7f9e4f7fe640 1 -- 172.21.15.150:0/2872207006 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9e5c0a8500 con 0x7f9e60049d50 2024-09-17T05:14:00.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.730+0000 7f9e4f7fe640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9e54073200 0x7f9e54075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:00.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.730+0000 7f9e4f7fe640 1 -- 172.21.15.150:0/2872207006 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7f9e5c12f100 con 0x7f9e60049d50 2024-09-17T05:14:00.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.730+0000 7f9e66d9d640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9e54073200 0x7f9e54075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:00.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.730+0000 7f9e66d9d640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9e54073200 0x7f9e54075710 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9e58001540 tx=0x7f9e58009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:00.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:00.734+0000 7f9e4f7fe640 1 -- 172.21.15.150:0/2872207006 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9e5c0c1030 con 0x7f9e60049d50 2024-09-17T05:14:01.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.254+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2872207006 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e60174050 con 0x7f9e60049d50 2024-09-17T05:14:01.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.254+0000 7f9e4f7fe640 1 -- 172.21.15.150:0/2872207006 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6547 (secure 0 0 0) 0x7f9e5c0ac030 con 0x7f9e60049d50 2024-09-17T05:14:01.253 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:01.253 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":171,"btime":"2024-09-17T05:14:00:851032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7078,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:01.254 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 171 2024-09-17T05:14:01.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.258+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9e54073200 msgr2=0x7f9e54075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:01.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.262+0000 7f9e6d1f6640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9e54073200 0x7f9e54075710 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9e58001540 tx=0x7f9e58009040 comp rx=0 tx=0).stop 2024-09-17T05:14:01.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.262+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e60049d50 msgr2=0x7f9e60007960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:01.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.262+0000 7f9e6d1f6640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e60049d50 0x7f9e60007960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0ac000 tx=0x7f9e5c005ca0 comp rx=0 tx=0).stop 2024-09-17T05:14:01.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.262+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2872207006 shutdown_connections 2024-09-17T05:14:01.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.262+0000 7f9e6d1f6640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9e54073200 0x7f9e54075710 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:01.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.262+0000 7f9e6d1f6640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9e60049d50 0x7f9e60007960 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:01.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.262+0000 7f9e6d1f6640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9e600d1920 0x7f9e600d1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:01.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.262+0000 7f9e6d1f6640 1 --2- 172.21.15.150:0/2872207006 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9e60048980 0x7f9e60170710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:01.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.262+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2872207006 >> 172.21.15.150:0/2872207006 conn(0x7f9e6000a740 msgr2=0x7f9e6000cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:01.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.262+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2872207006 shutdown_connections 2024-09-17T05:14:01.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:01.262+0000 7f9e6d1f6640 1 -- 172.21.15.150:0/2872207006 wait complete. 2024-09-17T05:14:01.301 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-09-17T05:14:01.301 INFO:tasks.mds_thrash.fs.[b]:waiting for 52 secs before thrashing 2024-09-17T05:14:04.997 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:14:05.003 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:14:07.405 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:07.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.742+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3706920319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c4158520 msgr2=0x7f51c40c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:07.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.742+0000 7f51cb2d8640 1 --2- 172.21.15.150:0/3706920319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c4158520 0x7f51c40c7b40 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009840 tx=0x7f51b401cc40 comp rx=0 tx=0).stop 2024-09-17T05:14:07.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.750+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3706920319 shutdown_connections 2024-09-17T05:14:07.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.750+0000 7f51cb2d8640 1 --2- 172.21.15.150:0/3706920319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c4158520 0x7f51c40c7b40 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:07.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.750+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3706920319 >> 172.21.15.150:0/3706920319 conn(0x7f51c40ac0f0 msgr2=0x7f51c40c9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:07.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.750+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3706920319 shutdown_connections 2024-09-17T05:14:07.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.750+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3706920319 wait complete. 2024-09-17T05:14:07.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.754+0000 7f51cb2d8640 1 Processor -- start 2024-09-17T05:14:07.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.758+0000 7f51cb2d8640 1 -- start start 2024-09-17T05:14:07.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.758+0000 7f51cb2d8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51c40c5810 0x7f51c41f4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:07.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.758+0000 7f51cb2d8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c4154fc0 0x7f51c41535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:07.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.758+0000 7f51cb2d8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51c4158520 0x7f51c41f7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:07.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.758+0000 7f51cb2d8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f51c4162eb0 con 0x7f51c4154fc0 2024-09-17T05:14:07.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.758+0000 7f51cb2d8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f51c40c8160 con 0x7f51c4158520 2024-09-17T05:14:07.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.758+0000 7f51cb2d8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f51c4163030 con 0x7f51c40c5810 2024-09-17T05:14:07.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.758+0000 7f51c9074640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51c40c5810 0x7f51c41f4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:07.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.758+0000 7f51c9074640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51c40c5810 0x7f51c41f4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:39216/0 (socket says 172.21.15.150:39216) 2024-09-17T05:14:07.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.758+0000 7f51c9074640 1 -- 172.21.15.150:0/3003318518 learned_addr learned my addr 172.21.15.150:0/3003318518 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:07.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.762+0000 7f51c9074640 1 -- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51c4158520 msgr2=0x7f51c41f7ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:14:07.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.762+0000 7f51c9074640 1 --2- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51c4158520 0x7f51c41f7ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:07.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.762+0000 7f51c9074640 1 -- 172.21.15.150:0/3003318518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c4154fc0 msgr2=0x7f51c41535d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:14:07.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.762+0000 7f51c9074640 1 --2- 172.21.15.150:0/3003318518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c4154fc0 0x7f51c41535d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:07.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.762+0000 7f51c9074640 1 -- 172.21.15.150:0/3003318518 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c406f3b0 con 0x7f51c40c5810 2024-09-17T05:14:07.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.762+0000 7f51c9074640 1 --2- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51c40c5810 0x7f51c41f4940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f51c000c4d0 tx=0x7f51c000c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:07.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.762+0000 7f51b9ffb640 1 -- 172.21.15.150:0/3003318518 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0005480 con 0x7f51c40c5810 2024-09-17T05:14:07.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.762+0000 7f51b9ffb640 1 -- 172.21.15.150:0/3003318518 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51c0003600 con 0x7f51c40c5810 2024-09-17T05:14:07.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.762+0000 7f51b9ffb640 1 -- 172.21.15.150:0/3003318518 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0003a30 con 0x7f51c40c5810 2024-09-17T05:14:07.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.766+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3003318518 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51c406f690 con 0x7f51c40c5810 2024-09-17T05:14:07.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.766+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3003318518 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f51c41f8cc0 con 0x7f51c40c5810 2024-09-17T05:14:07.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.766+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3003318518 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51a40052a0 con 0x7f51c40c5810 2024-09-17T05:14:07.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.766+0000 7f51b9ffb640 1 -- 172.21.15.150:0/3003318518 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f51c0003bd0 con 0x7f51c40c5810 2024-09-17T05:14:07.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.766+0000 7f51b9ffb640 1 --2- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f51a8073230 0x7f51a8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:07.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.766+0000 7f51c8873640 1 --2- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f51a8073230 0x7f51a8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:07.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.770+0000 7f51b9ffb640 1 -- 172.21.15.150:0/3003318518 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7f51c00949b0 con 0x7f51c40c5810 2024-09-17T05:14:07.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.770+0000 7f51c8873640 1 --2- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f51a8073230 0x7f51a8075740 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f51b40ae000 tx=0x7f51b40192d0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:07.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:07.774+0000 7f51b9ffb640 1 -- 172.21.15.150:0/3003318518 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f51c005ec10 con 0x7f51c40c5810 2024-09-17T05:14:08.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.250+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3003318518 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51a4005590 con 0x7f51c40c5810 2024-09-17T05:14:08.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.250+0000 7f51b9ffb640 1 -- 172.21.15.150:0/3003318518 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6547 (secure 0 0 0) 0x7f51c005e510 con 0x7f51c40c5810 2024-09-17T05:14:08.249 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:08.249 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":171,"btime":"2024-09-17T05:14:00:851032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7078,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:08.249 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 171 2024-09-17T05:14:08.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.254+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f51a8073230 msgr2=0x7f51a8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:08.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.254+0000 7f51cb2d8640 1 --2- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f51a8073230 0x7f51a8075740 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f51b40ae000 tx=0x7f51b40192d0 comp rx=0 tx=0).stop 2024-09-17T05:14:08.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.254+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51c40c5810 msgr2=0x7f51c41f4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:08.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.254+0000 7f51cb2d8640 1 --2- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51c40c5810 0x7f51c41f4940 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f51c000c4d0 tx=0x7f51c000c990 comp rx=0 tx=0).stop 2024-09-17T05:14:08.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.254+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3003318518 shutdown_connections 2024-09-17T05:14:08.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.254+0000 7f51cb2d8640 1 --2- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f51a8073230 0x7f51a8075740 secure :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f51b40ae000 tx=0x7f51b40192d0 comp rx=0 tx=0).stop 2024-09-17T05:14:08.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.254+0000 7f51cb2d8640 1 --2- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51c4158520 0x7f51c41f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:08.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.254+0000 7f51cb2d8640 1 --2- 172.21.15.150:0/3003318518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c4154fc0 0x7f51c41535d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:08.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.254+0000 7f51cb2d8640 1 --2- 172.21.15.150:0/3003318518 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f51c40c5810 0x7f51c41f4940 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:08.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.254+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3003318518 >> 172.21.15.150:0/3003318518 conn(0x7f51c40ac0f0 msgr2=0x7f51c41507c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:08.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.254+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3003318518 shutdown_connections 2024-09-17T05:14:08.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.254+0000 7f51cb2d8640 1 -- 172.21.15.150:0/3003318518 wait complete. 2024-09-17T05:14:08.304 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-17T05:14:08.304 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T05:14:08.304 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:14:08.321 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T05:14:08.321 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:08.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.698+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3834103262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b08070e50 msgr2=0x7f7b08071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:08.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.698+0000 7f7b0f7fe640 1 --2- 172.21.15.150:0/3834103262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b08070e50 0x7f7b08071260 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009900 tx=0x7f7af801c850 comp rx=0 tx=0).stop 2024-09-17T05:14:08.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.698+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3834103262 shutdown_connections 2024-09-17T05:14:08.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.698+0000 7f7b0f7fe640 1 --2- 172.21.15.150:0/3834103262 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b080717a0 0x7f7b08074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:08.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.698+0000 7f7b0f7fe640 1 --2- 172.21.15.150:0/3834103262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b08070e50 0x7f7b08071260 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:08.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.698+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3834103262 >> 172.21.15.150:0/3834103262 conn(0x7f7b080c4a00 msgr2=0x7f7b080c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:08.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.698+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3834103262 shutdown_connections 2024-09-17T05:14:08.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.698+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3834103262 wait complete. 2024-09-17T05:14:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.702+0000 7f7b0f7fe640 1 Processor -- start 2024-09-17T05:14:08.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.702+0000 7f7b0f7fe640 1 -- start start 2024-09-17T05:14:08.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b08070e50 0x7f7b080bd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:08.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0f7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b080717a0 0x7f7b0814d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:08.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b08073440 0x7f7b08144660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0f7fe640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7b080ad190 con 0x7f7b08070e50 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0f7fe640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7b080ad010 con 0x7f7b080717a0 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0f7fe640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7b080ad310 con 0x7f7b08073440 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0cd99640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b080717a0 0x7f7b0814d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0dd9b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b08073440 0x7f7b08144660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0cd99640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b080717a0 0x7f7b0814d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:48322/0 (socket says 172.21.15.150:48322) 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0cd99640 1 -- 172.21.15.150:0/3270301370 learned_addr learned my addr 172.21.15.150:0/3270301370 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0d59a640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b08070e50 0x7f7b080bd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0d59a640 1 -- 172.21.15.150:0/3270301370 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b08073440 msgr2=0x7f7b08144660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0d59a640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b08073440 0x7f7b08144660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0d59a640 1 -- 172.21.15.150:0/3270301370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b080717a0 msgr2=0x7f7b0814d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0d59a640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b080717a0 0x7f7b0814d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0d59a640 1 -- 172.21.15.150:0/3270301370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b080c2370 con 0x7f7b08070e50 2024-09-17T05:14:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0d59a640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b08070e50 0x7f7b080bd440 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400c390 tx=0x7f7b0400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7afdffb640 1 -- 172.21.15.150:0/3270301370 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b04017070 con 0x7f7b08070e50 2024-09-17T05:14:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7afdffb640 1 -- 172.21.15.150:0/3270301370 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b040053c0 con 0x7f7b08070e50 2024-09-17T05:14:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7afdffb640 1 -- 172.21.15.150:0/3270301370 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b04003860 con 0x7f7b08070e50 2024-09-17T05:14:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3270301370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b08150420 con 0x7f7b08070e50 2024-09-17T05:14:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.706+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3270301370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b081506a0 con 0x7f7b08070e50 2024-09-17T05:14:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.714+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3270301370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b080c60f0 con 0x7f7b08070e50 2024-09-17T05:14:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.714+0000 7f7afdffb640 1 -- 172.21.15.150:0/3270301370 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7b0401a070 con 0x7f7b08070e50 2024-09-17T05:14:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.714+0000 7f7afdffb640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7aec0b7210 0x7f7aec0b9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.714+0000 7f7afdffb640 1 -- 172.21.15.150:0/3270301370 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7f7b04094290 con 0x7f7b08070e50 2024-09-17T05:14:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.718+0000 7f7b0cd99640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7aec0b7210 0x7f7aec0b9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.718+0000 7f7b0cd99640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7aec0b7210 0x7f7aec0b9720 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7b08151cf0 tx=0x7f7af80195f0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:08.726+0000 7f7afdffb640 1 -- 172.21.15.150:0/3270301370 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7b08150420 con 0x7f7b08070e50 2024-09-17T05:14:09.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.282+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3270301370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b080ad830 con 0x7f7b08070e50 2024-09-17T05:14:09.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.286+0000 7f7afdffb640 1 -- 172.21.15.150:0/3270301370 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6547 (secure 0 0 0) 0x7f7b080c2370 con 0x7f7b08070e50 2024-09-17T05:14:09.285 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:09.285 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":171,"btime":"2024-09-17T05:14:00:851032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7078,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:09.286 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 171 2024-09-17T05:14:09.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.290+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3270301370 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7aec0b7210 msgr2=0x7f7aec0b9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:09.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.290+0000 7f7b0f7fe640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7aec0b7210 0x7f7aec0b9720 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7b08151cf0 tx=0x7f7af80195f0 comp rx=0 tx=0).stop 2024-09-17T05:14:09.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.290+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3270301370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b08070e50 msgr2=0x7f7b080bd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:09.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.290+0000 7f7b0f7fe640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b08070e50 0x7f7b080bd440 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400c390 tx=0x7f7b0400c850 comp rx=0 tx=0).stop 2024-09-17T05:14:09.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.294+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3270301370 shutdown_connections 2024-09-17T05:14:09.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.294+0000 7f7b0f7fe640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7aec0b7210 0x7f7aec0b9720 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:09.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.294+0000 7f7b0f7fe640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b080717a0 0x7f7b0814d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:09.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.294+0000 7f7b0f7fe640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7b08073440 0x7f7b08144660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:09.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.294+0000 7f7b0f7fe640 1 --2- 172.21.15.150:0/3270301370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b08070e50 0x7f7b080bd440 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:09.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.294+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3270301370 >> 172.21.15.150:0/3270301370 conn(0x7f7b080c4a00 msgr2=0x7f7b080a7300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:09.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.294+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3270301370 shutdown_connections 2024-09-17T05:14:09.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:09.294+0000 7f7b0f7fe640 1 -- 172.21.15.150:0/3270301370 wait complete. 2024-09-17T05:14:09.348 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:14:11.349 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:11.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.618+0000 7f5983300640 1 -- 172.21.15.150:0/2694441663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f597c0c3900 msgr2=0x7f597c074920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:11.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.618+0000 7f5983300640 1 --2- 172.21.15.150:0/2694441663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f597c0c3900 0x7f597c074920 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f5978009840 tx=0x7f597801c840 comp rx=0 tx=0).stop 2024-09-17T05:14:11.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.618+0000 7f5983300640 1 -- 172.21.15.150:0/2694441663 shutdown_connections 2024-09-17T05:14:11.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.618+0000 7f5983300640 1 --2- 172.21.15.150:0/2694441663 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f597c0c3900 0x7f597c074920 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:11.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.618+0000 7f5983300640 1 --2- 172.21.15.150:0/2694441663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f597c0c4250 0x7f597c153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:11.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.618+0000 7f5983300640 1 -- 172.21.15.150:0/2694441663 >> 172.21.15.150:0/2694441663 conn(0x7f597c06e360 msgr2=0x7f597c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:11.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.618+0000 7f5983300640 1 -- 172.21.15.150:0/2694441663 shutdown_connections 2024-09-17T05:14:11.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.618+0000 7f5983300640 1 -- 172.21.15.150:0/2694441663 wait complete. 2024-09-17T05:14:11.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.622+0000 7f5983300640 1 Processor -- start 2024-09-17T05:14:11.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f5983300640 1 -- start start 2024-09-17T05:14:11.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f5983300640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f597c074e60 0x7f597c14c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:11.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f5983300640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f597c0c3900 0x7f597c156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:11.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f5983300640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f597c0c4250 0x7f597c0a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:11.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f5983300640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f597c072f30 con 0x7f597c0c3900 2024-09-17T05:14:11.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f5983300640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f597c072db0 con 0x7f597c074e60 2024-09-17T05:14:11.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f5983300640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f597c160f90 con 0x7f597c0c4250 2024-09-17T05:14:11.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f598189d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f597c0c4250 0x7f597c0a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:11.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f598189d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f597c0c4250 0x7f597c0a99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:59834/0 (socket says 172.21.15.150:59834) 2024-09-17T05:14:11.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f598189d640 1 -- 172.21.15.150:0/2519540546 learned_addr learned my addr 172.21.15.150:0/2519540546 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:11.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f598189d640 1 -- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f597c074e60 msgr2=0x7f597c14c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:11.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f598189d640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f597c074e60 0x7f597c14c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:11.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f598189d640 1 -- 172.21.15.150:0/2519540546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f597c0c3900 msgr2=0x7f597c156440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:14:11.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.626+0000 7f598089b640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f597c0c3900 0x7f597c156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:11.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.630+0000 7f598189d640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f597c0c3900 0x7f597c156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:11.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.630+0000 7f598189d640 1 -- 172.21.15.150:0/2519540546 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597c153a30 con 0x7f597c0c4250 2024-09-17T05:14:11.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.630+0000 7f598089b640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f597c0c3900 0x7f597c156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:14:11.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.630+0000 7f598189d640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f597c0c4250 0x7f597c0a99b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f596c00c3f0 tx=0x7f596c00c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:11.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.630+0000 7f5971ffb640 1 -- 172.21.15.150:0/2519540546 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596c011790 con 0x7f597c0c4250 2024-09-17T05:14:11.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.630+0000 7f5971ffb640 1 -- 172.21.15.150:0/2519540546 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f596c017070 con 0x7f597c0c4250 2024-09-17T05:14:11.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.630+0000 7f5971ffb640 1 -- 172.21.15.150:0/2519540546 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596c011ca0 con 0x7f597c0c4250 2024-09-17T05:14:11.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.630+0000 7f5983300640 1 -- 172.21.15.150:0/2519540546 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f597c0ac870 con 0x7f597c0c4250 2024-09-17T05:14:11.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.630+0000 7f5983300640 1 -- 172.21.15.150:0/2519540546 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f597c156110 con 0x7f597c0c4250 2024-09-17T05:14:11.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.630+0000 7f5983300640 1 -- 172.21.15.150:0/2519540546 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f597c074920 con 0x7f597c0c4250 2024-09-17T05:14:11.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.630+0000 7f5971ffb640 1 -- 172.21.15.150:0/2519540546 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f596c01c460 con 0x7f597c0c4250 2024-09-17T05:14:11.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.634+0000 7f5971ffb640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5960073230 0x7f5960075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:11.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.634+0000 7f5971ffb640 1 -- 172.21.15.150:0/2519540546 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7f596c095950 con 0x7f597c0c4250 2024-09-17T05:14:11.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.634+0000 7f598109c640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5960073230 0x7f5960075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:11.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.634+0000 7f598109c640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5960073230 0x7f5960075740 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5974005fd0 tx=0x7f5974005ee0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:11.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:11.638+0000 7f5971ffb640 1 -- 172.21.15.150:0/2519540546 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f596c05fb10 con 0x7f597c0c4250 2024-09-17T05:14:12.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.094+0000 7f5983300640 1 -- 172.21.15.150:0/2519540546 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f597c0acae0 con 0x7f597c0c4250 2024-09-17T05:14:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.098+0000 7f5971ffb640 1 -- 172.21.15.150:0/2519540546 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6547 (secure 0 0 0) 0x7f596c05f410 con 0x7f597c0c4250 2024-09-17T05:14:12.095 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:12.095 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":171,"btime":"2024-09-17T05:14:00:851032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7078,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 171 2024-09-17T05:14:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.098+0000 7f5983300640 1 -- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5960073230 msgr2=0x7f5960075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.102+0000 7f5983300640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5960073230 0x7f5960075740 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5974005fd0 tx=0x7f5974005ee0 comp rx=0 tx=0).stop 2024-09-17T05:14:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.102+0000 7f5983300640 1 -- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f597c0c4250 msgr2=0x7f597c0a99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.102+0000 7f5983300640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f597c0c4250 0x7f597c0a99b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f596c00c3f0 tx=0x7f596c00c8b0 comp rx=0 tx=0).stop 2024-09-17T05:14:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.102+0000 7f5983300640 1 -- 172.21.15.150:0/2519540546 shutdown_connections 2024-09-17T05:14:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.102+0000 7f5983300640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5960073230 0x7f5960075740 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.102+0000 7f5983300640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f597c0c4250 0x7f597c0a99b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.102+0000 7f5983300640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f597c0c3900 0x7f597c156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.102+0000 7f5983300640 1 --2- 172.21.15.150:0/2519540546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f597c074e60 0x7f597c14c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.102+0000 7f5983300640 1 -- 172.21.15.150:0/2519540546 >> 172.21.15.150:0/2519540546 conn(0x7f597c06e360 msgr2=0x7f597c150300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.102+0000 7f5983300640 1 -- 172.21.15.150:0/2519540546 shutdown_connections 2024-09-17T05:14:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:12.102+0000 7f5983300640 1 -- 172.21.15.150:0/2519540546 wait complete. 2024-09-17T05:14:12.158 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:14:14.159 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:14.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.550+0000 7fde31e69640 1 -- 172.21.15.150:0/1503951141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde2c0c3ac0 msgr2=0x7fde2c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:14.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.550+0000 7fde31e69640 1 --2- 172.21.15.150:0/1503951141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde2c0c3ac0 0x7fde2c14fff0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009840 tx=0x7fde1c01c320 comp rx=0 tx=0).stop 2024-09-17T05:14:14.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.550+0000 7fde31e69640 1 -- 172.21.15.150:0/1503951141 shutdown_connections 2024-09-17T05:14:14.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.550+0000 7fde31e69640 1 --2- 172.21.15.150:0/1503951141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde2c0c3ac0 0x7fde2c14fff0 secure :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fde1c009840 tx=0x7fde1c01c320 comp rx=0 tx=0).stop 2024-09-17T05:14:14.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.550+0000 7fde31e69640 1 -- 172.21.15.150:0/1503951141 >> 172.21.15.150:0/1503951141 conn(0x7fde2c0c4a00 msgr2=0x7fde2c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:14.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.554+0000 7fde31e69640 1 -- 172.21.15.150:0/1503951141 shutdown_connections 2024-09-17T05:14:14.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.554+0000 7fde31e69640 1 -- 172.21.15.150:0/1503951141 wait complete. 2024-09-17T05:14:14.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.558+0000 7fde31e69640 1 Processor -- start 2024-09-17T05:14:14.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.558+0000 7fde31e69640 1 -- start start 2024-09-17T05:14:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde31e69640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde2c071310 0x7fde2c152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde31e69640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde2c071720 0x7fde2c2014f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde31e69640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde2c1541d0 0x7fde2c1fae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde31e69640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c1583b0 con 0x7fde2c071720 2024-09-17T05:14:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde31e69640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fde2c071120 con 0x7fde2c1541d0 2024-09-17T05:14:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde31e69640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fde2c158530 con 0x7fde2c071310 2024-09-17T05:14:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde2bfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde2c1541d0 0x7fde2c1fae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde2bfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde2c1541d0 0x7fde2c1fae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:34374/0 (socket says 172.21.15.150:34374) 2024-09-17T05:14:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde2bfff640 1 -- 172.21.15.150:0/1943128312 learned_addr learned my addr 172.21.15.150:0/1943128312 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde2bfff640 1 -- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde2c071310 msgr2=0x7fde2c152360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde2bfff640 1 --2- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde2c071310 0x7fde2c152360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde2bfff640 1 -- 172.21.15.150:0/1943128312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde2c071720 msgr2=0x7fde2c2014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde2bfff640 1 --2- 172.21.15.150:0/1943128312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde2c071720 0x7fde2c2014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde2bfff640 1 -- 172.21.15.150:0/1943128312 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2c1fe540 con 0x7fde2c1541d0 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde2bfff640 1 --2- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde2c1541d0 0x7fde2c1fae30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fde2400c6b0 tx=0x7fde2400cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde28ff9640 1 -- 172.21.15.150:0/1943128312 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24003b80 con 0x7fde2c1541d0 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde31e69640 1 -- 172.21.15.150:0/1943128312 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde2c1fe7e0 con 0x7fde2c1541d0 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde28ff9640 1 -- 172.21.15.150:0/1943128312 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde24003d40 con 0x7fde2c1541d0 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde28ff9640 1 -- 172.21.15.150:0/1943128312 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24005330 con 0x7fde2c1541d0 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde31e69640 1 -- 172.21.15.150:0/1943128312 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde2c0ad360 con 0x7fde2c1541d0 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.562+0000 7fde21ffb640 1 -- 172.21.15.150:0/1943128312 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde040052a0 con 0x7fde2c1541d0 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.566+0000 7fde28ff9640 1 -- 172.21.15.150:0/1943128312 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fde240054d0 con 0x7fde2c1541d0 2024-09-17T05:14:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.566+0000 7fde28ff9640 1 --2- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fde0c073200 0x7fde0c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:14.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.566+0000 7fde28ff9640 1 -- 172.21.15.150:0/1943128312 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fde24094ec0 con 0x7fde2c1541d0 2024-09-17T05:14:14.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.566+0000 7fde2b7fe640 1 --2- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fde0c073200 0x7fde0c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:14.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.574+0000 7fde28ff9640 1 -- 172.21.15.150:0/1943128312 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fde2c1fe540 con 0x7fde2c1541d0 2024-09-17T05:14:14.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.582+0000 7fde2b7fe640 1 --2- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fde0c073200 0x7fde0c075710 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0ae000 tx=0x7fde1c01ab60 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:14.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.882+0000 7fde21ffb640 1 -- 172.21.15.150:0/1943128312 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde04005590 con 0x7fde2c1541d0 2024-09-17T05:14:14.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.882+0000 7fde28ff9640 1 -- 172.21.15.150:0/1943128312 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6547 (secure 0 0 0) 0x7fde2405f090 con 0x7fde2c1541d0 2024-09-17T05:14:14.880 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:14.881 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":171,"btime":"2024-09-17T05:14:00:851032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7078,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:14.881 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 171 2024-09-17T05:14:14.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.886+0000 7fde21ffb640 1 -- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fde0c073200 msgr2=0x7fde0c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:14.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.886+0000 7fde21ffb640 1 --2- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fde0c073200 0x7fde0c075710 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fde1c0ae000 tx=0x7fde1c01ab60 comp rx=0 tx=0).stop 2024-09-17T05:14:14.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.886+0000 7fde21ffb640 1 -- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde2c1541d0 msgr2=0x7fde2c1fae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:14.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.886+0000 7fde21ffb640 1 --2- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde2c1541d0 0x7fde2c1fae30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fde2400c6b0 tx=0x7fde2400cb70 comp rx=0 tx=0).stop 2024-09-17T05:14:14.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.886+0000 7fde21ffb640 1 -- 172.21.15.150:0/1943128312 shutdown_connections 2024-09-17T05:14:14.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.886+0000 7fde21ffb640 1 --2- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fde0c073200 0x7fde0c075710 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:14.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.886+0000 7fde21ffb640 1 --2- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fde2c1541d0 0x7fde2c1fae30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:14.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.886+0000 7fde21ffb640 1 --2- 172.21.15.150:0/1943128312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde2c071720 0x7fde2c2014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:14.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.886+0000 7fde21ffb640 1 --2- 172.21.15.150:0/1943128312 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fde2c071310 0x7fde2c152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:14.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.886+0000 7fde21ffb640 1 -- 172.21.15.150:0/1943128312 >> 172.21.15.150:0/1943128312 conn(0x7fde2c0c4a00 msgr2=0x7fde2c0a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:14.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.886+0000 7fde21ffb640 1 -- 172.21.15.150:0/1943128312 shutdown_connections 2024-09-17T05:14:14.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:14.886+0000 7fde21ffb640 1 -- 172.21.15.150:0/1943128312 wait complete. 2024-09-17T05:14:14.905 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:14:16.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:17.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.074+0000 7fe666dea640 1 -- 172.21.15.150:0/1925343155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe660157810 msgr2=0x7fe660153b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:17.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.074+0000 7fe666dea640 1 --2- 172.21.15.150:0/1925343155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe660157810 0x7fe660153b20 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fe6600c9e10 tx=0x7fe65001c8c0 comp rx=0 tx=0).stop 2024-09-17T05:14:17.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.074+0000 7fe666dea640 1 -- 172.21.15.150:0/1925343155 shutdown_connections 2024-09-17T05:14:17.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.074+0000 7fe666dea640 1 --2- 172.21.15.150:0/1925343155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe660157810 0x7fe660153b20 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:17.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.074+0000 7fe666dea640 1 --2- 172.21.15.150:0/1925343155 >> v2:172.21.15.176:3300/0 conn(0x7fe6600c37b0 0x7fe6601535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:17.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.074+0000 7fe666dea640 1 -- 172.21.15.150:0/1925343155 >> 172.21.15.150:0/1925343155 conn(0x7fe6600aab60 msgr2=0x7fe6600aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:17.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.074+0000 7fe666dea640 1 -- 172.21.15.150:0/1925343155 shutdown_connections 2024-09-17T05:14:17.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.074+0000 7fe666dea640 1 -- 172.21.15.150:0/1925343155 wait complete. 2024-09-17T05:14:17.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe666dea640 1 Processor -- start 2024-09-17T05:14:17.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe666dea640 1 -- start start 2024-09-17T05:14:17.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe666dea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6600a8120 0x7fe6600ad150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:17.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe666dea640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6600c37b0 0x7fe660148aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:17.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe666dea640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe660157810 0x7fe660148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:17.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe666dea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe660160350 con 0x7fe6600a8120 2024-09-17T05:14:17.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe666dea640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe6601601d0 con 0x7fe660157810 2024-09-17T05:14:17.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe666dea640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe6601604d0 con 0x7fe6600c37b0 2024-09-17T05:14:17.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe664b86640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6600a8120 0x7fe6600ad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:17.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe664b86640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6600a8120 0x7fe6600ad150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54076/0 (socket says 172.21.15.150:54076) 2024-09-17T05:14:17.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe664b86640 1 -- 172.21.15.150:0/1533461484 learned_addr learned my addr 172.21.15.150:0/1533461484 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:17.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe665387640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe660157810 0x7fe660148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:17.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe65ffff640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6600c37b0 0x7fe660148aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe664b86640 1 -- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6600c37b0 msgr2=0x7fe660148aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe664b86640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6600c37b0 0x7fe660148aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe664b86640 1 -- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe660157810 msgr2=0x7fe660148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe664b86640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe660157810 0x7fe660148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe664b86640 1 -- 172.21.15.150:0/1533461484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe660149520 con 0x7fe6600a8120 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe665387640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe660157810 0x7fe660148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe664b86640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6600a8120 0x7fe6600ad150 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fe65800c3f0 tx=0x7fe65800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe65d7fa640 1 -- 172.21.15.150:0/1533461484 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe658017070 con 0x7fe6600a8120 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe65d7fa640 1 -- 172.21.15.150:0/1533461484 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6580053c0 con 0x7fe6600a8120 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe666dea640 1 -- 172.21.15.150:0/1533461484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6600adb60 con 0x7fe6600a8120 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe65d7fa640 1 -- 172.21.15.150:0/1533461484 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe658003860 con 0x7fe6600a8120 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe666dea640 1 -- 172.21.15.150:0/1533461484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe660150e90 con 0x7fe6600a8120 2024-09-17T05:14:17.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.078+0000 7fe666dea640 1 -- 172.21.15.150:0/1533461484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe660149f40 con 0x7fe6600a8120 2024-09-17T05:14:17.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.082+0000 7fe65d7fa640 1 -- 172.21.15.150:0/1533461484 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe65801a070 con 0x7fe6600a8120 2024-09-17T05:14:17.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.082+0000 7fe65d7fa640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe64c073230 0x7fe64c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:17.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.082+0000 7fe65d7fa640 1 -- 172.21.15.150:0/1533461484 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fe658094290 con 0x7fe6600a8120 2024-09-17T05:14:17.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.082+0000 7fe65ffff640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe64c073230 0x7fe64c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:17.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.082+0000 7fe65ffff640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe64c073230 0x7fe64c075740 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe654005c50 tx=0x7fe654005be0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:17.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.086+0000 7fe65d7fa640 1 -- 172.21.15.150:0/1533461484 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe65805e550 con 0x7fe6600a8120 2024-09-17T05:14:17.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.294+0000 7fe666dea640 1 -- 172.21.15.150:0/1533461484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe66006e870 con 0x7fe6600a8120 2024-09-17T05:14:17.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.294+0000 7fe65d7fa640 1 -- 172.21.15.150:0/1533461484 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6547 (secure 0 0 0) 0x7fe65805de50 con 0x7fe6600a8120 2024-09-17T05:14:17.293 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:17.293 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":171,"btime":"2024-09-17T05:14:00:851032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7078,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:17.293 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 171 2024-09-17T05:14:17.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.298+0000 7fe666dea640 1 -- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe64c073230 msgr2=0x7fe64c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:17.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.298+0000 7fe666dea640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe64c073230 0x7fe64c075740 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe654005c50 tx=0x7fe654005be0 comp rx=0 tx=0).stop 2024-09-17T05:14:17.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.298+0000 7fe666dea640 1 -- 172.21.15.150:0/1533461484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6600a8120 msgr2=0x7fe6600ad150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:17.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.298+0000 7fe666dea640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6600a8120 0x7fe6600ad150 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fe65800c3f0 tx=0x7fe65800c8b0 comp rx=0 tx=0).stop 2024-09-17T05:14:17.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.298+0000 7fe666dea640 1 -- 172.21.15.150:0/1533461484 shutdown_connections 2024-09-17T05:14:17.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.298+0000 7fe666dea640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe64c073230 0x7fe64c075740 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:17.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.298+0000 7fe666dea640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe660157810 0x7fe660148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:17.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.298+0000 7fe666dea640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe6600c37b0 0x7fe660148aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:17.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.298+0000 7fe666dea640 1 --2- 172.21.15.150:0/1533461484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe6600a8120 0x7fe6600ad150 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:17.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.298+0000 7fe666dea640 1 -- 172.21.15.150:0/1533461484 >> 172.21.15.150:0/1533461484 conn(0x7fe6600aab60 msgr2=0x7fe660156400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:17.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.298+0000 7fe666dea640 1 -- 172.21.15.150:0/1533461484 shutdown_connections 2024-09-17T05:14:17.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:17.298+0000 7fe666dea640 1 -- 172.21.15.150:0/1533461484 wait complete. 2024-09-17T05:14:17.316 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:14:19.317 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.486+0000 7fc5e387a640 1 -- 172.21.15.150:0/1812032750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c8005920 msgr2=0x7fc5c8005d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.486+0000 7fc5e387a640 1 --2- 172.21.15.150:0/1812032750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c8005920 0x7fc5c8005d30 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8006fd0 tx=0x7fc5d801f660 comp rx=0 tx=0).stop 2024-09-17T05:14:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.486+0000 7fc5e387a640 1 -- 172.21.15.150:0/1812032750 shutdown_connections 2024-09-17T05:14:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.486+0000 7fc5e387a640 1 --2- 172.21.15.150:0/1812032750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5c8005920 0x7fc5c8005d30 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.486+0000 7fc5e387a640 1 -- 172.21.15.150:0/1812032750 >> 172.21.15.150:0/1812032750 conn(0x7fc5dc0c4a00 msgr2=0x7fc5dc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.486+0000 7fc5e387a640 1 -- 172.21.15.150:0/1812032750 shutdown_connections 2024-09-17T05:14:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.486+0000 7fc5e387a640 1 -- 172.21.15.150:0/1812032750 wait complete. 2024-09-17T05:14:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.486+0000 7fc5e387a640 1 Processor -- start 2024-09-17T05:14:19.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.486+0000 7fc5e387a640 1 -- start start 2024-09-17T05:14:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e387a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5c8005920 0x7fc5dc154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e387a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5dc1e84c0 0x7fc5dc148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e387a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5dc149330 0x7fc5dc0ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e387a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc5dc070da0 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e387a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc5dc070ac0 con 0x7fc5dc149330 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e387a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc5dc0c6690 con 0x7fc5c8005920 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e0e15640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5dc1e84c0 0x7fc5dc148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e0e15640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5dc1e84c0 0x7fc5dc148df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42032/0 (socket says 172.21.15.150:42032) 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e0e15640 1 -- 172.21.15.150:0/1341932747 learned_addr learned my addr 172.21.15.150:0/1341932747 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e1616640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5c8005920 0x7fc5dc154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e1e17640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5dc149330 0x7fc5dc0ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e0e15640 1 -- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5c8005920 msgr2=0x7fc5dc154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e0e15640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5c8005920 0x7fc5dc154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e0e15640 1 -- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5dc149330 msgr2=0x7fc5dc0ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e0e15640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5dc149330 0x7fc5dc0ac6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e0e15640 1 -- 172.21.15.150:0/1341932747 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5dc14f920 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e0e15640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5dc1e84c0 0x7fc5dc148df0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8026370 tx=0x7fc5d8006ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5d1ffb640 1 -- 172.21.15.150:0/1341932747 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8006420 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5d1ffb640 1 -- 172.21.15.150:0/1341932747 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5d80065c0 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5d1ffb640 1 -- 172.21.15.150:0/1341932747 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8003600 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e387a640 1 -- 172.21.15.150:0/1341932747 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5dc156290 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e387a640 1 -- 172.21.15.150:0/1341932747 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5dc1e7fa0 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.490+0000 7fc5e387a640 1 -- 172.21.15.150:0/1341932747 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5dc1e7d90 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.494+0000 7fc5d1ffb640 1 -- 172.21.15.150:0/1341932747 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc5d80c1030 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.494+0000 7fc5d1ffb640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc5c00731f0 0x7fc5c0075700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:19.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.494+0000 7fc5d1ffb640 1 -- 172.21.15.150:0/1341932747 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fc5d8132930 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.494+0000 7fc5e1616640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc5c00731f0 0x7fc5c0075700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:19.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.494+0000 7fc5e1616640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc5c00731f0 0x7fc5c0075700 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc000c10 tx=0x7fc5cc006b80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:19.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.498+0000 7fc5d1ffb640 1 -- 172.21.15.150:0/1341932747 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc5d80c8060 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.706+0000 7fc5e387a640 1 -- 172.21.15.150:0/1341932747 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5dc0c6bf0 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5d1ffb640 1 -- 172.21.15.150:0/1341932747 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6547 (secure 0 0 0) 0x7fc5d8019410 con 0x7fc5dc1e84c0 2024-09-17T05:14:19.707 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:19.707 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":171,"btime":"2024-09-17T05:14:00:851032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7078,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:19.707 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 171 2024-09-17T05:14:19.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5e387a640 1 -- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc5c00731f0 msgr2=0x7fc5c0075700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:19.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5e387a640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc5c00731f0 0x7fc5c0075700 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc000c10 tx=0x7fc5cc006b80 comp rx=0 tx=0).stop 2024-09-17T05:14:19.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5e387a640 1 -- 172.21.15.150:0/1341932747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5dc1e84c0 msgr2=0x7fc5dc148df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:19.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5e387a640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5dc1e84c0 0x7fc5dc148df0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8026370 tx=0x7fc5d8006ca0 comp rx=0 tx=0).stop 2024-09-17T05:14:19.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5e387a640 1 -- 172.21.15.150:0/1341932747 shutdown_connections 2024-09-17T05:14:19.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5e387a640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc5c00731f0 0x7fc5c0075700 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:19.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5e387a640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5dc149330 0x7fc5dc0ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:19.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5e387a640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5dc1e84c0 0x7fc5dc148df0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:19.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5e387a640 1 --2- 172.21.15.150:0/1341932747 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc5c8005920 0x7fc5dc154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:19.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5e387a640 1 -- 172.21.15.150:0/1341932747 >> 172.21.15.150:0/1341932747 conn(0x7fc5dc0c4a00 msgr2=0x7fc5dc153100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:19.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5e387a640 1 -- 172.21.15.150:0/1341932747 shutdown_connections 2024-09-17T05:14:19.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:19.710+0000 7fc5e387a640 1 -- 172.21.15.150:0/1341932747 wait complete. 2024-09-17T05:14:19.729 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:14:21.731 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:21.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.898+0000 7f730bf77640 1 -- 172.21.15.150:0/1329243622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f0005e60 msgr2=0x7f72f0006270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:21.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.898+0000 7f730bf77640 1 --2- 172.21.15.150:0/1329243622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f0005e60 0x7f72f0006270 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f72fc001ec0 tx=0x7f72fc01cf60 comp rx=0 tx=0).stop 2024-09-17T05:14:21.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.898+0000 7f730a514640 1 -- 172.21.15.150:0/1329243622 reap_dead start 2024-09-17T05:14:21.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.898+0000 7f730bf77640 1 -- 172.21.15.150:0/1329243622 shutdown_connections 2024-09-17T05:14:21.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.898+0000 7f730bf77640 1 -- 172.21.15.150:0/1329243622 >> 172.21.15.150:0/1329243622 conn(0x7f73040c4a00 msgr2=0x7f73040c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:21.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.898+0000 7f730bf77640 1 -- 172.21.15.150:0/1329243622 shutdown_connections 2024-09-17T05:14:21.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.898+0000 7f730bf77640 1 -- 172.21.15.150:0/1329243622 wait complete. 2024-09-17T05:14:21.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730bf77640 1 Processor -- start 2024-09-17T05:14:21.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730bf77640 1 -- start start 2024-09-17T05:14:21.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730bf77640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72f0005e60 0x7f730414b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:21.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730bf77640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7304156b70 0x7f73040aaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:21.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730bf77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73040ab0e0 0x7f73040ad070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:21.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730bf77640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7304070ac0 con 0x7f73040ab0e0 2024-09-17T05:14:21.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730bf77640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f73040707e0 con 0x7f7304156b70 2024-09-17T05:14:21.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730bf77640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7304070da0 con 0x7f72f0005e60 2024-09-17T05:14:21.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730a514640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73040ab0e0 0x7f73040ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:21.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730a514640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73040ab0e0 0x7f73040ad070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42046/0 (socket says 172.21.15.150:42046) 2024-09-17T05:14:21.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730a514640 1 -- 172.21.15.150:0/1644526475 learned_addr learned my addr 172.21.15.150:0/1644526475 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:21.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f7309d13640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72f0005e60 0x7f730414b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730a514640 1 -- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72f0005e60 msgr2=0x7f730414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730a514640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72f0005e60 0x7f730414b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730a514640 1 -- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7304156b70 msgr2=0x7f73040aaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730a514640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7304156b70 0x7f73040aaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730a514640 1 -- 172.21.15.150:0/1644526475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7304155d90 con 0x7f73040ab0e0 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f7309d13640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72f0005e60 0x7f730414b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730a514640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73040ab0e0 0x7f73040ad070 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f72fc01ca20 tx=0x7f72fc01d1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f72f67fc640 1 -- 172.21.15.150:0/1644526475 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc006c60 con 0x7f73040ab0e0 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f72f67fc640 1 -- 172.21.15.150:0/1644526475 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72fc006090 con 0x7f73040ab0e0 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f72f67fc640 1 -- 172.21.15.150:0/1644526475 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72fc003600 con 0x7f73040ab0e0 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730bf77640 1 -- 172.21.15.150:0/1644526475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7304156070 con 0x7f73040ab0e0 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.902+0000 7f730bf77640 1 -- 172.21.15.150:0/1644526475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73040ab560 con 0x7f73040ab0e0 2024-09-17T05:14:21.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.906+0000 7f730bf77640 1 -- 172.21.15.150:0/1644526475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7304156240 con 0x7f73040ab0e0 2024-09-17T05:14:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.906+0000 7f72f67fc640 1 -- 172.21.15.150:0/1644526475 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f72fc0bf020 con 0x7f73040ab0e0 2024-09-17T05:14:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.906+0000 7f72f67fc640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e8073200 0x7f72e8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.906+0000 7f72f67fc640 1 -- 172.21.15.150:0/1644526475 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7f72fc131b60 con 0x7f73040ab0e0 2024-09-17T05:14:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.906+0000 7f7309d13640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e8073200 0x7f72e8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:21.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.906+0000 7f7309d13640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e8073200 0x7f72e8075710 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f7300009e00 tx=0x7f73000097e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:21.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:21.910+0000 7f72f67fc640 1 -- 172.21.15.150:0/1644526475 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f72fc131e20 con 0x7f73040ab0e0 2024-09-17T05:14:22.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.110+0000 7f730bf77640 1 -- 172.21.15.150:0/1644526475 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73040c6690 con 0x7f73040ab0e0 2024-09-17T05:14:22.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f72f67fc640 1 -- 172.21.15.150:0/1644526475 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6547 (secure 0 0 0) 0x7f72fc016d10 con 0x7f73040ab0e0 2024-09-17T05:14:22.110 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:22.110 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":171,"btime":"2024-09-17T05:14:00:851032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7078,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:22.111 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 171 2024-09-17T05:14:22.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f730bf77640 1 -- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e8073200 msgr2=0x7f72e8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:22.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f730bf77640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e8073200 0x7f72e8075710 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f7300009e00 tx=0x7f73000097e0 comp rx=0 tx=0).stop 2024-09-17T05:14:22.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f730bf77640 1 -- 172.21.15.150:0/1644526475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73040ab0e0 msgr2=0x7f73040ad070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:22.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f730bf77640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73040ab0e0 0x7f73040ad070 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f72fc01ca20 tx=0x7f72fc01d1b0 comp rx=0 tx=0).stop 2024-09-17T05:14:22.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f730bf77640 1 -- 172.21.15.150:0/1644526475 shutdown_connections 2024-09-17T05:14:22.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f730bf77640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f72e8073200 0x7f72e8075710 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:22.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f730bf77640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7304156b70 0x7f73040aaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:22.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f730bf77640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f73040ab0e0 0x7f73040ad070 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:22.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f730bf77640 1 --2- 172.21.15.150:0/1644526475 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f72f0005e60 0x7f730414b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:22.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f730bf77640 1 -- 172.21.15.150:0/1644526475 >> 172.21.15.150:0/1644526475 conn(0x7f73040c4a00 msgr2=0x7f73040a7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:22.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f730bf77640 1 -- 172.21.15.150:0/1644526475 shutdown_connections 2024-09-17T05:14:22.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:22.114+0000 7f730bf77640 1 -- 172.21.15.150:0/1644526475 wait complete. 2024-09-17T05:14:22.134 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:14:24.135 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:24.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 -- 172.21.15.150:0/1548838670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6234074980 msgr2=0x7f62340c3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:24.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 --2- 172.21.15.150:0/1548838670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6234074980 0x7f62340c3fb0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f62340c3ca0 tx=0x7f622c01de40 comp rx=0 tx=0).stop 2024-09-17T05:14:24.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 -- 172.21.15.150:0/1548838670 shutdown_connections 2024-09-17T05:14:24.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 --2- 172.21.15.150:0/1548838670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6234074980 0x7f62340c3fb0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:24.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 -- 172.21.15.150:0/1548838670 >> 172.21.15.150:0/1548838670 conn(0x7f62340c5ae0 msgr2=0x7f6234073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:24.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 -- 172.21.15.150:0/1548838670 shutdown_connections 2024-09-17T05:14:24.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 -- 172.21.15.150:0/1548838670 wait complete. 2024-09-17T05:14:24.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 Processor -- start 2024-09-17T05:14:24.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 -- start start 2024-09-17T05:14:24.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f623406fca0 0x7f623414c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:24.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6234074570 0x7f623414a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:24.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6234074980 0x7f62341449d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:24.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f62340c4b00 con 0x7f6234074980 2024-09-17T05:14:24.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f62340c4980 con 0x7f623406fca0 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f623b620640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f62340c4c80 con 0x7f6234074570 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f6239bbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6234074980 0x7f62341449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f6239bbd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6234074980 0x7f62341449d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42052/0 (socket says 172.21.15.150:42052) 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f6239bbd640 1 -- 172.21.15.150:0/3615648979 learned_addr learned my addr 172.21.15.150:0/3615648979 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f62393bc640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f623406fca0 0x7f623414c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f6238bbb640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6234074570 0x7f623414a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f6239bbd640 1 -- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6234074570 msgr2=0x7f623414a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f6239bbd640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6234074570 0x7f623414a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f6239bbd640 1 -- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f623406fca0 msgr2=0x7f623414c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f6239bbd640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f623406fca0 0x7f623414c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f6239bbd640 1 -- 172.21.15.150:0/3615648979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f623414e650 con 0x7f6234074980 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.318+0000 7f6239bbd640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6234074980 0x7f62341449d0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f622c099910 tx=0x7f622c005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.322+0000 7f6229ffb640 1 -- 172.21.15.150:0/3615648979 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c0be070 con 0x7f6234074980 2024-09-17T05:14:24.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.322+0000 7f6229ffb640 1 -- 172.21.15.150:0/3615648979 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f622c004330 con 0x7f6234074980 2024-09-17T05:14:24.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.322+0000 7f623b620640 1 -- 172.21.15.150:0/3615648979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f623414e930 con 0x7f6234074980 2024-09-17T05:14:24.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.322+0000 7f6229ffb640 1 -- 172.21.15.150:0/3615648979 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c0b3040 con 0x7f6234074980 2024-09-17T05:14:24.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.322+0000 7f623b620640 1 -- 172.21.15.150:0/3615648979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f623414ad40 con 0x7f6234074980 2024-09-17T05:14:24.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.322+0000 7f623b620640 1 -- 172.21.15.150:0/3615648979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f623414f400 con 0x7f6234074980 2024-09-17T05:14:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.322+0000 7f6229ffb640 1 -- 172.21.15.150:0/3615648979 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f622c0c1070 con 0x7f6234074980 2024-09-17T05:14:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.322+0000 7f6229ffb640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6218073200 0x7f6218075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.322+0000 7f6229ffb640 1 -- 172.21.15.150:0/3615648979 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7f622c12f9b0 con 0x7f6234074980 2024-09-17T05:14:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.322+0000 7f62393bc640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6218073200 0x7f6218075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:24.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.322+0000 7f62393bc640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6218073200 0x7f6218075710 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6230001640 tx=0x7f6230009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:24.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.326+0000 7f6229ffb640 1 -- 172.21.15.150:0/3615648979 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f622c017ac0 con 0x7f6234074980 2024-09-17T05:14:24.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 -- 172.21.15.150:0/3615648979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f623414eb00 con 0x7f6234074980 2024-09-17T05:14:24.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f6229ffb640 1 -- 172.21.15.150:0/3615648979 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+6547 (secure 0 0 0) 0x7f623414eb00 con 0x7f6234074980 2024-09-17T05:14:24.531 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:24.531 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":171,"btime":"2024-09-17T05:14:00:851032+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7078,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:12:04.469086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7407},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7407":{"gid":7407,"name":"c","rank":0,"incarnation":157,"state":"up:active","state_seq":19,"addr":"172.21.15.150:6835/3179072684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3179072684},{"type":"v1","addr":"172.21.15.150:6835","nonce":3179072684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7407,"qdb_cluster":[7407]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:24.532 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 171 2024-09-17T05:14:24.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 -- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6218073200 msgr2=0x7f6218075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:24.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6218073200 0x7f6218075710 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6230001640 tx=0x7f6230009040 comp rx=0 tx=0).stop 2024-09-17T05:14:24.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 -- 172.21.15.150:0/3615648979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6234074980 msgr2=0x7f62341449d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:24.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6234074980 0x7f62341449d0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f622c099910 tx=0x7f622c005ae0 comp rx=0 tx=0).stop 2024-09-17T05:14:24.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 -- 172.21.15.150:0/3615648979 shutdown_connections 2024-09-17T05:14:24.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6218073200 0x7f6218075710 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:24.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6234074980 0x7f62341449d0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:24.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6234074570 0x7f623414a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:24.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 --2- 172.21.15.150:0/3615648979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f623406fca0 0x7f623414c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:24.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 -- 172.21.15.150:0/3615648979 >> 172.21.15.150:0/3615648979 conn(0x7f62340c5ae0 msgr2=0x7f62340ad4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:24.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 -- 172.21.15.150:0/3615648979 shutdown_connections 2024-09-17T05:14:24.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:24.534+0000 7f623b620640 1 -- 172.21.15.150:0/3615648979 wait complete. 2024-09-17T05:14:24.555 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:14:26.556 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:26.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.714+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/347354722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b30004d50 msgr2=0x7f5b30005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:26.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.714+0000 7f5b4d3bc640 1 --2- 172.21.15.150:0/347354722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b30004d50 0x7f5b30005220 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f5b480719d0 tx=0x7f5b4001d2a0 comp rx=0 tx=0).stop 2024-09-17T05:14:26.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.714+0000 7f5b477fe640 1 -- 172.21.15.150:0/347354722 reap_dead start 2024-09-17T05:14:26.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.714+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/347354722 shutdown_connections 2024-09-17T05:14:26.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.714+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/347354722 >> 172.21.15.150:0/347354722 conn(0x7f5b4806e360 msgr2=0x7f5b4806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:26.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.714+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/347354722 shutdown_connections 2024-09-17T05:14:26.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.714+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/347354722 wait complete. 2024-09-17T05:14:26.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.714+0000 7f5b4d3bc640 1 Processor -- start 2024-09-17T05:14:26.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.714+0000 7f5b4d3bc640 1 -- start start 2024-09-17T05:14:26.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b4d3bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480be930 0x7f5b4814f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:26.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b4d3bc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b481562f0 0x7f5b48154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:26.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b4d3bc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b48155360 0x7f5b481edce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:26.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b4d3bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5b48071120 con 0x7f5b480be930 2024-09-17T05:14:26.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b4d3bc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5b48070fa0 con 0x7f5b48155360 2024-09-17T05:14:26.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b4d3bc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5b48072db0 con 0x7f5b481562f0 2024-09-17T05:14:26.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b46ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480be930 0x7f5b4814f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:26.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b46ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480be930 0x7f5b4814f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42072/0 (socket says 172.21.15.150:42072) 2024-09-17T05:14:26.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b46ffd640 1 -- 172.21.15.150:0/4031193759 learned_addr learned my addr 172.21.15.150:0/4031193759 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:26.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b467fc640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b481562f0 0x7f5b48154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:26.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b477fe640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b48155360 0x7f5b481edce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:26.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b46ffd640 1 -- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b481562f0 msgr2=0x7f5b48154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:26.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b46ffd640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b481562f0 0x7f5b48154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:26.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b46ffd640 1 -- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b48155360 msgr2=0x7f5b481edce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:26.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b46ffd640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b48155360 0x7f5b481edce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:26.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b46ffd640 1 -- 172.21.15.150:0/4031193759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b48155c00 con 0x7f5b480be930 2024-09-17T05:14:26.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b46ffd640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480be930 0x7f5b4814f220 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f5b38004eb0 tx=0x7f5b38001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:26.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b2f7fe640 1 -- 172.21.15.150:0/4031193759 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38015020 con 0x7f5b480be930 2024-09-17T05:14:26.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b2f7fe640 1 -- 172.21.15.150:0/4031193759 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b38011750 con 0x7f5b480be930 2024-09-17T05:14:26.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b2f7fe640 1 -- 172.21.15.150:0/4031193759 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38011b60 con 0x7f5b480be930 2024-09-17T05:14:26.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/4031193759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b4814fc50 con 0x7f5b480be930 2024-09-17T05:14:26.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.718+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/4031193759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b481ed7c0 con 0x7f5b480be930 2024-09-17T05:14:26.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.722+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/4031193759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b480a88e0 con 0x7f5b480be930 2024-09-17T05:14:26.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.722+0000 7f5b2f7fe640 1 -- 172.21.15.150:0/4031193759 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5b38018020 con 0x7f5b480be930 2024-09-17T05:14:26.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.722+0000 7f5b2f7fe640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b34073200 0x7f5b34075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:26.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.722+0000 7f5b2f7fe640 1 -- 172.21.15.150:0/4031193759 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f5b38096110 con 0x7f5b480be930 2024-09-17T05:14:26.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.722+0000 7f5b467fc640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b34073200 0x7f5b34075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:26.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.726+0000 7f5b467fc640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b34073200 0x7f5b34075710 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5b480a88b0 tx=0x7f5b3c0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:26.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.726+0000 7f5b2f7fe640 1 -- 172.21.15.150:0/4031193759 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5b3801f020 con 0x7f5b480be930 2024-09-17T05:14:26.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/4031193759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b480c3d10 con 0x7f5b480be930 2024-09-17T05:14:26.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b2f7fe640 1 -- 172.21.15.150:0/4031193759 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5710 (secure 0 0 0) 0x7f5b380604c0 con 0x7f5b480be930 2024-09-17T05:14:26.939 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:26.940 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":172,"btime":"2024-09-17T05:14:26:221649+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:26.221635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:26.940 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 172 2024-09-17T05:14:26.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b34073200 msgr2=0x7f5b34075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:26.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b34073200 0x7f5b34075710 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5b480a88b0 tx=0x7f5b3c0091e0 comp rx=0 tx=0).stop 2024-09-17T05:14:26.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/4031193759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480be930 msgr2=0x7f5b4814f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:26.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480be930 0x7f5b4814f220 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f5b38004eb0 tx=0x7f5b38001640 comp rx=0 tx=0).stop 2024-09-17T05:14:26.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/4031193759 shutdown_connections 2024-09-17T05:14:26.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b34073200 0x7f5b34075710 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:26.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b48155360 0x7f5b481edce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:26.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b481562f0 0x7f5b48154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:26.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 --2- 172.21.15.150:0/4031193759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b480be930 0x7f5b4814f220 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:26.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/4031193759 >> 172.21.15.150:0/4031193759 conn(0x7f5b4806e360 msgr2=0x7f5b4806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:26.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/4031193759 shutdown_connections 2024-09-17T05:14:26.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:26.942+0000 7f5b4d3bc640 1 -- 172.21.15.150:0/4031193759 wait complete. 2024-09-17T05:14:26.967 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-17T05:14:26.967 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:14:26.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:27.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.114+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/4150347734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe680c4250 msgr2=0x7fbe68153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:27.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.114+0000 7fbe6fd4b640 1 --2- 172.21.15.150:0/4150347734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe680c4250 0x7fbe68153ee0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009840 tx=0x7fbe6401c820 comp rx=0 tx=0).stop 2024-09-17T05:14:27.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.114+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/4150347734 shutdown_connections 2024-09-17T05:14:27.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.114+0000 7fbe6fd4b640 1 --2- 172.21.15.150:0/4150347734 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe680c3900 0x7fbe68074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:27.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 --2- 172.21.15.150:0/4150347734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe680c4250 0x7fbe68153ee0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:27.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/4150347734 >> 172.21.15.150:0/4150347734 conn(0x7fbe6806e360 msgr2=0x7fbe6806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:27.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/4150347734 shutdown_connections 2024-09-17T05:14:27.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/4150347734 wait complete. 2024-09-17T05:14:27.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 Processor -- start 2024-09-17T05:14:27.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 -- start start 2024-09-17T05:14:27.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe68074e60 0x7fbe68158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:27.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe680c3900 0x7fbe6806d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:27.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe680c4250 0x7fbe681ef560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:27.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbe68072f30 con 0x7fbe680c4250 2024-09-17T05:14:27.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbe68072db0 con 0x7fbe68074e60 2024-09-17T05:14:27.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbe68160f90 con 0x7fbe680c3900 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6e2e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe680c4250 0x7fbe681ef560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6e2e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe680c4250 0x7fbe681ef560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42088/0 (socket says 172.21.15.150:42088) 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6e2e8640 1 -- 172.21.15.150:0/3132829725 learned_addr learned my addr 172.21.15.150:0/3132829725 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6d2e6640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe680c3900 0x7fbe6806d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6dae7640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe68074e60 0x7fbe68158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6e2e8640 1 -- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe680c3900 msgr2=0x7fbe6806d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6e2e8640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe680c3900 0x7fbe6806d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6e2e8640 1 -- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe68074e60 msgr2=0x7fbe68158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6e2e8640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe68074e60 0x7fbe68158f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6e2e8640 1 -- 172.21.15.150:0/3132829725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe68157d30 con 0x7fbe680c4250 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6e2e8640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe680c4250 0x7fbe681ef560 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000cae0 tx=0x7fbe6000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe5e7fc640 1 -- 172.21.15.150:0/3132829725 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60016020 con 0x7fbe680c4250 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe5e7fc640 1 -- 172.21.15.150:0/3132829725 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbe6000ab40 con 0x7fbe680c4250 2024-09-17T05:14:27.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/3132829725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe68158010 con 0x7fbe680c4250 2024-09-17T05:14:27.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe5e7fc640 1 -- 172.21.15.150:0/3132829725 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe6000ce20 con 0x7fbe680c4250 2024-09-17T05:14:27.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.118+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/3132829725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe681eed60 con 0x7fbe680c4250 2024-09-17T05:14:27.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.122+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/3132829725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe68074920 con 0x7fbe680c4250 2024-09-17T05:14:27.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.122+0000 7fbe5e7fc640 1 -- 172.21.15.150:0/3132829725 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbe60019020 con 0x7fbe680c4250 2024-09-17T05:14:27.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.122+0000 7fbe5e7fc640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe4c073230 0x7fbe4c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:27.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.122+0000 7fbe5e7fc640 1 -- 172.21.15.150:0/3132829725 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7fbe60095380 con 0x7fbe680c4250 2024-09-17T05:14:27.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.122+0000 7fbe6dae7640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe4c073230 0x7fbe4c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:27.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.122+0000 7fbe6dae7640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe4c073230 0x7fbe4c075740 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbe640af000 tx=0x7fbe6401b060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:27.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.126+0000 7fbe5e7fc640 1 -- 172.21.15.150:0/3132829725 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbe60020020 con 0x7fbe680c4250 2024-09-17T05:14:27.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.330+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/3132829725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe6806dc80 con 0x7fbe680c4250 2024-09-17T05:14:27.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.330+0000 7fbe5e7fc640 1 -- 172.21.15.150:0/3132829725 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) ==== 95+0+5713 (secure 0 0 0) 0x7fbe6005f4d0 con 0x7fbe680c4250 2024-09-17T05:14:27.328 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:27.328 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":173,"btime":"2024-09-17T05:14:27:322796+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:26.732255+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:27.328 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 173 2024-09-17T05:14:27.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.330+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe4c073230 msgr2=0x7fbe4c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:27.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.330+0000 7fbe6fd4b640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe4c073230 0x7fbe4c075740 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbe640af000 tx=0x7fbe6401b060 comp rx=0 tx=0).stop 2024-09-17T05:14:27.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.330+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/3132829725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe680c4250 msgr2=0x7fbe681ef560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.330+0000 7fbe6fd4b640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe680c4250 0x7fbe681ef560 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000cae0 tx=0x7fbe6000cbd0 comp rx=0 tx=0).stop 2024-09-17T05:14:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.334+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/3132829725 shutdown_connections 2024-09-17T05:14:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.334+0000 7fbe6fd4b640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbe4c073230 0x7fbe4c075740 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.334+0000 7fbe6fd4b640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe680c4250 0x7fbe681ef560 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.334+0000 7fbe6fd4b640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbe680c3900 0x7fbe6806d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.334+0000 7fbe6fd4b640 1 --2- 172.21.15.150:0/3132829725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe68074e60 0x7fbe68158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.334+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/3132829725 >> 172.21.15.150:0/3132829725 conn(0x7fbe6806e360 msgr2=0x7fbe681547b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.334+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/3132829725 shutdown_connections 2024-09-17T05:14:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:27.334+0000 7fbe6fd4b640 1 -- 172.21.15.150:0/3132829725 wait complete. 2024-09-17T05:14:27.352 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:14:27:322796+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 173, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 173, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7078": { "addr": "172.21.15.150:6833/3933379792", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 3933379792, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 3933379792, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7078, "incarnation": 172, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:14:26.732255+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7078 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 170, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7015": { "addr": "172.21.15.176:6837/2338457617", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 2338457617, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 2338457617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7015, "incarnation": 167, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:13:49.801008+0000", "qdb_cluster": [ 7015 ], "qdb_leader": 7015, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7015 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:14:29.353 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:29.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.554+0000 7f8b906d3640 1 -- 172.21.15.150:0/143432909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b88071670 msgr2=0x7f8b880c5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:29.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.554+0000 7f8b906d3640 1 --2- 172.21.15.150:0/143432909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b88071670 0x7f8b880c5680 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f8b880b00f0 tx=0x7f8b8401c830 comp rx=0 tx=0).stop 2024-09-17T05:14:29.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.554+0000 7f8b906d3640 1 -- 172.21.15.150:0/143432909 shutdown_connections 2024-09-17T05:14:29.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.554+0000 7f8b906d3640 1 --2- 172.21.15.150:0/143432909 >> v2:172.21.15.176:3300/0 conn(0x7f8b8814e490 0x7f8b880c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:29.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.554+0000 7f8b906d3640 1 --2- 172.21.15.150:0/143432909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b88071670 0x7f8b880c5680 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:29.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.554+0000 7f8b906d3640 1 -- 172.21.15.150:0/143432909 >> 172.21.15.150:0/143432909 conn(0x7f8b880ca460 msgr2=0x7f8b880c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:29.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.554+0000 7f8b906d3640 1 -- 172.21.15.150:0/143432909 shutdown_connections 2024-09-17T05:14:29.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.554+0000 7f8b906d3640 1 -- 172.21.15.150:0/143432909 wait complete. 2024-09-17T05:14:29.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b906d3640 1 Processor -- start 2024-09-17T05:14:29.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b906d3640 1 -- start start 2024-09-17T05:14:29.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b906d3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b88071670 0x7f8b8814fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:29.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b906d3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b8814e490 0x7f8b88147280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:29.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b906d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b88157c00 0x7f8b880c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:29.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b906d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8b880c6740 con 0x7f8b88157c00 2024-09-17T05:14:29.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b906d3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8b880c65c0 con 0x7f8b88071670 2024-09-17T05:14:29.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b906d3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8b8814e8a0 con 0x7f8b8814e490 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8e46f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b88071670 0x7f8b8814fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8ec70640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b88157c00 0x7f8b880c3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8e46f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b88071670 0x7f8b8814fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:52726/0 (socket says 172.21.15.150:52726) 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8ec70640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b88157c00 0x7f8b880c3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57610/0 (socket says 172.21.15.150:57610) 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8e46f640 1 -- 172.21.15.150:0/1899844314 learned_addr learned my addr 172.21.15.150:0/1899844314 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8dc6e640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b8814e490 0x7f8b88147280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8ec70640 1 -- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b8814e490 msgr2=0x7f8b88147280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8ec70640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b8814e490 0x7f8b88147280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8ec70640 1 -- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b88071670 msgr2=0x7f8b8814fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8ec70640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b88071670 0x7f8b8814fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8ec70640 1 -- 172.21.15.150:0/1899844314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b88152060 con 0x7f8b88157c00 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8e46f640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b88071670 0x7f8b8814fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:14:29.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b8ec70640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b88157c00 0x7f8b880c3a30 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000c700 tx=0x7f8b8000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:29.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b7effd640 1 -- 172.21.15.150:0/1899844314 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b80003b80 con 0x7f8b88157c00 2024-09-17T05:14:29.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b7effd640 1 -- 172.21.15.150:0/1899844314 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b80003d20 con 0x7f8b88157c00 2024-09-17T05:14:29.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b906d3640 1 -- 172.21.15.150:0/1899844314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b88147c00 con 0x7f8b88157c00 2024-09-17T05:14:29.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b7effd640 1 -- 172.21.15.150:0/1899844314 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b80005330 con 0x7f8b88157c00 2024-09-17T05:14:29.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.558+0000 7f8b906d3640 1 -- 172.21.15.150:0/1899844314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b880bf9e0 con 0x7f8b88157c00 2024-09-17T05:14:29.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.562+0000 7f8b906d3640 1 -- 172.21.15.150:0/1899844314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b88072180 con 0x7f8b88157c00 2024-09-17T05:14:29.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.562+0000 7f8b7effd640 1 -- 172.21.15.150:0/1899844314 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8b800054d0 con 0x7f8b88157c00 2024-09-17T05:14:29.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.562+0000 7f8b7effd640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b6c073200 0x7f8b6c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:29.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.562+0000 7f8b7effd640 1 -- 172.21.15.150:0/1899844314 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f8b80094bd0 con 0x7f8b88157c00 2024-09-17T05:14:29.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.562+0000 7f8b8e46f640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b6c073200 0x7f8b6c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:29.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.562+0000 7f8b8e46f640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b6c073200 0x7f8b6c075710 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009850 tx=0x7f8b8401b070 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:29.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:29.570+0000 7f8b7effd640 1 -- 172.21.15.150:0/1899844314 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8b8005e770 con 0x7f8b88157c00 2024-09-17T05:14:30.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.137+0000 7f8b906d3640 1 -- 172.21.15.150:0/1899844314 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b880c9aa0 con 0x7f8b88157c00 2024-09-17T05:14:30.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.137+0000 7f8b7effd640 1 -- 172.21.15.150:0/1899844314 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+5718 (secure 0 0 0) 0x7f8b800631a0 con 0x7f8b88157c00 2024-09-17T05:14:30.137 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:30.137 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":175,"btime":"2024-09-17T05:14:29:347292+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:30.138 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 175 2024-09-17T05:14:30.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.141+0000 7f8b906d3640 1 -- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b6c073200 msgr2=0x7f8b6c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:30.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.141+0000 7f8b906d3640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b6c073200 0x7f8b6c075710 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009850 tx=0x7f8b8401b070 comp rx=0 tx=0).stop 2024-09-17T05:14:30.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.141+0000 7f8b906d3640 1 -- 172.21.15.150:0/1899844314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b88157c00 msgr2=0x7f8b880c3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:30.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.141+0000 7f8b906d3640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b88157c00 0x7f8b880c3a30 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000c700 tx=0x7f8b8000cbc0 comp rx=0 tx=0).stop 2024-09-17T05:14:30.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.141+0000 7f8b906d3640 1 -- 172.21.15.150:0/1899844314 shutdown_connections 2024-09-17T05:14:30.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.141+0000 7f8b906d3640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b6c073200 0x7f8b6c075710 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:30.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.141+0000 7f8b906d3640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b88157c00 0x7f8b880c3a30 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:30.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.141+0000 7f8b906d3640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8b8814e490 0x7f8b88147280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:30.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.141+0000 7f8b906d3640 1 --2- 172.21.15.150:0/1899844314 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b88071670 0x7f8b8814fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:30.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.141+0000 7f8b906d3640 1 -- 172.21.15.150:0/1899844314 >> 172.21.15.150:0/1899844314 conn(0x7f8b880ca460 msgr2=0x7f8b880c7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:30.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.141+0000 7f8b906d3640 1 -- 172.21.15.150:0/1899844314 shutdown_connections 2024-09-17T05:14:30.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.141+0000 7f8b906d3640 1 -- 172.21.15.150:0/1899844314 wait complete. 2024-09-17T05:14:30.163 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=7407 2024-09-17T05:14:30.163 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.c 2024-09-17T05:14:30.163 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-17T05:14:30.163 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T05:14:30.164 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T05:14:30.166 INFO:tasks.ceph.mds.c:Started 2024-09-17T05:14:30.166 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:30.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.405+0000 7f084563f640 1 -- 172.21.15.150:0/222744705 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0840156d90 msgr2=0x7f084014c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:30.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.405+0000 7f084563f640 1 --2- 172.21.15.150:0/222744705 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0840156d90 0x7f084014c940 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0838009f60 tx=0x7f083801ca70 comp rx=0 tx=0).stop 2024-09-17T05:14:30.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.405+0000 7f084563f640 1 -- 172.21.15.150:0/222744705 shutdown_connections 2024-09-17T05:14:30.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.405+0000 7f084563f640 1 --2- 172.21.15.150:0/222744705 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0840156d90 0x7f084014c940 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:30.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.405+0000 7f084563f640 1 --2- 172.21.15.150:0/222744705 >> v2:172.21.15.176:3300/0 conn(0x7f08400c4810 0x7f08400c2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:30.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.405+0000 7f084563f640 1 -- 172.21.15.150:0/222744705 >> 172.21.15.150:0/222744705 conn(0x7f08400c57d0 msgr2=0x7f0840073b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:30.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.405+0000 7f084563f640 1 -- 172.21.15.150:0/222744705 shutdown_connections 2024-09-17T05:14:30.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.405+0000 7f084563f640 1 -- 172.21.15.150:0/222744705 wait complete. 2024-09-17T05:14:30.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.409+0000 7f084563f640 1 Processor -- start 2024-09-17T05:14:30.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.409+0000 7f084563f640 1 -- start start 2024-09-17T05:14:30.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f084563f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f084006fba0 0x7f08401597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:30.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f084563f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08400c4810 0x7f08401448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:30.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f084563f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0840156d90 0x7f0840163a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:30.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f084563f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0840167b90 con 0x7f08400c4810 2024-09-17T05:14:30.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f084563f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0840167a10 con 0x7f0840156d90 2024-09-17T05:14:30.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f084563f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0840167d10 con 0x7f084006fba0 2024-09-17T05:14:30.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f083effd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f084006fba0 0x7f08401597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:30.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f083effd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f084006fba0 0x7f08401597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:35632/0 (socket says 172.21.15.150:35632) 2024-09-17T05:14:30.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f083effd640 1 -- 172.21.15.150:0/22093579 learned_addr learned my addr 172.21.15.150:0/22093579 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:30.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f083ffff640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0840156d90 0x7f0840163a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:30.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f083e7fc640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08400c4810 0x7f08401448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:30.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f083effd640 1 -- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0840156d90 msgr2=0x7f0840163a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:30.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f083effd640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0840156d90 0x7f0840163a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:30.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f083effd640 1 -- 172.21.15.150:0/22093579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08400c4810 msgr2=0x7f08401448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:30.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f083effd640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08400c4810 0x7f08401448e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:30.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f083effd640 1 -- 172.21.15.150:0/22093579 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08401540f0 con 0x7f084006fba0 2024-09-17T05:14:30.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f083e7fc640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08400c4810 0x7f08401448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:14:30.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f083effd640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f084006fba0 0x7f08401597d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f083000c6e0 tx=0x7f083000cba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:30.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f08277fe640 1 -- 172.21.15.150:0/22093579 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0830016020 con 0x7f084006fba0 2024-09-17T05:14:30.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f08277fe640 1 -- 172.21.15.150:0/22093579 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f083000a930 con 0x7f084006fba0 2024-09-17T05:14:30.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f084563f640 1 -- 172.21.15.150:0/22093579 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0840145dc0 con 0x7f084006fba0 2024-09-17T05:14:30.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f08277fe640 1 -- 172.21.15.150:0/22093579 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083000ad80 con 0x7f084006fba0 2024-09-17T05:14:30.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.413+0000 7f084563f640 1 -- 172.21.15.150:0/22093579 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0840153e50 con 0x7f084006fba0 2024-09-17T05:14:30.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.417+0000 7f084563f640 1 -- 172.21.15.150:0/22093579 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08200052a0 con 0x7f084006fba0 2024-09-17T05:14:30.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.421+0000 7f08277fe640 1 -- 172.21.15.150:0/22093579 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0830019020 con 0x7f084006fba0 2024-09-17T05:14:30.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.421+0000 7f08277fe640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f082c073200 0x7f082c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:30.419 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T05:14:30.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.421+0000 7f083e7fc640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f082c073200 0x7f082c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:30.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.421+0000 7f08277fe640 1 -- 172.21.15.150:0/22093579 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f0830095240 con 0x7f084006fba0 2024-09-17T05:14:30.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.421+0000 7f083e7fc640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f082c073200 0x7f082c075710 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0834005fd0 tx=0x7f0834005df0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:30.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:30.425+0000 7f08277fe640 1 -- 172.21.15.150:0/22093579 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0830020020 con 0x7f084006fba0 2024-09-17T05:14:30.858 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T05:14:30.861+0000 7f8557346640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:14:31.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.137+0000 7f084563f640 1 -- 172.21.15.150:0/22093579 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0820005590 con 0x7f084006fba0 2024-09-17T05:14:31.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.137+0000 7f08277fe640 1 -- 172.21.15.150:0/22093579 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+5718 (secure 0 0 0) 0x7f083005f3c0 con 0x7f084006fba0 2024-09-17T05:14:31.138 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:31.138 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":175,"btime":"2024-09-17T05:14:29:347292+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:31.138 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 175 2024-09-17T05:14:31.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.153+0000 7f084563f640 1 -- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f082c073200 msgr2=0x7f082c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:31.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.153+0000 7f084563f640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f082c073200 0x7f082c075710 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0834005fd0 tx=0x7f0834005df0 comp rx=0 tx=0).stop 2024-09-17T05:14:31.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.153+0000 7f084563f640 1 -- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f084006fba0 msgr2=0x7f08401597d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:31.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.153+0000 7f084563f640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f084006fba0 0x7f08401597d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f083000c6e0 tx=0x7f083000cba0 comp rx=0 tx=0).stop 2024-09-17T05:14:31.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.153+0000 7f084563f640 1 -- 172.21.15.150:0/22093579 shutdown_connections 2024-09-17T05:14:31.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.153+0000 7f084563f640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f082c073200 0x7f082c075710 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:31.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.153+0000 7f084563f640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0840156d90 0x7f0840163a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:31.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.153+0000 7f084563f640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08400c4810 0x7f08401448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:31.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.153+0000 7f084563f640 1 --2- 172.21.15.150:0/22093579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f084006fba0 0x7f08401597d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:31.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.153+0000 7f084563f640 1 -- 172.21.15.150:0/22093579 >> 172.21.15.150:0/22093579 conn(0x7f08400c57d0 msgr2=0x7f08401544d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:31.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.153+0000 7f084563f640 1 -- 172.21.15.150:0/22093579 shutdown_connections 2024-09-17T05:14:31.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:31.153+0000 7f084563f640 1 -- 172.21.15.150:0/22093579 wait complete. 2024-09-17T05:14:31.198 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-17T05:14:33.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:33.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.581+0000 7f7023fc0640 1 -- 172.21.15.150:0/1305664562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70140fc570 msgr2=0x7f70140fc980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:33.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.581+0000 7f7023fc0640 1 --2- 172.21.15.150:0/1305664562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70140fc570 0x7f70140fc980 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f701c000b70 tx=0x7f701c088570 comp rx=0 tx=0).stop 2024-09-17T05:14:33.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.581+0000 7f7023fc0640 1 -- 172.21.15.150:0/1305664562 shutdown_connections 2024-09-17T05:14:33.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.581+0000 7f7023fc0640 1 --2- 172.21.15.150:0/1305664562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70140fc570 0x7f70140fc980 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:33.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.581+0000 7f7023fc0640 1 --2- 172.21.15.150:0/1305664562 >> v2:172.21.15.176:3300/0 conn(0x7f701400a880 0x7f701400ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:33.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.581+0000 7f7023fc0640 1 -- 172.21.15.150:0/1305664562 >> 172.21.15.150:0/1305664562 conn(0x7f70140482a0 msgr2=0x7f70140486b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:33.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.581+0000 7f7023fc0640 1 -- 172.21.15.150:0/1305664562 shutdown_connections 2024-09-17T05:14:33.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.581+0000 7f7023fc0640 1 -- 172.21.15.150:0/1305664562 wait complete. 2024-09-17T05:14:33.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.585+0000 7f7023fc0640 1 Processor -- start 2024-09-17T05:14:33.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.585+0000 7f7023fc0640 1 -- start start 2024-09-17T05:14:33.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.589+0000 7f7023fc0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f701400a880 0x7f7014042720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.589+0000 7f7023fc0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7014030f40 0x7f70140f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.597+0000 7f7023fc0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70140fc570 0x7f70141956e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.597+0000 7f7023fc0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f70141094f0 con 0x7f701400a880 2024-09-17T05:14:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.597+0000 7f7023fc0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7014109370 con 0x7f7014030f40 2024-09-17T05:14:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.597+0000 7f7023fc0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7014109670 con 0x7f70140fc570 2024-09-17T05:14:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.597+0000 7f702255d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70140fc570 0x7f70141956e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.597+0000 7f702255d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70140fc570 0x7f70141956e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:35650/0 (socket says 172.21.15.150:35650) 2024-09-17T05:14:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.597+0000 7f702255d640 1 -- 172.21.15.150:0/3778303527 learned_addr learned my addr 172.21.15.150:0/3778303527 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.597+0000 7f702155b640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7014030f40 0x7f70140f4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.597+0000 7f702255d640 1 -- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7014030f40 msgr2=0x7f70140f4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.597+0000 7f702255d640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7014030f40 0x7f70140f4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.597+0000 7f702255d640 1 -- 172.21.15.150:0/3778303527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f701400a880 msgr2=0x7f7014042720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.589+0000 7f7021d5c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f701400a880 0x7f7014042720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.601+0000 7f702255d640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f701400a880 0x7f7014042720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:33.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.601+0000 7f702255d640 1 -- 172.21.15.150:0/3778303527 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70140f9d60 con 0x7f70140fc570 2024-09-17T05:14:33.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.601+0000 7f7021d5c640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f701400a880 0x7f7014042720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:14:33.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.601+0000 7f702255d640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70140fc570 0x7f70141956e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f701c02ebe0 tx=0x7f701c06b490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:33.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.601+0000 7f70127fc640 1 -- 172.21.15.150:0/3778303527 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c11c040 con 0x7f70140fc570 2024-09-17T05:14:33.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.601+0000 7f70127fc640 1 -- 172.21.15.150:0/3778303527 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f701c121480 con 0x7f70140fc570 2024-09-17T05:14:33.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.601+0000 7f7023fc0640 1 -- 172.21.15.150:0/3778303527 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70140f2640 con 0x7f70140fc570 2024-09-17T05:14:33.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.601+0000 7f70127fc640 1 -- 172.21.15.150:0/3778303527 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c06de60 con 0x7f70140fc570 2024-09-17T05:14:33.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.601+0000 7f7023fc0640 1 -- 172.21.15.150:0/3778303527 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f70140efbb0 con 0x7f70140fc570 2024-09-17T05:14:33.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.605+0000 7f7023fc0640 1 -- 172.21.15.150:0/3778303527 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70040052a0 con 0x7f70140fc570 2024-09-17T05:14:33.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.605+0000 7f70127fc640 1 -- 172.21.15.150:0/3778303527 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f701c129030 con 0x7f70140fc570 2024-09-17T05:14:33.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.609+0000 7f70127fc640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7000073200 0x7f7000075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:33.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.609+0000 7f7021d5c640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7000073200 0x7f7000075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:33.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.609+0000 7f70127fc640 1 -- 172.21.15.150:0/3778303527 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f701c19a420 con 0x7f70140fc570 2024-09-17T05:14:33.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.613+0000 7f7021d5c640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7000073200 0x7f7000075710 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f700c001230 tx=0x7f700c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:33.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:33.617+0000 7f70127fc640 1 -- 172.21.15.150:0/3778303527 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f701c130060 con 0x7f70140fc570 2024-09-17T05:14:34.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.333+0000 7f7023fc0640 1 -- 172.21.15.150:0/3778303527 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7004005590 con 0x7f70140fc570 2024-09-17T05:14:34.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.333+0000 7f70127fc640 1 -- 172.21.15.150:0/3778303527 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7f701c082380 con 0x7f70140fc570 2024-09-17T05:14:34.331 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:34.332 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:34.332 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:14:34.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.337+0000 7f7023fc0640 1 -- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7000073200 msgr2=0x7f7000075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:34.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.337+0000 7f7023fc0640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7000073200 0x7f7000075710 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f700c001230 tx=0x7f700c009040 comp rx=0 tx=0).stop 2024-09-17T05:14:34.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.337+0000 7f7023fc0640 1 -- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70140fc570 msgr2=0x7f70141956e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:34.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.337+0000 7f7023fc0640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70140fc570 0x7f70141956e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f701c02ebe0 tx=0x7f701c06b490 comp rx=0 tx=0).stop 2024-09-17T05:14:34.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.337+0000 7f7023fc0640 1 -- 172.21.15.150:0/3778303527 shutdown_connections 2024-09-17T05:14:34.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.337+0000 7f7023fc0640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7000073200 0x7f7000075710 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:34.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.337+0000 7f7023fc0640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7014030f40 0x7f70140f4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:34.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.337+0000 7f7023fc0640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f70140fc570 0x7f70141956e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:34.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.337+0000 7f7023fc0640 1 --2- 172.21.15.150:0/3778303527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f701400a880 0x7f7014042720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:34.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.337+0000 7f7023fc0640 1 -- 172.21.15.150:0/3778303527 >> 172.21.15.150:0/3778303527 conn(0x7f70140482a0 msgr2=0x7f70140fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:34.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.345+0000 7f7023fc0640 1 -- 172.21.15.150:0/3778303527 shutdown_connections 2024-09-17T05:14:34.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:34.345+0000 7f7023fc0640 1 -- 172.21.15.150:0/3778303527 wait complete. 2024-09-17T05:14:34.391 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-17T05:14:34.391 INFO:tasks.mds_thrash.fs.[a]:waiting for 15 secs before thrashing 2024-09-17T05:14:35.114 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:14:35.118 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:14:49.392 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.805+0000 7fc0369d5640 1 -- 172.21.15.150:0/3252833738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc03006fca0 msgr2=0x7fc030153790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.805+0000 7fc0369d5640 1 --2- 172.21.15.150:0/3252833738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc03006fca0 0x7fc030153790 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fc0300c2be0 tx=0x7fc02001db80 comp rx=0 tx=0).stop 2024-09-17T05:14:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.805+0000 7fc0369d5640 1 -- 172.21.15.150:0/3252833738 shutdown_connections 2024-09-17T05:14:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.805+0000 7fc0369d5640 1 --2- 172.21.15.150:0/3252833738 >> v2:172.21.15.176:3300/0 conn(0x7fc030074570 0x7fc0300aaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.805+0000 7fc0369d5640 1 --2- 172.21.15.150:0/3252833738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc03006fca0 0x7fc030153790 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.805+0000 7fc0369d5640 1 -- 172.21.15.150:0/3252833738 >> 172.21.15.150:0/3252833738 conn(0x7fc0300c5ae0 msgr2=0x7fc030073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.805+0000 7fc0369d5640 1 -- 172.21.15.150:0/3252833738 shutdown_connections 2024-09-17T05:14:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.805+0000 7fc0369d5640 1 -- 172.21.15.150:0/3252833738 wait complete. 2024-09-17T05:14:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.805+0000 7fc0369d5640 1 Processor -- start 2024-09-17T05:14:49.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.805+0000 7fc0369d5640 1 -- start start 2024-09-17T05:14:49.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.809+0000 7fc0369d5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc03006fca0 0x7fc03014ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:49.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.809+0000 7fc0369d5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc030074570 0x7fc03014c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:49.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.809+0000 7fc0369d5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc030074980 0x7fc03014e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:49.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.809+0000 7fc0369d5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc0300c4ce0 con 0x7fc030074980 2024-09-17T05:14:49.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.809+0000 7fc02f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc030074570 0x7fc03014c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:49.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.809+0000 7fc02f7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc030074570 0x7fc03014c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:37796/0 (socket says 172.21.15.150:37796) 2024-09-17T05:14:49.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.809+0000 7fc02f7fe640 1 -- 172.21.15.150:0/622395268 learned_addr learned my addr 172.21.15.150:0/622395268 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:49.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.809+0000 7fc0369d5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc03015f6a0 con 0x7fc03006fca0 2024-09-17T05:14:49.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.809+0000 7fc0369d5640 1 -- 172.21.15.150:0/622395268 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc03014fa80 con 0x7fc030074570 2024-09-17T05:14:49.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc02ffff640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc03006fca0 0x7fc03014ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:49.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc02f7fe640 1 -- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc03006fca0 msgr2=0x7fc03014ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:49.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc02f7fe640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc03006fca0 0x7fc03014ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:49.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc02f7fe640 1 -- 172.21.15.150:0/622395268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc030074980 msgr2=0x7fc03014e1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:14:49.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc02f7fe640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc030074980 0x7fc03014e1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:49.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc02f7fe640 1 -- 172.21.15.150:0/622395268 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03014b410 con 0x7fc030074570 2024-09-17T05:14:49.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc02ffff640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc03006fca0 0x7fc03014ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:14:49.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc02f7fe640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc030074570 0x7fc03014c570 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc02400b3f0 tx=0x7fc02400b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:49.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc02cff9640 1 -- 172.21.15.150:0/622395268 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc024015020 con 0x7fc030074570 2024-09-17T05:14:49.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc02cff9640 1 -- 172.21.15.150:0/622395268 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc024011920 con 0x7fc030074570 2024-09-17T05:14:49.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc02cff9640 1 -- 172.21.15.150:0/622395268 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc024011d70 con 0x7fc030074570 2024-09-17T05:14:49.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc0369d5640 1 -- 172.21.15.150:0/622395268 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc03014b6f0 con 0x7fc030074570 2024-09-17T05:14:49.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc0369d5640 1 -- 172.21.15.150:0/622395268 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc03014ec20 con 0x7fc030074570 2024-09-17T05:14:49.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.813+0000 7fc0369d5640 1 -- 172.21.15.150:0/622395268 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc030153790 con 0x7fc030074570 2024-09-17T05:14:49.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.821+0000 7fc02cff9640 1 -- 172.21.15.150:0/622395268 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc024018020 con 0x7fc030074570 2024-09-17T05:14:49.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.821+0000 7fc02cff9640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc01c073200 0x7fc01c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.821+0000 7fc02cff9640 1 -- 172.21.15.150:0/622395268 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7fc0240952c0 con 0x7fc030074570 2024-09-17T05:14:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.821+0000 7fc02ffff640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc01c073200 0x7fc01c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.821+0000 7fc02ffff640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc01c073200 0x7fc01c075710 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc0200b0000 tx=0x7fc02001c3c0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:49.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:49.825+0000 7fc02cff9640 1 -- 172.21.15.150:0/622395268 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc02401f020 con 0x7fc030074570 2024-09-17T05:14:50.351 INFO:tasks.workunit.client.1.smithi176.stdout:fs setup took 1378 secs 2024-09-17T05:14:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.377+0000 7fc0369d5640 1 -- 172.21.15.150:0/622395268 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0300c31e0 con 0x7fc030074570 2024-09-17T05:14:50.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.381+0000 7fc02cff9640 1 -- 172.21.15.150:0/622395268 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7fc02405f550 con 0x7fc030074570 2024-09-17T05:14:50.380 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:50.380 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:50.381 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:14:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.385+0000 7fc0369d5640 1 -- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc01c073200 msgr2=0x7fc01c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.385+0000 7fc0369d5640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc01c073200 0x7fc01c075710 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc0200b0000 tx=0x7fc02001c3c0 comp rx=0 tx=0).stop 2024-09-17T05:14:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.385+0000 7fc0369d5640 1 -- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc030074570 msgr2=0x7fc03014c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.385+0000 7fc0369d5640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc030074570 0x7fc03014c570 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc02400b3f0 tx=0x7fc02400b8b0 comp rx=0 tx=0).stop 2024-09-17T05:14:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.385+0000 7fc0369d5640 1 -- 172.21.15.150:0/622395268 shutdown_connections 2024-09-17T05:14:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.385+0000 7fc0369d5640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc01c073200 0x7fc01c075710 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.385+0000 7fc0369d5640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc030074980 0x7fc03014e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.385+0000 7fc0369d5640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc030074570 0x7fc03014c570 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.385+0000 7fc0369d5640 1 --2- 172.21.15.150:0/622395268 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc03006fca0 0x7fc03014ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.385+0000 7fc0369d5640 1 -- 172.21.15.150:0/622395268 >> 172.21.15.150:0/622395268 conn(0x7fc0300c5ae0 msgr2=0x7fc030068db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.385+0000 7fc0369d5640 1 -- 172.21.15.150:0/622395268 shutdown_connections 2024-09-17T05:14:50.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.385+0000 7fc0369d5640 1 -- 172.21.15.150:0/622395268 wait complete. 2024-09-17T05:14:50.423 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-17T05:14:50.424 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T05:14:50.424 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:14:50.443 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T05:14:50.443 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:50.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.781+0000 7f36244a1640 1 -- 172.21.15.150:0/2088193564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36080051d0 msgr2=0x7f360800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:50.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.781+0000 7f36244a1640 1 --2- 172.21.15.150:0/2088193564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36080051d0 0x7f360800b9d0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f361c074350 tx=0x7f361401d530 comp rx=0 tx=0).stop 2024-09-17T05:14:50.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.781+0000 7f36244a1640 1 -- 172.21.15.150:0/2088193564 shutdown_connections 2024-09-17T05:14:50.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.785+0000 7f36244a1640 1 --2- 172.21.15.150:0/2088193564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36080051d0 0x7f360800b9d0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:50.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.785+0000 7f36244a1640 1 -- 172.21.15.150:0/2088193564 >> 172.21.15.150:0/2088193564 conn(0x7f361c0aab60 msgr2=0x7f361c0aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:50.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.785+0000 7f36244a1640 1 -- 172.21.15.150:0/2088193564 shutdown_connections 2024-09-17T05:14:50.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.785+0000 7f36244a1640 1 -- 172.21.15.150:0/2088193564 wait complete. 2024-09-17T05:14:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.785+0000 7f36244a1640 1 Processor -- start 2024-09-17T05:14:50.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.785+0000 7f36244a1640 1 -- start start 2024-09-17T05:14:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.789+0000 7f36244a1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36080051d0 0x7f361c1e0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:50.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.789+0000 7f36244a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f361c1df7d0 0x7f361c0718a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.789+0000 7f36244a1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f361c1ee060 0x7f361c14e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.789+0000 7f362223d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36080051d0 0x7f361c1e0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.789+0000 7f3621a3c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f361c1df7d0 0x7f361c0718a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.789+0000 7f362223d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36080051d0 0x7f361c1e0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:55776/0 (socket says 172.21.15.150:55776) 2024-09-17T05:14:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.789+0000 7f3621a3c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f361c1df7d0 0x7f361c0718a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48318/0 (socket says 172.21.15.150:48318) 2024-09-17T05:14:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.789+0000 7f36244a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f361c1589d0 con 0x7f361c1df7d0 2024-09-17T05:14:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.789+0000 7f36244a1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f361c072740 con 0x7f36080051d0 2024-09-17T05:14:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.789+0000 7f36244a1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f361c06d800 con 0x7f361c1ee060 2024-09-17T05:14:50.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.789+0000 7f362223d640 1 -- 172.21.15.150:0/3001612854 learned_addr learned my addr 172.21.15.150:0/3001612854 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.789+0000 7f3622a3e640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f361c1ee060 0x7f361c14e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:50.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.793+0000 7f362223d640 1 -- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f361c1ee060 msgr2=0x7f361c14e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.793+0000 7f362223d640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f361c1ee060 0x7f361c14e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.793+0000 7f362223d640 1 -- 172.21.15.150:0/3001612854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f361c1df7d0 msgr2=0x7f361c0718a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.793+0000 7f362223d640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f361c1df7d0 0x7f361c0718a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:50.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.793+0000 7f362223d640 1 -- 172.21.15.150:0/3001612854 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f361c1ed8d0 con 0x7f36080051d0 2024-09-17T05:14:50.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.797+0000 7f362223d640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36080051d0 0x7f361c1e0750 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f361800c770 tx=0x7f361800cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:50.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.797+0000 7f3612ffd640 1 -- 172.21.15.150:0/3001612854 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3618005480 con 0x7f36080051d0 2024-09-17T05:14:50.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.797+0000 7f36244a1640 1 -- 172.21.15.150:0/3001612854 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f361c1ea2c0 con 0x7f36080051d0 2024-09-17T05:14:50.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.801+0000 7f36244a1640 1 -- 172.21.15.150:0/3001612854 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f361c1e16f0 con 0x7f36080051d0 2024-09-17T05:14:50.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.801+0000 7f36244a1640 1 -- 172.21.15.150:0/3001612854 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f361c0c7fc0 con 0x7f36080051d0 2024-09-17T05:14:50.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.801+0000 7f3612ffd640 1 -- 172.21.15.150:0/3001612854 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f361800ce80 con 0x7f36080051d0 2024-09-17T05:14:50.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.801+0000 7f3612ffd640 1 -- 172.21.15.150:0/3001612854 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3618003840 con 0x7f36080051d0 2024-09-17T05:14:50.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.809+0000 7f3612ffd640 1 -- 172.21.15.150:0/3001612854 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f36180039e0 con 0x7f36080051d0 2024-09-17T05:14:50.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.809+0000 7f3612ffd640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3600073200 0x7f3600075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:50.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.809+0000 7f3612ffd640 1 -- 172.21.15.150:0/3001612854 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f3618095900 con 0x7f36080051d0 2024-09-17T05:14:50.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.809+0000 7f3621a3c640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3600073200 0x7f3600075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:50.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.809+0000 7f3621a3c640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3600073200 0x7f3600075710 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f360c0046c0 tx=0x7f360c008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:50.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:50.813+0000 7f3612ffd640 1 -- 172.21.15.150:0/3001612854 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f361805fb60 con 0x7f36080051d0 2024-09-17T05:14:51.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.513+0000 7f36244a1640 1 -- 172.21.15.150:0/3001612854 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f361c06fa10 con 0x7f36080051d0 2024-09-17T05:14:51.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.513+0000 7f3612ffd640 1 -- 172.21.15.150:0/3001612854 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7f361805f460 con 0x7f36080051d0 2024-09-17T05:14:51.512 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:51.512 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:51.513 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:14:51.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.517+0000 7f36244a1640 1 -- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3600073200 msgr2=0x7f3600075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:51.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.517+0000 7f36244a1640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3600073200 0x7f3600075710 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f360c0046c0 tx=0x7f360c008040 comp rx=0 tx=0).stop 2024-09-17T05:14:51.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.517+0000 7f36244a1640 1 -- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36080051d0 msgr2=0x7f361c1e0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:51.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.517+0000 7f36244a1640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36080051d0 0x7f361c1e0750 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f361800c770 tx=0x7f361800cc30 comp rx=0 tx=0).stop 2024-09-17T05:14:51.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.517+0000 7f36244a1640 1 -- 172.21.15.150:0/3001612854 shutdown_connections 2024-09-17T05:14:51.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.517+0000 7f36244a1640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3600073200 0x7f3600075710 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:51.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.517+0000 7f36244a1640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f361c1ee060 0x7f361c14e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:51.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.517+0000 7f36244a1640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f361c1df7d0 0x7f361c0718a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:51.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.517+0000 7f36244a1640 1 --2- 172.21.15.150:0/3001612854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f36080051d0 0x7f361c1e0750 secure :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f361800c770 tx=0x7f361800cc30 comp rx=0 tx=0).stop 2024-09-17T05:14:51.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.517+0000 7f36244a1640 1 -- 172.21.15.150:0/3001612854 >> 172.21.15.150:0/3001612854 conn(0x7f361c0aab60 msgr2=0x7f361c0c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:51.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.517+0000 7f36244a1640 1 -- 172.21.15.150:0/3001612854 shutdown_connections 2024-09-17T05:14:51.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:51.517+0000 7f36244a1640 1 -- 172.21.15.150:0/3001612854 wait complete. 2024-09-17T05:14:51.546 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:14:53.302 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:53.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.545+0000 7fc819c92640 1 -- 172.21.15.150:0/2906754885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc81414e490 msgr2=0x7fc8140c7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:53.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.545+0000 7fc819c92640 1 --2- 172.21.15.150:0/2906754885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc81414e490 0x7fc8140c7d60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc804009840 tx=0x7fc80401c820 comp rx=0 tx=0).stop 2024-09-17T05:14:53.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.545+0000 7fc819c92640 1 -- 172.21.15.150:0/2906754885 shutdown_connections 2024-09-17T05:14:53.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.545+0000 7fc819c92640 1 --2- 172.21.15.150:0/2906754885 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc81414e490 0x7fc8140c7d60 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:53.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.545+0000 7fc819c92640 1 --2- 172.21.15.150:0/2906754885 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc814071670 0x7fc8140c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:53.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.545+0000 7fc819c92640 1 -- 172.21.15.150:0/2906754885 >> 172.21.15.150:0/2906754885 conn(0x7fc8140ca460 msgr2=0x7fc8140c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:53.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.549+0000 7fc819c92640 1 -- 172.21.15.150:0/2906754885 shutdown_connections 2024-09-17T05:14:53.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.549+0000 7fc819c92640 1 -- 172.21.15.150:0/2906754885 wait complete. 2024-09-17T05:14:53.547 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:53.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.549+0000 7fc819c92640 1 Processor -- start 2024-09-17T05:14:53.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.549+0000 7fc819c92640 1 -- start start 2024-09-17T05:14:53.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc819c92640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc814071670 0x7fc8140beea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:53.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc819c92640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81414e490 0x7fc8140c3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:53.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc812ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81414e490 0x7fc8140c3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:53.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc812ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81414e490 0x7fc8140c3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48326/0 (socket says 172.21.15.150:48326) 2024-09-17T05:14:53.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc812ffd640 1 -- 172.21.15.150:0/2385324372 learned_addr learned my addr 172.21.15.150:0/2385324372 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:53.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc819c92640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc814157c00 0x7fc8140c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:53.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc8140c6740 con 0x7fc81414e490 2024-09-17T05:14:53.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc8140c65c0 con 0x7fc814071670 2024-09-17T05:14:53.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc81414e8a0 con 0x7fc814157c00 2024-09-17T05:14:53.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc812ffd640 1 -- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc814157c00 msgr2=0x7fc8140c22b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:14:53.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc812ffd640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc814157c00 0x7fc8140c22b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:53.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc812ffd640 1 -- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc814071670 msgr2=0x7fc8140beea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:53.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc812ffd640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc814071670 0x7fc8140beea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:53.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc812ffd640 1 -- 172.21.15.150:0/2385324372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8140c3de0 con 0x7fc81414e490 2024-09-17T05:14:53.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc812ffd640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81414e490 0x7fc8140c3120 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fc80401aed0 tx=0x7fc8040a6620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:53.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2385324372 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8040bac90 con 0x7fc81414e490 2024-09-17T05:14:53.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2385324372 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc804004030 con 0x7fc81414e490 2024-09-17T05:14:53.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8140c3b00 con 0x7fc81414e490 2024-09-17T05:14:53.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2385324372 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804004480 con 0x7fc81414e490 2024-09-17T05:14:53.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.553+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc814156040 con 0x7fc81414e490 2024-09-17T05:14:53.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.557+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8140722f0 con 0x7fc81414e490 2024-09-17T05:14:53.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.557+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2385324372 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc8040ba120 con 0x7fc81414e490 2024-09-17T05:14:53.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.557+0000 7fc7fbfff640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc800084210 0x7fc800086720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:53.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.557+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2385324372 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7fc80412f5b0 con 0x7fc81414e490 2024-09-17T05:14:53.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.561+0000 7fc8137fe640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc800084210 0x7fc800086720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:53.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.565+0000 7fc8137fe640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc800084210 0x7fc800086720 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc8080058d0 tx=0x7fc808005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:53.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:53.593+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2385324372 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc804016410 con 0x7fc81414e490 2024-09-17T05:14:54.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.001+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/1277570874 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f940da1a0 msgr2=0x7f4f940d4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:54.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.001+0000 7f4fa1c7c640 1 --2- 172.21.15.150:0/1277570874 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f940da1a0 0x7f4f940d4250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009870 tx=0x7f4f9001c710 comp rx=0 tx=0).stop 2024-09-17T05:14:54.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.001+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/1277570874 shutdown_connections 2024-09-17T05:14:54.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.001+0000 7f4fa1c7c640 1 --2- 172.21.15.150:0/1277570874 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f940da1a0 0x7f4f940d4250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.001+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/1277570874 >> 172.21.15.150:0/1277570874 conn(0x7f4f940482a0 msgr2=0x7f4f940486b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:54.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.001+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/1277570874 shutdown_connections 2024-09-17T05:14:54.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.001+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/1277570874 wait complete. 2024-09-17T05:14:54.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.005+0000 7f4fa1c7c640 1 Processor -- start 2024-09-17T05:14:54.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.005+0000 7f4fa1c7c640 1 -- start start 2024-09-17T05:14:54.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4fa1c7c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f94031140 0x7f4f940e7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:54.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4fa1c7c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f940467b0 0x7f4f9417b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:54.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4fa1c7c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f940da1a0 0x7f4f9417b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:54.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4fa1c7c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9400af90 con 0x7f4f940da1a0 2024-09-17T05:14:54.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4fa1c7c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4f94045ec0 con 0x7f4f940467b0 2024-09-17T05:14:54.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4fa1c7c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4f9400b110 con 0x7f4f94031140 2024-09-17T05:14:54.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4f9b7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f94031140 0x7f4f940e7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:54.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4f9b7fe640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f94031140 0x7f4f940e7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:37824/0 (socket says 172.21.15.150:37824) 2024-09-17T05:14:54.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4f9b7fe640 1 -- 172.21.15.150:0/2309215591 learned_addr learned my addr 172.21.15.150:0/2309215591 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:54.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4f9b7fe640 1 -- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f940467b0 msgr2=0x7f4f9417b0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:14:54.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4f9b7fe640 1 --2- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f940467b0 0x7f4f9417b0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4f9b7fe640 1 -- 172.21.15.150:0/2309215591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f940da1a0 msgr2=0x7f4f9417b5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:14:54.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4f9b7fe640 1 --2- 172.21.15.150:0/2309215591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f940da1a0 0x7f4f9417b5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4f9b7fe640 1 -- 172.21.15.150:0/2309215591 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9402d690 con 0x7f4f94031140 2024-09-17T05:14:54.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4f9b7fe640 1 --2- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f94031140 0x7f4f940e7600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c06add0 tx=0x7f4f9c076ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:54.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.009+0000 7f4f87fff640 1 -- 172.21.15.150:0/2309215591 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c091070 con 0x7f4f94031140 2024-09-17T05:14:54.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.013+0000 7f4f87fff640 1 -- 172.21.15.150:0/2309215591 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f9c08d580 con 0x7f4f94031140 2024-09-17T05:14:54.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.013+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/2309215591 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f9417bb20 con 0x7f4f94031140 2024-09-17T05:14:54.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.013+0000 7f4f87fff640 1 -- 172.21.15.150:0/2309215591 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c08da50 con 0x7f4f94031140 2024-09-17T05:14:54.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.013+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/2309215591 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f9402ae20 con 0x7f4f94031140 2024-09-17T05:14:54.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.013+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/2309215591 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f9400a880 con 0x7f4f94031140 2024-09-17T05:14:54.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.017+0000 7f4f87fff640 1 -- 172.21.15.150:0/2309215591 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4f9c094070 con 0x7f4f94031140 2024-09-17T05:14:54.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.017+0000 7f4f87fff640 1 --2- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f800732d0 0x7f4f800757e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:54.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.017+0000 7f4f87fff640 1 -- 172.21.15.150:0/2309215591 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f4f9c111520 con 0x7f4f94031140 2024-09-17T05:14:54.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.021+0000 7f4f9affd640 1 --2- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f800732d0 0x7f4f800757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:54.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.021+0000 7f4f9affd640 1 --2- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f800732d0 0x7f4f800757e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4f900af000 tx=0x7f4f9001af50 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:54.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.021+0000 7f4f87fff640 1 -- 172.21.15.150:0/2309215591 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4f9c0db720 con 0x7f4f94031140 2024-09-17T05:14:54.139 INFO:tasks.workunit.client.1.smithi176.stdout:Syncing()...3 sec 2024-09-17T05:14:54.140 INFO:tasks.workunit.client.1.smithi176.stdout:Starting Actual Benchmark At: Tue Sep 17 05:14:54 2024 2024-09-17T05:14:54.140 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T05:14:54.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.313+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8140c7da0 con 0x7fc81414e490 2024-09-17T05:14:54.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.317+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2385324372 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7fc8040f9820 con 0x7fc81414e490 2024-09-17T05:14:54.315 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:54.315 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:54.316 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:14:54.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.321+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc800084210 msgr2=0x7fc800086720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:54.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.321+0000 7fc819c92640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc800084210 0x7fc800086720 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc8080058d0 tx=0x7fc808005860 comp rx=0 tx=0).stop 2024-09-17T05:14:54.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.321+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81414e490 msgr2=0x7fc8140c3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:54.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.321+0000 7fc819c92640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81414e490 0x7fc8140c3120 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fc80401aed0 tx=0x7fc8040a6620 comp rx=0 tx=0).stop 2024-09-17T05:14:54.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.321+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 shutdown_connections 2024-09-17T05:14:54.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.321+0000 7fc819c92640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc800084210 0x7fc800086720 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.321+0000 7fc819c92640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc814157c00 0x7fc8140c22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.321+0000 7fc819c92640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81414e490 0x7fc8140c3120 secure :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fc80401aed0 tx=0x7fc8040a6620 comp rx=0 tx=0).stop 2024-09-17T05:14:54.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.321+0000 7fc819c92640 1 --2- 172.21.15.150:0/2385324372 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc814071670 0x7fc8140beea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.321+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 >> 172.21.15.150:0/2385324372 conn(0x7fc8140ca460 msgr2=0x7fc814154490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:54.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.321+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 shutdown_connections 2024-09-17T05:14:54.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.321+0000 7fc819c92640 1 -- 172.21.15.150:0/2385324372 wait complete. 2024-09-17T05:14:54.390 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-17T05:14:54.390 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T05:14:54.390 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:14:54.415 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T05:14:54.415 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:54.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.817+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/2309215591 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f94046590 con 0x7f4f94031140 2024-09-17T05:14:54.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.821+0000 7f4f87fff640 1 -- 172.21.15.150:0/2309215591 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7f4f9c0db020 con 0x7f4f94031140 2024-09-17T05:14:54.819 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:54.819 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:54.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.821+0000 7f7c94e37640 1 -- 172.21.15.150:0/2534045709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c880d6a10 msgr2=0x7f7c880d6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:54.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.821+0000 7f7c94e37640 1 --2- 172.21.15.150:0/2534045709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c880d6a10 0x7f7c880d6e20 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7c9006b7b0 tx=0x7f7c9009a9f0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.819 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:14:54.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.821+0000 7f7c94e37640 1 -- 172.21.15.150:0/2534045709 shutdown_connections 2024-09-17T05:14:54.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.821+0000 7f7c94e37640 1 --2- 172.21.15.150:0/2534045709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c880d6a10 0x7f7c880d6e20 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.821+0000 7f7c94e37640 1 -- 172.21.15.150:0/2534045709 >> 172.21.15.150:0/2534045709 conn(0x7f7c88048b10 msgr2=0x7f7c88048f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:54.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.821+0000 7f7c94e37640 1 -- 172.21.15.150:0/2534045709 shutdown_connections 2024-09-17T05:14:54.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.821+0000 7f7c94e37640 1 -- 172.21.15.150:0/2534045709 wait complete. 2024-09-17T05:14:54.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.825+0000 7f7c94e37640 1 Processor -- start 2024-09-17T05:14:54.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.825+0000 7f7c94e37640 1 -- start start 2024-09-17T05:14:54.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.825+0000 7f7c94e37640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c88047020 0x7f7c880d3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:54.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.825+0000 7f7c94e37640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c880d3ff0 0x7f7c8816fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:54.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.825+0000 7f7c94e37640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c880d6a10 0x7f7c880c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:54.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.825+0000 7f7c94e37640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7c880e4330 con 0x7f7c880d6a10 2024-09-17T05:14:54.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.825+0000 7f7c94e37640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7c880e41b0 con 0x7f7c880d3ff0 2024-09-17T05:14:54.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.825+0000 7f7c94e37640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7c880e44b0 con 0x7f7c88047020 2024-09-17T05:14:54.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.825+0000 7f7c8e59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c88047020 0x7f7c880d3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:54.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.829+0000 7f7c8e59c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c88047020 0x7f7c880d3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:37832/0 (socket says 172.21.15.150:37832) 2024-09-17T05:14:54.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.829+0000 7f7c8e59c640 1 -- 172.21.15.150:0/1776853538 learned_addr learned my addr 172.21.15.150:0/1776853538 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:54.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.829+0000 7f7c8e59c640 1 -- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c880d3ff0 msgr2=0x7f7c8816fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:54.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.829+0000 7f7c8e59c640 1 --2- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c880d3ff0 0x7f7c8816fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.829+0000 7f7c8e59c640 1 -- 172.21.15.150:0/1776853538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c880d6a10 msgr2=0x7f7c880c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:54.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.829+0000 7f7c8e59c640 1 --2- 172.21.15.150:0/1776853538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c880d6a10 0x7f7c880c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.829+0000 7f7c8e59c640 1 -- 172.21.15.150:0/1776853538 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c880cb380 con 0x7f7c88047020 2024-09-17T05:14:54.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.829+0000 7f7c8e59c640 1 --2- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c88047020 0x7f7c880d3160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000c430 tx=0x7f7c8000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:54.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.833+0000 7f7c7effd640 1 -- 172.21.15.150:0/1776853538 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c80011790 con 0x7f7c88047020 2024-09-17T05:14:54.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.833+0000 7f7c7effd640 1 -- 172.21.15.150:0/1776853538 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c80017070 con 0x7f7c88047020 2024-09-17T05:14:54.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.833+0000 7f7c7effd640 1 -- 172.21.15.150:0/1776853538 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c80011cc0 con 0x7f7c88047020 2024-09-17T05:14:54.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.833+0000 7f7c94e37640 1 -- 172.21.15.150:0/1776853538 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c8802bf70 con 0x7f7c88047020 2024-09-17T05:14:54.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.833+0000 7f7c94e37640 1 -- 172.21.15.150:0/1776853538 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c88045920 con 0x7f7c88047020 2024-09-17T05:14:54.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.837+0000 7f7c94e37640 1 -- 172.21.15.150:0/1776853538 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c880d6e20 con 0x7f7c88047020 2024-09-17T05:14:54.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.837+0000 7f7c7effd640 1 -- 172.21.15.150:0/1776853538 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7c8001c460 con 0x7f7c88047020 2024-09-17T05:14:54.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.837+0000 7f7c7effd640 1 --2- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c70075440 0x7f7c70077950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:54.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.837+0000 7f7c7effd640 1 -- 172.21.15.150:0/1776853538 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f7c80094e30 con 0x7f7c88047020 2024-09-17T05:14:54.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.837+0000 7f7c8dd9b640 1 --2- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c70075440 0x7f7c70077950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:54.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.837+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f800732d0 msgr2=0x7f4f800757e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:54.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.837+0000 7f4fa1c7c640 1 --2- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f800732d0 0x7f4f800757e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4f900af000 tx=0x7f4f9001af50 comp rx=0 tx=0).stop 2024-09-17T05:14:54.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.837+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f94031140 msgr2=0x7f4f940e7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:54.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.837+0000 7f4fa1c7c640 1 --2- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f94031140 0x7f4f940e7600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c06add0 tx=0x7f4f9c076ea0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.841+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/2309215591 shutdown_connections 2024-09-17T05:14:54.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.841+0000 7f4fa1c7c640 1 --2- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4f800732d0 0x7f4f800757e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.841+0000 7f4fa1c7c640 1 --2- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f940467b0 0x7f4f9417b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.841+0000 7f4fa1c7c640 1 --2- 172.21.15.150:0/2309215591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f940da1a0 0x7f4f9417b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.841+0000 7f4fa1c7c640 1 --2- 172.21.15.150:0/2309215591 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4f94031140 0x7f4f940e7600 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:54.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.841+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/2309215591 >> 172.21.15.150:0/2309215591 conn(0x7f4f940482a0 msgr2=0x7f4f94005b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:54.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.841+0000 7f7c8dd9b640 1 --2- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c70075440 0x7f7c70077950 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7c84007fd0 tx=0x7f7c84007980 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:54.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.841+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/2309215591 shutdown_connections 2024-09-17T05:14:54.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.845+0000 7f4fa1c7c640 1 -- 172.21.15.150:0/2309215591 wait complete. 2024-09-17T05:14:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:54.857+0000 7f7c7effd640 1 -- 172.21.15.150:0/1776853538 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7c8005f010 con 0x7f7c88047020 2024-09-17T05:14:54.901 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:14:55.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.413+0000 7f7c94e37640 1 -- 172.21.15.150:0/1776853538 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c8804ad90 con 0x7f7c88047020 2024-09-17T05:14:55.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.417+0000 7f7c7effd640 1 -- 172.21.15.150:0/1776853538 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7f7c8005e910 con 0x7f7c88047020 2024-09-17T05:14:55.422 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:55.422 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:55.422 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:14:55.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.425+0000 7f7c94e37640 1 -- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c70075440 msgr2=0x7f7c70077950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:55.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.425+0000 7f7c94e37640 1 --2- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c70075440 0x7f7c70077950 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7c84007fd0 tx=0x7f7c84007980 comp rx=0 tx=0).stop 2024-09-17T05:14:55.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.425+0000 7f7c94e37640 1 -- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c88047020 msgr2=0x7f7c880d3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:55.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.425+0000 7f7c94e37640 1 --2- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c88047020 0x7f7c880d3160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000c430 tx=0x7f7c8000c8f0 comp rx=0 tx=0).stop 2024-09-17T05:14:55.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.425+0000 7f7c94e37640 1 -- 172.21.15.150:0/1776853538 shutdown_connections 2024-09-17T05:14:55.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.425+0000 7f7c94e37640 1 --2- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7c70075440 0x7f7c70077950 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:55.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.425+0000 7f7c94e37640 1 --2- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c880d3ff0 0x7f7c8816fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:55.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.425+0000 7f7c94e37640 1 --2- 172.21.15.150:0/1776853538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c880d6a10 0x7f7c880c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:55.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.425+0000 7f7c94e37640 1 --2- 172.21.15.150:0/1776853538 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7c88047020 0x7f7c880d3160 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:55.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.425+0000 7f7c94e37640 1 -- 172.21.15.150:0/1776853538 >> 172.21.15.150:0/1776853538 conn(0x7f7c88048b10 msgr2=0x7f7c8802f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:55.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.425+0000 7f7c94e37640 1 -- 172.21.15.150:0/1776853538 shutdown_connections 2024-09-17T05:14:55.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:55.425+0000 7f7c94e37640 1 -- 172.21.15.150:0/1776853538 wait complete. 2024-09-17T05:14:55.469 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:14:56.902 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:57.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.093+0000 7fcf4273f640 1 -- 172.21.15.150:0/3863604152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf3c071670 msgr2=0x7fcf3c0c5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:57.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.093+0000 7fcf4273f640 1 --2- 172.21.15.150:0/3863604152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf3c071670 0x7fcf3c0c5680 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009900 tx=0x7fcf2c01c900 comp rx=0 tx=0).stop 2024-09-17T05:14:57.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.093+0000 7fcf4273f640 1 -- 172.21.15.150:0/3863604152 shutdown_connections 2024-09-17T05:14:57.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.093+0000 7fcf4273f640 1 --2- 172.21.15.150:0/3863604152 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf3c14e490 0x7fcf3c0c7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.093+0000 7fcf4273f640 1 --2- 172.21.15.150:0/3863604152 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf3c071670 0x7fcf3c0c5680 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.093+0000 7fcf4273f640 1 -- 172.21.15.150:0/3863604152 >> 172.21.15.150:0/3863604152 conn(0x7fcf3c0ca460 msgr2=0x7fcf3c0c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:57.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.093+0000 7fcf4273f640 1 -- 172.21.15.150:0/3863604152 shutdown_connections 2024-09-17T05:14:57.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.093+0000 7fcf4273f640 1 -- 172.21.15.150:0/3863604152 wait complete. 2024-09-17T05:14:57.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf4273f640 1 Processor -- start 2024-09-17T05:14:57.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf4273f640 1 -- start start 2024-09-17T05:14:57.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf4273f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf3c071670 0x7fcf3c0c11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:57.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf4273f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf3c14e490 0x7fcf3c150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:57.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf4273f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf3c157c00 0x7fcf3c0c3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:57.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf3c071670 0x7fcf3c0c11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:57.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf3c071670 0x7fcf3c0c11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48352/0 (socket says 172.21.15.150:48352) 2024-09-17T05:14:57.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf3bfff640 1 -- 172.21.15.150:0/179544331 learned_addr learned my addr 172.21.15.150:0/179544331 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:57.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3c0c6740 con 0x7fcf3c071670 2024-09-17T05:14:57.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3c0c65c0 con 0x7fcf3c157c00 2024-09-17T05:14:57.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcf3c14e8a0 con 0x7fcf3c14e490 2024-09-17T05:14:57.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf337fe640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf3c14e490 0x7fcf3c150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:57.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.097+0000 7fcf40cdc640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf3c157c00 0x7fcf3c0c3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:57.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf3bfff640 1 -- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf3c14e490 msgr2=0x7fcf3c150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:57.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf3bfff640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf3c14e490 0x7fcf3c150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf3bfff640 1 -- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf3c157c00 msgr2=0x7fcf3c0c3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:57.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf3bfff640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf3c157c00 0x7fcf3c0c3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf3bfff640 1 -- 172.21.15.150:0/179544331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf3c0c1710 con 0x7fcf3c071670 2024-09-17T05:14:57.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf3bfff640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf3c071670 0x7fcf3c0c11d0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009850 tx=0x7fcf2c0a5650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:57.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf397fa640 1 -- 172.21.15.150:0/179544331 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c0b2040 con 0x7fcf3c071670 2024-09-17T05:14:57.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf397fa640 1 -- 172.21.15.150:0/179544331 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf2c004430 con 0x7fcf3c071670 2024-09-17T05:14:57.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf397fa640 1 -- 172.21.15.150:0/179544331 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c003860 con 0x7fcf3c071670 2024-09-17T05:14:57.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf3c0c3f90 con 0x7fcf3c071670 2024-09-17T05:14:57.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf3c0c2800 con 0x7fcf3c071670 2024-09-17T05:14:57.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf3c072180 con 0x7fcf3c071670 2024-09-17T05:14:57.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.101+0000 7fcf397fa640 1 -- 172.21.15.150:0/179544331 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fcf2c0bd060 con 0x7fcf3c071670 2024-09-17T05:14:57.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.105+0000 7fcf397fa640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf1c073200 0x7fcf1c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:57.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.105+0000 7fcf397fa640 1 -- 172.21.15.150:0/179544331 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7fcf2c12e940 con 0x7fcf3c071670 2024-09-17T05:14:57.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.105+0000 7fcf337fe640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf1c073200 0x7fcf1c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:57.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.105+0000 7fcf337fe640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf1c073200 0x7fcf1c075710 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c14c670 tx=0x7fcf24006f90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:57.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.109+0000 7fcf397fa640 1 -- 172.21.15.150:0/179544331 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcf2c0164f0 con 0x7fcf3c071670 2024-09-17T05:14:57.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.409+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf3c070ac0 con 0x7fcf3c071670 2024-09-17T05:14:57.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.409+0000 7fcf397fa640 1 -- 172.21.15.150:0/179544331 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7fcf2c0f8560 con 0x7fcf3c071670 2024-09-17T05:14:57.409 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:57.413 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:57.414 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:14:57.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.413+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf1c073200 msgr2=0x7fcf1c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:57.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.413+0000 7fcf4273f640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf1c073200 0x7fcf1c075710 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c14c670 tx=0x7fcf24006f90 comp rx=0 tx=0).stop 2024-09-17T05:14:57.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.413+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf3c071670 msgr2=0x7fcf3c0c11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:57.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.413+0000 7fcf4273f640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf3c071670 0x7fcf3c0c11d0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009850 tx=0x7fcf2c0a5650 comp rx=0 tx=0).stop 2024-09-17T05:14:57.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.413+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 shutdown_connections 2024-09-17T05:14:57.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.413+0000 7fcf4273f640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcf1c073200 0x7fcf1c075710 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.413+0000 7fcf4273f640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcf3c157c00 0x7fcf3c0c3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.413+0000 7fcf4273f640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcf3c14e490 0x7fcf3c150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.413+0000 7fcf4273f640 1 --2- 172.21.15.150:0/179544331 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf3c071670 0x7fcf3c0c11d0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.413+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 >> 172.21.15.150:0/179544331 conn(0x7fcf3c0ca460 msgr2=0x7fcf3c154490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:57.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.413+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 shutdown_connections 2024-09-17T05:14:57.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.413+0000 7fcf4273f640 1 -- 172.21.15.150:0/179544331 wait complete. 2024-09-17T05:14:57.443 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:14:57.470 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:57.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.661+0000 7f300cd16640 1 -- 172.21.15.150:0/446997245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3008154f90 msgr2=0x7f300806d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:57.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.661+0000 7f300cd16640 1 --2- 172.21.15.150:0/446997245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3008154f90 0x7f300806d560 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f3000009bc0 tx=0x7f300001cb10 comp rx=0 tx=0).stop 2024-09-17T05:14:57.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.661+0000 7f300cd16640 1 -- 172.21.15.150:0/446997245 shutdown_connections 2024-09-17T05:14:57.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.661+0000 7f300cd16640 1 --2- 172.21.15.150:0/446997245 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3008151f30 0x7f30080c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.661+0000 7f300cd16640 1 --2- 172.21.15.150:0/446997245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3008154f90 0x7f300806d560 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.661+0000 7f300cd16640 1 -- 172.21.15.150:0/446997245 >> 172.21.15.150:0/446997245 conn(0x7f30080c4e30 msgr2=0x7f300806ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:57.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.661+0000 7f300cd16640 1 -- 172.21.15.150:0/446997245 shutdown_connections 2024-09-17T05:14:57.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.661+0000 7f300cd16640 1 -- 172.21.15.150:0/446997245 wait complete. 2024-09-17T05:14:57.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f300cd16640 1 Processor -- start 2024-09-17T05:14:57.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f300cd16640 1 -- start start 2024-09-17T05:14:57.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f300cd16640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3008151f30 0x7f30081f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:57.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f300cd16640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3008154f90 0x7f3008149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:57.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f300cd16640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30081ee6c0 0x7f300814f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:57.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f300cd16640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3008161700 con 0x7f30081ee6c0 2024-09-17T05:14:57.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f300cd16640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f30081553a0 con 0x7f3008151f30 2024-09-17T05:14:57.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f300cd16640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3008161880 con 0x7f3008154f90 2024-09-17T05:14:57.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f3006d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30081ee6c0 0x7f300814f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:57.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f3006d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30081ee6c0 0x7f300814f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48370/0 (socket says 172.21.15.150:48370) 2024-09-17T05:14:57.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f3006d9d640 1 -- 172.21.15.150:0/1755152480 learned_addr learned my addr 172.21.15.150:0/1755152480 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:57.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f3006d9d640 1 -- 172.21.15.150:0/1755152480 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3008154f90 msgr2=0x7f3008149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:57.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f3006d9d640 1 --2- 172.21.15.150:0/1755152480 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3008154f90 0x7f3008149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f3006d9d640 1 -- 172.21.15.150:0/1755152480 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3008151f30 msgr2=0x7f30081f1060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:14:57.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f3006d9d640 1 --2- 172.21.15.150:0/1755152480 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3008151f30 0x7f30081f1060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f3006d9d640 1 -- 172.21.15.150:0/1755152480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30081558c0 con 0x7f30081ee6c0 2024-09-17T05:14:57.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.665+0000 7f3006d9d640 1 --2- 172.21.15.150:0/1755152480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30081ee6c0 0x7f300814f3b0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f30000b0040 tx=0x7f3000009330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:57.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.669+0000 7f2ffeffd640 1 -- 172.21.15.150:0/1755152480 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30000bc070 con 0x7f30081ee6c0 2024-09-17T05:14:57.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.669+0000 7f2ffeffd640 1 -- 172.21.15.150:0/1755152480 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30000042f0 con 0x7f30081ee6c0 2024-09-17T05:14:57.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.669+0000 7f300cd16640 1 -- 172.21.15.150:0/1755152480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30081f0860 con 0x7f30081ee6c0 2024-09-17T05:14:57.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.669+0000 7f2ffeffd640 1 -- 172.21.15.150:0/1755152480 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30000b2040 con 0x7f30081ee6c0 2024-09-17T05:14:57.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.669+0000 7f300cd16640 1 -- 172.21.15.150:0/1755152480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3008155ba0 con 0x7f30081ee6c0 2024-09-17T05:14:57.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.669+0000 7f2ffeffd640 1 -- 172.21.15.150:0/1755152480 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f30000bf070 con 0x7f30081ee6c0 2024-09-17T05:14:57.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.669+0000 7f300cd16640 1 -- 172.21.15.150:0/1755152480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30080c3440 con 0x7f30081ee6c0 2024-09-17T05:14:57.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.669+0000 7f2ffeffd640 1 --2- 172.21.15.150:0/1755152480 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2fe8073280 0x7f2fe8075790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:57.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.669+0000 7f2ffeffd640 1 -- 172.21.15.150:0/1755152480 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f3000132170 con 0x7f30081ee6c0 2024-09-17T05:14:57.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.673+0000 7f300659c640 1 --2- 172.21.15.150:0/1755152480 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2fe8073280 0x7f2fe8075790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:57.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.673+0000 7f300659c640 1 --2- 172.21.15.150:0/1755152480 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2fe8073280 0x7f2fe8075790 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f30080adff0 tx=0x7f2ff8007a40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:57.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.677+0000 7f2ffeffd640 1 -- 172.21.15.150:0/1755152480 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3000016790 con 0x7f30081ee6c0 2024-09-17T05:14:57.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.981+0000 7f300cd16640 1 -- 172.21.15.150:0/1755152480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f300806e900 con 0x7f30081ee6c0 2024-09-17T05:14:57.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.981+0000 7f2ffeffd640 1 -- 172.21.15.150:0/1755152480 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7f30000f9000 con 0x7f30081ee6c0 2024-09-17T05:14:57.980 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:57.980 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:57.981 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:14:57.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.985+0000 7f300cd16640 1 -- 172.21.15.150:0/1755152480 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2fe8073280 msgr2=0x7f2fe8075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:57.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.985+0000 7f300cd16640 1 --2- 172.21.15.150:0/1755152480 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2fe8073280 0x7f2fe8075790 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f30080adff0 tx=0x7f2ff8007a40 comp rx=0 tx=0).stop 2024-09-17T05:14:57.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.985+0000 7f300cd16640 1 -- 172.21.15.150:0/1755152480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30081ee6c0 msgr2=0x7f300814f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:57.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.985+0000 7f300cd16640 1 --2- 172.21.15.150:0/1755152480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30081ee6c0 0x7f300814f3b0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f30000b0040 tx=0x7f3000009330 comp rx=0 tx=0).stop 2024-09-17T05:14:57.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.985+0000 7f300cd16640 1 -- 172.21.15.150:0/1755152480 shutdown_connections 2024-09-17T05:14:57.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.985+0000 7f300cd16640 1 --2- 172.21.15.150:0/1755152480 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2fe8073280 0x7f2fe8075790 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.985+0000 7f300cd16640 1 --2- 172.21.15.150:0/1755152480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30081ee6c0 0x7f300814f3b0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.985+0000 7f300cd16640 1 --2- 172.21.15.150:0/1755152480 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3008154f90 0x7f3008149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.985+0000 7f300cd16640 1 --2- 172.21.15.150:0/1755152480 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3008151f30 0x7f30081f1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:57.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.985+0000 7f300cd16640 1 -- 172.21.15.150:0/1755152480 >> 172.21.15.150:0/1755152480 conn(0x7f30080c4e30 msgr2=0x7f300806f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:57.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.985+0000 7f300cd16640 1 -- 172.21.15.150:0/1755152480 shutdown_connections 2024-09-17T05:14:57.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:57.985+0000 7f300cd16640 1 -- 172.21.15.150:0/1755152480 wait complete. 2024-09-17T05:14:58.011 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:14:59.444 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:14:59.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.637+0000 7f28b27e0640 1 -- 172.21.15.150:0/1884704135 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ac1568c0 msgr2=0x7f28ac0c6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:59.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.637+0000 7f28b27e0640 1 --2- 172.21.15.150:0/1884704135 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ac1568c0 0x7f28ac0c6a20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f289c009840 tx=0x7f289c01c820 comp rx=0 tx=0).stop 2024-09-17T05:14:59.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.637+0000 7f28b27e0640 1 -- 172.21.15.150:0/1884704135 shutdown_connections 2024-09-17T05:14:59.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.637+0000 7f28b27e0640 1 --2- 172.21.15.150:0/1884704135 >> v2:172.21.15.150:3300/0 conn(0x7f28ac150700 0x7f28ac0c2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:59.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.637+0000 7f28b27e0640 1 --2- 172.21.15.150:0/1884704135 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ac1568c0 0x7f28ac0c6a20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:59.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.637+0000 7f28b27e0640 1 -- 172.21.15.150:0/1884704135 >> 172.21.15.150:0/1884704135 conn(0x7f28ac074b80 msgr2=0x7f28ac06f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:59.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.637+0000 7f28b27e0640 1 -- 172.21.15.150:0/1884704135 shutdown_connections 2024-09-17T05:14:59.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.637+0000 7f28b27e0640 1 -- 172.21.15.150:0/1884704135 wait complete. 2024-09-17T05:14:59.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.637+0000 7f28b27e0640 1 Processor -- start 2024-09-17T05:14:59.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.637+0000 7f28b27e0640 1 -- start start 2024-09-17T05:14:59.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28b27e0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac0c8ab0 0x7f28ac0cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:59.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28b27e0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28ac150700 0x7f28ac1f3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:59.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28b27e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ac1568c0 0x7f28ac14a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:59.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28b27e0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac06fe30 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28b27e0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac150b10 con 0x7f28ac150700 2024-09-17T05:14:59.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28b27e0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f28ac150c90 con 0x7f28ac1568c0 2024-09-17T05:14:59.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28abfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac0c8ab0 0x7f28ac0cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:59.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28abfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac0c8ab0 0x7f28ac0cb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53748/0 (socket says 172.21.15.150:53748) 2024-09-17T05:14:59.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28abfff640 1 -- 172.21.15.150:0/1841097553 learned_addr learned my addr 172.21.15.150:0/1841097553 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:14:59.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28b0d7d640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ac1568c0 0x7f28ac14a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:59.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28a3fff640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28ac150700 0x7f28ac1f3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:59.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28abfff640 1 -- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ac1568c0 msgr2=0x7f28ac14a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:59.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28abfff640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ac1568c0 0x7f28ac14a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:59.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28abfff640 1 -- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28ac150700 msgr2=0x7f28ac1f3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:59.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28abfff640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28ac150700 0x7f28ac1f3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:59.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28abfff640 1 -- 172.21.15.150:0/1841097553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ac14caa0 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28abfff640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac0c8ab0 0x7f28ac0cb680 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f289c0b0040 tx=0x7f289c0a6760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:59.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28a97fa640 1 -- 172.21.15.150:0/1841097553 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c0bd070 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28b27e0640 1 -- 172.21.15.150:0/1841097553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28ac1501a0 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28a97fa640 1 -- 172.21.15.150:0/1841097553 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f289c005120 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28a97fa640 1 -- 172.21.15.150:0/1841097553 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c005550 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.641+0000 7f28b27e0640 1 -- 172.21.15.150:0/1841097553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28ac1f3cc0 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.645+0000 7f28b27e0640 1 -- 172.21.15.150:0/1841097553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28ac06a6e0 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.645+0000 7f28a97fa640 1 -- 172.21.15.150:0/1841097553 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f289c0046d0 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.649+0000 7f28a97fa640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f288c073230 0x7f288c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:14:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.649+0000 7f28a97fa640 1 -- 172.21.15.150:0/1841097553 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f289c12f010 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.649+0000 7f28a3fff640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f288c073230 0x7f288c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:14:59.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.649+0000 7f28a3fff640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f288c073230 0x7f288c075740 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0c24b0 tx=0x7f2894006f90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:14:59.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.653+0000 7f28a97fa640 1 -- 172.21.15.150:0/1841097553 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f289c016410 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.945+0000 7f28b27e0640 1 -- 172.21.15.150:0/1841097553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28ac0c8490 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.945+0000 7f28a97fa640 1 -- 172.21.15.150:0/1841097553 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7f289c0f8b50 con 0x7f28ac0c8ab0 2024-09-17T05:14:59.945 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:14:59.945 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:14:59.945 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:14:59.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.949+0000 7f28b27e0640 1 -- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f288c073230 msgr2=0x7f288c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:59.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.949+0000 7f28b27e0640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f288c073230 0x7f288c075740 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0c24b0 tx=0x7f2894006f90 comp rx=0 tx=0).stop 2024-09-17T05:14:59.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.949+0000 7f28b27e0640 1 -- 172.21.15.150:0/1841097553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac0c8ab0 msgr2=0x7f28ac0cb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:14:59.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.949+0000 7f28b27e0640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac0c8ab0 0x7f28ac0cb680 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f289c0b0040 tx=0x7f289c0a6760 comp rx=0 tx=0).stop 2024-09-17T05:14:59.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.949+0000 7f28b27e0640 1 -- 172.21.15.150:0/1841097553 shutdown_connections 2024-09-17T05:14:59.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.949+0000 7f28b27e0640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f288c073230 0x7f288c075740 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:59.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.949+0000 7f28b27e0640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28ac150700 0x7f28ac1f3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:59.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.949+0000 7f28b27e0640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28ac1568c0 0x7f28ac14a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:59.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.949+0000 7f28b27e0640 1 --2- 172.21.15.150:0/1841097553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28ac0c8ab0 0x7f28ac0cb680 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:14:59.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.949+0000 7f28b27e0640 1 -- 172.21.15.150:0/1841097553 >> 172.21.15.150:0/1841097553 conn(0x7f28ac074b80 msgr2=0x7f28ac0c4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:14:59.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.949+0000 7f28b27e0640 1 -- 172.21.15.150:0/1841097553 shutdown_connections 2024-09-17T05:14:59.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:14:59.949+0000 7f28b27e0640 1 -- 172.21.15.150:0/1841097553 wait complete. 2024-09-17T05:14:59.983 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:15:00.012 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.197+0000 7f6ff42dc640 1 -- 172.21.15.150:0/1904383986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fec0c4250 msgr2=0x7f6fec153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.197+0000 7f6ff42dc640 1 --2- 172.21.15.150:0/1904383986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fec0c4250 0x7f6fec153ee0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009840 tx=0x7f6fe801c820 comp rx=0 tx=0).stop 2024-09-17T05:15:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.197+0000 7f6ff42dc640 1 -- 172.21.15.150:0/1904383986 shutdown_connections 2024-09-17T05:15:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.197+0000 7f6ff42dc640 1 --2- 172.21.15.150:0/1904383986 >> v2:172.21.15.150:3300/0 conn(0x7f6fec0c3900 0x7f6fec074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:00.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.197+0000 7f6ff42dc640 1 --2- 172.21.15.150:0/1904383986 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fec0c4250 0x7f6fec153ee0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.197+0000 7f6ff42dc640 1 -- 172.21.15.150:0/1904383986 >> 172.21.15.150:0/1904383986 conn(0x7f6fec06e360 msgr2=0x7f6fec06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.197+0000 7f6ff42dc640 1 -- 172.21.15.150:0/1904383986 shutdown_connections 2024-09-17T05:15:00.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.197+0000 7f6ff42dc640 1 -- 172.21.15.150:0/1904383986 wait complete. 2024-09-17T05:15:00.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff42dc640 1 Processor -- start 2024-09-17T05:15:00.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff42dc640 1 -- start start 2024-09-17T05:15:00.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff42dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fec074e60 0x7f6fec0ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:00.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff42dc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fec0c3900 0x7f6fec1ed810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:00.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff42dc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fec0c4250 0x7f6fec0c2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:00.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff42dc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec072f30 con 0x7f6fec074e60 2024-09-17T05:15:00.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff42dc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec072db0 con 0x7f6fec0c4250 2024-09-17T05:15:00.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff42dc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6fec160f90 con 0x7f6fec0c3900 2024-09-17T05:15:00.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff1877640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fec0c3900 0x7f6fec1ed810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:00.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff1877640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fec0c3900 0x7f6fec1ed810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:49320/0 (socket says 172.21.15.150:49320) 2024-09-17T05:15:00.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff2078640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fec074e60 0x7f6fec0ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:00.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff1877640 1 -- 172.21.15.150:0/2127107937 learned_addr learned my addr 172.21.15.150:0/2127107937 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:00.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff2879640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fec0c4250 0x7f6fec0c2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:00.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff2078640 1 -- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fec0c3900 msgr2=0x7f6fec1ed810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:00.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff2078640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fec0c3900 0x7f6fec1ed810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:00.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff2078640 1 -- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fec0c4250 msgr2=0x7f6fec0c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:00.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff2078640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fec0c4250 0x7f6fec0c2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:00.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.201+0000 7f6ff2078640 1 -- 172.21.15.150:0/2127107937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fec1ed350 con 0x7f6fec074e60 2024-09-17T05:15:00.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.205+0000 7f6ff2078640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fec074e60 0x7f6fec0ae180 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f6fe80ac000 tx=0x7f6fe8005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:00.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.205+0000 7f6fe2ffd640 1 -- 172.21.15.150:0/2127107937 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8004040 con 0x7f6fec074e60 2024-09-17T05:15:00.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.205+0000 7f6fe2ffd640 1 -- 172.21.15.150:0/2127107937 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6fe80042d0 con 0x7f6fec074e60 2024-09-17T05:15:00.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.205+0000 7f6fe2ffd640 1 -- 172.21.15.150:0/2127107937 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8003800 con 0x7f6fec074e60 2024-09-17T05:15:00.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.205+0000 7f6ff42dc640 1 -- 172.21.15.150:0/2127107937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fec06d0a0 con 0x7f6fec074e60 2024-09-17T05:15:00.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.205+0000 7f6ff42dc640 1 -- 172.21.15.150:0/2127107937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fec06d660 con 0x7f6fec074e60 2024-09-17T05:15:00.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.205+0000 7f6ff42dc640 1 -- 172.21.15.150:0/2127107937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fec1ee930 con 0x7f6fec074e60 2024-09-17T05:15:00.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.209+0000 7f6fe2ffd640 1 -- 172.21.15.150:0/2127107937 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6fe80c1030 con 0x7f6fec074e60 2024-09-17T05:15:00.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.209+0000 7f6fe2ffd640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6fd0073200 0x7f6fd0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:00.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.209+0000 7f6fe2ffd640 1 -- 172.21.15.150:0/2127107937 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f6fe8130060 con 0x7f6fec074e60 2024-09-17T05:15:00.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.209+0000 7f6ff1877640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6fd0073200 0x7f6fd0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:00.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.209+0000 7f6ff1877640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6fd0073200 0x7f6fd0075710 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc005fd0 tx=0x7f6fdc005ea0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:00.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.213+0000 7f6fe2ffd640 1 -- 172.21.15.150:0/2127107937 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6fe80c8020 con 0x7f6fec074e60 2024-09-17T05:15:00.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.517+0000 7f6ff42dc640 1 -- 172.21.15.150:0/2127107937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fec074920 con 0x7f6fec074e60 2024-09-17T05:15:00.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.517+0000 7f6fe2ffd640 1 -- 172.21.15.150:0/2127107937 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7f6fec1ed350 con 0x7f6fec074e60 2024-09-17T05:15:00.516 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:00.516 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:15:00.516 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:15:00.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.521+0000 7f6ff42dc640 1 -- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6fd0073200 msgr2=0x7f6fd0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:00.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.521+0000 7f6ff42dc640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6fd0073200 0x7f6fd0075710 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc005fd0 tx=0x7f6fdc005ea0 comp rx=0 tx=0).stop 2024-09-17T05:15:00.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.521+0000 7f6ff42dc640 1 -- 172.21.15.150:0/2127107937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fec074e60 msgr2=0x7f6fec0ae180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:00.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.521+0000 7f6ff42dc640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fec074e60 0x7f6fec0ae180 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f6fe80ac000 tx=0x7f6fe8005db0 comp rx=0 tx=0).stop 2024-09-17T05:15:00.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.521+0000 7f6ff42dc640 1 -- 172.21.15.150:0/2127107937 shutdown_connections 2024-09-17T05:15:00.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.521+0000 7f6ff42dc640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6fd0073200 0x7f6fd0075710 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:00.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.521+0000 7f6ff42dc640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6fec0c4250 0x7f6fec0c2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:00.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.521+0000 7f6ff42dc640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6fec0c3900 0x7f6fec1ed810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:00.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.521+0000 7f6ff42dc640 1 --2- 172.21.15.150:0/2127107937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fec074e60 0x7f6fec0ae180 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:00.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.521+0000 7f6ff42dc640 1 -- 172.21.15.150:0/2127107937 >> 172.21.15.150:0/2127107937 conn(0x7f6fec06e360 msgr2=0x7f6fec0c68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:00.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.521+0000 7f6ff42dc640 1 -- 172.21.15.150:0/2127107937 shutdown_connections 2024-09-17T05:15:00.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:00.521+0000 7f6ff42dc640 1 -- 172.21.15.150:0/2127107937 wait complete. 2024-09-17T05:15:00.548 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:15:01.984 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.153+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/2834248131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf4071680 msgr2=0x7fbdf413c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.153+0000 7fbdf8b4a640 1 --2- 172.21.15.150:0/2834248131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf4071680 0x7fbdf413c8f0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fbde8009900 tx=0x7fbde801c850 comp rx=0 tx=0).stop 2024-09-17T05:15:02.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.153+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/2834248131 shutdown_connections 2024-09-17T05:15:02.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.153+0000 7fbdf8b4a640 1 --2- 172.21.15.150:0/2834248131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf4071680 0x7fbdf413c8f0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:02.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.153+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/2834248131 >> 172.21.15.150:0/2834248131 conn(0x7fbdf40abaa0 msgr2=0x7fbdf40abeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:02.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.153+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/2834248131 shutdown_connections 2024-09-17T05:15:02.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.153+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/2834248131 wait complete. 2024-09-17T05:15:02.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.153+0000 7fbdf8b4a640 1 Processor -- start 2024-09-17T05:15:02.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.153+0000 7fbdf8b4a640 1 -- start start 2024-09-17T05:15:02.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf8b4a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf4071680 0x7fbdf4154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:02.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf8b4a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdf413ce30 0x7fbdf4152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:02.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf8b4a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbdf413db50 0x7fbdf4152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf8b4a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf41627b0 con 0x7fbdf4071680 2024-09-17T05:15:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf8b4a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf4162930 con 0x7fbdf413ce30 2024-09-17T05:15:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf8b4a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbdf4162ab0 con 0x7fbdf413db50 2024-09-17T05:15:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf4071680 0x7fbdf4154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf4071680 0x7fbdf4154d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53792/0 (socket says 172.21.15.150:53792) 2024-09-17T05:15:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf259c640 1 -- 172.21.15.150:0/391925333 learned_addr learned my addr 172.21.15.150:0/391925333 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf1d9b640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdf413ce30 0x7fbdf4152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf259c640 1 -- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbdf413db50 msgr2=0x7fbdf4152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf2d9d640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbdf413db50 0x7fbdf4152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf259c640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbdf413db50 0x7fbdf4152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf259c640 1 -- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdf413ce30 msgr2=0x7fbdf4152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf259c640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdf413ce30 0x7fbdf4152890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:02.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf259c640 1 -- 172.21.15.150:0/391925333 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf41f0000 con 0x7fbdf4071680 2024-09-17T05:15:02.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf1d9b640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdf413ce30 0x7fbdf4152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:15:02.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf259c640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf4071680 0x7fbdf4154d30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fbde400c3a0 tx=0x7fbde400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:02.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbde2ffd640 1 -- 172.21.15.150:0/391925333 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde4017070 con 0x7fbdf4071680 2024-09-17T05:15:02.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbde2ffd640 1 -- 172.21.15.150:0/391925333 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbde40053e0 con 0x7fbdf4071680 2024-09-17T05:15:02.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/391925333 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf4138530 con 0x7fbdf4071680 2024-09-17T05:15:02.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbde2ffd640 1 -- 172.21.15.150:0/391925333 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde4003880 con 0x7fbdf4071680 2024-09-17T05:15:02.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/391925333 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf4154810 con 0x7fbdf4071680 2024-09-17T05:15:02.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/391925333 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf413c8f0 con 0x7fbdf4071680 2024-09-17T05:15:02.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.157+0000 7fbde2ffd640 1 -- 172.21.15.150:0/391925333 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbde401a070 con 0x7fbdf4071680 2024-09-17T05:15:02.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.161+0000 7fbde2ffd640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbdd4073200 0x7fbdd4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:02.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.161+0000 7fbde2ffd640 1 -- 172.21.15.150:0/391925333 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7fbde4094300 con 0x7fbdf4071680 2024-09-17T05:15:02.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.161+0000 7fbdf1d9b640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbdd4073200 0x7fbdd4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:02.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.161+0000 7fbdf1d9b640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbdd4073200 0x7fbdd4075710 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fbde80af000 tx=0x7fbde801b090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:02.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.165+0000 7fbde2ffd640 1 -- 172.21.15.150:0/391925333 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbde405e570 con 0x7fbdf4071680 2024-09-17T05:15:02.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/391925333 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdf413c1f0 con 0x7fbdf4071680 2024-09-17T05:15:02.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbde2ffd640 1 -- 172.21.15.150:0/391925333 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7fbde405de70 con 0x7fbdf4071680 2024-09-17T05:15:02.371 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:02.371 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:15:02.372 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:15:02.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbdd4073200 msgr2=0x7fbdd4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:02.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbdf8b4a640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbdd4073200 0x7fbdd4075710 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fbde80af000 tx=0x7fbde801b090 comp rx=0 tx=0).stop 2024-09-17T05:15:02.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/391925333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf4071680 msgr2=0x7fbdf4154d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:02.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbdf8b4a640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf4071680 0x7fbdf4154d30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fbde400c3a0 tx=0x7fbde400c860 comp rx=0 tx=0).stop 2024-09-17T05:15:02.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/391925333 shutdown_connections 2024-09-17T05:15:02.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbdf8b4a640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbdd4073200 0x7fbdd4075710 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:02.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbdf8b4a640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbdf413ce30 0x7fbdf4152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:02.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbdf8b4a640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbdf413db50 0x7fbdf4152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:02.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbdf8b4a640 1 --2- 172.21.15.150:0/391925333 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbdf4071680 0x7fbdf4154d30 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:02.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/391925333 >> 172.21.15.150:0/391925333 conn(0x7fbdf40abaa0 msgr2=0x7fbdf40c3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.373+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/391925333 shutdown_connections 2024-09-17T05:15:02.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.377+0000 7fbdf8b4a640 1 -- 172.21.15.150:0/391925333 wait complete. 2024-09-17T05:15:02.395 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:15:02.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:02.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.805+0000 7f3039503640 1 -- 172.21.15.150:0/12020983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c003a10 msgr2=0x7f301c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:02.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.805+0000 7f3039503640 1 --2- 172.21.15.150:0/12020983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c003a10 0x7f301c003e80 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f30340c2500 tx=0x7f302401cf20 comp rx=0 tx=0).stop 2024-09-17T05:15:02.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.805+0000 7f30337fe640 1 -- 172.21.15.150:0/12020983 reap_dead start 2024-09-17T05:15:02.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.805+0000 7f3039503640 1 -- 172.21.15.150:0/12020983 shutdown_connections 2024-09-17T05:15:02.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.805+0000 7f3039503640 1 -- 172.21.15.150:0/12020983 >> 172.21.15.150:0/12020983 conn(0x7f30340c4a00 msgr2=0x7f30340c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:02.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.805+0000 7f3039503640 1 -- 172.21.15.150:0/12020983 shutdown_connections 2024-09-17T05:15:02.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.805+0000 7f3039503640 1 -- 172.21.15.150:0/12020983 wait complete. 2024-09-17T05:15:02.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.805+0000 7f3039503640 1 Processor -- start 2024-09-17T05:15:02.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.805+0000 7f3039503640 1 -- start start 2024-09-17T05:15:02.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3039503640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c003a10 0x7f30341eda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:02.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3039503640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30341584f0 0x7f30340ac380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:02.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3039503640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303414d2f0 0x7f303414d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:02.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3039503640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f30340adb30 con 0x7f301c003a10 2024-09-17T05:15:02.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3039503640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f30340ad9b0 con 0x7f303414d2f0 2024-09-17T05:15:02.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3039503640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f30340a75f0 con 0x7f30341584f0 2024-09-17T05:15:02.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3032ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c003a10 0x7f30341eda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:02.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3032ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c003a10 0x7f30341eda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53810/0 (socket says 172.21.15.150:53810) 2024-09-17T05:15:02.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3032ffd640 1 -- 172.21.15.150:0/4101579596 learned_addr learned my addr 172.21.15.150:0/4101579596 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:02.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f30327fc640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30341584f0 0x7f30340ac380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:02.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f30337fe640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303414d2f0 0x7f303414d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:02.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3032ffd640 1 -- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30341584f0 msgr2=0x7f30340ac380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:02.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3032ffd640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30341584f0 0x7f30340ac380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:02.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3032ffd640 1 -- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303414d2f0 msgr2=0x7f303414d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:02.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3032ffd640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303414d2f0 0x7f303414d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:02.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3032ffd640 1 -- 172.21.15.150:0/4101579596 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3034150bc0 con 0x7f301c003a10 2024-09-17T05:15:02.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3032ffd640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c003a10 0x7f30341eda20 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f302401d400 tx=0x7f3024006d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:02.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f301b7fe640 1 -- 172.21.15.150:0/4101579596 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30240a5bb0 con 0x7f301c003a10 2024-09-17T05:15:02.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f301b7fe640 1 -- 172.21.15.150:0/4101579596 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30240014e0 con 0x7f301c003a10 2024-09-17T05:15:02.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f301b7fe640 1 -- 172.21.15.150:0/4101579596 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30240060d0 con 0x7f301c003a10 2024-09-17T05:15:02.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3039503640 1 -- 172.21.15.150:0/4101579596 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f303414e0c0 con 0x7f301c003a10 2024-09-17T05:15:02.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3039503640 1 -- 172.21.15.150:0/4101579596 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30340acd60 con 0x7f301c003a10 2024-09-17T05:15:02.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f3039503640 1 -- 172.21.15.150:0/4101579596 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3034070400 con 0x7f301c003a10 2024-09-17T05:15:02.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.809+0000 7f301b7fe640 1 -- 172.21.15.150:0/4101579596 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3024006290 con 0x7f301c003a10 2024-09-17T05:15:02.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.813+0000 7f301b7fe640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3020073230 0x7f3020075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:02.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.813+0000 7f301b7fe640 1 -- 172.21.15.150:0/4101579596 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f302412ef10 con 0x7f301c003a10 2024-09-17T05:15:02.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.813+0000 7f30327fc640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3020073230 0x7f3020075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:02.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.813+0000 7f30327fc640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3020073230 0x7f3020075740 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3028005fd0 tx=0x7f3028005ec0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:02.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:02.817+0000 7f301b7fe640 1 -- 172.21.15.150:0/4101579596 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f30240c8080 con 0x7f301c003a10 2024-09-17T05:15:03.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.021+0000 7f3039503640 1 -- 172.21.15.150:0/4101579596 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30340a72c0 con 0x7f301c003a10 2024-09-17T05:15:03.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.021+0000 7f301b7fe640 1 -- 172.21.15.150:0/4101579596 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7f3024016cd0 con 0x7f301c003a10 2024-09-17T05:15:03.021 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:03.022 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:15:03.022 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:15:03.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.025+0000 7f3039503640 1 -- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3020073230 msgr2=0x7f3020075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:03.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.025+0000 7f3039503640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3020073230 0x7f3020075740 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3028005fd0 tx=0x7f3028005ec0 comp rx=0 tx=0).stop 2024-09-17T05:15:03.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.025+0000 7f3039503640 1 -- 172.21.15.150:0/4101579596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c003a10 msgr2=0x7f30341eda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:03.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.025+0000 7f3039503640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c003a10 0x7f30341eda20 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f302401d400 tx=0x7f3024006d80 comp rx=0 tx=0).stop 2024-09-17T05:15:03.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.025+0000 7f3039503640 1 -- 172.21.15.150:0/4101579596 shutdown_connections 2024-09-17T05:15:03.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.025+0000 7f3039503640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3020073230 0x7f3020075740 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:03.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.025+0000 7f3039503640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303414d2f0 0x7f303414d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:03.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.025+0000 7f3039503640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30341584f0 0x7f30340ac380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:03.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.025+0000 7f3039503640 1 --2- 172.21.15.150:0/4101579596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f301c003a10 0x7f30341eda20 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:03.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.025+0000 7f3039503640 1 -- 172.21.15.150:0/4101579596 >> 172.21.15.150:0/4101579596 conn(0x7f30340c4a00 msgr2=0x7f303415e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:03.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.025+0000 7f3039503640 1 -- 172.21.15.150:0/4101579596 shutdown_connections 2024-09-17T05:15:03.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:03.025+0000 7f3039503640 1 -- 172.21.15.150:0/4101579596 wait complete. 2024-09-17T05:15:03.045 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:15:04.395 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:04.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.565+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/634735579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e9406fab0 msgr2=0x7f7e9406fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:04.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.565+0000 7f7e9ac0a640 1 --2- 172.21.15.150:0/634735579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e9406fab0 0x7f7e9406fec0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f7e940c78a0 tx=0x7f7e8c01cd80 comp rx=0 tx=0).stop 2024-09-17T05:15:04.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.565+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/634735579 shutdown_connections 2024-09-17T05:15:04.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.565+0000 7f7e9ac0a640 1 --2- 172.21.15.150:0/634735579 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e9406fab0 0x7f7e9406fec0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:04.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.565+0000 7f7e9ac0a640 1 --2- 172.21.15.150:0/634735579 >> v2:172.21.15.176:3300/0 conn(0x7f7e940c41c0 0x7f7e940ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:04.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.565+0000 7f7e9ac0a640 1 --2- 172.21.15.150:0/634735579 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e940c3db0 0x7f7e940ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:04.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.565+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/634735579 >> 172.21.15.150:0/634735579 conn(0x7f7e940c4a00 msgr2=0x7f7e940c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:04.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.565+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/634735579 shutdown_connections 2024-09-17T05:15:04.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.565+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/634735579 wait complete. 2024-09-17T05:15:04.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.565+0000 7f7e9ac0a640 1 Processor -- start 2024-09-17T05:15:04.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.565+0000 7f7e9ac0a640 1 -- start start 2024-09-17T05:15:04.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e9ac0a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e9406fab0 0x7f7e94150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:04.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e9ac0a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e940c3db0 0x7f7e94151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e9ac0a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e940c41c0 0x7f7e9414b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e9ac0a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7e940a7770 con 0x7f7e940c41c0 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e9ac0a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7e940a75f0 con 0x7f7e9406fab0 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e9ac0a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7e94158480 con 0x7f7e940c3db0 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e991a7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e940c41c0 0x7f7e9414b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e991a7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e940c41c0 0x7f7e9414b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53818/0 (socket says 172.21.15.150:53818) 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e991a7640 1 -- 172.21.15.150:0/2363244458 learned_addr learned my addr 172.21.15.150:0/2363244458 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e989a6640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e9406fab0 0x7f7e94150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e93fff640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e940c3db0 0x7f7e94151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e991a7640 1 -- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e940c3db0 msgr2=0x7f7e94151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e991a7640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e940c3db0 0x7f7e94151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e991a7640 1 -- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e9406fab0 msgr2=0x7f7e94150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:04.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e991a7640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e9406fab0 0x7f7e94150950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:04.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e991a7640 1 -- 172.21.15.150:0/2363244458 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9414b330 con 0x7f7e940c41c0 2024-09-17T05:15:04.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e93fff640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e940c3db0 0x7f7e94151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:15:04.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e991a7640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e940c41c0 0x7f7e9414b7c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009bd0 tx=0x7f7e8c005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:04.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e917fa640 1 -- 172.21.15.150:0/2363244458 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c0bb070 con 0x7f7e940c41c0 2024-09-17T05:15:04.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e917fa640 1 -- 172.21.15.150:0/2363244458 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e8c004d40 con 0x7f7e940c41c0 2024-09-17T05:15:04.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e917fa640 1 -- 172.21.15.150:0/2363244458 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c0b2040 con 0x7f7e940c41c0 2024-09-17T05:15:04.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/2363244458 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e94149190 con 0x7f7e940c41c0 2024-09-17T05:15:04.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/2363244458 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e94157940 con 0x7f7e940c41c0 2024-09-17T05:15:04.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/2363244458 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e94158930 con 0x7f7e940c41c0 2024-09-17T05:15:04.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e917fa640 1 -- 172.21.15.150:0/2363244458 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7e8c0be070 con 0x7f7e940c41c0 2024-09-17T05:15:04.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e917fa640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7e74073200 0x7f7e74075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:04.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e917fa640 1 -- 172.21.15.150:0/2363244458 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f7e8c12f080 con 0x7f7e940c41c0 2024-09-17T05:15:04.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.569+0000 7f7e989a6640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7e74073200 0x7f7e74075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:04.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.573+0000 7f7e989a6640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7e74073200 0x7f7e74075710 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7e84005fd0 tx=0x7f7e84005de0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:04.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.573+0000 7f7e917fa640 1 -- 172.21.15.150:0/2363244458 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7e8c016970 con 0x7f7e940c41c0 2024-09-17T05:15:04.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/2363244458 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e94153b40 con 0x7f7e940c41c0 2024-09-17T05:15:04.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e917fa640 1 -- 172.21.15.150:0/2363244458 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7f7e8c0f92f0 con 0x7f7e940c41c0 2024-09-17T05:15:04.779 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:04.780 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:15:04.780 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:15:04.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7e74073200 msgr2=0x7f7e74075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:04.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e9ac0a640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7e74073200 0x7f7e74075710 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f7e84005fd0 tx=0x7f7e84005de0 comp rx=0 tx=0).stop 2024-09-17T05:15:04.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/2363244458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e940c41c0 msgr2=0x7f7e9414b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:04.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e9ac0a640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e940c41c0 0x7f7e9414b7c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009bd0 tx=0x7f7e8c005b00 comp rx=0 tx=0).stop 2024-09-17T05:15:04.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/2363244458 shutdown_connections 2024-09-17T05:15:04.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e9ac0a640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7e74073200 0x7f7e74075710 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:04.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e9ac0a640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e940c41c0 0x7f7e9414b7c0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:04.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e9ac0a640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7e940c3db0 0x7f7e94151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:04.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e9ac0a640 1 --2- 172.21.15.150:0/2363244458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e9406fab0 0x7f7e94150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:04.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/2363244458 >> 172.21.15.150:0/2363244458 conn(0x7f7e940c4a00 msgr2=0x7f7e9414e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:04.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.781+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/2363244458 shutdown_connections 2024-09-17T05:15:04.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:04.785+0000 7f7e9ac0a640 1 -- 172.21.15.150:0/2363244458 wait complete. 2024-09-17T05:15:04.805 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:15:05.045 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:05.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.209+0000 7f122ddd6640 1 -- 172.21.15.150:0/2622980493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12280ad7e0 msgr2=0x7f12280701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:05.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.209+0000 7f122ddd6640 1 --2- 172.21.15.150:0/2622980493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12280ad7e0 0x7f12280701c0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f1220009900 tx=0x7f122001c8e0 comp rx=0 tx=0).stop 2024-09-17T05:15:05.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.209+0000 7f122ddd6640 1 -- 172.21.15.150:0/2622980493 shutdown_connections 2024-09-17T05:15:05.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.209+0000 7f122ddd6640 1 --2- 172.21.15.150:0/2622980493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12280ad7e0 0x7f12280701c0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:05.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.209+0000 7f122ddd6640 1 --2- 172.21.15.150:0/2622980493 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12280c3db0 0x7f12280ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:05.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.209+0000 7f122ddd6640 1 -- 172.21.15.150:0/2622980493 >> 172.21.15.150:0/2622980493 conn(0x7f12280c4a00 msgr2=0x7f12280c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:05.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.209+0000 7f122ddd6640 1 -- 172.21.15.150:0/2622980493 shutdown_connections 2024-09-17T05:15:05.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.209+0000 7f122ddd6640 1 -- 172.21.15.150:0/2622980493 wait complete. 2024-09-17T05:15:05.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.209+0000 7f122ddd6640 1 Processor -- start 2024-09-17T05:15:05.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.209+0000 7f122ddd6640 1 -- start start 2024-09-17T05:15:05.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f122ddd6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12280ad7e0 0x7f12281eda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:05.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f122ddd6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12280c3db0 0x7f122814d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:05.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f122ddd6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12280c41c0 0x7f1228150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:05.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f122ddd6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f12280a75f0 con 0x7f12280c41c0 2024-09-17T05:15:05.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f122ddd6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f12280add70 con 0x7f12280ad7e0 2024-09-17T05:15:05.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f122ddd6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f12280a7770 con 0x7f12280c3db0 2024-09-17T05:15:05.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f1227fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12280c41c0 0x7f1228150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:05.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f1227fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12280c41c0 0x7f1228150db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53846/0 (socket says 172.21.15.150:53846) 2024-09-17T05:15:05.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f1227fff640 1 -- 172.21.15.150:0/2031686147 learned_addr learned my addr 172.21.15.150:0/2031686147 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:05.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f1226ffd640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12280c3db0 0x7f122814d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:05.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f12277fe640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12280ad7e0 0x7f12281eda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:05.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f1227fff640 1 -- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12280c3db0 msgr2=0x7f122814d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:05.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f1227fff640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12280c3db0 0x7f122814d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:05.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f1227fff640 1 -- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12280ad7e0 msgr2=0x7f12281eda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:05.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f1227fff640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12280ad7e0 0x7f12281eda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:05.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f1227fff640 1 -- 172.21.15.150:0/2031686147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12281ecf80 con 0x7f12280c41c0 2024-09-17T05:15:05.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f1227fff640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12280c41c0 0x7f1228150db0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f1220009230 tx=0x7f12200a55a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:05.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f121ffff640 1 -- 172.21.15.150:0/2031686147 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12200b2040 con 0x7f12280c41c0 2024-09-17T05:15:05.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f121ffff640 1 -- 172.21.15.150:0/2031686147 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12200041b0 con 0x7f12280c41c0 2024-09-17T05:15:05.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f122ddd6640 1 -- 172.21.15.150:0/2031686147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1228152bb0 con 0x7f12280c41c0 2024-09-17T05:15:05.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f121ffff640 1 -- 172.21.15.150:0/2031686147 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220003600 con 0x7f12280c41c0 2024-09-17T05:15:05.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f122ddd6640 1 -- 172.21.15.150:0/2031686147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12280ac870 con 0x7f12280c41c0 2024-09-17T05:15:05.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f122ddd6640 1 -- 172.21.15.150:0/2031686147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12280ad2a0 con 0x7f12280c41c0 2024-09-17T05:15:05.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f121ffff640 1 -- 172.21.15.150:0/2031686147 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f12200bf020 con 0x7f12280c41c0 2024-09-17T05:15:05.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f121ffff640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1208073230 0x7f1208075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:05.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.213+0000 7f121ffff640 1 -- 172.21.15.150:0/2031686147 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f122012f9d0 con 0x7f12280c41c0 2024-09-17T05:15:05.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.217+0000 7f12277fe640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1208073230 0x7f1208075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:05.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.217+0000 7f12277fe640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1208073230 0x7f1208075740 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f12280c2500 tx=0x7f1210005e50 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:05.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.217+0000 7f121ffff640 1 -- 172.21.15.150:0/2031686147 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f12200c6020 con 0x7f12280c41c0 2024-09-17T05:15:05.223 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:15:05.225 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:15:05.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.429+0000 7f122ddd6640 1 -- 172.21.15.150:0/2031686147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12281528d0 con 0x7f12280c41c0 2024-09-17T05:15:05.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f121ffff640 1 -- 172.21.15.150:0/2031686147 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+6544 (secure 0 0 0) 0x7f12200164d0 con 0x7f12280c41c0 2024-09-17T05:15:05.430 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:05.431 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":176,"btime":"2024-09-17T05:14:31:375977+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":7719,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:14:29.347280+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7078":{"gid":7078,"name":"a","rank":0,"incarnation":172,"state":"up:active","state_seq":10,"addr":"172.21.15.150:6833/3933379792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3933379792},{"type":"v1","addr":"172.21.15.150:6833","nonce":3933379792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7078,"qdb_cluster":[7078]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:15:05.431 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 176 2024-09-17T05:15:05.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f122ddd6640 1 -- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1208073230 msgr2=0x7f1208075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:05.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f122ddd6640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1208073230 0x7f1208075740 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f12280c2500 tx=0x7f1210005e50 comp rx=0 tx=0).stop 2024-09-17T05:15:05.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f122ddd6640 1 -- 172.21.15.150:0/2031686147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12280c41c0 msgr2=0x7f1228150db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:05.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f122ddd6640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12280c41c0 0x7f1228150db0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f1220009230 tx=0x7f12200a55a0 comp rx=0 tx=0).stop 2024-09-17T05:15:05.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f122ddd6640 1 -- 172.21.15.150:0/2031686147 shutdown_connections 2024-09-17T05:15:05.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f122ddd6640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1208073230 0x7f1208075740 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:05.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f122ddd6640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12280c41c0 0x7f1228150db0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:05.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f122ddd6640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f12280c3db0 0x7f122814d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:05.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f122ddd6640 1 --2- 172.21.15.150:0/2031686147 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12280ad7e0 0x7f12281eda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:05.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f122ddd6640 1 -- 172.21.15.150:0/2031686147 >> 172.21.15.150:0/2031686147 conn(0x7f12280c4a00 msgr2=0x7f122815e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:05.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f122ddd6640 1 -- 172.21.15.150:0/2031686147 shutdown_connections 2024-09-17T05:15:05.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:05.433+0000 7f122ddd6640 1 -- 172.21.15.150:0/2031686147 wait complete. 2024-09-17T05:15:05.453 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:15:06.806 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.969+0000 7f9708d58640 1 -- 172.21.15.150:0/2079481030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96ec0040c0 msgr2=0x7f96ec004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.969+0000 7f9708d58640 1 --2- 172.21.15.150:0/2079481030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96ec0040c0 0x7f96ec004810 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f96f800bb10 tx=0x7f96f801cda0 comp rx=0 tx=0).stop 2024-09-17T05:15:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.969+0000 7f9702d9d640 1 -- 172.21.15.150:0/2079481030 reap_dead start 2024-09-17T05:15:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.969+0000 7f9708d58640 1 -- 172.21.15.150:0/2079481030 shutdown_connections 2024-09-17T05:15:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.969+0000 7f9708d58640 1 -- 172.21.15.150:0/2079481030 >> 172.21.15.150:0/2079481030 conn(0x7f97040abaa0 msgr2=0x7f97040abeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.969+0000 7f9708d58640 1 -- 172.21.15.150:0/2079481030 shutdown_connections 2024-09-17T05:15:06.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.969+0000 7f9708d58640 1 -- 172.21.15.150:0/2079481030 wait complete. 2024-09-17T05:15:06.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9708d58640 1 Processor -- start 2024-09-17T05:15:06.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9708d58640 1 -- start start 2024-09-17T05:15:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9708d58640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96ec0040c0 0x7f9704138950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9708d58640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97041f36c0 0x7f97041f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9708d58640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97041559f0 0x7f9704155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9708d58640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f970413d240 con 0x7f96ec0040c0 2024-09-17T05:15:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9708d58640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f97040c8bb0 con 0x7f97041f36c0 2024-09-17T05:15:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9708d58640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f970413d3c0 con 0x7f97041559f0 2024-09-17T05:15:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f970259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96ec0040c0 0x7f9704138950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f970259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96ec0040c0 0x7f9704138950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53860/0 (socket says 172.21.15.150:53860) 2024-09-17T05:15:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f970259c640 1 -- 172.21.15.150:0/2347489644 learned_addr learned my addr 172.21.15.150:0/2347489644 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9701d9b640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97041f36c0 0x7f97041f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9702d9d640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97041559f0 0x7f9704155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:06.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f970259c640 1 -- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97041559f0 msgr2=0x7f9704155e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f970259c640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97041559f0 0x7f9704155e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f970259c640 1 -- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97041f36c0 msgr2=0x7f97041f3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f970259c640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97041f36c0 0x7f97041f3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f970259c640 1 -- 172.21.15.150:0/2347489644 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97040c3d40 con 0x7f96ec0040c0 2024-09-17T05:15:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f970259c640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96ec0040c0 0x7f9704138950 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f96f400d6d0 tx=0x7f96f400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f96f2ffd640 1 -- 172.21.15.150:0/2347489644 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f40060d0 con 0x7f96ec0040c0 2024-09-17T05:15:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f96f2ffd640 1 -- 172.21.15.150:0/2347489644 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96f4015020 con 0x7f96ec0040c0 2024-09-17T05:15:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9708d58640 1 -- 172.21.15.150:0/2347489644 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9704156ac0 con 0x7f96ec0040c0 2024-09-17T05:15:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f96f2ffd640 1 -- 172.21.15.150:0/2347489644 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f400b830 con 0x7f96ec0040c0 2024-09-17T05:15:06.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9708d58640 1 -- 172.21.15.150:0/2347489644 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97040ab7c0 con 0x7f96ec0040c0 2024-09-17T05:15:06.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.973+0000 7f9708d58640 1 -- 172.21.15.150:0/2347489644 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97040c7e80 con 0x7f96ec0040c0 2024-09-17T05:15:06.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.977+0000 7f96f2ffd640 1 -- 172.21.15.150:0/2347489644 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f96f400bbe0 con 0x7f96ec0040c0 2024-09-17T05:15:06.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.977+0000 7f96f2ffd640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f96e4073230 0x7f96e4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:06.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.977+0000 7f96f2ffd640 1 -- 172.21.15.150:0/2347489644 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7f96f4095cb0 con 0x7f96ec0040c0 2024-09-17T05:15:06.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.977+0000 7f9701d9b640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f96e4073230 0x7f96e4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:06.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.977+0000 7f9701d9b640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f96e4073230 0x7f96e4075740 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f96f80b0000 tx=0x7f96f80a7040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:06.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:06.981+0000 7f96f2ffd640 1 -- 172.21.15.150:0/2347489644 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f96f401f030 con 0x7f96ec0040c0 2024-09-17T05:15:07.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.189+0000 7f9708d58640 1 -- 172.21.15.150:0/2347489644 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f970413b290 con 0x7f96ec0040c0 2024-09-17T05:15:07.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.189+0000 7f96f2ffd640 1 -- 172.21.15.150:0/2347489644 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) ==== 95+0+5707 (secure 0 0 0) 0x7f96f405fec0 con 0x7f96ec0040c0 2024-09-17T05:15:07.189 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:07.189 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":177,"btime":"2024-09-17T05:15:06:234697+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:06.234683+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:15:07.189 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 177 2024-09-17T05:15:07.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.193+0000 7f9708d58640 1 -- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f96e4073230 msgr2=0x7f96e4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:07.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.193+0000 7f9708d58640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f96e4073230 0x7f96e4075740 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f96f80b0000 tx=0x7f96f80a7040 comp rx=0 tx=0).stop 2024-09-17T05:15:07.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.193+0000 7f9708d58640 1 -- 172.21.15.150:0/2347489644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96ec0040c0 msgr2=0x7f9704138950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:07.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.193+0000 7f9708d58640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96ec0040c0 0x7f9704138950 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f96f400d6d0 tx=0x7f96f400db90 comp rx=0 tx=0).stop 2024-09-17T05:15:07.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.193+0000 7f9708d58640 1 -- 172.21.15.150:0/2347489644 shutdown_connections 2024-09-17T05:15:07.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.193+0000 7f9708d58640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f96e4073230 0x7f96e4075740 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.193+0000 7f9708d58640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97041f36c0 0x7f97041f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.193+0000 7f9708d58640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f97041559f0 0x7f9704155e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.193+0000 7f9708d58640 1 --2- 172.21.15.150:0/2347489644 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96ec0040c0 0x7f9704138950 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.193+0000 7f9708d58640 1 -- 172.21.15.150:0/2347489644 >> 172.21.15.150:0/2347489644 conn(0x7f97040abaa0 msgr2=0x7f9704139180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:07.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.193+0000 7f9708d58640 1 -- 172.21.15.150:0/2347489644 shutdown_connections 2024-09-17T05:15:07.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.193+0000 7f9708d58640 1 -- 172.21.15.150:0/2347489644 wait complete. 2024-09-17T05:15:07.211 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-17T05:15:07.211 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:15:07.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:07.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.353+0000 7fdefa187640 1 -- 172.21.15.150:0/2413449773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee0004e70 msgr2=0x7fdee00052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:07.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.353+0000 7fdefa187640 1 --2- 172.21.15.150:0/2413449773 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee0004e70 0x7fdee00052c0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fdedc0016f0 tx=0x7fdedc01c7b0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.353+0000 7fdef3fff640 1 -- 172.21.15.150:0/2413449773 reap_dead start 2024-09-17T05:15:07.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.353+0000 7fdefa187640 1 -- 172.21.15.150:0/2413449773 shutdown_connections 2024-09-17T05:15:07.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.353+0000 7fdefa187640 1 -- 172.21.15.150:0/2413449773 >> 172.21.15.150:0/2413449773 conn(0x7fdef40abb00 msgr2=0x7fdef40abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:07.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.353+0000 7fdefa187640 1 -- 172.21.15.150:0/2413449773 shutdown_connections 2024-09-17T05:15:07.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.353+0000 7fdefa187640 1 -- 172.21.15.150:0/2413449773 wait complete. 2024-09-17T05:15:07.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdefa187640 1 Processor -- start 2024-09-17T05:15:07.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdefa187640 1 -- start start 2024-09-17T05:15:07.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdefa187640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee0004e70 0x7fdef41de480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:07.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdefa187640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdef40ae0d0 0x7fdef40ae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:07.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdefa187640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdef41df170 0x7fdef41df580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:07.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdefa187640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdef41553f0 con 0x7fdee0004e70 2024-09-17T05:15:07.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdefa187640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdef4155110 con 0x7fdef40ae0d0 2024-09-17T05:15:07.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdefa187640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdef40c4be0 con 0x7fdef41df170 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdef37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee0004e70 0x7fdef41de480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdef37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee0004e70 0x7fdef41de480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53876/0 (socket says 172.21.15.150:53876) 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdef37fe640 1 -- 172.21.15.150:0/133957542 learned_addr learned my addr 172.21.15.150:0/133957542 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdef3fff640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdef41df170 0x7fdef41df580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdef2ffd640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdef40ae0d0 0x7fdef40ae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdef37fe640 1 -- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdef41df170 msgr2=0x7fdef41df580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdef37fe640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdef41df170 0x7fdef41df580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdef37fe640 1 -- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdef40ae0d0 msgr2=0x7fdef40ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdef37fe640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdef40ae0d0 0x7fdef40ae4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdef37fe640 1 -- 172.21.15.150:0/133957542 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef40ad030 con 0x7fdee0004e70 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdef2ffd640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdef40ae0d0 0x7fdef40ae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdef37fe640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee0004e70 0x7fdef41de480 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fdeec010b50 tx=0x7fdeec00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:07.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdeebfff640 1 -- 172.21.15.150:0/133957542 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeec003b80 con 0x7fdee0004e70 2024-09-17T05:15:07.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdeebfff640 1 -- 172.21.15.150:0/133957542 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdeec00ee90 con 0x7fdee0004e70 2024-09-17T05:15:07.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdefa187640 1 -- 172.21.15.150:0/133957542 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef40ad310 con 0x7fdee0004e70 2024-09-17T05:15:07.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdeebfff640 1 -- 172.21.15.150:0/133957542 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeec004290 con 0x7fdee0004e70 2024-09-17T05:15:07.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdefa187640 1 -- 172.21.15.150:0/133957542 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef40ad8d0 con 0x7fdee0004e70 2024-09-17T05:15:07.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.357+0000 7fdefa187640 1 -- 172.21.15.150:0/133957542 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef4151d10 con 0x7fdee0004e70 2024-09-17T05:15:07.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.361+0000 7fdeebfff640 1 -- 172.21.15.150:0/133957542 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fdeec00c050 con 0x7fdee0004e70 2024-09-17T05:15:07.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.361+0000 7fdeebfff640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fded4073200 0x7fded4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:07.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.361+0000 7fdeebfff640 1 -- 172.21.15.150:0/133957542 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7fdeec095bd0 con 0x7fdee0004e70 2024-09-17T05:15:07.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.361+0000 7fdef2ffd640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fded4073200 0x7fded4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:07.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.361+0000 7fdef2ffd640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fded4073200 0x7fded4075710 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdee4001a40 tx=0x7fdee4000fa0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:07.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.365+0000 7fdeebfff640 1 -- 172.21.15.150:0/133957542 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdeec05f7a0 con 0x7fdee0004e70 2024-09-17T05:15:07.455 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:07.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.565+0000 7fdefa187640 1 -- 172.21.15.150:0/133957542 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdef40c40f0 con 0x7fdee0004e70 2024-09-17T05:15:07.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.569+0000 7fdeebfff640 1 -- 172.21.15.150:0/133957542 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) ==== 95+0+5711 (secure 0 0 0) 0x7fdeec0641d0 con 0x7fdee0004e70 2024-09-17T05:15:07.567 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:07.567 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":178,"btime":"2024-09-17T05:15:07:238694+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:06.731596+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:15:07.567 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 178 2024-09-17T05:15:07.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.573+0000 7fdefa187640 1 -- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fded4073200 msgr2=0x7fded4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:07.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.573+0000 7fdefa187640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fded4073200 0x7fded4075710 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdee4001a40 tx=0x7fdee4000fa0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.573+0000 7fdefa187640 1 -- 172.21.15.150:0/133957542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee0004e70 msgr2=0x7fdef41de480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:07.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.573+0000 7fdefa187640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee0004e70 0x7fdef41de480 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fdeec010b50 tx=0x7fdeec00e6c0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.573+0000 7fdefa187640 1 -- 172.21.15.150:0/133957542 shutdown_connections 2024-09-17T05:15:07.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.573+0000 7fdefa187640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fded4073200 0x7fded4075710 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.573+0000 7fdefa187640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdef40ae0d0 0x7fdef40ae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.573+0000 7fdefa187640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdef41df170 0x7fdef41df580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.573+0000 7fdefa187640 1 --2- 172.21.15.150:0/133957542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdee0004e70 0x7fdef41de480 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.573+0000 7fdefa187640 1 -- 172.21.15.150:0/133957542 >> 172.21.15.150:0/133957542 conn(0x7fdef40abb00 msgr2=0x7fdef414e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:07.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.573+0000 7fdefa187640 1 -- 172.21.15.150:0/133957542 shutdown_connections 2024-09-17T05:15:07.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.573+0000 7fdefa187640 1 -- 172.21.15.150:0/133957542 wait complete. 2024-09-17T05:15:07.595 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:15:07:238694+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 178, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 178, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7719": { "addr": "172.21.15.150:6835/178583496", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 178583496, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 178583496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7719, "incarnation": 177, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:15:06.731596+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7719 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 170, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7015": { "addr": "172.21.15.176:6837/2338457617", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 2338457617, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 2338457617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7015, "incarnation": 167, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:13:49.801008+0000", "qdb_cluster": [ 7015 ], "qdb_leader": 7015, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7015 } } } ], "standbys": [ { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:15:07.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.605+0000 7fdffafa6640 1 -- 172.21.15.150:0/2894886366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfe0003a10 msgr2=0x7fdfe0003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:07.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.605+0000 7fdffafa6640 1 --2- 172.21.15.150:0/2894886366 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfe0003a10 0x7fdfe0003e80 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00bbe0 tx=0x7fdfdc01ce50 comp rx=0 tx=0).stop 2024-09-17T05:15:07.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.605+0000 7fdff9543640 1 -- 172.21.15.150:0/2894886366 reap_dead start 2024-09-17T05:15:07.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.605+0000 7fdffafa6640 1 -- 172.21.15.150:0/2894886366 shutdown_connections 2024-09-17T05:15:07.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.605+0000 7fdffafa6640 1 -- 172.21.15.150:0/2894886366 >> 172.21.15.150:0/2894886366 conn(0x7fdff4074b80 msgr2=0x7fdff406f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:07.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.605+0000 7fdffafa6640 1 -- 172.21.15.150:0/2894886366 shutdown_connections 2024-09-17T05:15:07.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.605+0000 7fdffafa6640 1 -- 172.21.15.150:0/2894886366 wait complete. 2024-09-17T05:15:07.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdffafa6640 1 Processor -- start 2024-09-17T05:15:07.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdffafa6640 1 -- start start 2024-09-17T05:15:07.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdffafa6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfe0003a10 0x7fdff41f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:07.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdffafa6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdff414f9f0 0x7fdff414fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:07.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdffafa6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdff414bca0 0x7fdff414c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:07.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdffafa6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdff40c9310 con 0x7fdff414bca0 2024-09-17T05:15:07.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdffafa6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdff40c9190 con 0x7fdfe0003a10 2024-09-17T05:15:07.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdffafa6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdff40c39c0 con 0x7fdff414f9f0 2024-09-17T05:15:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff9543640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdff414bca0 0x7fdff414c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff9543640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdff414bca0 0x7fdff414c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53904/0 (socket says 172.21.15.150:53904) 2024-09-17T05:15:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff9543640 1 -- 172.21.15.150:0/3695742861 learned_addr learned my addr 172.21.15.150:0/3695742861 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff9543640 1 -- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdff414f9f0 msgr2=0x7fdff414fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff9543640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdff414f9f0 0x7fdff414fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff9543640 1 -- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfe0003a10 msgr2=0x7fdff41f3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff8d42640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfe0003a10 0x7fdff41f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff9543640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfe0003a10 0x7fdff41f3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff9543640 1 -- 172.21.15.150:0/3695742861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff41f4200 con 0x7fdff414bca0 2024-09-17T05:15:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff9543640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdff414bca0 0x7fdff414c0b0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fdff415ae50 tx=0x7fdfec00fd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff17fa640 1 -- 172.21.15.150:0/3695742861 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec004d30 con 0x7fdff414bca0 2024-09-17T05:15:07.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff17fa640 1 -- 172.21.15.150:0/3695742861 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdfec00ea30 con 0x7fdff414bca0 2024-09-17T05:15:07.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdffafa6640 1 -- 172.21.15.150:0/3695742861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdff41f3f20 con 0x7fdff414bca0 2024-09-17T05:15:07.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdff17fa640 1 -- 172.21.15.150:0/3695742861 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec00ee80 con 0x7fdff414bca0 2024-09-17T05:15:07.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdffafa6640 1 -- 172.21.15.150:0/3695742861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff41514f0 con 0x7fdff414bca0 2024-09-17T05:15:07.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.609+0000 7fdffafa6640 1 -- 172.21.15.150:0/3695742861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdff40c5810 con 0x7fdff414bca0 2024-09-17T05:15:07.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.613+0000 7fdff17fa640 1 -- 172.21.15.150:0/3695742861 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fdfec019030 con 0x7fdff414bca0 2024-09-17T05:15:07.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.613+0000 7fdff17fa640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfd4073200 0x7fdfd4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:07.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.613+0000 7fdff17fa640 1 -- 172.21.15.150:0/3695742861 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7fdfec0958a0 con 0x7fdff414bca0 2024-09-17T05:15:07.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.613+0000 7fdff8d42640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfd4073200 0x7fdfd4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:07.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.613+0000 7fdff8d42640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfd4073200 0x7fdfd4075710 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc0b0000 tx=0x7fdfdc019bf0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:07.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.617+0000 7fdff17fa640 1 -- 172.21.15.150:0/3695742861 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdfec05f3e0 con 0x7fdff414bca0 2024-09-17T05:15:07.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.821+0000 7fdffafa6640 1 -- 172.21.15.150:0/3695742861 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdff4157790 con 0x7fdff414bca0 2024-09-17T05:15:07.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.821+0000 7fdff17fa640 1 -- 172.21.15.150:0/3695742861 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) ==== 95+0+5711 (secure 0 0 0) 0x7fdfec063e10 con 0x7fdff414bca0 2024-09-17T05:15:07.820 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:07.820 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":178,"btime":"2024-09-17T05:15:07:238694+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4355,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:06.731596+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:13:49.801008+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7015},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7015":{"gid":7015,"name":"d","rank":0,"incarnation":167,"state":"up:active","state_seq":24,"addr":"172.21.15.176:6837/2338457617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2338457617},{"type":"v1","addr":"172.21.15.176:6837","nonce":2338457617}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7015,"qdb_cluster":[7015]},"id":2}]} 2024-09-17T05:15:07.821 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 178 2024-09-17T05:15:07.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.825+0000 7fdffafa6640 1 -- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfd4073200 msgr2=0x7fdfd4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:07.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.825+0000 7fdffafa6640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfd4073200 0x7fdfd4075710 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc0b0000 tx=0x7fdfdc019bf0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.825+0000 7fdffafa6640 1 -- 172.21.15.150:0/3695742861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdff414bca0 msgr2=0x7fdff414c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:07.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.825+0000 7fdffafa6640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdff414bca0 0x7fdff414c0b0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fdff415ae50 tx=0x7fdfec00fd90 comp rx=0 tx=0).stop 2024-09-17T05:15:07.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.825+0000 7fdffafa6640 1 -- 172.21.15.150:0/3695742861 shutdown_connections 2024-09-17T05:15:07.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.825+0000 7fdffafa6640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfd4073200 0x7fdfd4075710 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.825+0000 7fdffafa6640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdff414bca0 0x7fdff414c0b0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.825+0000 7fdffafa6640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdff414f9f0 0x7fdff414fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.825+0000 7fdffafa6640 1 --2- 172.21.15.150:0/3695742861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfe0003a10 0x7fdff41f3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:07.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.825+0000 7fdffafa6640 1 -- 172.21.15.150:0/3695742861 >> 172.21.15.150:0/3695742861 conn(0x7fdff4074b80 msgr2=0x7fdff4159e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:07.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.825+0000 7fdffafa6640 1 -- 172.21.15.150:0/3695742861 shutdown_connections 2024-09-17T05:15:07.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:07.825+0000 7fdffafa6640 1 -- 172.21.15.150:0/3695742861 wait complete. 2024-09-17T05:15:07.843 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:15:09.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:09.843 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:09.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.885+0000 7feec52a0640 1 -- 172.21.15.150:0/2065741688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec006e5d0 msgr2=0x7feec0153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:09.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.885+0000 7feec52a0640 1 --2- 172.21.15.150:0/2065741688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec006e5d0 0x7feec0153660 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7feec0075620 tx=0x7feeb801cd30 comp rx=0 tx=0).stop 2024-09-17T05:15:09.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.889+0000 7feec52a0640 1 -- 172.21.15.150:0/2065741688 shutdown_connections 2024-09-17T05:15:09.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.889+0000 7feec52a0640 1 --2- 172.21.15.150:0/2065741688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec006e5d0 0x7feec0153660 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:09.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.889+0000 7feec52a0640 1 --2- 172.21.15.150:0/2065741688 >> v2:172.21.15.150:3300/0 conn(0x7feec006fca0 0x7feec006e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:09.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.889+0000 7feec52a0640 1 --2- 172.21.15.150:0/2065741688 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feec00745c0 0x7feec0150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:09.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.889+0000 7feec52a0640 1 -- 172.21.15.150:0/2065741688 >> 172.21.15.150:0/2065741688 conn(0x7feec00c5ae0 msgr2=0x7feec0073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:09.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.889+0000 7feec52a0640 1 -- 172.21.15.150:0/2065741688 shutdown_connections 2024-09-17T05:15:09.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.889+0000 7feec52a0640 1 -- 172.21.15.150:0/2065741688 wait complete. 2024-09-17T05:15:09.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.901+0000 7feec52a0640 1 Processor -- start 2024-09-17T05:15:09.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.901+0000 7feec52a0640 1 -- start start 2024-09-17T05:15:09.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feec52a0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec006e5d0 0x7feec0151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:09.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feec52a0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec006fca0 0x7feec00c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:09.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feec52a0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feec00745c0 0x7feec014c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:09.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feebe7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec006fca0 0x7feec00c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:09.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feebe7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec006fca0 0x7feec00c6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33454/0 (socket says 172.21.15.150:33454) 2024-09-17T05:15:09.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feebe7fc640 1 -- 172.21.15.150:0/3052204048 learned_addr learned my addr 172.21.15.150:0/3052204048 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:09.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feebf7fe640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feec00745c0 0x7feec014c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:09.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feec00c3020 con 0x7feec006fca0 2024-09-17T05:15:09.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7feec015c010 con 0x7feec006e5d0 2024-09-17T05:15:09.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7feec00c31a0 con 0x7feec00745c0 2024-09-17T05:15:09.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feebe7fc640 1 -- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feec00745c0 msgr2=0x7feec014c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:09.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feebe7fc640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feec00745c0 0x7feec014c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:09.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feebe7fc640 1 -- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec006e5d0 msgr2=0x7feec0151c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:15:09.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feebe7fc640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec006e5d0 0x7feec0151c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:09.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feebe7fc640 1 -- 172.21.15.150:0/3052204048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec01f7fa0 con 0x7feec006fca0 2024-09-17T05:15:09.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.905+0000 7feebe7fc640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec006fca0 0x7feec00c6bb0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7feeb400ca20 tx=0x7feeb400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:09.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.909+0000 7feea77fe640 1 -- 172.21.15.150:0/3052204048 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb4007c70 con 0x7feec006fca0 2024-09-17T05:15:09.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.909+0000 7feea77fe640 1 -- 172.21.15.150:0/3052204048 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feeb400f450 con 0x7feec006fca0 2024-09-17T05:15:09.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.909+0000 7feea77fe640 1 -- 172.21.15.150:0/3052204048 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb400f8a0 con 0x7feec006fca0 2024-09-17T05:15:09.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.909+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec01f8280 con 0x7feec006fca0 2024-09-17T05:15:09.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.909+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec00c70f0 con 0x7feec006fca0 2024-09-17T05:15:09.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.913+0000 7feea77fe640 1 -- 172.21.15.150:0/3052204048 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7feeb4018400 con 0x7feec006fca0 2024-09-17T05:15:09.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.913+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea00052a0 con 0x7feec006fca0 2024-09-17T05:15:09.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.913+0000 7feea77fe640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feeac073250 0x7feeac075760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:09.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.913+0000 7feea77fe640 1 -- 172.21.15.150:0/3052204048 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7feeb4095940 con 0x7feec006fca0 2024-09-17T05:15:09.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.913+0000 7feebeffd640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feeac073250 0x7feeac075760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:09.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.921+0000 7feebeffd640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feeac073250 0x7feeac075760 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7feeb000b880 tx=0x7feeb000b260 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:09.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:09.921+0000 7feea77fe640 1 -- 172.21.15.150:0/3052204048 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7feeb4014030 con 0x7feec006fca0 2024-09-17T05:15:10.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.225+0000 7f0786dc3640 1 -- 172.21.15.150:0/1104217560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07800734d0 msgr2=0x7f078006d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:10.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.225+0000 7f0786dc3640 1 --2- 172.21.15.150:0/1104217560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07800734d0 0x7f078006d560 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0778009bf0 tx=0x7f077801cb10 comp rx=0 tx=0).stop 2024-09-17T05:15:10.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.225+0000 7f0786dc3640 1 -- 172.21.15.150:0/1104217560 shutdown_connections 2024-09-17T05:15:10.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.225+0000 7f0786dc3640 1 --2- 172.21.15.150:0/1104217560 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f07800734d0 0x7f078006d560 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:10.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.225+0000 7f0786dc3640 1 -- 172.21.15.150:0/1104217560 >> 172.21.15.150:0/1104217560 conn(0x7f07800c4e30 msgr2=0x7f078006ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:10.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.225+0000 7f0786dc3640 1 -- 172.21.15.150:0/1104217560 shutdown_connections 2024-09-17T05:15:10.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.225+0000 7f0786dc3640 1 -- 172.21.15.150:0/1104217560 wait complete. 2024-09-17T05:15:10.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.225+0000 7f0786dc3640 1 Processor -- start 2024-09-17T05:15:10.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.229+0000 7f0786dc3640 1 -- start start 2024-09-17T05:15:10.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.229+0000 7f0786dc3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f078006daa0 0x7f07800c6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:10.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f0786dc3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07800734d0 0x7f0780159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:10.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f0786dc3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0780074f70 0x7f07801546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:10.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f0786dc3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0780071dd0 con 0x7f07800734d0 2024-09-17T05:15:10.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f0786dc3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0780071c50 con 0x7f078006daa0 2024-09-17T05:15:10.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f0786dc3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f07800a72a0 con 0x7f0780074f70 2024-09-17T05:15:10.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f077ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07800734d0 0x7f0780159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:10.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f077ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07800734d0 0x7f0780159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33466/0 (socket says 172.21.15.150:33466) 2024-09-17T05:15:10.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f077ffff640 1 -- 172.21.15.150:0/1901714656 learned_addr learned my addr 172.21.15.150:0/1901714656 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:10.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f077ffff640 1 -- 172.21.15.150:0/1901714656 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0780074f70 msgr2=0x7f07801546b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:10.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f077ffff640 1 --2- 172.21.15.150:0/1901714656 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0780074f70 0x7f07801546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:10.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f077ffff640 1 -- 172.21.15.150:0/1901714656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f078006daa0 msgr2=0x7f07800c6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:10.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f077ffff640 1 --2- 172.21.15.150:0/1901714656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f078006daa0 0x7f07800c6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:10.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f077ffff640 1 -- 172.21.15.150:0/1901714656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07800ab160 con 0x7f07800734d0 2024-09-17T05:15:10.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.233+0000 7f077ffff640 1 --2- 172.21.15.150:0/1901714656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07800734d0 0x7f0780159010 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f07780b0040 tx=0x7f0778005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:10.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.245+0000 7f077d7fa640 1 -- 172.21.15.150:0/1901714656 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07780ba020 con 0x7f07800734d0 2024-09-17T05:15:10.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.245+0000 7f077d7fa640 1 -- 172.21.15.150:0/1901714656 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0778003c70 con 0x7f07800734d0 2024-09-17T05:15:10.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.245+0000 7f077d7fa640 1 -- 172.21.15.150:0/1901714656 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07780b2040 con 0x7f07800734d0 2024-09-17T05:15:10.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.245+0000 7f0786dc3640 1 -- 172.21.15.150:0/1901714656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0780157a00 con 0x7f07800734d0 2024-09-17T05:15:10.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.245+0000 7f0786dc3640 1 -- 172.21.15.150:0/1901714656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0780155fc0 con 0x7f07800734d0 2024-09-17T05:15:10.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.253+0000 7f077d7fa640 1 -- 172.21.15.150:0/1901714656 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f07780054f0 con 0x7f07800734d0 2024-09-17T05:15:10.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.253+0000 7f077d7fa640 1 --2- 172.21.15.150:0/1901714656 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f076c073230 0x7f076c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:10.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.253+0000 7f077d7fa640 1 -- 172.21.15.150:0/1901714656 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f077812cfc0 con 0x7f07800734d0 2024-09-17T05:15:10.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.253+0000 7f0786dc3640 1 -- 172.21.15.150:0/1901714656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07800c20d0 con 0x7f07800734d0 2024-09-17T05:15:10.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.253+0000 7f0784b5f640 1 --2- 172.21.15.150:0/1901714656 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f076c073230 0x7f076c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:10.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.269+0000 7f077d7fa640 1 -- 172.21.15.150:0/1901714656 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f07780c4020 con 0x7f07800734d0 2024-09-17T05:15:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.293+0000 7f0784b5f640 1 --2- 172.21.15.150:0/1901714656 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f076c073230 0x7f076c075740 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f0774001590 tx=0x7f0774007a00 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:10.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.553+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feea0005860 con 0x7feec006fca0 2024-09-17T05:15:10.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.553+0000 7feea77fe640 1 -- 172.21.15.150:0/3052204048 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) ==== 95+0+4883 (secure 0 0 0) 0x7feeb405faf0 con 0x7feec006fca0 2024-09-17T05:15:10.552 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:10.552 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":182,"btime":"2024-09-17T05:15:10:275241+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:09.264728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7719,"qdb_cluster":[7719]},"id":1},{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:09.275615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:rejoin","state_seq":388,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:15:10.553 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 182 2024-09-17T05:15:10.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.557+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feeac073250 msgr2=0x7feeac075760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:10.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.557+0000 7feec52a0640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feeac073250 0x7feeac075760 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7feeb000b880 tx=0x7feeb000b260 comp rx=0 tx=0).stop 2024-09-17T05:15:10.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.557+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec006fca0 msgr2=0x7feec00c6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:10.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.557+0000 7feec52a0640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec006fca0 0x7feec00c6bb0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7feeb400ca20 tx=0x7feeb400cee0 comp rx=0 tx=0).stop 2024-09-17T05:15:10.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.557+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 shutdown_connections 2024-09-17T05:15:10.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.557+0000 7feec52a0640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feeac073250 0x7feeac075760 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:10.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.557+0000 7feec52a0640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7feec00745c0 0x7feec014c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:10.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.557+0000 7feec52a0640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feec006fca0 0x7feec00c6bb0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:10.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.557+0000 7feec52a0640 1 --2- 172.21.15.150:0/3052204048 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feec006e5d0 0x7feec0151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:10.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.557+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 >> 172.21.15.150:0/3052204048 conn(0x7feec00c5ae0 msgr2=0x7feec0157d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:10.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.557+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 shutdown_connections 2024-09-17T05:15:10.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.557+0000 7feec52a0640 1 -- 172.21.15.150:0/3052204048 wait complete. 2024-09-17T05:15:10.631 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=7078 2024-09-17T05:15:10.631 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before reviving mds.a 2024-09-17T05:15:10.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.809+0000 7f0786dc3640 1 -- 172.21.15.150:0/1901714656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07800a83e0 con 0x7f07800734d0 2024-09-17T05:15:10.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.813+0000 7f077d7fa640 1 -- 172.21.15.150:0/1901714656 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) ==== 95+0+4883 (secure 0 0 0) 0x7f07780167f0 con 0x7f07800734d0 2024-09-17T05:15:10.811 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:10.811 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":182,"btime":"2024-09-17T05:15:10:275241+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:09.264728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7719,"qdb_cluster":[7719]},"id":1},{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:09.275615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:rejoin","state_seq":388,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:15:10.811 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 182 2024-09-17T05:15:10.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.817+0000 7f0786dc3640 1 -- 172.21.15.150:0/1901714656 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f076c073230 msgr2=0x7f076c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:10.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.817+0000 7f0786dc3640 1 --2- 172.21.15.150:0/1901714656 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f076c073230 0x7f076c075740 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f0774001590 tx=0x7f0774007a00 comp rx=0 tx=0).stop 2024-09-17T05:15:10.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.817+0000 7f0786dc3640 1 -- 172.21.15.150:0/1901714656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07800734d0 msgr2=0x7f0780159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:10.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.817+0000 7f0786dc3640 1 --2- 172.21.15.150:0/1901714656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07800734d0 0x7f0780159010 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f07780b0040 tx=0x7f0778005040 comp rx=0 tx=0).stop 2024-09-17T05:15:10.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.821+0000 7f0786dc3640 1 -- 172.21.15.150:0/1901714656 shutdown_connections 2024-09-17T05:15:10.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.821+0000 7f0786dc3640 1 --2- 172.21.15.150:0/1901714656 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f076c073230 0x7f076c075740 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:10.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.821+0000 7f0786dc3640 1 --2- 172.21.15.150:0/1901714656 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0780074f70 0x7f07801546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:10.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.821+0000 7f0786dc3640 1 --2- 172.21.15.150:0/1901714656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07800734d0 0x7f0780159010 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:10.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.821+0000 7f0786dc3640 1 --2- 172.21.15.150:0/1901714656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f078006daa0 0x7f07800c6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:10.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.821+0000 7f0786dc3640 1 -- 172.21.15.150:0/1901714656 >> 172.21.15.150:0/1901714656 conn(0x7f07800c4e30 msgr2=0x7f07800a8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:10.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.821+0000 7f0786dc3640 1 -- 172.21.15.150:0/1901714656 shutdown_connections 2024-09-17T05:15:10.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:10.821+0000 7f0786dc3640 1 -- 172.21.15.150:0/1901714656 wait complete. 2024-09-17T05:15:10.886 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-17T05:15:10.886 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:15:10.886 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:11.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.229+0000 7f03df2f2640 1 -- 172.21.15.150:0/3102469996 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03d8074e60 msgr2=0x7f03d815df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:11.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.229+0000 7f03df2f2640 1 --2- 172.21.15.150:0/3102469996 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03d8074e60 0x7f03d815df70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009bc0 tx=0x7f03d001cbc0 comp rx=0 tx=0).stop 2024-09-17T05:15:11.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.229+0000 7f03df2f2640 1 -- 172.21.15.150:0/3102469996 shutdown_connections 2024-09-17T05:15:11.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.229+0000 7f03df2f2640 1 --2- 172.21.15.150:0/3102469996 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03d8074e60 0x7f03d815df70 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:11.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.229+0000 7f03df2f2640 1 --2- 172.21.15.150:0/3102469996 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03d80c3900 0x7f03d8074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:11.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.229+0000 7f03df2f2640 1 -- 172.21.15.150:0/3102469996 >> 172.21.15.150:0/3102469996 conn(0x7f03d806e360 msgr2=0x7f03d806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:11.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.229+0000 7f03df2f2640 1 -- 172.21.15.150:0/3102469996 shutdown_connections 2024-09-17T05:15:11.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.229+0000 7f03df2f2640 1 -- 172.21.15.150:0/3102469996 wait complete. 2024-09-17T05:15:11.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.233+0000 7f03df2f2640 1 Processor -- start 2024-09-17T05:15:11.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.233+0000 7f03df2f2640 1 -- start start 2024-09-17T05:15:11.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.233+0000 7f03df2f2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03d8074e60 0x7f03d80ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:11.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.233+0000 7f03df2f2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03d80c3900 0x7f03d80bdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:11.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.233+0000 7f03df2f2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d80c4250 0x7f03d8148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:11.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.233+0000 7f03df2f2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f03d8160cd0 con 0x7f03d80c4250 2024-09-17T05:15:11.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.233+0000 7f03df2f2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f03d8072fc0 con 0x7f03d8074e60 2024-09-17T05:15:11.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.233+0000 7f03df2f2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f03d8160e50 con 0x7f03d80c3900 2024-09-17T05:15:11.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.233+0000 7f03dd88f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d80c4250 0x7f03d8148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:11.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.233+0000 7f03dd88f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d80c4250 0x7f03d8148060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33480/0 (socket says 172.21.15.150:33480) 2024-09-17T05:15:11.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.233+0000 7f03dc88d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03d80c3900 0x7f03d80bdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:11.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.233+0000 7f03dd88f640 1 -- 172.21.15.150:0/1944722888 learned_addr learned my addr 172.21.15.150:0/1944722888 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:11.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.237+0000 7f03dc88d640 1 -- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03d8074e60 msgr2=0x7f03d80ad4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:15:11.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.237+0000 7f03dc88d640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03d8074e60 0x7f03d80ad4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:11.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.237+0000 7f03dc88d640 1 -- 172.21.15.150:0/1944722888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d80c4250 msgr2=0x7f03d8148060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:11.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.237+0000 7f03dc88d640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d80c4250 0x7f03d8148060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:11.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.237+0000 7f03dc88d640 1 -- 172.21.15.150:0/1944722888 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d815dae0 con 0x7f03d80c3900 2024-09-17T05:15:11.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.237+0000 7f03dd88f640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d80c4250 0x7f03d8148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:15:11.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.237+0000 7f03dc88d640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03d80c3900 0x7f03d80bdd60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f03d80a88b0 tx=0x7f03d4009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:11.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.237+0000 7f03cdffb640 1 -- 172.21.15.150:0/1944722888 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d4015020 con 0x7f03d80c3900 2024-09-17T05:15:11.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.237+0000 7f03cdffb640 1 -- 172.21.15.150:0/1944722888 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03d4011920 con 0x7f03d80c3900 2024-09-17T05:15:11.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.237+0000 7f03cdffb640 1 -- 172.21.15.150:0/1944722888 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d4011d70 con 0x7f03d80c3900 2024-09-17T05:15:11.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.237+0000 7f03df2f2640 1 -- 172.21.15.150:0/1944722888 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d80acfc0 con 0x7f03d80c3900 2024-09-17T05:15:11.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.237+0000 7f03df2f2640 1 -- 172.21.15.150:0/1944722888 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d814ebe0 con 0x7f03d80c3900 2024-09-17T05:15:11.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.241+0000 7f03bb7fe640 1 -- 172.21.15.150:0/1944722888 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d8074920 con 0x7f03d80c3900 2024-09-17T05:15:11.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.241+0000 7f03cdffb640 1 -- 172.21.15.150:0/1944722888 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f03d4018020 con 0x7f03d80c3900 2024-09-17T05:15:11.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.241+0000 7f03cdffb640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03bc073280 0x7f03bc075790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:11.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.241+0000 7f03cdffb640 1 -- 172.21.15.150:0/1944722888 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f03d40954d0 con 0x7f03d80c3900 2024-09-17T05:15:11.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.241+0000 7f03dd08e640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03bc073280 0x7f03bc075790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:11.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.245+0000 7f03dd08e640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03bc073280 0x7f03bc075790 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f03c80014e0 tx=0x7f03c80078b0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:11.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:11.253+0000 7f03cdffb640 1 -- 172.21.15.150:0/1944722888 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f03d401f030 con 0x7f03d80c3900 2024-09-17T05:15:11.395 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T05:15:11.397+0000 7ff3775b2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:15:12.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.085+0000 7f03bb7fe640 1 -- 172.21.15.150:0/1944722888 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03d8153900 con 0x7f03d80c3900 2024-09-17T05:15:12.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.085+0000 7f03cdffb640 1 -- 172.21.15.150:0/1944722888 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) ==== 95+0+4890 (secure 0 0 0) 0x7f03d405f6b0 con 0x7f03d80c3900 2024-09-17T05:15:12.086 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:12.086 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":183,"btime":"2024-09-17T05:15:11:291895+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:09.264728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7719,"qdb_cluster":[7719]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:12.086 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 183 2024-09-17T05:15:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.097+0000 7f03df2f2640 1 -- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03bc073280 msgr2=0x7f03bc075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.097+0000 7f03df2f2640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03bc073280 0x7f03bc075790 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f03c80014e0 tx=0x7f03c80078b0 comp rx=0 tx=0).stop 2024-09-17T05:15:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.097+0000 7f03df2f2640 1 -- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03d80c3900 msgr2=0x7f03d80bdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.097+0000 7f03df2f2640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03d80c3900 0x7f03d80bdd60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f03d80a88b0 tx=0x7f03d4009f90 comp rx=0 tx=0).stop 2024-09-17T05:15:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.097+0000 7f03df2f2640 1 -- 172.21.15.150:0/1944722888 shutdown_connections 2024-09-17T05:15:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.097+0000 7f03df2f2640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f03bc073280 0x7f03bc075790 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.097+0000 7f03df2f2640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03d80c4250 0x7f03d8148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.097+0000 7f03df2f2640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f03d80c3900 0x7f03d80bdd60 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f03d80a88b0 tx=0x7f03d4009f90 comp rx=0 tx=0).stop 2024-09-17T05:15:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.097+0000 7f03df2f2640 1 --2- 172.21.15.150:0/1944722888 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f03d8074e60 0x7f03d80ad4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.097+0000 7f03df2f2640 1 -- 172.21.15.150:0/1944722888 >> 172.21.15.150:0/1944722888 conn(0x7f03d806e360 msgr2=0x7f03d80c7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.097+0000 7f03df2f2640 1 -- 172.21.15.150:0/1944722888 shutdown_connections 2024-09-17T05:15:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:12.097+0000 7f03df2f2640 1 -- 172.21.15.150:0/1944722888 wait complete. 2024-09-17T05:15:12.182 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=7015 2024-09-17T05:15:12.182 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.d 2024-09-17T05:15:12.221 INFO:tasks.ceph.mds.b.smithi176.stderr:2024-09-17T05:15:12.216+0000 7fd8dbde5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:15:15.183 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-17T05:15:15.184 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T05:15:15.184 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T05:15:15.186 INFO:tasks.ceph.mds.d:Started 2024-09-17T05:15:15.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:15.413 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-09-17T05:15:15.632 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-17T05:15:15.632 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T05:15:15.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T05:15:15.634 INFO:tasks.ceph.mds.a:Started 2024-09-17T05:15:15.635 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:15.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.905+0000 7fb4cfc52640 1 -- 172.21.15.150:0/1176588658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4c80c4d70 msgr2=0x7fb4c80c5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:15.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.905+0000 7fb4cfc52640 1 --2- 172.21.15.150:0/1176588658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4c80c4d70 0x7fb4c80c5180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009bc0 tx=0x7fb4c001cb00 comp rx=0 tx=0).stop 2024-09-17T05:15:15.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.913+0000 7fb4cfc52640 1 -- 172.21.15.150:0/1176588658 shutdown_connections 2024-09-17T05:15:15.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.913+0000 7fb4cfc52640 1 --2- 172.21.15.150:0/1176588658 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4c80c4d70 0x7fb4c80c5180 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:15.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.913+0000 7fb4cfc52640 1 --2- 172.21.15.150:0/1176588658 >> v2:172.21.15.176:3300/0 conn(0x7fb4c81565f0 0x7fb4c806ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:15.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.913+0000 7fb4cfc52640 1 -- 172.21.15.150:0/1176588658 >> 172.21.15.150:0/1176588658 conn(0x7fb4c8074cc0 msgr2=0x7fb4c80cbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:15.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.933+0000 7fb4cfc52640 1 -- 172.21.15.150:0/1176588658 shutdown_connections 2024-09-17T05:15:15.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.933+0000 7fb4cfc52640 1 -- 172.21.15.150:0/1176588658 wait complete. 2024-09-17T05:15:15.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.933+0000 7fb4cfc52640 1 Processor -- start 2024-09-17T05:15:15.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.937+0000 7fb4cfc52640 1 -- start start 2024-09-17T05:15:15.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4cfc52640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4c806f290 0x7fb4c80cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:15.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4cfc52640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4c80c4d70 0x7fb4c8149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:15.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4cfc52640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4c81565f0 0x7fb4c814dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:15.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4cfc52640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c80c3520 con 0x7fb4c81565f0 2024-09-17T05:15:15.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4cd1ed640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4c80c4d70 0x7fb4c8149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:15.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4cfc52640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c80c33a0 con 0x7fb4c806f290 2024-09-17T05:15:15.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4cfc52640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb4c80c67a0 con 0x7fb4c80c4d70 2024-09-17T05:15:15.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4cd1ed640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4c80c4d70 0x7fb4c8149000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:36782/0 (socket says 172.21.15.150:36782) 2024-09-17T05:15:15.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4cd1ed640 1 -- 172.21.15.150:0/946261291 learned_addr learned my addr 172.21.15.150:0/946261291 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:15.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4ce1ef640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4c81565f0 0x7fb4c814dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:15.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4ce1ef640 1 -- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4c80c4d70 msgr2=0x7fb4c8149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:15.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4ce1ef640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4c80c4d70 0x7fb4c8149000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:15.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4ce1ef640 1 -- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4c806f290 msgr2=0x7fb4c80cb490 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:15:15.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4ce1ef640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4c806f290 0x7fb4c80cb490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:15.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4ce1ef640 1 -- 172.21.15.150:0/946261291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c81fca00 con 0x7fb4c81565f0 2024-09-17T05:15:15.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.941+0000 7fb4cd1ed640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4c80c4d70 0x7fb4c8149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:15:15.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.945+0000 7fb4ce1ef640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4c81565f0 0x7fb4c814dea0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00ac000 tx=0x7fb4c00a5a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:15.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.953+0000 7fb4be7fc640 1 -- 172.21.15.150:0/946261291 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c00b2040 con 0x7fb4c81565f0 2024-09-17T05:15:15.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.953+0000 7fb4cfc52640 1 -- 172.21.15.150:0/946261291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c80c9b90 con 0x7fb4c81565f0 2024-09-17T05:15:15.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.953+0000 7fb4cfc52640 1 -- 172.21.15.150:0/946261291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c814ee50 con 0x7fb4c81565f0 2024-09-17T05:15:15.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.953+0000 7fb4cfc52640 1 -- 172.21.15.150:0/946261291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4ac0052a0 con 0x7fb4c81565f0 2024-09-17T05:15:15.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.969+0000 7fb4be7fc640 1 -- 172.21.15.150:0/946261291 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb4c0003a70 con 0x7fb4c81565f0 2024-09-17T05:15:15.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.973+0000 7fb4be7fc640 1 -- 172.21.15.150:0/946261291 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c00ae770 con 0x7fb4c81565f0 2024-09-17T05:15:15.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.977+0000 7fb4be7fc640 1 -- 172.21.15.150:0/946261291 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb4c00be020 con 0x7fb4c81565f0 2024-09-17T05:15:15.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.985+0000 7fb4be7fc640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb4b4073450 0x7fb4b4075960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:15.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.985+0000 7fb4be7fc640 1 -- 172.21.15.150:0/946261291 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7fb4c012f430 con 0x7fb4c81565f0 2024-09-17T05:15:15.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.985+0000 7fb4cd9ee640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb4b4073450 0x7fb4b4075960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:15.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.989+0000 7fb4cd9ee640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb4b4073450 0x7fb4b4075960 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40099d0 tx=0x7fb4c4009260 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:15.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:15.993+0000 7fb4be7fc640 1 -- 172.21.15.150:0/946261291 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb4c00167b0 con 0x7fb4c81565f0 2024-09-17T05:15:16.119 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T05:15:16.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.337+0000 7f243791d640 1 -- 172.21.15.150:0/1989553548 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24300a73f0 msgr2=0x7f2430070610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:16.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.337+0000 7f243791d640 1 --2- 172.21.15.150:0/1989553548 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24300a73f0 0x7f2430070610 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f242c009fd0 tx=0x7f242c01da70 comp rx=0 tx=0).stop 2024-09-17T05:15:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.353+0000 7f243791d640 1 -- 172.21.15.150:0/1989553548 shutdown_connections 2024-09-17T05:15:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.353+0000 7f243791d640 1 --2- 172.21.15.150:0/1989553548 >> v2:172.21.15.176:3300/0 conn(0x7f24300c2e30 0x7f243016db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.353+0000 7f243791d640 1 --2- 172.21.15.150:0/1989553548 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f24300a73f0 0x7f2430070610 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.353+0000 7f243791d640 1 -- 172.21.15.150:0/1989553548 >> 172.21.15.150:0/1989553548 conn(0x7f24300c7f30 msgr2=0x7f24300c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.357+0000 7f243791d640 1 -- 172.21.15.150:0/1989553548 shutdown_connections 2024-09-17T05:15:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.357+0000 7f243791d640 1 -- 172.21.15.150:0/1989553548 wait complete. 2024-09-17T05:15:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.357+0000 7f243791d640 1 Processor -- start 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.357+0000 7f243791d640 1 -- start start 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.361+0000 7f243791d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2430070b50 0x7f24300ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.361+0000 7f243791d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24300a73f0 0x7f24300c0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.361+0000 7f243791d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24300c2e30 0x7f24300a8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.361+0000 7f243791d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f243008c050 con 0x7f24300c2e30 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.361+0000 7f243791d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f243008bed0 con 0x7f24300a73f0 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.361+0000 7f243791d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f243008bb50 con 0x7f2430070b50 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.361+0000 7f2435eba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24300c2e30 0x7f24300a8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.361+0000 7f2435eba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24300c2e30 0x7f24300a8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33526/0 (socket says 172.21.15.150:33526) 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.361+0000 7f2435eba640 1 -- 172.21.15.150:0/2132018094 learned_addr learned my addr 172.21.15.150:0/2132018094 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.365+0000 7f2435eba640 1 -- 172.21.15.150:0/2132018094 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2430070b50 msgr2=0x7f24300ad010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.365+0000 7f2435eba640 1 --2- 172.21.15.150:0/2132018094 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2430070b50 0x7f24300ad010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.365+0000 7f2435eba640 1 -- 172.21.15.150:0/2132018094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24300a73f0 msgr2=0x7f24300c0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:16.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.365+0000 7f2435eba640 1 --2- 172.21.15.150:0/2132018094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24300a73f0 0x7f24300c0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:16.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.365+0000 7f2435eba640 1 -- 172.21.15.150:0/2132018094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24300ad8b0 con 0x7f24300c2e30 2024-09-17T05:15:16.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.365+0000 7f2435eba640 1 --2- 172.21.15.150:0/2132018094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24300c2e30 0x7f24300a8b20 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f242800b960 tx=0x7f242800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.369+0000 7f2425ffb640 1 -- 172.21.15.150:0/2132018094 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2428019070 con 0x7f24300c2e30 2024-09-17T05:15:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.369+0000 7f2425ffb640 1 -- 172.21.15.150:0/2132018094 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f242800cc10 con 0x7f24300c2e30 2024-09-17T05:15:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.369+0000 7f2425ffb640 1 -- 172.21.15.150:0/2132018094 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2428014440 con 0x7f24300c2e30 2024-09-17T05:15:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.373+0000 7f243791d640 1 -- 172.21.15.150:0/2132018094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2430161b60 con 0x7f24300c2e30 2024-09-17T05:15:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.373+0000 7f243791d640 1 -- 172.21.15.150:0/2132018094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f243008a0b0 con 0x7f24300c2e30 2024-09-17T05:15:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.373+0000 7f2425ffb640 1 -- 172.21.15.150:0/2132018094 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f242801c070 con 0x7f24300c2e30 2024-09-17T05:15:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.377+0000 7f2425ffb640 1 --2- 172.21.15.150:0/2132018094 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2414073200 0x7f2414075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.377+0000 7f2425ffb640 1 -- 172.21.15.150:0/2132018094 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f2428094d40 con 0x7f24300c2e30 2024-09-17T05:15:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.377+0000 7f24356b9640 1 --2- 172.21.15.150:0/2132018094 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2414073200 0x7f2414075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.377+0000 7f24356b9640 1 --2- 172.21.15.150:0/2132018094 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2414073200 0x7f2414075710 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f242c0af000 tx=0x7f242c01c2b0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:16.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.389+0000 7f243791d640 1 -- 172.21.15.150:0/2132018094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24300c3320 con 0x7f24300c2e30 2024-09-17T05:15:16.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:16.397+0000 7f2425ffb640 1 -- 172.21.15.150:0/2132018094 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f242805e850 con 0x7f24300c2e30 2024-09-17T05:15:17.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.505+0000 7f243791d640 1 -- 172.21.15.150:0/2132018094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2430074220 con 0x7f24300c2e30 2024-09-17T05:15:17.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.509+0000 7f2425ffb640 1 -- 172.21.15.150:0/2132018094 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+6544 (secure 0 0 0) 0x7f2428063280 con 0x7f24300c2e30 2024-09-17T05:15:17.515 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:17.515 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":184,"btime":"2024-09-17T05:15:16:325986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7854,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:09.264728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7719,"qdb_cluster":[7719]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:17.515 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 184 2024-09-17T05:15:17.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.529+0000 7f243791d640 1 -- 172.21.15.150:0/2132018094 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2414073200 msgr2=0x7f2414075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:17.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.529+0000 7f243791d640 1 --2- 172.21.15.150:0/2132018094 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2414073200 0x7f2414075710 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f242c0af000 tx=0x7f242c01c2b0 comp rx=0 tx=0).stop 2024-09-17T05:15:17.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.529+0000 7f243791d640 1 -- 172.21.15.150:0/2132018094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24300c2e30 msgr2=0x7f24300a8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:17.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.529+0000 7f243791d640 1 --2- 172.21.15.150:0/2132018094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24300c2e30 0x7f24300a8b20 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f242800b960 tx=0x7f242800be20 comp rx=0 tx=0).stop 2024-09-17T05:15:17.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.529+0000 7f243791d640 1 -- 172.21.15.150:0/2132018094 shutdown_connections 2024-09-17T05:15:17.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.529+0000 7f243791d640 1 --2- 172.21.15.150:0/2132018094 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2414073200 0x7f2414075710 secure :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f242c0af000 tx=0x7f242c01c2b0 comp rx=0 tx=0).stop 2024-09-17T05:15:17.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.529+0000 7f243791d640 1 --2- 172.21.15.150:0/2132018094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24300a73f0 0x7f24300c0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:17.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.529+0000 7f243791d640 1 --2- 172.21.15.150:0/2132018094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24300c2e30 0x7f24300a8b20 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:17.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.529+0000 7f243791d640 1 --2- 172.21.15.150:0/2132018094 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2430070b50 0x7f24300ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:17.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.529+0000 7f243791d640 1 -- 172.21.15.150:0/2132018094 >> 172.21.15.150:0/2132018094 conn(0x7f24300c7f30 msgr2=0x7f2430162dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:17.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.537+0000 7f243791d640 1 -- 172.21.15.150:0/2132018094 shutdown_connections 2024-09-17T05:15:17.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.537+0000 7f243791d640 1 -- 172.21.15.150:0/2132018094 wait complete. 2024-09-17T05:15:17.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.549+0000 7fb4cfc52640 1 -- 172.21.15.150:0/946261291 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4ac005590 con 0x7fb4c81565f0 2024-09-17T05:15:17.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.549+0000 7fb4be7fc640 1 -- 172.21.15.150:0/946261291 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+6544 (secure 0 0 0) 0x7fb4c00f94f0 con 0x7fb4c81565f0 2024-09-17T05:15:17.553 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:17.553 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":184,"btime":"2024-09-17T05:15:16:325986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7854,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:09.264728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7719,"qdb_cluster":[7719]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:17.554 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 184 2024-09-17T05:15:17.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.565+0000 7fb4abfff640 1 -- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb4b4073450 msgr2=0x7fb4b4075960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:17.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.565+0000 7fb4abfff640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb4b4073450 0x7fb4b4075960 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40099d0 tx=0x7fb4c4009260 comp rx=0 tx=0).stop 2024-09-17T05:15:17.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.565+0000 7fb4abfff640 1 -- 172.21.15.150:0/946261291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4c81565f0 msgr2=0x7fb4c814dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:17.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.565+0000 7fb4abfff640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4c81565f0 0x7fb4c814dea0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00ac000 tx=0x7fb4c00a5a70 comp rx=0 tx=0).stop 2024-09-17T05:15:17.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.565+0000 7fb4abfff640 1 -- 172.21.15.150:0/946261291 shutdown_connections 2024-09-17T05:15:17.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.565+0000 7fb4abfff640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb4b4073450 0x7fb4b4075960 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:17.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.565+0000 7fb4abfff640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4c81565f0 0x7fb4c814dea0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:17.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.565+0000 7fb4abfff640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb4c80c4d70 0x7fb4c8149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:17.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.565+0000 7fb4abfff640 1 --2- 172.21.15.150:0/946261291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb4c806f290 0x7fb4c80cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:17.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.565+0000 7fb4abfff640 1 -- 172.21.15.150:0/946261291 >> 172.21.15.150:0/946261291 conn(0x7fb4c8074cc0 msgr2=0x7fb4c81465b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:17.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.565+0000 7fb4abfff640 1 -- 172.21.15.150:0/946261291 shutdown_connections 2024-09-17T05:15:17.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:17.565+0000 7fb4abfff640 1 -- 172.21.15.150:0/946261291 wait complete. 2024-09-17T05:15:17.732 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-17T05:15:17.732 INFO:tasks.mds_thrash.fs.[b]:waiting for 82 secs before thrashing 2024-09-17T05:15:17.748 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-17T05:15:17.748 INFO:tasks.mds_thrash.fs.[a]:waiting for 16 secs before thrashing 2024-09-17T05:15:33.750 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:34.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.129+0000 7f8cab3db640 1 -- 172.21.15.150:0/3013845140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ca40c8560 msgr2=0x7f8ca40abdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:34.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.129+0000 7f8cab3db640 1 --2- 172.21.15.150:0/3013845140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ca40c8560 0x7f8ca40abdd0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009890 tx=0x7f8ca001c8a0 comp rx=0 tx=0).stop 2024-09-17T05:15:34.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.133+0000 7f8cab3db640 1 -- 172.21.15.150:0/3013845140 shutdown_connections 2024-09-17T05:15:34.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.133+0000 7f8cab3db640 1 --2- 172.21.15.150:0/3013845140 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ca4167630 0x7f8ca4167a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:34.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.133+0000 7f8cab3db640 1 --2- 172.21.15.150:0/3013845140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ca40c8560 0x7f8ca40abdd0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:34.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.133+0000 7f8cab3db640 1 -- 172.21.15.150:0/3013845140 >> 172.21.15.150:0/3013845140 conn(0x7f8ca40a9ca0 msgr2=0x7f8ca40aa0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:34.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.133+0000 7f8cab3db640 1 -- 172.21.15.150:0/3013845140 shutdown_connections 2024-09-17T05:15:34.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.133+0000 7f8cab3db640 1 -- 172.21.15.150:0/3013845140 wait complete. 2024-09-17T05:15:34.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.133+0000 7f8cab3db640 1 Processor -- start 2024-09-17T05:15:34.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.133+0000 7f8cab3db640 1 -- start start 2024-09-17T05:15:34.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8cab3db640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ca40c8150 0x7f8ca40be120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:34.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8cab3db640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ca40c8560 0x7f8ca415ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:34.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8cab3db640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ca4167630 0x7f8ca415e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:34.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8ca9978640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ca4167630 0x7f8ca415e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:34.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8ca9978640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ca4167630 0x7f8ca415e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:57626/0 (socket says 172.21.15.150:57626) 2024-09-17T05:15:34.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8ca9978640 1 -- 172.21.15.150:0/3793310338 learned_addr learned my addr 172.21.15.150:0/3793310338 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:34.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca40c2430 con 0x7f8ca4167630 2024-09-17T05:15:34.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41684a0 con 0x7f8ca40c8150 2024-09-17T05:15:34.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8ca40c25b0 con 0x7f8ca40c8560 2024-09-17T05:15:34.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8ca9978640 1 -- 172.21.15.150:0/3793310338 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ca40c8560 msgr2=0x7f8ca415ce40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:15:34.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8ca9978640 1 --2- 172.21.15.150:0/3793310338 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ca40c8560 0x7f8ca415ce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:34.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8ca9978640 1 -- 172.21.15.150:0/3793310338 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ca40c8150 msgr2=0x7f8ca40be120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:34.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8ca9978640 1 --2- 172.21.15.150:0/3793310338 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ca40c8150 0x7f8ca40be120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:34.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.137+0000 7f8ca9978640 1 -- 172.21.15.150:0/3793310338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca40be660 con 0x7f8ca4167630 2024-09-17T05:15:34.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.141+0000 7f8ca9978640 1 --2- 172.21.15.150:0/3793310338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ca4167630 0x7f8ca415e310 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400b410 tx=0x7f8c9400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:34.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.145+0000 7f8c99ffb640 1 -- 172.21.15.150:0/3793310338 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c94015020 con 0x7f8ca4167630 2024-09-17T05:15:34.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.145+0000 7f8c99ffb640 1 -- 172.21.15.150:0/3793310338 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c940119e0 con 0x7f8ca4167630 2024-09-17T05:15:34.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.145+0000 7f8c99ffb640 1 -- 172.21.15.150:0/3793310338 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c94011e30 con 0x7f8ca4167630 2024-09-17T05:15:34.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.145+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ca40c13a0 con 0x7f8ca4167630 2024-09-17T05:15:34.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.145+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ca40c1680 con 0x7f8ca4167630 2024-09-17T05:15:34.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.149+0000 7f8c99ffb640 1 -- 172.21.15.150:0/3793310338 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8c94018020 con 0x7f8ca4167630 2024-09-17T05:15:34.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.149+0000 7f8c99ffb640 1 --2- 172.21.15.150:0/3793310338 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c880732d0 0x7f8c880757e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:34.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.149+0000 7f8c99ffb640 1 -- 172.21.15.150:0/3793310338 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f8c940950c0 con 0x7f8ca4167630 2024-09-17T05:15:34.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.149+0000 7f8ca9177640 1 --2- 172.21.15.150:0/3793310338 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c880732d0 0x7f8c880757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:34.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.161+0000 7f8ca9177640 1 --2- 172.21.15.150:0/3793310338 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c880732d0 0x7f8c880757e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40c4440 tx=0x7f8c9c001660 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:34.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.169+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c840052a0 con 0x7f8ca4167630 2024-09-17T05:15:34.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:34.181+0000 7f8c99ffb640 1 -- 172.21.15.150:0/3793310338 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8c9401f020 con 0x7f8ca4167630 2024-09-17T05:15:35.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.081+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c84005590 con 0x7f8ca4167630 2024-09-17T05:15:35.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.081+0000 7f8c99ffb640 1 -- 172.21.15.150:0/3793310338 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+6544 (secure 0 0 0) 0x7f8c9405f2a0 con 0x7f8ca4167630 2024-09-17T05:15:35.089 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:35.089 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":184,"btime":"2024-09-17T05:15:16:325986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7854,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:09.264728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7719,"qdb_cluster":[7719]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:35.089 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 184 2024-09-17T05:15:35.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.101+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c880732d0 msgr2=0x7f8c880757e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:35.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.101+0000 7f8cab3db640 1 --2- 172.21.15.150:0/3793310338 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c880732d0 0x7f8c880757e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40c4440 tx=0x7f8c9c001660 comp rx=0 tx=0).stop 2024-09-17T05:15:35.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.101+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ca4167630 msgr2=0x7f8ca415e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:35.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.101+0000 7f8cab3db640 1 --2- 172.21.15.150:0/3793310338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ca4167630 0x7f8ca415e310 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f8c9400b410 tx=0x7f8c9400b8d0 comp rx=0 tx=0).stop 2024-09-17T05:15:35.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.121+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 shutdown_connections 2024-09-17T05:15:35.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.121+0000 7f8cab3db640 1 --2- 172.21.15.150:0/3793310338 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c880732d0 0x7f8c880757e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:35.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.121+0000 7f8cab3db640 1 --2- 172.21.15.150:0/3793310338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ca4167630 0x7f8ca415e310 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:35.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.121+0000 7f8cab3db640 1 --2- 172.21.15.150:0/3793310338 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ca40c8560 0x7f8ca415ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:35.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.121+0000 7f8cab3db640 1 --2- 172.21.15.150:0/3793310338 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ca40c8150 0x7f8ca40be120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:35.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.121+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 >> 172.21.15.150:0/3793310338 conn(0x7f8ca40a9ca0 msgr2=0x7f8ca40c3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:35.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.121+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 shutdown_connections 2024-09-17T05:15:35.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:35.121+0000 7f8cab3db640 1 -- 172.21.15.150:0/3793310338 wait complete. 2024-09-17T05:15:35.269 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-17T05:15:35.270 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T05:15:35.270 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:15:35.309 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T05:15:35.309 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:35.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:15:35.395 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:15:36.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.472+0000 7f29048d0640 1 -- 172.21.15.150:0/3936536385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28e800b800 msgr2=0x7f28e800dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:36.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.472+0000 7f28fdd9b640 1 -- 172.21.15.150:0/3936536385 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0006cb0 con 0x7f28e800b800 2024-09-17T05:15:36.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.472+0000 7f29048d0640 1 --2- 172.21.15.150:0/3936536385 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28e800b800 0x7f28e800dce0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f28f000c2e0 tx=0x7f28f001d060 comp rx=0 tx=0).stop 2024-09-17T05:15:36.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.472+0000 7f28ff59e640 1 -- 172.21.15.150:0/3936536385 reap_dead start 2024-09-17T05:15:36.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.472+0000 7f29048d0640 1 -- 172.21.15.150:0/3936536385 shutdown_connections 2024-09-17T05:15:36.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.472+0000 7f29048d0640 1 -- 172.21.15.150:0/3936536385 >> 172.21.15.150:0/3936536385 conn(0x7f290006ce50 msgr2=0x7f290006d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:36.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.476+0000 7f29048d0640 1 -- 172.21.15.150:0/3936536385 shutdown_connections 2024-09-17T05:15:36.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.476+0000 7f29048d0640 1 -- 172.21.15.150:0/3936536385 wait complete. 2024-09-17T05:15:36.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.480+0000 7f29048d0640 1 Processor -- start 2024-09-17T05:15:36.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.480+0000 7f29048d0640 1 -- start start 2024-09-17T05:15:36.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f29048d0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28e800b800 0x7f29001599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:36.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f29048d0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29000aa220 0x7f2900151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:36.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f29048d0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29001ef150 0x7f290014e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:36.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f29048d0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f29000ac0d0 con 0x7f29000aa220 2024-09-17T05:15:36.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f29048d0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f29000c4ec0 con 0x7f29001ef150 2024-09-17T05:15:36.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f29048d0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f29000ac250 con 0x7f28e800b800 2024-09-17T05:15:36.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28fed9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28e800b800 0x7f29001599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:36.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28fed9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28e800b800 0x7f29001599c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:56658/0 (socket says 172.21.15.150:56658) 2024-09-17T05:15:36.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28fed9d640 1 -- 172.21.15.150:0/400111720 learned_addr learned my addr 172.21.15.150:0/400111720 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:36.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28ff59e640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29001ef150 0x7f290014e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:36.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28fe59c640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29000aa220 0x7f2900151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:36.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28fed9d640 1 -- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29001ef150 msgr2=0x7f290014e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:36.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28fed9d640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29001ef150 0x7f290014e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:36.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28fed9d640 1 -- 172.21.15.150:0/400111720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29000aa220 msgr2=0x7f2900151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:36.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28fed9d640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29000aa220 0x7f2900151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:36.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28fed9d640 1 -- 172.21.15.150:0/400111720 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290014f3a0 con 0x7f28e800b800 2024-09-17T05:15:36.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28fed9d640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28e800b800 0x7f29001599c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f28f8007db0 tx=0x7f28f800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:36.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28fe59c640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29000aa220 0x7f2900151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:15:36.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.484+0000 7f28ff59e640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29001ef150 0x7f290014e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:15:36.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.496+0000 7f28e77fe640 1 -- 172.21.15.150:0/400111720 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f800edd0 con 0x7f28e800b800 2024-09-17T05:15:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.496+0000 7f29048d0640 1 -- 172.21.15.150:0/400111720 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29001fda90 con 0x7f28e800b800 2024-09-17T05:15:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.496+0000 7f29048d0640 1 -- 172.21.15.150:0/400111720 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f290014f730 con 0x7f28e800b800 2024-09-17T05:15:36.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.500+0000 7f28e77fe640 1 -- 172.21.15.150:0/400111720 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28f8003a70 con 0x7f28e800b800 2024-09-17T05:15:36.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.500+0000 7f28e77fe640 1 -- 172.21.15.150:0/400111720 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f8004020 con 0x7f28e800b800 2024-09-17T05:15:36.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.512+0000 7f28e77fe640 1 -- 172.21.15.150:0/400111720 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f28f801b020 con 0x7f28e800b800 2024-09-17T05:15:36.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.512+0000 7f29048d0640 1 -- 172.21.15.150:0/400111720 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e00052a0 con 0x7f28e800b800 2024-09-17T05:15:36.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.512+0000 7f28e77fe640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f28ec0732d0 0x7f28ec0757e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:36.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.512+0000 7f28e77fe640 1 -- 172.21.15.150:0/400111720 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f28f8095320 con 0x7f28e800b800 2024-09-17T05:15:36.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.516+0000 7f28fe59c640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f28ec0732d0 0x7f28ec0757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:36.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.516+0000 7f28fe59c640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f28ec0732d0 0x7f28ec0757e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f29001efc40 tx=0x7f28f4008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:36.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:36.544+0000 7f28e77fe640 1 -- 172.21.15.150:0/400111720 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f28f805f500 con 0x7f28e800b800 2024-09-17T05:15:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.940+0000 7f29048d0640 1 -- 172.21.15.150:0/400111720 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28e0005590 con 0x7f28e800b800 2024-09-17T05:15:37.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.944+0000 7f28e77fe640 1 -- 172.21.15.150:0/400111720 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+6544 (secure 0 0 0) 0x7f28f805ee00 con 0x7f28e800b800 2024-09-17T05:15:37.956 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:37.956 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":184,"btime":"2024-09-17T05:15:16:325986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7854,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:09.264728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7719,"qdb_cluster":[7719]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:37.956 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 184 2024-09-17T05:15:37.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.988+0000 7f29048d0640 1 -- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f28ec0732d0 msgr2=0x7f28ec0757e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:37.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.988+0000 7f29048d0640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f28ec0732d0 0x7f28ec0757e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f29001efc40 tx=0x7f28f4008040 comp rx=0 tx=0).stop 2024-09-17T05:15:37.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.988+0000 7f29048d0640 1 -- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28e800b800 msgr2=0x7f29001599c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:37.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.988+0000 7f29048d0640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28e800b800 0x7f29001599c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f28f8007db0 tx=0x7f28f800e4b0 comp rx=0 tx=0).stop 2024-09-17T05:15:37.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.988+0000 7f29048d0640 1 -- 172.21.15.150:0/400111720 shutdown_connections 2024-09-17T05:15:37.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.988+0000 7f29048d0640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f28ec0732d0 0x7f28ec0757e0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:37.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.992+0000 7f29048d0640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29001ef150 0x7f290014e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:37.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.992+0000 7f29048d0640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29000aa220 0x7f2900151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:37.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.992+0000 7f29048d0640 1 --2- 172.21.15.150:0/400111720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28e800b800 0x7f29001599c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:37.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.992+0000 7f29048d0640 1 -- 172.21.15.150:0/400111720 >> 172.21.15.150:0/400111720 conn(0x7f290006ce50 msgr2=0x7f2900073a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:37.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.992+0000 7f29048d0640 1 -- 172.21.15.150:0/400111720 shutdown_connections 2024-09-17T05:15:37.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:37.992+0000 7f29048d0640 1 -- 172.21.15.150:0/400111720 wait complete. 2024-09-17T05:15:38.145 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:15:40.146 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:40.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.440+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2510800296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf880c8230 msgr2=0x7fdf88074870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:40.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.440+0000 7fdf8cf66640 1 --2- 172.21.15.150:0/2510800296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf880c8230 0x7fdf88074870 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009900 tx=0x7fdf7801c8e0 comp rx=0 tx=0).stop 2024-09-17T05:15:40.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.440+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2510800296 shutdown_connections 2024-09-17T05:15:40.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.440+0000 7fdf8cf66640 1 --2- 172.21.15.150:0/2510800296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf880c7140 0x7fdf880c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:40.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.440+0000 7fdf8cf66640 1 --2- 172.21.15.150:0/2510800296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf880c8230 0x7fdf88074870 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:40.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.440+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2510800296 >> 172.21.15.150:0/2510800296 conn(0x7fdf8806e360 msgr2=0x7fdf8806e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:40.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.440+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2510800296 shutdown_connections 2024-09-17T05:15:40.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.440+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2510800296 wait complete. 2024-09-17T05:15:40.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8cf66640 1 Processor -- start 2024-09-17T05:15:40.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8cf66640 1 -- start start 2024-09-17T05:15:40.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8cf66640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf88074db0 0x7fdf881549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:40.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8cf66640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf880c7140 0x7fdf881f0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:40.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8cf66640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf880c8230 0x7fdf88153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:40.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8cf66640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdf881617b0 con 0x7fdf88074db0 2024-09-17T05:15:40.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8cf66640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdf88157d80 con 0x7fdf880c8230 2024-09-17T05:15:40.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8cf66640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdf88161930 con 0x7fdf880c7140 2024-09-17T05:15:40.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf88074db0 0x7fdf881549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:40.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf88074db0 0x7fdf881549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32816/0 (socket says 172.21.15.150:32816) 2024-09-17T05:15:40.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8659c640 1 -- 172.21.15.150:0/2250542011 learned_addr learned my addr 172.21.15.150:0/2250542011 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:40.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf86d9d640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf880c8230 0x7fdf88153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:40.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf85d9b640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf880c7140 0x7fdf881f0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:40.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8659c640 1 -- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf880c7140 msgr2=0x7fdf881f0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:40.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8659c640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf880c7140 0x7fdf881f0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:40.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8659c640 1 -- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf880c8230 msgr2=0x7fdf88153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:40.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8659c640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf880c8230 0x7fdf88153880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:40.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8659c640 1 -- 172.21.15.150:0/2250542011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8815a230 con 0x7fdf88074db0 2024-09-17T05:15:40.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf8659c640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf88074db0 0x7fdf881549b0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fdf780b0040 tx=0x7fdf78005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:40.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.444+0000 7fdf76ffd640 1 -- 172.21.15.150:0/2250542011 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780b2040 con 0x7fdf88074db0 2024-09-17T05:15:40.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.448+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2250542011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf88154ef0 con 0x7fdf88074db0 2024-09-17T05:15:40.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.448+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2250542011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf88158ef0 con 0x7fdf88074db0 2024-09-17T05:15:40.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.448+0000 7fdf76ffd640 1 -- 172.21.15.150:0/2250542011 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf78004230 con 0x7fdf88074db0 2024-09-17T05:15:40.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.448+0000 7fdf76ffd640 1 -- 172.21.15.150:0/2250542011 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78005020 con 0x7fdf88074db0 2024-09-17T05:15:40.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.448+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2250542011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf6c0052a0 con 0x7fdf88074db0 2024-09-17T05:15:40.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.448+0000 7fdf76ffd640 1 -- 172.21.15.150:0/2250542011 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fdf780051c0 con 0x7fdf88074db0 2024-09-17T05:15:40.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.448+0000 7fdf76ffd640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdf68073230 0x7fdf68075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:40.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.448+0000 7fdf76ffd640 1 -- 172.21.15.150:0/2250542011 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7fdf7812ea30 con 0x7fdf88074db0 2024-09-17T05:15:40.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.448+0000 7fdf85d9b640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdf68073230 0x7fdf68075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:40.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.452+0000 7fdf85d9b640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdf68073230 0x7fdf68075740 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fdf880aeff0 tx=0x7fdf7c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:40.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.456+0000 7fdf76ffd640 1 -- 172.21.15.150:0/2250542011 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdf780164d0 con 0x7fdf88074db0 2024-09-17T05:15:40.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.752+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2250542011 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf6c005590 con 0x7fdf88074db0 2024-09-17T05:15:40.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf76ffd640 1 -- 172.21.15.150:0/2250542011 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+6544 (secure 0 0 0) 0x7fdf780f8520 con 0x7fdf88074db0 2024-09-17T05:15:40.755 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:40.755 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":184,"btime":"2024-09-17T05:15:16:325986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7854,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:09.264728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7719,"qdb_cluster":[7719]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:40.755 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 184 2024-09-17T05:15:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdf68073230 msgr2=0x7fdf68075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf8cf66640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdf68073230 0x7fdf68075740 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fdf880aeff0 tx=0x7fdf7c009040 comp rx=0 tx=0).stop 2024-09-17T05:15:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2250542011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf88074db0 msgr2=0x7fdf881549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf8cf66640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf88074db0 0x7fdf881549b0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fdf780b0040 tx=0x7fdf78005760 comp rx=0 tx=0).stop 2024-09-17T05:15:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2250542011 shutdown_connections 2024-09-17T05:15:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf8cf66640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdf68073230 0x7fdf68075740 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf8cf66640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdf880c8230 0x7fdf88153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf8cf66640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdf880c7140 0x7fdf881f0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:40.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf8cf66640 1 --2- 172.21.15.150:0/2250542011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdf88074db0 0x7fdf881549b0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:40.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2250542011 >> 172.21.15.150:0/2250542011 conn(0x7fdf8806e360 msgr2=0x7fdf88157460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:40.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2250542011 shutdown_connections 2024-09-17T05:15:40.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:40.756+0000 7fdf8cf66640 1 -- 172.21.15.150:0/2250542011 wait complete. 2024-09-17T05:15:40.783 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:15:42.784 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:42.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.976+0000 7f5b3bac6640 1 -- 172.21.15.150:0/62739388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b34074980 msgr2=0x7f5b340c3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:42.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.976+0000 7f5b3bac6640 1 --2- 172.21.15.150:0/62739388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b34074980 0x7f5b340c3fb0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5b340c3a90 tx=0x7f5b2c01dc20 comp rx=0 tx=0).stop 2024-09-17T05:15:42.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.976+0000 7f5b3bac6640 1 -- 172.21.15.150:0/62739388 shutdown_connections 2024-09-17T05:15:42.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.976+0000 7f5b3bac6640 1 --2- 172.21.15.150:0/62739388 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b34074980 0x7f5b340c3fb0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:42.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.976+0000 7f5b3bac6640 1 --2- 172.21.15.150:0/62739388 >> v2:172.21.15.150:3300/0 conn(0x7f5b3406fca0 0x7f5b34153790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:42.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.976+0000 7f5b3bac6640 1 -- 172.21.15.150:0/62739388 >> 172.21.15.150:0/62739388 conn(0x7f5b340c5ae0 msgr2=0x7f5b34073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:42.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.976+0000 7f5b3bac6640 1 -- 172.21.15.150:0/62739388 shutdown_connections 2024-09-17T05:15:42.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.976+0000 7f5b3bac6640 1 -- 172.21.15.150:0/62739388 wait complete. 2024-09-17T05:15:42.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b3bac6640 1 Processor -- start 2024-09-17T05:15:42.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b3bac6640 1 -- start start 2024-09-17T05:15:42.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b3bac6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b3406fca0 0x7f5b341503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:42.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b3bac6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b34074570 0x7f5b341f4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:42.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b3bac6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b34074980 0x7f5b3414bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:42.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b3bac6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5b340c49c0 con 0x7f5b34074570 2024-09-17T05:15:42.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b3bac6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5b340c4840 con 0x7f5b3406fca0 2024-09-17T05:15:42.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b3bac6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5b340c4b40 con 0x7f5b34074980 2024-09-17T05:15:42.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b39862640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b3406fca0 0x7f5b341503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:42.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b39061640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b34074570 0x7f5b341f4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:42.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b39862640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b3406fca0 0x7f5b341503c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:51510/0 (socket says 172.21.15.150:51510) 2024-09-17T05:15:42.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b39862640 1 -- 172.21.15.150:0/3711826456 learned_addr learned my addr 172.21.15.150:0/3711826456 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:42.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b3a063640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b34074980 0x7f5b3414bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:42.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b39061640 1 -- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b34074980 msgr2=0x7f5b3414bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:42.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b39061640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b34074980 0x7f5b3414bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:42.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b39061640 1 -- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b3406fca0 msgr2=0x7f5b341503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:42.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b39061640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b3406fca0 0x7f5b341503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:42.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b39061640 1 -- 172.21.15.150:0/3711826456 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3415d240 con 0x7f5b34074570 2024-09-17T05:15:42.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b3a063640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b34074980 0x7f5b3414bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:15:42.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b39061640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b34074570 0x7f5b341f4150 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400c3c0 tx=0x7f5b24001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:42.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b2a7fc640 1 -- 172.21.15.150:0/3711826456 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b24011790 con 0x7f5b34074570 2024-09-17T05:15:42.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b2a7fc640 1 -- 172.21.15.150:0/3711826456 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b24017070 con 0x7f5b34074570 2024-09-17T05:15:42.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.980+0000 7f5b3bac6640 1 -- 172.21.15.150:0/3711826456 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b341f4950 con 0x7f5b34074570 2024-09-17T05:15:42.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.984+0000 7f5b2a7fc640 1 -- 172.21.15.150:0/3711826456 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b24011cc0 con 0x7f5b34074570 2024-09-17T05:15:42.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.984+0000 7f5b3bac6640 1 -- 172.21.15.150:0/3711826456 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b341f4bf0 con 0x7f5b34074570 2024-09-17T05:15:42.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.984+0000 7f5b2a7fc640 1 -- 172.21.15.150:0/3711826456 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5b2401c460 con 0x7f5b34074570 2024-09-17T05:15:42.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.984+0000 7f5b2a7fc640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b18073200 0x7f5b18075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:42.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.984+0000 7f5b2a7fc640 1 -- 172.21.15.150:0/3711826456 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f5b24098060 con 0x7f5b34074570 2024-09-17T05:15:42.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.984+0000 7f5b39862640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b18073200 0x7f5b18075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:42.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.984+0000 7f5b3bac6640 1 -- 172.21.15.150:0/3711826456 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b34156f50 con 0x7f5b34074570 2024-09-17T05:15:42.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.984+0000 7f5b39862640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b18073200 0x7f5b18075710 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009a00 tx=0x7f5b30009110 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:42.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:42.992+0000 7f5b2a7fc640 1 -- 172.21.15.150:0/3711826456 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5b34153cd0 con 0x7f5b34074570 2024-09-17T05:15:43.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.304+0000 7f5b3bac6640 1 -- 172.21.15.150:0/3711826456 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b340c8140 con 0x7f5b34074570 2024-09-17T05:15:43.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.304+0000 7f5b2a7fc640 1 -- 172.21.15.150:0/3711826456 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+6544 (secure 0 0 0) 0x7f5b2405dfc0 con 0x7f5b34074570 2024-09-17T05:15:43.306 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:43.306 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":184,"btime":"2024-09-17T05:15:16:325986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7854,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:09.264728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7719,"qdb_cluster":[7719]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:43.306 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 184 2024-09-17T05:15:43.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.308+0000 7f5b3bac6640 1 -- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b18073200 msgr2=0x7f5b18075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:43.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.308+0000 7f5b3bac6640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b18073200 0x7f5b18075710 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009a00 tx=0x7f5b30009110 comp rx=0 tx=0).stop 2024-09-17T05:15:43.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.308+0000 7f5b3bac6640 1 -- 172.21.15.150:0/3711826456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b34074570 msgr2=0x7f5b341f4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:43.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.308+0000 7f5b3bac6640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b34074570 0x7f5b341f4150 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400c3c0 tx=0x7f5b24001e80 comp rx=0 tx=0).stop 2024-09-17T05:15:43.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.308+0000 7f5b3bac6640 1 -- 172.21.15.150:0/3711826456 shutdown_connections 2024-09-17T05:15:43.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.308+0000 7f5b3bac6640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5b18073200 0x7f5b18075710 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:43.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.308+0000 7f5b3bac6640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5b34074980 0x7f5b3414bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:43.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.308+0000 7f5b3bac6640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b34074570 0x7f5b341f4150 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:43.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.308+0000 7f5b3bac6640 1 --2- 172.21.15.150:0/3711826456 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5b3406fca0 0x7f5b341503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:43.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.308+0000 7f5b3bac6640 1 -- 172.21.15.150:0/3711826456 >> 172.21.15.150:0/3711826456 conn(0x7f5b340c5ae0 msgr2=0x7f5b34151690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:43.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.312+0000 7f5b3bac6640 1 -- 172.21.15.150:0/3711826456 shutdown_connections 2024-09-17T05:15:43.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:43.312+0000 7f5b3bac6640 1 -- 172.21.15.150:0/3711826456 wait complete. 2024-09-17T05:15:43.338 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:15:45.339 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:45.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.528+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/4103483407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2a00039f0 msgr2=0x7fb2a0003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:45.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.528+0000 7fb2bc0a4640 1 --2- 172.21.15.150:0/4103483407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2a00039f0 0x7fb2a0003e40 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4073460 tx=0x7fb2b001d2d0 comp rx=0 tx=0).stop 2024-09-17T05:15:45.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.528+0000 7fb2ba641640 1 -- 172.21.15.150:0/4103483407 reap_dead start 2024-09-17T05:15:45.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.532+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/4103483407 shutdown_connections 2024-09-17T05:15:45.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.532+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/4103483407 >> 172.21.15.150:0/4103483407 conn(0x7fb2b40c5ae0 msgr2=0x7fb2b4073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:45.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.532+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/4103483407 shutdown_connections 2024-09-17T05:15:45.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.532+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/4103483407 wait complete. 2024-09-17T05:15:45.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.532+0000 7fb2bc0a4640 1 Processor -- start 2024-09-17T05:15:45.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.532+0000 7fb2bc0a4640 1 -- start start 2024-09-17T05:15:45.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.532+0000 7fb2bc0a4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2b414ec20 0x7fb2b4145420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:45.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.532+0000 7fb2bc0a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b40c6dd0 0x7fb2b40c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:45.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.532+0000 7fb2bc0a4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2b40c6210 0x7fb2b40c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:45.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.532+0000 7fb2bc0a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b406fca0 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.532+0000 7fb2bc0a4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b4070820 con 0x7fb2b40c6210 2024-09-17T05:15:45.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.532+0000 7fb2bc0a4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb2b406fe20 con 0x7fb2b414ec20 2024-09-17T05:15:45.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2b963f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b40c6dd0 0x7fb2b40c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:45.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2b963f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b40c6dd0 0x7fb2b40c7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32838/0 (socket says 172.21.15.150:32838) 2024-09-17T05:15:45.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2b963f640 1 -- 172.21.15.150:0/1291478854 learned_addr learned my addr 172.21.15.150:0/1291478854 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:45.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2b9e40640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2b414ec20 0x7fb2b4145420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:45.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2ba641640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2b40c6210 0x7fb2b40c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:45.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2b963f640 1 -- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2b414ec20 msgr2=0x7fb2b4145420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:45.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2b963f640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2b414ec20 0x7fb2b4145420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:45.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2b963f640 1 -- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2b40c6210 msgr2=0x7fb2b40c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:45.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2b963f640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2b40c6210 0x7fb2b40c6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:45.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2b963f640 1 -- 172.21.15.150:0/1291478854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b4142120 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2ba641640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2b40c6210 0x7fb2b40c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:15:45.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2b9e40640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2b414ec20 0x7fb2b4145420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:15:45.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2b963f640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b40c6dd0 0x7fb2b40c7b90 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb2a400f6c0 tx=0x7fb2a400fb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:45.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2aa7fc640 1 -- 172.21.15.150:0/1291478854 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4015070 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2aa7fc640 1 -- 172.21.15.150:0/1291478854 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2a4006170 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/1291478854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2b4141c40 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2aa7fc640 1 -- 172.21.15.150:0/1291478854 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4003840 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/1291478854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2b40a9b80 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.536+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/1291478854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2b414fa60 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.540+0000 7fb2aa7fc640 1 -- 172.21.15.150:0/1291478854 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb2a4018070 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.540+0000 7fb2aa7fc640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb298073230 0x7fb298075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:45.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.540+0000 7fb2b9e40640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb298073230 0x7fb298075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:45.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.540+0000 7fb2aa7fc640 1 -- 172.21.15.150:0/1291478854 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7fb2a4095220 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.540+0000 7fb2b9e40640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb298073230 0x7fb298075740 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb2b00ae000 tx=0x7fb2b001a070 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:45.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.548+0000 7fb2aa7fc640 1 -- 172.21.15.150:0/1291478854 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb2a405ed90 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.840+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/1291478854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2b406e740 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.840+0000 7fb2aa7fc640 1 -- 172.21.15.150:0/1291478854 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+6544 (secure 0 0 0) 0x7fb2a40637c0 con 0x7fb2b40c6dd0 2024-09-17T05:15:45.841 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:45.841 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":184,"btime":"2024-09-17T05:15:16:325986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7854,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:09.264728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7719,"qdb_cluster":[7719]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:45.841 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 184 2024-09-17T05:15:45.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.844+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb298073230 msgr2=0x7fb298075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:45.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.844+0000 7fb2bc0a4640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb298073230 0x7fb298075740 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb2b00ae000 tx=0x7fb2b001a070 comp rx=0 tx=0).stop 2024-09-17T05:15:45.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.844+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/1291478854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b40c6dd0 msgr2=0x7fb2b40c7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:45.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.844+0000 7fb2bc0a4640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b40c6dd0 0x7fb2b40c7b90 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fb2a400f6c0 tx=0x7fb2a400fb80 comp rx=0 tx=0).stop 2024-09-17T05:15:45.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.844+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/1291478854 shutdown_connections 2024-09-17T05:15:45.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.844+0000 7fb2bc0a4640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb298073230 0x7fb298075740 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:45.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.844+0000 7fb2bc0a4640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2b40c6210 0x7fb2b40c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:45.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.844+0000 7fb2bc0a4640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2b40c6dd0 0x7fb2b40c7b90 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:45.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.844+0000 7fb2bc0a4640 1 --2- 172.21.15.150:0/1291478854 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2b414ec20 0x7fb2b4145420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:45.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.844+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/1291478854 >> 172.21.15.150:0/1291478854 conn(0x7fb2b40c5ae0 msgr2=0x7fb2b414dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:45.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.844+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/1291478854 shutdown_connections 2024-09-17T05:15:45.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:45.844+0000 7fb2bc0a4640 1 -- 172.21.15.150:0/1291478854 wait complete. 2024-09-17T05:15:45.868 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:15:47.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:48.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.060+0000 7f9dafb77640 1 -- 172.21.15.150:0/68865752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9da80c7eb0 msgr2=0x7f9da81540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:48.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.060+0000 7f9dafb77640 1 --2- 172.21.15.150:0/68865752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9da80c7eb0 0x7f9da81540b0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f9da4009870 tx=0x7f9da401c820 comp rx=0 tx=0).stop 2024-09-17T05:15:48.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.060+0000 7f9dafb77640 1 -- 172.21.15.150:0/68865752 shutdown_connections 2024-09-17T05:15:48.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.060+0000 7f9dafb77640 1 --2- 172.21.15.150:0/68865752 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9da80c7eb0 0x7f9da81540b0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:48.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.060+0000 7f9dafb77640 1 -- 172.21.15.150:0/68865752 >> 172.21.15.150:0/68865752 conn(0x7f9da80aab60 msgr2=0x7f9da80aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:48.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.060+0000 7f9dafb77640 1 -- 172.21.15.150:0/68865752 shutdown_connections 2024-09-17T05:15:48.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.060+0000 7f9dafb77640 1 -- 172.21.15.150:0/68865752 wait complete. 2024-09-17T05:15:48.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.060+0000 7f9dafb77640 1 Processor -- start 2024-09-17T05:15:48.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.060+0000 7f9dafb77640 1 -- start start 2024-09-17T05:15:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dafb77640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9da80c7eb0 0x7f9da80c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dafb77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9da8149d70 0x7f9da81f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dafb77640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9da814a180 0x7f9da8148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dafb77640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9da8156540 con 0x7f9da8149d70 2024-09-17T05:15:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dafb77640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9da81566c0 con 0x7f9da80c7eb0 2024-09-17T05:15:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dafb77640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9da8156840 con 0x7f9da814a180 2024-09-17T05:15:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dad112640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9da8149d70 0x7f9da81f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dad112640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9da8149d70 0x7f9da81f3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32856/0 (socket says 172.21.15.150:32856) 2024-09-17T05:15:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dad112640 1 -- 172.21.15.150:0/2088552260 learned_addr learned my addr 172.21.15.150:0/2088552260 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:48.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dae114640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9da814a180 0x7f9da8148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:48.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dae114640 1 -- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9da80c7eb0 msgr2=0x7f9da80c23b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:15:48.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dae114640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9da80c7eb0 0x7f9da80c23b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:48.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dae114640 1 -- 172.21.15.150:0/2088552260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9da8149d70 msgr2=0x7f9da81f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:48.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dae114640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9da8149d70 0x7f9da81f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:48.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.064+0000 7f9dae114640 1 -- 172.21.15.150:0/2088552260 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da81f3e80 con 0x7f9da814a180 2024-09-17T05:15:48.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.068+0000 7f9dae114640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9da814a180 0x7f9da8148d60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9da000c730 tx=0x7f9da000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:48.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.068+0000 7f9d9e7fc640 1 -- 172.21.15.150:0/2088552260 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da0003b80 con 0x7f9da814a180 2024-09-17T05:15:48.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.068+0000 7f9d9e7fc640 1 -- 172.21.15.150:0/2088552260 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9da0003d20 con 0x7f9da814a180 2024-09-17T05:15:48.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.068+0000 7f9d9e7fc640 1 -- 172.21.15.150:0/2088552260 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da00052d0 con 0x7f9da814a180 2024-09-17T05:15:48.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.068+0000 7f9dafb77640 1 -- 172.21.15.150:0/2088552260 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9da81f0820 con 0x7f9da814a180 2024-09-17T05:15:48.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.068+0000 7f9dafb77640 1 -- 172.21.15.150:0/2088552260 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9da806d4b0 con 0x7f9da814a180 2024-09-17T05:15:48.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.068+0000 7f9dafb77640 1 -- 172.21.15.150:0/2088552260 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9da81540b0 con 0x7f9da814a180 2024-09-17T05:15:48.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.072+0000 7f9d9e7fc640 1 -- 172.21.15.150:0/2088552260 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9da0005470 con 0x7f9da814a180 2024-09-17T05:15:48.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.072+0000 7f9d9e7fc640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9d8c073230 0x7f9d8c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:48.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.072+0000 7f9d9e7fc640 1 -- 172.21.15.150:0/2088552260 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f9da0095830 con 0x7f9da814a180 2024-09-17T05:15:48.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.072+0000 7f9dad913640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9d8c073230 0x7f9d8c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:48.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.072+0000 7f9dad913640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9d8c073230 0x7f9d8c075740 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9da40af000 tx=0x7f9da401b060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:48.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.080+0000 7f9d9e7fc640 1 -- 172.21.15.150:0/2088552260 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9da005f9e0 con 0x7f9da814a180 2024-09-17T05:15:48.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.384+0000 7f9dafb77640 1 -- 172.21.15.150:0/2088552260 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9da81f0f40 con 0x7f9da814a180 2024-09-17T05:15:48.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.388+0000 7f9d9e7fc640 1 -- 172.21.15.150:0/2088552260 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+6544 (secure 0 0 0) 0x7f9da005f2e0 con 0x7f9da814a180 2024-09-17T05:15:48.388 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:48.388 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":184,"btime":"2024-09-17T05:15:16:325986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7854,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:09.264728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7719,"qdb_cluster":[7719]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:48.389 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 184 2024-09-17T05:15:48.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.392+0000 7f9dafb77640 1 -- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9d8c073230 msgr2=0x7f9d8c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:48.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.392+0000 7f9dafb77640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9d8c073230 0x7f9d8c075740 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9da40af000 tx=0x7f9da401b060 comp rx=0 tx=0).stop 2024-09-17T05:15:48.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.392+0000 7f9dafb77640 1 -- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9da814a180 msgr2=0x7f9da8148d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:48.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.392+0000 7f9dafb77640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9da814a180 0x7f9da8148d60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9da000c730 tx=0x7f9da000cbf0 comp rx=0 tx=0).stop 2024-09-17T05:15:48.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.392+0000 7f9dafb77640 1 -- 172.21.15.150:0/2088552260 shutdown_connections 2024-09-17T05:15:48.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.392+0000 7f9dafb77640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9d8c073230 0x7f9d8c075740 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:48.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.392+0000 7f9dafb77640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9da814a180 0x7f9da8148d60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:48.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.392+0000 7f9dafb77640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9da8149d70 0x7f9da81f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:48.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.392+0000 7f9dafb77640 1 --2- 172.21.15.150:0/2088552260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9da80c7eb0 0x7f9da80c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:48.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.392+0000 7f9dafb77640 1 -- 172.21.15.150:0/2088552260 >> 172.21.15.150:0/2088552260 conn(0x7f9da80aab60 msgr2=0x7f9da8156b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:48.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.392+0000 7f9dafb77640 1 -- 172.21.15.150:0/2088552260 shutdown_connections 2024-09-17T05:15:48.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:48.392+0000 7f9dafb77640 1 -- 172.21.15.150:0/2088552260 wait complete. 2024-09-17T05:15:48.419 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:15:50.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:50.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.604+0000 7efe3cefd640 1 -- 172.21.15.150:0/716073401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe380a9e20 msgr2=0x7efe38150bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:50.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.604+0000 7efe3cefd640 1 --2- 172.21.15.150:0/716073401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe380a9e20 0x7efe38150bb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7efe2c009840 tx=0x7efe2c01c790 comp rx=0 tx=0).stop 2024-09-17T05:15:50.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.604+0000 7efe3cefd640 1 -- 172.21.15.150:0/716073401 shutdown_connections 2024-09-17T05:15:50.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.604+0000 7efe3cefd640 1 --2- 172.21.15.150:0/716073401 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe380a9e20 0x7efe38150bb0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:50.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.604+0000 7efe3cefd640 1 -- 172.21.15.150:0/716073401 >> 172.21.15.150:0/716073401 conn(0x7efe38070b50 msgr2=0x7efe38070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:50.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.604+0000 7efe3cefd640 1 -- 172.21.15.150:0/716073401 shutdown_connections 2024-09-17T05:15:50.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.604+0000 7efe3cefd640 1 -- 172.21.15.150:0/716073401 wait complete. 2024-09-17T05:15:50.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe3cefd640 1 Processor -- start 2024-09-17T05:15:50.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe3cefd640 1 -- start start 2024-09-17T05:15:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe3cefd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe3806e530 0x7efe38155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe3cefd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe38159f90 0x7efe3813fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe3cefd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe380c7780 0x7efe380c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe3cefd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efe380aa290 con 0x7efe380c7780 2024-09-17T05:15:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe3cefd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efe3815a7b0 con 0x7efe3806e530 2024-09-17T05:15:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe3cefd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efe380aa410 con 0x7efe38159f90 2024-09-17T05:15:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe36d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe380c7780 0x7efe380c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe36d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe380c7780 0x7efe380c7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53782/0 (socket says 172.21.15.150:53782) 2024-09-17T05:15:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe36d9d640 1 -- 172.21.15.150:0/118173977 learned_addr learned my addr 172.21.15.150:0/118173977 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe3659c640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe3806e530 0x7efe38155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe35d9b640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe38159f90 0x7efe3813fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:50.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe36d9d640 1 -- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe38159f90 msgr2=0x7efe3813fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe36d9d640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe38159f90 0x7efe3813fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe36d9d640 1 -- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe3806e530 msgr2=0x7efe38155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe36d9d640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe3806e530 0x7efe38155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe36d9d640 1 -- 172.21.15.150:0/118173977 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe38154b50 con 0x7efe380c7780 2024-09-17T05:15:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe35d9b640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe38159f90 0x7efe3813fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:15:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe36d9d640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe380c7780 0x7efe380c7b90 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7efe3000cae0 tx=0x7efe3000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe26ffd640 1 -- 172.21.15.150:0/118173977 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30016020 con 0x7efe380c7780 2024-09-17T05:15:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe26ffd640 1 -- 172.21.15.150:0/118173977 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe3000aae0 con 0x7efe380c7780 2024-09-17T05:15:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe3cefd640 1 -- 172.21.15.150:0/118173977 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe381548b0 con 0x7efe380c7780 2024-09-17T05:15:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe26ffd640 1 -- 172.21.15.150:0/118173977 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3000ce20 con 0x7efe380c7780 2024-09-17T05:15:50.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe3cefd640 1 -- 172.21.15.150:0/118173977 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe381f3c40 con 0x7efe380c7780 2024-09-17T05:15:50.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.608+0000 7efe3cefd640 1 -- 172.21.15.150:0/118173977 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe380a9e20 con 0x7efe380c7780 2024-09-17T05:15:50.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.612+0000 7efe26ffd640 1 -- 172.21.15.150:0/118173977 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7efe30019020 con 0x7efe380c7780 2024-09-17T05:15:50.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.612+0000 7efe26ffd640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efe18073230 0x7efe18075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:50.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.612+0000 7efe26ffd640 1 -- 172.21.15.150:0/118173977 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7efe30095020 con 0x7efe380c7780 2024-09-17T05:15:50.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.612+0000 7efe3659c640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efe18073230 0x7efe18075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:50.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.612+0000 7efe3659c640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efe18073230 0x7efe18075740 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7efe2c0af000 tx=0x7efe2c01afd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:50.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.620+0000 7efe26ffd640 1 -- 172.21.15.150:0/118173977 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7efe30020030 con 0x7efe380c7780 2024-09-17T05:15:50.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.916+0000 7efe3cefd640 1 -- 172.21.15.150:0/118173977 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe380ae800 con 0x7efe380c7780 2024-09-17T05:15:50.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe26ffd640 1 -- 172.21.15.150:0/118173977 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+6544 (secure 0 0 0) 0x7efe3005f160 con 0x7efe380c7780 2024-09-17T05:15:50.918 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:50.918 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":184,"btime":"2024-09-17T05:15:16:325986+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7854,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:09.264728+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7719},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7719":{"gid":7719,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":13,"addr":"172.21.15.150:6835/178583496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":178583496},{"type":"v1","addr":"172.21.15.150:6835","nonce":178583496}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7719,"qdb_cluster":[7719]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:50.919 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 184 2024-09-17T05:15:50.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe3cefd640 1 -- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efe18073230 msgr2=0x7efe18075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:50.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe3cefd640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efe18073230 0x7efe18075740 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7efe2c0af000 tx=0x7efe2c01afd0 comp rx=0 tx=0).stop 2024-09-17T05:15:50.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe3cefd640 1 -- 172.21.15.150:0/118173977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe380c7780 msgr2=0x7efe380c7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:50.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe3cefd640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe380c7780 0x7efe380c7b90 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7efe3000cae0 tx=0x7efe3000cbd0 comp rx=0 tx=0).stop 2024-09-17T05:15:50.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe3cefd640 1 -- 172.21.15.150:0/118173977 shutdown_connections 2024-09-17T05:15:50.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe3cefd640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efe18073230 0x7efe18075740 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:50.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe3cefd640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe380c7780 0x7efe380c7b90 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:50.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe3cefd640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efe38159f90 0x7efe3813fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:50.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe3cefd640 1 --2- 172.21.15.150:0/118173977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe3806e530 0x7efe38155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:50.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe3cefd640 1 -- 172.21.15.150:0/118173977 >> 172.21.15.150:0/118173977 conn(0x7efe38070b50 msgr2=0x7efe3814a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:50.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe3cefd640 1 -- 172.21.15.150:0/118173977 shutdown_connections 2024-09-17T05:15:50.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:50.920+0000 7efe3cefd640 1 -- 172.21.15.150:0/118173977 wait complete. 2024-09-17T05:15:50.952 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:15:52.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:53.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.144+0000 7f48a6b41640 1 -- 172.21.15.150:0/3498513390 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a00c5760 msgr2=0x7f48a00c5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:53.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.144+0000 7f48a6b41640 1 --2- 172.21.15.150:0/3498513390 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a00c5760 0x7f48a00c5b70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f4890009840 tx=0x7f489001c700 comp rx=0 tx=0).stop 2024-09-17T05:15:53.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.144+0000 7f48a6b41640 1 -- 172.21.15.150:0/3498513390 shutdown_connections 2024-09-17T05:15:53.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.144+0000 7f48a6b41640 1 --2- 172.21.15.150:0/3498513390 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a00c5760 0x7f48a00c5b70 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:53.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.144+0000 7f48a6b41640 1 -- 172.21.15.150:0/3498513390 >> 172.21.15.150:0/3498513390 conn(0x7f48a00aaa80 msgr2=0x7f48a00aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:53.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.144+0000 7f48a6b41640 1 -- 172.21.15.150:0/3498513390 shutdown_connections 2024-09-17T05:15:53.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.144+0000 7f48a6b41640 1 -- 172.21.15.150:0/3498513390 wait complete. 2024-09-17T05:15:53.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a6b41640 1 Processor -- start 2024-09-17T05:15:53.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a6b41640 1 -- start start 2024-09-17T05:15:53.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a6b41640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48a00a7f10 0x7f48a0148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:53.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a6b41640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a00c5760 0x7f48a0147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a6b41640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48a01535b0 0x7f48a0155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a6b41640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f48a00c5370 con 0x7f48a00a7f10 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a6b41640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f48a00718d0 con 0x7f48a00c5760 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a6b41640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f48a00c54f0 con 0x7f48a01535b0 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a48dd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48a00a7f10 0x7f48a0148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a48dd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48a00a7f10 0x7f48a0148700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53794/0 (socket says 172.21.15.150:53794) 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a48dd640 1 -- 172.21.15.150:0/3743874230 learned_addr learned my addr 172.21.15.150:0/3743874230 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a50de640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48a01535b0 0x7f48a0155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f4897fff640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a00c5760 0x7f48a0147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a48dd640 1 -- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48a01535b0 msgr2=0x7f48a0155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a48dd640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48a01535b0 0x7f48a0155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a48dd640 1 -- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a00c5760 msgr2=0x7f48a0147270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:53.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a48dd640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a00c5760 0x7f48a0147270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:53.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a48dd640 1 -- 172.21.15.150:0/3743874230 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a00c79e0 con 0x7f48a00a7f10 2024-09-17T05:15:53.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a48dd640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48a00a7f10 0x7f48a0148700 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f489c00c440 tx=0x7f489c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:53.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48957fa640 1 -- 172.21.15.150:0/3743874230 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c011790 con 0x7f48a00a7f10 2024-09-17T05:15:53.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48957fa640 1 -- 172.21.15.150:0/3743874230 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f489c017070 con 0x7f48a00a7f10 2024-09-17T05:15:53.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a6b41640 1 -- 172.21.15.150:0/3743874230 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48a0155ec0 con 0x7f48a00a7f10 2024-09-17T05:15:53.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48957fa640 1 -- 172.21.15.150:0/3743874230 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489c011cc0 con 0x7f48a00a7f10 2024-09-17T05:15:53.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.148+0000 7f48a6b41640 1 -- 172.21.15.150:0/3743874230 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48a00ac810 con 0x7f48a00a7f10 2024-09-17T05:15:53.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.152+0000 7f48957fa640 1 -- 172.21.15.150:0/3743874230 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f489c01c460 con 0x7f48a00a7f10 2024-09-17T05:15:53.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.152+0000 7f48957fa640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f488c073230 0x7f488c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:53.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.152+0000 7f48957fa640 1 -- 172.21.15.150:0/3743874230 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f489c094e90 con 0x7f48a00a7f10 2024-09-17T05:15:53.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.152+0000 7f48a6b41640 1 -- 172.21.15.150:0/3743874230 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48a0153aa0 con 0x7f48a00a7f10 2024-09-17T05:15:53.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.156+0000 7f4897fff640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f488c073230 0x7f488c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:53.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.156+0000 7f4897fff640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f488c073230 0x7f488c075740 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f48900af000 tx=0x7f48900b2040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:53.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.160+0000 7f48957fa640 1 -- 172.21.15.150:0/3743874230 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f489c05e8d0 con 0x7f48a00a7f10 2024-09-17T05:15:53.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.452+0000 7f48a6b41640 1 -- 172.21.15.150:0/3743874230 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48a0153390 con 0x7f48a00a7f10 2024-09-17T05:15:53.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.452+0000 7f48957fa640 1 -- 172.21.15.150:0/3743874230 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+5714 (secure 0 0 0) 0x7f489c063300 con 0x7f48a00a7f10 2024-09-17T05:15:53.454 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:53.454 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":186,"btime":"2024-09-17T05:15:52:838345+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:51.988161+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:53.454 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 186 2024-09-17T05:15:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.456+0000 7f48a6b41640 1 -- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f488c073230 msgr2=0x7f488c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.456+0000 7f48a6b41640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f488c073230 0x7f488c075740 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f48900af000 tx=0x7f48900b2040 comp rx=0 tx=0).stop 2024-09-17T05:15:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.456+0000 7f48a6b41640 1 -- 172.21.15.150:0/3743874230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48a00a7f10 msgr2=0x7f48a0148700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:53.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.456+0000 7f48a6b41640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48a00a7f10 0x7f48a0148700 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f489c00c440 tx=0x7f489c00c900 comp rx=0 tx=0).stop 2024-09-17T05:15:53.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.456+0000 7f48a6b41640 1 -- 172.21.15.150:0/3743874230 shutdown_connections 2024-09-17T05:15:53.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.456+0000 7f48a6b41640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f488c073230 0x7f488c075740 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:53.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.456+0000 7f48a6b41640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f48a00c5760 0x7f48a0147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:53.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.456+0000 7f48a6b41640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f48a01535b0 0x7f48a0155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:53.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.456+0000 7f48a6b41640 1 --2- 172.21.15.150:0/3743874230 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48a00a7f10 0x7f48a0148700 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:53.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.456+0000 7f48a6b41640 1 -- 172.21.15.150:0/3743874230 >> 172.21.15.150:0/3743874230 conn(0x7f48a00aaa80 msgr2=0x7f48a0074bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:53.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.456+0000 7f48a6b41640 1 -- 172.21.15.150:0/3743874230 shutdown_connections 2024-09-17T05:15:53.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.456+0000 7f48a6b41640 1 -- 172.21.15.150:0/3743874230 wait complete. 2024-09-17T05:15:53.487 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-17T05:15:53.487 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:15:53.488 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:53.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 -- 172.21.15.150:0/3686267276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c4c0c97a0 msgr2=0x7f2c4c151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:53.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 --2- 172.21.15.150:0/3686267276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c4c0c97a0 0x7f2c4c151c70 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f2c48009840 tx=0x7f2c4801c750 comp rx=0 tx=0).stop 2024-09-17T05:15:53.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 -- 172.21.15.150:0/3686267276 shutdown_connections 2024-09-17T05:15:53.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 --2- 172.21.15.150:0/3686267276 >> v2:172.21.15.176:3300/0 conn(0x7f2c4c072ea0 0x7f2c4c0732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:53.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 --2- 172.21.15.150:0/3686267276 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c4c0c97a0 0x7f2c4c151c70 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:53.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 -- 172.21.15.150:0/3686267276 >> 172.21.15.150:0/3686267276 conn(0x7f2c4c0abb00 msgr2=0x7f2c4c0abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:53.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 -- 172.21.15.150:0/3686267276 shutdown_connections 2024-09-17T05:15:53.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 -- 172.21.15.150:0/3686267276 wait complete. 2024-09-17T05:15:53.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 Processor -- start 2024-09-17T05:15:53.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 -- start start 2024-09-17T05:15:53.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c4c072ea0 0x7f2c4c0c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:53.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c4c0c97a0 0x7f2c4c1f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:53.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c4c157be0 0x7f2c4c145570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:53.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c161790 con 0x7f2c4c0c97a0 2024-09-17T05:15:53.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c0c4df0 con 0x7f2c4c157be0 2024-09-17T05:15:53.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.680+0000 7f2c5399d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2c4c161910 con 0x7f2c4c072ea0 2024-09-17T05:15:53.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c50f38640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c4c0c97a0 0x7f2c4c1f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:53.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c50f38640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c4c0c97a0 0x7f2c4c1f31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53814/0 (socket says 172.21.15.150:53814) 2024-09-17T05:15:53.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c50f38640 1 -- 172.21.15.150:0/3102915536 learned_addr learned my addr 172.21.15.150:0/3102915536 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:53.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c51739640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c4c072ea0 0x7f2c4c0c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:53.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c50f38640 1 -- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c4c072ea0 msgr2=0x7f2c4c0c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:53.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c50f38640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c4c072ea0 0x7f2c4c0c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:53.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c50f38640 1 -- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c4c157be0 msgr2=0x7f2c4c145570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:53.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c50f38640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c4c157be0 0x7f2c4c145570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:53.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c50f38640 1 -- 172.21.15.150:0/3102915536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4c1f3c40 con 0x7f2c4c0c97a0 2024-09-17T05:15:53.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c51739640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c4c072ea0 0x7f2c4c0c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:15:53.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c50f38640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c4c0c97a0 0x7f2c4c1f31e0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00e740 tx=0x7f2c3c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:53.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c41ffb640 1 -- 172.21.15.150:0/3102915536 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c00ccc0 con 0x7f2c4c0c97a0 2024-09-17T05:15:53.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c41ffb640 1 -- 172.21.15.150:0/3102915536 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2c3c00ee50 con 0x7f2c4c0c97a0 2024-09-17T05:15:53.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c5399d640 1 -- 172.21.15.150:0/3102915536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c4c153bf0 con 0x7f2c4c0c97a0 2024-09-17T05:15:53.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c41ffb640 1 -- 172.21.15.150:0/3102915536 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c010670 con 0x7f2c4c0c97a0 2024-09-17T05:15:53.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c5399d640 1 -- 172.21.15.150:0/3102915536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c4c14d9a0 con 0x7f2c4c0c97a0 2024-09-17T05:15:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c5399d640 1 -- 172.21.15.150:0/3102915536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c4c0732b0 con 0x7f2c4c0c97a0 2024-09-17T05:15:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c41ffb640 1 -- 172.21.15.150:0/3102915536 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2c3c010810 con 0x7f2c4c0c97a0 2024-09-17T05:15:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.684+0000 7f2c41ffb640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2c30073230 0x7f2c30075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:53.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.688+0000 7f2c41ffb640 1 -- 172.21.15.150:0/3102915536 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f2c3c095a70 con 0x7f2c4c0c97a0 2024-09-17T05:15:53.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.688+0000 7f2c51739640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2c30073230 0x7f2c30075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:53.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.688+0000 7f2c51739640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2c30073230 0x7f2c30075740 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2c480af000 tx=0x7f2c4801af90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:53.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:53.696+0000 7f2c41ffb640 1 -- 172.21.15.150:0/3102915536 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2c3c05f590 con 0x7f2c4c0c97a0 2024-09-17T05:15:54.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.012+0000 7f2c5399d640 1 -- 172.21.15.150:0/3102915536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c4c0c40f0 con 0x7f2c4c0c97a0 2024-09-17T05:15:54.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.012+0000 7f2c41ffb640 1 -- 172.21.15.150:0/3102915536 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+5711 (secure 0 0 0) 0x7f2c3c063fc0 con 0x7f2c4c0c97a0 2024-09-17T05:15:54.014 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:54.014 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":187,"btime":"2024-09-17T05:15:53:839615+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:52.843551+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:54.014 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 187 2024-09-17T05:15:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.016+0000 7f2c5399d640 1 -- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2c30073230 msgr2=0x7f2c30075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.016+0000 7f2c5399d640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2c30073230 0x7f2c30075740 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2c480af000 tx=0x7f2c4801af90 comp rx=0 tx=0).stop 2024-09-17T05:15:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.016+0000 7f2c5399d640 1 -- 172.21.15.150:0/3102915536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c4c0c97a0 msgr2=0x7f2c4c1f31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.016+0000 7f2c5399d640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c4c0c97a0 0x7f2c4c1f31e0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00e740 tx=0x7f2c3c00ec00 comp rx=0 tx=0).stop 2024-09-17T05:15:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.016+0000 7f2c5399d640 1 -- 172.21.15.150:0/3102915536 shutdown_connections 2024-09-17T05:15:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.016+0000 7f2c5399d640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2c30073230 0x7f2c30075740 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.016+0000 7f2c5399d640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2c4c157be0 0x7f2c4c145570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.016+0000 7f2c5399d640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c4c0c97a0 0x7f2c4c1f31e0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.016+0000 7f2c5399d640 1 --2- 172.21.15.150:0/3102915536 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2c4c072ea0 0x7f2c4c0c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.016+0000 7f2c5399d640 1 -- 172.21.15.150:0/3102915536 >> 172.21.15.150:0/3102915536 conn(0x7f2c4c0abb00 msgr2=0x7f2c4c0aed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:54.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.020+0000 7f2c5399d640 1 -- 172.21.15.150:0/3102915536 shutdown_connections 2024-09-17T05:15:54.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:54.020+0000 7f2c5399d640 1 -- 172.21.15.150:0/3102915536 wait complete. 2024-09-17T05:15:54.049 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:15:53:839615+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 187, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 187, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7854": { "addr": "172.21.15.150:6833/949083177", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 949083177, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 949083177, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7854, "incarnation": 185, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:15:52.843551+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7854 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 183, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4355": { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 180, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 389 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:15:11.291889+0000", "qdb_cluster": [ 4355 ], "qdb_leader": 4355, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4355 } } } ], "standbys": [ { "addr": "172.21.15.176:6837/1903363960", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 1903363960, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 1903363960, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 184, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6563, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:15:56.050 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:15:56.381 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T05:15:56.380+0000 7f05e26e7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:15:56.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.780+0000 7f863ffff640 1 -- 172.21.15.150:0/3070440163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86380d4870 msgr2=0x7f8638007d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:56.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.780+0000 7f863ffff640 1 --2- 172.21.15.150:0/3070440163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86380d4870 0x7f8638007d20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8630009840 tx=0x7f863001c7e0 comp rx=0 tx=0).stop 2024-09-17T05:15:56.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.784+0000 7f863ffff640 1 -- 172.21.15.150:0/3070440163 shutdown_connections 2024-09-17T05:15:56.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.784+0000 7f863ffff640 1 --2- 172.21.15.150:0/3070440163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86380d4870 0x7f8638007d20 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:56.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.784+0000 7f863ffff640 1 --2- 172.21.15.150:0/3070440163 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86380d79a0 0x7f86380d4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:56.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.784+0000 7f863ffff640 1 -- 172.21.15.150:0/3070440163 >> 172.21.15.150:0/3070440163 conn(0x7f8638049590 msgr2=0x7f863802e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:56.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.784+0000 7f863ffff640 1 -- 172.21.15.150:0/3070440163 shutdown_connections 2024-09-17T05:15:56.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.784+0000 7f863ffff640 1 -- 172.21.15.150:0/3070440163 wait complete. 2024-09-17T05:15:56.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.784+0000 7f863ffff640 1 Processor -- start 2024-09-17T05:15:56.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.784+0000 7f863ffff640 1 -- start start 2024-09-17T05:15:56.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86380478a0 0x7f8638044e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:56.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863ffff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86380d4870 0x7f86381765d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:56.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863ffff640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86380d79a0 0x7f86380cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:56.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86380478a0 0x7f8638044e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:56.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86380478a0 0x7f8638044e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53822/0 (socket says 172.21.15.150:53822) 2024-09-17T05:15:56.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863effd640 1 -- 172.21.15.150:0/2288596592 learned_addr learned my addr 172.21.15.150:0/2288596592 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:15:56.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863ffff640 1 -- 172.21.15.150:0/2288596592 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f86380e3c40 con 0x7f86380478a0 2024-09-17T05:15:56.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863ffff640 1 -- 172.21.15.150:0/2288596592 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f86380e3ac0 con 0x7f86380d4870 2024-09-17T05:15:56.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863ffff640 1 -- 172.21.15.150:0/2288596592 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f86380e3dc0 con 0x7f86380d79a0 2024-09-17T05:15:56.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863e7fc640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86380d4870 0x7f86381765d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:56.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863effd640 1 -- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86380d79a0 msgr2=0x7f86380cfc90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:15:56.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863effd640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86380d79a0 0x7f86380cfc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:56.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863effd640 1 -- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86380d4870 msgr2=0x7f86381765d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:56.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863effd640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86380d4870 0x7f86381765d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:56.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863effd640 1 -- 172.21.15.150:0/2288596592 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86380461c0 con 0x7f86380478a0 2024-09-17T05:15:56.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f863effd640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86380478a0 0x7f8638044e10 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f8640076440 tx=0x7f86400771f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:56.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.788+0000 7f86277fe640 1 -- 172.21.15.150:0/2288596592 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640091070 con 0x7f86380478a0 2024-09-17T05:15:56.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.792+0000 7f86277fe640 1 -- 172.21.15.150:0/2288596592 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86400754b0 con 0x7f86380478a0 2024-09-17T05:15:56.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.792+0000 7f86277fe640 1 -- 172.21.15.150:0/2288596592 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640091070 con 0x7f86380478a0 2024-09-17T05:15:56.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.792+0000 7f863ffff640 1 -- 172.21.15.150:0/2288596592 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86380cda60 con 0x7f86380478a0 2024-09-17T05:15:56.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.792+0000 7f863ffff640 1 -- 172.21.15.150:0/2288596592 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8638043ed0 con 0x7f86380478a0 2024-09-17T05:15:56.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.796+0000 7f86277fe640 1 -- 172.21.15.150:0/2288596592 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f86400638b0 con 0x7f86380478a0 2024-09-17T05:15:56.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.796+0000 7f86277fe640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f862c073600 0x7f862c075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:15:56.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.796+0000 7f86277fe640 1 -- 172.21.15.150:0/2288596592 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f86400751c0 con 0x7f86380478a0 2024-09-17T05:15:56.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.796+0000 7f86257fa640 1 -- 172.21.15.150:0/2288596592 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8638007d20 con 0x7f86380478a0 2024-09-17T05:15:56.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.804+0000 7f86277fe640 1 -- 172.21.15.150:0/2288596592 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f86380d7db0 con 0x7f86380478a0 2024-09-17T05:15:56.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.808+0000 7f863e7fc640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f862c073600 0x7f862c075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:15:56.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:56.812+0000 7f863e7fc640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f862c073600 0x7f862c075b10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8638043700 tx=0x7f86340091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:15:58.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.296+0000 7f86257fa640 1 -- 172.21.15.150:0/2288596592 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86380e11c0 con 0x7f86380478a0 2024-09-17T05:15:58.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.308+0000 7f86277fe640 1 -- 172.21.15.150:0/2288596592 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) ==== 95+0+5718 (secure 0 0 0) 0x7f86400da4b0 con 0x7f86380478a0 2024-09-17T05:15:58.312 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:15:58.312 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":188,"btime":"2024-09-17T05:15:54:841103+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:15:58.312 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 188 2024-09-17T05:15:58.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.316+0000 7f86257fa640 1 -- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f862c073600 msgr2=0x7f862c075b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:58.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.316+0000 7f86257fa640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f862c073600 0x7f862c075b10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8638043700 tx=0x7f86340091e0 comp rx=0 tx=0).stop 2024-09-17T05:15:58.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.316+0000 7f86257fa640 1 -- 172.21.15.150:0/2288596592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86380478a0 msgr2=0x7f8638044e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:15:58.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.316+0000 7f86257fa640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86380478a0 0x7f8638044e10 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f8640076440 tx=0x7f86400771f0 comp rx=0 tx=0).stop 2024-09-17T05:15:58.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.316+0000 7f86257fa640 1 -- 172.21.15.150:0/2288596592 shutdown_connections 2024-09-17T05:15:58.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.316+0000 7f86257fa640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f862c073600 0x7f862c075b10 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:58.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.316+0000 7f86257fa640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86380d4870 0x7f86381765d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:58.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.320+0000 7f86257fa640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f86380d79a0 0x7f86380cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:58.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.320+0000 7f86257fa640 1 --2- 172.21.15.150:0/2288596592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86380478a0 0x7f8638044e10 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:15:58.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.320+0000 7f86257fa640 1 -- 172.21.15.150:0/2288596592 >> 172.21.15.150:0/2288596592 conn(0x7f8638049590 msgr2=0x7f86380d2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:15:58.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.320+0000 7f86257fa640 1 -- 172.21.15.150:0/2288596592 shutdown_connections 2024-09-17T05:15:58.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:15:58.320+0000 7f86257fa640 1 -- 172.21.15.150:0/2288596592 wait complete. 2024-09-17T05:15:58.490 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=7719 2024-09-17T05:15:58.490 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before reviving mds.c 2024-09-17T05:16:05.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:16:05.606 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:16:06.491 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-17T05:16:06.491 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T05:16:06.491 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T05:16:06.494 INFO:tasks.ceph.mds.c:Started 2024-09-17T05:16:06.494 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:06.821 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T05:16:07.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.008+0000 7fa922870640 1 -- 172.21.15.150:0/3463518324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa90c009f00 msgr2=0x7fa90c00c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:07.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.008+0000 7fa922870640 1 --2- 172.21.15.150:0/3463518324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa90c009f00 0x7fa90c00c320 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fa91000c730 tx=0x7fa910024b60 comp rx=0 tx=0).stop 2024-09-17T05:16:07.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.012+0000 7fa922870640 1 -- 172.21.15.150:0/3463518324 shutdown_connections 2024-09-17T05:16:07.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.012+0000 7fa922870640 1 --2- 172.21.15.150:0/3463518324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa90c009f00 0x7fa90c00c320 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:07.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.012+0000 7fa922870640 1 -- 172.21.15.150:0/3463518324 >> 172.21.15.150:0/3463518324 conn(0x7fa91c0a9ca0 msgr2=0x7fa91c0aa0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.048+0000 7fa922870640 1 -- 172.21.15.150:0/3463518324 shutdown_connections 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.048+0000 7fa922870640 1 -- 172.21.15.150:0/3463518324 wait complete. 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.052+0000 7fa922870640 1 Processor -- start 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.052+0000 7fa922870640 1 -- start start 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.052+0000 7fa922870640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa90c009f00 0x7fa91c16a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.052+0000 7fa922870640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa91c0c1800 0x7fa91c15cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.052+0000 7fa922870640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c1629b0 0x7fa91c15d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.052+0000 7fa922870640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c073a60 con 0x7fa91c1629b0 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.052+0000 7fa922870640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c0c2f10 con 0x7fa90c009f00 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.052+0000 7fa922870640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa91c08c060 con 0x7fa91c0c1800 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.052+0000 7fa91bfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa90c009f00 0x7fa91c16a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.052+0000 7fa920e0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c1629b0 0x7fa91c15d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.052+0000 7fa920e0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c1629b0 0x7fa91c15d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58094/0 (socket says 172.21.15.150:58094) 2024-09-17T05:16:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.052+0000 7fa920e0d640 1 -- 172.21.15.150:0/2397110907 learned_addr learned my addr 172.21.15.150:0/2397110907 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:07.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.056+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa91c0c1800 0x7fa91c15cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:07.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.056+0000 7fa91bfff640 1 -- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa91c0c1800 msgr2=0x7fa91c15cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:07.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.056+0000 7fa91bfff640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa91c0c1800 0x7fa91c15cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:07.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.056+0000 7fa91bfff640 1 -- 172.21.15.150:0/2397110907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c1629b0 msgr2=0x7fa91c15d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:07.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.056+0000 7fa91bfff640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c1629b0 0x7fa91c15d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:07.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.056+0000 7fa91bfff640 1 -- 172.21.15.150:0/2397110907 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91c1667b0 con 0x7fa90c009f00 2024-09-17T05:16:07.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.056+0000 7fa91bfff640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa90c009f00 0x7fa91c16a520 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa91400c110 tx=0x7fa91400ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:07.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.056+0000 7fa918ff9640 1 -- 172.21.15.150:0/2397110907 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914018030 con 0x7fa90c009f00 2024-09-17T05:16:07.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.056+0000 7fa918ff9640 1 -- 172.21.15.150:0/2397110907 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa914003ac0 con 0x7fa90c009f00 2024-09-17T05:16:07.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.056+0000 7fa918ff9640 1 -- 172.21.15.150:0/2397110907 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914004020 con 0x7fa90c009f00 2024-09-17T05:16:07.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.060+0000 7fa922870640 1 -- 172.21.15.150:0/2397110907 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91c15d860 con 0x7fa90c009f00 2024-09-17T05:16:07.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.060+0000 7fa922870640 1 -- 172.21.15.150:0/2397110907 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91c16b4a0 con 0x7fa90c009f00 2024-09-17T05:16:07.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.060+0000 7fa922870640 1 -- 172.21.15.150:0/2397110907 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa91c0c2430 con 0x7fa90c009f00 2024-09-17T05:16:07.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.072+0000 7fa918ff9640 1 -- 172.21.15.150:0/2397110907 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa9140041c0 con 0x7fa90c009f00 2024-09-17T05:16:07.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.072+0000 7fa918ff9640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa900073250 0x7fa900075760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:07.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.072+0000 7fa918ff9640 1 -- 172.21.15.150:0/2397110907 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7fa914098330 con 0x7fa90c009f00 2024-09-17T05:16:07.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.072+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa900073250 0x7fa900075760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:07.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.072+0000 7fa91b7fe640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa900073250 0x7fa900075760 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0c2820 tx=0x7fa908011040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:07.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:07.084+0000 7fa918ff9640 1 -- 172.21.15.150:0/2397110907 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa914062400 con 0x7fa90c009f00 2024-09-17T05:16:08.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.072+0000 7fa922870640 1 -- 172.21.15.150:0/2397110907 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa91c070fb0 con 0x7fa90c009f00 2024-09-17T05:16:08.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa918ff9640 1 -- 172.21.15.150:0/2397110907 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+6547 (secure 0 0 0) 0x7fa914061d00 con 0x7fa90c009f00 2024-09-17T05:16:08.074 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:08.074 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":189,"btime":"2024-09-17T05:16:07:302153+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:08.074 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 189 2024-09-17T05:16:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa922870640 1 -- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa900073250 msgr2=0x7fa900075760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa922870640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa900073250 0x7fa900075760 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0c2820 tx=0x7fa908011040 comp rx=0 tx=0).stop 2024-09-17T05:16:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa922870640 1 -- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa90c009f00 msgr2=0x7fa91c16a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa922870640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa90c009f00 0x7fa91c16a520 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa91400c110 tx=0x7fa91400ef90 comp rx=0 tx=0).stop 2024-09-17T05:16:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa922870640 1 -- 172.21.15.150:0/2397110907 shutdown_connections 2024-09-17T05:16:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa922870640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa900073250 0x7fa900075760 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa922870640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa91c1629b0 0x7fa91c15d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa922870640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa91c0c1800 0x7fa91c15cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa922870640 1 --2- 172.21.15.150:0/2397110907 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa90c009f00 0x7fa91c16a520 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa922870640 1 -- 172.21.15.150:0/2397110907 >> 172.21.15.150:0/2397110907 conn(0x7fa91c0a9ca0 msgr2=0x7fa91c17d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:08.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa922870640 1 -- 172.21.15.150:0/2397110907 shutdown_connections 2024-09-17T05:16:08.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:08.076+0000 7fa922870640 1 -- 172.21.15.150:0/2397110907 wait complete. 2024-09-17T05:16:08.108 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-17T05:16:08.109 INFO:tasks.mds_thrash.fs.[a]:waiting for 27 secs before thrashing 2024-09-17T05:16:35.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:35.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.632+0000 7f77833a1640 1 -- 172.21.15.150:0/803705944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f777c070b50 msgr2=0x7f777c0c2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:35.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.632+0000 7f77833a1640 1 --2- 172.21.15.150:0/803705944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f777c070b50 0x7f777c0c2720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f777c0710c0 tx=0x7f777801cc60 comp rx=0 tx=0).stop 2024-09-17T05:16:35.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.632+0000 7f77833a1640 1 -- 172.21.15.150:0/803705944 shutdown_connections 2024-09-17T05:16:35.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.632+0000 7f77833a1640 1 --2- 172.21.15.150:0/803705944 >> v2:172.21.15.150:3300/0 conn(0x7f777c0c2c60 0x7f777c0c3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.632+0000 7f77833a1640 1 --2- 172.21.15.150:0/803705944 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f777c070b50 0x7f777c0c2720 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.632+0000 7f77833a1640 1 --2- 172.21.15.150:0/803705944 >> v2:172.21.15.176:3300/0 conn(0x7f777c0a73f0 0x7f777c070610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.632+0000 7f77833a1640 1 -- 172.21.15.150:0/803705944 >> 172.21.15.150:0/803705944 conn(0x7f777c0c7f30 msgr2=0x7f777c0c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.632+0000 7f77833a1640 1 -- 172.21.15.150:0/803705944 shutdown_connections 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.632+0000 7f77833a1640 1 -- 172.21.15.150:0/803705944 wait complete. 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.636+0000 7f77833a1640 1 Processor -- start 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.636+0000 7f77833a1640 1 -- start start 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.636+0000 7f77833a1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f777c070b50 0x7f777c0bce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.636+0000 7f77833a1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f777c0a73f0 0x7f777c087820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.636+0000 7f77833a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f777c0c2c60 0x7f777c0c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.636+0000 7f77833a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f777c08c050 con 0x7f777c0c2c60 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.636+0000 7f77833a1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f777c08bed0 con 0x7f777c0a73f0 2024-09-17T05:16:35.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.636+0000 7f77833a1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f777c08bb50 con 0x7f777c070b50 2024-09-17T05:16:35.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.636+0000 7f778193e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f777c0c2c60 0x7f777c0c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:35.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.636+0000 7f778193e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f777c0c2c60 0x7f777c0c6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54812/0 (socket says 172.21.15.150:54812) 2024-09-17T05:16:35.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.636+0000 7f778193e640 1 -- 172.21.15.150:0/2550585279 learned_addr learned my addr 172.21.15.150:0/2550585279 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:35.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.640+0000 7f778193e640 1 -- 172.21.15.150:0/2550585279 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f777c070b50 msgr2=0x7f777c0bce70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:16:35.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.640+0000 7f778193e640 1 --2- 172.21.15.150:0/2550585279 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f777c070b50 0x7f777c0bce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:35.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.640+0000 7f778193e640 1 -- 172.21.15.150:0/2550585279 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f777c0a73f0 msgr2=0x7f777c087820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:35.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.640+0000 7f778193e640 1 --2- 172.21.15.150:0/2550585279 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f777c0a73f0 0x7f777c087820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:35.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.640+0000 7f778193e640 1 -- 172.21.15.150:0/2550585279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777c08a380 con 0x7f777c0c2c60 2024-09-17T05:16:35.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.640+0000 7f778193e640 1 --2- 172.21.15.150:0/2550585279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f777c0c2c60 0x7f777c0c6fe0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f776c009f50 tx=0x7f776c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:35.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.648+0000 7f7771ffb640 1 -- 172.21.15.150:0/2550585279 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c059020 con 0x7f777c0c2c60 2024-09-17T05:16:35.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.652+0000 7f7771ffb640 1 -- 172.21.15.150:0/2550585279 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f776c055730 con 0x7f777c0c2c60 2024-09-17T05:16:35.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.652+0000 7f7771ffb640 1 -- 172.21.15.150:0/2550585279 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c055b80 con 0x7f777c0c2c60 2024-09-17T05:16:35.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.672+0000 7f77833a1640 1 -- 172.21.15.150:0/2550585279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777c1636c0 con 0x7f777c0c2c60 2024-09-17T05:16:35.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.672+0000 7f77833a1640 1 -- 172.21.15.150:0/2550585279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f777c087020 con 0x7f777c0c2c60 2024-09-17T05:16:35.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.680+0000 7f775f7fe640 1 -- 172.21.15.150:0/2550585279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f777c070610 con 0x7f777c0c2c60 2024-09-17T05:16:35.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.704+0000 7f7771ffb640 1 -- 172.21.15.150:0/2550585279 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f776c05c020 con 0x7f777c0c2c60 2024-09-17T05:16:35.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.708+0000 7f7771ffb640 1 --2- 172.21.15.150:0/2550585279 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7768073410 0x7f7768075920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.708+0000 7f7771ffb640 1 -- 172.21.15.150:0/2550585279 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f776c0d8f80 con 0x7f777c0c2c60 2024-09-17T05:16:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.708+0000 7f778113d640 1 --2- 172.21.15.150:0/2550585279 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7768073410 0x7f7768075920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.708+0000 7f7771ffb640 1 -- 172.21.15.150:0/2550585279 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f776c063020 con 0x7f777c0c2c60 2024-09-17T05:16:35.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:35.708+0000 7f778113d640 1 --2- 172.21.15.150:0/2550585279 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7768073410 0x7f7768075920 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f777400ab90 tx=0x7f7774009230 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:36.037 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:16:36.040 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:16:36.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.852+0000 7f775f7fe640 1 -- 172.21.15.150:0/2550585279 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f777c0c3940 con 0x7f777c0c2c60 2024-09-17T05:16:36.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.852+0000 7f7771ffb640 1 -- 172.21.15.150:0/2550585279 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+6547 (secure 0 0 0) 0x7f776c0a3110 con 0x7f777c0c2c60 2024-09-17T05:16:36.857 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:36.857 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":189,"btime":"2024-09-17T05:16:07:302153+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 189 2024-09-17T05:16:36.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.864+0000 7f775f7fe640 1 -- 172.21.15.150:0/2550585279 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7768073410 msgr2=0x7f7768075920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:36.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.864+0000 7f775f7fe640 1 --2- 172.21.15.150:0/2550585279 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7768073410 0x7f7768075920 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f777400ab90 tx=0x7f7774009230 comp rx=0 tx=0).stop 2024-09-17T05:16:36.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.868+0000 7f775f7fe640 1 -- 172.21.15.150:0/2550585279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f777c0c2c60 msgr2=0x7f777c0c6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:36.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.868+0000 7f775f7fe640 1 --2- 172.21.15.150:0/2550585279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f777c0c2c60 0x7f777c0c6fe0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f776c009f50 tx=0x7f776c00b8d0 comp rx=0 tx=0).stop 2024-09-17T05:16:36.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.868+0000 7f775f7fe640 1 -- 172.21.15.150:0/2550585279 shutdown_connections 2024-09-17T05:16:36.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.868+0000 7f775f7fe640 1 --2- 172.21.15.150:0/2550585279 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7768073410 0x7f7768075920 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:36.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.868+0000 7f775f7fe640 1 --2- 172.21.15.150:0/2550585279 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f777c0a73f0 0x7f777c087820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:36.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.868+0000 7f775f7fe640 1 --2- 172.21.15.150:0/2550585279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f777c0c2c60 0x7f777c0c6fe0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:36.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.868+0000 7f775f7fe640 1 --2- 172.21.15.150:0/2550585279 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f777c070b50 0x7f777c0bce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:36.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.868+0000 7f775f7fe640 1 -- 172.21.15.150:0/2550585279 >> 172.21.15.150:0/2550585279 conn(0x7f777c0c7f30 msgr2=0x7f777c0735f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:36.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.868+0000 7f775f7fe640 1 -- 172.21.15.150:0/2550585279 shutdown_connections 2024-09-17T05:16:36.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:36.868+0000 7f775f7fe640 1 -- 172.21.15.150:0/2550585279 wait complete. 2024-09-17T05:16:36.952 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-09-17T05:16:36.953 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T05:16:36.953 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:16:36.979 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T05:16:36.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:37.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.916+0000 7f33079a7640 1 -- 172.21.15.150:0/2397225555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33000c31b0 msgr2=0x7f33000c35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:37.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.916+0000 7f33079a7640 1 --2- 172.21.15.150:0/2397225555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33000c31b0 0x7f33000c35c0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f3300070140 tx=0x7f32f001c740 comp rx=0 tx=0).stop 2024-09-17T05:16:37.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.920+0000 7f33079a7640 1 -- 172.21.15.150:0/2397225555 shutdown_connections 2024-09-17T05:16:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.920+0000 7f33079a7640 1 --2- 172.21.15.150:0/2397225555 >> v2:172.21.15.176:3300/0 conn(0x7f33000c7f10 0x7f33000c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.920+0000 7f33079a7640 1 --2- 172.21.15.150:0/2397225555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33000c31b0 0x7f33000c35c0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.920+0000 7f33079a7640 1 -- 172.21.15.150:0/2397225555 >> 172.21.15.150:0/2397225555 conn(0x7f33000ca460 msgr2=0x7f33000c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.920+0000 7f33079a7640 1 -- 172.21.15.150:0/2397225555 shutdown_connections 2024-09-17T05:16:37.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.920+0000 7f33079a7640 1 -- 172.21.15.150:0/2397225555 wait complete. 2024-09-17T05:16:37.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f33079a7640 1 Processor -- start 2024-09-17T05:16:37.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f33079a7640 1 -- start start 2024-09-17T05:16:37.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f33079a7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33000c31b0 0x7f3300145220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:37.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f3305743640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33000c31b0 0x7f3300145220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:37.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f3305743640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33000c31b0 0x7f3300145220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:54824/0 (socket says 172.21.15.150:54824) 2024-09-17T05:16:37.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f3305743640 1 -- 172.21.15.150:0/1273729216 learned_addr learned my addr 172.21.15.150:0/1273729216 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:37.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f33079a7640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33000c7f10 0x7f330008b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:37.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f33079a7640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3300156800 0x7f330020d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:37.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f33000727a0 con 0x7f33000c31b0 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3300072620 con 0x7f3300156800 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f33000c15c0 con 0x7f33000c7f10 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f3304f42640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33000c7f10 0x7f330008b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f3305f44640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3300156800 0x7f330020d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f3305743640 1 -- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33000c7f10 msgr2=0x7f330008b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f3305743640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33000c7f10 0x7f330008b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f3305743640 1 -- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3300156800 msgr2=0x7f330020d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f3305743640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3300156800 0x7f330020d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.924+0000 7f3305743640 1 -- 172.21.15.150:0/1273729216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3300146fc0 con 0x7f33000c31b0 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.928+0000 7f3305743640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33000c31b0 0x7f3300145220 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f32f800cc50 tx=0x7f32f800cd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.928+0000 7f3304f42640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33000c7f10 0x7f330008b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.928+0000 7f3305f44640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3300156800 0x7f330020d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.928+0000 7f32f5ffb640 1 -- 172.21.15.150:0/1273729216 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f8016020 con 0x7f33000c31b0 2024-09-17T05:16:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.928+0000 7f32f5ffb640 1 -- 172.21.15.150:0/1273729216 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32f800ab60 con 0x7f33000c31b0 2024-09-17T05:16:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.928+0000 7f32f5ffb640 1 -- 172.21.15.150:0/1273729216 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f8012420 con 0x7f33000c31b0 2024-09-17T05:16:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.928+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3300211410 con 0x7f33000c31b0 2024-09-17T05:16:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.932+0000 7f32f5ffb640 1 -- 172.21.15.150:0/1273729216 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f32f8019020 con 0x7f33000c31b0 2024-09-17T05:16:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.932+0000 7f32f5ffb640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f32ec073560 0x7f32ec075a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.932+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f330014f8f0 con 0x7f33000c31b0 2024-09-17T05:16:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.932+0000 7f3304f42640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f32ec073560 0x7f32ec075a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.932+0000 7f32f5ffb640 1 -- 172.21.15.150:0/1273729216 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f32f8019df0 con 0x7f33000c31b0 2024-09-17T05:16:37.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.936+0000 7f3304f42640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f32ec073560 0x7f32ec075a70 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f32f00af000 tx=0x7f32f001af80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:37.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.936+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32e00052a0 con 0x7f33000c31b0 2024-09-17T05:16:37.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:37.944+0000 7f32f5ffb640 1 -- 172.21.15.150:0/1273729216 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f32f805e8e0 con 0x7f33000c31b0 2024-09-17T05:16:38.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.712+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32e0005860 con 0x7f33000c31b0 2024-09-17T05:16:38.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.712+0000 7f32f5ffb640 1 -- 172.21.15.150:0/1273729216 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+6547 (secure 0 0 0) 0x7f32f805e1e0 con 0x7f33000c31b0 2024-09-17T05:16:38.715 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:38.715 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":189,"btime":"2024-09-17T05:16:07:302153+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:38.715 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 189 2024-09-17T05:16:38.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.724+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f32ec073560 msgr2=0x7f32ec075a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:38.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.724+0000 7f33079a7640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f32ec073560 0x7f32ec075a70 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f32f00af000 tx=0x7f32f001af80 comp rx=0 tx=0).stop 2024-09-17T05:16:38.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.724+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33000c31b0 msgr2=0x7f3300145220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:38.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.724+0000 7f33079a7640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33000c31b0 0x7f3300145220 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f32f800cc50 tx=0x7f32f800cd40 comp rx=0 tx=0).stop 2024-09-17T05:16:38.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.724+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 shutdown_connections 2024-09-17T05:16:38.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.724+0000 7f33079a7640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f32ec073560 0x7f32ec075a70 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:38.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.724+0000 7f33079a7640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3300156800 0x7f330020d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:38.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.724+0000 7f33079a7640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f33000c7f10 0x7f330008b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:38.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.724+0000 7f33079a7640 1 --2- 172.21.15.150:0/1273729216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33000c31b0 0x7f3300145220 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:38.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.724+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 >> 172.21.15.150:0/1273729216 conn(0x7f33000ca460 msgr2=0x7f33000c9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:38.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.724+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 shutdown_connections 2024-09-17T05:16:38.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:38.724+0000 7f33079a7640 1 -- 172.21.15.150:0/1273729216 wait complete. 2024-09-17T05:16:38.845 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:16:39.734 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:40.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.408+0000 7f4a9612f640 1 -- 172.21.15.150:0/1076381784 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a90156c50 msgr2=0x7f4a900c6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:40.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.408+0000 7f4a9612f640 1 --2- 172.21.15.150:0/1076381784 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a90156c50 0x7f4a900c6c40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4a90075620 tx=0x7f4a8801cd30 comp rx=0 tx=0).stop 2024-09-17T05:16:40.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.408+0000 7f4a9612f640 1 -- 172.21.15.150:0/1076381784 shutdown_connections 2024-09-17T05:16:40.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.408+0000 7f4a9612f640 1 --2- 172.21.15.150:0/1076381784 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a90156c50 0x7f4a900c6c40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:40.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.408+0000 7f4a9612f640 1 --2- 172.21.15.150:0/1076381784 >> v2:172.21.15.150:3300/0 conn(0x7f4a90156840 0x7f4a9006d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:40.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.408+0000 7f4a9612f640 1 --2- 172.21.15.150:0/1076381784 >> v2:172.21.15.176:3300/0 conn(0x7f4a90074f70 0x7f4a900734d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:40.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.408+0000 7f4a9612f640 1 -- 172.21.15.150:0/1076381784 >> 172.21.15.150:0/1076381784 conn(0x7f4a900c4e30 msgr2=0x7f4a9006ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:40.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.420+0000 7f4a9612f640 1 -- 172.21.15.150:0/1076381784 shutdown_connections 2024-09-17T05:16:40.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.420+0000 7f4a9612f640 1 -- 172.21.15.150:0/1076381784 wait complete. 2024-09-17T05:16:40.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.420+0000 7f4a9612f640 1 Processor -- start 2024-09-17T05:16:40.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.420+0000 7f4a9612f640 1 -- start start 2024-09-17T05:16:40.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a9612f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a90074f70 0x7f4a90151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:40.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a9612f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a90156840 0x7f4a90159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:40.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a9612f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a90156c50 0x7f4a9014f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:40.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a9612f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4a90071c50 con 0x7f4a90156c50 2024-09-17T05:16:40.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a9612f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9015ac40 con 0x7f4a90074f70 2024-09-17T05:16:40.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a9612f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4a90071dd0 con 0x7f4a90156840 2024-09-17T05:16:40.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a8f7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a90074f70 0x7f4a90151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:40.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a8f7fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a90074f70 0x7f4a90151f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:49862/0 (socket says 172.21.15.150:49862) 2024-09-17T05:16:40.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a8f7fe640 1 -- 172.21.15.150:0/891092782 learned_addr learned my addr 172.21.15.150:0/891092782 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:40.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a8f7fe640 1 -- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a90156840 msgr2=0x7f4a90159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:40.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a8f7fe640 1 --2- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a90156840 0x7f4a90159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:40.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a8f7fe640 1 -- 172.21.15.150:0/891092782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a90156c50 msgr2=0x7f4a9014f3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:16:40.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a8f7fe640 1 --2- 172.21.15.150:0/891092782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a90156c50 0x7f4a9014f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:40.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a8f7fe640 1 -- 172.21.15.150:0/891092782 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a90158c00 con 0x7f4a90074f70 2024-09-17T05:16:40.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a8f7fe640 1 --2- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a90074f70 0x7f4a90151f50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b440 tx=0x7f4a8000b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:40.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a8cff9640 1 -- 172.21.15.150:0/891092782 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80015020 con 0x7f4a90074f70 2024-09-17T05:16:40.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a9612f640 1 -- 172.21.15.150:0/891092782 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9014fe70 con 0x7f4a90074f70 2024-09-17T05:16:40.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a9612f640 1 -- 172.21.15.150:0/891092782 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a900ab020 con 0x7f4a90074f70 2024-09-17T05:16:40.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a8cff9640 1 -- 172.21.15.150:0/891092782 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a80011980 con 0x7f4a90074f70 2024-09-17T05:16:40.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a8cff9640 1 -- 172.21.15.150:0/891092782 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80011dd0 con 0x7f4a90074f70 2024-09-17T05:16:40.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.424+0000 7f4a9612f640 1 -- 172.21.15.150:0/891092782 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a9006d560 con 0x7f4a90074f70 2024-09-17T05:16:40.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.428+0000 7f4a8cff9640 1 -- 172.21.15.150:0/891092782 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4a80018020 con 0x7f4a90074f70 2024-09-17T05:16:40.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.428+0000 7f4a8cff9640 1 --2- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a70073200 0x7f4a70075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:40.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.428+0000 7f4a8cff9640 1 -- 172.21.15.150:0/891092782 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f4a800957c0 con 0x7f4a90074f70 2024-09-17T05:16:40.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.428+0000 7f4a87fff640 1 --2- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a70073200 0x7f4a70075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:40.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.428+0000 7f4a87fff640 1 --2- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a70073200 0x7f4a70075710 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4a900ae060 tx=0x7f4a78006c00 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:40.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:40.444+0000 7f4a8cff9640 1 -- 172.21.15.150:0/891092782 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4a8001f020 con 0x7f4a90074f70 2024-09-17T05:16:40.846 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.268+0000 7fe7397e0640 1 -- 172.21.15.150:0/1943425828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72c00a850 msgr2=0x7fe72c00ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.268+0000 7fe7397e0640 1 --2- 172.21.15.150:0/1943425828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72c00a850 0x7fe72c00ac60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe728009900 tx=0x7fe72801c870 comp rx=0 tx=0).stop 2024-09-17T05:16:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.268+0000 7fe7397e0640 1 -- 172.21.15.150:0/1943425828 shutdown_connections 2024-09-17T05:16:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.268+0000 7fe7397e0640 1 --2- 172.21.15.150:0/1943425828 >> v2:172.21.15.176:3300/0 conn(0x7fe72c0fb370 0x7fe72c0fb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.268+0000 7fe7397e0640 1 --2- 172.21.15.150:0/1943425828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72c00a850 0x7fe72c00ac60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.268+0000 7fe7397e0640 1 -- 172.21.15.150:0/1943425828 >> 172.21.15.150:0/1943425828 conn(0x7fe72c048270 msgr2=0x7fe72c048680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:41.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.300+0000 7fe7397e0640 1 -- 172.21.15.150:0/1943425828 shutdown_connections 2024-09-17T05:16:41.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.304+0000 7fe7397e0640 1 -- 172.21.15.150:0/1943425828 wait complete. 2024-09-17T05:16:41.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.312+0000 7fe7397e0640 1 Processor -- start 2024-09-17T05:16:41.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.312+0000 7fe7397e0640 1 -- start start 2024-09-17T05:16:41.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.312+0000 7fe7397e0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe72c00a850 0x7fe72c19d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:41.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.312+0000 7fe7397e0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72c0463b0 0x7fe72c0448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:41.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.312+0000 7fe7397e0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72c0fb370 0x7fe72c044df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:41.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.312+0000 7fe732ffd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe72c00a850 0x7fe72c19d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:41.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.312+0000 7fe732ffd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe72c00a850 0x7fe72c19d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:49868/0 (socket says 172.21.15.150:49868) 2024-09-17T05:16:41.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.312+0000 7fe7397e0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c0fbcc0 con 0x7fe72c0fb370 2024-09-17T05:16:41.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.312+0000 7fe7337fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72c0fb370 0x7fe72c044df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:41.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.312+0000 7fe7337fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72c0fb370 0x7fe72c044df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49036/0 (socket says 172.21.15.150:49036) 2024-09-17T05:16:41.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.312+0000 7fe732ffd640 1 -- 172.21.15.150:0/2994574250 learned_addr learned my addr 172.21.15.150:0/2994574250 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:41.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.312+0000 7fe7397e0640 1 -- 172.21.15.150:0/2994574250 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c046970 con 0x7fe72c00a850 2024-09-17T05:16:41.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe7397e0640 1 -- 172.21.15.150:0/2994574250 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe72c046af0 con 0x7fe72c0463b0 2024-09-17T05:16:41.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe7327fc640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72c0463b0 0x7fe72c0448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:41.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe7337fe640 1 -- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72c0463b0 msgr2=0x7fe72c0448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:41.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe7337fe640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72c0463b0 0x7fe72c0448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:41.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe7337fe640 1 -- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe72c00a850 msgr2=0x7fe72c19d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:41.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe7337fe640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe72c00a850 0x7fe72c19d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:41.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe7337fe640 1 -- 172.21.15.150:0/2994574250 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72c19de80 con 0x7fe72c0fb370 2024-09-17T05:16:41.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe7337fe640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72c0fb370 0x7fe72c044df0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fe7340529c0 tx=0x7fe734076ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:41.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe71b7fe640 1 -- 172.21.15.150:0/2994574250 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734080070 con 0x7fe72c0fb370 2024-09-17T05:16:41.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe7397e0640 1 -- 172.21.15.150:0/2994574250 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe72c030190 con 0x7fe72c0fb370 2024-09-17T05:16:41.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe7397e0640 1 -- 172.21.15.150:0/2994574250 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe72c19dba0 con 0x7fe72c0fb370 2024-09-17T05:16:41.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe71b7fe640 1 -- 172.21.15.150:0/2994574250 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe73407c6c0 con 0x7fe72c0fb370 2024-09-17T05:16:41.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.316+0000 7fe71b7fe640 1 -- 172.21.15.150:0/2994574250 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73407cb50 con 0x7fe72c0fb370 2024-09-17T05:16:41.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.320+0000 7fe71b7fe640 1 -- 172.21.15.150:0/2994574250 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe734083070 con 0x7fe72c0fb370 2024-09-17T05:16:41.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.324+0000 7fe71b7fe640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe720073300 0x7fe720075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:41.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.324+0000 7fe732ffd640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe720073300 0x7fe720075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:41.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.324+0000 7fe71b7fe640 1 -- 172.21.15.150:0/2994574250 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7fe734100300 con 0x7fe72c0fb370 2024-09-17T05:16:41.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.324+0000 7fe732ffd640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe720073300 0x7fe720075810 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe7240098b0 tx=0x7fe724008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:41.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.324+0000 7fe7397e0640 1 -- 172.21.15.150:0/2994574250 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7140052a0 con 0x7fe72c0fb370 2024-09-17T05:16:41.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.336+0000 7fe71b7fe640 1 -- 172.21.15.150:0/2994574250 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe7340ca400 con 0x7fe72c0fb370 2024-09-17T05:16:41.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.476+0000 7f4a9612f640 1 -- 172.21.15.150:0/891092782 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a90150900 con 0x7f4a90074f70 2024-09-17T05:16:41.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.480+0000 7f4a8cff9640 1 -- 172.21.15.150:0/891092782 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+6547 (secure 0 0 0) 0x7f4a8000d040 con 0x7f4a90074f70 2024-09-17T05:16:41.490 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:41.490 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":189,"btime":"2024-09-17T05:16:07:302153+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:41.490 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 189 2024-09-17T05:16:41.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.532+0000 7f4a85ffb640 1 -- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a70073200 msgr2=0x7f4a70075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:41.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.532+0000 7f4a85ffb640 1 --2- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a70073200 0x7f4a70075710 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4a900ae060 tx=0x7f4a78006c00 comp rx=0 tx=0).stop 2024-09-17T05:16:41.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.532+0000 7f4a85ffb640 1 -- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a90074f70 msgr2=0x7f4a90151f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:41.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.532+0000 7f4a85ffb640 1 --2- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a90074f70 0x7f4a90151f50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b440 tx=0x7f4a8000b900 comp rx=0 tx=0).stop 2024-09-17T05:16:41.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.532+0000 7f4a85ffb640 1 -- 172.21.15.150:0/891092782 shutdown_connections 2024-09-17T05:16:41.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.532+0000 7f4a85ffb640 1 --2- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4a70073200 0x7f4a70075710 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:41.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.532+0000 7f4a85ffb640 1 --2- 172.21.15.150:0/891092782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a90156c50 0x7f4a9014f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:41.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.532+0000 7f4a85ffb640 1 --2- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4a90156840 0x7f4a90159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:41.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.532+0000 7f4a85ffb640 1 --2- 172.21.15.150:0/891092782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a90074f70 0x7f4a90151f50 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:41.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.532+0000 7f4a85ffb640 1 -- 172.21.15.150:0/891092782 >> 172.21.15.150:0/891092782 conn(0x7f4a900c4e30 msgr2=0x7f4a900c69c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:41.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.532+0000 7f4a85ffb640 1 -- 172.21.15.150:0/891092782 shutdown_connections 2024-09-17T05:16:41.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:41.536+0000 7f4a85ffb640 1 -- 172.21.15.150:0/891092782 wait complete. 2024-09-17T05:16:41.655 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-09-17T05:16:41.655 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T05:16:41.656 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:16:41.689 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T05:16:41.689 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:42.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.132+0000 7fb1eb7fe640 1 -- 172.21.15.150:0/2584331858 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e00a6e20 con 0x7fb1e40463e0 2024-09-17T05:16:42.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.132+0000 7fb1f2a6f640 1 -- 172.21.15.150:0/2584331858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1e40463e0 msgr2=0x7fb1e40f5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:42.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.132+0000 7fb1f2a6f640 1 --2- 172.21.15.150:0/2584331858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1e40463e0 0x7fb1e40f5270 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400cd20 tx=0x7fb1e001c870 comp rx=0 tx=0).stop 2024-09-17T05:16:42.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.132+0000 7fb1f2a6f640 1 -- 172.21.15.150:0/2584331858 shutdown_connections 2024-09-17T05:16:42.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.132+0000 7fb1f2a6f640 1 --2- 172.21.15.150:0/2584331858 >> v2:172.21.15.176:3300/0 conn(0x7fb1e40fb3a0 0x7fb1e40fb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.132+0000 7fb1f2a6f640 1 --2- 172.21.15.150:0/2584331858 >> v2:172.21.15.150:3300/0 conn(0x7fb1e400a880 0x7fb1e400ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.132+0000 7fb1f2a6f640 1 --2- 172.21.15.150:0/2584331858 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1e40463e0 0x7fb1e40f5270 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.132+0000 7fb1f2a6f640 1 -- 172.21.15.150:0/2584331858 >> 172.21.15.150:0/2584331858 conn(0x7fb1e40482a0 msgr2=0x7fb1e40486b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:42.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.132+0000 7fb1f2a6f640 1 -- 172.21.15.150:0/2584331858 shutdown_connections 2024-09-17T05:16:42.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.132+0000 7fb1f2a6f640 1 -- 172.21.15.150:0/2584331858 wait complete. 2024-09-17T05:16:42.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1f2a6f640 1 Processor -- start 2024-09-17T05:16:42.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1f2a6f640 1 -- start start 2024-09-17T05:16:42.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1f2a6f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1e400a880 0x7fb1e4042b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:42.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1f2a6f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1e40463e0 0x7fb1e40f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:42.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1f080b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1e400a880 0x7fb1e4042b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:42.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1f080b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1e400a880 0x7fb1e4042b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:50358/0 (socket says 172.21.15.150:50358) 2024-09-17T05:16:42.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1ebfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1e40463e0 0x7fb1e40f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:42.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1ebfff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1e40463e0 0x7fb1e40f3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:49876/0 (socket says 172.21.15.150:49876) 2024-09-17T05:16:42.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1f2a6f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1e40fb3a0 0x7fb1e40ff870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:42.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1f080b640 1 -- 172.21.15.150:0/4176032534 learned_addr learned my addr 172.21.15.150:0/4176032534 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:42.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1f2a6f640 1 -- 172.21.15.150:0/4176032534 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e40fbcf0 con 0x7fb1e40fb3a0 2024-09-17T05:16:42.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1f2a6f640 1 -- 172.21.15.150:0/4176032534 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e404a460 con 0x7fb1e40463e0 2024-09-17T05:16:42.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.136+0000 7fb1f2a6f640 1 -- 172.21.15.150:0/4176032534 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb1e40469a0 con 0x7fb1e400a880 2024-09-17T05:16:42.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.140+0000 7fb1f100c640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1e40fb3a0 0x7fb1e40ff870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:42.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.140+0000 7fb1f080b640 1 -- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1e40463e0 msgr2=0x7fb1e40f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:42.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.140+0000 7fb1f080b640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1e40463e0 0x7fb1e40f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.140+0000 7fb1f080b640 1 -- 172.21.15.150:0/4176032534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1e40fb3a0 msgr2=0x7fb1e40ff870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:42.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.140+0000 7fb1f080b640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1e40fb3a0 0x7fb1e40ff870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.140+0000 7fb1f080b640 1 -- 172.21.15.150:0/4176032534 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e40f4890 con 0x7fb1e400a880 2024-09-17T05:16:42.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.140+0000 7fb1f080b640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1e400a880 0x7fb1e4042b30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb1e001af20 tx=0x7fb1e00a6660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:42.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.140+0000 7fb1e97fa640 1 -- 172.21.15.150:0/4176032534 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e00bb020 con 0x7fb1e400a880 2024-09-17T05:16:42.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.140+0000 7fb1f2a6f640 1 -- 172.21.15.150:0/4176032534 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e40fc990 con 0x7fb1e400a880 2024-09-17T05:16:42.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.140+0000 7fb1f2a6f640 1 -- 172.21.15.150:0/4176032534 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1e40f45b0 con 0x7fb1e400a880 2024-09-17T05:16:42.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.144+0000 7fb1e97fa640 1 -- 172.21.15.150:0/4176032534 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1e0004240 con 0x7fb1e400a880 2024-09-17T05:16:42.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.148+0000 7fb1e97fa640 1 -- 172.21.15.150:0/4176032534 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0003600 con 0x7fb1e400a880 2024-09-17T05:16:42.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.148+0000 7fb1e97fa640 1 -- 172.21.15.150:0/4176032534 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb1e00be020 con 0x7fb1e400a880 2024-09-17T05:16:42.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.148+0000 7fb1e97fa640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1d0073330 0x7fb1d0075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:42.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.148+0000 7fb1ebfff640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1d0073330 0x7fb1d0075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:42.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.148+0000 7fb1e97fa640 1 -- 172.21.15.150:0/4176032534 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7fb1e012fdb0 con 0x7fb1e400a880 2024-09-17T05:16:42.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.148+0000 7fb1ebfff640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1d0073330 0x7fb1d0075840 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0668e0 tx=0x7fb1ec070040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:42.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.152+0000 7fb1d6ffd640 1 -- 172.21.15.150:0/4176032534 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1cc0052a0 con 0x7fb1e400a880 2024-09-17T05:16:42.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.164+0000 7fb1e97fa640 1 -- 172.21.15.150:0/4176032534 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb1e00c5030 con 0x7fb1e400a880 2024-09-17T05:16:42.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.364+0000 7fe7397e0640 1 -- 172.21.15.150:0/2994574250 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe714005590 con 0x7fe72c0fb370 2024-09-17T05:16:42.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.364+0000 7fe71b7fe640 1 -- 172.21.15.150:0/2994574250 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+6547 (secure 0 0 0) 0x7fe7340c9d00 con 0x7fe72c0fb370 2024-09-17T05:16:42.365 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:42.366 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":189,"btime":"2024-09-17T05:16:07:302153+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:42.366 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 189 2024-09-17T05:16:42.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.384+0000 7fe7197fa640 1 -- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe720073300 msgr2=0x7fe720075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:42.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.384+0000 7fe7197fa640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe720073300 0x7fe720075810 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe7240098b0 tx=0x7fe724008040 comp rx=0 tx=0).stop 2024-09-17T05:16:42.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.384+0000 7fe7197fa640 1 -- 172.21.15.150:0/2994574250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72c0fb370 msgr2=0x7fe72c044df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:42.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.384+0000 7fe7197fa640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72c0fb370 0x7fe72c044df0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fe7340529c0 tx=0x7fe734076ea0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.388+0000 7fe7197fa640 1 -- 172.21.15.150:0/2994574250 shutdown_connections 2024-09-17T05:16:42.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.388+0000 7fe7197fa640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe720073300 0x7fe720075810 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.388+0000 7fe7197fa640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe72c0fb370 0x7fe72c044df0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.388+0000 7fe7197fa640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe72c0463b0 0x7fe72c0448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.388+0000 7fe7197fa640 1 --2- 172.21.15.150:0/2994574250 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe72c00a850 0x7fe72c19d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.388+0000 7fe7197fa640 1 -- 172.21.15.150:0/2994574250 >> 172.21.15.150:0/2994574250 conn(0x7fe72c048270 msgr2=0x7fe72c0fa110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:42.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.388+0000 7fe7197fa640 1 -- 172.21.15.150:0/2994574250 shutdown_connections 2024-09-17T05:16:42.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.388+0000 7fe7197fa640 1 -- 172.21.15.150:0/2994574250 wait complete. 2024-09-17T05:16:42.426 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:16:42.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.816+0000 7fb1d6ffd640 1 -- 172.21.15.150:0/4176032534 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1cc005590 con 0x7fb1e400a880 2024-09-17T05:16:42.818 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:42.818 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":189,"btime":"2024-09-17T05:16:07:302153+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:42.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.816+0000 7fb1e97fa640 1 -- 172.21.15.150:0/4176032534 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+6547 (secure 0 0 0) 0x7fb1e00164f0 con 0x7fb1e400a880 2024-09-17T05:16:42.818 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 189 2024-09-17T05:16:42.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.820+0000 7fb1d6ffd640 1 -- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1d0073330 msgr2=0x7fb1d0075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:42.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.820+0000 7fb1d6ffd640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1d0073330 0x7fb1d0075840 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0668e0 tx=0x7fb1ec070040 comp rx=0 tx=0).stop 2024-09-17T05:16:42.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.820+0000 7fb1d6ffd640 1 -- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1e400a880 msgr2=0x7fb1e4042b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:42.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.820+0000 7fb1d6ffd640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1e400a880 0x7fb1e4042b30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb1e001af20 tx=0x7fb1e00a6660 comp rx=0 tx=0).stop 2024-09-17T05:16:42.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.820+0000 7fb1d6ffd640 1 -- 172.21.15.150:0/4176032534 shutdown_connections 2024-09-17T05:16:42.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.820+0000 7fb1d6ffd640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb1d0073330 0x7fb1d0075840 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.820+0000 7fb1d6ffd640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1e40463e0 0x7fb1e40f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.820+0000 7fb1d6ffd640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb1e40fb3a0 0x7fb1e40ff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.820+0000 7fb1d6ffd640 1 --2- 172.21.15.150:0/4176032534 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb1e400a880 0x7fb1e4042b30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:42.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.820+0000 7fb1d6ffd640 1 -- 172.21.15.150:0/4176032534 >> 172.21.15.150:0/4176032534 conn(0x7fb1e40482a0 msgr2=0x7fb1e4030070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:42.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.820+0000 7fb1d6ffd640 1 -- 172.21.15.150:0/4176032534 shutdown_connections 2024-09-17T05:16:42.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:42.820+0000 7fb1d6ffd640 1 -- 172.21.15.150:0/4176032534 wait complete. 2024-09-17T05:16:42.852 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:16:44.427 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:44.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.592+0000 7f871840a640 1 -- 172.21.15.150:0/4236921362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86fc004810 msgr2=0x7f86fc004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:44.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.592+0000 7f871840a640 1 --2- 172.21.15.150:0/4236921362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86fc004810 0x7f86fc004c20 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f8700009b90 tx=0x7f870001d2b0 comp rx=0 tx=0).stop 2024-09-17T05:16:44.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.592+0000 7f87169a7640 1 -- 172.21.15.150:0/4236921362 reap_dead start 2024-09-17T05:16:44.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.592+0000 7f871840a640 1 -- 172.21.15.150:0/4236921362 shutdown_connections 2024-09-17T05:16:44.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.592+0000 7f871840a640 1 -- 172.21.15.150:0/4236921362 >> 172.21.15.150:0/4236921362 conn(0x7f8710070b50 msgr2=0x7f8710070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:44.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.592+0000 7f871840a640 1 -- 172.21.15.150:0/4236921362 shutdown_connections 2024-09-17T05:16:44.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.592+0000 7f871840a640 1 -- 172.21.15.150:0/4236921362 wait complete. 2024-09-17T05:16:44.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.592+0000 7f871840a640 1 Processor -- start 2024-09-17T05:16:44.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.592+0000 7f871840a640 1 -- start start 2024-09-17T05:16:44.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f871840a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86fc004810 0x7f871015c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f871840a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f871006d020 0x7f87100ae330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f871840a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f871006d430 0x7f8710150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f871840a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f87100c3ac0 con 0x7f871006d020 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f871840a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8710069e00 con 0x7f86fc004810 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f871840a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8710154200 con 0x7f871006d430 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f87159a5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f871006d020 0x7f87100ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f87159a5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f871006d020 0x7f87100ae330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49068/0 (socket says 172.21.15.150:49068) 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f87159a5640 1 -- 172.21.15.150:0/1105844948 learned_addr learned my addr 172.21.15.150:0/1105844948 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f87161a6640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86fc004810 0x7f871015c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f87169a7640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f871006d430 0x7f8710150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f87159a5640 1 -- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f871006d430 msgr2=0x7f8710150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f87159a5640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f871006d430 0x7f8710150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:44.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f87159a5640 1 -- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86fc004810 msgr2=0x7f871015c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:44.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f87159a5640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86fc004810 0x7f871015c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:44.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f87159a5640 1 -- 172.21.15.150:0/1105844948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87100ad690 con 0x7f871006d020 2024-09-17T05:16:44.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f87169a7640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f871006d430 0x7f8710150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:16:44.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f87159a5640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f871006d020 0x7f87100ae330 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f87000af040 tx=0x7f87000055d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:44.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f8706ffd640 1 -- 172.21.15.150:0/1105844948 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87000b1040 con 0x7f871006d020 2024-09-17T05:16:44.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f8706ffd640 1 -- 172.21.15.150:0/1105844948 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87000bb070 con 0x7f871006d020 2024-09-17T05:16:44.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f871840a640 1 -- 172.21.15.150:0/1105844948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87100adc70 con 0x7f871006d020 2024-09-17T05:16:44.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f8706ffd640 1 -- 172.21.15.150:0/1105844948 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8700004380 con 0x7f871006d020 2024-09-17T05:16:44.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f871840a640 1 -- 172.21.15.150:0/1105844948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87101f46f0 con 0x7f871006d020 2024-09-17T05:16:44.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f871840a640 1 -- 172.21.15.150:0/1105844948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f871006e370 con 0x7f871006d020 2024-09-17T05:16:44.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f8706ffd640 1 -- 172.21.15.150:0/1105844948 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f87000046d0 con 0x7f871006d020 2024-09-17T05:16:44.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.596+0000 7f8706ffd640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f86f4073200 0x7f86f4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:44.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.600+0000 7f8706ffd640 1 -- 172.21.15.150:0/1105844948 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f87001303a0 con 0x7f871006d020 2024-09-17T05:16:44.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.600+0000 7f87161a6640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f86f4073200 0x7f86f4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:44.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.600+0000 7f87161a6640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f86f4073200 0x7f86f4075710 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f870c009420 tx=0x7f870c0099e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:44.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.604+0000 7f8706ffd640 1 -- 172.21.15.150:0/1105844948 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8700017000 con 0x7f871006d020 2024-09-17T05:16:44.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 -- 172.21.15.150:0/1105844948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87100aa9c0 con 0x7f871006d020 2024-09-17T05:16:44.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f8706ffd640 1 -- 172.21.15.150:0/1105844948 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+6547 (secure 0 0 0) 0x7f87000fa480 con 0x7f871006d020 2024-09-17T05:16:44.795 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:44.795 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":189,"btime":"2024-09-17T05:16:07:302153+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:44.795 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 189 2024-09-17T05:16:44.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 -- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f86f4073200 msgr2=0x7f86f4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:44.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f86f4073200 0x7f86f4075710 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f870c009420 tx=0x7f870c0099e0 comp rx=0 tx=0).stop 2024-09-17T05:16:44.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 -- 172.21.15.150:0/1105844948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f871006d020 msgr2=0x7f87100ae330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:44.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f871006d020 0x7f87100ae330 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f87000af040 tx=0x7f87000055d0 comp rx=0 tx=0).stop 2024-09-17T05:16:44.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 -- 172.21.15.150:0/1105844948 shutdown_connections 2024-09-17T05:16:44.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f86f4073200 0x7f86f4075710 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:44.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f871006d430 0x7f8710150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:44.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f871006d020 0x7f87100ae330 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:44.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 --2- 172.21.15.150:0/1105844948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f86fc004810 0x7f871015c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:44.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 -- 172.21.15.150:0/1105844948 >> 172.21.15.150:0/1105844948 conn(0x7f8710070b50 msgr2=0x7f8710149550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:44.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 -- 172.21.15.150:0/1105844948 shutdown_connections 2024-09-17T05:16:44.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:44.796+0000 7f871840a640 1 -- 172.21.15.150:0/1105844948 wait complete. 2024-09-17T05:16:44.816 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:16:44.852 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:45.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.003+0000 7ff451447640 1 -- 172.21.15.150:0/3849648786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c074450 msgr2=0x7ff44c074860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:45.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.003+0000 7ff451447640 1 --2- 172.21.15.150:0/3849648786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c074450 0x7ff44c074860 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff44c072370 tx=0x7ff44401ccf0 comp rx=0 tx=0).stop 2024-09-17T05:16:45.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.003+0000 7ff451447640 1 -- 172.21.15.150:0/3849648786 shutdown_connections 2024-09-17T05:16:45.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.003+0000 7ff451447640 1 --2- 172.21.15.150:0/3849648786 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c074450 0x7ff44c074860 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:45.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.003+0000 7ff451447640 1 --2- 172.21.15.150:0/3849648786 >> v2:172.21.15.176:3300/0 conn(0x7ff44c073440 0x7ff44c0a7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:45.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.003+0000 7ff451447640 1 -- 172.21.15.150:0/3849648786 >> 172.21.15.150:0/3849648786 conn(0x7ff44c0c4a00 msgr2=0x7ff44c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:45.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.003+0000 7ff451447640 1 -- 172.21.15.150:0/3849648786 shutdown_connections 2024-09-17T05:16:45.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.003+0000 7ff451447640 1 -- 172.21.15.150:0/3849648786 wait complete. 2024-09-17T05:16:45.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff451447640 1 Processor -- start 2024-09-17T05:16:45.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff451447640 1 -- start start 2024-09-17T05:16:45.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff451447640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff44c070e50 0x7ff44c0c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:45.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff451447640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff44c073440 0x7ff44c1f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:45.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff451447640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c074450 0x7ff44c143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:45.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff451447640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c0ade60 con 0x7ff44c074450 2024-09-17T05:16:45.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff451447640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff44c0adce0 con 0x7ff44c070e50 2024-09-17T05:16:45.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff451447640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff44c0ad610 con 0x7ff44c073440 2024-09-17T05:16:45.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff44b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c074450 0x7ff44c143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:45.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff44b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c074450 0x7ff44c143710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49096/0 (socket says 172.21.15.150:49096) 2024-09-17T05:16:45.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff44b7fe640 1 -- 172.21.15.150:0/560122220 learned_addr learned my addr 172.21.15.150:0/560122220 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:45.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff44affd640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff44c070e50 0x7ff44c0c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:45.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff44a7fc640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff44c073440 0x7ff44c1f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:45.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff44b7fe640 1 -- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff44c073440 msgr2=0x7ff44c1f3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:45.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff44b7fe640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff44c073440 0x7ff44c1f3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:45.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff44b7fe640 1 -- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff44c070e50 msgr2=0x7ff44c0c1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:45.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff44b7fe640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff44c070e50 0x7ff44c0c1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:45.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff44b7fe640 1 -- 172.21.15.150:0/560122220 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44c14c200 con 0x7ff44c074450 2024-09-17T05:16:45.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff44b7fe640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c074450 0x7ff44c143710 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7ff44401c760 tx=0x7ff444005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:45.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff4337fe640 1 -- 172.21.15.150:0/560122220 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4440bc070 con 0x7ff44c074450 2024-09-17T05:16:45.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff4337fe640 1 -- 172.21.15.150:0/560122220 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4440042f0 con 0x7ff44c074450 2024-09-17T05:16:45.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff4337fe640 1 -- 172.21.15.150:0/560122220 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4440b2040 con 0x7ff44c074450 2024-09-17T05:16:45.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff451447640 1 -- 172.21.15.150:0/560122220 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff44c1444e0 con 0x7ff44c074450 2024-09-17T05:16:45.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff451447640 1 -- 172.21.15.150:0/560122220 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff44c14d160 con 0x7ff44c074450 2024-09-17T05:16:45.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.007+0000 7ff451447640 1 -- 172.21.15.150:0/560122220 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff44c14d360 con 0x7ff44c074450 2024-09-17T05:16:45.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.011+0000 7ff4337fe640 1 -- 172.21.15.150:0/560122220 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff4440bf070 con 0x7ff44c074450 2024-09-17T05:16:45.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.011+0000 7ff4337fe640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff438073230 0x7ff438075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:45.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.011+0000 7ff4337fe640 1 -- 172.21.15.150:0/560122220 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7ff44412f050 con 0x7ff44c074450 2024-09-17T05:16:45.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.011+0000 7ff44affd640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff438073230 0x7ff438075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:45.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.011+0000 7ff44affd640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff438073230 0x7ff438075740 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0058d0 tx=0x7ff43c005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:45.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.015+0000 7ff4337fe640 1 -- 172.21.15.150:0/560122220 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff444016970 con 0x7ff44c074450 2024-09-17T05:16:45.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.227+0000 7ff451447640 1 -- 172.21.15.150:0/560122220 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff44c0ad9d0 con 0x7ff44c074450 2024-09-17T05:16:45.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.227+0000 7ff4337fe640 1 -- 172.21.15.150:0/560122220 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+6547 (secure 0 0 0) 0x7ff4440f9020 con 0x7ff44c074450 2024-09-17T05:16:45.228 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:45.228 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":189,"btime":"2024-09-17T05:16:07:302153+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:45.228 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 189 2024-09-17T05:16:45.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.231+0000 7ff451447640 1 -- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff438073230 msgr2=0x7ff438075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:45.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.231+0000 7ff451447640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff438073230 0x7ff438075740 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0058d0 tx=0x7ff43c005860 comp rx=0 tx=0).stop 2024-09-17T05:16:45.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.231+0000 7ff451447640 1 -- 172.21.15.150:0/560122220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c074450 msgr2=0x7ff44c143710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:45.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.231+0000 7ff451447640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c074450 0x7ff44c143710 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7ff44401c760 tx=0x7ff444005b10 comp rx=0 tx=0).stop 2024-09-17T05:16:45.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.231+0000 7ff451447640 1 -- 172.21.15.150:0/560122220 shutdown_connections 2024-09-17T05:16:45.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.231+0000 7ff451447640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff438073230 0x7ff438075740 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:45.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.231+0000 7ff451447640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff44c074450 0x7ff44c143710 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:45.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.231+0000 7ff451447640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff44c073440 0x7ff44c1f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:45.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.231+0000 7ff451447640 1 --2- 172.21.15.150:0/560122220 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff44c070e50 0x7ff44c0c1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:45.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.231+0000 7ff451447640 1 -- 172.21.15.150:0/560122220 >> 172.21.15.150:0/560122220 conn(0x7ff44c0c4a00 msgr2=0x7ff44c15e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:45.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.231+0000 7ff451447640 1 -- 172.21.15.150:0/560122220 shutdown_connections 2024-09-17T05:16:45.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:45.231+0000 7ff451447640 1 -- 172.21.15.150:0/560122220 wait complete. 2024-09-17T05:16:45.250 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:16:46.816 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:46.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.983+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1085489909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d040c8fc0 msgr2=0x7f7d040c20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:46.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.983+0000 7f7d0c7d7640 1 --2- 172.21.15.150:0/1085489909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d040c8fc0 0x7f7d040c20a0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f7d040c5f20 tx=0x7f7cf401c890 comp rx=0 tx=0).stop 2024-09-17T05:16:46.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.983+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1085489909 shutdown_connections 2024-09-17T05:16:46.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.983+0000 7f7d0c7d7640 1 --2- 172.21.15.150:0/1085489909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d040c8fc0 0x7f7d040c20a0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:46.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.983+0000 7f7d0c7d7640 1 --2- 172.21.15.150:0/1085489909 >> v2:172.21.15.176:3300/0 conn(0x7f7d040c35b0 0x7f7d040c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:46.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.983+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1085489909 >> 172.21.15.150:0/1085489909 conn(0x7f7d04074b80 msgr2=0x7f7d0406f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:46.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.983+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1085489909 shutdown_connections 2024-09-17T05:16:46.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.983+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1085489909 wait complete. 2024-09-17T05:16:46.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d0c7d7640 1 Processor -- start 2024-09-17T05:16:46.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d0c7d7640 1 -- start start 2024-09-17T05:16:46.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d0c7d7640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d040c35b0 0x7f7d0414cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:46.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d0c7d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d040c8fc0 0x7f7d041f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:46.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d0c7d7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d041568a0 0x7f7d04144e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:46.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d0c7d7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7d040c6b70 con 0x7f7d040c8fc0 2024-09-17T05:16:46.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d0c7d7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7d040c69f0 con 0x7f7d041568a0 2024-09-17T05:16:46.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d0c7d7640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7d040b0fd0 con 0x7f7d040c35b0 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d09d72640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d040c8fc0 0x7f7d041f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d09d72640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d040c8fc0 0x7f7d041f3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49114/0 (socket says 172.21.15.150:49114) 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d09d72640 1 -- 172.21.15.150:0/1875502862 learned_addr learned my addr 172.21.15.150:0/1875502862 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d0a573640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d040c35b0 0x7f7d0414cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d0ad74640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d041568a0 0x7f7d04144e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d09d72640 1 -- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d040c35b0 msgr2=0x7f7d0414cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d09d72640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d040c35b0 0x7f7d0414cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d09d72640 1 -- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d041568a0 msgr2=0x7f7d04144e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d09d72640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d041568a0 0x7f7d04144e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d09d72640 1 -- 172.21.15.150:0/1875502862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0406a920 con 0x7f7d040c8fc0 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d09d72640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d040c8fc0 0x7f7d041f3c30 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800d6a0 tx=0x7f7cf800db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d02ffd640 1 -- 172.21.15.150:0/1875502862 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8017070 con 0x7f7d040c8fc0 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d02ffd640 1 -- 172.21.15.150:0/1875502862 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cf8009b10 con 0x7f7d040c8fc0 2024-09-17T05:16:46.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d02ffd640 1 -- 172.21.15.150:0/1875502862 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8010460 con 0x7f7d040c8fc0 2024-09-17T05:16:46.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1875502862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d040ccbe0 con 0x7f7d040c8fc0 2024-09-17T05:16:46.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.987+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1875502862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d04153dc0 con 0x7f7d040c8fc0 2024-09-17T05:16:46.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.991+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1875502862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d04154230 con 0x7f7d040c8fc0 2024-09-17T05:16:46.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.991+0000 7f7d02ffd640 1 -- 172.21.15.150:0/1875502862 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7cf801a070 con 0x7f7d040c8fc0 2024-09-17T05:16:46.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.991+0000 7f7d02ffd640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7cf0073200 0x7f7cf0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:46.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.991+0000 7f7d02ffd640 1 -- 172.21.15.150:0/1875502862 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f7cf8095330 con 0x7f7d040c8fc0 2024-09-17T05:16:46.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.991+0000 7f7d0a573640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7cf0073200 0x7f7cf0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:46.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.991+0000 7f7d0a573640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7cf0073200 0x7f7cf0075710 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc007bf0 tx=0x7f7cfc007240 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:46.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:46.995+0000 7f7d02ffd640 1 -- 172.21.15.150:0/1875502862 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7cf805f4a0 con 0x7f7d040c8fc0 2024-09-17T05:16:47.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.203+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1875502862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d04157790 con 0x7f7d040c8fc0 2024-09-17T05:16:47.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.203+0000 7f7d02ffd640 1 -- 172.21.15.150:0/1875502862 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+6547 (secure 0 0 0) 0x7f7cf805eda0 con 0x7f7d040c8fc0 2024-09-17T05:16:47.204 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:47.204 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":189,"btime":"2024-09-17T05:16:07:302153+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:47.204 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 189 2024-09-17T05:16:47.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.207+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7cf0073200 msgr2=0x7f7cf0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:47.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.207+0000 7f7d0c7d7640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7cf0073200 0x7f7cf0075710 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc007bf0 tx=0x7f7cfc007240 comp rx=0 tx=0).stop 2024-09-17T05:16:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.207+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1875502862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d040c8fc0 msgr2=0x7f7d041f3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.207+0000 7f7d0c7d7640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d040c8fc0 0x7f7d041f3c30 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800d6a0 tx=0x7f7cf800db60 comp rx=0 tx=0).stop 2024-09-17T05:16:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.207+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1875502862 shutdown_connections 2024-09-17T05:16:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.207+0000 7f7d0c7d7640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7cf0073200 0x7f7cf0075710 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.207+0000 7f7d0c7d7640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d041568a0 0x7f7d04144e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.207+0000 7f7d0c7d7640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d040c8fc0 0x7f7d041f3c30 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.207+0000 7f7d0c7d7640 1 --2- 172.21.15.150:0/1875502862 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7d040c35b0 0x7f7d0414cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.207+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1875502862 >> 172.21.15.150:0/1875502862 conn(0x7f7d04074b80 msgr2=0x7f7d041618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.207+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1875502862 shutdown_connections 2024-09-17T05:16:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.207+0000 7f7d0c7d7640 1 -- 172.21.15.150:0/1875502862 wait complete. 2024-09-17T05:16:47.225 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:16:47.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:47.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.391+0000 7faec2890640 1 -- 172.21.15.150:0/3243759275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faebc0c3db0 msgr2=0x7faebc0ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:47.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.391+0000 7faec2890640 1 --2- 172.21.15.150:0/3243759275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faebc0c3db0 0x7faebc0ad2a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7faebc06f7a0 tx=0x7faeb001c8a0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.391+0000 7faec2890640 1 -- 172.21.15.150:0/3243759275 shutdown_connections 2024-09-17T05:16:47.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.391+0000 7faec2890640 1 --2- 172.21.15.150:0/3243759275 >> v2:172.21.15.176:3300/0 conn(0x7faebc0ad7e0 0x7faebc0701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.391+0000 7faec2890640 1 --2- 172.21.15.150:0/3243759275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faebc0c3db0 0x7faebc0ad2a0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.391+0000 7faec2890640 1 -- 172.21.15.150:0/3243759275 >> 172.21.15.150:0/3243759275 conn(0x7faebc0c4a00 msgr2=0x7faebc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:47.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.391+0000 7faec2890640 1 -- 172.21.15.150:0/3243759275 shutdown_connections 2024-09-17T05:16:47.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.391+0000 7faec2890640 1 -- 172.21.15.150:0/3243759275 wait complete. 2024-09-17T05:16:47.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.391+0000 7faec2890640 1 Processor -- start 2024-09-17T05:16:47.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.391+0000 7faec2890640 1 -- start start 2024-09-17T05:16:47.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faec2890640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faebc0ad7e0 0x7faebc158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:47.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faec2890640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faebc0c3db0 0x7faebc14e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:47.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faec2890640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faebc0c41c0 0x7faebc14a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:47.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faec2890640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7faebc0a75f0 con 0x7faebc0c3db0 2024-09-17T05:16:47.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faec2890640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7faebc0add70 con 0x7faebc0c41c0 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faec2890640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7faebc0a7770 con 0x7faebc0ad7e0 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faebc0c3db0 0x7faebc14e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faebb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faebc0c3db0 0x7faebc14e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49134/0 (socket says 172.21.15.150:49134) 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faebb7fe640 1 -- 172.21.15.150:0/3812690918 learned_addr learned my addr 172.21.15.150:0/3812690918 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faebbfff640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faebc0ad7e0 0x7faebc158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faec0e2d640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faebc0c41c0 0x7faebc14a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faebb7fe640 1 -- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faebc0ad7e0 msgr2=0x7faebc158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faebb7fe640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faebc0ad7e0 0x7faebc158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faebb7fe640 1 -- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faebc0c41c0 msgr2=0x7faebc14a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faebb7fe640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faebc0c41c0 0x7faebc14a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faebb7fe640 1 -- 172.21.15.150:0/3812690918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faebc14b010 con 0x7faebc0c3db0 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faebb7fe640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faebc0c3db0 0x7faebc14e170 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7faeac00b410 tx=0x7faeac00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:47.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faeb8ff9640 1 -- 172.21.15.150:0/3812690918 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac015030 con 0x7faebc0c3db0 2024-09-17T05:16:47.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faeb8ff9640 1 -- 172.21.15.150:0/3812690918 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faeac004020 con 0x7faebc0c3db0 2024-09-17T05:16:47.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faec2890640 1 -- 172.21.15.150:0/3812690918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faebc145ea0 con 0x7faebc0c3db0 2024-09-17T05:16:47.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faeb8ff9640 1 -- 172.21.15.150:0/3812690918 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac004470 con 0x7faebc0c3db0 2024-09-17T05:16:47.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faec2890640 1 -- 172.21.15.150:0/3812690918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faebc157b60 con 0x7faebc0c3db0 2024-09-17T05:16:47.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faec2890640 1 -- 172.21.15.150:0/3812690918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faebc0ad2a0 con 0x7faebc0c3db0 2024-09-17T05:16:47.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faeb8ff9640 1 -- 172.21.15.150:0/3812690918 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7faeac018070 con 0x7faebc0c3db0 2024-09-17T05:16:47.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faeb8ff9640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faea8073200 0x7faea8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:47.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.395+0000 7faeb8ff9640 1 -- 172.21.15.150:0/3812690918 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7faeac094d40 con 0x7faebc0c3db0 2024-09-17T05:16:47.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.399+0000 7faebbfff640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faea8073200 0x7faea8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:47.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.399+0000 7faebbfff640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faea8073200 0x7faea8075710 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7faeb00af000 tx=0x7faeb001b0e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:47.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.403+0000 7faeb8ff9640 1 -- 172.21.15.150:0/3812690918 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7faeac05ee20 con 0x7faebc0c3db0 2024-09-17T05:16:47.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.591+0000 7faec2890640 1 -- 172.21.15.150:0/3812690918 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faebc0c6c20 con 0x7faebc0c3db0 2024-09-17T05:16:47.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faeb8ff9640 1 -- 172.21.15.150:0/3812690918 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+6547 (secure 0 0 0) 0x7faeac05e720 con 0x7faebc0c3db0 2024-09-17T05:16:47.594 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:47.594 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":189,"btime":"2024-09-17T05:16:07:302153+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:47.594 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 189 2024-09-17T05:16:47.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faec2890640 1 -- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faea8073200 msgr2=0x7faea8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:47.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faec2890640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faea8073200 0x7faea8075710 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7faeb00af000 tx=0x7faeb001b0e0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faec2890640 1 -- 172.21.15.150:0/3812690918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faebc0c3db0 msgr2=0x7faebc14e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faec2890640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faebc0c3db0 0x7faebc14e170 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7faeac00b410 tx=0x7faeac00b8d0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faec2890640 1 -- 172.21.15.150:0/3812690918 shutdown_connections 2024-09-17T05:16:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faec2890640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7faea8073200 0x7faea8075710 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faec2890640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7faebc0c41c0 0x7faebc14a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faec2890640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faebc0c3db0 0x7faebc14e170 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faec2890640 1 --2- 172.21.15.150:0/3812690918 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7faebc0ad7e0 0x7faebc158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faec2890640 1 -- 172.21.15.150:0/3812690918 >> 172.21.15.150:0/3812690918 conn(0x7faebc0c4a00 msgr2=0x7faebc15e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faec2890640 1 -- 172.21.15.150:0/3812690918 shutdown_connections 2024-09-17T05:16:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:47.595+0000 7faec2890640 1 -- 172.21.15.150:0/3812690918 wait complete. 2024-09-17T05:16:47.615 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:16:49.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:49.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.387+0000 7f2286487640 1 -- 172.21.15.150:0/2798292734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22800c5b70 msgr2=0x7f2280149b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:49.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.387+0000 7f2286487640 1 --2- 172.21.15.150:0/2798292734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22800c5b70 0x7f2280149b20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2278009bc0 tx=0x7f227801cb10 comp rx=0 tx=0).stop 2024-09-17T05:16:49.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.387+0000 7f2286487640 1 -- 172.21.15.150:0/2798292734 shutdown_connections 2024-09-17T05:16:49.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.387+0000 7f2286487640 1 --2- 172.21.15.150:0/2798292734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22800c5b70 0x7f2280149b20 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.387+0000 7f2286487640 1 --2- 172.21.15.150:0/2798292734 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22801535b0 0x7f2280157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.387+0000 7f2286487640 1 -- 172.21.15.150:0/2798292734 >> 172.21.15.150:0/2798292734 conn(0x7f22800aaa80 msgr2=0x7f22800aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:49.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.387+0000 7f2286487640 1 -- 172.21.15.150:0/2798292734 shutdown_connections 2024-09-17T05:16:49.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.387+0000 7f2286487640 1 -- 172.21.15.150:0/2798292734 wait complete. 2024-09-17T05:16:49.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.387+0000 7f2286487640 1 Processor -- start 2024-09-17T05:16:49.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.387+0000 7f2286487640 1 -- start start 2024-09-17T05:16:49.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f2286487640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22800c5760 0x7f2280150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:49.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f2286487640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22800c5b70 0x7f22801ec900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:49.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f2286487640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22801535b0 0x7f2280141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:49.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f2286487640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f22800718d0 con 0x7f22800c5b70 2024-09-17T05:16:49.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f2286487640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2280157a00 con 0x7f22801535b0 2024-09-17T05:16:49.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f2286487640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f22800c5370 con 0x7f22800c5760 2024-09-17T05:16:49.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22800c5b70 0x7f22801ec900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:49.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22800c5b70 0x7f22801ec900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47906/0 (socket says 172.21.15.150:47906) 2024-09-17T05:16:49.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227f7fe640 1 -- 172.21.15.150:0/1675907538 learned_addr learned my addr 172.21.15.150:0/1675907538 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:49.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f2284a24640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22801535b0 0x7f2280141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:49.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227ffff640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22800c5760 0x7f2280150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:49.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227f7fe640 1 -- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22800c5760 msgr2=0x7f2280150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:49.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227f7fe640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22800c5760 0x7f2280150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227f7fe640 1 -- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22801535b0 msgr2=0x7f2280141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:49.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227f7fe640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22801535b0 0x7f2280141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227f7fe640 1 -- 172.21.15.150:0/1675907538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2280152920 con 0x7f22800c5b70 2024-09-17T05:16:49.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227f7fe640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22800c5b70 0x7f22801ec900 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f227400e640 tx=0x7f227400eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:49.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227cff9640 1 -- 172.21.15.150:0/1675907538 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274015070 con 0x7f22800c5b70 2024-09-17T05:16:49.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227cff9640 1 -- 172.21.15.150:0/1675907538 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2274003600 con 0x7f22800c5b70 2024-09-17T05:16:49.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f2286487640 1 -- 172.21.15.150:0/1675907538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f228014e840 con 0x7f22800c5b70 2024-09-17T05:16:49.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227cff9640 1 -- 172.21.15.150:0/1675907538 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274003a50 con 0x7f22800c5b70 2024-09-17T05:16:49.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f2286487640 1 -- 172.21.15.150:0/1675907538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f228014e040 con 0x7f22800c5b70 2024-09-17T05:16:49.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f2286487640 1 -- 172.21.15.150:0/1675907538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22801539c0 con 0x7f22800c5b70 2024-09-17T05:16:49.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227cff9640 1 -- 172.21.15.150:0/1675907538 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2274018070 con 0x7f22800c5b70 2024-09-17T05:16:49.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227cff9640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f226c073200 0x7f226c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:49.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227cff9640 1 -- 172.21.15.150:0/1675907538 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f22740973c0 con 0x7f22800c5b70 2024-09-17T05:16:49.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.391+0000 7f227ffff640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f226c073200 0x7f226c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:49.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.395+0000 7f227ffff640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f226c073200 0x7f226c075710 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f2270009ad0 tx=0x7f2270009260 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:49.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.395+0000 7f227cff9640 1 -- 172.21.15.150:0/1675907538 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2274061530 con 0x7f22800c5b70 2024-09-17T05:16:49.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.587+0000 7f2286487640 1 -- 172.21.15.150:0/1675907538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22800c3070 con 0x7f22800c5b70 2024-09-17T05:16:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.587+0000 7f227cff9640 1 -- 172.21.15.150:0/1675907538 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+6547 (secure 0 0 0) 0x7f2274060e30 con 0x7f22800c5b70 2024-09-17T05:16:49.587 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:49.588 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":189,"btime":"2024-09-17T05:16:07:302153+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 189 2024-09-17T05:16:49.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.587+0000 7f2286487640 1 -- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f226c073200 msgr2=0x7f226c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:49.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.587+0000 7f2286487640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f226c073200 0x7f226c075710 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f2270009ad0 tx=0x7f2270009260 comp rx=0 tx=0).stop 2024-09-17T05:16:49.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.587+0000 7f2286487640 1 -- 172.21.15.150:0/1675907538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22800c5b70 msgr2=0x7f22801ec900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:49.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.587+0000 7f2286487640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22800c5b70 0x7f22801ec900 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f227400e640 tx=0x7f227400eb00 comp rx=0 tx=0).stop 2024-09-17T05:16:49.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.587+0000 7f2286487640 1 -- 172.21.15.150:0/1675907538 shutdown_connections 2024-09-17T05:16:49.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.587+0000 7f2286487640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f226c073200 0x7f226c075710 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.587+0000 7f2286487640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22801535b0 0x7f2280141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.591+0000 7f2286487640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22800c5b70 0x7f22801ec900 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.591+0000 7f2286487640 1 --2- 172.21.15.150:0/1675907538 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22800c5760 0x7f2280150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.591+0000 7f2286487640 1 -- 172.21.15.150:0/1675907538 >> 172.21.15.150:0/1675907538 conn(0x7f22800aaa80 msgr2=0x7f2280152010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:49.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.591+0000 7f2286487640 1 -- 172.21.15.150:0/1675907538 shutdown_connections 2024-09-17T05:16:49.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.591+0000 7f2286487640 1 -- 172.21.15.150:0/1675907538 wait complete. 2024-09-17T05:16:49.608 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:16:49.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:49.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.767+0000 7fc8508fc640 1 -- 172.21.15.150:0/773542524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc84c0c3660 msgr2=0x7fc84c156c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:49.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.767+0000 7fc8508fc640 1 --2- 172.21.15.150:0/773542524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc84c0c3660 0x7fc84c156c50 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fc84c06a2c0 tx=0x7fc84401f580 comp rx=0 tx=0).stop 2024-09-17T05:16:49.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.767+0000 7fc8508fc640 1 -- 172.21.15.150:0/773542524 shutdown_connections 2024-09-17T05:16:49.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.767+0000 7fc8508fc640 1 --2- 172.21.15.150:0/773542524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc84c0c3660 0x7fc84c156c50 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.767+0000 7fc8508fc640 1 --2- 172.21.15.150:0/773542524 >> v2:172.21.15.176:3300/0 conn(0x7fc84c071740 0x7fc84c0c3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.767+0000 7fc8508fc640 1 --2- 172.21.15.150:0/773542524 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc84c150d40 0x7fc84c0a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.767+0000 7fc8508fc640 1 -- 172.21.15.150:0/773542524 >> 172.21.15.150:0/773542524 conn(0x7fc84c070c20 msgr2=0x7fc84c071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:49.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.767+0000 7fc8508fc640 1 -- 172.21.15.150:0/773542524 shutdown_connections 2024-09-17T05:16:49.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.767+0000 7fc8508fc640 1 -- 172.21.15.150:0/773542524 wait complete. 2024-09-17T05:16:49.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc8508fc640 1 Processor -- start 2024-09-17T05:16:49.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc8508fc640 1 -- start start 2024-09-17T05:16:49.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc8508fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc84c071740 0x7fc84c0c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc8508fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc84c0c3660 0x7fc84c0acab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc8508fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc84c150d40 0x7fc84c0c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc8508fc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c06e170 con 0x7fc84c071740 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc8508fc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c073390 con 0x7fc84c0c3660 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc8508fc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc84c0ae0d0 con 0x7fc84c150d40 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc84a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc84c071740 0x7fc84c0c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc84a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc84c071740 0x7fc84c0c7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47936/0 (socket says 172.21.15.150:47936) 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc84a59c640 1 -- 172.21.15.150:0/3890731968 learned_addr learned my addr 172.21.15.150:0/3890731968 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc84ad9d640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc84c150d40 0x7fc84c0c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc843fff640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc84c0c3660 0x7fc84c0acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc84a59c640 1 -- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc84c150d40 msgr2=0x7fc84c0c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc84a59c640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc84c150d40 0x7fc84c0c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc84a59c640 1 -- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc84c0c3660 msgr2=0x7fc84c0acab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:49.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc84a59c640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc84c0c3660 0x7fc84c0acab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc84a59c640 1 -- 172.21.15.150:0/3890731968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c14f010 con 0x7fc84c071740 2024-09-17T05:16:49.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc84a59c640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc84c071740 0x7fc84c0c7e90 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00c9f0 tx=0x7fc83c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:49.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc842ffd640 1 -- 172.21.15.150:0/3890731968 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c007b50 con 0x7fc84c071740 2024-09-17T05:16:49.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc8508fc640 1 -- 172.21.15.150:0/3890731968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc84c0c72e0 con 0x7fc84c071740 2024-09-17T05:16:49.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc842ffd640 1 -- 172.21.15.150:0/3890731968 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc83c007de0 con 0x7fc84c071740 2024-09-17T05:16:49.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc842ffd640 1 -- 172.21.15.150:0/3890731968 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83c00f6b0 con 0x7fc84c071740 2024-09-17T05:16:49.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc8508fc640 1 -- 172.21.15.150:0/3890731968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc84c0ac0f0 con 0x7fc84c071740 2024-09-17T05:16:49.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.771+0000 7fc8508fc640 1 -- 172.21.15.150:0/3890731968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc84c0c3a70 con 0x7fc84c071740 2024-09-17T05:16:49.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.775+0000 7fc842ffd640 1 -- 172.21.15.150:0/3890731968 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc83c00f850 con 0x7fc84c071740 2024-09-17T05:16:49.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.775+0000 7fc842ffd640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc82c073200 0x7fc82c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:49.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.775+0000 7fc842ffd640 1 -- 172.21.15.150:0/3890731968 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7fc83c095c60 con 0x7fc84c071740 2024-09-17T05:16:49.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.775+0000 7fc843fff640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc82c073200 0x7fc82c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:49.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.775+0000 7fc843fff640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc82c073200 0x7fc82c075710 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0c4270 tx=0x7fc834005ca0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:49.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.779+0000 7fc842ffd640 1 -- 172.21.15.150:0/3890731968 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc83c014030 con 0x7fc84c071740 2024-09-17T05:16:49.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.987+0000 7fc8508fc640 1 -- 172.21.15.150:0/3890731968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc84c0c7c70 con 0x7fc84c071740 2024-09-17T05:16:49.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.987+0000 7fc842ffd640 1 -- 172.21.15.150:0/3890731968 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+6547 (secure 0 0 0) 0x7fc83c05fdd0 con 0x7fc84c071740 2024-09-17T05:16:49.989 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:49.989 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":189,"btime":"2024-09-17T05:16:07:302153+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184},{"gid":7920,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:15:54.841091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7854},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7854":{"gid":7854,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":12,"addr":"172.21.15.150:6833/949083177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":949083177},{"type":"v1","addr":"172.21.15.150:6833","nonce":949083177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7854,"qdb_cluster":[7854]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:49.989 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 189 2024-09-17T05:16:49.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.991+0000 7fc8508fc640 1 -- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc82c073200 msgr2=0x7fc82c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:49.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.991+0000 7fc8508fc640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc82c073200 0x7fc82c075710 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0c4270 tx=0x7fc834005ca0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.991+0000 7fc8508fc640 1 -- 172.21.15.150:0/3890731968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc84c071740 msgr2=0x7fc84c0c7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:49.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.991+0000 7fc8508fc640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc84c071740 0x7fc84c0c7e90 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00c9f0 tx=0x7fc83c00ceb0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.991+0000 7fc8508fc640 1 -- 172.21.15.150:0/3890731968 shutdown_connections 2024-09-17T05:16:49.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.991+0000 7fc8508fc640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc82c073200 0x7fc82c075710 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.991+0000 7fc8508fc640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc84c0c3660 0x7fc84c0acab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.991+0000 7fc8508fc640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc84c150d40 0x7fc84c0c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.991+0000 7fc8508fc640 1 --2- 172.21.15.150:0/3890731968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc84c071740 0x7fc84c0c7e90 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.991+0000 7fc8508fc640 1 -- 172.21.15.150:0/3890731968 >> 172.21.15.150:0/3890731968 conn(0x7fc84c070c20 msgr2=0x7fc84c147470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.991+0000 7fc8508fc640 1 -- 172.21.15.150:0/3890731968 shutdown_connections 2024-09-17T05:16:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:49.991+0000 7fc8508fc640 1 -- 172.21.15.150:0/3890731968 wait complete. 2024-09-17T05:16:50.012 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:16:51.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.775+0000 7f19a1519640 1 -- 172.21.15.150:0/775494989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c155690 msgr2=0x7f199c0c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.775+0000 7f19a1519640 1 --2- 172.21.15.150:0/775494989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c155690 0x7f199c0c6fc0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f199c073110 tx=0x7f199401ccf0 comp rx=0 tx=0).stop 2024-09-17T05:16:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.775+0000 7f19a1519640 1 -- 172.21.15.150:0/775494989 shutdown_connections 2024-09-17T05:16:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.775+0000 7f19a1519640 1 --2- 172.21.15.150:0/775494989 >> v2:172.21.15.176:3300/0 conn(0x7f199c0c87e0 0x7f199c155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.775+0000 7f19a1519640 1 --2- 172.21.15.150:0/775494989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c155690 0x7f199c0c6fc0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.775+0000 7f19a1519640 1 -- 172.21.15.150:0/775494989 >> 172.21.15.150:0/775494989 conn(0x7f199c06e360 msgr2=0x7f199c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.775+0000 7f19a1519640 1 -- 172.21.15.150:0/775494989 shutdown_connections 2024-09-17T05:16:51.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.775+0000 7f19a1519640 1 -- 172.21.15.150:0/775494989 wait complete. 2024-09-17T05:16:51.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.779+0000 7f19a1519640 1 Processor -- start 2024-09-17T05:16:51.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.779+0000 7f19a1519640 1 -- start start 2024-09-17T05:16:51.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f19a1519640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f199c0c87e0 0x7f199c157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f19a1519640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c155690 0x7f199c1ed580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f19a1519640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f199c153010 0x7f199c1486e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f19a1519640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f199c161be0 con 0x7f199c155690 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f19a1519640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f199c161a60 con 0x7f199c0c87e0 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f19a1519640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f199c161d60 con 0x7f199c153010 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f199a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c155690 0x7f199c1ed580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f199a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c155690 0x7f199c1ed580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47942/0 (socket says 172.21.15.150:47942) 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f199a7fc640 1 -- 172.21.15.150:0/1264520386 learned_addr learned my addr 172.21.15.150:0/1264520386 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f199affd640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f199c0c87e0 0x7f199c157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f199b7fe640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f199c153010 0x7f199c1486e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f199a7fc640 1 -- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f199c153010 msgr2=0x7f199c1486e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f199a7fc640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f199c153010 0x7f199c1486e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:51.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f199a7fc640 1 -- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f199c0c87e0 msgr2=0x7f199c157190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:51.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f199a7fc640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f199c0c87e0 0x7f199c157190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:51.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f199a7fc640 1 -- 172.21.15.150:0/1264520386 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199c1567e0 con 0x7f199c155690 2024-09-17T05:16:51.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f199a7fc640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c155690 0x7f199c1ed580 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f199000e7d0 tx=0x7f199000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:51.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f19837fe640 1 -- 172.21.15.150:0/1264520386 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199000cc60 con 0x7f199c155690 2024-09-17T05:16:51.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f19837fe640 1 -- 172.21.15.150:0/1264520386 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1990010430 con 0x7f199c155690 2024-09-17T05:16:51.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f19837fe640 1 -- 172.21.15.150:0/1264520386 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1990010860 con 0x7f199c155690 2024-09-17T05:16:51.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f19a1519640 1 -- 172.21.15.150:0/1264520386 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199c154a90 con 0x7f199c155690 2024-09-17T05:16:51.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f19a1519640 1 -- 172.21.15.150:0/1264520386 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199c15e830 con 0x7f199c155690 2024-09-17T05:16:51.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.783+0000 7f19a1519640 1 -- 172.21.15.150:0/1264520386 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f199c0c3740 con 0x7f199c155690 2024-09-17T05:16:51.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.787+0000 7f19837fe640 1 -- 172.21.15.150:0/1264520386 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1990010a00 con 0x7f199c155690 2024-09-17T05:16:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.787+0000 7f19837fe640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1988073200 0x7f1988075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.787+0000 7f19837fe640 1 -- 172.21.15.150:0/1264520386 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7f1990095be0 con 0x7f199c155690 2024-09-17T05:16:51.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.787+0000 7f199affd640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1988073200 0x7f1988075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:51.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.787+0000 7f199affd640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1988073200 0x7f1988075710 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f198c001650 tx=0x7f198c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:51.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.795+0000 7f19837fe640 1 -- 172.21.15.150:0/1264520386 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f199005fc70 con 0x7f199c155690 2024-09-17T05:16:51.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.999+0000 7f19a1519640 1 -- 172.21.15.150:0/1264520386 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f199c069500 con 0x7f199c155690 2024-09-17T05:16:51.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.999+0000 7f19837fe640 1 -- 172.21.15.150:0/1264520386 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+5713 (secure 0 0 0) 0x7f199005f570 con 0x7f199c155690 2024-09-17T05:16:51.999 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:52.000 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":190,"btime":"2024-09-17T05:16:51:271329+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:51.271318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:52.000 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 190 2024-09-17T05:16:52.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.999+0000 7f19a1519640 1 -- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1988073200 msgr2=0x7f1988075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:52.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:51.999+0000 7f19a1519640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1988073200 0x7f1988075710 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f198c001650 tx=0x7f198c009040 comp rx=0 tx=0).stop 2024-09-17T05:16:52.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.003+0000 7f19a1519640 1 -- 172.21.15.150:0/1264520386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c155690 msgr2=0x7f199c1ed580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:52.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.003+0000 7f19a1519640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c155690 0x7f199c1ed580 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f199000e7d0 tx=0x7f199000ec90 comp rx=0 tx=0).stop 2024-09-17T05:16:52.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.003+0000 7f19a1519640 1 -- 172.21.15.150:0/1264520386 shutdown_connections 2024-09-17T05:16:52.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.003+0000 7f19a1519640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1988073200 0x7f1988075710 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.003+0000 7f19a1519640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f199c153010 0x7f199c1486e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.003+0000 7f19a1519640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f199c155690 0x7f199c1ed580 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.003+0000 7f19a1519640 1 --2- 172.21.15.150:0/1264520386 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f199c0c87e0 0x7f199c157190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.003+0000 7f19a1519640 1 -- 172.21.15.150:0/1264520386 >> 172.21.15.150:0/1264520386 conn(0x7f199c06e360 msgr2=0x7f199c0c5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:52.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.003+0000 7f19a1519640 1 -- 172.21.15.150:0/1264520386 shutdown_connections 2024-09-17T05:16:52.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.003+0000 7f19a1519640 1 -- 172.21.15.150:0/1264520386 wait complete. 2024-09-17T05:16:52.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:52.022 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-09-17T05:16:52.022 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:16:52.022 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:52.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 -- 172.21.15.150:0/1944160396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f923c0053b0 msgr2=0x7f923c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:52.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 --2- 172.21.15.150:0/1944160396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f923c0053b0 0x7f923c00ba60 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f92500c9b70 tx=0x7f924801cad0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 -- 172.21.15.150:0/1944160396 shutdown_connections 2024-09-17T05:16:52.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 --2- 172.21.15.150:0/1944160396 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f923c0053b0 0x7f923c00ba60 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 -- 172.21.15.150:0/1944160396 >> 172.21.15.150:0/1944160396 conn(0x7f92500ca460 msgr2=0x7f92500c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:52.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 -- 172.21.15.150:0/1944160396 shutdown_connections 2024-09-17T05:16:52.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 -- 172.21.15.150:0/1944160396 wait complete. 2024-09-17T05:16:52.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 Processor -- start 2024-09-17T05:16:52.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 -- start start 2024-09-17T05:16:52.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f923c0053b0 0x7f9250151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:52.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f925014b770 0x7f92501f1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:52.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925013e900 0x7f925013ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:52.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f92500722a0 con 0x7f925013e900 2024-09-17T05:16:52.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9250072120 con 0x7f925014b770 2024-09-17T05:16:52.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9255056640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f925014b0b0 con 0x7f923c0053b0 2024-09-17T05:16:52.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f924f59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925013e900 0x7f925013ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:52.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f924f59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925013e900 0x7f925013ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47958/0 (socket says 172.21.15.150:47958) 2024-09-17T05:16:52.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f924f59e640 1 -- 172.21.15.150:0/3474532968 learned_addr learned my addr 172.21.15.150:0/3474532968 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:52.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9247fff640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f925014b770 0x7f92501f1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:52.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f924ed9d640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f923c0053b0 0x7f9250151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:52.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f924f59e640 1 -- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f923c0053b0 msgr2=0x7f9250151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f924f59e640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f923c0053b0 0x7f9250151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f924f59e640 1 -- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f925014b770 msgr2=0x7f92501f1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f924f59e640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f925014b770 0x7f92501f1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f924f59e640 1 -- 172.21.15.150:0/3474532968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9250160480 con 0x7f925013e900 2024-09-17T05:16:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.159+0000 7f9247fff640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f925014b770 0x7f92501f1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:16:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.163+0000 7f924f59e640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925013e900 0x7f925013ed10 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f9248098440 tx=0x7f9248006d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.163+0000 7f92477fe640 1 -- 172.21.15.150:0/3474532968 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480a5720 con 0x7f925013e900 2024-09-17T05:16:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.163+0000 7f92477fe640 1 -- 172.21.15.150:0/3474532968 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92480b4020 con 0x7f925013e900 2024-09-17T05:16:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.163+0000 7f9255056640 1 -- 172.21.15.150:0/3474532968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f925013f660 con 0x7f925013e900 2024-09-17T05:16:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.163+0000 7f92477fe640 1 -- 172.21.15.150:0/3474532968 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248003cf0 con 0x7f925013e900 2024-09-17T05:16:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.163+0000 7f9255056640 1 -- 172.21.15.150:0/3474532968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f925014bc20 con 0x7f925013e900 2024-09-17T05:16:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.163+0000 7f9255056640 1 -- 172.21.15.150:0/3474532968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f925013f880 con 0x7f925013e900 2024-09-17T05:16:52.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.163+0000 7f92477fe640 1 -- 172.21.15.150:0/3474532968 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9248006170 con 0x7f925013e900 2024-09-17T05:16:52.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.163+0000 7f92477fe640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9230073200 0x7f9230075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:52.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.163+0000 7f92477fe640 1 -- 172.21.15.150:0/3474532968 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7f924812f5c0 con 0x7f925013e900 2024-09-17T05:16:52.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.163+0000 7f924ed9d640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9230073200 0x7f9230075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:52.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.167+0000 7f924ed9d640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9230073200 0x7f9230075710 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f9240009a10 tx=0x7f92400091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.167+0000 7f92477fe640 1 -- 172.21.15.150:0/3474532968 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f92480be030 con 0x7f925013e900 2024-09-17T05:16:52.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.211+0000 7f3931239640 1 -- 172.21.15.150:0/3331996256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f392c0c3900 msgr2=0x7f392c074920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:52.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.211+0000 7f3931239640 1 --2- 172.21.15.150:0/3331996256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f392c0c3900 0x7f392c074920 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f391c009870 tx=0x7f391c01c820 comp rx=0 tx=0).stop 2024-09-17T05:16:52.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 -- 172.21.15.150:0/3331996256 shutdown_connections 2024-09-17T05:16:52.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 --2- 172.21.15.150:0/3331996256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f392c0c3900 0x7f392c074920 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 -- 172.21.15.150:0/3331996256 >> 172.21.15.150:0/3331996256 conn(0x7f392c06e360 msgr2=0x7f392c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:52.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 -- 172.21.15.150:0/3331996256 shutdown_connections 2024-09-17T05:16:52.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 -- 172.21.15.150:0/3331996256 wait complete. 2024-09-17T05:16:52.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 Processor -- start 2024-09-17T05:16:52.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 -- start start 2024-09-17T05:16:52.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f392c074e60 0x7f392c14d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:52.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f392c0c3900 0x7f392c1efca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:52.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f392c0c4250 0x7f392c1554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:52.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f392c161eb0 con 0x7f392c0c4250 2024-09-17T05:16:52.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f392c072fc0 con 0x7f392c0c3900 2024-09-17T05:16:52.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f392c162030 con 0x7f392c074e60 2024-09-17T05:16:52.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f392b59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f392c0c4250 0x7f392c1554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:52.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f392b59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f392c0c4250 0x7f392c1554c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47982/0 (socket says 172.21.15.150:47982) 2024-09-17T05:16:52.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f392b59e640 1 -- 172.21.15.150:0/501318424 learned_addr learned my addr 172.21.15.150:0/501318424 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:52.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f392ad9d640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f392c074e60 0x7f392c14d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:52.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f392a59c640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f392c0c3900 0x7f392c1efca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:52.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f392b59e640 1 -- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f392c074e60 msgr2=0x7f392c14d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:52.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f392b59e640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f392c074e60 0x7f392c14d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f392b59e640 1 -- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f392c0c3900 msgr2=0x7f392c1efca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:52.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f392b59e640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f392c0c3900 0x7f392c1efca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f392b59e640 1 -- 172.21.15.150:0/501318424 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392c157760 con 0x7f392c0c4250 2024-09-17T05:16:52.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f392b59e640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f392c0c4250 0x7f392c1554c0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f392400c790 tx=0x7f392400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:52.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f39137fe640 1 -- 172.21.15.150:0/501318424 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3924003b80 con 0x7f392c0c4250 2024-09-17T05:16:52.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f39137fe640 1 -- 172.21.15.150:0/501318424 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3924003d20 con 0x7f392c0c4250 2024-09-17T05:16:52.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f3931239640 1 -- 172.21.15.150:0/501318424 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f392c154e10 con 0x7f392c0c4250 2024-09-17T05:16:52.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.215+0000 7f39137fe640 1 -- 172.21.15.150:0/501318424 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392400aab0 con 0x7f392c0c4250 2024-09-17T05:16:52.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.219+0000 7f3931239640 1 -- 172.21.15.150:0/501318424 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f392c155f20 con 0x7f392c0c4250 2024-09-17T05:16:52.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.219+0000 7f3931239640 1 -- 172.21.15.150:0/501318424 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f392c0a88e0 con 0x7f392c0c4250 2024-09-17T05:16:52.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.219+0000 7f39137fe640 1 -- 172.21.15.150:0/501318424 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f392400ac50 con 0x7f392c0c4250 2024-09-17T05:16:52.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.219+0000 7f39137fe640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3918073230 0x7f3918075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:52.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.219+0000 7f39137fe640 1 -- 172.21.15.150:0/501318424 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7f3924094ee0 con 0x7f392c0c4250 2024-09-17T05:16:52.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.219+0000 7f392ad9d640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3918073230 0x7f3918075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:52.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.219+0000 7f392ad9d640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3918073230 0x7f3918075740 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3920001640 tx=0x7f3920009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:52.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.223+0000 7f39137fe640 1 -- 172.21.15.150:0/501318424 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f392405eff0 con 0x7f392c0c4250 2024-09-17T05:16:52.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.367+0000 7f9255056640 1 -- 172.21.15.150:0/3474532968 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92500c7d60 con 0x7f925013e900 2024-09-17T05:16:52.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.367+0000 7f92477fe640 1 -- 172.21.15.150:0/3474532968 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+5713 (secure 0 0 0) 0x7f9248016830 con 0x7f925013e900 2024-09-17T05:16:52.368 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:52.368 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":190,"btime":"2024-09-17T05:16:51:271329+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:51.271318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:52.368 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 190 2024-09-17T05:16:52.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.371+0000 7f9255056640 1 -- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9230073200 msgr2=0x7f9230075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:52.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.371+0000 7f9255056640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9230073200 0x7f9230075710 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f9240009a10 tx=0x7f92400091e0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.371+0000 7f9255056640 1 -- 172.21.15.150:0/3474532968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925013e900 msgr2=0x7f925013ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:52.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.371+0000 7f9255056640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925013e900 0x7f925013ed10 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f9248098440 tx=0x7f9248006d40 comp rx=0 tx=0).stop 2024-09-17T05:16:52.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.371+0000 7f9255056640 1 -- 172.21.15.150:0/3474532968 shutdown_connections 2024-09-17T05:16:52.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.371+0000 7f9255056640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9230073200 0x7f9230075710 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.371+0000 7f9255056640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f925014b770 0x7f92501f1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.371+0000 7f9255056640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925013e900 0x7f925013ed10 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.371+0000 7f9255056640 1 --2- 172.21.15.150:0/3474532968 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f923c0053b0 0x7f9250151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.371+0000 7f9255056640 1 -- 172.21.15.150:0/3474532968 >> 172.21.15.150:0/3474532968 conn(0x7f92500ca460 msgr2=0x7f92500c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:52.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.371+0000 7f9255056640 1 -- 172.21.15.150:0/3474532968 shutdown_connections 2024-09-17T05:16:52.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.371+0000 7f9255056640 1 -- 172.21.15.150:0/3474532968 wait complete. 2024-09-17T05:16:52.391 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:16:52.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.431+0000 7f3931239640 1 -- 172.21.15.150:0/501318424 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f392c074920 con 0x7f392c0c4250 2024-09-17T05:16:52.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.431+0000 7f39137fe640 1 -- 172.21.15.150:0/501318424 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+5713 (secure 0 0 0) 0x7f392405e8f0 con 0x7f392c0c4250 2024-09-17T05:16:52.432 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:52.432 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":190,"btime":"2024-09-17T05:16:51:271329+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6563,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:51.271318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:15:11.291889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4355":{"gid":4355,"name":"b","rank":0,"incarnation":180,"state":"up:active","state_seq":389,"addr":"172.21.15.176:6835/524846644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":524846644},{"type":"v1","addr":"172.21.15.176:6835","nonce":524846644}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4355,"qdb_cluster":[4355]},"id":2}]} 2024-09-17T05:16:52.433 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 190 2024-09-17T05:16:52.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.435+0000 7f3931239640 1 -- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3918073230 msgr2=0x7f3918075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:52.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.435+0000 7f3931239640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3918073230 0x7f3918075740 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3920001640 tx=0x7f3920009040 comp rx=0 tx=0).stop 2024-09-17T05:16:52.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.435+0000 7f3931239640 1 -- 172.21.15.150:0/501318424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f392c0c4250 msgr2=0x7f392c1554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:52.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.435+0000 7f3931239640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f392c0c4250 0x7f392c1554c0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f392400c790 tx=0x7f392400cc50 comp rx=0 tx=0).stop 2024-09-17T05:16:52.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.435+0000 7f3931239640 1 -- 172.21.15.150:0/501318424 shutdown_connections 2024-09-17T05:16:52.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.435+0000 7f3931239640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3918073230 0x7f3918075740 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.435+0000 7f3931239640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f392c0c3900 0x7f392c1efca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.435+0000 7f3931239640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f392c0c4250 0x7f392c1554c0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.435+0000 7f3931239640 1 --2- 172.21.15.150:0/501318424 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f392c074e60 0x7f392c14d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:52.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.435+0000 7f3931239640 1 -- 172.21.15.150:0/501318424 >> 172.21.15.150:0/501318424 conn(0x7f392c06e360 msgr2=0x7f392c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:52.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.435+0000 7f3931239640 1 -- 172.21.15.150:0/501318424 shutdown_connections 2024-09-17T05:16:52.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:52.435+0000 7f3931239640 1 -- 172.21.15.150:0/501318424 wait complete. 2024-09-17T05:16:52.456 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:16:51:271329+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 190, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 190, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7920": { "addr": "172.21.15.150:6835/3297154462", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 3297154462, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 3297154462, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7920, "incarnation": 190, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:16:51.271318+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7920 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 183, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4355": { "addr": "172.21.15.176:6835/524846644", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 524846644, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 524846644, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 4355, "incarnation": 180, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 389 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:15:11.291889+0000", "qdb_cluster": [ 4355 ], "qdb_leader": 4355, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4355 } } } ], "standbys": [ { "addr": "172.21.15.176:6837/1903363960", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 1903363960, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 1903363960, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 184, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6563, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:16:54.392 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:54.456 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:54.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.559+0000 7f140790b640 1 -- 172.21.15.150:0/4258609030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13ec003a10 msgr2=0x7f13ec003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:54.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.559+0000 7f140790b640 1 --2- 172.21.15.150:0/4258609030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13ec003a10 0x7f13ec003e80 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f13f800c640 tx=0x7f13f801d300 comp rx=0 tx=0).stop 2024-09-17T05:16:54.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.559+0000 7f1405ea8640 1 -- 172.21.15.150:0/4258609030 reap_dead start 2024-09-17T05:16:54.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.559+0000 7f140790b640 1 -- 172.21.15.150:0/4258609030 shutdown_connections 2024-09-17T05:16:54.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.559+0000 7f140790b640 1 -- 172.21.15.150:0/4258609030 >> 172.21.15.150:0/4258609030 conn(0x7f14000aab60 msgr2=0x7f14000aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:54.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.559+0000 7f140790b640 1 -- 172.21.15.150:0/4258609030 shutdown_connections 2024-09-17T05:16:54.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.559+0000 7f140790b640 1 -- 172.21.15.150:0/4258609030 wait complete. 2024-09-17T05:16:54.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f140790b640 1 Processor -- start 2024-09-17T05:16:54.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f140790b640 1 -- start start 2024-09-17T05:16:54.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f140790b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13ec003a10 0x7f14001f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:54.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f140790b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1400150920 0x7f14000add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:54.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f140790b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1400152600 0x7f1400152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:54.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f140790b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f14000c79a0 con 0x7f1400150920 2024-09-17T05:16:54.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f140790b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f140014a500 con 0x7f13ec003a10 2024-09-17T05:16:54.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f140790b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f14000c7b20 con 0x7f1400152600 2024-09-17T05:16:54.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f1404ea6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1400150920 0x7f14000add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:54.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f1404ea6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1400150920 0x7f14000add90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48006/0 (socket says 172.21.15.150:48006) 2024-09-17T05:16:54.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f1404ea6640 1 -- 172.21.15.150:0/4185471306 learned_addr learned my addr 172.21.15.150:0/4185471306 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:54.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f14056a7640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13ec003a10 0x7f14001f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:54.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f1405ea8640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1400152600 0x7f1400152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:54.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f1404ea6640 1 -- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1400152600 msgr2=0x7f1400152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:54.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f1404ea6640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1400152600 0x7f1400152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f1404ea6640 1 -- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13ec003a10 msgr2=0x7f14001f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:54.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f1404ea6640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13ec003a10 0x7f14001f28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f1404ea6640 1 -- 172.21.15.150:0/4185471306 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1400150500 con 0x7f1400150920 2024-09-17T05:16:54.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f14056a7640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13ec003a10 0x7f14001f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:16:54.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f1404ea6640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1400150920 0x7f14000add90 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f13f0011a50 tx=0x7f13f0011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:54.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f13f5ffb640 1 -- 172.21.15.150:0/4185471306 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f001f070 con 0x7f1400150920 2024-09-17T05:16:54.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f13f5ffb640 1 -- 172.21.15.150:0/4185471306 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13f000fe80 con 0x7f1400150920 2024-09-17T05:16:54.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f140790b640 1 -- 172.21.15.150:0/4185471306 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14001f2e00 con 0x7f1400150920 2024-09-17T05:16:54.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f13f5ffb640 1 -- 172.21.15.150:0/4185471306 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f0012690 con 0x7f1400150920 2024-09-17T05:16:54.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f140790b640 1 -- 172.21.15.150:0/4185471306 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14001490c0 con 0x7f1400150920 2024-09-17T05:16:54.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.563+0000 7f140790b640 1 -- 172.21.15.150:0/4185471306 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14000aee20 con 0x7f1400150920 2024-09-17T05:16:54.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.567+0000 7f13f5ffb640 1 -- 172.21.15.150:0/4185471306 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f13f0022070 con 0x7f1400150920 2024-09-17T05:16:54.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.567+0000 7f13f5ffb640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f13e4073230 0x7f13e4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:54.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.567+0000 7f13f5ffb640 1 -- 172.21.15.150:0/4185471306 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f13f0095ce0 con 0x7f1400150920 2024-09-17T05:16:54.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.567+0000 7f14056a7640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f13e4073230 0x7f13e4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:54.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.567+0000 7f14056a7640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f13e4073230 0x7f13e4075740 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f13f800c640 tx=0x7f13f801a140 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:54.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.571+0000 7f13f5ffb640 1 -- 172.21.15.150:0/4185471306 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f13f005f6c0 con 0x7f1400150920 2024-09-17T05:16:54.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.607+0000 7f82ca282640 1 -- 172.21.15.150:0/1528549632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82c40699b0 msgr2=0x7f82c40719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:54.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.607+0000 7f82ca282640 1 --2- 172.21.15.150:0/1528549632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82c40699b0 0x7f82c40719e0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f82c40c2ae0 tx=0x7f82b801c6e0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.607+0000 7f82ca282640 1 -- 172.21.15.150:0/1528549632 shutdown_connections 2024-09-17T05:16:54.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.607+0000 7f82ca282640 1 --2- 172.21.15.150:0/1528549632 >> v2:172.21.15.176:3300/0 conn(0x7f82c4151d10 0x7f82c40adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.607+0000 7f82ca282640 1 --2- 172.21.15.150:0/1528549632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82c40699b0 0x7f82c40719e0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.607+0000 7f82ca282640 1 -- 172.21.15.150:0/1528549632 >> 172.21.15.150:0/1528549632 conn(0x7f82c4070b50 msgr2=0x7f82c4070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:54.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.607+0000 7f82ca282640 1 -- 172.21.15.150:0/1528549632 shutdown_connections 2024-09-17T05:16:54.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.607+0000 7f82ca282640 1 -- 172.21.15.150:0/1528549632 wait complete. 2024-09-17T05:16:54.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82ca282640 1 Processor -- start 2024-09-17T05:16:54.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82ca282640 1 -- start start 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82ca282640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82c40699b0 0x7f82c40c1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82ca282640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82c406e590 0x7f82c40c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82ca282640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82c4151d10 0x7f82c413f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82ca282640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f82c4161990 con 0x7f82c40699b0 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82ca282640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f82c40a9fb0 con 0x7f82c4151d10 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82ca282640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f82c4161b10 con 0x7f82c406e590 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82c40699b0 0x7f82c40c1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82c40699b0 0x7f82c40c1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48028/0 (socket says 172.21.15.150:48028) 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c3fff640 1 -- 172.21.15.150:0/51951302 learned_addr learned my addr 172.21.15.150:0/51951302 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c37fe640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82c406e590 0x7f82c40c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c881f640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82c4151d10 0x7f82c413f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c3fff640 1 -- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82c406e590 msgr2=0x7f82c40c64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:54.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c3fff640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82c406e590 0x7f82c40c64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c3fff640 1 -- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82c4151d10 msgr2=0x7f82c413f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:54.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c3fff640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82c4151d10 0x7f82c413f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c3fff640 1 -- 172.21.15.150:0/51951302 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c40c4b40 con 0x7f82c40699b0 2024-09-17T05:16:54.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c3fff640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82c40699b0 0x7f82c40c1cb0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f82b400c3f0 tx=0x7f82b400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:54.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c0ff9640 1 -- 172.21.15.150:0/51951302 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b4011790 con 0x7f82c40699b0 2024-09-17T05:16:54.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c0ff9640 1 -- 172.21.15.150:0/51951302 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82b4017070 con 0x7f82c40699b0 2024-09-17T05:16:54.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82ca282640 1 -- 172.21.15.150:0/51951302 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82c41f3a60 con 0x7f82c40699b0 2024-09-17T05:16:54.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.611+0000 7f82c0ff9640 1 -- 172.21.15.150:0/51951302 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b4011cc0 con 0x7f82c40699b0 2024-09-17T05:16:54.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.615+0000 7f82ca282640 1 -- 172.21.15.150:0/51951302 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82c40c7b80 con 0x7f82c40699b0 2024-09-17T05:16:54.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.615+0000 7f82ca282640 1 -- 172.21.15.150:0/51951302 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82c40719e0 con 0x7f82c40699b0 2024-09-17T05:16:54.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.615+0000 7f82c0ff9640 1 -- 172.21.15.150:0/51951302 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f82b401c460 con 0x7f82c40699b0 2024-09-17T05:16:54.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.615+0000 7f82c0ff9640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82b0073200 0x7f82b0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:54.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.615+0000 7f82c0ff9640 1 -- 172.21.15.150:0/51951302 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f82b4095b20 con 0x7f82c40699b0 2024-09-17T05:16:54.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.615+0000 7f82c37fe640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82b0073200 0x7f82b0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:54.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.615+0000 7f82c37fe640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82b0073200 0x7f82b0075710 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f82b80af000 tx=0x7f82b801af20 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:54.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.619+0000 7f82c0ff9640 1 -- 172.21.15.150:0/51951302 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f82b4095e20 con 0x7f82c40699b0 2024-09-17T05:16:54.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.843+0000 7f140790b640 1 -- 172.21.15.150:0/4185471306 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14000c8330 con 0x7f1400150920 2024-09-17T05:16:54.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.847+0000 7f13f5ffb640 1 -- 172.21.15.150:0/4185471306 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) ==== 95+0+4891 (secure 0 0 0) 0x7f13f00640f0 con 0x7f1400150920 2024-09-17T05:16:54.846 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:54.846 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":194,"btime":"2024-09-17T05:16:54:689084+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:54.294991+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:16:54.847 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 194 2024-09-17T05:16:54.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.851+0000 7f140790b640 1 -- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f13e4073230 msgr2=0x7f13e4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:54.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.851+0000 7f140790b640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f13e4073230 0x7f13e4075740 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f13f800c640 tx=0x7f13f801a140 comp rx=0 tx=0).stop 2024-09-17T05:16:54.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.851+0000 7f140790b640 1 -- 172.21.15.150:0/4185471306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1400150920 msgr2=0x7f14000add90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:54.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.851+0000 7f140790b640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1400150920 0x7f14000add90 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f13f0011a50 tx=0x7f13f0011f10 comp rx=0 tx=0).stop 2024-09-17T05:16:54.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.851+0000 7f140790b640 1 -- 172.21.15.150:0/4185471306 shutdown_connections 2024-09-17T05:16:54.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.851+0000 7f140790b640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f13e4073230 0x7f13e4075740 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.851+0000 7f140790b640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1400152600 0x7f1400152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.851+0000 7f140790b640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1400150920 0x7f14000add90 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.851+0000 7f140790b640 1 --2- 172.21.15.150:0/4185471306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13ec003a10 0x7f14001f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.851+0000 7f140790b640 1 -- 172.21.15.150:0/4185471306 >> 172.21.15.150:0/4185471306 conn(0x7f14000aab60 msgr2=0x7f1400156c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:54.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.851+0000 7f140790b640 1 -- 172.21.15.150:0/4185471306 shutdown_connections 2024-09-17T05:16:54.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.851+0000 7f140790b640 1 -- 172.21.15.150:0/4185471306 wait complete. 2024-09-17T05:16:54.876 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-09-17T05:16:54.876 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:16:54.877 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:54.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.927+0000 7f82ca282640 1 -- 172.21.15.150:0/51951302 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82c41536a0 con 0x7f82c40699b0 2024-09-17T05:16:54.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.927+0000 7f82c0ff9640 1 -- 172.21.15.150:0/51951302 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) ==== 95+0+4891 (secure 0 0 0) 0x7f82b405fbe0 con 0x7f82c40699b0 2024-09-17T05:16:54.926 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:54.926 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":194,"btime":"2024-09-17T05:16:54:689084+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:54.294991+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:16:54.927 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 194 2024-09-17T05:16:54.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.927+0000 7f82ca282640 1 -- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82b0073200 msgr2=0x7f82b0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:54.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.927+0000 7f82ca282640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82b0073200 0x7f82b0075710 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f82b80af000 tx=0x7f82b801af20 comp rx=0 tx=0).stop 2024-09-17T05:16:54.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.927+0000 7f82ca282640 1 -- 172.21.15.150:0/51951302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82c40699b0 msgr2=0x7f82c40c1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:54.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.927+0000 7f82ca282640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82c40699b0 0x7f82c40c1cb0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f82b400c3f0 tx=0x7f82b400c8b0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.931+0000 7f82ca282640 1 -- 172.21.15.150:0/51951302 shutdown_connections 2024-09-17T05:16:54.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.931+0000 7f82ca282640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f82b0073200 0x7f82b0075710 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.931+0000 7f82ca282640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82c4151d10 0x7f82c413f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.931+0000 7f82ca282640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f82c406e590 0x7f82c40c64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.931+0000 7f82ca282640 1 --2- 172.21.15.150:0/51951302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82c40699b0 0x7f82c40c1cb0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:54.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.931+0000 7f82ca282640 1 -- 172.21.15.150:0/51951302 >> 172.21.15.150:0/51951302 conn(0x7f82c4070b50 msgr2=0x7f82c40a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:54.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.931+0000 7f82ca282640 1 -- 172.21.15.150:0/51951302 shutdown_connections 2024-09-17T05:16:54.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:54.931+0000 7f82ca282640 1 -- 172.21.15.150:0/51951302 wait complete. 2024-09-17T05:16:54.963 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=7854 2024-09-17T05:16:54.963 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before reviving mds.a 2024-09-17T05:16:55.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.151+0000 7fe3d1287640 1 -- 172.21.15.150:0/945996334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3cc06d390 msgr2=0x7fe3cc0c8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:55.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.151+0000 7fe3c9ffb640 1 -- 172.21.15.150:0/945996334 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc0b2070 con 0x7fe3cc06d390 2024-09-17T05:16:55.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.151+0000 7fe3d1287640 1 --2- 172.21.15.150:0/945996334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3cc06d390 0x7fe3cc0c8110 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc0c3140 tx=0x7fe3bc01c8c0 comp rx=0 tx=0).stop 2024-09-17T05:16:55.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.151+0000 7fe3d1287640 1 -- 172.21.15.150:0/945996334 shutdown_connections 2024-09-17T05:16:55.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.151+0000 7fe3d1287640 1 --2- 172.21.15.150:0/945996334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3cc06d390 0x7fe3cc0c8110 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:55.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.151+0000 7fe3d1287640 1 --2- 172.21.15.150:0/945996334 >> v2:172.21.15.150:3300/0 conn(0x7fe3cc15a3d0 0x7fe3cc06ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:55.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.151+0000 7fe3d1287640 1 --2- 172.21.15.150:0/945996334 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3cc156720 0x7fe3cc0a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:55.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.151+0000 7fe3d1287640 1 -- 172.21.15.150:0/945996334 >> 172.21.15.150:0/945996334 conn(0x7fe3cc0c5a10 msgr2=0x7fe3cc0c5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:55.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.155+0000 7fe3d1287640 1 -- 172.21.15.150:0/945996334 shutdown_connections 2024-09-17T05:16:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.155+0000 7fe3d1287640 1 -- 172.21.15.150:0/945996334 wait complete. 2024-09-17T05:16:55.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.159+0000 7fe3d1287640 1 Processor -- start 2024-09-17T05:16:55.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.159+0000 7fe3d1287640 1 -- start start 2024-09-17T05:16:55.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.159+0000 7fe3d1287640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3cc06d390 0x7fe3cc1646f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:55.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3d1287640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3cc156720 0x7fe3cc14f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:55.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3d1287640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3cc15a3d0 0x7fe3cc165c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:55.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3d1287640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe3cc0a87e0 con 0x7fe3cc06d390 2024-09-17T05:16:55.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3d1287640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe3cc0a8660 con 0x7fe3cc15a3d0 2024-09-17T05:16:55.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3d1287640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe3cc074d70 con 0x7fe3cc156720 2024-09-17T05:16:55.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3ca7fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3cc156720 0x7fe3cc14f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:55.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3ca7fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3cc156720 0x7fe3cc14f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:52828/0 (socket says 172.21.15.150:52828) 2024-09-17T05:16:55.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3ca7fc640 1 -- 172.21.15.150:0/2558376835 learned_addr learned my addr 172.21.15.150:0/2558376835 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:55.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3caffd640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3cc06d390 0x7fe3cc1646f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:55.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3cb7fe640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3cc15a3d0 0x7fe3cc165c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:55.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3caffd640 1 -- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3cc156720 msgr2=0x7fe3cc14f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:55.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3caffd640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3cc156720 0x7fe3cc14f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:55.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3caffd640 1 -- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3cc15a3d0 msgr2=0x7fe3cc165c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:55.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3caffd640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3cc15a3d0 0x7fe3cc165c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:55.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.163+0000 7fe3caffd640 1 -- 172.21.15.150:0/2558376835 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3cc150210 con 0x7fe3cc06d390 2024-09-17T05:16:55.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.167+0000 7fe3caffd640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3cc06d390 0x7fe3cc1646f0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400ea20 tx=0x7fe3c400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:55.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.167+0000 7fe3b37fe640 1 -- 172.21.15.150:0/2558376835 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c400cde0 con 0x7fe3cc06d390 2024-09-17T05:16:55.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.167+0000 7fe3b37fe640 1 -- 172.21.15.150:0/2558376835 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3c4010520 con 0x7fe3cc06d390 2024-09-17T05:16:55.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.167+0000 7fe3b37fe640 1 -- 172.21.15.150:0/2558376835 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4010930 con 0x7fe3cc06d390 2024-09-17T05:16:55.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.179+0000 7fe3d1287640 1 -- 172.21.15.150:0/2558376835 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3cc1638a0 con 0x7fe3cc06d390 2024-09-17T05:16:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.183+0000 7fe3b37fe640 1 -- 172.21.15.150:0/2558376835 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe3c4010ad0 con 0x7fe3cc06d390 2024-09-17T05:16:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.183+0000 7fe3b37fe640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3b8073280 0x7fe3b8075790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:55.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.183+0000 7fe3ca7fc640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3b8073280 0x7fe3b8075790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:55.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.183+0000 7fe3d1287640 1 -- 172.21.15.150:0/2558376835 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3cc0a8240 con 0x7fe3cc06d390 2024-09-17T05:16:55.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.187+0000 7fe3b37fe640 1 -- 172.21.15.150:0/2558376835 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7fe3c4010dc0 con 0x7fe3cc06d390 2024-09-17T05:16:55.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.187+0000 7fe3ca7fc640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3b8073280 0x7fe3b8075790 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc166d00 tx=0x7fe3c000b390 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:55.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.187+0000 7fe3d1287640 1 -- 172.21.15.150:0/2558376835 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3cc075240 con 0x7fe3cc06d390 2024-09-17T05:16:55.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.203+0000 7fe3b37fe640 1 -- 172.21.15.150:0/2558376835 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe3c405f5a0 con 0x7fe3cc06d390 2024-09-17T05:16:55.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 -- 172.21.15.150:0/2558376835 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3cc0c6070 con 0x7fe3cc06d390 2024-09-17T05:16:55.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3b37fe640 1 -- 172.21.15.150:0/2558376835 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) ==== 95+0+4888 (secure 0 0 0) 0x7fe3c4063fd0 con 0x7fe3cc06d390 2024-09-17T05:16:55.799 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:16:55.799 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":195,"btime":"2024-09-17T05:16:55:703844+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:54.711313+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:16:55.799 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 195 2024-09-17T05:16:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 -- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3b8073280 msgr2=0x7fe3b8075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3b8073280 0x7fe3b8075790 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc166d00 tx=0x7fe3c000b390 comp rx=0 tx=0).stop 2024-09-17T05:16:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 -- 172.21.15.150:0/2558376835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3cc06d390 msgr2=0x7fe3cc1646f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3cc06d390 0x7fe3cc1646f0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400ea20 tx=0x7fe3c400eee0 comp rx=0 tx=0).stop 2024-09-17T05:16:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 -- 172.21.15.150:0/2558376835 shutdown_connections 2024-09-17T05:16:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe3b8073280 0x7fe3b8075790 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe3cc15a3d0 0x7fe3cc165c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe3cc156720 0x7fe3cc14f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 --2- 172.21.15.150:0/2558376835 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe3cc06d390 0x7fe3cc1646f0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:55.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 -- 172.21.15.150:0/2558376835 >> 172.21.15.150:0/2558376835 conn(0x7fe3cc0c5a10 msgr2=0x7fe3cc158890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:55.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 -- 172.21.15.150:0/2558376835 shutdown_connections 2024-09-17T05:16:55.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:55.799+0000 7fe3d1287640 1 -- 172.21.15.150:0/2558376835 wait complete. 2024-09-17T05:16:55.830 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:16:55:703844+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 195, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 194, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7920": { "addr": "172.21.15.150:6835/3297154462", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 3297154462, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 3297154462, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7920, "incarnation": 190, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:16:54.689070+0000", "qdb_cluster": [ 7920 ], "qdb_leader": 7920, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7920 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 195, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6563": { "addr": "172.21.15.176:6837/1903363960", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 1903363960, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 1903363960, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6563, "incarnation": 193, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:16:54.711313+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6563 } } } ], "standbys": [] } 2024-09-17T05:16:57.341 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T05:16:57.339+0000 7faf2f63c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:16:57.831 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:16:59.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.483+0000 7f3fef027640 1 -- 172.21.15.150:0/3875320387 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3fd4004810 msgr2=0x7f3fd4004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:59.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.483+0000 7f3fef027640 1 --2- 172.21.15.150:0/3875320387 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3fd4004810 0x7f3fd4004c20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3fe80aac60 tx=0x7f3fd801d760 comp rx=0 tx=0).stop 2024-09-17T05:16:59.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.483+0000 7f3fee826640 1 -- 172.21.15.150:0/3875320387 reap_dead start 2024-09-17T05:16:59.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.495+0000 7f3fef027640 1 -- 172.21.15.150:0/3875320387 shutdown_connections 2024-09-17T05:16:59.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.495+0000 7f3fef027640 1 -- 172.21.15.150:0/3875320387 >> 172.21.15.150:0/3875320387 conn(0x7f3fe806d250 msgr2=0x7f3fe806d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:16:59.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.503+0000 7f3fef027640 1 -- 172.21.15.150:0/3875320387 shutdown_connections 2024-09-17T05:16:59.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.503+0000 7f3fef027640 1 -- 172.21.15.150:0/3875320387 wait complete. 2024-09-17T05:16:59.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.507+0000 7f3fef027640 1 Processor -- start 2024-09-17T05:16:59.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fef027640 1 -- start start 2024-09-17T05:16:59.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fef027640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3fd4004810 0x7f3fe814ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:59.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fef027640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fe81468a0 0x7f3fe8144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:59.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fef027640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3fe8146cb0 0x7f3fe81454d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:59.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fed824640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fe81468a0 0x7f3fe8144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:59.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fed824640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fe81468a0 0x7f3fe8144f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58824/0 (socket says 172.21.15.150:58824) 2024-09-17T05:16:59.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fed824640 1 -- 172.21.15.150:0/3448202828 learned_addr learned my addr 172.21.15.150:0/3448202828 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:16:59.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fef027640 1 -- 172.21.15.150:0/3448202828 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe81620f0 con 0x7f3fe81468a0 2024-09-17T05:16:59.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fef027640 1 -- 172.21.15.150:0/3448202828 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe8161f70 con 0x7f3fe8146cb0 2024-09-17T05:16:59.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fef027640 1 -- 172.21.15.150:0/3448202828 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3fe8162270 con 0x7f3fd4004810 2024-09-17T05:16:59.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fee826640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3fe8146cb0 0x7f3fe81454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:59.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fed824640 1 -- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3fd4004810 msgr2=0x7f3fe814ec70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:16:59.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fed824640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3fd4004810 0x7f3fe814ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:59.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fed824640 1 -- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3fe8146cb0 msgr2=0x7f3fe81454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:16:59.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.511+0000 7f3fed824640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3fe8146cb0 0x7f3fe81454d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:16:59.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.515+0000 7f3fed824640 1 -- 172.21.15.150:0/3448202828 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe8073410 con 0x7f3fe81468a0 2024-09-17T05:16:59.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.515+0000 7f3fed824640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fe81468a0 0x7f3fe8144f90 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f3fd801be60 tx=0x7f3fd8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:59.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.515+0000 7f3fdeffd640 1 -- 172.21.15.150:0/3448202828 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8001440 con 0x7f3fe81468a0 2024-09-17T05:16:59.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.515+0000 7f3fdeffd640 1 -- 172.21.15.150:0/3448202828 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3fd80aa050 con 0x7f3fe81468a0 2024-09-17T05:16:59.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.515+0000 7f3fdeffd640 1 -- 172.21.15.150:0/3448202828 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd80048b0 con 0x7f3fe81468a0 2024-09-17T05:16:59.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.515+0000 7f3fef027640 1 -- 172.21.15.150:0/3448202828 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe80c3290 con 0x7f3fe81468a0 2024-09-17T05:16:59.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.515+0000 7f3fef027640 1 -- 172.21.15.150:0/3448202828 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe814b090 con 0x7f3fe81468a0 2024-09-17T05:16:59.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.515+0000 7f3fef027640 1 -- 172.21.15.150:0/3448202828 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fd00052a0 con 0x7f3fe81468a0 2024-09-17T05:16:59.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.519+0000 7f3fdeffd640 1 -- 172.21.15.150:0/3448202828 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3fd8004a50 con 0x7f3fe81468a0 2024-09-17T05:16:59.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.519+0000 7f3fdeffd640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3fcc0732d0 0x7f3fcc0757e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:16:59.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.519+0000 7f3fdeffd640 1 -- 172.21.15.150:0/3448202828 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f3fd812f870 con 0x7f3fe81468a0 2024-09-17T05:16:59.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.519+0000 7f3fee025640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3fcc0732d0 0x7f3fcc0757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:16:59.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.523+0000 7f3fee025640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3fcc0732d0 0x7f3fcc0757e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3fe80c3920 tx=0x7f3fe0009310 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:16:59.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:16:59.555+0000 7f3fdeffd640 1 -- 172.21.15.150:0/3448202828 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3fd8017510 con 0x7f3fe81468a0 2024-09-17T05:17:00.964 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-09-17T05:17:00.965 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T05:17:00.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T05:17:00.970 INFO:tasks.ceph.mds.a:Started 2024-09-17T05:17:00.970 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:17:01.378 INFO:tasks.ceph.mds.d.smithi176.stderr:2024-09-17T05:17:01.375+0000 7f3ad356f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:17:01.396 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-09-17T05:17:01.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.627+0000 7f3fef027640 1 -- 172.21.15.150:0/3448202828 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fd0005590 con 0x7f3fe81468a0 2024-09-17T05:17:01.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.627+0000 7f3fdeffd640 1 -- 172.21.15.150:0/3448202828 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) ==== 95+0+4895 (secure 0 0 0) 0x7f3fd80f9270 con 0x7f3fe81468a0 2024-09-17T05:17:01.638 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:17:01.638 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":196,"btime":"2024-09-17T05:16:56:716453+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:17:01.639 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 196 2024-09-17T05:17:01.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.643+0000 7f3fdcef9640 1 -- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3fcc0732d0 msgr2=0x7f3fcc0757e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:17:01.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.643+0000 7f3fdcef9640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3fcc0732d0 0x7f3fcc0757e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3fe80c3920 tx=0x7f3fe0009310 comp rx=0 tx=0).stop 2024-09-17T05:17:01.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.643+0000 7f3fdcef9640 1 -- 172.21.15.150:0/3448202828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fe81468a0 msgr2=0x7f3fe8144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:17:01.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.643+0000 7f3fdcef9640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fe81468a0 0x7f3fe8144f90 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f3fd801be60 tx=0x7f3fd8003c20 comp rx=0 tx=0).stop 2024-09-17T05:17:01.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.643+0000 7f3fdcef9640 1 -- 172.21.15.150:0/3448202828 shutdown_connections 2024-09-17T05:17:01.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.643+0000 7f3fdcef9640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3fcc0732d0 0x7f3fcc0757e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:01.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.643+0000 7f3fdcef9640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3fe8146cb0 0x7f3fe81454d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:01.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.643+0000 7f3fdcef9640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3fe81468a0 0x7f3fe8144f90 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:01.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.643+0000 7f3fdcef9640 1 --2- 172.21.15.150:0/3448202828 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3fd4004810 0x7f3fe814ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:01.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.643+0000 7f3fdcef9640 1 -- 172.21.15.150:0/3448202828 >> 172.21.15.150:0/3448202828 conn(0x7f3fe806d250 msgr2=0x7f3fe80c4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:17:01.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.643+0000 7f3fdcef9640 1 -- 172.21.15.150:0/3448202828 shutdown_connections 2024-09-17T05:17:01.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.643+0000 7f3fdcef9640 1 -- 172.21.15.150:0/3448202828 wait complete. 2024-09-17T05:17:01.778 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=4355 2024-09-17T05:17:01.779 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.b 2024-09-17T05:17:01.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.859+0000 7f773f2b3640 1 -- 172.21.15.150:0/1652904663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77380c7180 msgr2=0x7f7738152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:17:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.859+0000 7f773f2b3640 1 --2- 172.21.15.150:0/1652904663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77380c7180 0x7f7738152900 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f77380731f0 tx=0x7f772801c870 comp rx=0 tx=0).stop 2024-09-17T05:17:01.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.863+0000 7f773f2b3640 1 -- 172.21.15.150:0/1652904663 shutdown_connections 2024-09-17T05:17:01.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.863+0000 7f773f2b3640 1 --2- 172.21.15.150:0/1652904663 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77380c7180 0x7f7738152900 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:01.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.863+0000 7f773f2b3640 1 --2- 172.21.15.150:0/1652904663 >> v2:172.21.15.150:3300/0 conn(0x7f7738071b60 0x7f77380c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:01.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.863+0000 7f773f2b3640 1 --2- 172.21.15.150:0/1652904663 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f77380711c0 0x7f7738150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:01.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.863+0000 7f773f2b3640 1 -- 172.21.15.150:0/1652904663 >> 172.21.15.150:0/1652904663 conn(0x7f77380c4e30 msgr2=0x7f773806ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:17:01.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.883+0000 7f773f2b3640 1 -- 172.21.15.150:0/1652904663 shutdown_connections 2024-09-17T05:17:01.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.883+0000 7f773f2b3640 1 -- 172.21.15.150:0/1652904663 wait complete. 2024-09-17T05:17:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.887+0000 7f773f2b3640 1 Processor -- start 2024-09-17T05:17:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.887+0000 7f773f2b3640 1 -- start start 2024-09-17T05:17:01.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.899+0000 7f773f2b3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77380711c0 0x7f7738157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:17:01.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773f2b3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7738071b60 0x7f77380acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:17:01.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773d04f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77380711c0 0x7f7738157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:17:01.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773d04f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77380711c0 0x7f7738157c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:48784/0 (socket says 172.21.15.150:48784) 2024-09-17T05:17:01.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773d04f640 1 -- 172.21.15.150:0/2527527815 learned_addr learned my addr 172.21.15.150:0/2527527815 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:17:01.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773f2b3640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77380c7180 0x7f773814de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:17:01.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773f2b3640 1 -- 172.21.15.150:0/2527527815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7738156a90 con 0x7f77380c7180 2024-09-17T05:17:01.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773f2b3640 1 -- 172.21.15.150:0/2527527815 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7738156910 con 0x7f77380711c0 2024-09-17T05:17:01.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773d850640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77380c7180 0x7f773814de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:17:01.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773f2b3640 1 -- 172.21.15.150:0/2527527815 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7738156c10 con 0x7f7738071b60 2024-09-17T05:17:01.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773c84e640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7738071b60 0x7f77380acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:17:01.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773d850640 1 -- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7738071b60 msgr2=0x7f77380acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:17:01.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773d850640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7738071b60 0x7f77380acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:01.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773d850640 1 -- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77380711c0 msgr2=0x7f7738157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:17:01.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773d850640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77380711c0 0x7f7738157c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:01.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773d850640 1 -- 172.21.15.150:0/2527527815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77381586b0 con 0x7f77380c7180 2024-09-17T05:17:01.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773d850640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77380c7180 0x7f773814de20 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f7738070860 tx=0x7f7728004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:17:01.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773d04f640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77380711c0 0x7f7738157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:17:01.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f772dffb640 1 -- 172.21.15.150:0/2527527815 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280bc070 con 0x7f77380c7180 2024-09-17T05:17:01.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f772dffb640 1 -- 172.21.15.150:0/2527527815 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7728003600 con 0x7f77380c7180 2024-09-17T05:17:01.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f772dffb640 1 -- 172.21.15.150:0/2527527815 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77280b2040 con 0x7f77380c7180 2024-09-17T05:17:01.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.903+0000 7f773f2b3640 1 -- 172.21.15.150:0/2527527815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f773814f050 con 0x7f77380c7180 2024-09-17T05:17:01.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.907+0000 7f773f2b3640 1 -- 172.21.15.150:0/2527527815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7738158d40 con 0x7f77380c7180 2024-09-17T05:17:01.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.911+0000 7f772dffb640 1 -- 172.21.15.150:0/2527527815 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7728003a10 con 0x7f77380c7180 2024-09-17T05:17:01.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.915+0000 7f772dffb640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7724073440 0x7f7724075950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:17:01.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.915+0000 7f773d04f640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7724073440 0x7f7724075950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:17:01.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.915+0000 7f772dffb640 1 -- 172.21.15.150:0/2527527815 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f7728132170 con 0x7f77380c7180 2024-09-17T05:17:01.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.915+0000 7f773d04f640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7724073440 0x7f7724075950 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7730009d90 tx=0x7f7730009790 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:17:01.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.963+0000 7f773f2b3640 1 -- 172.21.15.150:0/2527527815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77180052a0 con 0x7f77380c7180 2024-09-17T05:17:01.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:01.975+0000 7f772dffb640 1 -- 172.21.15.150:0/2527527815 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f77280164f0 con 0x7f77380c7180 2024-09-17T05:17:03.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.827+0000 7f773f2b3640 1 -- 172.21.15.150:0/2527527815 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7718005860 con 0x7f77380c7180 2024-09-17T05:17:03.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.831+0000 7f772dffb640 1 -- 172.21.15.150:0/2527527815 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+5723 (secure 0 0 0) 0x7f77280f8a40 con 0x7f77380c7180 2024-09-17T05:17:03.833 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:17:03.834 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":197,"btime":"2024-09-17T05:17:01:822457+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:17:03.834 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 197 2024-09-17T05:17:03.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.843+0000 7f77237fe640 1 -- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7724073440 msgr2=0x7f7724075950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:17:03.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.847+0000 7f77237fe640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7724073440 0x7f7724075950 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7730009d90 tx=0x7f7730009790 comp rx=0 tx=0).stop 2024-09-17T05:17:03.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.847+0000 7f77237fe640 1 -- 172.21.15.150:0/2527527815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77380c7180 msgr2=0x7f773814de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:17:03.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.847+0000 7f77237fe640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77380c7180 0x7f773814de20 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f7738070860 tx=0x7f7728004040 comp rx=0 tx=0).stop 2024-09-17T05:17:03.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.847+0000 7f77237fe640 1 -- 172.21.15.150:0/2527527815 shutdown_connections 2024-09-17T05:17:03.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.847+0000 7f77237fe640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7724073440 0x7f7724075950 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:03.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.847+0000 7f77237fe640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77380c7180 0x7f773814de20 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:03.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.847+0000 7f77237fe640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7738071b60 0x7f77380acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:03.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.847+0000 7f77237fe640 1 --2- 172.21.15.150:0/2527527815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77380711c0 0x7f7738157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:03.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.847+0000 7f77237fe640 1 -- 172.21.15.150:0/2527527815 >> 172.21.15.150:0/2527527815 conn(0x7f77380c4e30 msgr2=0x7f77381555c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:17:03.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.847+0000 7f77237fe640 1 -- 172.21.15.150:0/2527527815 shutdown_connections 2024-09-17T05:17:03.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:03.847+0000 7f77237fe640 1 -- 172.21.15.150:0/2527527815 wait complete. 2024-09-17T05:17:03.953 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-09-17T05:17:03.953 INFO:tasks.mds_thrash.fs.[a]:waiting for 71 secs before thrashing 2024-09-17T05:17:04.779 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-09-17T05:17:04.780 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T05:17:04.780 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T05:17:04.782 INFO:tasks.ceph.mds.b:Started 2024-09-17T05:17:04.782 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:17:05.275 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-09-17T05:17:05.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.655+0000 7f283359e640 1 -- 172.21.15.150:0/1239707966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28340c1390 msgr2=0x7f28340c17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:17:05.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.655+0000 7f283359e640 1 --2- 172.21.15.150:0/1239707966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28340c1390 0x7f28340c17a0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f282c009c10 tx=0x7f282c01c700 comp rx=0 tx=0).stop 2024-09-17T05:17:05.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.655+0000 7f283359e640 1 -- 172.21.15.150:0/1239707966 shutdown_connections 2024-09-17T05:17:05.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.655+0000 7f283359e640 1 --2- 172.21.15.150:0/1239707966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f28340c1390 0x7f28340c17a0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:05.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.655+0000 7f283359e640 1 --2- 172.21.15.150:0/1239707966 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2834070170 0x7f283406d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:05.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.655+0000 7f283359e640 1 -- 172.21.15.150:0/1239707966 >> 172.21.15.150:0/1239707966 conn(0x7f28340a9ad0 msgr2=0x7f28340a9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:17:05.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.655+0000 7f283359e640 1 -- 172.21.15.150:0/1239707966 shutdown_connections 2024-09-17T05:17:05.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.655+0000 7f283359e640 1 -- 172.21.15.150:0/1239707966 wait complete. 2024-09-17T05:17:05.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.659+0000 7f283359e640 1 Processor -- start 2024-09-17T05:17:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.659+0000 7f283359e640 1 -- start start 2024-09-17T05:17:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.659+0000 7f283359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834070170 0x7f2834154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:17:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.659+0000 7f283259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834070170 0x7f2834154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:17:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.659+0000 7f283259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834070170 0x7f2834154800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:58862/0 (socket says 172.21.15.150:58862) 2024-09-17T05:17:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.659+0000 7f283259c640 1 -- 172.21.15.150:0/3791426962 learned_addr learned my addr 172.21.15.150:0/3791426962 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:17:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.659+0000 7f283359e640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28340c1390 0x7f283414ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:17:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.659+0000 7f283359e640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28340c1ce0 0x7f2834147990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:17:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.659+0000 7f283359e640 1 -- 172.21.15.150:0/3791426962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f283415aa20 con 0x7f2834070170 2024-09-17T05:17:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.663+0000 7f283359e640 1 -- 172.21.15.150:0/3791426962 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f283414fe40 con 0x7f28340c1390 2024-09-17T05:17:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.663+0000 7f2832d9d640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28340c1ce0 0x7f2834147990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:17:05.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.663+0000 7f283359e640 1 -- 172.21.15.150:0/3791426962 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f28341f5300 con 0x7f28340c1ce0 2024-09-17T05:17:05.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.663+0000 7f283259c640 1 -- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28340c1ce0 msgr2=0x7f2834147990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:17:05.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.663+0000 7f283259c640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28340c1ce0 0x7f2834147990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:05.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.663+0000 7f283259c640 1 -- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28340c1390 msgr2=0x7f283414ae30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:17:05.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.663+0000 7f283259c640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28340c1390 0x7f283414ae30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:05.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.663+0000 7f283259c640 1 -- 172.21.15.150:0/3791426962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2834142e10 con 0x7f2834070170 2024-09-17T05:17:05.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.663+0000 7f283259c640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834070170 0x7f2834154800 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f282400e9f0 tx=0x7f282400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:17:05.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.663+0000 7f2822ffd640 1 -- 172.21.15.150:0/3791426962 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282400ccc0 con 0x7f2834070170 2024-09-17T05:17:05.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.671+0000 7f2822ffd640 1 -- 172.21.15.150:0/3791426962 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2824010430 con 0x7f2834070170 2024-09-17T05:17:05.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.671+0000 7f283359e640 1 -- 172.21.15.150:0/3791426962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28341430f0 con 0x7f2834070170 2024-09-17T05:17:05.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.671+0000 7f283359e640 1 -- 172.21.15.150:0/3791426962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f283413e410 con 0x7f2834070170 2024-09-17T05:17:05.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.675+0000 7f2822ffd640 1 -- 172.21.15.150:0/3791426962 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2824010880 con 0x7f2834070170 2024-09-17T05:17:05.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.679+0000 7f2822ffd640 1 -- 172.21.15.150:0/3791426962 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2824010a20 con 0x7f2834070170 2024-09-17T05:17:05.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.679+0000 7f2822ffd640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2814073300 0x7f2814075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:17:05.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.679+0000 7f2822ffd640 1 -- 172.21.15.150:0/3791426962 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f2824095ad0 con 0x7f2834070170 2024-09-17T05:17:05.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.679+0000 7f2831d9b640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2814073300 0x7f2814075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:17:05.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.679+0000 7f2831d9b640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2814073300 0x7f2814075810 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f282c0ae000 tx=0x7f282c01af40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:17:05.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.691+0000 7f283359e640 1 -- 172.21.15.150:0/3791426962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28180052a0 con 0x7f2834070170 2024-09-17T05:17:05.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:05.723+0000 7f2822ffd640 1 -- 172.21.15.150:0/3791426962 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f282405fbb0 con 0x7f2834070170 2024-09-17T05:17:06.314 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:17:06.318 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:17:07.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.071+0000 7f283359e640 1 -- 172.21.15.150:0/3791426962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2818005590 con 0x7f2834070170 2024-09-17T05:17:07.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.071+0000 7f2822ffd640 1 -- 172.21.15.150:0/3791426962 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7f282405f4b0 con 0x7f2834070170 2024-09-17T05:17:07.084 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:17:07.084 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:17:07.084 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:17:07.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.099+0000 7f2820ff9640 1 -- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2814073300 msgr2=0x7f2814075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:17:07.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.099+0000 7f2820ff9640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2814073300 0x7f2814075810 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f282c0ae000 tx=0x7f282c01af40 comp rx=0 tx=0).stop 2024-09-17T05:17:07.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.099+0000 7f2820ff9640 1 -- 172.21.15.150:0/3791426962 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834070170 msgr2=0x7f2834154800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:17:07.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.099+0000 7f2820ff9640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834070170 0x7f2834154800 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f282400e9f0 tx=0x7f282400eeb0 comp rx=0 tx=0).stop 2024-09-17T05:17:07.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.107+0000 7f2820ff9640 1 -- 172.21.15.150:0/3791426962 shutdown_connections 2024-09-17T05:17:07.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.107+0000 7f2820ff9640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2814073300 0x7f2814075810 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:07.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.107+0000 7f2820ff9640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28340c1390 0x7f283414ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:07.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.107+0000 7f2820ff9640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f28340c1ce0 0x7f2834147990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:07.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.107+0000 7f2820ff9640 1 --2- 172.21.15.150:0/3791426962 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2834070170 0x7f2834154800 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:17:07.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.107+0000 7f2820ff9640 1 -- 172.21.15.150:0/3791426962 >> 172.21.15.150:0/3791426962 conn(0x7f28340a9ad0 msgr2=0x7f2834074720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:17:07.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.107+0000 7f2820ff9640 1 -- 172.21.15.150:0/3791426962 shutdown_connections 2024-09-17T05:17:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:17:07.123+0000 7f2820ff9640 1 -- 172.21.15.150:0/3791426962 wait complete. 2024-09-17T05:17:07.179 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-09-17T05:17:07.179 INFO:tasks.mds_thrash.fs.[b]:waiting for 69 secs before thrashing 2024-09-17T05:17:19.303 INFO:tasks.workunit.client.0.smithi150.stdout:Syncing()...5 sec 2024-09-17T05:17:19.304 INFO:tasks.workunit.client.0.smithi150.stdout:FFSB benchmark finished at: Tue Sep 17 05:17:19 2024 2024-09-17T05:17:19.304 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T05:17:19.304 INFO:tasks.workunit.client.0.smithi150.stdout:Results: 2024-09-17T05:17:19.304 INFO:tasks.workunit.client.0.smithi150.stdout:Benchmark took 309.81 sec 2024-09-17T05:17:19.304 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T05:17:19.305 INFO:tasks.workunit.client.0.smithi150.stdout:Total Results 2024-09-17T05:17:19.305 INFO:tasks.workunit.client.0.smithi150.stdout:=============== 2024-09-17T05:17:19.305 INFO:tasks.workunit.client.0.smithi150.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-09-17T05:17:19.305 INFO:tasks.workunit.client.0.smithi150.stdout: ======= ============ ========= ======= =========== ========== 2024-09-17T05:17:19.305 INFO:tasks.workunit.client.0.smithi150.stdout: write : 1361920 4396.05 100.000% 100.000% 17.2MB/sec 2024-09-17T05:17:19.305 INFO:tasks.workunit.client.0.smithi150.stdout:- 2024-09-17T05:17:19.306 INFO:tasks.workunit.client.0.smithi150.stdout:4396.05 Transactions per Second 2024-09-17T05:17:19.306 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T05:17:19.306 INFO:tasks.workunit.client.0.smithi150.stdout:Throughput Results 2024-09-17T05:17:19.306 INFO:tasks.workunit.client.0.smithi150.stdout:=================== 2024-09-17T05:17:19.306 INFO:tasks.workunit.client.0.smithi150.stdout:Write Throughput: 17.2MB/sec 2024-09-17T05:17:19.311 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T05:17:19.311 INFO:tasks.workunit.client.0.smithi150.stdout:System Call Latency statistics in millisecs 2024-09-17T05:17:19.311 INFO:tasks.workunit.client.0.smithi150.stdout:===== 2024-09-17T05:17:19.311 INFO:tasks.workunit.client.0.smithi150.stdout: Min Avg Max Total Calls 2024-09-17T05:17:19.312 INFO:tasks.workunit.client.0.smithi150.stdout: ======== ======== ======== ============ 2024-09-17T05:17:19.312 INFO:tasks.workunit.client.0.smithi150.stdout:[ open] 0.497000 1962.676126 20743.966797 1064 2024-09-17T05:17:19.312 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-17T05:17:19.312 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-17T05:17:19.312 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-17T05:17:19.312 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-17T05:17:19.312 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-17T05:17:19.312 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[5] 100.000000 - 200.000000 : 12 2024-09-17T05:17:19.312 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[6] 50.000000 - 100.000000 : 48 2024-09-17T05:17:19.312 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[7] 20.000000 - 50.000000 : 295 2024-09-17T05:17:19.312 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[8] 10.000000 - 20.000000 : 378 2024-09-17T05:17:19.312 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[9] 5.000000 - 10.000000 : 135 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[10] 2.000000 - 5.000000 : 43 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[11] 1.000000 - 2.000000 : 16 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[12] 0.500000 - 1.000000 : 8 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[13] 0.200000 - 0.500000 : 1 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout:[ write] 0.088000 5.568385 283.002991 1361920 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-17T05:17:19.313 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[4] 200.000000 - 500.000000 : 7 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[5] 100.000000 - 200.000000 : 324 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[6] 50.000000 - 100.000000 : 7793 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[7] 20.000000 - 50.000000 : 63282 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[8] 10.000000 - 20.000000 : 139827 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[9] 5.000000 - 10.000000 : 265733 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[10] 2.000000 - 5.000000 : 319220 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[11] 1.000000 - 2.000000 : 120940 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[12] 0.500000 - 1.000000 : 60866 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[13] 0.200000 - 0.500000 : 211634 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[14] 0.100000 - 0.200000 : 171806 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[15] 0.050000 - 0.100000 : 488 2024-09-17T05:17:19.314 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout:[ lseek] 0.000000 0.001038 7.781000 1361920 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-09-17T05:17:19.315 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[9] 5.000000 - 10.000000 : 2 2024-09-17T05:17:19.316 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[10] 2.000000 - 5.000000 : 3 2024-09-17T05:17:19.316 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[11] 1.000000 - 2.000000 : 3 2024-09-17T05:17:19.316 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[12] 0.500000 - 1.000000 : 4 2024-09-17T05:17:19.316 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[13] 0.200000 - 0.500000 : 30 2024-09-17T05:17:19.316 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[14] 0.100000 - 0.200000 : 40 2024-09-17T05:17:19.316 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[15] 0.050000 - 0.100000 : 52 2024-09-17T05:17:19.316 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[16] 0.020000 - 0.050000 : 352 2024-09-17T05:17:19.316 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[17] 0.010000 - 0.020000 : 842 2024-09-17T05:17:19.316 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[18] 0.000000 - 0.010000 : 1360592 2024-09-17T05:17:19.316 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout:[ close] 0.022000 2.558006 63.583000 1064 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[6] 50.000000 - 100.000000 : 4 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[7] 20.000000 - 50.000000 : 23 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[8] 10.000000 - 20.000000 : 36 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[9] 5.000000 - 10.000000 : 81 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[10] 2.000000 - 5.000000 : 145 2024-09-17T05:17:19.317 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[11] 1.000000 - 2.000000 : 104 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[12] 0.500000 - 1.000000 : 63 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[13] 0.200000 - 0.500000 : 75 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[14] 0.100000 - 0.200000 : 120 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[15] 0.050000 - 0.100000 : 102 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[16] 0.020000 - 0.050000 : 311 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout: 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout:0.4% User Time 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout:48.8% System Time 2024-09-17T05:17:19.318 INFO:tasks.workunit.client.0.smithi150.stdout:49.2% CPU Utilization 2024-09-17T05:17:19.319 INFO:tasks.workunit.client.0.smithi150.stderr:+ cd .. 2024-09-17T05:17:19.320 INFO:tasks.workunit.client.0.smithi150.stderr:+ rm -r tmp ffsb 2024-09-17T05:17:22.125 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T05:17:22.126 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T05:17:22.164 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-09-17T05:17:22.164 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-17T05:17:22.558 DEBUG:teuthology.parallel:result is None 2024-09-17T05:17:36.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:17:36.621 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:18:06.750 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:18:06.754 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:18:14.955 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:15.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.222+0000 7f042bc92640 1 -- 172.21.15.150:0/2498426075 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0410003a10 msgr2=0x7f0410003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:15.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.222+0000 7f042bc92640 1 --2- 172.21.15.150:0/2498426075 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0410003a10 0x7f0410003e80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f042000c310 tx=0x7f042001d000 comp rx=0 tx=0).stop 2024-09-17T05:18:15.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.222+0000 7f042a22f640 1 -- 172.21.15.150:0/2498426075 reap_dead start 2024-09-17T05:18:15.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.222+0000 7f042bc92640 1 -- 172.21.15.150:0/2498426075 shutdown_connections 2024-09-17T05:18:15.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.222+0000 7f042bc92640 1 -- 172.21.15.150:0/2498426075 >> 172.21.15.150:0/2498426075 conn(0x7f04240c5ae0 msgr2=0x7f0424073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:15.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.222+0000 7f042bc92640 1 -- 172.21.15.150:0/2498426075 shutdown_connections 2024-09-17T05:18:15.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.222+0000 7f042bc92640 1 -- 172.21.15.150:0/2498426075 wait complete. 2024-09-17T05:18:15.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042bc92640 1 Processor -- start 2024-09-17T05:18:15.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042bc92640 1 -- start start 2024-09-17T05:18:15.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042bc92640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0410003a10 0x7f04240c7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042bc92640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c77a0 0x7f04240abc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042bc92640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04240ac150 0x7f0424148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042bc92640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0424070610 con 0x7f04240ac150 2024-09-17T05:18:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042bc92640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f04240ab1d0 con 0x7f0410003a10 2024-09-17T05:18:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042bc92640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0424070790 con 0x7f04240c77a0 2024-09-17T05:18:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042a22f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04240ac150 0x7f0424148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042a22f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04240ac150 0x7f0424148a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53152/0 (socket says 172.21.15.150:53152) 2024-09-17T05:18:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042a22f640 1 -- 172.21.15.150:0/1771231698 learned_addr learned my addr 172.21.15.150:0/1771231698 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042922d640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c77a0 0x7f04240abc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042a22f640 1 -- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c77a0 msgr2=0x7f04240abc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042a22f640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c77a0 0x7f04240abc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042a22f640 1 -- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0410003a10 msgr2=0x7f04240c7260 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:18:15.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042a22f640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0410003a10 0x7f04240c7260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:15.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042a22f640 1 -- 172.21.15.150:0/1771231698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04241f0510 con 0x7f04240ac150 2024-09-17T05:18:15.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.226+0000 7f042a22f640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04240ac150 0x7f0424148a10 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f04240adac0 tx=0x7f041c00db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:15.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.230+0000 7f041a7fc640 1 -- 172.21.15.150:0/1771231698 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c00ba10 con 0x7f04240ac150 2024-09-17T05:18:15.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.230+0000 7f041a7fc640 1 -- 172.21.15.150:0/1771231698 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f041c019070 con 0x7f04240ac150 2024-09-17T05:18:15.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.230+0000 7f042bc92640 1 -- 172.21.15.150:0/1771231698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04240c7e70 con 0x7f04240ac150 2024-09-17T05:18:15.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.230+0000 7f041a7fc640 1 -- 172.21.15.150:0/1771231698 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c010460 con 0x7f04240ac150 2024-09-17T05:18:15.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.230+0000 7f042bc92640 1 -- 172.21.15.150:0/1771231698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04240acdc0 con 0x7f04240ac150 2024-09-17T05:18:15.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.230+0000 7f042bc92640 1 -- 172.21.15.150:0/1771231698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f042406e620 con 0x7f04240ac150 2024-09-17T05:18:15.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.230+0000 7f041a7fc640 1 -- 172.21.15.150:0/1771231698 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f041c010600 con 0x7f04240ac150 2024-09-17T05:18:15.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.230+0000 7f041a7fc640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0408073200 0x7f0408075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.230+0000 7f041a7fc640 1 -- 172.21.15.150:0/1771231698 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f041c095520 con 0x7f04240ac150 2024-09-17T05:18:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.230+0000 7f0429a2e640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0408073200 0x7f0408075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:15.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.238+0000 7f0429a2e640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0408073200 0x7f0408075710 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f04200ab000 tx=0x7f0420019e40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:15.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.242+0000 7f041a7fc640 1 -- 172.21.15.150:0/1771231698 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f04240c7e70 con 0x7f04240ac150 2024-09-17T05:18:15.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.542+0000 7f042bc92640 1 -- 172.21.15.150:0/1771231698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04240748b0 con 0x7f04240ac150 2024-09-17T05:18:15.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.542+0000 7f041a7fc640 1 -- 172.21.15.150:0/1771231698 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7f041c05f550 con 0x7f04240ac150 2024-09-17T05:18:15.550 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:15.550 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:15.550 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:15.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.554+0000 7f042bc92640 1 -- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0408073200 msgr2=0x7f0408075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:15.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.554+0000 7f042bc92640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0408073200 0x7f0408075710 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f04200ab000 tx=0x7f0420019e40 comp rx=0 tx=0).stop 2024-09-17T05:18:15.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.554+0000 7f042bc92640 1 -- 172.21.15.150:0/1771231698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04240ac150 msgr2=0x7f0424148a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:15.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.554+0000 7f042bc92640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04240ac150 0x7f0424148a10 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f04240adac0 tx=0x7f041c00db60 comp rx=0 tx=0).stop 2024-09-17T05:18:15.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.554+0000 7f042bc92640 1 -- 172.21.15.150:0/1771231698 shutdown_connections 2024-09-17T05:18:15.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.554+0000 7f042bc92640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0408073200 0x7f0408075710 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:15.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.554+0000 7f042bc92640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04240ac150 0x7f0424148a10 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:15.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.554+0000 7f042bc92640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f04240c77a0 0x7f04240abc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:15.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.554+0000 7f042bc92640 1 --2- 172.21.15.150:0/1771231698 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0410003a10 0x7f04240c7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:15.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.554+0000 7f042bc92640 1 -- 172.21.15.150:0/1771231698 >> 172.21.15.150:0/1771231698 conn(0x7f04240c5ae0 msgr2=0x7f042414cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:15.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.554+0000 7f042bc92640 1 -- 172.21.15.150:0/1771231698 shutdown_connections 2024-09-17T05:18:15.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.554+0000 7f042bc92640 1 -- 172.21.15.150:0/1771231698 wait complete. 2024-09-17T05:18:15.579 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-09-17T05:18:15.579 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T05:18:15.579 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:18:15.602 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T05:18:15.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:15.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.818+0000 7fc689ccd640 1 -- 172.21.15.150:0/1340125682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6840a7350 msgr2=0x7fc684150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:15.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.818+0000 7fc689ccd640 1 --2- 172.21.15.150:0/1340125682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6840a7350 0x7fc684150060 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fc678009900 tx=0x7fc67801c880 comp rx=0 tx=0).stop 2024-09-17T05:18:15.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.818+0000 7fc689ccd640 1 -- 172.21.15.150:0/1340125682 shutdown_connections 2024-09-17T05:18:15.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.818+0000 7fc689ccd640 1 --2- 172.21.15.150:0/1340125682 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6841505a0 0x7fc684152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:15.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.818+0000 7fc689ccd640 1 --2- 172.21.15.150:0/1340125682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6840a7350 0x7fc684150060 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.818+0000 7fc689ccd640 1 -- 172.21.15.150:0/1340125682 >> 172.21.15.150:0/1340125682 conn(0x7fc6840c4a00 msgr2=0x7fc6840c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:15.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.822+0000 7fc689ccd640 1 -- 172.21.15.150:0/1340125682 shutdown_connections 2024-09-17T05:18:15.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.822+0000 7fc689ccd640 1 -- 172.21.15.150:0/1340125682 wait complete. 2024-09-17T05:18:15.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.822+0000 7fc689ccd640 1 Processor -- start 2024-09-17T05:18:15.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc689ccd640 1 -- start start 2024-09-17T05:18:15.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc689ccd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6840a7350 0x7fc6840aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:15.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc689ccd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6840c42e0 0x7fc6840ad180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:15.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc689ccd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6841505a0 0x7fc6841fa470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:15.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc689ccd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc6841537c0 con 0x7fc6840c42e0 2024-09-17T05:18:15.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc689ccd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc684153640 con 0x7fc6840a7350 2024-09-17T05:18:15.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc689ccd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc6840c5c30 con 0x7fc6841505a0 2024-09-17T05:18:15.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc682ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6840c42e0 0x7fc6840ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:15.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc682ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6840c42e0 0x7fc6840ad180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53174/0 (socket says 172.21.15.150:53174) 2024-09-17T05:18:15.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc682ffd640 1 -- 172.21.15.150:0/3140979763 learned_addr learned my addr 172.21.15.150:0/3140979763 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:15.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc6837fe640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6840a7350 0x7fc6840aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:15.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc683fff640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6841505a0 0x7fc6841fa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:15.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc682ffd640 1 -- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6841505a0 msgr2=0x7fc6841fa470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:15.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc682ffd640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6841505a0 0x7fc6841fa470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:15.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc682ffd640 1 -- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6840a7350 msgr2=0x7fc6840aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:15.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc682ffd640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6840a7350 0x7fc6840aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:15.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc682ffd640 1 -- 172.21.15.150:0/3140979763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6840c1b00 con 0x7fc6840c42e0 2024-09-17T05:18:15.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc682ffd640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6840c42e0 0x7fc6840ad180 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fc678009850 tx=0x7fc678005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:15.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc66bfff640 1 -- 172.21.15.150:0/3140979763 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6780b2040 con 0x7fc6840c42e0 2024-09-17T05:18:15.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc689ccd640 1 -- 172.21.15.150:0/3140979763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc684155eb0 con 0x7fc6840c42e0 2024-09-17T05:18:15.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc689ccd640 1 -- 172.21.15.150:0/3140979763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc684156e30 con 0x7fc6840c42e0 2024-09-17T05:18:15.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc66bfff640 1 -- 172.21.15.150:0/3140979763 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc678004310 con 0x7fc6840c42e0 2024-09-17T05:18:15.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.826+0000 7fc66bfff640 1 -- 172.21.15.150:0/3140979763 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc678005290 con 0x7fc6840c42e0 2024-09-17T05:18:15.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.830+0000 7fc689ccd640 1 -- 172.21.15.150:0/3140979763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6840c6190 con 0x7fc6840c42e0 2024-09-17T05:18:15.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.830+0000 7fc66bfff640 1 -- 172.21.15.150:0/3140979763 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc678005430 con 0x7fc6840c42e0 2024-09-17T05:18:15.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.830+0000 7fc66bfff640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc670073200 0x7fc670075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:15.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.830+0000 7fc66bfff640 1 -- 172.21.15.150:0/3140979763 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7fc67812f1d0 con 0x7fc6840c42e0 2024-09-17T05:18:15.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.830+0000 7fc6837fe640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc670073200 0x7fc670075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:15.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.830+0000 7fc6837fe640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc670073200 0x7fc670075710 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fc674007fd0 tx=0x7fc674009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:15.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:15.842+0000 7fc66bfff640 1 -- 172.21.15.150:0/3140979763 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc6780164d0 con 0x7fc6840c42e0 2024-09-17T05:18:16.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.138+0000 7fc689ccd640 1 -- 172.21.15.150:0/3140979763 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6840c67b0 con 0x7fc6840c42e0 2024-09-17T05:18:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.138+0000 7fc66bfff640 1 -- 172.21.15.150:0/3140979763 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7fc6780f8140 con 0x7fc6840c42e0 2024-09-17T05:18:16.139 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:16.139 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:16.140 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.142+0000 7fc689ccd640 1 -- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc670073200 msgr2=0x7fc670075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.142+0000 7fc689ccd640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc670073200 0x7fc670075710 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fc674007fd0 tx=0x7fc674009040 comp rx=0 tx=0).stop 2024-09-17T05:18:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.142+0000 7fc689ccd640 1 -- 172.21.15.150:0/3140979763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6840c42e0 msgr2=0x7fc6840ad180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.142+0000 7fc689ccd640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6840c42e0 0x7fc6840ad180 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fc678009850 tx=0x7fc678005de0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.142+0000 7fc689ccd640 1 -- 172.21.15.150:0/3140979763 shutdown_connections 2024-09-17T05:18:16.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.142+0000 7fc689ccd640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc670073200 0x7fc670075710 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.142+0000 7fc689ccd640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc6841505a0 0x7fc6841fa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.142+0000 7fc689ccd640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc6840c42e0 0x7fc6840ad180 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.142+0000 7fc689ccd640 1 --2- 172.21.15.150:0/3140979763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc6840a7350 0x7fc6840aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.142+0000 7fc689ccd640 1 -- 172.21.15.150:0/3140979763 >> 172.21.15.150:0/3140979763 conn(0x7fc6840c4a00 msgr2=0x7fc6840c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:16.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.142+0000 7fc689ccd640 1 -- 172.21.15.150:0/3140979763 shutdown_connections 2024-09-17T05:18:16.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.142+0000 7fc689ccd640 1 -- 172.21.15.150:0/3140979763 wait complete. 2024-09-17T05:18:16.168 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:18:16.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:16.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.366+0000 7f3bc956b640 1 -- 172.21.15.150:0/3897299642 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc40c4250 msgr2=0x7f3bc4153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:16.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.366+0000 7f3bc956b640 1 --2- 172.21.15.150:0/3897299642 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc40c4250 0x7f3bc4153ee0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4009870 tx=0x7f3bb401c860 comp rx=0 tx=0).stop 2024-09-17T05:18:16.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.366+0000 7f3bc956b640 1 -- 172.21.15.150:0/3897299642 shutdown_connections 2024-09-17T05:18:16.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.366+0000 7f3bc956b640 1 --2- 172.21.15.150:0/3897299642 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc40c4250 0x7f3bc4153ee0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.366+0000 7f3bc956b640 1 -- 172.21.15.150:0/3897299642 >> 172.21.15.150:0/3897299642 conn(0x7f3bc406e360 msgr2=0x7f3bc406e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:16.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.366+0000 7f3bc956b640 1 -- 172.21.15.150:0/3897299642 shutdown_connections 2024-09-17T05:18:16.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.366+0000 7f3bc956b640 1 -- 172.21.15.150:0/3897299642 wait complete. 2024-09-17T05:18:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.366+0000 7f3bc956b640 1 Processor -- start 2024-09-17T05:18:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.366+0000 7f3bc956b640 1 -- start start 2024-09-17T05:18:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bc956b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc40c3900 0x7f3bc4159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bc956b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc40c3d10 0x7f3bc415b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bc956b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bc40c4250 0x7f3bc40c2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bc956b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4071120 con 0x7f3bc40c4250 2024-09-17T05:18:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bc956b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4070fa0 con 0x7f3bc40c3900 2024-09-17T05:18:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bc956b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc4072db0 con 0x7f3bc40c3d10 2024-09-17T05:18:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bc37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bc40c4250 0x7f3bc40c2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bc37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bc40c4250 0x7f3bc40c2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53188/0 (socket says 172.21.15.150:53188) 2024-09-17T05:18:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bc37fe640 1 -- 172.21.15.150:0/3423760019 learned_addr learned my addr 172.21.15.150:0/3423760019 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:16.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bbbfff640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc40c3d10 0x7f3bc415b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:16.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bbbfff640 1 -- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc40c3900 msgr2=0x7f3bc4159050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:18:16.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bbbfff640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc40c3900 0x7f3bc4159050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bbbfff640 1 -- 172.21.15.150:0/3423760019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bc40c4250 msgr2=0x7f3bc40c2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:16.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bbbfff640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bc40c4250 0x7f3bc40c2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bbbfff640 1 -- 172.21.15.150:0/3423760019 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc41ed4b0 con 0x7f3bc40c3d10 2024-09-17T05:18:16.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bc37fe640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bc40c4250 0x7f3bc40c2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:18:16.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bbbfff640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc40c3d10 0x7f3bc415b220 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4074510 tx=0x7f3bac00c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bbb7fe640 1 -- 172.21.15.150:0/3423760019 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac017070 con 0x7f3bc40c3d10 2024-09-17T05:18:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bbb7fe640 1 -- 172.21.15.150:0/3423760019 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bac003600 con 0x7f3bc40c3d10 2024-09-17T05:18:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bbb7fe640 1 -- 172.21.15.150:0/3423760019 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac003a50 con 0x7f3bc40c3d10 2024-09-17T05:18:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bc956b640 1 -- 172.21.15.150:0/3423760019 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc4158af0 con 0x7f3bc40c3d10 2024-09-17T05:18:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.370+0000 7f3bc956b640 1 -- 172.21.15.150:0/3423760019 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc41f2f90 con 0x7f3bc40c3d10 2024-09-17T05:18:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.374+0000 7f3bc956b640 1 -- 172.21.15.150:0/3423760019 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc4153ee0 con 0x7f3bc40c3d10 2024-09-17T05:18:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.374+0000 7f3bbb7fe640 1 -- 172.21.15.150:0/3423760019 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3bac01a070 con 0x7f3bc40c3d10 2024-09-17T05:18:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.374+0000 7f3bbb7fe640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3ba4073200 0x7f3ba4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.374+0000 7f3bbb7fe640 1 -- 172.21.15.150:0/3423760019 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f3bac094770 con 0x7f3bc40c3d10 2024-09-17T05:18:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.374+0000 7f3bc2ffd640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3ba4073200 0x7f3ba4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.374+0000 7f3bc2ffd640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3ba4073200 0x7f3ba4075710 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40af000 tx=0x7f3bb401b0a0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.382+0000 7f3bbb7fe640 1 -- 172.21.15.150:0/3423760019 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3bac05e1d0 con 0x7f3bc40c3d10 2024-09-17T05:18:16.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.670+0000 7f3bc956b640 1 -- 172.21.15.150:0/3423760019 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bc4153800 con 0x7f3bc40c3d10 2024-09-17T05:18:16.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.670+0000 7f3bbb7fe640 1 -- 172.21.15.150:0/3423760019 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7f3bac062c00 con 0x7f3bc40c3d10 2024-09-17T05:18:16.673 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:16.673 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:16.673 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:16.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.674+0000 7f3bc956b640 1 -- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3ba4073200 msgr2=0x7f3ba4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:16.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.674+0000 7f3bc956b640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3ba4073200 0x7f3ba4075710 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40af000 tx=0x7f3bb401b0a0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.674+0000 7f3bc956b640 1 -- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc40c3d10 msgr2=0x7f3bc415b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:16.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.674+0000 7f3bc956b640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc40c3d10 0x7f3bc415b220 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4074510 tx=0x7f3bac00c820 comp rx=0 tx=0).stop 2024-09-17T05:18:16.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.674+0000 7f3bc956b640 1 -- 172.21.15.150:0/3423760019 shutdown_connections 2024-09-17T05:18:16.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.674+0000 7f3bc956b640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3ba4073200 0x7f3ba4075710 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.674+0000 7f3bc956b640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bc40c4250 0x7f3bc40c2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.674+0000 7f3bc956b640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3bc40c3d10 0x7f3bc415b220 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.674+0000 7f3bc956b640 1 --2- 172.21.15.150:0/3423760019 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3bc40c3900 0x7f3bc4159050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.674+0000 7f3bc956b640 1 -- 172.21.15.150:0/3423760019 >> 172.21.15.150:0/3423760019 conn(0x7f3bc406e360 msgr2=0x7f3bc40ae1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:16.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.674+0000 7f3bc956b640 1 -- 172.21.15.150:0/3423760019 shutdown_connections 2024-09-17T05:18:16.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.674+0000 7f3bc956b640 1 -- 172.21.15.150:0/3423760019 wait complete. 2024-09-17T05:18:16.708 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-09-17T05:18:16.708 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T05:18:16.708 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:18:16.729 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T05:18:16.729 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.926+0000 7fae82fa3640 1 -- 172.21.15.150:0/1074790231 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae680099c0 msgr2=0x7fae6800bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.926+0000 7fae82fa3640 1 --2- 172.21.15.150:0/1074790231 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae680099c0 0x7fae6800bde0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fae7c06fcd0 tx=0x7fae6401ce90 comp rx=0 tx=0).stop 2024-09-17T05:18:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.926+0000 7fae82fa3640 1 -- 172.21.15.150:0/1074790231 shutdown_connections 2024-09-17T05:18:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.926+0000 7fae82fa3640 1 --2- 172.21.15.150:0/1074790231 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae680099c0 0x7fae6800bde0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.926+0000 7fae82fa3640 1 -- 172.21.15.150:0/1074790231 >> 172.21.15.150:0/1074790231 conn(0x7fae7c0c4e30 msgr2=0x7fae7c06ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.926+0000 7fae82fa3640 1 -- 172.21.15.150:0/1074790231 shutdown_connections 2024-09-17T05:18:16.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.926+0000 7fae82fa3640 1 -- 172.21.15.150:0/1074790231 wait complete. 2024-09-17T05:18:16.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.930+0000 7fae82fa3640 1 Processor -- start 2024-09-17T05:18:16.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.930+0000 7fae82fa3640 1 -- start start 2024-09-17T05:18:16.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.930+0000 7fae82fa3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae680099c0 0x7fae7c158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:16.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.930+0000 7fae82fa3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae7c0ad020 0x7fae7c0bcd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:16.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.930+0000 7fae82fa3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae7c0ad430 0x7fae7c14c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:16.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.930+0000 7fae82fa3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c151000 con 0x7fae680099c0 2024-09-17T05:18:16.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.930+0000 7fae82fa3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c074710 con 0x7fae7c0ad430 2024-09-17T05:18:16.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.930+0000 7fae82fa3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fae7c073e50 con 0x7fae7c0ad020 2024-09-17T05:18:16.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.930+0000 7fae81540640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae7c0ad430 0x7fae7c14c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:16.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.930+0000 7fae81540640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae7c0ad430 0x7fae7c14c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:35684/0 (socket says 172.21.15.150:35684) 2024-09-17T05:18:16.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.930+0000 7fae81540640 1 -- 172.21.15.150:0/2461612347 learned_addr learned my addr 172.21.15.150:0/2461612347 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.934+0000 7fae7bfff640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae7c0ad020 0x7fae7c0bcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.934+0000 7fae80d3f640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae680099c0 0x7fae7c158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.934+0000 7fae81540640 1 -- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae7c0ad020 msgr2=0x7fae7c0bcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.934+0000 7fae81540640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae7c0ad020 0x7fae7c0bcd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.934+0000 7fae81540640 1 -- 172.21.15.150:0/2461612347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae680099c0 msgr2=0x7fae7c158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:16.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.934+0000 7fae81540640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae680099c0 0x7fae7c158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:16.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.934+0000 7fae81540640 1 -- 172.21.15.150:0/2461612347 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae7c0becb0 con 0x7fae7c0ad430 2024-09-17T05:18:16.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.938+0000 7fae81540640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae7c0ad430 0x7fae7c14c0c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fae6401c950 tx=0x7fae64006c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:16.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.938+0000 7fae797fa640 1 -- 172.21.15.150:0/2461612347 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae640017c0 con 0x7fae7c0ad430 2024-09-17T05:18:16.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.938+0000 7fae797fa640 1 -- 172.21.15.150:0/2461612347 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae640061e0 con 0x7fae7c0ad430 2024-09-17T05:18:16.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.938+0000 7fae797fa640 1 -- 172.21.15.150:0/2461612347 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae64003600 con 0x7fae7c0ad430 2024-09-17T05:18:16.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.938+0000 7fae82fa3640 1 -- 172.21.15.150:0/2461612347 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae7c150340 con 0x7fae7c0ad430 2024-09-17T05:18:16.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.938+0000 7fae82fa3640 1 -- 172.21.15.150:0/2461612347 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae7c0bd7b0 con 0x7fae7c0ad430 2024-09-17T05:18:16.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.938+0000 7fae82fa3640 1 -- 172.21.15.150:0/2461612347 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae7c0bda10 con 0x7fae7c0ad430 2024-09-17T05:18:16.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.938+0000 7fae797fa640 1 -- 172.21.15.150:0/2461612347 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fae640037a0 con 0x7fae7c0ad430 2024-09-17T05:18:16.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.942+0000 7fae797fa640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae5c073200 0x7fae5c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:16.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.942+0000 7fae797fa640 1 -- 172.21.15.150:0/2461612347 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7fae6412f370 con 0x7fae7c0ad430 2024-09-17T05:18:16.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.942+0000 7fae80d3f640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae5c073200 0x7fae5c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:16.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.942+0000 7fae80d3f640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae5c073200 0x7fae5c075710 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fae74000c10 tx=0x7fae74006b80 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:16.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:16.958+0000 7fae797fa640 1 -- 172.21.15.150:0/2461612347 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fae64016c90 con 0x7fae7c0ad430 2024-09-17T05:18:17.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.250+0000 7fae82fa3640 1 -- 172.21.15.150:0/2461612347 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae7c153fa0 con 0x7fae7c0ad430 2024-09-17T05:18:17.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae797fa640 1 -- 172.21.15.150:0/2461612347 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7fae640f94a0 con 0x7fae7c0ad430 2024-09-17T05:18:17.253 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:17.253 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:17.254 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:17.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae82fa3640 1 -- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae5c073200 msgr2=0x7fae5c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:17.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae82fa3640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae5c073200 0x7fae5c075710 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fae74000c10 tx=0x7fae74006b80 comp rx=0 tx=0).stop 2024-09-17T05:18:17.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae82fa3640 1 -- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae7c0ad430 msgr2=0x7fae7c14c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:17.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae82fa3640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae7c0ad430 0x7fae7c14c0c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fae6401c950 tx=0x7fae64006c00 comp rx=0 tx=0).stop 2024-09-17T05:18:17.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae82fa3640 1 -- 172.21.15.150:0/2461612347 shutdown_connections 2024-09-17T05:18:17.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae82fa3640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fae5c073200 0x7fae5c075710 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:17.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae82fa3640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fae7c0ad430 0x7fae7c14c0c0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:17.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae82fa3640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fae7c0ad020 0x7fae7c0bcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:17.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae82fa3640 1 --2- 172.21.15.150:0/2461612347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae680099c0 0x7fae7c158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:17.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae82fa3640 1 -- 172.21.15.150:0/2461612347 >> 172.21.15.150:0/2461612347 conn(0x7fae7c0c4e30 msgr2=0x7fae7c1512c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:17.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae82fa3640 1 -- 172.21.15.150:0/2461612347 shutdown_connections 2024-09-17T05:18:17.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:17.254+0000 7fae82fa3640 1 -- 172.21.15.150:0/2461612347 wait complete. 2024-09-17T05:18:17.285 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:18:18.168 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:18.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.358+0000 7f0580ce1640 1 -- 172.21.15.150:0/2050493447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f057c06fca0 msgr2=0x7f057c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:18.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.358+0000 7f0580ce1640 1 --2- 172.21.15.150:0/2050493447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f057c06fca0 0x7f057c06e090 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f057c072bc0 tx=0x7f057001c830 comp rx=0 tx=0).stop 2024-09-17T05:18:18.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.358+0000 7f0580ce1640 1 -- 172.21.15.150:0/2050493447 shutdown_connections 2024-09-17T05:18:18.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.358+0000 7f0580ce1640 1 --2- 172.21.15.150:0/2050493447 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f057c06e5d0 0x7f057c153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:18.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.358+0000 7f0580ce1640 1 --2- 172.21.15.150:0/2050493447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f057c06fca0 0x7f057c06e090 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:18.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.358+0000 7f0580ce1640 1 --2- 172.21.15.150:0/2050493447 >> v2:172.21.15.176:3300/0 conn(0x7f057c0745c0 0x7f057c150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:18.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.358+0000 7f0580ce1640 1 -- 172.21.15.150:0/2050493447 >> 172.21.15.150:0/2050493447 conn(0x7f057c0c5ae0 msgr2=0x7f057c073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:18.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.358+0000 7f0580ce1640 1 -- 172.21.15.150:0/2050493447 shutdown_connections 2024-09-17T05:18:18.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.358+0000 7f0580ce1640 1 -- 172.21.15.150:0/2050493447 wait complete. 2024-09-17T05:18:18.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.362+0000 7f0580ce1640 1 Processor -- start 2024-09-17T05:18:18.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.362+0000 7f0580ce1640 1 -- start start 2024-09-17T05:18:18.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.362+0000 7f0580ce1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f057c06e5d0 0x7f057c148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:18.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.362+0000 7f0580ce1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f057c06fca0 0x7f057c0ad0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:18.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.362+0000 7f0580ce1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f057c0745c0 0x7f057c0ab9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:18.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.362+0000 7f0580ce1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f057c1512f0 con 0x7f057c0745c0 2024-09-17T05:18:18.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.362+0000 7f0580ce1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f057c070610 con 0x7f057c06fca0 2024-09-17T05:18:18.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.362+0000 7f0580ce1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f057c070790 con 0x7f057c06e5d0 2024-09-17T05:18:18.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f057ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f057c0745c0 0x7f057c0ab9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:18.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f057ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f057c0745c0 0x7f057c0ab9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53234/0 (socket says 172.21.15.150:53234) 2024-09-17T05:18:18.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f057ad9d640 1 -- 172.21.15.150:0/3033155994 learned_addr learned my addr 172.21.15.150:0/3033155994 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:18.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f057a59c640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f057c06e5d0 0x7f057c148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f057ad9d640 1 -- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f057c06e5d0 msgr2=0x7f057c148f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f057ad9d640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f057c06e5d0 0x7f057c148f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f057ad9d640 1 -- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f057c06fca0 msgr2=0x7f057c0ad0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:18:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f057ad9d640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f057c06fca0 0x7f057c0ad0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f057ad9d640 1 -- 172.21.15.150:0/3033155994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f057c151fe0 con 0x7f057c0745c0 2024-09-17T05:18:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f057ad9d640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f057c0745c0 0x7f057c0ab9a0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f057400b970 tx=0x7f057400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f056affd640 1 -- 172.21.15.150:0/3033155994 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0574019070 con 0x7f057c0745c0 2024-09-17T05:18:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f0580ce1640 1 -- 172.21.15.150:0/3033155994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f057c0c7ec0 con 0x7f057c0745c0 2024-09-17T05:18:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f0580ce1640 1 -- 172.21.15.150:0/3033155994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f057c0c4150 con 0x7f057c0745c0 2024-09-17T05:18:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f056affd640 1 -- 172.21.15.150:0/3033155994 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f057400cc30 con 0x7f057c0745c0 2024-09-17T05:18:18.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f056affd640 1 -- 172.21.15.150:0/3033155994 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05740146a0 con 0x7f057c0745c0 2024-09-17T05:18:18.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.366+0000 7f0580ce1640 1 -- 172.21.15.150:0/3033155994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05600052a0 con 0x7f057c0745c0 2024-09-17T05:18:18.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.370+0000 7f056affd640 1 -- 172.21.15.150:0/3033155994 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f057401c070 con 0x7f057c0745c0 2024-09-17T05:18:18.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.370+0000 7f056affd640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f055c073280 0x7f055c075790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:18.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.370+0000 7f056affd640 1 -- 172.21.15.150:0/3033155994 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f0574094cc0 con 0x7f057c0745c0 2024-09-17T05:18:18.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.370+0000 7f057a59c640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f055c073280 0x7f055c075790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:18.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.370+0000 7f057a59c640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f055c073280 0x7f055c075790 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f056c009b40 tx=0x7f056c0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:18.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.374+0000 7f056affd640 1 -- 172.21.15.150:0/3033155994 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f057405e7d0 con 0x7f057c0745c0 2024-09-17T05:18:18.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.686+0000 7f0580ce1640 1 -- 172.21.15.150:0/3033155994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0560005860 con 0x7f057c0745c0 2024-09-17T05:18:18.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.686+0000 7f056affd640 1 -- 172.21.15.150:0/3033155994 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7f0574063200 con 0x7f057c0745c0 2024-09-17T05:18:18.688 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:18.688 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:18.689 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:18.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.690+0000 7f0580ce1640 1 -- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f055c073280 msgr2=0x7f055c075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:18.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.690+0000 7f0580ce1640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f055c073280 0x7f055c075790 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f056c009b40 tx=0x7f056c0091e0 comp rx=0 tx=0).stop 2024-09-17T05:18:18.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.690+0000 7f0580ce1640 1 -- 172.21.15.150:0/3033155994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f057c0745c0 msgr2=0x7f057c0ab9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:18.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.690+0000 7f0580ce1640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f057c0745c0 0x7f057c0ab9a0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f057400b970 tx=0x7f057400be30 comp rx=0 tx=0).stop 2024-09-17T05:18:18.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.690+0000 7f0580ce1640 1 -- 172.21.15.150:0/3033155994 shutdown_connections 2024-09-17T05:18:18.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.690+0000 7f0580ce1640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f055c073280 0x7f055c075790 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:18.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.690+0000 7f0580ce1640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f057c06fca0 0x7f057c0ad0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:18.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.690+0000 7f0580ce1640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f057c0745c0 0x7f057c0ab9a0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:18.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.690+0000 7f0580ce1640 1 --2- 172.21.15.150:0/3033155994 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f057c06e5d0 0x7f057c148f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:18.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.690+0000 7f0580ce1640 1 -- 172.21.15.150:0/3033155994 >> 172.21.15.150:0/3033155994 conn(0x7f057c0c5ae0 msgr2=0x7f057c14e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:18.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.690+0000 7f0580ce1640 1 -- 172.21.15.150:0/3033155994 shutdown_connections 2024-09-17T05:18:18.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:18.690+0000 7f0580ce1640 1 -- 172.21.15.150:0/3033155994 wait complete. 2024-09-17T05:18:18.715 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:18:19.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:19.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.466+0000 7f1a54360640 1 -- 172.21.15.150:0/3679609397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c06fca0 msgr2=0x7f1a4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:19.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.466+0000 7f1a54360640 1 --2- 172.21.15.150:0/3679609397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c06fca0 0x7f1a4c06e090 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0c2ea0 tx=0x7f1a3c01c770 comp rx=0 tx=0).stop 2024-09-17T05:18:19.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.466+0000 7f1a54360640 1 -- 172.21.15.150:0/3679609397 shutdown_connections 2024-09-17T05:18:19.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.466+0000 7f1a54360640 1 --2- 172.21.15.150:0/3679609397 >> v2:172.21.15.176:3300/0 conn(0x7f1a4c06e5d0 0x7f1a4c15dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:19.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.466+0000 7f1a54360640 1 --2- 172.21.15.150:0/3679609397 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c06fca0 0x7f1a4c06e090 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:19.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.466+0000 7f1a54360640 1 -- 172.21.15.150:0/3679609397 >> 172.21.15.150:0/3679609397 conn(0x7f1a4c0c5ae0 msgr2=0x7f1a4c073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:19.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.466+0000 7f1a54360640 1 -- 172.21.15.150:0/3679609397 shutdown_connections 2024-09-17T05:18:19.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.466+0000 7f1a54360640 1 -- 172.21.15.150:0/3679609397 wait complete. 2024-09-17T05:18:19.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.466+0000 7f1a54360640 1 Processor -- start 2024-09-17T05:18:19.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.466+0000 7f1a54360640 1 -- start start 2024-09-17T05:18:19.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a54360640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a4c06e5d0 0x7f1a4c0c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:19.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a54360640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c06fca0 0x7f1a4c0ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:19.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a54360640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4c0745c0 0x7f1a4c1ee510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:19.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a54360640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c070610 con 0x7f1a4c06fca0 2024-09-17T05:18:19.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a54360640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c070790 con 0x7f1a4c06e5d0 2024-09-17T05:18:19.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a54360640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1a4c0ab140 con 0x7f1a4c0745c0 2024-09-17T05:18:19.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a518fb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c06fca0 0x7f1a4c0ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:19.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a518fb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c06fca0 0x7f1a4c0ab9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46798/0 (socket says 172.21.15.150:46798) 2024-09-17T05:18:19.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a518fb640 1 -- 172.21.15.150:0/3613827739 learned_addr learned my addr 172.21.15.150:0/3613827739 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a528fd640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4c0745c0 0x7f1a4c1ee510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a520fc640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a4c06e5d0 0x7f1a4c0c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a528fd640 1 -- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a4c06e5d0 msgr2=0x7f1a4c0c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a528fd640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a4c06e5d0 0x7f1a4c0c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a528fd640 1 -- 172.21.15.150:0/3613827739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c06fca0 msgr2=0x7f1a4c0ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a528fd640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c06fca0 0x7f1a4c0ab9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a528fd640 1 -- 172.21.15.150:0/3613827739 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4c142110 con 0x7f1a4c0745c0 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a518fb640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c06fca0 0x7f1a4c0ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a528fd640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4c0745c0 0x7f1a4c1ee510 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400e9c0 tx=0x7f1a4400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a42ffd640 1 -- 172.21.15.150:0/3613827739 <== mon.2 v2:172.21.15.176:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4400cd80 con 0x7f1a4c0745c0 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a42ffd640 1 -- 172.21.15.150:0/3613827739 <== mon.2 v2:172.21.15.176:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a44004b30 con 0x7f1a4c0745c0 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a42ffd640 1 -- 172.21.15.150:0/3613827739 <== mon.2 v2:172.21.15.176:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44005020 con 0x7f1a4c0745c0 2024-09-17T05:18:19.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a54360640 1 -- 172.21.15.150:0/3613827739 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a4c142ef0 con 0x7f1a4c0745c0 2024-09-17T05:18:19.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.470+0000 7f1a54360640 1 -- 172.21.15.150:0/3613827739 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a4c141c60 con 0x7f1a4c0745c0 2024-09-17T05:18:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.482+0000 7f1a42ffd640 1 -- 172.21.15.150:0/3613827739 <== mon.2 v2:172.21.15.176:3301/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1a440051c0 con 0x7f1a4c0745c0 2024-09-17T05:18:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.482+0000 7f1a42ffd640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1a30073230 0x7f1a30075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.482+0000 7f1a42ffd640 1 -- 172.21.15.150:0/3613827739 <== mon.2 v2:172.21.15.176:3301/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f1a44095ec0 con 0x7f1a4c0745c0 2024-09-17T05:18:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.482+0000 7f1a54360640 1 -- 172.21.15.150:0/3613827739 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a4c06e090 con 0x7f1a4c0745c0 2024-09-17T05:18:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.486+0000 7f1a520fc640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1a30073230 0x7f1a30075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.486+0000 7f1a520fc640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1a30073230 0x7f1a30075740 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1a48007fd0 tx=0x7f1a48007ea0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:19.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.490+0000 7f1a42ffd640 1 -- 172.21.15.150:0/3613827739 <== mon.2 v2:172.21.15.176:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1a4405f7c0 con 0x7f1a4c0745c0 2024-09-17T05:18:19.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.782+0000 7f1a54360640 1 -- 172.21.15.150:0/3613827739 --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a4c0c7cf0 con 0x7f1a4c0745c0 2024-09-17T05:18:19.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.786+0000 7f1a42ffd640 1 -- 172.21.15.150:0/3613827739 <== mon.2 v2:172.21.15.176:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7f1a440641f0 con 0x7f1a4c0745c0 2024-09-17T05:18:19.788 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:19.788 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:19.788 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:19.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.790+0000 7f1a54360640 1 -- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1a30073230 msgr2=0x7f1a30075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:19.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.790+0000 7f1a54360640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1a30073230 0x7f1a30075740 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f1a48007fd0 tx=0x7f1a48007ea0 comp rx=0 tx=0).stop 2024-09-17T05:18:19.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.790+0000 7f1a54360640 1 -- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4c0745c0 msgr2=0x7f1a4c1ee510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:19.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.790+0000 7f1a54360640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4c0745c0 0x7f1a4c1ee510 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400e9c0 tx=0x7f1a4400ee80 comp rx=0 tx=0).stop 2024-09-17T05:18:19.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.794+0000 7f1a54360640 1 -- 172.21.15.150:0/3613827739 shutdown_connections 2024-09-17T05:18:19.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.794+0000 7f1a54360640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1a30073230 0x7f1a30075740 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:19.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.794+0000 7f1a54360640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1a4c0745c0 0x7f1a4c1ee510 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:19.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.794+0000 7f1a54360640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4c06fca0 0x7f1a4c0ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:19.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.794+0000 7f1a54360640 1 --2- 172.21.15.150:0/3613827739 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1a4c06e5d0 0x7f1a4c0c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:19.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.794+0000 7f1a54360640 1 -- 172.21.15.150:0/3613827739 >> 172.21.15.150:0/3613827739 conn(0x7f1a4c0c5ae0 msgr2=0x7f1a4c14ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:19.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.794+0000 7f1a54360640 1 -- 172.21.15.150:0/3613827739 shutdown_connections 2024-09-17T05:18:19.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:19.794+0000 7f1a54360640 1 -- 172.21.15.150:0/3613827739 wait complete. 2024-09-17T05:18:19.818 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:18:20.716 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:20.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.886+0000 7f7da6a6e640 1 -- 172.21.15.150:0/4058051818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da0157c00 msgr2=0x7f7da0158010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:20.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.886+0000 7f7da6a6e640 1 --2- 172.21.15.150:0/4058051818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da0157c00 0x7f7da0158010 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7da00a9a30 tx=0x7f7d9001c690 comp rx=0 tx=0).stop 2024-09-17T05:18:20.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.886+0000 7f7da6a6e640 1 -- 172.21.15.150:0/4058051818 shutdown_connections 2024-09-17T05:18:20.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.886+0000 7f7da6a6e640 1 --2- 172.21.15.150:0/4058051818 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da0157c00 0x7f7da0158010 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:20.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.886+0000 7f7da6a6e640 1 --2- 172.21.15.150:0/4058051818 >> v2:172.21.15.150:3300/0 conn(0x7f7da0071670 0x7f7da00c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:20.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.886+0000 7f7da6a6e640 1 -- 172.21.15.150:0/4058051818 >> 172.21.15.150:0/4058051818 conn(0x7f7da00ca460 msgr2=0x7f7da00c68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:20.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.886+0000 7f7da6a6e640 1 -- 172.21.15.150:0/4058051818 shutdown_connections 2024-09-17T05:18:20.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.886+0000 7f7da6a6e640 1 -- 172.21.15.150:0/4058051818 wait complete. 2024-09-17T05:18:20.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da6a6e640 1 Processor -- start 2024-09-17T05:18:20.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da6a6e640 1 -- start start 2024-09-17T05:18:20.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da6a6e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da0071670 0x7f7da0144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:20.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da6a6e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da014e490 0x7f7da01deb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:20.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da6a6e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da0157c00 0x7f7da01edd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:20.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da6a6e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7da00c62d0 con 0x7f7da0071670 2024-09-17T05:18:20.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da6a6e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7da00c51f0 con 0x7f7da014e490 2024-09-17T05:18:20.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da6a6e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7da00c65c0 con 0x7f7da0157c00 2024-09-17T05:18:20.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da480a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da0071670 0x7f7da0144e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:20.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da480a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da0071670 0x7f7da0144e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46808/0 (socket says 172.21.15.150:46808) 2024-09-17T05:18:20.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da480a640 1 -- 172.21.15.150:0/862448876 learned_addr learned my addr 172.21.15.150:0/862448876 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:20.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da480a640 1 -- 172.21.15.150:0/862448876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da0157c00 msgr2=0x7f7da01edd70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:18:20.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da480a640 1 --2- 172.21.15.150:0/862448876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da0157c00 0x7f7da01edd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:20.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da480a640 1 -- 172.21.15.150:0/862448876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da014e490 msgr2=0x7f7da01deb20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:18:20.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da480a640 1 --2- 172.21.15.150:0/862448876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da014e490 0x7f7da01deb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:20.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da480a640 1 -- 172.21.15.150:0/862448876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da01eb3b0 con 0x7f7da0071670 2024-09-17T05:18:20.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da480a640 1 --2- 172.21.15.150:0/862448876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da0071670 0x7f7da0144e00 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f7d9800ed00 tx=0x7f7d9800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:20.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7d9d7fa640 1 -- 172.21.15.150:0/862448876 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98003b80 con 0x7f7da0071670 2024-09-17T05:18:20.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7d9d7fa640 1 -- 172.21.15.150:0/862448876 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d98003d20 con 0x7f7da0071670 2024-09-17T05:18:20.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da6a6e640 1 -- 172.21.15.150:0/862448876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da01eb690 con 0x7f7da0071670 2024-09-17T05:18:20.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7d9d7fa640 1 -- 172.21.15.150:0/862448876 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98004280 con 0x7f7da0071670 2024-09-17T05:18:20.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.890+0000 7f7da6a6e640 1 -- 172.21.15.150:0/862448876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da01de320 con 0x7f7da0071670 2024-09-17T05:18:20.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.894+0000 7f7da6a6e640 1 -- 172.21.15.150:0/862448876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d840052a0 con 0x7f7da0071670 2024-09-17T05:18:20.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.894+0000 7f7d9d7fa640 1 -- 172.21.15.150:0/862448876 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7d98004420 con 0x7f7da0071670 2024-09-17T05:18:20.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.894+0000 7f7d9d7fa640 1 --2- 172.21.15.150:0/862448876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d80073230 0x7f7d80075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:20.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.894+0000 7f7d9d7fa640 1 -- 172.21.15.150:0/862448876 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f7d98095ee0 con 0x7f7da0071670 2024-09-17T05:18:20.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.894+0000 7f7d9ffff640 1 --2- 172.21.15.150:0/862448876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d80073230 0x7f7d80075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:20.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.898+0000 7f7d9ffff640 1 --2- 172.21.15.150:0/862448876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d80073230 0x7f7d80075740 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7da01e0c70 tx=0x7f7d8800aa00 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:20.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:20.906+0000 7f7d9d7fa640 1 -- 172.21.15.150:0/862448876 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7d9805fe80 con 0x7f7da0071670 2024-09-17T05:18:21.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.210+0000 7f7da6a6e640 1 -- 172.21.15.150:0/862448876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d84005590 con 0x7f7da0071670 2024-09-17T05:18:21.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.210+0000 7f7d9d7fa640 1 -- 172.21.15.150:0/862448876 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7f7d9805f780 con 0x7f7da0071670 2024-09-17T05:18:21.213 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:21.213 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:21.214 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:21.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.214+0000 7f7d96ffd640 1 -- 172.21.15.150:0/862448876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d80073230 msgr2=0x7f7d80075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:21.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.214+0000 7f7d96ffd640 1 --2- 172.21.15.150:0/862448876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d80073230 0x7f7d80075740 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7da01e0c70 tx=0x7f7d8800aa00 comp rx=0 tx=0).stop 2024-09-17T05:18:21.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.214+0000 7f7d96ffd640 1 -- 172.21.15.150:0/862448876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da0071670 msgr2=0x7f7da0144e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:21.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.214+0000 7f7d96ffd640 1 --2- 172.21.15.150:0/862448876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da0071670 0x7f7da0144e00 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f7d9800ed00 tx=0x7f7d9800c6a0 comp rx=0 tx=0).stop 2024-09-17T05:18:21.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.214+0000 7f7d96ffd640 1 -- 172.21.15.150:0/862448876 shutdown_connections 2024-09-17T05:18:21.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.214+0000 7f7d96ffd640 1 --2- 172.21.15.150:0/862448876 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d80073230 0x7f7d80075740 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:21.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.214+0000 7f7d96ffd640 1 --2- 172.21.15.150:0/862448876 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7da014e490 0x7f7da01deb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:21.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.214+0000 7f7d96ffd640 1 --2- 172.21.15.150:0/862448876 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da0157c00 0x7f7da01edd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:21.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.214+0000 7f7d96ffd640 1 --2- 172.21.15.150:0/862448876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da0071670 0x7f7da0144e00 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:21.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.214+0000 7f7d96ffd640 1 -- 172.21.15.150:0/862448876 >> 172.21.15.150:0/862448876 conn(0x7f7da00ca460 msgr2=0x7f7da014d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:21.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.214+0000 7f7d96ffd640 1 -- 172.21.15.150:0/862448876 shutdown_connections 2024-09-17T05:18:21.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.214+0000 7f7d96ffd640 1 -- 172.21.15.150:0/862448876 wait complete. 2024-09-17T05:18:21.241 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:18:21.819 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:21.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 -- 172.21.15.150:0/2650464872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae40c7cd0 msgr2=0x7f9ae40c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:21.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 --2- 172.21.15.150:0/2650464872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae40c7cd0 0x7f9ae40c80e0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009c80 tx=0x7f9adc01ccd0 comp rx=0 tx=0).stop 2024-09-17T05:18:21.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 -- 172.21.15.150:0/2650464872 shutdown_connections 2024-09-17T05:18:21.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 --2- 172.21.15.150:0/2650464872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae40c7cd0 0x7f9ae40c80e0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:21.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 --2- 172.21.15.150:0/2650464872 >> v2:172.21.15.176:3300/0 conn(0x7f9ae4072ea0 0x7f9ae40732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:21.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 -- 172.21.15.150:0/2650464872 >> 172.21.15.150:0/2650464872 conn(0x7f9ae40abb00 msgr2=0x7f9ae40abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:21.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 -- 172.21.15.150:0/2650464872 shutdown_connections 2024-09-17T05:18:21.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 -- 172.21.15.150:0/2650464872 wait complete. 2024-09-17T05:18:21.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 Processor -- start 2024-09-17T05:18:21.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 -- start start 2024-09-17T05:18:22.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4072ea0 0x7f9ae406df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:22.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ae40c7cd0 0x7f9ae40c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:22.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae40c97a0 0x7f9ae406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:22.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae4161790 con 0x7f9ae4072ea0 2024-09-17T05:18:22.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae40c4df0 con 0x7f9ae40c7cd0 2024-09-17T05:18:22.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9ae4161910 con 0x7f9ae40c97a0 2024-09-17T05:18:22.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae8ff5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4072ea0 0x7f9ae406df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:22.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae8ff5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4072ea0 0x7f9ae406df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46822/0 (socket says 172.21.15.150:46822) 2024-09-17T05:18:22.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae8ff5640 1 -- 172.21.15.150:0/2282703958 learned_addr learned my addr 172.21.15.150:0/2282703958 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:22.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae3fff640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ae40c7cd0 0x7f9ae40c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:22.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae97f6640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae40c97a0 0x7f9ae406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:22.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae8ff5640 1 -- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae40c97a0 msgr2=0x7f9ae406ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:22.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae8ff5640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae40c97a0 0x7f9ae406ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:22.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae8ff5640 1 -- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ae40c7cd0 msgr2=0x7f9ae40c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:22.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae8ff5640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ae40c7cd0 0x7f9ae40c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:22.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae8ff5640 1 -- 172.21.15.150:0/2282703958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae4152a50 con 0x7f9ae4072ea0 2024-09-17T05:18:22.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae97f6640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae40c97a0 0x7f9ae406ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:18:22.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae8ff5640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4072ea0 0x7f9ae406df60 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400c3c0 tx=0x7f9ad400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:22.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae17fa640 1 -- 172.21.15.150:0/2282703958 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4017070 con 0x7f9ae4072ea0 2024-09-17T05:18:22.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae17fa640 1 -- 172.21.15.150:0/2282703958 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ad40053c0 con 0x7f9ae4072ea0 2024-09-17T05:18:22.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 -- 172.21.15.150:0/2282703958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae414f7f0 con 0x7f9ae4072ea0 2024-09-17T05:18:22.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9ae17fa640 1 -- 172.21.15.150:0/2282703958 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4003860 con 0x7f9ae4072ea0 2024-09-17T05:18:22.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:21.998+0000 7f9aeb259640 1 -- 172.21.15.150:0/2282703958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae414e350 con 0x7f9ae4072ea0 2024-09-17T05:18:22.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.002+0000 7f9aeb259640 1 -- 172.21.15.150:0/2282703958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ae40732b0 con 0x7f9ae4072ea0 2024-09-17T05:18:22.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.002+0000 7f9ae17fa640 1 -- 172.21.15.150:0/2282703958 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9ad401a070 con 0x7f9ae4072ea0 2024-09-17T05:18:22.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.002+0000 7f9ae17fa640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ad0073200 0x7f9ad0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:22.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.002+0000 7f9ae17fa640 1 -- 172.21.15.150:0/2282703958 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f9ad4094600 con 0x7f9ae4072ea0 2024-09-17T05:18:22.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.002+0000 7f9ae3fff640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ad0073200 0x7f9ad0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:22.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.002+0000 7f9ae3fff640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ad0073200 0x7f9ad0075710 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9ae415e300 tx=0x7f9ad800b390 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:22.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.006+0000 7f9ae17fa640 1 -- 172.21.15.150:0/2282703958 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9ad4021080 con 0x7f9ae4072ea0 2024-09-17T05:18:22.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.314+0000 7f9aeb259640 1 -- 172.21.15.150:0/2282703958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ae40c40f0 con 0x7f9ae4072ea0 2024-09-17T05:18:22.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.318+0000 7f9ae17fa640 1 -- 172.21.15.150:0/2282703958 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7f9ad405e0f0 con 0x7f9ae4072ea0 2024-09-17T05:18:22.319 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:22.319 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:22.320 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:22.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.322+0000 7f9aeb259640 1 -- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ad0073200 msgr2=0x7f9ad0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:22.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.322+0000 7f9aeb259640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ad0073200 0x7f9ad0075710 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9ae415e300 tx=0x7f9ad800b390 comp rx=0 tx=0).stop 2024-09-17T05:18:22.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.322+0000 7f9aeb259640 1 -- 172.21.15.150:0/2282703958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4072ea0 msgr2=0x7f9ae406df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:22.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.322+0000 7f9aeb259640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4072ea0 0x7f9ae406df60 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400c3c0 tx=0x7f9ad400c880 comp rx=0 tx=0).stop 2024-09-17T05:18:22.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.322+0000 7f9aeb259640 1 -- 172.21.15.150:0/2282703958 shutdown_connections 2024-09-17T05:18:22.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.322+0000 7f9aeb259640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ad0073200 0x7f9ad0075710 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:22.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.322+0000 7f9aeb259640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ae40c7cd0 0x7f9ae40c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:22.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.322+0000 7f9aeb259640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ae40c97a0 0x7f9ae406ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:22.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.322+0000 7f9aeb259640 1 --2- 172.21.15.150:0/2282703958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ae4072ea0 0x7f9ae406df60 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:22.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.322+0000 7f9aeb259640 1 -- 172.21.15.150:0/2282703958 >> 172.21.15.150:0/2282703958 conn(0x7f9ae40abb00 msgr2=0x7f9ae415e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:22.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.322+0000 7f9aeb259640 1 -- 172.21.15.150:0/2282703958 shutdown_connections 2024-09-17T05:18:22.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:22.322+0000 7f9aeb259640 1 -- 172.21.15.150:0/2282703958 wait complete. 2024-09-17T05:18:22.349 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:18:23.242 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:23.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.410+0000 7ffba5973640 1 -- 172.21.15.150:0/103662930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb8c0040c0 msgr2=0x7ffb8c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:23.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.410+0000 7ffba5973640 1 --2- 172.21.15.150:0/103662930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb8c0040c0 0x7ffb8c004810 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7ffb8800c310 tx=0x7ffb8801cf60 comp rx=0 tx=0).stop 2024-09-17T05:18:23.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.410+0000 7ffb9f7fe640 1 -- 172.21.15.150:0/103662930 reap_dead start 2024-09-17T05:18:23.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.410+0000 7ffba5973640 1 -- 172.21.15.150:0/103662930 shutdown_connections 2024-09-17T05:18:23.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.410+0000 7ffba5973640 1 -- 172.21.15.150:0/103662930 >> 172.21.15.150:0/103662930 conn(0x7ffba00c4a00 msgr2=0x7ffba00c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:23.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.410+0000 7ffba5973640 1 -- 172.21.15.150:0/103662930 shutdown_connections 2024-09-17T05:18:23.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.410+0000 7ffba5973640 1 -- 172.21.15.150:0/103662930 wait complete. 2024-09-17T05:18:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.410+0000 7ffba5973640 1 Processor -- start 2024-09-17T05:18:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.410+0000 7ffba5973640 1 -- start start 2024-09-17T05:18:23.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffba5973640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb8c0040c0 0x7ffba014d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:23.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffba5973640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba0155c00 0x7ffba014aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:23.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffba5973640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffba014b000 0x7ffba01f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:23.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffba5973640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffba00ad010 con 0x7ffba0155c00 2024-09-17T05:18:23.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffba5973640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffba00adef0 con 0x7ffba014b000 2024-09-17T05:18:23.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffba5973640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffba00ad190 con 0x7ffb8c0040c0 2024-09-17T05:18:23.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba0155c00 0x7ffba014aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:23.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba0155c00 0x7ffba014aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46832/0 (socket says 172.21.15.150:46832) 2024-09-17T05:18:23.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9e7fc640 1 -- 172.21.15.150:0/1861279600 learned_addr learned my addr 172.21.15.150:0/1861279600 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:23.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9effd640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb8c0040c0 0x7ffba014d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:23.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9f7fe640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffba014b000 0x7ffba01f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:23.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9e7fc640 1 -- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb8c0040c0 msgr2=0x7ffba014d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:23.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9e7fc640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb8c0040c0 0x7ffba014d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:23.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9e7fc640 1 -- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffba014b000 msgr2=0x7ffba01f3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9e7fc640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffba014b000 0x7ffba01f3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9e7fc640 1 -- 172.21.15.150:0/1861279600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba0156f20 con 0x7ffba0155c00 2024-09-17T05:18:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9f7fe640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffba014b000 0x7ffba01f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:18:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9effd640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb8c0040c0 0x7ffba014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:18:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb9e7fc640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba0155c00 0x7ffba014aac0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7ffb8801b660 tx=0x7ffb88001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb97fff640 1 -- 172.21.15.150:0/1861279600 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb880014c0 con 0x7ffba0155c00 2024-09-17T05:18:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb97fff640 1 -- 172.21.15.150:0/1861279600 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb88001660 con 0x7ffba0155c00 2024-09-17T05:18:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffba5973640 1 -- 172.21.15.150:0/1861279600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffba01548e0 con 0x7ffba0155c00 2024-09-17T05:18:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb97fff640 1 -- 172.21.15.150:0/1861279600 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb88006330 con 0x7ffba0155c00 2024-09-17T05:18:23.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffba5973640 1 -- 172.21.15.150:0/1861279600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffba01ed190 con 0x7ffba0155c00 2024-09-17T05:18:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffba5973640 1 -- 172.21.15.150:0/1861279600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffba00ad720 con 0x7ffba0155c00 2024-09-17T05:18:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb97fff640 1 -- 172.21.15.150:0/1861279600 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ffb880b7030 con 0x7ffba0155c00 2024-09-17T05:18:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb97fff640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffb80073200 0x7ffb80075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.414+0000 7ffb97fff640 1 -- 172.21.15.150:0/1861279600 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7ffb8812edf0 con 0x7ffba0155c00 2024-09-17T05:18:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.418+0000 7ffb9effd640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffb80073200 0x7ffb80075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:23.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.418+0000 7ffb9effd640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffb80073200 0x7ffb80075710 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ffb98000f90 tx=0x7ffb98006f90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:23.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.418+0000 7ffb97fff640 1 -- 172.21.15.150:0/1861279600 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ffb88016d10 con 0x7ffba0155c00 2024-09-17T05:18:23.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.614+0000 7ffba5973640 1 -- 172.21.15.150:0/1861279600 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffba00c60f0 con 0x7ffba0155c00 2024-09-17T05:18:23.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.614+0000 7ffb97fff640 1 -- 172.21.15.150:0/1861279600 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7ffb880f8ea0 con 0x7ffba0155c00 2024-09-17T05:18:23.611 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:23.611 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:23.611 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.618+0000 7ffba5973640 1 -- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffb80073200 msgr2=0x7ffb80075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.618+0000 7ffba5973640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffb80073200 0x7ffb80075710 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ffb98000f90 tx=0x7ffb98006f90 comp rx=0 tx=0).stop 2024-09-17T05:18:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.618+0000 7ffba5973640 1 -- 172.21.15.150:0/1861279600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba0155c00 msgr2=0x7ffba014aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.618+0000 7ffba5973640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba0155c00 0x7ffba014aac0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7ffb8801b660 tx=0x7ffb88001d00 comp rx=0 tx=0).stop 2024-09-17T05:18:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.618+0000 7ffba5973640 1 -- 172.21.15.150:0/1861279600 shutdown_connections 2024-09-17T05:18:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.618+0000 7ffba5973640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffb80073200 0x7ffb80075710 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.618+0000 7ffba5973640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffba014b000 0x7ffba01f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.618+0000 7ffba5973640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba0155c00 0x7ffba014aac0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.618+0000 7ffba5973640 1 --2- 172.21.15.150:0/1861279600 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffb8c0040c0 0x7ffba014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.618+0000 7ffba5973640 1 -- 172.21.15.150:0/1861279600 >> 172.21.15.150:0/1861279600 conn(0x7ffba00c4a00 msgr2=0x7ffba0071260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.618+0000 7ffba5973640 1 -- 172.21.15.150:0/1861279600 shutdown_connections 2024-09-17T05:18:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:23.618+0000 7ffba5973640 1 -- 172.21.15.150:0/1861279600 wait complete. 2024-09-17T05:18:23.631 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:18:24.349 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:24.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.518+0000 7f3c83c26640 1 -- 172.21.15.150:0/1579570567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c64005260 msgr2=0x7f3c6400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:24.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.518+0000 7f3c83c26640 1 --2- 172.21.15.150:0/1579570567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c64005260 0x7f3c6400b9d0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009b40 tx=0x7f3c7401d290 comp rx=0 tx=0).stop 2024-09-17T05:18:24.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.518+0000 7f3c83c26640 1 -- 172.21.15.150:0/1579570567 shutdown_connections 2024-09-17T05:18:24.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.518+0000 7f3c83c26640 1 --2- 172.21.15.150:0/1579570567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c64005260 0x7f3c6400b9d0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:24.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.518+0000 7f3c83c26640 1 -- 172.21.15.150:0/1579570567 >> 172.21.15.150:0/1579570567 conn(0x7f3c7c0c4a00 msgr2=0x7f3c7c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:24.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.518+0000 7f3c83c26640 1 -- 172.21.15.150:0/1579570567 shutdown_connections 2024-09-17T05:18:24.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.518+0000 7f3c83c26640 1 -- 172.21.15.150:0/1579570567 wait complete. 2024-09-17T05:18:24.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.518+0000 7f3c83c26640 1 Processor -- start 2024-09-17T05:18:24.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.518+0000 7f3c83c26640 1 -- start start 2024-09-17T05:18:24.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c83c26640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c64005260 0x7f3c7c0ad3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:24.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c83c26640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c0abb50 0x7f3c7c0ac700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:24.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c83c26640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c1ecb00 0x7f3c7c1ecf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:24.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c83c26640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c070da0 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c83c26640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c0c1fb0 con 0x7f3c7c0abb50 2024-09-17T05:18:24.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c83c26640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3c7c0c6690 con 0x7f3c64005260 2024-09-17T05:18:24.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c821c3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c1ecb00 0x7f3c7c1ecf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:24.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c821c3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c1ecb00 0x7f3c7c1ecf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46850/0 (socket says 172.21.15.150:46850) 2024-09-17T05:18:24.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c821c3640 1 -- 172.21.15.150:0/3118620847 learned_addr learned my addr 172.21.15.150:0/3118620847 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:24.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c811c1640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c0abb50 0x7f3c7c0ac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:24.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c819c2640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c64005260 0x7f3c7c0ad3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:24.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c821c3640 1 -- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c64005260 msgr2=0x7f3c7c0ad3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:24.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c821c3640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c64005260 0x7f3c7c0ad3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:24.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c821c3640 1 -- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c0abb50 msgr2=0x7f3c7c0ac700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:24.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c821c3640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c0abb50 0x7f3c7c0ac700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:24.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c821c3640 1 -- 172.21.15.150:0/3118620847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c7c1ed670 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c811c1640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c0abb50 0x7f3c7c0ac700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:18:24.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c821c3640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c1ecb00 0x7f3c7c1ecf10 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f3c74000c10 tx=0x7f3c74005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:24.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c7a7fc640 1 -- 172.21.15.150:0/3118620847 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74004a20 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c7a7fc640 1 -- 172.21.15.150:0/3118620847 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c740bc070 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c83c26640 1 -- 172.21.15.150:0/3118620847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c7c0aabd0 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c7a7fc640 1 -- 172.21.15.150:0/3118620847 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74004f30 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c83c26640 1 -- 172.21.15.150:0/3118620847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c7c0a7860 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c83c26640 1 -- 172.21.15.150:0/3118620847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c7c0ab0f0 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.522+0000 7f3c7a7fc640 1 -- 172.21.15.150:0/3118620847 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3c740b5040 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.526+0000 7f3c7a7fc640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c68073200 0x7f3c68075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:24.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.526+0000 7f3c7a7fc640 1 -- 172.21.15.150:0/3118620847 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f3c74130560 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.526+0000 7f3c819c2640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c68073200 0x7f3c68075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:24.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.526+0000 7f3c819c2640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c68073200 0x7f3c68075710 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c001c30 tx=0x7f3c6c0014a0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:24.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.530+0000 7f3c7a7fc640 1 -- 172.21.15.150:0/3118620847 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3c7c0ab0f0 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.734+0000 7f3c83c26640 1 -- 172.21.15.150:0/3118620847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c7c0c6bf0 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.734+0000 7f3c7a7fc640 1 -- 172.21.15.150:0/3118620847 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7f3c74016fa0 con 0x7f3c7c1ecb00 2024-09-17T05:18:24.732 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:24.733 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:24.733 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:24.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.738+0000 7f3c83c26640 1 -- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c68073200 msgr2=0x7f3c68075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:24.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.738+0000 7f3c83c26640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c68073200 0x7f3c68075710 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c001c30 tx=0x7f3c6c0014a0 comp rx=0 tx=0).stop 2024-09-17T05:18:24.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.738+0000 7f3c83c26640 1 -- 172.21.15.150:0/3118620847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c1ecb00 msgr2=0x7f3c7c1ecf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:24.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.738+0000 7f3c83c26640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c1ecb00 0x7f3c7c1ecf10 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f3c74000c10 tx=0x7f3c74005dd0 comp rx=0 tx=0).stop 2024-09-17T05:18:24.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.738+0000 7f3c83c26640 1 -- 172.21.15.150:0/3118620847 shutdown_connections 2024-09-17T05:18:24.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.738+0000 7f3c83c26640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3c68073200 0x7f3c68075710 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:24.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.738+0000 7f3c83c26640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c7c0abb50 0x7f3c7c0ac700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:24.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.738+0000 7f3c83c26640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c7c1ecb00 0x7f3c7c1ecf10 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:24.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.738+0000 7f3c83c26640 1 --2- 172.21.15.150:0/3118620847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3c64005260 0x7f3c7c0ad3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:24.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.738+0000 7f3c83c26640 1 -- 172.21.15.150:0/3118620847 >> 172.21.15.150:0/3118620847 conn(0x7f3c7c0c4a00 msgr2=0x7f3c7c15dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:24.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.738+0000 7f3c83c26640 1 -- 172.21.15.150:0/3118620847 shutdown_connections 2024-09-17T05:18:24.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:24.738+0000 7f3c83c26640 1 -- 172.21.15.150:0/3118620847 wait complete. 2024-09-17T05:18:24.754 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:18:25.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:25.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 -- 172.21.15.150:0/2865431476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9130003a60 msgr2=0x7f9130003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:25.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 --2- 172.21.15.150:0/2865431476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9130003a60 0x7f9130003ed0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f9138001dc0 tx=0x7f913801cad0 comp rx=0 tx=0).stop 2024-09-17T05:18:25.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 -- 172.21.15.150:0/2865431476 shutdown_connections 2024-09-17T05:18:25.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 --2- 172.21.15.150:0/2865431476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9130003a60 0x7f9130003ed0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:25.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 -- 172.21.15.150:0/2865431476 >> 172.21.15.150:0/2865431476 conn(0x7f91480aadc0 msgr2=0x7f91480ab1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:25.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 -- 172.21.15.150:0/2865431476 shutdown_connections 2024-09-17T05:18:25.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 -- 172.21.15.150:0/2865431476 wait complete. 2024-09-17T05:18:25.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 Processor -- start 2024-09-17T05:18:25.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 -- start start 2024-09-17T05:18:25.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9130003a60 0x7f91480a9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:25.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9148153e00 0x7f91481f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:25.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91481f4330 0x7f9148147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:25.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9148158550 con 0x7f9130003a60 2024-09-17T05:18:25.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9148152420 con 0x7f9148153e00 2024-09-17T05:18:25.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914e442640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f91481586d0 con 0x7f91481f4330 2024-09-17T05:18:25.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f9147fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9130003a60 0x7f91480a9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:25.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f9147fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9130003a60 0x7f91480a9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46868/0 (socket says 172.21.15.150:46868) 2024-09-17T05:18:25.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f9147fff640 1 -- 172.21.15.150:0/2972017846 learned_addr learned my addr 172.21.15.150:0/2972017846 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:25.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.806+0000 7f914c9df640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91481f4330 0x7f9148147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f91477fe640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9148153e00 0x7f91481f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f9147fff640 1 -- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91481f4330 msgr2=0x7f9148147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f9147fff640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91481f4330 0x7f9148147f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f9147fff640 1 -- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9148153e00 msgr2=0x7f91481f3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f9147fff640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9148153e00 0x7f91481f3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f9147fff640 1 -- 172.21.15.150:0/2972017846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91481525d0 con 0x7f9130003a60 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f91477fe640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9148153e00 0x7f91481f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f9147fff640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9130003a60 0x7f91480a9aa0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f913801b1d0 tx=0x7f9138003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f9144ff9640 1 -- 172.21.15.150:0/2972017846 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91380a5920 con 0x7f9130003a60 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f9144ff9640 1 -- 172.21.15.150:0/2972017846 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91380a5bb0 con 0x7f9130003a60 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f9144ff9640 1 -- 172.21.15.150:0/2972017846 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91380060d0 con 0x7f9130003a60 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f914e442640 1 -- 172.21.15.150:0/2972017846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91481f11a0 con 0x7f9130003a60 2024-09-17T05:18:25.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f914e442640 1 -- 172.21.15.150:0/2972017846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91481f0ec0 con 0x7f9130003a60 2024-09-17T05:18:25.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f914e442640 1 -- 172.21.15.150:0/2972017846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91481527a0 con 0x7f9130003a60 2024-09-17T05:18:25.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f9144ff9640 1 -- 172.21.15.150:0/2972017846 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9138006290 con 0x7f9130003a60 2024-09-17T05:18:25.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f9144ff9640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9134073200 0x7f9134075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:25.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f9144ff9640 1 -- 172.21.15.150:0/2972017846 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f913812f2c0 con 0x7f9130003a60 2024-09-17T05:18:25.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.810+0000 7f91477fe640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9134073200 0x7f9134075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:25.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.814+0000 7f91477fe640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9134073200 0x7f9134075710 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f913c005ec0 tx=0x7f913c0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:25.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:25.814+0000 7f9144ff9640 1 -- 172.21.15.150:0/2972017846 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9138016830 con 0x7f9130003a60 2024-09-17T05:18:26.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.026+0000 7f914e442640 1 -- 172.21.15.150:0/2972017846 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9148077180 con 0x7f9130003a60 2024-09-17T05:18:26.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f9144ff9640 1 -- 172.21.15.150:0/2972017846 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7f91380f94c0 con 0x7f9130003a60 2024-09-17T05:18:26.028 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:26.028 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:26.029 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:26.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f914e442640 1 -- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9134073200 msgr2=0x7f9134075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:26.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f914e442640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9134073200 0x7f9134075710 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f913c005ec0 tx=0x7f913c0091e0 comp rx=0 tx=0).stop 2024-09-17T05:18:26.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f914e442640 1 -- 172.21.15.150:0/2972017846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9130003a60 msgr2=0x7f91480a9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:26.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f914e442640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9130003a60 0x7f91480a9aa0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f913801b1d0 tx=0x7f9138003f60 comp rx=0 tx=0).stop 2024-09-17T05:18:26.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f914e442640 1 -- 172.21.15.150:0/2972017846 shutdown_connections 2024-09-17T05:18:26.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f914e442640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9134073200 0x7f9134075710 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:26.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f914e442640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9148153e00 0x7f91481f3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:26.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f914e442640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f91481f4330 0x7f9148147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:26.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f914e442640 1 --2- 172.21.15.150:0/2972017846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9130003a60 0x7f91480a9aa0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:26.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f914e442640 1 -- 172.21.15.150:0/2972017846 >> 172.21.15.150:0/2972017846 conn(0x7f91480aadc0 msgr2=0x7f91480c2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:26.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f914e442640 1 -- 172.21.15.150:0/2972017846 shutdown_connections 2024-09-17T05:18:26.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.030+0000 7f914e442640 1 -- 172.21.15.150:0/2972017846 wait complete. 2024-09-17T05:18:26.048 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:18:26.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:26.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.926+0000 7f820bb5e640 1 -- 172.21.15.150:0/1874256087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82040717a0 msgr2=0x7f8204074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:26.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.926+0000 7f820bb5e640 1 --2- 172.21.15.150:0/1874256087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82040717a0 0x7f8204074450 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f82040c25a0 tx=0x7f81fc01ccc0 comp rx=0 tx=0).stop 2024-09-17T05:18:26.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.926+0000 7f820bb5e640 1 -- 172.21.15.150:0/1874256087 shutdown_connections 2024-09-17T05:18:26.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.926+0000 7f820bb5e640 1 --2- 172.21.15.150:0/1874256087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82040717a0 0x7f8204074450 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:26.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.926+0000 7f820bb5e640 1 --2- 172.21.15.150:0/1874256087 >> v2:172.21.15.176:3300/0 conn(0x7f8204070e50 0x7f8204071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:26.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.926+0000 7f820bb5e640 1 --2- 172.21.15.150:0/1874256087 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8204073440 0x7f82040a7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:26.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.926+0000 7f820bb5e640 1 -- 172.21.15.150:0/1874256087 >> 172.21.15.150:0/1874256087 conn(0x7f82040c4a00 msgr2=0x7f82040c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:26.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.926+0000 7f820bb5e640 1 -- 172.21.15.150:0/1874256087 shutdown_connections 2024-09-17T05:18:26.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.926+0000 7f820bb5e640 1 -- 172.21.15.150:0/1874256087 wait complete. 2024-09-17T05:18:26.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.926+0000 7f820bb5e640 1 Processor -- start 2024-09-17T05:18:26.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.926+0000 7f820bb5e640 1 -- start start 2024-09-17T05:18:26.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f820bb5e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8204070e50 0x7f8204149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:26.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f820bb5e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82040717a0 0x7f820414c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:26.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f820bb5e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8204073440 0x7f82041ef9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:26.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f820bb5e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f82040ace00 con 0x7f8204070e50 2024-09-17T05:18:26.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f820bb5e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f82040adef0 con 0x7f82040717a0 2024-09-17T05:18:26.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f820bb5e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f82040acf80 con 0x7f8204073440 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f82098fa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8204070e50 0x7f8204149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f82098fa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8204070e50 0x7f8204149280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46878/0 (socket says 172.21.15.150:46878) 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f82098fa640 1 -- 172.21.15.150:0/4257228675 learned_addr learned my addr 172.21.15.150:0/4257228675 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f82090f9640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82040717a0 0x7f820414c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f820a0fb640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8204073440 0x7f82041ef9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f82098fa640 1 -- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8204073440 msgr2=0x7f82041ef9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f82098fa640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8204073440 0x7f82041ef9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f82098fa640 1 -- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82040717a0 msgr2=0x7f820414c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f82098fa640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82040717a0 0x7f820414c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f82098fa640 1 -- 172.21.15.150:0/4257228675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82041ed570 con 0x7f8204070e50 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f82098fa640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8204070e50 0x7f8204149280 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f820000c9a0 tx=0x7f820000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f81fa7fc640 1 -- 172.21.15.150:0/4257228675 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200004220 con 0x7f8204070e50 2024-09-17T05:18:26.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f81fa7fc640 1 -- 172.21.15.150:0/4257228675 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82000044b0 con 0x7f8204070e50 2024-09-17T05:18:26.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f81fa7fc640 1 -- 172.21.15.150:0/4257228675 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200007bb0 con 0x7f8204070e50 2024-09-17T05:18:26.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f820bb5e640 1 -- 172.21.15.150:0/4257228675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8204150860 con 0x7f8204070e50 2024-09-17T05:18:26.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f820bb5e640 1 -- 172.21.15.150:0/4257228675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82040c2890 con 0x7f8204070e50 2024-09-17T05:18:26.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f820bb5e640 1 -- 172.21.15.150:0/4257228675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8204074450 con 0x7f8204070e50 2024-09-17T05:18:26.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.930+0000 7f81fa7fc640 1 -- 172.21.15.150:0/4257228675 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8200003620 con 0x7f8204070e50 2024-09-17T05:18:26.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.934+0000 7f81fa7fc640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f81e8073230 0x7f81e8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:26.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.934+0000 7f81fa7fc640 1 -- 172.21.15.150:0/4257228675 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f82000955c0 con 0x7f8204070e50 2024-09-17T05:18:26.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.934+0000 7f82090f9640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f81e8073230 0x7f81e8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:26.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.934+0000 7f82090f9640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f81e8073230 0x7f81e8075740 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f81f4005fd0 tx=0x7f81f4005d10 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:26.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:26.938+0000 7f81fa7fc640 1 -- 172.21.15.150:0/4257228675 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f82041ed570 con 0x7f8204070e50 2024-09-17T05:18:27.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.138+0000 7f820bb5e640 1 -- 172.21.15.150:0/4257228675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8204150c40 con 0x7f8204070e50 2024-09-17T05:18:27.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f81fa7fc640 1 -- 172.21.15.150:0/4257228675 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7f8200014030 con 0x7f8204070e50 2024-09-17T05:18:27.138 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:27.138 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:27.139 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:27.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f820bb5e640 1 -- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f81e8073230 msgr2=0x7f81e8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:27.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f820bb5e640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f81e8073230 0x7f81e8075740 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f81f4005fd0 tx=0x7f81f4005d10 comp rx=0 tx=0).stop 2024-09-17T05:18:27.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f820bb5e640 1 -- 172.21.15.150:0/4257228675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8204070e50 msgr2=0x7f8204149280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:27.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f820bb5e640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8204070e50 0x7f8204149280 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f820000c9a0 tx=0x7f820000ce60 comp rx=0 tx=0).stop 2024-09-17T05:18:27.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f820bb5e640 1 -- 172.21.15.150:0/4257228675 shutdown_connections 2024-09-17T05:18:27.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f820bb5e640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f81e8073230 0x7f81e8075740 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:27.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f820bb5e640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f82040717a0 0x7f820414c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:27.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f820bb5e640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8204073440 0x7f82041ef9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:27.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f820bb5e640 1 --2- 172.21.15.150:0/4257228675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8204070e50 0x7f8204149280 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:27.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f820bb5e640 1 -- 172.21.15.150:0/4257228675 >> 172.21.15.150:0/4257228675 conn(0x7f82040c4a00 msgr2=0x7f820414d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:27.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f820bb5e640 1 -- 172.21.15.150:0/4257228675 shutdown_connections 2024-09-17T05:18:27.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:27.142+0000 7f820bb5e640 1 -- 172.21.15.150:0/4257228675 wait complete. 2024-09-17T05:18:27.158 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:18:28.049 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:28.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.218+0000 7f0a6ba81640 1 -- 172.21.15.150:0/3601406169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a50003a10 msgr2=0x7f0a50003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:28.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.218+0000 7f0a6ba81640 1 --2- 172.21.15.150:0/3601406169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a50003a10 0x7f0a50003e80 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400bbc0 tx=0x7f0a5401ce60 comp rx=0 tx=0).stop 2024-09-17T05:18:28.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.218+0000 7f0a6a01e640 1 -- 172.21.15.150:0/3601406169 reap_dead start 2024-09-17T05:18:28.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.218+0000 7f0a6ba81640 1 -- 172.21.15.150:0/3601406169 shutdown_connections 2024-09-17T05:18:28.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.218+0000 7f0a6ba81640 1 -- 172.21.15.150:0/3601406169 >> 172.21.15.150:0/3601406169 conn(0x7f0a640c4a00 msgr2=0x7f0a640c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:28.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.218+0000 7f0a6ba81640 1 -- 172.21.15.150:0/3601406169 shutdown_connections 2024-09-17T05:18:28.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.218+0000 7f0a6ba81640 1 -- 172.21.15.150:0/3601406169 wait complete. 2024-09-17T05:18:28.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6ba81640 1 Processor -- start 2024-09-17T05:18:28.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6ba81640 1 -- start start 2024-09-17T05:18:28.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6ba81640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a50003a10 0x7f0a6414f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:28.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6ba81640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a6414e6d0 0x7f0a641f1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:28.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6ba81640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a641f1790 0x7f0a64140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:28.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6ba81640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0a64070750 con 0x7f0a641f1790 2024-09-17T05:18:28.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6ba81640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0a640705d0 con 0x7f0a50003a10 2024-09-17T05:18:28.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6ba81640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0a640a75f0 con 0x7f0a6414e6d0 2024-09-17T05:18:28.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6a01e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a641f1790 0x7f0a64140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6a01e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a641f1790 0x7f0a64140e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46908/0 (socket says 172.21.15.150:46908) 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6a01e640 1 -- 172.21.15.150:0/2370716247 learned_addr learned my addr 172.21.15.150:0/2370716247 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6981d640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a50003a10 0x7f0a6414f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6901c640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a6414e6d0 0x7f0a641f1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6a01e640 1 -- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a6414e6d0 msgr2=0x7f0a641f1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6a01e640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a6414e6d0 0x7f0a641f1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6a01e640 1 -- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a50003a10 msgr2=0x7f0a6414f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6a01e640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a50003a10 0x7f0a6414f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6a01e640 1 -- 172.21.15.150:0/2370716247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6414e320 con 0x7f0a641f1790 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6a01e640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a641f1790 0x7f0a64140e70 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f0a50004790 tx=0x7f0a5c010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a5a7fc640 1 -- 172.21.15.150:0/2370716247 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c017070 con 0x7f0a641f1790 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a5a7fc640 1 -- 172.21.15.150:0/2370716247 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a5c015070 con 0x7f0a641f1790 2024-09-17T05:18:28.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6ba81640 1 -- 172.21.15.150:0/2370716247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a641f1cf0 con 0x7f0a641f1790 2024-09-17T05:18:28.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a5a7fc640 1 -- 172.21.15.150:0/2370716247 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c004d90 con 0x7f0a641f1790 2024-09-17T05:18:28.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6ba81640 1 -- 172.21.15.150:0/2370716247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a641eb690 con 0x7f0a641f1790 2024-09-17T05:18:28.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.222+0000 7f0a6ba81640 1 -- 172.21.15.150:0/2370716247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a6414ede0 con 0x7f0a641f1790 2024-09-17T05:18:28.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.226+0000 7f0a5a7fc640 1 -- 172.21.15.150:0/2370716247 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0a5c003980 con 0x7f0a641f1790 2024-09-17T05:18:28.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.226+0000 7f0a5a7fc640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a48073200 0x7f0a48075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:28.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.226+0000 7f0a5a7fc640 1 -- 172.21.15.150:0/2370716247 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f0a5c095650 con 0x7f0a641f1790 2024-09-17T05:18:28.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.226+0000 7f0a6981d640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a48073200 0x7f0a48075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:28.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.226+0000 7f0a6981d640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a48073200 0x7f0a48075710 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0a540af000 tx=0x7f0a54019c00 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:28.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.230+0000 7f0a5a7fc640 1 -- 172.21.15.150:0/2370716247 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0a5c05f710 con 0x7f0a641f1790 2024-09-17T05:18:28.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.438+0000 7f0a6ba81640 1 -- 172.21.15.150:0/2370716247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a6414e4f0 con 0x7f0a641f1790 2024-09-17T05:18:28.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.438+0000 7f0a5a7fc640 1 -- 172.21.15.150:0/2370716247 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7f0a6414e4f0 con 0x7f0a641f1790 2024-09-17T05:18:28.436 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:28.436 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:28.436 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:28.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.442+0000 7f0a6ba81640 1 -- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a48073200 msgr2=0x7f0a48075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:28.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.442+0000 7f0a6ba81640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a48073200 0x7f0a48075710 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0a540af000 tx=0x7f0a54019c00 comp rx=0 tx=0).stop 2024-09-17T05:18:28.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.442+0000 7f0a6ba81640 1 -- 172.21.15.150:0/2370716247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a641f1790 msgr2=0x7f0a64140e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:28.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.442+0000 7f0a6ba81640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a641f1790 0x7f0a64140e70 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f0a50004790 tx=0x7f0a5c010e90 comp rx=0 tx=0).stop 2024-09-17T05:18:28.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.442+0000 7f0a6ba81640 1 -- 172.21.15.150:0/2370716247 shutdown_connections 2024-09-17T05:18:28.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.442+0000 7f0a6ba81640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a48073200 0x7f0a48075710 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:28.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.442+0000 7f0a6ba81640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a641f1790 0x7f0a64140e70 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:28.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.442+0000 7f0a6ba81640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a6414e6d0 0x7f0a641f1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:28.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.442+0000 7f0a6ba81640 1 --2- 172.21.15.150:0/2370716247 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a50003a10 0x7f0a6414f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:28.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.442+0000 7f0a6ba81640 1 -- 172.21.15.150:0/2370716247 >> 172.21.15.150:0/2370716247 conn(0x7f0a640c4a00 msgr2=0x7f0a640c5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:28.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.442+0000 7f0a6ba81640 1 -- 172.21.15.150:0/2370716247 shutdown_connections 2024-09-17T05:18:28.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:28.442+0000 7f0a6ba81640 1 -- 172.21.15.150:0/2370716247 wait complete. 2024-09-17T05:18:28.458 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:18:29.159 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:29.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.330+0000 7fe57a889640 1 -- 172.21.15.150:0/270468038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe560004d50 msgr2=0x7fe560005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:29.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.330+0000 7fe57a889640 1 --2- 172.21.15.150:0/270468038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe560004d50 0x7fe560005160 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009eb0 tx=0x7fe56c01d5f0 comp rx=0 tx=0).stop 2024-09-17T05:18:29.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.330+0000 7fe578e26640 1 -- 172.21.15.150:0/270468038 reap_dead start 2024-09-17T05:18:29.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.330+0000 7fe57a889640 1 -- 172.21.15.150:0/270468038 shutdown_connections 2024-09-17T05:18:29.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.330+0000 7fe57a889640 1 -- 172.21.15.150:0/270468038 >> 172.21.15.150:0/270468038 conn(0x7fe5740c4a00 msgr2=0x7fe5740c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:29.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.330+0000 7fe57a889640 1 -- 172.21.15.150:0/270468038 shutdown_connections 2024-09-17T05:18:29.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.330+0000 7fe57a889640 1 -- 172.21.15.150:0/270468038 wait complete. 2024-09-17T05:18:29.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe57a889640 1 Processor -- start 2024-09-17T05:18:29.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe57a889640 1 -- start start 2024-09-17T05:18:29.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe57a889640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe57414d030 0x7fe57414d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:29.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe57a889640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5740ac690 0x7fe5740acaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:29.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe57a889640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5741de2a0 0x7fe5741de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:29.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe57a889640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe5740ade60 con 0x7fe57414d030 2024-09-17T05:18:29.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe57a889640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe5740adce0 con 0x7fe5740ac690 2024-09-17T05:18:29.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe57a889640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe574071260 con 0x7fe5741de2a0 2024-09-17T05:18:29.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe573fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe57414d030 0x7fe57414d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:29.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe573fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe57414d030 0x7fe57414d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48840/0 (socket says 172.21.15.150:48840) 2024-09-17T05:18:29.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe573fff640 1 -- 172.21.15.150:0/993890989 learned_addr learned my addr 172.21.15.150:0/993890989 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:29.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe5737fe640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5740ac690 0x7fe5740acaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:29.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe578e26640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5741de2a0 0x7fe5741de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:29.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe573fff640 1 -- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5741de2a0 msgr2=0x7fe5741de6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:29.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe573fff640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5741de2a0 0x7fe5741de6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:29.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe573fff640 1 -- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5740ac690 msgr2=0x7fe5740acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:29.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe573fff640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5740ac690 0x7fe5740acaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:29.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe573fff640 1 -- 172.21.15.150:0/993890989 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5741e9700 con 0x7fe57414d030 2024-09-17T05:18:29.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe578e26640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5741de2a0 0x7fe5741de6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:18:29.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe573fff640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe57414d030 0x7fe57414d440 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fe56400e360 tx=0x7fe56400e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:29.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe570ff9640 1 -- 172.21.15.150:0/993890989 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564012910 con 0x7fe57414d030 2024-09-17T05:18:29.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe570ff9640 1 -- 172.21.15.150:0/993890989 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe564012bc0 con 0x7fe57414d030 2024-09-17T05:18:29.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe57a889640 1 -- 172.21.15.150:0/993890989 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe57414d980 con 0x7fe57414d030 2024-09-17T05:18:29.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe570ff9640 1 -- 172.21.15.150:0/993890989 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56401a400 con 0x7fe57414d030 2024-09-17T05:18:29.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe57a889640 1 -- 172.21.15.150:0/993890989 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5741ddd80 con 0x7fe57414d030 2024-09-17T05:18:29.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.334+0000 7fe57a889640 1 -- 172.21.15.150:0/993890989 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe574154020 con 0x7fe57414d030 2024-09-17T05:18:29.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.338+0000 7fe570ff9640 1 -- 172.21.15.150:0/993890989 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe56400d030 con 0x7fe57414d030 2024-09-17T05:18:29.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.338+0000 7fe570ff9640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe554073200 0x7fe554075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:29.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.338+0000 7fe570ff9640 1 -- 172.21.15.150:0/993890989 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7fe564098920 con 0x7fe57414d030 2024-09-17T05:18:29.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.338+0000 7fe5737fe640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe554073200 0x7fe554075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:29.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.338+0000 7fe5737fe640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe554073200 0x7fe554075710 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe5740a9110 tx=0x7fe55c0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:29.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.342+0000 7fe570ff9640 1 -- 172.21.15.150:0/993890989 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe57414d980 con 0x7fe57414d030 2024-09-17T05:18:29.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.550+0000 7fe57a889640 1 -- 172.21.15.150:0/993890989 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5740c60f0 con 0x7fe57414d030 2024-09-17T05:18:29.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.550+0000 7fe570ff9640 1 -- 172.21.15.150:0/993890989 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7fe564016030 con 0x7fe57414d030 2024-09-17T05:18:29.548 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:29.548 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:29.548 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:29.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.554+0000 7fe57a889640 1 -- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe554073200 msgr2=0x7fe554075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:29.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.554+0000 7fe57a889640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe554073200 0x7fe554075710 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe5740a9110 tx=0x7fe55c0091e0 comp rx=0 tx=0).stop 2024-09-17T05:18:29.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.554+0000 7fe57a889640 1 -- 172.21.15.150:0/993890989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe57414d030 msgr2=0x7fe57414d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:29.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.554+0000 7fe57a889640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe57414d030 0x7fe57414d440 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fe56400e360 tx=0x7fe56400e820 comp rx=0 tx=0).stop 2024-09-17T05:18:29.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.554+0000 7fe57a889640 1 -- 172.21.15.150:0/993890989 shutdown_connections 2024-09-17T05:18:29.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.554+0000 7fe57a889640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe554073200 0x7fe554075710 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:29.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.554+0000 7fe57a889640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5740ac690 0x7fe5740acaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:29.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.554+0000 7fe57a889640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5741de2a0 0x7fe5741de6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:29.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.554+0000 7fe57a889640 1 --2- 172.21.15.150:0/993890989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe57414d030 0x7fe57414d440 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:29.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.554+0000 7fe57a889640 1 -- 172.21.15.150:0/993890989 >> 172.21.15.150:0/993890989 conn(0x7fe5740c4a00 msgr2=0x7fe57414fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:29.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.554+0000 7fe57a889640 1 -- 172.21.15.150:0/993890989 shutdown_connections 2024-09-17T05:18:29.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:29.554+0000 7fe57a889640 1 -- 172.21.15.150:0/993890989 wait complete. 2024-09-17T05:18:29.570 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-09-17T05:18:30.459 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:30.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.630+0000 7f710f73c640 1 -- 172.21.15.150:0/491947670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7108074450 msgr2=0x7f7108074860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:30.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.630+0000 7f710f73c640 1 --2- 172.21.15.150:0/491947670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7108074450 0x7f7108074860 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f7100009bc0 tx=0x7f710001cb10 comp rx=0 tx=0).stop 2024-09-17T05:18:30.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.630+0000 7f710f73c640 1 -- 172.21.15.150:0/491947670 shutdown_connections 2024-09-17T05:18:30.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.630+0000 7f710f73c640 1 --2- 172.21.15.150:0/491947670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7108074450 0x7f7108074860 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:30.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.630+0000 7f710f73c640 1 -- 172.21.15.150:0/491947670 >> 172.21.15.150:0/491947670 conn(0x7f71080c4a00 msgr2=0x7f71080c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:30.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.630+0000 7f710f73c640 1 -- 172.21.15.150:0/491947670 shutdown_connections 2024-09-17T05:18:30.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.630+0000 7f710f73c640 1 -- 172.21.15.150:0/491947670 wait complete. 2024-09-17T05:18:30.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.630+0000 7f710f73c640 1 Processor -- start 2024-09-17T05:18:30.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.630+0000 7f710f73c640 1 -- start start 2024-09-17T05:18:30.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710f73c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7108070e50 0x7f71080bf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:30.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710f73c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7108073440 0x7f71080bea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:30.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710f73c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7108074450 0x7f71080c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:30.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710f73c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f71080ad820 con 0x7f7108074450 2024-09-17T05:18:30.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710f73c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f71080adef0 con 0x7f7108073440 2024-09-17T05:18:30.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710f73c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f71080ad9a0 con 0x7f7108070e50 2024-09-17T05:18:30.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710dcd9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7108074450 0x7f71080c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:30.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710dcd9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7108074450 0x7f71080c0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48864/0 (socket says 172.21.15.150:48864) 2024-09-17T05:18:30.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710dcd9640 1 -- 172.21.15.150:0/1934131275 learned_addr learned my addr 172.21.15.150:0/1934131275 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:30.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710ccd7640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7108073440 0x7f71080bea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:30.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710dcd9640 1 -- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7108070e50 msgr2=0x7f71080bf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:30.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710d4d8640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7108070e50 0x7f71080bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:30.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710dcd9640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7108070e50 0x7f71080bf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:30.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710dcd9640 1 -- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7108073440 msgr2=0x7f71080bea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:30.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710dcd9640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7108073440 0x7f71080bea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:30.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710dcd9640 1 -- 172.21.15.150:0/1934131275 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71080c1cb0 con 0x7f7108074450 2024-09-17T05:18:30.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710d4d8640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7108070e50 0x7f71080bf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:18:30.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710dcd9640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7108074450 0x7f71080c0eb0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f7100009b10 tx=0x7f7100009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:30.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f70fdffb640 1 -- 172.21.15.150:0/1934131275 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71000bb070 con 0x7f7108074450 2024-09-17T05:18:30.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f70fdffb640 1 -- 172.21.15.150:0/1934131275 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7100004d40 con 0x7f7108074450 2024-09-17T05:18:30.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710f73c640 1 -- 172.21.15.150:0/1934131275 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71080bd370 con 0x7f7108074450 2024-09-17T05:18:30.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f70fdffb640 1 -- 172.21.15.150:0/1934131275 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71000b2040 con 0x7f7108074450 2024-09-17T05:18:30.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710f73c640 1 -- 172.21.15.150:0/1934131275 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71080bd610 con 0x7f7108074450 2024-09-17T05:18:30.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f710f73c640 1 -- 172.21.15.150:0/1934131275 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f710814d5a0 con 0x7f7108074450 2024-09-17T05:18:30.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f70fdffb640 1 -- 172.21.15.150:0/1934131275 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f71000be070 con 0x7f7108074450 2024-09-17T05:18:30.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f70fdffb640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f70ec073200 0x7f70ec075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:30.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.634+0000 7f70fdffb640 1 -- 172.21.15.150:0/1934131275 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f710012f150 con 0x7f7108074450 2024-09-17T05:18:30.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.638+0000 7f710d4d8640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f70ec073200 0x7f70ec075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:30.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.638+0000 7f710d4d8640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f70ec073200 0x7f70ec075710 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7104001230 tx=0x7f7104009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:30.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.642+0000 7f70fdffb640 1 -- 172.21.15.150:0/1934131275 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7100016790 con 0x7f7108074450 2024-09-17T05:18:30.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.842+0000 7f710f73c640 1 -- 172.21.15.150:0/1934131275 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7108074860 con 0x7f7108074450 2024-09-17T05:18:30.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.842+0000 7f70fdffb640 1 -- 172.21.15.150:0/1934131275 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+6552 (secure 0 0 0) 0x7f71000f9210 con 0x7f7108074450 2024-09-17T05:18:30.840 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:30.840 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":198,"btime":"2024-09-17T05:17:05:888759+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7384,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198},{"gid":8031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:16:54.689070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"c","rank":0,"incarnation":190,"state":"up:active","state_seq":15,"addr":"172.21.15.150:6835/3297154462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3297154462},{"type":"v1","addr":"172.21.15.150:6835","nonce":3297154462}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:16:56.716447+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6563":{"gid":6563,"name":"d","rank":0,"incarnation":193,"state":"up:active","state_seq":28,"addr":"172.21.15.176:6837/1903363960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":1903363960},{"type":"v1","addr":"172.21.15.176:6837","nonce":1903363960}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6563,"qdb_cluster":[6563]},"id":2}]} 2024-09-17T05:18:30.840 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 198 2024-09-17T05:18:30.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.846+0000 7f710f73c640 1 -- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f70ec073200 msgr2=0x7f70ec075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:30.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.846+0000 7f710f73c640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f70ec073200 0x7f70ec075710 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7104001230 tx=0x7f7104009040 comp rx=0 tx=0).stop 2024-09-17T05:18:30.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.846+0000 7f710f73c640 1 -- 172.21.15.150:0/1934131275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7108074450 msgr2=0x7f71080c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:30.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.846+0000 7f710f73c640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7108074450 0x7f71080c0eb0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f7100009b10 tx=0x7f7100009260 comp rx=0 tx=0).stop 2024-09-17T05:18:30.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.846+0000 7f710f73c640 1 -- 172.21.15.150:0/1934131275 shutdown_connections 2024-09-17T05:18:30.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.846+0000 7f710f73c640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f70ec073200 0x7f70ec075710 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:30.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.846+0000 7f710f73c640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7108073440 0x7f71080bea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:30.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.846+0000 7f710f73c640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7108074450 0x7f71080c0eb0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:30.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.846+0000 7f710f73c640 1 --2- 172.21.15.150:0/1934131275 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7108070e50 0x7f71080bf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:30.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.846+0000 7f710f73c640 1 -- 172.21.15.150:0/1934131275 >> 172.21.15.150:0/1934131275 conn(0x7f71080c4a00 msgr2=0x7f71080c35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:30.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.846+0000 7f710f73c640 1 -- 172.21.15.150:0/1934131275 shutdown_connections 2024-09-17T05:18:30.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:30.846+0000 7f710f73c640 1 -- 172.21.15.150:0/1934131275 wait complete. 2024-09-17T05:18:30.861 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-09-17T05:18:31.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:31.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.722+0000 7f833ea81640 1 -- 172.21.15.150:0/3821406308 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8338159f90 msgr2=0x7f833815a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:31.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.722+0000 7f833ea81640 1 --2- 172.21.15.150:0/3821406308 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8338159f90 0x7f833815a3a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8330009c80 tx=0x7f833001cd10 comp rx=0 tx=0).stop 2024-09-17T05:18:31.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.722+0000 7f833ea81640 1 -- 172.21.15.150:0/3821406308 shutdown_connections 2024-09-17T05:18:31.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.722+0000 7f833ea81640 1 --2- 172.21.15.150:0/3821406308 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8338159f90 0x7f833815a3a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:31.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.722+0000 7f833ea81640 1 --2- 172.21.15.150:0/3821406308 >> v2:172.21.15.176:3300/0 conn(0x7f833806e530 0x7f8338150bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:31.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.722+0000 7f833ea81640 1 -- 172.21.15.150:0/3821406308 >> 172.21.15.150:0/3821406308 conn(0x7f8338070b50 msgr2=0x7f8338070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:31.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.722+0000 7f833ea81640 1 -- 172.21.15.150:0/3821406308 shutdown_connections 2024-09-17T05:18:31.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.722+0000 7f833ea81640 1 -- 172.21.15.150:0/3821406308 wait complete. 2024-09-17T05:18:31.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833ea81640 1 Processor -- start 2024-09-17T05:18:31.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833ea81640 1 -- start start 2024-09-17T05:18:31.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833ea81640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f833806e530 0x7f83380ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:31.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833ea81640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8338159f90 0x7f83381590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:31.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833ea81640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83380ac540 0x7f83380ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:31.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833ea81640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f83380aa410 con 0x7f833806e530 2024-09-17T05:18:31.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833ea81640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f83380aa290 con 0x7f83380ac540 2024-09-17T05:18:31.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833ea81640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8338163b40 con 0x7f8338159f90 2024-09-17T05:18:31.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833c81d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f833806e530 0x7f83380ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:31.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833c81d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f833806e530 0x7f83380ab5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48866/0 (socket says 172.21.15.150:48866) 2024-09-17T05:18:31.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833c81d640 1 -- 172.21.15.150:0/607338090 learned_addr learned my addr 172.21.15.150:0/607338090 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f8337fff640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8338159f90 0x7f83381590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833d01e640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83380ac540 0x7f83380ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833c81d640 1 -- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8338159f90 msgr2=0x7f83381590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833c81d640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8338159f90 0x7f83381590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833c81d640 1 -- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83380ac540 msgr2=0x7f83380ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833c81d640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83380ac540 0x7f83380ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833c81d640 1 -- 172.21.15.150:0/607338090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83380c8340 con 0x7f833806e530 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833c81d640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f833806e530 0x7f83380ab5c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f832400c420 tx=0x7f832400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f83357fa640 1 -- 172.21.15.150:0/607338090 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8324017070 con 0x7f833806e530 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f83357fa640 1 -- 172.21.15.150:0/607338090 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83240053c0 con 0x7f833806e530 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833ea81640 1 -- 172.21.15.150:0/607338090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83380c62f0 con 0x7f833806e530 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f83357fa640 1 -- 172.21.15.150:0/607338090 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8324003860 con 0x7f833806e530 2024-09-17T05:18:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.726+0000 7f833ea81640 1 -- 172.21.15.150:0/607338090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83380abdf0 con 0x7f833806e530 2024-09-17T05:18:31.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.730+0000 7f833ea81640 1 -- 172.21.15.150:0/607338090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83380a9e20 con 0x7f833806e530 2024-09-17T05:18:31.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.730+0000 7f83357fa640 1 -- 172.21.15.150:0/607338090 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f832401a070 con 0x7f833806e530 2024-09-17T05:18:31.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.730+0000 7f83357fa640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f831c073230 0x7f831c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:31.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.730+0000 7f83357fa640 1 -- 172.21.15.150:0/607338090 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7f83240945b0 con 0x7f833806e530 2024-09-17T05:18:31.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.730+0000 7f8337fff640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f831c073230 0x7f831c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:31.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.734+0000 7f8337fff640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f831c073230 0x7f831c075740 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8338144640 tx=0x7f832c0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:31.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.734+0000 7f83357fa640 1 -- 172.21.15.150:0/607338090 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f832405dee0 con 0x7f833806e530 2024-09-17T05:18:31.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.934+0000 7f833ea81640 1 -- 172.21.15.150:0/607338090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83380ae800 con 0x7f833806e530 2024-09-17T05:18:31.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.934+0000 7f83357fa640 1 -- 172.21.15.150:0/607338090 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) ==== 95+0+4879 (secure 0 0 0) 0x7f8324062910 con 0x7f833806e530 2024-09-17T05:18:31.932 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:31.932 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":199,"btime":"2024-09-17T05:18:31:295338+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:31.295328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:replay","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:31.295328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:18:31.933 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 199 2024-09-17T05:18:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.938+0000 7f833ea81640 1 -- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f831c073230 msgr2=0x7f831c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.938+0000 7f833ea81640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f831c073230 0x7f831c075740 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8338144640 tx=0x7f832c0091e0 comp rx=0 tx=0).stop 2024-09-17T05:18:31.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.938+0000 7f833ea81640 1 -- 172.21.15.150:0/607338090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f833806e530 msgr2=0x7f83380ab5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.938+0000 7f833ea81640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f833806e530 0x7f83380ab5c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f832400c420 tx=0x7f832400c8e0 comp rx=0 tx=0).stop 2024-09-17T05:18:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.938+0000 7f833ea81640 1 -- 172.21.15.150:0/607338090 shutdown_connections 2024-09-17T05:18:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.938+0000 7f833ea81640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f831c073230 0x7f831c075740 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.938+0000 7f833ea81640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83380ac540 0x7f83380ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.938+0000 7f833ea81640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8338159f90 0x7f83381590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.938+0000 7f833ea81640 1 --2- 172.21.15.150:0/607338090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f833806e530 0x7f83380ab5c0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.938+0000 7f833ea81640 1 -- 172.21.15.150:0/607338090 >> 172.21.15.150:0/607338090 conn(0x7f8338070b50 msgr2=0x7f833814a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.938+0000 7f833ea81640 1 -- 172.21.15.150:0/607338090 shutdown_connections 2024-09-17T05:18:31.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:31.938+0000 7f833ea81640 1 -- 172.21.15.150:0/607338090 wait complete. 2024-09-17T05:18:31.953 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-09-17T05:18:31.953 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:18:31.954 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:32.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.098+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/3955924608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc152cf0 msgr2=0x7fe9cc153100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:32.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.098+0000 7fe9d3aa8640 1 --2- 172.21.15.150:0/3955924608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc152cf0 0x7fe9cc153100 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc0c7010 tx=0x7fe9c401cd50 comp rx=0 tx=0).stop 2024-09-17T05:18:32.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.098+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/3955924608 shutdown_connections 2024-09-17T05:18:32.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.098+0000 7fe9d3aa8640 1 --2- 172.21.15.150:0/3955924608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc152cf0 0x7fe9cc153100 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:32.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.098+0000 7fe9d3aa8640 1 --2- 172.21.15.150:0/3955924608 >> v2:172.21.15.176:3300/0 conn(0x7fe9cc0c42e0 0x7fe9cc0c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:32.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.098+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/3955924608 >> 172.21.15.150:0/3955924608 conn(0x7fe9cc0c4a00 msgr2=0x7fe9cc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:32.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.098+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/3955924608 shutdown_connections 2024-09-17T05:18:32.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.098+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/3955924608 wait complete. 2024-09-17T05:18:32.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d3aa8640 1 Processor -- start 2024-09-17T05:18:32.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d3aa8640 1 -- start start 2024-09-17T05:18:32.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d3aa8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc0a7350 0x7fe9cc0ab620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:32.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d3aa8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9cc0c42e0 0x7fe9cc1f0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:32.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d3aa8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9cc152cf0 0x7fe9cc146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:32.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d3aa8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc0c6d50 con 0x7fe9cc0a7350 2024-09-17T05:18:32.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d3aa8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc0c6bd0 con 0x7fe9cc0c42e0 2024-09-17T05:18:32.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d3aa8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe9cc153640 con 0x7fe9cc152cf0 2024-09-17T05:18:32.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d1844640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc0a7350 0x7fe9cc0ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:32.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d1844640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc0a7350 0x7fe9cc0ab620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48874/0 (socket says 172.21.15.150:48874) 2024-09-17T05:18:32.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d1844640 1 -- 172.21.15.150:0/4213916855 learned_addr learned my addr 172.21.15.150:0/4213916855 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:32.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d1043640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9cc0c42e0 0x7fe9cc1f0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:32.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d2045640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9cc152cf0 0x7fe9cc146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:32.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d1844640 1 -- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9cc152cf0 msgr2=0x7fe9cc146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:32.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d1844640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9cc152cf0 0x7fe9cc146a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:32.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d1844640 1 -- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9cc0c42e0 msgr2=0x7fe9cc1f0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:32.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d1844640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9cc0c42e0 0x7fe9cc1f0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:32.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d1844640 1 -- 172.21.15.150:0/4213916855 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc155e30 con 0x7fe9cc0a7350 2024-09-17T05:18:32.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d1043640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9cc0c42e0 0x7fe9cc1f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:18:32.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d1844640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc0a7350 0x7fe9cc0ab620 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800d720 tx=0x7fe9c800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:32.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9c27fc640 1 -- 172.21.15.150:0/4213916855 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8017070 con 0x7fe9cc0a7350 2024-09-17T05:18:32.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9c27fc640 1 -- 172.21.15.150:0/4213916855 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9c8009cc0 con 0x7fe9cc0a7350 2024-09-17T05:18:32.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/4213916855 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9cc1f1160 con 0x7fe9cc0a7350 2024-09-17T05:18:32.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9c27fc640 1 -- 172.21.15.150:0/4213916855 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8005280 con 0x7fe9cc0a7350 2024-09-17T05:18:32.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/4213916855 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9cc0ac060 con 0x7fe9cc0a7350 2024-09-17T05:18:32.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.102+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/4213916855 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9cc0c6690 con 0x7fe9cc0a7350 2024-09-17T05:18:32.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.106+0000 7fe9c27fc640 1 -- 172.21.15.150:0/4213916855 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe9c8005420 con 0x7fe9cc0a7350 2024-09-17T05:18:32.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.106+0000 7fe9c27fc640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe9b0073200 0x7fe9b0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:32.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.106+0000 7fe9c27fc640 1 -- 172.21.15.150:0/4213916855 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7fe9c80954a0 con 0x7fe9cc0a7350 2024-09-17T05:18:32.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.106+0000 7fe9d1043640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe9b0073200 0x7fe9b0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:32.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.106+0000 7fe9d1043640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe9b0073200 0x7fe9b0075710 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc000f70 tx=0x7fe9bc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:32.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.110+0000 7fe9c27fc640 1 -- 172.21.15.150:0/4213916855 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe9c805f390 con 0x7fe9cc0a7350 2024-09-17T05:18:32.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.298+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/4213916855 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9cc06e900 con 0x7fe9cc0a7350 2024-09-17T05:18:32.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9c27fc640 1 -- 172.21.15.150:0/4213916855 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) ==== 95+0+4879 (secure 0 0 0) 0x7fe9c805ec90 con 0x7fe9cc0a7350 2024-09-17T05:18:32.297 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:32.297 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":199,"btime":"2024-09-17T05:18:31:295338+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:31.295328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:replay","state_seq":1,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:31.295328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:18:32.297 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 199 2024-09-17T05:18:32.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe9b0073200 msgr2=0x7fe9b0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:32.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9d3aa8640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe9b0073200 0x7fe9b0075710 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc000f70 tx=0x7fe9bc009040 comp rx=0 tx=0).stop 2024-09-17T05:18:32.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/4213916855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc0a7350 msgr2=0x7fe9cc0ab620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:32.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9d3aa8640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc0a7350 0x7fe9cc0ab620 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800d720 tx=0x7fe9c800dbe0 comp rx=0 tx=0).stop 2024-09-17T05:18:32.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/4213916855 shutdown_connections 2024-09-17T05:18:32.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9d3aa8640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe9b0073200 0x7fe9b0075710 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:32.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9d3aa8640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe9cc0c42e0 0x7fe9cc1f0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:32.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9d3aa8640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe9cc152cf0 0x7fe9cc146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:32.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9d3aa8640 1 --2- 172.21.15.150:0/4213916855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9cc0a7350 0x7fe9cc0ab620 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:32.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/4213916855 >> 172.21.15.150:0/4213916855 conn(0x7fe9cc0c4a00 msgr2=0x7fe9cc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:32.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/4213916855 shutdown_connections 2024-09-17T05:18:32.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:32.302+0000 7fe9d3aa8640 1 -- 172.21.15.150:0/4213916855 wait complete. 2024-09-17T05:18:32.317 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:18:31:295338+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 199, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 199, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7384": { "addr": "172.21.15.176:6835/1690935336", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 1690935336, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 1690935336, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7384, "incarnation": 199, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:18:31.295328+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7384 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 199, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8031": { "addr": "172.21.15.150:6833/2857695220", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2857695220, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 2857695220, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 8031, "incarnation": 199, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:18:31.295328+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8031 } } } ], "standbys": [] } 2024-09-17T05:18:32.862 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:33.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.034+0000 7f8bb143d640 1 -- 172.21.15.150:0/497415875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b94003a40 msgr2=0x7f8b94003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:33.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.034+0000 7f8bb143d640 1 --2- 172.21.15.150:0/497415875 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b94003a40 0x7f8b94003eb0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00c260 tx=0x7f8b9c01cf10 comp rx=0 tx=0).stop 2024-09-17T05:18:33.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.034+0000 7f8bab7fe640 1 -- 172.21.15.150:0/497415875 reap_dead start 2024-09-17T05:18:33.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.034+0000 7f8bb143d640 1 -- 172.21.15.150:0/497415875 shutdown_connections 2024-09-17T05:18:33.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.034+0000 7f8bb143d640 1 -- 172.21.15.150:0/497415875 >> 172.21.15.150:0/497415875 conn(0x7f8bac0c4a00 msgr2=0x7f8bac0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:33.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.034+0000 7f8bb143d640 1 -- 172.21.15.150:0/497415875 shutdown_connections 2024-09-17T05:18:33.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.034+0000 7f8bb143d640 1 -- 172.21.15.150:0/497415875 wait complete. 2024-09-17T05:18:33.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.034+0000 7f8bb143d640 1 Processor -- start 2024-09-17T05:18:33.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.034+0000 7f8bb143d640 1 -- start start 2024-09-17T05:18:33.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8bb143d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b94003a40 0x7f8bac0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:33.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8bb143d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac0c6690 0x7f8bac1f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:33.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8bb143d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8bac0c6aa0 0x7f8bac14d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:33.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8bb143d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac0ad820 con 0x7f8bac0c6690 2024-09-17T05:18:33.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8bb143d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac0adef0 con 0x7f8b94003a40 2024-09-17T05:18:33.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8bb143d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8bac0ad9a0 con 0x7f8bac0c6aa0 2024-09-17T05:18:33.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8baa7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac0c6690 0x7f8bac1f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:33.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8baa7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac0c6690 0x7f8bac1f44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48878/0 (socket says 172.21.15.150:48878) 2024-09-17T05:18:33.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8baa7fc640 1 -- 172.21.15.150:0/1966718923 learned_addr learned my addr 172.21.15.150:0/1966718923 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:33.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8bab7fe640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8bac0c6aa0 0x7f8bac14d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:33.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8baaffd640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b94003a40 0x7f8bac0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:33.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8baa7fc640 1 -- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8bac0c6aa0 msgr2=0x7f8bac14d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:33.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8baa7fc640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8bac0c6aa0 0x7f8bac14d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8baa7fc640 1 -- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b94003a40 msgr2=0x7f8bac0a3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:33.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8baa7fc640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b94003a40 0x7f8bac0a3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8baa7fc640 1 -- 172.21.15.150:0/1966718923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bac150650 con 0x7f8bac0c6690 2024-09-17T05:18:33.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8baa7fc640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac0c6690 0x7f8bac1f44d0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0011a50 tx=0x7f8ba0011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:33.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8b937fe640 1 -- 172.21.15.150:0/1966718923 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba000fc70 con 0x7f8bac0c6690 2024-09-17T05:18:33.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8b937fe640 1 -- 172.21.15.150:0/1966718923 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ba0004020 con 0x7f8bac0c6690 2024-09-17T05:18:33.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8bb143d640 1 -- 172.21.15.150:0/1966718923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bac150090 con 0x7f8bac0c6690 2024-09-17T05:18:33.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8b937fe640 1 -- 172.21.15.150:0/1966718923 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0004470 con 0x7f8bac0c6690 2024-09-17T05:18:33.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8bb143d640 1 -- 172.21.15.150:0/1966718923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bac0ac3b0 con 0x7f8bac0c6690 2024-09-17T05:18:33.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8bb143d640 1 -- 172.21.15.150:0/1966718923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bac074450 con 0x7f8bac0c6690 2024-09-17T05:18:33.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.038+0000 7f8b937fe640 1 -- 172.21.15.150:0/1966718923 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8ba0003600 con 0x7f8bac0c6690 2024-09-17T05:18:33.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.042+0000 7f8b937fe640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b98073200 0x7f8b98075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:33.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.042+0000 7f8b937fe640 1 -- 172.21.15.150:0/1966718923 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7f8ba0095e50 con 0x7f8bac0c6690 2024-09-17T05:18:33.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.042+0000 7f8baaffd640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b98073200 0x7f8b98075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:33.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.042+0000 7f8baaffd640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b98073200 0x7f8b98075710 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0ab000 tx=0x7f8b9c019d60 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:33.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.046+0000 7f8b937fe640 1 -- 172.21.15.150:0/1966718923 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8bac150650 con 0x7f8bac0c6690 2024-09-17T05:18:33.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.238+0000 7f8bb143d640 1 -- 172.21.15.150:0/1966718923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bac0c2260 con 0x7f8bac0c6690 2024-09-17T05:18:33.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.238+0000 7f8b937fe640 1 -- 172.21.15.150:0/1966718923 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) ==== 95+0+4887 (secure 0 0 0) 0x7f8ba005fe60 con 0x7f8bac0c6690 2024-09-17T05:18:33.235 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:33.235 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":200,"btime":"2024-09-17T05:18:32:620287+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:31.955991+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:31.865134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:18:33.235 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 200 2024-09-17T05:18:33.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.242+0000 7f8bb143d640 1 -- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b98073200 msgr2=0x7f8b98075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:33.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.242+0000 7f8bb143d640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b98073200 0x7f8b98075710 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0ab000 tx=0x7f8b9c019d60 comp rx=0 tx=0).stop 2024-09-17T05:18:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.242+0000 7f8bb143d640 1 -- 172.21.15.150:0/1966718923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac0c6690 msgr2=0x7f8bac1f44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.242+0000 7f8bb143d640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac0c6690 0x7f8bac1f44d0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0011a50 tx=0x7f8ba0011f10 comp rx=0 tx=0).stop 2024-09-17T05:18:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.242+0000 7f8bb143d640 1 -- 172.21.15.150:0/1966718923 shutdown_connections 2024-09-17T05:18:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.242+0000 7f8bb143d640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8b98073200 0x7f8b98075710 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.242+0000 7f8bb143d640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8bac0c6aa0 0x7f8bac14d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.242+0000 7f8bb143d640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bac0c6690 0x7f8bac1f44d0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.242+0000 7f8bb143d640 1 --2- 172.21.15.150:0/1966718923 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b94003a40 0x7f8bac0a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.242+0000 7f8bb143d640 1 -- 172.21.15.150:0/1966718923 >> 172.21.15.150:0/1966718923 conn(0x7f8bac0c4a00 msgr2=0x7f8bac0a8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.242+0000 7f8bb143d640 1 -- 172.21.15.150:0/1966718923 shutdown_connections 2024-09-17T05:18:33.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.242+0000 7f8bb143d640 1 -- 172.21.15.150:0/1966718923 wait complete. 2024-09-17T05:18:33.255 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-09-17T05:18:33.255 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:18:33.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:33.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 -- 172.21.15.150:0/863473896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa800ad7e0 msgr2=0x7ffa800701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:33.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 --2- 172.21.15.150:0/863473896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa800ad7e0 0x7ffa800701c0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7ffa800c4740 tx=0x7ffa7801cd80 comp rx=0 tx=0).stop 2024-09-17T05:18:33.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 -- 172.21.15.150:0/863473896 shutdown_connections 2024-09-17T05:18:33.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 --2- 172.21.15.150:0/863473896 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa800ad7e0 0x7ffa800701c0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 --2- 172.21.15.150:0/863473896 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa800c3db0 0x7ffa800ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 -- 172.21.15.150:0/863473896 >> 172.21.15.150:0/863473896 conn(0x7ffa800c4a00 msgr2=0x7ffa800c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:33.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 -- 172.21.15.150:0/863473896 shutdown_connections 2024-09-17T05:18:33.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 -- 172.21.15.150:0/863473896 wait complete. 2024-09-17T05:18:33.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 Processor -- start 2024-09-17T05:18:33.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 -- start start 2024-09-17T05:18:33.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa800ad7e0 0x7ffa8014b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:33.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa800c3db0 0x7ffa80149bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:33.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa800c41c0 0x7ffa80147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:33.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffa800a75f0 con 0x7ffa800ad7e0 2024-09-17T05:18:33.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffa800add70 con 0x7ffa800c3db0 2024-09-17T05:18:33.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa8849d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffa800a7770 con 0x7ffa800c41c0 2024-09-17T05:18:33.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa86239640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa800ad7e0 0x7ffa8014b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:33.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa86239640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa800ad7e0 0x7ffa8014b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48896/0 (socket says 172.21.15.150:48896) 2024-09-17T05:18:33.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.414+0000 7ffa86239640 1 -- 172.21.15.150:0/1371115259 learned_addr learned my addr 172.21.15.150:0/1371115259 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:33.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa85a38640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa800c3db0 0x7ffa80149bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:33.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa86a3a640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa800c41c0 0x7ffa80147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:33.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa86239640 1 -- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa800c41c0 msgr2=0x7ffa80147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:33.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa86239640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa800c41c0 0x7ffa80147320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa86239640 1 -- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa800c3db0 msgr2=0x7ffa80149bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:33.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa86239640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa800c3db0 0x7ffa80149bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa86239640 1 -- 172.21.15.150:0/1371115259 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa80074960 con 0x7ffa800ad7e0 2024-09-17T05:18:33.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa86239640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa800ad7e0 0x7ffa8014b9b0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00c9a0 tx=0x7ffa7c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:33.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa76ffd640 1 -- 172.21.15.150:0/1371115259 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c004220 con 0x7ffa800ad7e0 2024-09-17T05:18:33.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa76ffd640 1 -- 172.21.15.150:0/1371115259 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa7c0079b0 con 0x7ffa800ad7e0 2024-09-17T05:18:33.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa76ffd640 1 -- 172.21.15.150:0/1371115259 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c0044d0 con 0x7ffa800ad7e0 2024-09-17T05:18:33.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa8849d640 1 -- 172.21.15.150:0/1371115259 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa80148110 con 0x7ffa800ad7e0 2024-09-17T05:18:33.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa8849d640 1 -- 172.21.15.150:0/1371115259 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa80075060 con 0x7ffa800ad7e0 2024-09-17T05:18:33.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa8849d640 1 -- 172.21.15.150:0/1371115259 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa80073440 con 0x7ffa800ad7e0 2024-09-17T05:18:33.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa76ffd640 1 -- 172.21.15.150:0/1371115259 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ffa7c003620 con 0x7ffa800ad7e0 2024-09-17T05:18:33.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa76ffd640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa64073200 0x7ffa64075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:33.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa76ffd640 1 -- 172.21.15.150:0/1371115259 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7ffa7c095360 con 0x7ffa800ad7e0 2024-09-17T05:18:33.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.418+0000 7ffa85a38640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa64073200 0x7ffa64075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:33.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.422+0000 7ffa85a38640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa64073200 0x7ffa64075710 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ffa8006a990 tx=0x7ffa70008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:33.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.426+0000 7ffa76ffd640 1 -- 172.21.15.150:0/1371115259 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ffa7c014030 con 0x7ffa800ad7e0 2024-09-17T05:18:33.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.626+0000 7ffa8849d640 1 -- 172.21.15.150:0/1371115259 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa800ad2a0 con 0x7ffa800ad7e0 2024-09-17T05:18:33.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.626+0000 7ffa76ffd640 1 -- 172.21.15.150:0/1371115259 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) ==== 95+0+4887 (secure 0 0 0) 0x7ffa7c05f370 con 0x7ffa800ad7e0 2024-09-17T05:18:33.623 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:33.624 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":200,"btime":"2024-09-17T05:18:32:620287+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:31.955991+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:31.865134+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:18:33.624 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 200 2024-09-17T05:18:33.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.630+0000 7ffa8849d640 1 -- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa64073200 msgr2=0x7ffa64075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:33.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.630+0000 7ffa8849d640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa64073200 0x7ffa64075710 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ffa8006a990 tx=0x7ffa70008040 comp rx=0 tx=0).stop 2024-09-17T05:18:33.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.630+0000 7ffa8849d640 1 -- 172.21.15.150:0/1371115259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa800ad7e0 msgr2=0x7ffa8014b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:33.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.630+0000 7ffa8849d640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa800ad7e0 0x7ffa8014b9b0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00c9a0 tx=0x7ffa7c00ce60 comp rx=0 tx=0).stop 2024-09-17T05:18:33.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.630+0000 7ffa8849d640 1 -- 172.21.15.150:0/1371115259 shutdown_connections 2024-09-17T05:18:33.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.630+0000 7ffa8849d640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa64073200 0x7ffa64075710 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.630+0000 7ffa8849d640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa800c3db0 0x7ffa80149bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.630+0000 7ffa8849d640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa800c41c0 0x7ffa80147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.630+0000 7ffa8849d640 1 --2- 172.21.15.150:0/1371115259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa800ad7e0 0x7ffa8014b9b0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:33.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.630+0000 7ffa8849d640 1 -- 172.21.15.150:0/1371115259 >> 172.21.15.150:0/1371115259 conn(0x7ffa800c4a00 msgr2=0x7ffa800c1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:33.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.630+0000 7ffa8849d640 1 -- 172.21.15.150:0/1371115259 shutdown_connections 2024-09-17T05:18:33.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:33.630+0000 7ffa8849d640 1 -- 172.21.15.150:0/1371115259 wait complete. 2024-09-17T05:18:33.649 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:18:32:620287+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 200, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 200, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7384": { "addr": "172.21.15.176:6835/1690935336", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 1690935336, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 1690935336, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7384, "incarnation": 199, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:18:31.955991+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7384 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 200, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8031": { "addr": "172.21.15.150:6833/2857695220", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2857695220, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 2857695220, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 8031, "incarnation": 199, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:18:31.865134+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8031 } } } ], "standbys": [] } 2024-09-17T05:18:34.318 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.486+0000 7fc83f0e2640 1 -- 172.21.15.150:0/872274690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8381506d0 msgr2=0x7fc8380c2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.486+0000 7fc83f0e2640 1 --2- 172.21.15.150:0/872274690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8381506d0 0x7fc8380c2240 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fc83808db10 tx=0x7fc82401c8c0 comp rx=0 tx=0).stop 2024-09-17T05:18:34.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.486+0000 7fc83f0e2640 1 -- 172.21.15.150:0/872274690 shutdown_connections 2024-09-17T05:18:34.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.486+0000 7fc83f0e2640 1 --2- 172.21.15.150:0/872274690 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8380c2780 0x7fc8380c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:34.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.486+0000 7fc83f0e2640 1 --2- 172.21.15.150:0/872274690 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8381506d0 0x7fc8380c2240 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:34.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.486+0000 7fc83f0e2640 1 --2- 172.21.15.150:0/872274690 >> v2:172.21.15.176:3300/0 conn(0x7fc838156820 0x7fc8380c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:34.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.486+0000 7fc83f0e2640 1 -- 172.21.15.150:0/872274690 >> 172.21.15.150:0/872274690 conn(0x7fc838074b80 msgr2=0x7fc83806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:34.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.486+0000 7fc83f0e2640 1 -- 172.21.15.150:0/872274690 shutdown_connections 2024-09-17T05:18:34.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.486+0000 7fc83f0e2640 1 -- 172.21.15.150:0/872274690 wait complete. 2024-09-17T05:18:34.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83f0e2640 1 Processor -- start 2024-09-17T05:18:34.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83f0e2640 1 -- start start 2024-09-17T05:18:34.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83f0e2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8380c2780 0x7fc838151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:34.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83f0e2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8381506d0 0x7fc83814f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:34.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83f0e2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc838156820 0x7fc838147950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:34.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83f0e2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc83806ff00 con 0x7fc838156820 2024-09-17T05:18:34.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83f0e2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc83806fd80 con 0x7fc8380c2780 2024-09-17T05:18:34.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83f0e2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc8380c5790 con 0x7fc8381506d0 2024-09-17T05:18:34.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83d67f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc838156820 0x7fc838147950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:34.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83d67f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc838156820 0x7fc838147950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48908/0 (socket says 172.21.15.150:48908) 2024-09-17T05:18:34.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83d67f640 1 -- 172.21.15.150:0/815130737 learned_addr learned my addr 172.21.15.150:0/815130737 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:34.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83ce7e640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8380c2780 0x7fc838151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:34.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc837fff640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8381506d0 0x7fc83814f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:34.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83d67f640 1 -- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8381506d0 msgr2=0x7fc83814f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:34.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83d67f640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8381506d0 0x7fc83814f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:34.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83d67f640 1 -- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8380c2780 msgr2=0x7fc838151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:34.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83d67f640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8380c2780 0x7fc838151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:34.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83d67f640 1 -- 172.21.15.150:0/815130737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc838148bf0 con 0x7fc838156820 2024-09-17T05:18:34.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83d67f640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc838156820 0x7fc838147950 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc83000ca60 tx=0x7fc83000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:34.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc8357fa640 1 -- 172.21.15.150:0/815130737 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc830007d90 con 0x7fc838156820 2024-09-17T05:18:34.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc8357fa640 1 -- 172.21.15.150:0/815130737 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc83000f560 con 0x7fc838156820 2024-09-17T05:18:34.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83f0e2640 1 -- 172.21.15.150:0/815130737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc838148ed0 con 0x7fc838156820 2024-09-17T05:18:34.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc8357fa640 1 -- 172.21.15.150:0/815130737 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83000f9b0 con 0x7fc838156820 2024-09-17T05:18:34.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.490+0000 7fc83f0e2640 1 -- 172.21.15.150:0/815130737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8381501f0 con 0x7fc838156820 2024-09-17T05:18:34.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.494+0000 7fc83f0e2640 1 -- 172.21.15.150:0/815130737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8380b0260 con 0x7fc838156820 2024-09-17T05:18:34.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.494+0000 7fc8357fa640 1 -- 172.21.15.150:0/815130737 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc83000fb50 con 0x7fc838156820 2024-09-17T05:18:34.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.494+0000 7fc8357fa640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc81c073230 0x7fc81c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:34.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.494+0000 7fc8357fa640 1 -- 172.21.15.150:0/815130737 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7fc830095d60 con 0x7fc838156820 2024-09-17T05:18:34.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.494+0000 7fc83ce7e640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc81c073230 0x7fc81c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:34.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.494+0000 7fc83ce7e640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc81c073230 0x7fc81c075740 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00ca60 tx=0x7fc82c00c440 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:34.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.498+0000 7fc8357fa640 1 -- 172.21.15.150:0/815130737 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc830014070 con 0x7fc838156820 2024-09-17T05:18:34.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 -- 172.21.15.150:0/815130737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc838150f00 con 0x7fc838156820 2024-09-17T05:18:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc8357fa640 1 -- 172.21.15.150:0/815130737 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) ==== 95+0+4895 (secure 0 0 0) 0x7fc83005fc50 con 0x7fc838156820 2024-09-17T05:18:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:34.734 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":202,"btime":"2024-09-17T05:18:34:641262+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:18:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 202 2024-09-17T05:18:34.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 -- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc81c073230 msgr2=0x7fc81c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:34.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc81c073230 0x7fc81c075740 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00ca60 tx=0x7fc82c00c440 comp rx=0 tx=0).stop 2024-09-17T05:18:34.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 -- 172.21.15.150:0/815130737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc838156820 msgr2=0x7fc838147950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:34.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc838156820 0x7fc838147950 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fc83000ca60 tx=0x7fc83000cf20 comp rx=0 tx=0).stop 2024-09-17T05:18:34.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 -- 172.21.15.150:0/815130737 shutdown_connections 2024-09-17T05:18:34.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc81c073230 0x7fc81c075740 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:34.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc838156820 0x7fc838147950 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:34.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8381506d0 0x7fc83814f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:34.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 --2- 172.21.15.150:0/815130737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8380c2780 0x7fc838151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:34.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 -- 172.21.15.150:0/815130737 >> 172.21.15.150:0/815130737 conn(0x7fc838074b80 msgr2=0x7fc838161700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:34.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 -- 172.21.15.150:0/815130737 shutdown_connections 2024-09-17T05:18:34.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:34.738+0000 7fc83f0e2640 1 -- 172.21.15.150:0/815130737 wait complete. 2024-09-17T05:18:34.764 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=6563 2024-09-17T05:18:34.764 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.d 2024-09-17T05:18:35.650 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:35.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.814+0000 7f3e0de9f640 1 -- 172.21.15.150:0/187910163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df0006330 msgr2=0x7f3df000cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:35.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.814+0000 7f3e0de9f640 1 --2- 172.21.15.150:0/187910163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3df0006330 0x7f3df000cae0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f3e0806fcc0 tx=0x7f3e0001f9a0 comp rx=0 tx=0).stop 2024-09-17T05:18:35.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.814+0000 7f3e07fff640 1 -- 172.21.15.150:0/187910163 reap_dead start 2024-09-17T05:18:35.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.814+0000 7f3e0de9f640 1 -- 172.21.15.150:0/187910163 shutdown_connections 2024-09-17T05:18:35.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.814+0000 7f3e0de9f640 1 -- 172.21.15.150:0/187910163 >> 172.21.15.150:0/187910163 conn(0x7f3e080c4a00 msgr2=0x7f3e080c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:35.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.814+0000 7f3e0de9f640 1 -- 172.21.15.150:0/187910163 shutdown_connections 2024-09-17T05:18:35.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.814+0000 7f3e0de9f640 1 -- 172.21.15.150:0/187910163 wait complete. 2024-09-17T05:18:35.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e0de9f640 1 Processor -- start 2024-09-17T05:18:35.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e0de9f640 1 -- start start 2024-09-17T05:18:35.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e0de9f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3df0006330 0x7f3e081eb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:35.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e0de9f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e081543c0 0x7f3e081547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:35.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e0de9f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e08152e40 0x7f3e08153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:35.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e0de9f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3e08070ac0 con 0x7f3e081543c0 2024-09-17T05:18:35.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e0de9f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3e081509b0 con 0x7f3e08152e40 2024-09-17T05:18:35.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e0de9f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3e08070da0 con 0x7f3df0006330 2024-09-17T05:18:35.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e06ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e081543c0 0x7f3e081547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:35.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e06ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e081543c0 0x7f3e081547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48932/0 (socket says 172.21.15.150:48932) 2024-09-17T05:18:35.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e06ffd640 1 -- 172.21.15.150:0/3653477406 learned_addr learned my addr 172.21.15.150:0/3653477406 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:35.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e077fe640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3df0006330 0x7f3e081eb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:35.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e06ffd640 1 -- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3df0006330 msgr2=0x7f3e081eb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:35.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e06ffd640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3df0006330 0x7f3e081eb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:35.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e06ffd640 1 -- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e08152e40 msgr2=0x7f3e08153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:35.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e07fff640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e08152e40 0x7f3e08153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:35.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e06ffd640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e08152e40 0x7f3e08153250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:35.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e06ffd640 1 -- 172.21.15.150:0/3653477406 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e081ec0c0 con 0x7f3e081543c0 2024-09-17T05:18:35.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e06ffd640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e081543c0 0x7f3e081547d0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f3df8010940 tx=0x7f3df8010e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:35.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3deffff640 1 -- 172.21.15.150:0/3653477406 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df800ec80 con 0x7f3e081543c0 2024-09-17T05:18:35.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3deffff640 1 -- 172.21.15.150:0/3653477406 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3df800ee20 con 0x7f3e081543c0 2024-09-17T05:18:35.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e0de9f640 1 -- 172.21.15.150:0/3653477406 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e080ab0a0 con 0x7f3e081543c0 2024-09-17T05:18:35.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3deffff640 1 -- 172.21.15.150:0/3653477406 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8004290 con 0x7f3e081543c0 2024-09-17T05:18:35.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.818+0000 7f3e0de9f640 1 -- 172.21.15.150:0/3653477406 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e08149a20 con 0x7f3e081543c0 2024-09-17T05:18:35.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.822+0000 7f3e0de9f640 1 -- 172.21.15.150:0/3653477406 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e080ab920 con 0x7f3e081543c0 2024-09-17T05:18:35.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.822+0000 7f3deffff640 1 -- 172.21.15.150:0/3653477406 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3df8003600 con 0x7f3e081543c0 2024-09-17T05:18:35.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.822+0000 7f3deffff640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3df4073200 0x7f3df4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:35.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.822+0000 7f3deffff640 1 -- 172.21.15.150:0/3653477406 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7f3df8096010 con 0x7f3e081543c0 2024-09-17T05:18:35.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.822+0000 7f3e077fe640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3df4073200 0x7f3df4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:35.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.822+0000 7f3e077fe640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3df4073200 0x7f3df4075710 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc000c10 tx=0x7f3dfc006ab0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:35.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:35.826+0000 7f3deffff640 1 -- 172.21.15.150:0/3653477406 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3df8060020 con 0x7f3e081543c0 2024-09-17T05:18:35.915 INFO:tasks.ceph.mds.b.smithi176.stderr:2024-09-17T05:18:35.914+0000 7fde083b2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:18:36.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.030+0000 7f3e0de9f640 1 -- 172.21.15.150:0/3653477406 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e081502e0 con 0x7f3e081543c0 2024-09-17T05:18:36.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3deffff640 1 -- 172.21.15.150:0/3653477406 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) ==== 95+0+4895 (secure 0 0 0) 0x7f3df805f920 con 0x7f3e081543c0 2024-09-17T05:18:36.030 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:36.030 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":202,"btime":"2024-09-17T05:18:34:641262+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:18:36.030 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 202 2024-09-17T05:18:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3e0de9f640 1 -- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3df4073200 msgr2=0x7f3df4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3e0de9f640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3df4073200 0x7f3df4075710 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc000c10 tx=0x7f3dfc006ab0 comp rx=0 tx=0).stop 2024-09-17T05:18:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3e0de9f640 1 -- 172.21.15.150:0/3653477406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e081543c0 msgr2=0x7f3e081547d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3e0de9f640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e081543c0 0x7f3e081547d0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f3df8010940 tx=0x7f3df8010e00 comp rx=0 tx=0).stop 2024-09-17T05:18:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3e0de9f640 1 -- 172.21.15.150:0/3653477406 shutdown_connections 2024-09-17T05:18:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3e0de9f640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3df4073200 0x7f3df4075710 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3e0de9f640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3e08152e40 0x7f3e08153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:36.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3e0de9f640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e081543c0 0x7f3e081547d0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:36.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3e0de9f640 1 --2- 172.21.15.150:0/3653477406 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3df0006330 0x7f3e081eb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:36.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3e0de9f640 1 -- 172.21.15.150:0/3653477406 >> 172.21.15.150:0/3653477406 conn(0x7f3e080c4a00 msgr2=0x7f3e0814f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:36.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3e0de9f640 1 -- 172.21.15.150:0/3653477406 shutdown_connections 2024-09-17T05:18:36.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:36.034+0000 7f3e0de9f640 1 -- 172.21.15.150:0/3653477406 wait complete. 2024-09-17T05:18:36.050 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=7920 2024-09-17T05:18:36.050 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before reviving mds.c 2024-09-17T05:18:36.882 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-09-17T05:18:36.886+0000 7fe3e1b7d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:18:37.124 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:18:37.127 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:18:37.765 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-09-17T05:18:37.766 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T05:18:37.766 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T05:18:37.769 INFO:tasks.ceph.mds.d:Started 2024-09-17T05:18:37.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:37.876 INFO:tasks.ceph.mds.d.smithi176.stdout:starting mds.d at 2024-09-17T05:18:37.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.938+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/2437127229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d580c97a0 msgr2=0x7f2d58151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:37.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.938+0000 7f2d5d9cf640 1 --2- 172.21.15.150:0/2437127229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d580c97a0 0x7f2d58151c70 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f2d48009840 tx=0x7f2d4801c740 comp rx=0 tx=0).stop 2024-09-17T05:18:37.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.938+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/2437127229 shutdown_connections 2024-09-17T05:18:37.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.938+0000 7f2d5d9cf640 1 --2- 172.21.15.150:0/2437127229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d580c97a0 0x7f2d58151c70 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:37.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.938+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/2437127229 >> 172.21.15.150:0/2437127229 conn(0x7f2d580abb00 msgr2=0x7f2d580abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:37.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.938+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/2437127229 shutdown_connections 2024-09-17T05:18:37.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.938+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/2437127229 wait complete. 2024-09-17T05:18:37.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d5d9cf640 1 Processor -- start 2024-09-17T05:18:37.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d5d9cf640 1 -- start start 2024-09-17T05:18:37.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d5d9cf640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d58072ea0 0x7f2d58147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:37.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d5d9cf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d580732b0 0x7f2d581f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d5d9cf640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d580c97a0 0x7f2d5806f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d5d9cf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2d580c4be0 con 0x7f2d580732b0 2024-09-17T05:18:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d5d9cf640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2d58155470 con 0x7f2d580c97a0 2024-09-17T05:18:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d5d9cf640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2d580c4d60 con 0x7f2d58072ea0 2024-09-17T05:18:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d567fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d580732b0 0x7f2d581f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d567fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d580732b0 0x7f2d581f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:48938/0 (socket says 172.21.15.150:48938) 2024-09-17T05:18:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d567fc640 1 -- 172.21.15.150:0/1642296062 learned_addr learned my addr 172.21.15.150:0/1642296062 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d56ffd640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d58072ea0 0x7f2d58147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d577fe640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d580c97a0 0x7f2d5806f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d567fc640 1 -- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d58072ea0 msgr2=0x7f2d58147ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d567fc640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d58072ea0 0x7f2d58147ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d567fc640 1 -- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d580c97a0 msgr2=0x7f2d5806f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d567fc640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d580c97a0 0x7f2d5806f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d567fc640 1 -- 172.21.15.150:0/1642296062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d58157a30 con 0x7f2d580732b0 2024-09-17T05:18:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d567fc640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d580732b0 0x7f2d581f3970 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000cc60 tx=0x7f2d5000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d4ffff640 1 -- 172.21.15.150:0/1642296062 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50016020 con 0x7f2d580732b0 2024-09-17T05:18:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d4ffff640 1 -- 172.21.15.150:0/1642296062 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d5000aae0 con 0x7f2d580732b0 2024-09-17T05:18:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/1642296062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d581efe00 con 0x7f2d580732b0 2024-09-17T05:18:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d4ffff640 1 -- 172.21.15.150:0/1642296062 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50012420 con 0x7f2d580732b0 2024-09-17T05:18:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/1642296062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d581f3490 con 0x7f2d580732b0 2024-09-17T05:18:37.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.942+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/1642296062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d5814de20 con 0x7f2d580732b0 2024-09-17T05:18:37.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.946+0000 7f2d4ffff640 1 -- 172.21.15.150:0/1642296062 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f2d50019020 con 0x7f2d580732b0 2024-09-17T05:18:37.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.946+0000 7f2d4ffff640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d38073230 0x7f2d38075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:37.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.946+0000 7f2d4ffff640 1 -- 172.21.15.150:0/1642296062 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7f2d50095050 con 0x7f2d580732b0 2024-09-17T05:18:37.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.946+0000 7f2d56ffd640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d38073230 0x7f2d38075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:37.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.946+0000 7f2d56ffd640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d38073230 0x7f2d38075740 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2d480af000 tx=0x7f2d480b2040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:37.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:37.950+0000 7f2d4ffff640 1 -- 172.21.15.150:0/1642296062 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2d50020070 con 0x7f2d580732b0 2024-09-17T05:18:38.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.154+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/1642296062 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d58151c70 con 0x7f2d580732b0 2024-09-17T05:18:38.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.154+0000 7f2d4ffff640 1 -- 172.21.15.150:0/1642296062 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) ==== 95+0+4895 (secure 0 0 0) 0x7f2d580736c0 con 0x7f2d580732b0 2024-09-17T05:18:38.153 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:38.153 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":202,"btime":"2024-09-17T05:18:34:641262+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:18:38.153 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 202 2024-09-17T05:18:38.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.158+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d38073230 msgr2=0x7f2d38075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:38.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.158+0000 7f2d5d9cf640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d38073230 0x7f2d38075740 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2d480af000 tx=0x7f2d480b2040 comp rx=0 tx=0).stop 2024-09-17T05:18:38.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.158+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/1642296062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d580732b0 msgr2=0x7f2d581f3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:38.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.158+0000 7f2d5d9cf640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d580732b0 0x7f2d581f3970 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f2d5000cc60 tx=0x7f2d5000cd50 comp rx=0 tx=0).stop 2024-09-17T05:18:38.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.158+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/1642296062 shutdown_connections 2024-09-17T05:18:38.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.158+0000 7f2d5d9cf640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2d38073230 0x7f2d38075740 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:38.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.158+0000 7f2d5d9cf640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d580c97a0 0x7f2d5806f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:38.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.158+0000 7f2d5d9cf640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d580732b0 0x7f2d581f3970 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:38.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.158+0000 7f2d5d9cf640 1 --2- 172.21.15.150:0/1642296062 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f2d58072ea0 0x7f2d58147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:38.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.158+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/1642296062 >> 172.21.15.150:0/1642296062 conn(0x7f2d580abb00 msgr2=0x7f2d58154620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:38.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.158+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/1642296062 shutdown_connections 2024-09-17T05:18:38.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:38.158+0000 7f2d5d9cf640 1 -- 172.21.15.150:0/1642296062 wait complete. 2024-09-17T05:18:38.173 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-09-17T05:18:40.174 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:40.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.350+0000 7f133b9c6640 1 -- 172.21.15.150:0/3440092008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1334152cf0 msgr2=0x7f1334153100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:40.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.350+0000 7f133b9c6640 1 --2- 172.21.15.150:0/3440092008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1334152cf0 0x7f1334153100 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f1324009bc0 tx=0x7f132401cba0 comp rx=0 tx=0).stop 2024-09-17T05:18:40.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.350+0000 7f133b9c6640 1 -- 172.21.15.150:0/3440092008 shutdown_connections 2024-09-17T05:18:40.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.350+0000 7f133b9c6640 1 --2- 172.21.15.150:0/3440092008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1334152cf0 0x7f1334153100 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:40.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.350+0000 7f133b9c6640 1 -- 172.21.15.150:0/3440092008 >> 172.21.15.150:0/3440092008 conn(0x7f13340c4a00 msgr2=0x7f13340c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:40.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.350+0000 7f133b9c6640 1 -- 172.21.15.150:0/3440092008 shutdown_connections 2024-09-17T05:18:40.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.350+0000 7f133b9c6640 1 -- 172.21.15.150:0/3440092008 wait complete. 2024-09-17T05:18:40.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.350+0000 7f133b9c6640 1 Processor -- start 2024-09-17T05:18:40.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.350+0000 7f133b9c6640 1 -- start start 2024-09-17T05:18:40.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f133b9c6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13340a7350 0x7f13340c17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:40.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f133b9c6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13340c42e0 0x7f1334148c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:40.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f133b9c6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1334152cf0 0x7f1334144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:40.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f133b9c6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1334150770 con 0x7f1334152cf0 2024-09-17T05:18:40.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f133b9c6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f13341508f0 con 0x7f13340a7350 2024-09-17T05:18:40.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f133b9c6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1334160ea0 con 0x7f13340c42e0 2024-09-17T05:18:40.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1339f63640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1334152cf0 0x7f1334144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:40.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1339f63640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1334152cf0 0x7f1334144d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53030/0 (socket says 172.21.15.150:53030) 2024-09-17T05:18:40.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1339f63640 1 -- 172.21.15.150:0/3905376951 learned_addr learned my addr 172.21.15.150:0/3905376951 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:40.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1339762640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13340a7350 0x7f13340c17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:40.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1338f61640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13340c42e0 0x7f1334148c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:40.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1339f63640 1 -- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13340c42e0 msgr2=0x7f1334148c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:40.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1339f63640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13340c42e0 0x7f1334148c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:40.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1339f63640 1 -- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13340a7350 msgr2=0x7f13340c17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:40.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1339f63640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13340a7350 0x7f13340c17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:40.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1339f63640 1 -- 172.21.15.150:0/3905376951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13341f2950 con 0x7f1334152cf0 2024-09-17T05:18:40.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1338f61640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13340c42e0 0x7f1334148c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:18:40.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1339f63640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1334152cf0 0x7f1334144d20 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f13240b0040 tx=0x7f1324009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:40.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1331ffb640 1 -- 172.21.15.150:0/3905376951 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13240b2040 con 0x7f1334152cf0 2024-09-17T05:18:40.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1331ffb640 1 -- 172.21.15.150:0/3905376951 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f13240bb070 con 0x7f1334152cf0 2024-09-17T05:18:40.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f133b9c6640 1 -- 172.21.15.150:0/3905376951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f133414f240 con 0x7f1334152cf0 2024-09-17T05:18:40.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1331ffb640 1 -- 172.21.15.150:0/3905376951 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324004630 con 0x7f1334152cf0 2024-09-17T05:18:40.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f133b9c6640 1 -- 172.21.15.150:0/3905376951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f133414cdf0 con 0x7f1334152cf0 2024-09-17T05:18:40.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f133b9c6640 1 -- 172.21.15.150:0/3905376951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f133414cfc0 con 0x7f1334152cf0 2024-09-17T05:18:40.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.354+0000 7f1331ffb640 1 -- 172.21.15.150:0/3905376951 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f13240a5d10 con 0x7f1334152cf0 2024-09-17T05:18:40.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.358+0000 7f1331ffb640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1320073200 0x7f1320075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:40.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.358+0000 7f1331ffb640 1 -- 172.21.15.150:0/3905376951 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7f132412fc70 con 0x7f1334152cf0 2024-09-17T05:18:40.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.358+0000 7f1339762640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1320073200 0x7f1320075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:40.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.358+0000 7f1339762640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1320073200 0x7f1320075710 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f132c00b810 tx=0x7f132c008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:40.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.362+0000 7f1331ffb640 1 -- 172.21.15.150:0/3905376951 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1324016790 con 0x7f1334152cf0 2024-09-17T05:18:40.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.566+0000 7f133b9c6640 1 -- 172.21.15.150:0/3905376951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1334153100 con 0x7f1334152cf0 2024-09-17T05:18:40.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f1331ffb640 1 -- 172.21.15.150:0/3905376951 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) ==== 95+0+5723 (secure 0 0 0) 0x7f13240f9bf0 con 0x7f1334152cf0 2024-09-17T05:18:40.565 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:40.565 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":203,"btime":"2024-09-17T05:18:38:866863+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:18:40.566 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 203 2024-09-17T05:18:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f133b9c6640 1 -- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1320073200 msgr2=0x7f1320075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f133b9c6640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1320073200 0x7f1320075710 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f132c00b810 tx=0x7f132c008040 comp rx=0 tx=0).stop 2024-09-17T05:18:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f133b9c6640 1 -- 172.21.15.150:0/3905376951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1334152cf0 msgr2=0x7f1334144d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f133b9c6640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1334152cf0 0x7f1334144d20 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f13240b0040 tx=0x7f1324009310 comp rx=0 tx=0).stop 2024-09-17T05:18:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f133b9c6640 1 -- 172.21.15.150:0/3905376951 shutdown_connections 2024-09-17T05:18:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f133b9c6640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1320073200 0x7f1320075710 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f133b9c6640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1334152cf0 0x7f1334144d20 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f133b9c6640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f13340c42e0 0x7f1334148c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f133b9c6640 1 --2- 172.21.15.150:0/3905376951 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f13340a7350 0x7f13340c17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f133b9c6640 1 -- 172.21.15.150:0/3905376951 >> 172.21.15.150:0/3905376951 conn(0x7f13340c4a00 msgr2=0x7f13340c6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f133b9c6640 1 -- 172.21.15.150:0/3905376951 shutdown_connections 2024-09-17T05:18:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:40.570+0000 7f133b9c6640 1 -- 172.21.15.150:0/3905376951 wait complete. 2024-09-17T05:18:40.587 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-09-17T05:18:40.588 INFO:tasks.mds_thrash.fs.[b]:waiting for 97 secs before thrashing 2024-09-17T05:18:41.050 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-09-17T05:18:41.051 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T05:18:41.051 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T05:18:41.053 INFO:tasks.ceph.mds.c:Started 2024-09-17T05:18:41.054 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:41.154 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T05:18:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 -- 172.21.15.150:0/3573302329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080c97a0 msgr2=0x7f0008151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 --2- 172.21.15.150:0/3573302329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080c97a0 0x7f0008151c70 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7efff8009900 tx=0x7efff801c700 comp rx=0 tx=0).stop 2024-09-17T05:18:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 -- 172.21.15.150:0/3573302329 shutdown_connections 2024-09-17T05:18:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 --2- 172.21.15.150:0/3573302329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00080732b0 0x7f00081582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 --2- 172.21.15.150:0/3573302329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080c97a0 0x7f0008151c70 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 -- 172.21.15.150:0/3573302329 >> 172.21.15.150:0/3573302329 conn(0x7f00080abb00 msgr2=0x7f00080abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 -- 172.21.15.150:0/3573302329 shutdown_connections 2024-09-17T05:18:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 -- 172.21.15.150:0/3573302329 wait complete. 2024-09-17T05:18:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 Processor -- start 2024-09-17T05:18:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 -- start start 2024-09-17T05:18:41.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0008072ea0 0x7f00081edd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:41.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00080732b0 0x7f00081568c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:41.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00080c97a0 0x7f000814d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:41.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0008152260 con 0x7f0008072ea0 2024-09-17T05:18:41.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f00080c4df0 con 0x7f00080c97a0 2024-09-17T05:18:41.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f000de28640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0008158830 con 0x7f00080732b0 2024-09-17T05:18:41.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f00077fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0008072ea0 0x7f00081edd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:41.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f00077fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0008072ea0 0x7f00081edd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53060/0 (socket says 172.21.15.150:53060) 2024-09-17T05:18:41.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.226+0000 7f00077fe640 1 -- 172.21.15.150:0/628459118 learned_addr learned my addr 172.21.15.150:0/628459118 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:41.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7f0006ffd640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00080732b0 0x7f00081568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:41.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7f0007fff640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00080c97a0 0x7f000814d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:41.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7f00077fe640 1 -- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00080732b0 msgr2=0x7f00081568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:41.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7f00077fe640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00080732b0 0x7f00081568c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:41.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7f00077fe640 1 -- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00080c97a0 msgr2=0x7f000814d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:41.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7f00077fe640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00080c97a0 0x7f000814d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:41.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7f00077fe640 1 -- 172.21.15.150:0/628459118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0008156e00 con 0x7f0008072ea0 2024-09-17T05:18:41.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7f0007fff640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00080c97a0 0x7f000814d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:18:41.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7f00077fe640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0008072ea0 0x7f00081edd80 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7efff8000c10 tx=0x7efff80a6c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:41.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7effeffff640 1 -- 172.21.15.150:0/628459118 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff80ac070 con 0x7f0008072ea0 2024-09-17T05:18:41.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7effeffff640 1 -- 172.21.15.150:0/628459118 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efff8005120 con 0x7f0008072ea0 2024-09-17T05:18:41.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7effeffff640 1 -- 172.21.15.150:0/628459118 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff8005570 con 0x7f0008072ea0 2024-09-17T05:18:41.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7f000de28640 1 -- 172.21.15.150:0/628459118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00081f1cb0 con 0x7f0008072ea0 2024-09-17T05:18:41.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7f000de28640 1 -- 172.21.15.150:0/628459118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00081f1f50 con 0x7f0008072ea0 2024-09-17T05:18:41.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7f000de28640 1 -- 172.21.15.150:0/628459118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0008151c70 con 0x7f0008072ea0 2024-09-17T05:18:41.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7effeffff640 1 -- 172.21.15.150:0/628459118 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7efff80af070 con 0x7f0008072ea0 2024-09-17T05:18:41.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.230+0000 7effeffff640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efff4073230 0x7efff4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:41.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.234+0000 7effeffff640 1 -- 172.21.15.150:0/628459118 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7efff812e990 con 0x7f0008072ea0 2024-09-17T05:18:41.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.234+0000 7f0006ffd640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efff4073230 0x7efff4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:41.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.234+0000 7f0006ffd640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efff4073230 0x7efff4075740 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f00081516d0 tx=0x7efffc008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:41.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.238+0000 7effeffff640 1 -- 172.21.15.150:0/628459118 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7efff80c0020 con 0x7f0008072ea0 2024-09-17T05:18:41.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.450+0000 7f000de28640 1 -- 172.21.15.150:0/628459118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00080c40f0 con 0x7f0008072ea0 2024-09-17T05:18:41.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7effeffff640 1 -- 172.21.15.150:0/628459118 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) ==== 95+0+5723 (secure 0 0 0) 0x7f00080736c0 con 0x7f0008072ea0 2024-09-17T05:18:41.449 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:41.449 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":203,"btime":"2024-09-17T05:18:38:866863+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:18:41.449 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 203 2024-09-17T05:18:41.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7f000de28640 1 -- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efff4073230 msgr2=0x7efff4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7f000de28640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efff4073230 0x7efff4075740 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f00081516d0 tx=0x7efffc008040 comp rx=0 tx=0).stop 2024-09-17T05:18:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7f000de28640 1 -- 172.21.15.150:0/628459118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0008072ea0 msgr2=0x7f00081edd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7f000de28640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0008072ea0 0x7f00081edd80 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7efff8000c10 tx=0x7efff80a6c20 comp rx=0 tx=0).stop 2024-09-17T05:18:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7f000de28640 1 -- 172.21.15.150:0/628459118 shutdown_connections 2024-09-17T05:18:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7f000de28640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efff4073230 0x7efff4075740 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7f000de28640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00080c97a0 0x7f000814d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7f000de28640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00080732b0 0x7f00081568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7f000de28640 1 --2- 172.21.15.150:0/628459118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0008072ea0 0x7f00081edd80 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7f000de28640 1 -- 172.21.15.150:0/628459118 >> 172.21.15.150:0/628459118 conn(0x7f00080abb00 msgr2=0x7f0008157ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7f000de28640 1 -- 172.21.15.150:0/628459118 shutdown_connections 2024-09-17T05:18:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:41.454+0000 7f000de28640 1 -- 172.21.15.150:0/628459118 wait complete. 2024-09-17T05:18:41.473 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-09-17T05:18:43.474 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:18:43.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.642+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/2861073526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e40ae320 msgr2=0x7fe8e4157e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:43.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.642+0000 7fe8ec3fa640 1 --2- 172.21.15.150:0/2861073526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e40ae320 0x7fe8e4157e00 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fe8e40ae910 tx=0x7fe8d401c810 comp rx=0 tx=0).stop 2024-09-17T05:18:43.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.642+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/2861073526 shutdown_connections 2024-09-17T05:18:43.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.642+0000 7fe8ec3fa640 1 --2- 172.21.15.150:0/2861073526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e40ae320 0x7fe8e4157e00 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:43.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.642+0000 7fe8ec3fa640 1 --2- 172.21.15.150:0/2861073526 >> v2:172.21.15.176:3300/0 conn(0x7fe8e40c5d80 0x7fe8e40adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:43.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.642+0000 7fe8ec3fa640 1 --2- 172.21.15.150:0/2861073526 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8e40c6ea0 0x7fe8e40c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:43.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.642+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/2861073526 >> 172.21.15.150:0/2861073526 conn(0x7fe8e40c8db0 msgr2=0x7fe8e40c91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:43.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.642+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/2861073526 shutdown_connections 2024-09-17T05:18:43.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.642+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/2861073526 wait complete. 2024-09-17T05:18:43.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ec3fa640 1 Processor -- start 2024-09-17T05:18:43.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ec3fa640 1 -- start start 2024-09-17T05:18:43.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ec3fa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8e40ae320 0x7fe8e40bded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:43.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ec3fa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e40c5d80 0x7fe8e40a7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:43.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ec3fa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8e40c6ea0 0x7fe8e4153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:43.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ec3fa640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e406cfd0 con 0x7fe8e40c5d80 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ec3fa640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e406ce50 con 0x7fe8e40ae320 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ec3fa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e406d150 con 0x7fe8e40c6ea0 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8e9995640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e40c5d80 0x7fe8e40a7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8e9995640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e40c5d80 0x7fe8e40a7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53078/0 (socket says 172.21.15.150:53078) 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8e9995640 1 -- 172.21.15.150:0/1049272434 learned_addr learned my addr 172.21.15.150:0/1049272434 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ea997640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8e40c6ea0 0x7fe8e4153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ea196640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8e40ae320 0x7fe8e40bded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8e9995640 1 -- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8e40c6ea0 msgr2=0x7fe8e4153be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8e9995640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8e40c6ea0 0x7fe8e4153be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8e9995640 1 -- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8e40ae320 msgr2=0x7fe8e40bded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8e9995640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8e40ae320 0x7fe8e40bded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8e9995640 1 -- 172.21.15.150:0/1049272434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e41558b0 con 0x7fe8e40c5d80 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ea196640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8e40ae320 0x7fe8e40bded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ea997640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8e40c6ea0 0x7fe8e4153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:18:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8e9995640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e40c5d80 0x7fe8e40a7ce0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800d660 tx=0x7fe8d800db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:43.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8e2ffd640 1 -- 172.21.15.150:0/1049272434 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8017070 con 0x7fe8e40c5d80 2024-09-17T05:18:43.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8e2ffd640 1 -- 172.21.15.150:0/1049272434 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe8d8009b10 con 0x7fe8e40c5d80 2024-09-17T05:18:43.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/1049272434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e40c00f0 con 0x7fe8e40c5d80 2024-09-17T05:18:43.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8e2ffd640 1 -- 172.21.15.150:0/1049272434 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8010460 con 0x7fe8e40c5d80 2024-09-17T05:18:43.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.646+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/1049272434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e40c06b0 con 0x7fe8e40c5d80 2024-09-17T05:18:43.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.650+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/1049272434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8e4155620 con 0x7fe8e40c5d80 2024-09-17T05:18:43.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.650+0000 7fe8e2ffd640 1 -- 172.21.15.150:0/1049272434 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe8d801a070 con 0x7fe8e40c5d80 2024-09-17T05:18:43.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.650+0000 7fe8e2ffd640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe8d0073200 0x7fe8d0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:18:43.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.650+0000 7fe8e2ffd640 1 -- 172.21.15.150:0/1049272434 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7fe8d8095430 con 0x7fe8e40c5d80 2024-09-17T05:18:43.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.650+0000 7fe8ea196640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe8d0073200 0x7fe8d0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:18:43.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.650+0000 7fe8ea196640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe8d0073200 0x7fe8d0075710 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc008d60 tx=0x7fe8dc0082b0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:18:43.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.654+0000 7fe8e2ffd640 1 -- 172.21.15.150:0/1049272434 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe8d805f3b0 con 0x7fe8e40c5d80 2024-09-17T05:18:43.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.862+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/1049272434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8e40adde0 con 0x7fe8e40c5d80 2024-09-17T05:18:43.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.862+0000 7fe8e2ffd640 1 -- 172.21.15.150:0/1049272434 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+6552 (secure 0 0 0) 0x7fe8d805ecb0 con 0x7fe8e40c5d80 2024-09-17T05:18:43.859 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:18:43.860 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":204,"btime":"2024-09-17T05:18:41:892667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":8199,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:18:43.860 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 204 2024-09-17T05:18:43.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.866+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe8d0073200 msgr2=0x7fe8d0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:43.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.866+0000 7fe8ec3fa640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe8d0073200 0x7fe8d0075710 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc008d60 tx=0x7fe8dc0082b0 comp rx=0 tx=0).stop 2024-09-17T05:18:43.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.866+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/1049272434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e40c5d80 msgr2=0x7fe8e40a7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:18:43.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.866+0000 7fe8ec3fa640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e40c5d80 0x7fe8e40a7ce0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800d660 tx=0x7fe8d800db20 comp rx=0 tx=0).stop 2024-09-17T05:18:43.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.866+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/1049272434 shutdown_connections 2024-09-17T05:18:43.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.866+0000 7fe8ec3fa640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe8d0073200 0x7fe8d0075710 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:43.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.866+0000 7fe8ec3fa640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe8e40c6ea0 0x7fe8e4153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:43.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.866+0000 7fe8ec3fa640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e40c5d80 0x7fe8e40a7ce0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:43.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.866+0000 7fe8ec3fa640 1 --2- 172.21.15.150:0/1049272434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8e40ae320 0x7fe8e40bded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:18:43.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.866+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/1049272434 >> 172.21.15.150:0/1049272434 conn(0x7fe8e40c8db0 msgr2=0x7fe8e40c91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:18:43.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.866+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/1049272434 shutdown_connections 2024-09-17T05:18:43.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:18:43.866+0000 7fe8ec3fa640 1 -- 172.21.15.150:0/1049272434 wait complete. 2024-09-17T05:18:43.882 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-09-17T05:18:43.882 INFO:tasks.mds_thrash.fs.[a]:waiting for 50 secs before thrashing 2024-09-17T05:19:07.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:19:07.182 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:19:33.884 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:19:34.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.053+0000 7f83e76cd640 1 -- 172.21.15.150:0/2423380057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00732b0 msgr2=0x7f83e00c83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:34.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.053+0000 7f83e76cd640 1 --2- 172.21.15.150:0/2423380057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00732b0 0x7f83e00c83e0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009bf0 tx=0x7f83d801cb10 comp rx=0 tx=0).stop 2024-09-17T05:19:34.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.053+0000 7f83e76cd640 1 -- 172.21.15.150:0/2423380057 shutdown_connections 2024-09-17T05:19:34.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.053+0000 7f83e76cd640 1 --2- 172.21.15.150:0/2423380057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00732b0 0x7f83e00c83e0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.053+0000 7f83e76cd640 1 -- 172.21.15.150:0/2423380057 >> 172.21.15.150:0/2423380057 conn(0x7f83e00abb00 msgr2=0x7f83e00abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:34.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.053+0000 7f83e76cd640 1 -- 172.21.15.150:0/2423380057 shutdown_connections 2024-09-17T05:19:34.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.053+0000 7f83e76cd640 1 -- 172.21.15.150:0/2423380057 wait complete. 2024-09-17T05:19:34.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e76cd640 1 Processor -- start 2024-09-17T05:19:34.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e76cd640 1 -- start start 2024-09-17T05:19:34.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e76cd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83e0072ea0 0x7f83e01473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:34.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e76cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00732b0 0x7f83e0150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:34.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e76cd640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83e00c97a0 0x7f83e0148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:34.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e76cd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f83e00c8aa0 con 0x7f83e00732b0 2024-09-17T05:19:34.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e76cd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f83e00c8920 con 0x7f83e0072ea0 2024-09-17T05:19:34.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e76cd640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f83e01627a0 con 0x7f83e00c97a0 2024-09-17T05:19:34.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e4c68640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00732b0 0x7f83e0150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:34.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e4c68640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00732b0 0x7f83e0150580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43356/0 (socket says 172.21.15.150:43356) 2024-09-17T05:19:34.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e4c68640 1 -- 172.21.15.150:0/556826080 learned_addr learned my addr 172.21.15.150:0/556826080 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:19:34.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e5c6a640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83e00c97a0 0x7f83e0148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:34.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e5469640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83e0072ea0 0x7f83e01473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:34.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e4c68640 1 -- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83e00c97a0 msgr2=0x7f83e0148950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:34.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e4c68640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83e00c97a0 0x7f83e0148950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e4c68640 1 -- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83e0072ea0 msgr2=0x7f83e01473c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:34.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e4c68640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83e0072ea0 0x7f83e01473c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e4c68640 1 -- 172.21.15.150:0/556826080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e01f1ea0 con 0x7f83e00732b0 2024-09-17T05:19:34.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e5469640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83e0072ea0 0x7f83e01473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:19:34.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e4c68640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00732b0 0x7f83e0150580 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f83d000c760 tx=0x7f83d000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:34.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83d5ffb640 1 -- 172.21.15.150:0/556826080 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d0006aa0 con 0x7f83e00732b0 2024-09-17T05:19:34.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83d5ffb640 1 -- 172.21.15.150:0/556826080 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83d0006c40 con 0x7f83e00732b0 2024-09-17T05:19:34.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e76cd640 1 -- 172.21.15.150:0/556826080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e0149eb0 con 0x7f83e00732b0 2024-09-17T05:19:34.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83d5ffb640 1 -- 172.21.15.150:0/556826080 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d0007290 con 0x7f83e00732b0 2024-09-17T05:19:34.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e76cd640 1 -- 172.21.15.150:0/556826080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e014a470 con 0x7f83e00732b0 2024-09-17T05:19:34.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.057+0000 7f83e76cd640 1 -- 172.21.15.150:0/556826080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e01f1c10 con 0x7f83e00732b0 2024-09-17T05:19:34.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.061+0000 7f83d5ffb640 1 -- 172.21.15.150:0/556826080 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f83d0007430 con 0x7f83e00732b0 2024-09-17T05:19:34.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.061+0000 7f83d5ffb640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f83c4073230 0x7f83c4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:34.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.061+0000 7f83d5ffb640 1 -- 172.21.15.150:0/556826080 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7f83d0094ce0 con 0x7f83e00732b0 2024-09-17T05:19:34.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.061+0000 7f83e5469640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f83c4073230 0x7f83c4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:34.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.061+0000 7f83e5469640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f83c4073230 0x7f83c4075740 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f83dc001630 tx=0x7f83dc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:34.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.065+0000 7f83d5ffb640 1 -- 172.21.15.150:0/556826080 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f83e00736c0 con 0x7f83e00732b0 2024-09-17T05:19:34.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.269+0000 7f83e76cd640 1 -- 172.21.15.150:0/556826080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83e0151c70 con 0x7f83e00732b0 2024-09-17T05:19:34.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83d5ffb640 1 -- 172.21.15.150:0/556826080 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+6552 (secure 0 0 0) 0x7f83d005ecf0 con 0x7f83e00732b0 2024-09-17T05:19:34.269 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:19:34.269 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":204,"btime":"2024-09-17T05:18:41:892667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":8199,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:19:34.270 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 204 2024-09-17T05:19:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83e76cd640 1 -- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f83c4073230 msgr2=0x7f83c4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83e76cd640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f83c4073230 0x7f83c4075740 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f83dc001630 tx=0x7f83dc009040 comp rx=0 tx=0).stop 2024-09-17T05:19:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83e76cd640 1 -- 172.21.15.150:0/556826080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00732b0 msgr2=0x7f83e0150580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83e76cd640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00732b0 0x7f83e0150580 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f83d000c760 tx=0x7f83d000cc20 comp rx=0 tx=0).stop 2024-09-17T05:19:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83e76cd640 1 -- 172.21.15.150:0/556826080 shutdown_connections 2024-09-17T05:19:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83e76cd640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f83c4073230 0x7f83c4075740 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83e76cd640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f83e00c97a0 0x7f83e0148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83e76cd640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00732b0 0x7f83e0150580 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83e76cd640 1 --2- 172.21.15.150:0/556826080 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f83e0072ea0 0x7f83e01473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83e76cd640 1 -- 172.21.15.150:0/556826080 >> 172.21.15.150:0/556826080 conn(0x7f83e00abb00 msgr2=0x7f83e00c5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:34.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83e76cd640 1 -- 172.21.15.150:0/556826080 shutdown_connections 2024-09-17T05:19:34.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.273+0000 7f83e76cd640 1 -- 172.21.15.150:0/556826080 wait complete. 2024-09-17T05:19:34.291 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-09-17T05:19:34.292 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T05:19:34.292 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:19:34.322 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T05:19:34.322 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:19:34.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.481+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3918944887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b881505a0 msgr2=0x7f3b88152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:34.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.481+0000 7f3b8dc49640 1 --2- 172.21.15.150:0/3918944887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b881505a0 0x7f3b88152cf0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f3b8806ffd0 tx=0x7f3b8001dc70 comp rx=0 tx=0).stop 2024-09-17T05:19:34.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.481+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3918944887 shutdown_connections 2024-09-17T05:19:34.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.481+0000 7f3b8dc49640 1 --2- 172.21.15.150:0/3918944887 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b881505a0 0x7f3b88152cf0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.481+0000 7f3b8dc49640 1 --2- 172.21.15.150:0/3918944887 >> v2:172.21.15.176:3300/0 conn(0x7f3b880a7350 0x7f3b88150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.481+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3918944887 >> 172.21.15.150:0/3918944887 conn(0x7f3b880c4a00 msgr2=0x7f3b880c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:34.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.481+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3918944887 shutdown_connections 2024-09-17T05:19:34.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.481+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3918944887 wait complete. 2024-09-17T05:19:34.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.481+0000 7f3b8dc49640 1 Processor -- start 2024-09-17T05:19:34.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.481+0000 7f3b8dc49640 1 -- start start 2024-09-17T05:19:34.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b8dc49640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b880a7350 0x7f3b880c11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:34.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b8dc49640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b880c42e0 0x7f3b88073860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:34.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b8dc49640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b881505a0 0x7f3b8814c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:34.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b8dc49640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3b881537c0 con 0x7f3b880c42e0 2024-09-17T05:19:34.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b8dc49640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3b88153640 con 0x7f3b880a7350 2024-09-17T05:19:34.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b8dc49640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3b8815f680 con 0x7f3b881505a0 2024-09-17T05:19:34.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b7ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b880c42e0 0x7f3b88073860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:34.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b7ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b880c42e0 0x7f3b88073860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43374/0 (socket says 172.21.15.150:43374) 2024-09-17T05:19:34.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b7ffff640 1 -- 172.21.15.150:0/3735326877 learned_addr learned my addr 172.21.15.150:0/3735326877 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:19:34.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b877fe640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b880a7350 0x7f3b880c11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:34.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b87fff640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b881505a0 0x7f3b8814c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b7ffff640 1 -- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b881505a0 msgr2=0x7f3b8814c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b7ffff640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b881505a0 0x7f3b8814c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b7ffff640 1 -- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b880a7350 msgr2=0x7f3b880c11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b7ffff640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b880a7350 0x7f3b880c11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b7ffff640 1 -- 172.21.15.150:0/3735326877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b88156740 con 0x7f3b880c42e0 2024-09-17T05:19:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b7ffff640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b880c42e0 0x7f3b88073860 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000d720 tx=0x7f3b7000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b84ff9640 1 -- 172.21.15.150:0/3735326877 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70017070 con 0x7f3b880c42e0 2024-09-17T05:19:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b84ff9640 1 -- 172.21.15.150:0/3735326877 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b70009d20 con 0x7f3b880c42e0 2024-09-17T05:19:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3735326877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b880bf810 con 0x7f3b880c42e0 2024-09-17T05:19:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b84ff9640 1 -- 172.21.15.150:0/3735326877 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70005280 con 0x7f3b880c42e0 2024-09-17T05:19:34.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3735326877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b880acd40 con 0x7f3b880c42e0 2024-09-17T05:19:34.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3735326877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b8814d580 con 0x7f3b880c42e0 2024-09-17T05:19:34.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b84ff9640 1 -- 172.21.15.150:0/3735326877 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3b70005420 con 0x7f3b880c42e0 2024-09-17T05:19:34.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.485+0000 7f3b84ff9640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3b68073200 0x7f3b68075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:34.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.489+0000 7f3b84ff9640 1 -- 172.21.15.150:0/3735326877 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7f3b700954e0 con 0x7f3b880c42e0 2024-09-17T05:19:34.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.489+0000 7f3b877fe640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3b68073200 0x7f3b68075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:34.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.489+0000 7f3b877fe640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3b68073200 0x7f3b68075710 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f3b7800b810 tx=0x7f3b78008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:34.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.493+0000 7f3b84ff9640 1 -- 172.21.15.150:0/3735326877 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3b7005f350 con 0x7f3b880c42e0 2024-09-17T05:19:34.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.697+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3735326877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b88150060 con 0x7f3b880c42e0 2024-09-17T05:19:34.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.697+0000 7f3b84ff9640 1 -- 172.21.15.150:0/3735326877 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+6552 (secure 0 0 0) 0x7f3b7005ec50 con 0x7f3b880c42e0 2024-09-17T05:19:34.696 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:19:34.697 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":204,"btime":"2024-09-17T05:18:41:892667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":8199,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:19:34.697 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 204 2024-09-17T05:19:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.701+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3b68073200 msgr2=0x7f3b68075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.701+0000 7f3b8dc49640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3b68073200 0x7f3b68075710 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f3b7800b810 tx=0x7f3b78008040 comp rx=0 tx=0).stop 2024-09-17T05:19:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.701+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3735326877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b880c42e0 msgr2=0x7f3b88073860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.701+0000 7f3b8dc49640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b880c42e0 0x7f3b88073860 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000d720 tx=0x7f3b7000dbe0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.701+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3735326877 shutdown_connections 2024-09-17T05:19:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.701+0000 7f3b8dc49640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3b68073200 0x7f3b68075710 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.701+0000 7f3b8dc49640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3b881505a0 0x7f3b8814c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.701+0000 7f3b8dc49640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b880c42e0 0x7f3b88073860 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.701+0000 7f3b8dc49640 1 --2- 172.21.15.150:0/3735326877 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b880a7350 0x7f3b880c11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:34.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.701+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3735326877 >> 172.21.15.150:0/3735326877 conn(0x7f3b880c4a00 msgr2=0x7f3b880c35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:34.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.701+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3735326877 shutdown_connections 2024-09-17T05:19:34.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:34.701+0000 7f3b8dc49640 1 -- 172.21.15.150:0/3735326877 wait complete. 2024-09-17T05:19:34.718 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:19:36.720 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:19:36.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.889+0000 7fdd1b61a640 1 -- 172.21.15.150:0/3814141510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140717a0 msgr2=0x7fdd14074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:36.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.889+0000 7fdd1b61a640 1 --2- 172.21.15.150:0/3814141510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140717a0 0x7fdd14074450 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fdd14070590 tx=0x7fdd0401c830 comp rx=0 tx=0).stop 2024-09-17T05:19:36.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.889+0000 7fdd1b61a640 1 -- 172.21.15.150:0/3814141510 shutdown_connections 2024-09-17T05:19:36.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.889+0000 7fdd1b61a640 1 --2- 172.21.15.150:0/3814141510 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140717a0 0x7fdd14074450 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:36.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.889+0000 7fdd1b61a640 1 --2- 172.21.15.150:0/3814141510 >> v2:172.21.15.176:3300/0 conn(0x7fdd14070e50 0x7fdd14071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:36.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.889+0000 7fdd1b61a640 1 -- 172.21.15.150:0/3814141510 >> 172.21.15.150:0/3814141510 conn(0x7fdd140c4a00 msgr2=0x7fdd140c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:36.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.889+0000 7fdd1b61a640 1 -- 172.21.15.150:0/3814141510 shutdown_connections 2024-09-17T05:19:36.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.889+0000 7fdd1b61a640 1 -- 172.21.15.150:0/3814141510 wait complete. 2024-09-17T05:19:36.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd1b61a640 1 Processor -- start 2024-09-17T05:19:36.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd1b61a640 1 -- start start 2024-09-17T05:19:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd1b61a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd14070e50 0x7fdd141f0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd1b61a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd140717a0 0x7fdd14155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd1b61a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd14073440 0x7fdd14150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd1b61a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdd140ad190 con 0x7fdd14073440 2024-09-17T05:19:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd1b61a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd140ad010 con 0x7fdd14070e50 2024-09-17T05:19:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd1b61a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd140ad310 con 0x7fdd140717a0 2024-09-17T05:19:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd19bb7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd14073440 0x7fdd14150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd19bb7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd14073440 0x7fdd14150440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43386/0 (socket says 172.21.15.150:43386) 2024-09-17T05:19:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd19bb7640 1 -- 172.21.15.150:0/535457889 learned_addr learned my addr 172.21.15.150:0/535457889 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:19:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd18bb5640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd140717a0 0x7fdd14155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:36.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd193b6640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd14070e50 0x7fdd141f0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:36.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd19bb7640 1 -- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd140717a0 msgr2=0x7fdd14155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:36.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd19bb7640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd140717a0 0x7fdd14155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:36.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd19bb7640 1 -- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd14070e50 msgr2=0x7fdd141f0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:36.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd19bb7640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd14070e50 0x7fdd141f0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:36.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd19bb7640 1 -- 172.21.15.150:0/535457889 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd141571e0 con 0x7fdd14073440 2024-09-17T05:19:36.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd19bb7640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd14073440 0x7fdd14150440 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009850 tx=0x7fdd040a5640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:36.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd11ffb640 1 -- 172.21.15.150:0/535457889 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd040bb070 con 0x7fdd14073440 2024-09-17T05:19:36.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd11ffb640 1 -- 172.21.15.150:0/535457889 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd040043b0 con 0x7fdd14073440 2024-09-17T05:19:36.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd1b61a640 1 -- 172.21.15.150:0/535457889 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd141f1900 con 0x7fdd14073440 2024-09-17T05:19:36.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd11ffb640 1 -- 172.21.15.150:0/535457889 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd040b2040 con 0x7fdd14073440 2024-09-17T05:19:36.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd1b61a640 1 -- 172.21.15.150:0/535457889 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd14149e20 con 0x7fdd14073440 2024-09-17T05:19:36.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.893+0000 7fdd1b61a640 1 -- 172.21.15.150:0/535457889 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd140ad850 con 0x7fdd14073440 2024-09-17T05:19:36.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.897+0000 7fdd11ffb640 1 -- 172.21.15.150:0/535457889 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fdd040be070 con 0x7fdd14073440 2024-09-17T05:19:36.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.897+0000 7fdd11ffb640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd00073200 0x7fdd00075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:36.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.897+0000 7fdd11ffb640 1 -- 172.21.15.150:0/535457889 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7fdd0412ed70 con 0x7fdd14073440 2024-09-17T05:19:36.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.897+0000 7fdd193b6640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd00073200 0x7fdd00075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:36.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.897+0000 7fdd193b6640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd00073200 0x7fdd00075710 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fdd141513e0 tx=0x7fdd0c008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:36.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:36.901+0000 7fdd11ffb640 1 -- 172.21.15.150:0/535457889 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdd040164b0 con 0x7fdd14073440 2024-09-17T05:19:37.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.097+0000 7fdd1b61a640 1 -- 172.21.15.150:0/535457889 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd14150f50 con 0x7fdd14073440 2024-09-17T05:19:37.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd11ffb640 1 -- 172.21.15.150:0/535457889 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+6552 (secure 0 0 0) 0x7fdd040f8d00 con 0x7fdd14073440 2024-09-17T05:19:37.097 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:19:37.098 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":204,"btime":"2024-09-17T05:18:41:892667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":8199,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:19:37.098 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 204 2024-09-17T05:19:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd1b61a640 1 -- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd00073200 msgr2=0x7fdd00075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:37.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd1b61a640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd00073200 0x7fdd00075710 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fdd141513e0 tx=0x7fdd0c008040 comp rx=0 tx=0).stop 2024-09-17T05:19:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd1b61a640 1 -- 172.21.15.150:0/535457889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd14073440 msgr2=0x7fdd14150440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd1b61a640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd14073440 0x7fdd14150440 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009850 tx=0x7fdd040a5640 comp rx=0 tx=0).stop 2024-09-17T05:19:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd1b61a640 1 -- 172.21.15.150:0/535457889 shutdown_connections 2024-09-17T05:19:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd1b61a640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd00073200 0x7fdd00075710 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd1b61a640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd14073440 0x7fdd14150440 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd1b61a640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd140717a0 0x7fdd14155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd1b61a640 1 --2- 172.21.15.150:0/535457889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd14070e50 0x7fdd141f0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd1b61a640 1 -- 172.21.15.150:0/535457889 >> 172.21.15.150:0/535457889 conn(0x7fdd140c4a00 msgr2=0x7fdd14152d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd1b61a640 1 -- 172.21.15.150:0/535457889 shutdown_connections 2024-09-17T05:19:37.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:37.101+0000 7fdd1b61a640 1 -- 172.21.15.150:0/535457889 wait complete. 2024-09-17T05:19:37.117 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:19:37.231 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:19:37.234 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:19:39.118 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:19:39.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.285+0000 7f00d2f64640 1 -- 172.21.15.150:0/1528568549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00cc0c8cc0 msgr2=0x7f00cc150570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:39.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.285+0000 7f00d2f64640 1 --2- 172.21.15.150:0/1528568549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00cc0c8cc0 0x7f00cc150570 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f00c0009bc0 tx=0x7f00c001cae0 comp rx=0 tx=0).stop 2024-09-17T05:19:39.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.285+0000 7f00d2f64640 1 -- 172.21.15.150:0/1528568549 shutdown_connections 2024-09-17T05:19:39.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.285+0000 7f00d2f64640 1 --2- 172.21.15.150:0/1528568549 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00cc0c8cc0 0x7f00cc150570 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:39.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.285+0000 7f00d2f64640 1 --2- 172.21.15.150:0/1528568549 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00cc156900 0x7f00cc0c3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:39.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.285+0000 7f00d2f64640 1 -- 172.21.15.150:0/1528568549 >> 172.21.15.150:0/1528568549 conn(0x7f00cc06f760 msgr2=0x7f00cc074580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:39.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.285+0000 7f00d2f64640 1 -- 172.21.15.150:0/1528568549 shutdown_connections 2024-09-17T05:19:39.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.285+0000 7f00d2f64640 1 -- 172.21.15.150:0/1528568549 wait complete. 2024-09-17T05:19:39.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d2f64640 1 Processor -- start 2024-09-17T05:19:39.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d2f64640 1 -- start start 2024-09-17T05:19:39.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d2f64640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00cc0c8cc0 0x7f00cc1529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:39.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d2f64640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00cc0c90d0 0x7f00cc14fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:39.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d2f64640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00cc156900 0x7f00cc1f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:39.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d2f64640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc0b1040 con 0x7f00cc156900 2024-09-17T05:19:39.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d2f64640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc0c41f0 con 0x7f00cc0c90d0 2024-09-17T05:19:39.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d2f64640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f00cc0b11c0 con 0x7f00cc0c8cc0 2024-09-17T05:19:39.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d1501640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00cc156900 0x7f00cc1f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:39.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d1501640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00cc156900 0x7f00cc1f3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37440/0 (socket says 172.21.15.150:37440) 2024-09-17T05:19:39.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d1501640 1 -- 172.21.15.150:0/2595736743 learned_addr learned my addr 172.21.15.150:0/2595736743 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:19:39.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d0d00640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00cc0c8cc0 0x7f00cc1529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00cbfff640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00cc0c90d0 0x7f00cc14fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d1501640 1 -- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00cc0c8cc0 msgr2=0x7f00cc1529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d1501640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00cc0c8cc0 0x7f00cc1529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d1501640 1 -- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00cc0c90d0 msgr2=0x7f00cc14fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d1501640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00cc0c90d0 0x7f00cc14fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d1501640 1 -- 172.21.15.150:0/2595736743 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00cc153470 con 0x7f00cc156900 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00cbfff640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00cc0c90d0 0x7f00cc14fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d0d00640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00cc0c8cc0 0x7f00cc1529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00d1501640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00cc156900 0x7f00cc1f3300 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f00c400c3c0 tx=0x7f00c400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.289+0000 7f00c97fa640 1 -- 172.21.15.150:0/2595736743 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4011790 con 0x7f00cc156900 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.293+0000 7f00c97fa640 1 -- 172.21.15.150:0/2595736743 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00c4017070 con 0x7f00cc156900 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.293+0000 7f00d2f64640 1 -- 172.21.15.150:0/2595736743 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00cc153750 con 0x7f00cc156900 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.293+0000 7f00c97fa640 1 -- 172.21.15.150:0/2595736743 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4011cc0 con 0x7f00cc156900 2024-09-17T05:19:39.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.293+0000 7f00d2f64640 1 -- 172.21.15.150:0/2595736743 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00cc153d10 con 0x7f00cc156900 2024-09-17T05:19:39.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.293+0000 7f00d2f64640 1 -- 172.21.15.150:0/2595736743 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00cc150570 con 0x7f00cc156900 2024-09-17T05:19:39.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.293+0000 7f00c97fa640 1 -- 172.21.15.150:0/2595736743 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f00c401c460 con 0x7f00cc156900 2024-09-17T05:19:39.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.293+0000 7f00c97fa640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f00b8073230 0x7f00b8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:39.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.293+0000 7f00c97fa640 1 -- 172.21.15.150:0/2595736743 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7f00c4095b80 con 0x7f00cc156900 2024-09-17T05:19:39.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.293+0000 7f00d0d00640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f00b8073230 0x7f00b8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:39.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.293+0000 7f00d0d00640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f00b8073230 0x7f00b8075740 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f00cc06e040 tx=0x7f00bc0091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.297+0000 7f00c97fa640 1 -- 172.21.15.150:0/2595736743 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f00cc0c3bb0 con 0x7f00cc156900 2024-09-17T05:19:39.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.501+0000 7f00d2f64640 1 -- 172.21.15.150:0/2595736743 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00cc069130 con 0x7f00cc156900 2024-09-17T05:19:39.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.501+0000 7f00c97fa640 1 -- 172.21.15.150:0/2595736743 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+6552 (secure 0 0 0) 0x7f00c405fb80 con 0x7f00cc156900 2024-09-17T05:19:39.500 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:19:39.500 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":204,"btime":"2024-09-17T05:18:41:892667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":8199,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:19:39.501 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 204 2024-09-17T05:19:39.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.505+0000 7f00d2f64640 1 -- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f00b8073230 msgr2=0x7f00b8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:39.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.505+0000 7f00d2f64640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f00b8073230 0x7f00b8075740 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f00cc06e040 tx=0x7f00bc0091e0 comp rx=0 tx=0).stop 2024-09-17T05:19:39.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.505+0000 7f00d2f64640 1 -- 172.21.15.150:0/2595736743 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00cc156900 msgr2=0x7f00cc1f3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:39.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.505+0000 7f00d2f64640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00cc156900 0x7f00cc1f3300 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f00c400c3c0 tx=0x7f00c400c880 comp rx=0 tx=0).stop 2024-09-17T05:19:39.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.505+0000 7f00d2f64640 1 -- 172.21.15.150:0/2595736743 shutdown_connections 2024-09-17T05:19:39.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.505+0000 7f00d2f64640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f00b8073230 0x7f00b8075740 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:39.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.505+0000 7f00d2f64640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f00cc0c90d0 0x7f00cc14fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:39.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.505+0000 7f00d2f64640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00cc156900 0x7f00cc1f3300 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:39.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.505+0000 7f00d2f64640 1 --2- 172.21.15.150:0/2595736743 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f00cc0c8cc0 0x7f00cc1529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:39.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.505+0000 7f00d2f64640 1 -- 172.21.15.150:0/2595736743 >> 172.21.15.150:0/2595736743 conn(0x7f00cc06f760 msgr2=0x7f00cc0c58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:39.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.505+0000 7f00d2f64640 1 -- 172.21.15.150:0/2595736743 shutdown_connections 2024-09-17T05:19:39.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:39.505+0000 7f00d2f64640 1 -- 172.21.15.150:0/2595736743 wait complete. 2024-09-17T05:19:39.522 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:19:41.523 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:19:41.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.693+0000 7ffa01219640 1 -- 172.21.15.150:0/317537596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9fc06fab0 msgr2=0x7ff9fc06fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:41.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.693+0000 7ffa01219640 1 --2- 172.21.15.150:0/317537596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9fc06fab0 0x7ff9fc06fec0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc072ef0 tx=0x7ff9e401c8c0 comp rx=0 tx=0).stop 2024-09-17T05:19:41.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 -- 172.21.15.150:0/317537596 shutdown_connections 2024-09-17T05:19:41.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 --2- 172.21.15.150:0/317537596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9fc06fab0 0x7ff9fc06fec0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:41.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 --2- 172.21.15.150:0/317537596 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9fc0c41c0 0x7ff9fc0ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:41.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 --2- 172.21.15.150:0/317537596 >> v2:172.21.15.176:3300/0 conn(0x7ff9fc0c3db0 0x7ff9fc0ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:41.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 -- 172.21.15.150:0/317537596 >> 172.21.15.150:0/317537596 conn(0x7ff9fc0c4a00 msgr2=0x7ff9fc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:41.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 -- 172.21.15.150:0/317537596 shutdown_connections 2024-09-17T05:19:41.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 -- 172.21.15.150:0/317537596 wait complete. 2024-09-17T05:19:41.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 Processor -- start 2024-09-17T05:19:41.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 -- start start 2024-09-17T05:19:41.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9fc06fab0 0x7ff9fc073440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:41.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9fc0c3db0 0x7ff9fc0c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:41.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9fc0c41c0 0x7ff9fc1f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:41.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc0a7770 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff9fc0a75f0 con 0x7ff9fc0c3db0 2024-09-17T05:19:41.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ffa01219640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff9fc158480 con 0x7ff9fc06fab0 2024-09-17T05:19:41.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ff9fb59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9fc0c41c0 0x7ff9fc1f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:41.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ff9fb59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9fc0c41c0 0x7ff9fc1f00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37460/0 (socket says 172.21.15.150:37460) 2024-09-17T05:19:41.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ff9fb59e640 1 -- 172.21.15.150:0/2197746908 learned_addr learned my addr 172.21.15.150:0/2197746908 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ff9fa59c640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9fc0c3db0 0x7ff9fc0c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ff9fad9d640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9fc06fab0 0x7ff9fc073440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ff9fb59e640 1 -- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9fc06fab0 msgr2=0x7ff9fc073440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ff9fb59e640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9fc06fab0 0x7ff9fc073440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ff9fb59e640 1 -- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9fc0c3db0 msgr2=0x7ff9fc0c1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ff9fb59e640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9fc0c3db0 0x7ff9fc0c1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ff9fb59e640 1 -- 172.21.15.150:0/2197746908 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9fc073af0 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.697+0000 7ff9fb59e640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9fc0c41c0 0x7ff9fc1f00e0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4009850 tx=0x7ff9e40a66d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.701+0000 7ff9f37fe640 1 -- 172.21.15.150:0/2197746908 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e40bd070 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.701+0000 7ff9f37fe640 1 -- 172.21.15.150:0/2197746908 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9e4004470 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.701+0000 7ffa01219640 1 -- 172.21.15.150:0/2197746908 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9fc075270 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.701+0000 7ff9f37fe640 1 -- 172.21.15.150:0/2197746908 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4005290 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.701+0000 7ffa01219640 1 -- 172.21.15.150:0/2197746908 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9fc074d50 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.701+0000 7ffa01219640 1 -- 172.21.15.150:0/2197746908 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9fc158950 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.701+0000 7ff9f37fe640 1 -- 172.21.15.150:0/2197746908 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff9e4005430 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.701+0000 7ff9f37fe640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff9dc073200 0x7ff9dc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:41.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.701+0000 7ff9f37fe640 1 -- 172.21.15.150:0/2197746908 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7ff9e412f480 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.701+0000 7ff9fad9d640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff9dc073200 0x7ff9dc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:41.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.701+0000 7ff9fad9d640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff9dc073200 0x7ff9dc075710 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0c2500 tx=0x7ff9ec00a250 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:41.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.705+0000 7ff9f37fe640 1 -- 172.21.15.150:0/2197746908 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff9e40164b0 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 -- 172.21.15.150:0/2197746908 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9fc153a30 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ff9f37fe640 1 -- 172.21.15.150:0/2197746908 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+6552 (secure 0 0 0) 0x7ff9e40f8dc0 con 0x7ff9fc0c41c0 2024-09-17T05:19:41.910 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:19:41.910 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":204,"btime":"2024-09-17T05:18:41:892667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":8199,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:19:41.910 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 204 2024-09-17T05:19:41.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 -- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff9dc073200 msgr2=0x7ff9dc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:41.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff9dc073200 0x7ff9dc075710 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0c2500 tx=0x7ff9ec00a250 comp rx=0 tx=0).stop 2024-09-17T05:19:41.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 -- 172.21.15.150:0/2197746908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9fc0c41c0 msgr2=0x7ff9fc1f00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:41.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9fc0c41c0 0x7ff9fc1f00e0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4009850 tx=0x7ff9e40a66d0 comp rx=0 tx=0).stop 2024-09-17T05:19:41.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 -- 172.21.15.150:0/2197746908 shutdown_connections 2024-09-17T05:19:41.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff9dc073200 0x7ff9dc075710 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:41.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9fc0c3db0 0x7ff9fc0c1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:41.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9fc0c41c0 0x7ff9fc1f00e0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:41.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 --2- 172.21.15.150:0/2197746908 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff9fc06fab0 0x7ff9fc073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:41.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 -- 172.21.15.150:0/2197746908 >> 172.21.15.150:0/2197746908 conn(0x7ff9fc0c4a00 msgr2=0x7ff9fc156d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:41.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 -- 172.21.15.150:0/2197746908 shutdown_connections 2024-09-17T05:19:41.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:41.913+0000 7ffa01219640 1 -- 172.21.15.150:0/2197746908 wait complete. 2024-09-17T05:19:41.931 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:19:43.931 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:19:44.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.101+0000 7f3ddc69d640 1 -- 172.21.15.150:0/230516860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd40c7cd0 msgr2=0x7f3dd40c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:44.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.101+0000 7f3ddc69d640 1 --2- 172.21.15.150:0/230516860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd40c7cd0 0x7f3dd40c80e0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009840 tx=0x7f3dc401c820 comp rx=0 tx=0).stop 2024-09-17T05:19:44.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.101+0000 7f3ddc69d640 1 -- 172.21.15.150:0/230516860 shutdown_connections 2024-09-17T05:19:44.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.101+0000 7f3ddc69d640 1 --2- 172.21.15.150:0/230516860 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dd4072ea0 0x7f3dd40732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:44.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.101+0000 7f3ddc69d640 1 --2- 172.21.15.150:0/230516860 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd40c7cd0 0x7f3dd40c80e0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:44.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.101+0000 7f3ddc69d640 1 -- 172.21.15.150:0/230516860 >> 172.21.15.150:0/230516860 conn(0x7f3dd40abb00 msgr2=0x7f3dd40abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:44.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.101+0000 7f3ddc69d640 1 -- 172.21.15.150:0/230516860 shutdown_connections 2024-09-17T05:19:44.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.101+0000 7f3ddc69d640 1 -- 172.21.15.150:0/230516860 wait complete. 2024-09-17T05:19:44.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddc69d640 1 Processor -- start 2024-09-17T05:19:44.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddc69d640 1 -- start start 2024-09-17T05:19:44.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddc69d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dd4072ea0 0x7f3dd40c3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:44.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddc69d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd40c7cd0 0x7f3dd4149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:44.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddc69d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd40c97a0 0x7f3dd4146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:44.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddc69d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd4161910 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddc69d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd4161790 con 0x7f3dd40c97a0 2024-09-17T05:19:44.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddc69d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f3dd4161a90 con 0x7f3dd4072ea0 2024-09-17T05:19:44.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dd9c38640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd40c7cd0 0x7f3dd4149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:44.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dd9c38640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd40c7cd0 0x7f3dd4149ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37466/0 (socket says 172.21.15.150:37466) 2024-09-17T05:19:44.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dd9c38640 1 -- 172.21.15.150:0/1633111329 learned_addr learned my addr 172.21.15.150:0/1633111329 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:19:44.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dda439640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dd4072ea0 0x7f3dd40c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:44.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddac3a640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd40c97a0 0x7f3dd4146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:44.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dd9c38640 1 -- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dd4072ea0 msgr2=0x7f3dd40c3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:44.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dd9c38640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dd4072ea0 0x7f3dd40c3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:44.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dd9c38640 1 -- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd40c97a0 msgr2=0x7f3dd4146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:44.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dd9c38640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd40c97a0 0x7f3dd4146960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:44.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dd9c38640 1 -- 172.21.15.150:0/1633111329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd4149230 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dda439640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dd4072ea0 0x7f3dd40c3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:19:44.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddac3a640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd40c97a0 0x7f3dd4146960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:19:44.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dd9c38640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd40c7cd0 0x7f3dd4149ac0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f3dc800e9f0 tx=0x7f3dc800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:44.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dd2ffd640 1 -- 172.21.15.150:0/1633111329 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc800cd80 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dd2ffd640 1 -- 172.21.15.150:0/1633111329 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dc8003a70 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3dd2ffd640 1 -- 172.21.15.150:0/1633111329 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8004020 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddc69d640 1 -- 172.21.15.150:0/1633111329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dd40c3da0 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddc69d640 1 -- 172.21.15.150:0/1633111329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dd414a600 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.105+0000 7f3ddc69d640 1 -- 172.21.15.150:0/1633111329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dd40732b0 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.109+0000 7f3dd2ffd640 1 -- 172.21.15.150:0/1633111329 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f3dc80041c0 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.109+0000 7f3dd2ffd640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3dc0073230 0x7f3dc0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:44.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.109+0000 7f3dd2ffd640 1 -- 172.21.15.150:0/1633111329 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7f3dc8096120 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.109+0000 7f3dda439640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3dc0073230 0x7f3dc0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:44.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.109+0000 7f3dda439640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3dc0073230 0x7f3dc0075740 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc001230 tx=0x7f3dcc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:44.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.113+0000 7f3dd2ffd640 1 -- 172.21.15.150:0/1633111329 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3dc8016030 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3ddc69d640 1 -- 172.21.15.150:0/1633111329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3dd40c69f0 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3dd2ffd640 1 -- 172.21.15.150:0/1633111329 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+6552 (secure 0 0 0) 0x7f3dc80600c0 con 0x7f3dd40c7cd0 2024-09-17T05:19:44.318 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:19:44.318 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":204,"btime":"2024-09-17T05:18:41:892667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":8199,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:19:44.319 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 204 2024-09-17T05:19:44.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3ddc69d640 1 -- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3dc0073230 msgr2=0x7f3dc0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:44.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3ddc69d640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3dc0073230 0x7f3dc0075740 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc001230 tx=0x7f3dcc009040 comp rx=0 tx=0).stop 2024-09-17T05:19:44.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3ddc69d640 1 -- 172.21.15.150:0/1633111329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd40c7cd0 msgr2=0x7f3dd4149ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:44.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3ddc69d640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd40c7cd0 0x7f3dd4149ac0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f3dc800e9f0 tx=0x7f3dc800eeb0 comp rx=0 tx=0).stop 2024-09-17T05:19:44.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3ddc69d640 1 -- 172.21.15.150:0/1633111329 shutdown_connections 2024-09-17T05:19:44.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3ddc69d640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3dc0073230 0x7f3dc0075740 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:44.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3ddc69d640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd40c97a0 0x7f3dd4146960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:44.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3ddc69d640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd40c7cd0 0x7f3dd4149ac0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:44.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3ddc69d640 1 --2- 172.21.15.150:0/1633111329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f3dd4072ea0 0x7f3dd40c3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:44.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3ddc69d640 1 -- 172.21.15.150:0/1633111329 >> 172.21.15.150:0/1633111329 conn(0x7f3dd40abb00 msgr2=0x7f3dd40ab390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:44.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.321+0000 7f3ddc69d640 1 -- 172.21.15.150:0/1633111329 shutdown_connections 2024-09-17T05:19:44.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:44.325+0000 7f3ddc69d640 1 -- 172.21.15.150:0/1633111329 wait complete. 2024-09-17T05:19:44.341 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:19:46.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:19:46.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.513+0000 7f1e3e349640 1 -- 172.21.15.150:0/735726472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e380c41c0 msgr2=0x7f1e380ad7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:46.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.513+0000 7f1e3e349640 1 --2- 172.21.15.150:0/735726472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e380c41c0 0x7f1e380ad7e0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f1e3806f7a0 tx=0x7f1e2001cda0 comp rx=0 tx=0).stop 2024-09-17T05:19:46.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.513+0000 7f1e3e349640 1 -- 172.21.15.150:0/735726472 shutdown_connections 2024-09-17T05:19:46.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.513+0000 7f1e3e349640 1 --2- 172.21.15.150:0/735726472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e380c41c0 0x7f1e380ad7e0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:46.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.513+0000 7f1e3e349640 1 --2- 172.21.15.150:0/735726472 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e380c3db0 0x7f1e380ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:46.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.513+0000 7f1e3e349640 1 -- 172.21.15.150:0/735726472 >> 172.21.15.150:0/735726472 conn(0x7f1e380c4a00 msgr2=0x7f1e380c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:46.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.513+0000 7f1e3e349640 1 -- 172.21.15.150:0/735726472 shutdown_connections 2024-09-17T05:19:46.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.513+0000 7f1e3e349640 1 -- 172.21.15.150:0/735726472 wait complete. 2024-09-17T05:19:46.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.513+0000 7f1e3e349640 1 Processor -- start 2024-09-17T05:19:46.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.513+0000 7f1e3e349640 1 -- start start 2024-09-17T05:19:46.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3e349640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e3806fab0 0x7f1e3814d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:46.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3e349640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e380c3db0 0x7f1e38147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3e349640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e380c41c0 0x7f1e381ed0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3e349640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1e380ade50 con 0x7f1e380c41c0 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3e349640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1e380707e0 con 0x7f1e380c3db0 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3e349640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f1e380a75f0 con 0x7f1e3806fab0 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3c8e6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e380c41c0 0x7f1e381ed0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3c8e6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e380c41c0 0x7f1e381ed0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37484/0 (socket says 172.21.15.150:37484) 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3c8e6640 1 -- 172.21.15.150:0/884257423 learned_addr learned my addr 172.21.15.150:0/884257423 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e37fff640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e3806fab0 0x7f1e3814d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e377fe640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e380c3db0 0x7f1e38147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3c8e6640 1 -- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e3806fab0 msgr2=0x7f1e3814d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3c8e6640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e3806fab0 0x7f1e3814d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3c8e6640 1 -- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e380c3db0 msgr2=0x7f1e38147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:46.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3c8e6640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e380c3db0 0x7f1e38147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:46.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3c8e6640 1 -- 172.21.15.150:0/884257423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e381edc80 con 0x7f1e380c41c0 2024-09-17T05:19:46.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e377fe640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e380c3db0 0x7f1e38147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:19:46.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3c8e6640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e380c41c0 0x7f1e381ed0d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000b4c0 tx=0x7f1e3000b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:46.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e34ff9640 1 -- 172.21.15.150:0/884257423 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e30015020 con 0x7f1e380c41c0 2024-09-17T05:19:46.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e34ff9640 1 -- 172.21.15.150:0/884257423 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e30011920 con 0x7f1e380c41c0 2024-09-17T05:19:46.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3e349640 1 -- 172.21.15.150:0/884257423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e381edf60 con 0x7f1e380c41c0 2024-09-17T05:19:46.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e34ff9640 1 -- 172.21.15.150:0/884257423 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e30011d70 con 0x7f1e380c41c0 2024-09-17T05:19:46.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3e349640 1 -- 172.21.15.150:0/884257423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e38147d80 con 0x7f1e380c41c0 2024-09-17T05:19:46.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e3e349640 1 -- 172.21.15.150:0/884257423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e380ad590 con 0x7f1e380c41c0 2024-09-17T05:19:46.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.517+0000 7f1e34ff9640 1 -- 172.21.15.150:0/884257423 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f1e30018020 con 0x7f1e380c41c0 2024-09-17T05:19:46.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.521+0000 7f1e34ff9640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1e18073200 0x7f1e18075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:46.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.521+0000 7f1e34ff9640 1 -- 172.21.15.150:0/884257423 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7f1e300953c0 con 0x7f1e380c41c0 2024-09-17T05:19:46.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.521+0000 7f1e37fff640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1e18073200 0x7f1e18075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:46.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.521+0000 7f1e37fff640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1e18073200 0x7f1e18075710 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1e28005fd0 tx=0x7f1e2800c460 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:46.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.521+0000 7f1e34ff9640 1 -- 172.21.15.150:0/884257423 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1e380ad590 con 0x7f1e380c41c0 2024-09-17T05:19:46.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.717+0000 7f1e3e349640 1 -- 172.21.15.150:0/884257423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e3806fec0 con 0x7f1e380c41c0 2024-09-17T05:19:46.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e34ff9640 1 -- 172.21.15.150:0/884257423 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+6552 (secure 0 0 0) 0x7f1e3001f030 con 0x7f1e380c41c0 2024-09-17T05:19:46.717 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:19:46.717 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":204,"btime":"2024-09-17T05:18:41:892667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":8199,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:19:46.717 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 204 2024-09-17T05:19:46.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e3e349640 1 -- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1e18073200 msgr2=0x7f1e18075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:46.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e3e349640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1e18073200 0x7f1e18075710 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1e28005fd0 tx=0x7f1e2800c460 comp rx=0 tx=0).stop 2024-09-17T05:19:46.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e3e349640 1 -- 172.21.15.150:0/884257423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e380c41c0 msgr2=0x7f1e381ed0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:46.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e3e349640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e380c41c0 0x7f1e381ed0d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000b4c0 tx=0x7f1e3000b980 comp rx=0 tx=0).stop 2024-09-17T05:19:46.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e3e349640 1 -- 172.21.15.150:0/884257423 shutdown_connections 2024-09-17T05:19:46.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e3e349640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f1e18073200 0x7f1e18075710 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:46.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e3e349640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e380c3db0 0x7f1e38147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:46.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e3e349640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e380c41c0 0x7f1e381ed0d0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e3e349640 1 --2- 172.21.15.150:0/884257423 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f1e3806fab0 0x7f1e3814d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e3e349640 1 -- 172.21.15.150:0/884257423 >> 172.21.15.150:0/884257423 conn(0x7f1e380c4a00 msgr2=0x7f1e3806d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e3e349640 1 -- 172.21.15.150:0/884257423 shutdown_connections 2024-09-17T05:19:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:46.721+0000 7f1e3e349640 1 -- 172.21.15.150:0/884257423 wait complete. 2024-09-17T05:19:46.738 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:19:48.739 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:19:48.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.909+0000 7fcd171a4640 1 -- 172.21.15.150:0/3118777272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd101505a0 msgr2=0x7fcd10152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:48.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.909+0000 7fcd171a4640 1 --2- 172.21.15.150:0/3118777272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd101505a0 0x7fcd10152cf0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fcd100c7010 tx=0x7fcd0801cd60 comp rx=0 tx=0).stop 2024-09-17T05:19:48.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.909+0000 7fcd171a4640 1 -- 172.21.15.150:0/3118777272 shutdown_connections 2024-09-17T05:19:48.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.909+0000 7fcd171a4640 1 --2- 172.21.15.150:0/3118777272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd101505a0 0x7fcd10152cf0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:48.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.909+0000 7fcd171a4640 1 --2- 172.21.15.150:0/3118777272 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcd100a7350 0x7fcd10150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:48.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.909+0000 7fcd171a4640 1 --2- 172.21.15.150:0/3118777272 >> v2:172.21.15.176:3300/0 conn(0x7fcd100c42e0 0x7fcd100c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:48.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.909+0000 7fcd171a4640 1 -- 172.21.15.150:0/3118777272 >> 172.21.15.150:0/3118777272 conn(0x7fcd100c4a00 msgr2=0x7fcd100c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:48.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.909+0000 7fcd171a4640 1 -- 172.21.15.150:0/3118777272 shutdown_connections 2024-09-17T05:19:48.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.909+0000 7fcd171a4640 1 -- 172.21.15.150:0/3118777272 wait complete. 2024-09-17T05:19:48.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.909+0000 7fcd171a4640 1 Processor -- start 2024-09-17T05:19:48.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.909+0000 7fcd171a4640 1 -- start start 2024-09-17T05:19:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd171a4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd100a7350 0x7fcd100a9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd171a4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcd100c42e0 0x7fcd101eeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd171a4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcd101505a0 0x7fcd101480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd171a4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcd100c6bd0 con 0x7fcd100a7350 2024-09-17T05:19:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd171a4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcd10070b80 con 0x7fcd101505a0 2024-09-17T05:19:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd171a4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcd100c6d50 con 0x7fcd100c42e0 2024-09-17T05:19:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd14f40640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd100a7350 0x7fcd100a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd14f40640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd100a7350 0x7fcd100a9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:37508/0 (socket says 172.21.15.150:37508) 2024-09-17T05:19:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd14f40640 1 -- 172.21.15.150:0/2426714400 learned_addr learned my addr 172.21.15.150:0/2426714400 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:19:48.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd15741640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcd101505a0 0x7fcd101480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd07fff640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcd100c42e0 0x7fcd101eeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd14f40640 1 -- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcd100c42e0 msgr2=0x7fcd101eeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd14f40640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcd100c42e0 0x7fcd101eeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd14f40640 1 -- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcd101505a0 msgr2=0x7fcd101480b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd14f40640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcd101505a0 0x7fcd101480b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd14f40640 1 -- 172.21.15.150:0/2426714400 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd100adc60 con 0x7fcd100a7350 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd07fff640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcd100c42e0 0x7fcd101eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd14f40640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd100a7350 0x7fcd100a9150 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00db90 tx=0x7fcd0c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd057fa640 1 -- 172.21.15.150:0/2426714400 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c01b070 con 0x7fcd100a7350 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd057fa640 1 -- 172.21.15.150:0/2426714400 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd0c00ae50 con 0x7fcd100a7350 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd171a4640 1 -- 172.21.15.150:0/2426714400 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd10148fe0 con 0x7fcd100a7350 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd057fa640 1 -- 172.21.15.150:0/2426714400 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c00e9c0 con 0x7fcd100a7350 2024-09-17T05:19:48.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd171a4640 1 -- 172.21.15.150:0/2426714400 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd1014f2a0 con 0x7fcd100a7350 2024-09-17T05:19:48.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd171a4640 1 -- 172.21.15.150:0/2426714400 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd100c6690 con 0x7fcd100a7350 2024-09-17T05:19:48.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.913+0000 7fcd057fa640 1 -- 172.21.15.150:0/2426714400 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fcd0c01e070 con 0x7fcd100a7350 2024-09-17T05:19:48.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.917+0000 7fcd057fa640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fccfc073200 0x7fccfc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:48.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.917+0000 7fcd057fa640 1 -- 172.21.15.150:0/2426714400 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 10020+0+0 (secure 0 0 0) 0x7fcd0c095690 con 0x7fcd100a7350 2024-09-17T05:19:48.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.917+0000 7fcd07fff640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fccfc073200 0x7fccfc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:48.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.917+0000 7fcd07fff640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fccfc073200 0x7fccfc075710 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fcd00005fd0 tx=0x7fcd00005de0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:48.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:48.921+0000 7fcd057fa640 1 -- 172.21.15.150:0/2426714400 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcd0c05f050 con 0x7fcd100a7350 2024-09-17T05:19:49.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd171a4640 1 -- 172.21.15.150:0/2426714400 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd100a7130 con 0x7fcd100a7350 2024-09-17T05:19:49.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd057fa640 1 -- 172.21.15.150:0/2426714400 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+6552 (secure 0 0 0) 0x7fcd0c063a80 con 0x7fcd100a7350 2024-09-17T05:19:49.126 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:19:49.126 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":204,"btime":"2024-09-17T05:18:41:892667+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":8199,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":204}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:18:34.641247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"b","rank":0,"incarnation":199,"state":"up:active","state_seq":25,"addr":"172.21.15.176:6835/1690935336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1690935336},{"type":"v1","addr":"172.21.15.176:6835","nonce":1690935336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:19:49.126 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 204 2024-09-17T05:19:49.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd171a4640 1 -- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fccfc073200 msgr2=0x7fccfc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:49.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd171a4640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fccfc073200 0x7fccfc075710 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fcd00005fd0 tx=0x7fcd00005de0 comp rx=0 tx=0).stop 2024-09-17T05:19:49.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd171a4640 1 -- 172.21.15.150:0/2426714400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd100a7350 msgr2=0x7fcd100a9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:49.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd171a4640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd100a7350 0x7fcd100a9150 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00db90 tx=0x7fcd0c00a620 comp rx=0 tx=0).stop 2024-09-17T05:19:49.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd171a4640 1 -- 172.21.15.150:0/2426714400 shutdown_connections 2024-09-17T05:19:49.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd171a4640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fccfc073200 0x7fccfc075710 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:49.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd171a4640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcd101505a0 0x7fcd101480b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:49.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd171a4640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcd100c42e0 0x7fcd101eeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:49.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd171a4640 1 --2- 172.21.15.150:0/2426714400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd100a7350 0x7fcd100a9150 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:49.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd171a4640 1 -- 172.21.15.150:0/2426714400 >> 172.21.15.150:0/2426714400 conn(0x7fcd100c4a00 msgr2=0x7fcd10155140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:49.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.129+0000 7fcd171a4640 1 -- 172.21.15.150:0/2426714400 shutdown_connections 2024-09-17T05:19:49.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:49.133+0000 7fcd171a4640 1 -- 172.21.15.150:0/2426714400 wait complete. 2024-09-17T05:19:49.149 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-09-17T05:19:51.150 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:19:51.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.313+0000 7fef68b71640 1 -- 172.21.15.150:0/1701864957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef50005360 msgr2=0x7fef5000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:51.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.313+0000 7fef68b71640 1 --2- 172.21.15.150:0/1701864957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef50005360 0x7fef5000b9d0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fef5c009eb0 tx=0x7fef5c01d5f0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.313+0000 7fef68b71640 1 -- 172.21.15.150:0/1701864957 shutdown_connections 2024-09-17T05:19:51.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.313+0000 7fef68b71640 1 --2- 172.21.15.150:0/1701864957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef50005360 0x7fef5000b9d0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.313+0000 7fef68b71640 1 -- 172.21.15.150:0/1701864957 >> 172.21.15.150:0/1701864957 conn(0x7fef640c4a00 msgr2=0x7fef640c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:51.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.317+0000 7fef68b71640 1 -- 172.21.15.150:0/1701864957 shutdown_connections 2024-09-17T05:19:51.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.317+0000 7fef68b71640 1 -- 172.21.15.150:0/1701864957 wait complete. 2024-09-17T05:19:51.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.317+0000 7fef68b71640 1 Processor -- start 2024-09-17T05:19:51.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.317+0000 7fef68b71640 1 -- start start 2024-09-17T05:19:51.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef68b71640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef50005360 0x7fef640ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:51.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef68b71640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef640ab510 0x7fef640ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:51.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef68b71640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef641476c0 0x7fef64147ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:51.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef68b71640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fef6406f250 con 0x7fef50005360 2024-09-17T05:19:51.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef68b71640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fef640adef0 con 0x7fef640ab510 2024-09-17T05:19:51.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef68b71640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fef6406f810 con 0x7fef641476c0 2024-09-17T05:19:51.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef62d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef641476c0 0x7fef64147ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:51.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef62d9d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef641476c0 0x7fef64147ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3301/0 says I am v2:172.21.15.150:49934/0 (socket says 172.21.15.150:49934) 2024-09-17T05:19:51.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef62d9d640 1 -- 172.21.15.150:0/3554537970 learned_addr learned my addr 172.21.15.150:0/3554537970 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:19:51.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef6259c640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef50005360 0x7fef640ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:51.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef6259c640 1 -- 172.21.15.150:0/3554537970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef641476c0 msgr2=0x7fef64147ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:51.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef6259c640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef641476c0 0x7fef64147ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef6259c640 1 -- 172.21.15.150:0/3554537970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef640ab510 msgr2=0x7fef640ad3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T05:19:51.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef6259c640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef640ab510 0x7fef640ad3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef6259c640 1 -- 172.21.15.150:0/3554537970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef640acc10 con 0x7fef50005360 2024-09-17T05:19:51.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef6259c640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef50005360 0x7fef640ac6d0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fef54000fc0 tx=0x7fef5400d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:51.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef5b7fe640 1 -- 172.21.15.150:0/3554537970 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef540116a0 con 0x7fef50005360 2024-09-17T05:19:51.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef5b7fe640 1 -- 172.21.15.150:0/3554537970 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef54017020 con 0x7fef50005360 2024-09-17T05:19:51.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef68b71640 1 -- 172.21.15.150:0/3554537970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef640ab070 con 0x7fef50005360 2024-09-17T05:19:51.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef5b7fe640 1 -- 172.21.15.150:0/3554537970 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54011b70 con 0x7fef50005360 2024-09-17T05:19:51.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef68b71640 1 -- 172.21.15.150:0/3554537970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef64156440 con 0x7fef50005360 2024-09-17T05:19:51.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.321+0000 7fef68b71640 1 -- 172.21.15.150:0/3554537970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef480052a0 con 0x7fef50005360 2024-09-17T05:19:51.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.325+0000 7fef5b7fe640 1 -- 172.21.15.150:0/3554537970 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fef54011d10 con 0x7fef50005360 2024-09-17T05:19:51.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.325+0000 7fef5b7fe640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fef44073200 0x7fef44075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:51.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.325+0000 7fef5b7fe640 1 -- 172.21.15.150:0/3554537970 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7fef540988f0 con 0x7fef50005360 2024-09-17T05:19:51.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.329+0000 7fef59d9b640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fef44073200 0x7fef44075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:51.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.329+0000 7fef59d9b640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fef44073200 0x7fef44075710 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fef4c000f70 tx=0x7fef4c0066d0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:51.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.333+0000 7fef5b7fe640 1 -- 172.21.15.150:0/3554537970 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fef54021020 con 0x7fef50005360 2024-09-17T05:19:51.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.565+0000 7fef68b71640 1 -- 172.21.15.150:0/3554537970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef48005590 con 0x7fef50005360 2024-09-17T05:19:51.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.565+0000 7fef5b7fe640 1 -- 172.21.15.150:0/3554537970 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) ==== 95+0+5715 (secure 0 0 0) 0x7fef640acc10 con 0x7fef50005360 2024-09-17T05:19:51.563 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:19:51.563 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":205,"btime":"2024-09-17T05:19:51:316316+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203}],"filesystems":[{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:51.316311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:19:51.563 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 205 2024-09-17T05:19:51.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.569+0000 7fef68b71640 1 -- 172.21.15.150:0/3554537970 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fef44073200 msgr2=0x7fef44075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:51.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.569+0000 7fef68b71640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fef44073200 0x7fef44075710 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fef4c000f70 tx=0x7fef4c0066d0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.569+0000 7fef68b71640 1 -- 172.21.15.150:0/3554537970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef50005360 msgr2=0x7fef640ac6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:51.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.569+0000 7fef68b71640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef50005360 0x7fef640ac6d0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fef54000fc0 tx=0x7fef5400d9b0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.569+0000 7fef68b71640 1 -- 172.21.15.150:0/3554537970 shutdown_connections 2024-09-17T05:19:51.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.569+0000 7fef68b71640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fef44073200 0x7fef44075710 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.569+0000 7fef68b71640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef640ab510 0x7fef640ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.569+0000 7fef68b71640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fef641476c0 0x7fef64147ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.569+0000 7fef68b71640 1 --2- 172.21.15.150:0/3554537970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fef50005360 0x7fef640ac6d0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.569+0000 7fef68b71640 1 -- 172.21.15.150:0/3554537970 >> 172.21.15.150:0/3554537970 conn(0x7fef640c4a00 msgr2=0x7fef64156ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:51.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.569+0000 7fef68b71640 1 -- 172.21.15.150:0/3554537970 shutdown_connections 2024-09-17T05:19:51.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.569+0000 7fef68b71640 1 -- 172.21.15.150:0/3554537970 wait complete. 2024-09-17T05:19:51.584 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-09-17T05:19:51.584 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-09-17T05:19:51.584 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:19:51.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.725+0000 7f74333a8640 1 -- 172.21.15.150:0/3431091313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c0c5390 msgr2=0x7f742c074960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:51.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.725+0000 7f74333a8640 1 --2- 172.21.15.150:0/3431091313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c0c5390 0x7f742c074960 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f742c0c6cd0 tx=0x7f742801c800 comp rx=0 tx=0).stop 2024-09-17T05:19:51.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.725+0000 7f74333a8640 1 -- 172.21.15.150:0/3431091313 shutdown_connections 2024-09-17T05:19:51.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.725+0000 7f74333a8640 1 --2- 172.21.15.150:0/3431091313 >> v2:172.21.15.176:3300/0 conn(0x7f742c1576d0 0x7f742c0c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.725+0000 7f74333a8640 1 --2- 172.21.15.150:0/3431091313 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f742c0c8230 0x7f742c157190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.725+0000 7f74333a8640 1 --2- 172.21.15.150:0/3431091313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c0c5390 0x7f742c074960 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.725+0000 7f74333a8640 1 -- 172.21.15.150:0/3431091313 >> 172.21.15.150:0/3431091313 conn(0x7f742c06e360 msgr2=0x7f742c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:51.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.725+0000 7f74333a8640 1 -- 172.21.15.150:0/3431091313 shutdown_connections 2024-09-17T05:19:51.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.725+0000 7f74333a8640 1 -- 172.21.15.150:0/3431091313 wait complete. 2024-09-17T05:19:51.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f74333a8640 1 Processor -- start 2024-09-17T05:19:51.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f74333a8640 1 -- start start 2024-09-17T05:19:51.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f74333a8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f742c0c5390 0x7f742c159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:51.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f74333a8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c0c8230 0x7f742c0a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:51.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f74333a8640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f742c1576d0 0x7f742c0c16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:51.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f74333a8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f742c06d1e0 con 0x7f742c0c8230 2024-09-17T05:19:51.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f74333a8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f742c06d060 con 0x7f742c0c5390 2024-09-17T05:19:51.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f74333a8640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f742c06d360 con 0x7f742c1576d0 2024-09-17T05:19:51.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7430943640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c0c8230 0x7f742c0a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:51.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7430943640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c0c8230 0x7f742c0a9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51552/0 (socket says 172.21.15.150:51552) 2024-09-17T05:19:51.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7430943640 1 -- 172.21.15.150:0/37631021 learned_addr learned my addr 172.21.15.150:0/37631021 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:19:51.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7431144640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f742c0c5390 0x7f742c159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:51.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7431945640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f742c1576d0 0x7f742c0c16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:51.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7430943640 1 -- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f742c1576d0 msgr2=0x7f742c0c16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:51.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7430943640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f742c1576d0 0x7f742c0c16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7430943640 1 -- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f742c0c5390 msgr2=0x7f742c159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:51.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7430943640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f742c0c5390 0x7f742c159f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7430943640 1 -- 172.21.15.150:0/37631021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742c1f1d30 con 0x7f742c0c8230 2024-09-17T05:19:51.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7430943640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c0c8230 0x7f742c0a9b70 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f741c00d750 tx=0x7f741c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:51.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7421ffb640 1 -- 172.21.15.150:0/37631021 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c017070 con 0x7f742c0c8230 2024-09-17T05:19:51.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7421ffb640 1 -- 172.21.15.150:0/37631021 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f741c009d20 con 0x7f742c0c8230 2024-09-17T05:19:51.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f74333a8640 1 -- 172.21.15.150:0/37631021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742c0c0d80 con 0x7f742c0c8230 2024-09-17T05:19:51.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f7421ffb640 1 -- 172.21.15.150:0/37631021 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c005280 con 0x7f742c0c8230 2024-09-17T05:19:51.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.729+0000 7f74333a8640 1 -- 172.21.15.150:0/37631021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f742c158ef0 con 0x7f742c0c8230 2024-09-17T05:19:51.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.733+0000 7f74333a8640 1 -- 172.21.15.150:0/37631021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f742c157190 con 0x7f742c0c8230 2024-09-17T05:19:51.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.733+0000 7f7421ffb640 1 -- 172.21.15.150:0/37631021 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f741c005420 con 0x7f742c0c8230 2024-09-17T05:19:51.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.733+0000 7f7421ffb640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7410073200 0x7f7410075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:51.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.733+0000 7f7421ffb640 1 -- 172.21.15.150:0/37631021 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7f741c0953f0 con 0x7f742c0c8230 2024-09-17T05:19:51.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.733+0000 7f7431144640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7410073200 0x7f7410075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:51.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.733+0000 7f7431144640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7410073200 0x7f7410075710 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f74280af000 tx=0x7f742801b040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:51.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.737+0000 7f7421ffb640 1 -- 172.21.15.150:0/37631021 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f741c00de60 con 0x7f742c0c8230 2024-09-17T05:19:51.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.941+0000 7f74333a8640 1 -- 172.21.15.150:0/37631021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f742c069360 con 0x7f742c0c8230 2024-09-17T05:19:51.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.941+0000 7f7421ffb640 1 -- 172.21.15.150:0/37631021 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) ==== 95+0+5715 (secure 0 0 0) 0x7f741c05f3b0 con 0x7f742c0c8230 2024-09-17T05:19:51.939 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:19:51.939 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":205,"btime":"2024-09-17T05:19:51:316316+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203}],"filesystems":[{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:51.316311+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:replay","state_seq":1,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:19:51.940 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 205 2024-09-17T05:19:51.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.945+0000 7f74333a8640 1 -- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7410073200 msgr2=0x7f7410075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:51.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.945+0000 7f74333a8640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7410073200 0x7f7410075710 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f74280af000 tx=0x7f742801b040 comp rx=0 tx=0).stop 2024-09-17T05:19:51.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.945+0000 7f74333a8640 1 -- 172.21.15.150:0/37631021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c0c8230 msgr2=0x7f742c0a9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:51.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.945+0000 7f74333a8640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c0c8230 0x7f742c0a9b70 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f741c00d750 tx=0x7f741c00dc10 comp rx=0 tx=0).stop 2024-09-17T05:19:51.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.945+0000 7f74333a8640 1 -- 172.21.15.150:0/37631021 shutdown_connections 2024-09-17T05:19:51.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.945+0000 7f74333a8640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7410073200 0x7f7410075710 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.945+0000 7f74333a8640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f742c1576d0 0x7f742c0c16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.945+0000 7f74333a8640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f742c0c8230 0x7f742c0a9b70 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.945+0000 7f74333a8640 1 --2- 172.21.15.150:0/37631021 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f742c0c5390 0x7f742c159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:51.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.945+0000 7f74333a8640 1 -- 172.21.15.150:0/37631021 >> 172.21.15.150:0/37631021 conn(0x7f742c06e360 msgr2=0x7f742c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:51.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.945+0000 7f74333a8640 1 -- 172.21.15.150:0/37631021 shutdown_connections 2024-09-17T05:19:51.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:51.945+0000 7f74333a8640 1 -- 172.21.15.150:0/37631021 wait complete. 2024-09-17T05:19:51.960 INFO:tasks.mds_thrash.fs.[a]:mds map: { "btime": "2024-09-17T05:19:51:316316+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 205, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 205, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8199": { "addr": "172.21.15.150:6835/1799698362", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 1799698362, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 1799698362, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 8199, "incarnation": 205, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:19:51.316311+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8199 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 202, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8031": { "addr": "172.21.15.150:6833/2857695220", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6832", "nonce": 2857695220, "type": "v2" }, { "addr": "172.21.15.150:6833", "nonce": 2857695220, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 8031, "incarnation": 199, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:18:34.641250+0000", "qdb_cluster": [ 8031 ], "qdb_leader": 8031, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8031 } } } ], "standbys": [ { "addr": "172.21.15.176:6837/2843535555", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 2843535555, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 2843535555, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 203, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6749, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:19:53.960 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:19:54.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 -- 172.21.15.150:0/1165409847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c0a7330 msgr2=0x7ff71c15ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:54.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 --2- 172.21.15.150:0/1165409847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c0a7330 0x7ff71c15ab30 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7ff70c009900 tx=0x7ff70c01c7c0 comp rx=0 tx=0).stop 2024-09-17T05:19:54.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 -- 172.21.15.150:0/1165409847 shutdown_connections 2024-09-17T05:19:54.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 --2- 172.21.15.150:0/1165409847 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c0c1330 0x7ff71c156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:54.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 --2- 172.21.15.150:0/1165409847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c0a7330 0x7ff71c15ab30 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:54.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 -- 172.21.15.150:0/1165409847 >> 172.21.15.150:0/1165409847 conn(0x7ff71c0c4a00 msgr2=0x7ff71c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:54.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 -- 172.21.15.150:0/1165409847 shutdown_connections 2024-09-17T05:19:54.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 -- 172.21.15.150:0/1165409847 wait complete. 2024-09-17T05:19:54.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 Processor -- start 2024-09-17T05:19:54.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 -- start start 2024-09-17T05:19:54.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff71c0c1330 0x7ff71c155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:54.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c0c1740 0x7ff71c0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:54.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c0706a0 0x7ff71c1efe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:54.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c0735d0 con 0x7ff71c0706a0 2024-09-17T05:19:54.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c0c72f0 con 0x7ff71c0c1330 2024-09-17T05:19:54.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff722a2a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff71c163af0 con 0x7ff71c0c1740 2024-09-17T05:19:54.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff720fc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c0706a0 0x7ff71c1efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:54.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff720fc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c0706a0 0x7ff71c1efe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51564/0 (socket says 172.21.15.150:51564) 2024-09-17T05:19:54.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.129+0000 7ff720fc7640 1 -- 172.21.15.150:0/854843340 learned_addr learned my addr 172.21.15.150:0/854843340 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:19:54.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff71bfff640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff71c0c1330 0x7ff71c155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:54.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff720fc7640 1 -- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c0c1740 msgr2=0x7ff71c0ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:54.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff720fc7640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c0c1740 0x7ff71c0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:54.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff720fc7640 1 -- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff71c0c1330 msgr2=0x7ff71c155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:54.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff720fc7640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff71c0c1330 0x7ff71c155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:54.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff720fc7640 1 -- 172.21.15.150:0/854843340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71c156ac0 con 0x7ff71c0706a0 2024-09-17T05:19:54.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff720fc7640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c0706a0 0x7ff71c1efe30 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7ff71400b9e0 tx=0x7ff71400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:54.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff718ff9640 1 -- 172.21.15.150:0/854843340 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff714003b80 con 0x7ff71c0706a0 2024-09-17T05:19:54.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff718ff9640 1 -- 172.21.15.150:0/854843340 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff714003e10 con 0x7ff71c0706a0 2024-09-17T05:19:54.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff722a2a640 1 -- 172.21.15.150:0/854843340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff71c1f0d30 con 0x7ff71c0706a0 2024-09-17T05:19:54.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff718ff9640 1 -- 172.21.15.150:0/854843340 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71400cb80 con 0x7ff71c0706a0 2024-09-17T05:19:54.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff722a2a640 1 -- 172.21.15.150:0/854843340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff71c14da60 con 0x7ff71c0706a0 2024-09-17T05:19:54.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff722a2a640 1 -- 172.21.15.150:0/854843340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff71c15ab30 con 0x7ff71c0706a0 2024-09-17T05:19:54.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff718ff9640 1 -- 172.21.15.150:0/854843340 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff714006d10 con 0x7ff71c0706a0 2024-09-17T05:19:54.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff718ff9640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff708073200 0x7ff708075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:54.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff718ff9640 1 -- 172.21.15.150:0/854843340 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7ff714094f90 con 0x7ff71c0706a0 2024-09-17T05:19:54.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.133+0000 7ff71bfff640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff708073200 0x7ff708075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:54.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.137+0000 7ff71bfff640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff708073200 0x7ff708075710 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff70c0ae000 tx=0x7ff70c01b000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:54.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.137+0000 7ff718ff9640 1 -- 172.21.15.150:0/854843340 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff71c156ac0 con 0x7ff71c0706a0 2024-09-17T05:19:54.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.329+0000 7ff722a2a640 1 -- 172.21.15.150:0/854843340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff71c0addf0 con 0x7ff71c0706a0 2024-09-17T05:19:54.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.329+0000 7ff718ff9640 1 -- 172.21.15.150:0/854843340 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) ==== 95+0+5716 (secure 0 0 0) 0x7ff71405eec0 con 0x7ff71c0706a0 2024-09-17T05:19:54.328 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:19:54.328 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":207,"btime":"2024-09-17T05:19:53:327729+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:52.333173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:19:54.329 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 207 2024-09-17T05:19:54.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.333+0000 7ff722a2a640 1 -- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff708073200 msgr2=0x7ff708075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:54.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.333+0000 7ff722a2a640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff708073200 0x7ff708075710 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff70c0ae000 tx=0x7ff70c01b000 comp rx=0 tx=0).stop 2024-09-17T05:19:54.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.333+0000 7ff722a2a640 1 -- 172.21.15.150:0/854843340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c0706a0 msgr2=0x7ff71c1efe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:54.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.333+0000 7ff722a2a640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c0706a0 0x7ff71c1efe30 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7ff71400b9e0 tx=0x7ff71400bea0 comp rx=0 tx=0).stop 2024-09-17T05:19:54.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.333+0000 7ff722a2a640 1 -- 172.21.15.150:0/854843340 shutdown_connections 2024-09-17T05:19:54.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.333+0000 7ff722a2a640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff708073200 0x7ff708075710 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:54.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.333+0000 7ff722a2a640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff71c0706a0 0x7ff71c1efe30 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:54.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.333+0000 7ff722a2a640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff71c0c1740 0x7ff71c0ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:54.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.333+0000 7ff722a2a640 1 --2- 172.21.15.150:0/854843340 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff71c0c1330 0x7ff71c155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:54.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.333+0000 7ff722a2a640 1 -- 172.21.15.150:0/854843340 >> 172.21.15.150:0/854843340 conn(0x7ff71c0c4a00 msgr2=0x7ff71c0c62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:54.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.333+0000 7ff722a2a640 1 -- 172.21.15.150:0/854843340 shutdown_connections 2024-09-17T05:19:54.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:54.333+0000 7ff722a2a640 1 -- 172.21.15.150:0/854843340 wait complete. 2024-09-17T05:19:54.351 INFO:tasks.mds_thrash.fs.[a]:no change 2024-09-17T05:19:56.352 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:19:56.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.517+0000 7f8ed9108640 1 -- 172.21.15.150:0/2445252688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed40c35b0 msgr2=0x7f8ed40c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:56.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.517+0000 7f8ed9108640 1 --2- 172.21.15.150:0/2445252688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed40c35b0 0x7f8ed40c8a80 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f8ed408d8b0 tx=0x7f8ec801c890 comp rx=0 tx=0).stop 2024-09-17T05:19:56.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.517+0000 7f8ed9108640 1 -- 172.21.15.150:0/2445252688 shutdown_connections 2024-09-17T05:19:56.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.517+0000 7f8ed9108640 1 --2- 172.21.15.150:0/2445252688 >> v2:172.21.15.176:3300/0 conn(0x7f8ed40c8fc0 0x7f8ed40c20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:56.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.517+0000 7f8ed9108640 1 --2- 172.21.15.150:0/2445252688 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed40c35b0 0x7f8ed40c8a80 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:56.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.517+0000 7f8ed9108640 1 -- 172.21.15.150:0/2445252688 >> 172.21.15.150:0/2445252688 conn(0x7f8ed4074b80 msgr2=0x7f8ed406f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:56.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.517+0000 7f8ed9108640 1 -- 172.21.15.150:0/2445252688 shutdown_connections 2024-09-17T05:19:56.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.517+0000 7f8ed9108640 1 -- 172.21.15.150:0/2445252688 wait complete. 2024-09-17T05:19:56.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed9108640 1 Processor -- start 2024-09-17T05:19:56.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed9108640 1 -- start start 2024-09-17T05:19:56.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed9108640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ed40c35b0 0x7f8ed4152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed9108640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ed40c8fc0 0x7f8ed4158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed9108640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed41568a0 0x7f8ed41f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed9108640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed40c6b70 con 0x7f8ed41568a0 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed9108640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed40c69f0 con 0x7f8ed40c35b0 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed9108640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed40b0fd0 con 0x7f8ed40c8fc0 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed41568a0 0x7f8ed41f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed41568a0 0x7f8ed41f3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51578/0 (socket says 172.21.15.150:51578) 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed359e640 1 -- 172.21.15.150:0/160687418 learned_addr learned my addr 172.21.15.150:0/160687418 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed2d9d640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ed40c35b0 0x7f8ed4152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed259c640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ed40c8fc0 0x7f8ed4158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed359e640 1 -- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ed40c8fc0 msgr2=0x7f8ed4158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed359e640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ed40c8fc0 0x7f8ed4158840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:56.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed359e640 1 -- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ed40c35b0 msgr2=0x7f8ed4152790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:56.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed359e640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ed40c35b0 0x7f8ed4152790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:56.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed359e640 1 -- 172.21.15.150:0/160687418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed4152cd0 con 0x7f8ed41568a0 2024-09-17T05:19:56.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed2d9d640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ed40c35b0 0x7f8ed4152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:19:56.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed359e640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed41568a0 0x7f8ed41f3a10 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00e9a0 tx=0x7f8ecc00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:56.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ebb7fe640 1 -- 172.21.15.150:0/160687418 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc004d30 con 0x7f8ed41568a0 2024-09-17T05:19:56.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ebb7fe640 1 -- 172.21.15.150:0/160687418 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ecc00cb20 con 0x7f8ed41568a0 2024-09-17T05:19:56.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ebb7fe640 1 -- 172.21.15.150:0/160687418 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc005020 con 0x7f8ed41568a0 2024-09-17T05:19:56.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed9108640 1 -- 172.21.15.150:0/160687418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed41468d0 con 0x7f8ed41568a0 2024-09-17T05:19:56.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.521+0000 7f8ed9108640 1 -- 172.21.15.150:0/160687418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed4153d80 con 0x7f8ed41568a0 2024-09-17T05:19:56.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.525+0000 7f8ed9108640 1 -- 172.21.15.150:0/160687418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ed4146af0 con 0x7f8ed41568a0 2024-09-17T05:19:56.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.525+0000 7f8ebb7fe640 1 -- 172.21.15.150:0/160687418 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8ecc0051c0 con 0x7f8ed41568a0 2024-09-17T05:19:56.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.525+0000 7f8ebb7fe640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8ec0073200 0x7f8ec0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:19:56.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.525+0000 7f8ebb7fe640 1 -- 172.21.15.150:0/160687418 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7f8ecc096120 con 0x7f8ed41568a0 2024-09-17T05:19:56.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.525+0000 7f8ed2d9d640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8ec0073200 0x7f8ec0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:19:56.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.525+0000 7f8ed2d9d640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8ec0073200 0x7f8ec0075710 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40098b0 tx=0x7f8ec4008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:19:56.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.529+0000 7f8ebb7fe640 1 -- 172.21.15.150:0/160687418 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8ed4152cd0 con 0x7f8ed41568a0 2024-09-17T05:19:56.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.729+0000 7f8ed9108640 1 -- 172.21.15.150:0/160687418 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ed4153b50 con 0x7f8ed41568a0 2024-09-17T05:19:56.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.729+0000 7f8ebb7fe640 1 -- 172.21.15.150:0/160687418 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) ==== 95+0+5723 (secure 0 0 0) 0x7f8ecc016030 con 0x7f8ed41568a0 2024-09-17T05:19:56.728 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:19:56.728 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":208,"btime":"2024-09-17T05:19:54:337498+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:19:56.728 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 208 2024-09-17T05:19:56.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.733+0000 7f8ed9108640 1 -- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8ec0073200 msgr2=0x7f8ec0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:56.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.733+0000 7f8ed9108640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8ec0073200 0x7f8ec0075710 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40098b0 tx=0x7f8ec4008040 comp rx=0 tx=0).stop 2024-09-17T05:19:56.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.733+0000 7f8ed9108640 1 -- 172.21.15.150:0/160687418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed41568a0 msgr2=0x7f8ed41f3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:19:56.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.733+0000 7f8ed9108640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed41568a0 0x7f8ed41f3a10 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00e9a0 tx=0x7f8ecc00ee60 comp rx=0 tx=0).stop 2024-09-17T05:19:56.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.733+0000 7f8ed9108640 1 -- 172.21.15.150:0/160687418 shutdown_connections 2024-09-17T05:19:56.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.733+0000 7f8ed9108640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8ec0073200 0x7f8ec0075710 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:56.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.733+0000 7f8ed9108640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8ed41568a0 0x7f8ed41f3a10 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:56.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.733+0000 7f8ed9108640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8ed40c8fc0 0x7f8ed4158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:56.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.733+0000 7f8ed9108640 1 --2- 172.21.15.150:0/160687418 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ed40c35b0 0x7f8ed4152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:19:56.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.733+0000 7f8ed9108640 1 -- 172.21.15.150:0/160687418 >> 172.21.15.150:0/160687418 conn(0x7f8ed4074b80 msgr2=0x7f8ed40c80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:19:56.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.733+0000 7f8ed9108640 1 -- 172.21.15.150:0/160687418 shutdown_connections 2024-09-17T05:19:56.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:19:56.733+0000 7f8ed9108640 1 -- 172.21.15.150:0/160687418 wait complete. 2024-09-17T05:19:56.749 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=7384 2024-09-17T05:19:56.749 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before reviving mds.b 2024-09-17T05:19:56.905 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-09-17T05:19:56.909+0000 7fa135639640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:20:01.750 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-09-17T05:20:01.751 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T05:20:01.751 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T05:20:01.754 INFO:tasks.ceph.mds.b:Started 2024-09-17T05:20:01.755 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:01.863 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-09-17T05:20:01.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.925+0000 7f7b1120d640 1 -- 172.21.15.150:0/3550167226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af4003a10 msgr2=0x7f7af4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:01.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.925+0000 7f7b1120d640 1 --2- 172.21.15.150:0/3550167226 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af4003a10 0x7f7af4003e80 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009b60 tx=0x7f7afc01d2b0 comp rx=0 tx=0).stop 2024-09-17T05:20:01.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.925+0000 7f7b0b59e640 1 -- 172.21.15.150:0/3550167226 reap_dead start 2024-09-17T05:20:01.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.925+0000 7f7b1120d640 1 -- 172.21.15.150:0/3550167226 shutdown_connections 2024-09-17T05:20:01.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.925+0000 7f7b1120d640 1 -- 172.21.15.150:0/3550167226 >> 172.21.15.150:0/3550167226 conn(0x7f7b0c074750 msgr2=0x7f7b0c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:01.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.925+0000 7f7b1120d640 1 -- 172.21.15.150:0/3550167226 shutdown_connections 2024-09-17T05:20:01.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.925+0000 7f7b1120d640 1 -- 172.21.15.150:0/3550167226 wait complete. 2024-09-17T05:20:01.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.925+0000 7f7b1120d640 1 Processor -- start 2024-09-17T05:20:01.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.925+0000 7f7b1120d640 1 -- start start 2024-09-17T05:20:01.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b1120d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7af4003a10 0x7f7b0c159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:01.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b1120d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b0c0cc340 0x7f7b0c15dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b1120d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b0c1f3320 0x7f7b0c1f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b1120d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c1614f0 con 0x7f7b0c0cc340 2024-09-17T05:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b1120d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c161370 con 0x7f7b0c1f3320 2024-09-17T05:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b1120d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7b0c161670 con 0x7f7af4003a10 2024-09-17T05:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b0a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b0c0cc340 0x7f7b0c15dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b0a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b0c0cc340 0x7f7b0c15dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38224/0 (socket says 172.21.15.150:38224) 2024-09-17T05:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b0a59c640 1 -- 172.21.15.150:0/1247072706 learned_addr learned my addr 172.21.15.150:0/1247072706 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:01.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b0ad9d640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7af4003a10 0x7f7b0c159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b0b59e640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b0c1f3320 0x7f7b0c1f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b0a59c640 1 -- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7af4003a10 msgr2=0x7f7b0c159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b0a59c640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7af4003a10 0x7f7b0c159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b0a59c640 1 -- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b0c1f3320 msgr2=0x7f7b0c1f3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b0a59c640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b0c1f3320 0x7f7b0c1f3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b0a59c640 1 -- 172.21.15.150:0/1247072706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0c151450 con 0x7f7b0c0cc340 2024-09-17T05:20:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b0a59c640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b0c0cc340 0x7f7b0c15dd10 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f7b000066a0 tx=0x7f7b000049e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7af37fe640 1 -- 172.21.15.150:0/1247072706 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b00015070 con 0x7f7b0c0cc340 2024-09-17T05:20:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7af37fe640 1 -- 172.21.15.150:0/1247072706 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b00010980 con 0x7f7b0c0cc340 2024-09-17T05:20:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b1120d640 1 -- 172.21.15.150:0/1247072706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b0c14ba70 con 0x7f7b0c0cc340 2024-09-17T05:20:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7af37fe640 1 -- 172.21.15.150:0/1247072706 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b00010d90 con 0x7f7b0c0cc340 2024-09-17T05:20:01.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b1120d640 1 -- 172.21.15.150:0/1247072706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b0c1f3e40 con 0x7f7b0c0cc340 2024-09-17T05:20:01.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7b1120d640 1 -- 172.21.15.150:0/1247072706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b0c1f3c70 con 0x7f7b0c0cc340 2024-09-17T05:20:01.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.929+0000 7f7af37fe640 1 -- 172.21.15.150:0/1247072706 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7b00018070 con 0x7f7b0c0cc340 2024-09-17T05:20:01.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.933+0000 7f7af37fe640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7af8073200 0x7f7af8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:01.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.933+0000 7f7af37fe640 1 -- 172.21.15.150:0/1247072706 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7f7b00095de0 con 0x7f7b0c0cc340 2024-09-17T05:20:01.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.933+0000 7f7b0ad9d640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7af8073200 0x7f7af8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:01.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.933+0000 7f7b0ad9d640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7af8073200 0x7f7af8075710 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009b60 tx=0x7f7afc01a050 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:01.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:01.937+0000 7f7af37fe640 1 -- 172.21.15.150:0/1247072706 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7b0c0b20f0 con 0x7f7b0c0cc340 2024-09-17T05:20:02.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.137+0000 7f7b1120d640 1 -- 172.21.15.150:0/1247072706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b0c157330 con 0x7f7b0c0cc340 2024-09-17T05:20:02.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.137+0000 7f7af37fe640 1 -- 172.21.15.150:0/1247072706 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) ==== 95+0+5723 (secure 0 0 0) 0x7f7b0005f7d0 con 0x7f7b0c0cc340 2024-09-17T05:20:02.135 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:02.135 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":208,"btime":"2024-09-17T05:19:54:337498+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:20:02.135 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 208 2024-09-17T05:20:02.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.141+0000 7f7b1120d640 1 -- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7af8073200 msgr2=0x7f7af8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:02.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.141+0000 7f7b1120d640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7af8073200 0x7f7af8075710 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009b60 tx=0x7f7afc01a050 comp rx=0 tx=0).stop 2024-09-17T05:20:02.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.141+0000 7f7b1120d640 1 -- 172.21.15.150:0/1247072706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b0c0cc340 msgr2=0x7f7b0c15dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:02.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.141+0000 7f7b1120d640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b0c0cc340 0x7f7b0c15dd10 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f7b000066a0 tx=0x7f7b000049e0 comp rx=0 tx=0).stop 2024-09-17T05:20:02.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.141+0000 7f7b1120d640 1 -- 172.21.15.150:0/1247072706 shutdown_connections 2024-09-17T05:20:02.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.141+0000 7f7b1120d640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7af8073200 0x7f7af8075710 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:02.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.141+0000 7f7b1120d640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7b0c1f3320 0x7f7b0c1f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:02.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.141+0000 7f7b1120d640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b0c0cc340 0x7f7b0c15dd10 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:02.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.141+0000 7f7b1120d640 1 --2- 172.21.15.150:0/1247072706 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7af4003a10 0x7f7b0c159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:02.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.141+0000 7f7b1120d640 1 -- 172.21.15.150:0/1247072706 >> 172.21.15.150:0/1247072706 conn(0x7f7b0c074750 msgr2=0x7f7b0c150920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:02.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.141+0000 7f7b1120d640 1 -- 172.21.15.150:0/1247072706 shutdown_connections 2024-09-17T05:20:02.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:02.141+0000 7f7b1120d640 1 -- 172.21.15.150:0/1247072706 wait complete. 2024-09-17T05:20:02.156 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-09-17T05:20:04.157 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:04.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.329+0000 7fe009aba640 1 -- 172.21.15.150:0/4076874053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe004070e50 msgr2=0x7fe004071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:04.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.329+0000 7fe009aba640 1 --2- 172.21.15.150:0/4076874053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe004070e50 0x7fe004071260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009900 tx=0x7fdff801c850 comp rx=0 tx=0).stop 2024-09-17T05:20:04.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.329+0000 7fe009aba640 1 -- 172.21.15.150:0/4076874053 shutdown_connections 2024-09-17T05:20:04.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.329+0000 7fe009aba640 1 --2- 172.21.15.150:0/4076874053 >> v2:172.21.15.176:3300/0 conn(0x7fe0040717a0 0x7fe004074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:04.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.329+0000 7fe009aba640 1 --2- 172.21.15.150:0/4076874053 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe004070e50 0x7fe004071260 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:04.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.329+0000 7fe009aba640 1 -- 172.21.15.150:0/4076874053 >> 172.21.15.150:0/4076874053 conn(0x7fe0040c4a00 msgr2=0x7fe0040c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:04.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.329+0000 7fe009aba640 1 -- 172.21.15.150:0/4076874053 shutdown_connections 2024-09-17T05:20:04.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.329+0000 7fe009aba640 1 -- 172.21.15.150:0/4076874053 wait complete. 2024-09-17T05:20:04.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.329+0000 7fe009aba640 1 Processor -- start 2024-09-17T05:20:04.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.329+0000 7fe009aba640 1 -- start start 2024-09-17T05:20:04.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe009aba640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe004070e50 0x7fe00414cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:04.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe009aba640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0040717a0 0x7fe0040a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:04.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe009aba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe004073440 0x7fe004144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe009aba640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe0040ad210 con 0x7fe004073440 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe009aba640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe0040ad090 con 0x7fe0040717a0 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe009aba640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe0040ad390 con 0x7fe004070e50 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe003fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe004073440 0x7fe004144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe003fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe004073440 0x7fe004144620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38234/0 (socket says 172.21.15.150:38234) 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe003fff640 1 -- 172.21.15.150:0/1786715065 learned_addr learned my addr 172.21.15.150:0/1786715065 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe002ffd640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0040717a0 0x7fe0040a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe0037fe640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe004070e50 0x7fe00414cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe003fff640 1 -- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe004070e50 msgr2=0x7fe00414cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe003fff640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe004070e50 0x7fe00414cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe003fff640 1 -- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0040717a0 msgr2=0x7fe0040a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe003fff640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0040717a0 0x7fe0040a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:04.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe003fff640 1 -- 172.21.15.150:0/1786715065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe004147b40 con 0x7fe004073440 2024-09-17T05:20:04.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe003fff640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe004073440 0x7fe004144620 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fdffc00ea80 tx=0x7fdffc00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:04.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fdfebfff640 1 -- 172.21.15.150:0/1786715065 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdffc00ce40 con 0x7fe004073440 2024-09-17T05:20:04.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fdfebfff640 1 -- 172.21.15.150:0/1786715065 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdffc004b30 con 0x7fe004073440 2024-09-17T05:20:04.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe009aba640 1 -- 172.21.15.150:0/1786715065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe004147860 con 0x7fe004073440 2024-09-17T05:20:04.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fdfebfff640 1 -- 172.21.15.150:0/1786715065 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdffc005020 con 0x7fe004073440 2024-09-17T05:20:04.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe009aba640 1 -- 172.21.15.150:0/1786715065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0040a94a0 con 0x7fe004073440 2024-09-17T05:20:04.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fe009aba640 1 -- 172.21.15.150:0/1786715065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0040ad840 con 0x7fe004073440 2024-09-17T05:20:04.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.333+0000 7fdfebfff640 1 -- 172.21.15.150:0/1786715065 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fdffc0051c0 con 0x7fe004073440 2024-09-17T05:20:04.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.337+0000 7fdfebfff640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdff0073200 0x7fdff0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:04.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.337+0000 7fdfebfff640 1 -- 172.21.15.150:0/1786715065 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7fdffc0964c0 con 0x7fe004073440 2024-09-17T05:20:04.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.337+0000 7fe0037fe640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdff0073200 0x7fdff0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:04.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.337+0000 7fe0037fe640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdff0073200 0x7fdff0075710 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fe0041513e0 tx=0x7fdff4009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:04.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.341+0000 7fdfebfff640 1 -- 172.21.15.150:0/1786715065 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdffc060390 con 0x7fe004073440 2024-09-17T05:20:04.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.549+0000 7fe009aba640 1 -- 172.21.15.150:0/1786715065 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0040c2380 con 0x7fe004073440 2024-09-17T05:20:04.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.549+0000 7fdfebfff640 1 -- 172.21.15.150:0/1786715065 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+6552 (secure 0 0 0) 0x7fdffc05fc90 con 0x7fe004073440 2024-09-17T05:20:04.547 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:04.547 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":209,"btime":"2024-09-17T05:20:02:371612+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":7567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":209}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:20:04.547 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 209 2024-09-17T05:20:04.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.553+0000 7fe009aba640 1 -- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdff0073200 msgr2=0x7fdff0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:04.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.553+0000 7fe009aba640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdff0073200 0x7fdff0075710 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fe0041513e0 tx=0x7fdff4009040 comp rx=0 tx=0).stop 2024-09-17T05:20:04.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.553+0000 7fe009aba640 1 -- 172.21.15.150:0/1786715065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe004073440 msgr2=0x7fe004144620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:04.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.553+0000 7fe009aba640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe004073440 0x7fe004144620 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fdffc00ea80 tx=0x7fdffc00ef40 comp rx=0 tx=0).stop 2024-09-17T05:20:04.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.553+0000 7fe009aba640 1 -- 172.21.15.150:0/1786715065 shutdown_connections 2024-09-17T05:20:04.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.553+0000 7fe009aba640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdff0073200 0x7fdff0075710 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:04.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.553+0000 7fe009aba640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe0040717a0 0x7fe0040a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:04.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.553+0000 7fe009aba640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe004073440 0x7fe004144620 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:04.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.553+0000 7fe009aba640 1 --2- 172.21.15.150:0/1786715065 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe004070e50 0x7fe00414cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:04.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.553+0000 7fe009aba640 1 -- 172.21.15.150:0/1786715065 >> 172.21.15.150:0/1786715065 conn(0x7fe0040c4a00 msgr2=0x7fe0040c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:04.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.553+0000 7fe009aba640 1 -- 172.21.15.150:0/1786715065 shutdown_connections 2024-09-17T05:20:04.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:04.553+0000 7fe009aba640 1 -- 172.21.15.150:0/1786715065 wait complete. 2024-09-17T05:20:04.569 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-09-17T05:20:04.570 INFO:tasks.mds_thrash.fs.[a]:waiting for 112 secs before thrashing 2024-09-17T05:20:07.363 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:20:07.366 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:20:16.271 INFO:tasks.workunit.client.1.smithi176.stdout:Syncing()...1 sec 2024-09-17T05:20:16.272 INFO:tasks.workunit.client.1.smithi176.stdout:FFSB benchmark finished at: Tue Sep 17 05:20:16 2024 2024-09-17T05:20:16.272 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T05:20:16.272 INFO:tasks.workunit.client.1.smithi176.stdout:Results: 2024-09-17T05:20:16.272 INFO:tasks.workunit.client.1.smithi176.stdout:Benchmark took 322.13 sec 2024-09-17T05:20:16.272 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T05:20:16.272 INFO:tasks.workunit.client.1.smithi176.stdout:Total Results 2024-09-17T05:20:16.272 INFO:tasks.workunit.client.1.smithi176.stdout:=============== 2024-09-17T05:20:16.272 INFO:tasks.workunit.client.1.smithi176.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-09-17T05:20:16.272 INFO:tasks.workunit.client.1.smithi176.stdout: ======= ============ ========= ======= =========== ========== 2024-09-17T05:20:16.273 INFO:tasks.workunit.client.1.smithi176.stdout: write : 1084160 3365.58 100.000% 100.000% 13.1MB/sec 2024-09-17T05:20:16.273 INFO:tasks.workunit.client.1.smithi176.stdout:- 2024-09-17T05:20:16.273 INFO:tasks.workunit.client.1.smithi176.stdout:3365.58 Transactions per Second 2024-09-17T05:20:16.273 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T05:20:16.273 INFO:tasks.workunit.client.1.smithi176.stdout:Throughput Results 2024-09-17T05:20:16.273 INFO:tasks.workunit.client.1.smithi176.stdout:=================== 2024-09-17T05:20:16.273 INFO:tasks.workunit.client.1.smithi176.stdout:Write Throughput: 13.1MB/sec 2024-09-17T05:20:16.277 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T05:20:16.277 INFO:tasks.workunit.client.1.smithi176.stdout:System Call Latency statistics in millisecs 2024-09-17T05:20:16.277 INFO:tasks.workunit.client.1.smithi176.stdout:===== 2024-09-17T05:20:16.277 INFO:tasks.workunit.client.1.smithi176.stdout: Min Avg Max Total Calls 2024-09-17T05:20:16.277 INFO:tasks.workunit.client.1.smithi176.stdout: ======== ======== ======== ============ 2024-09-17T05:20:16.277 INFO:tasks.workunit.client.1.smithi176.stdout:[ open] 0.545000 1112.837972 17386.923828 847 2024-09-17T05:20:16.277 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-17T05:20:16.277 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-17T05:20:16.277 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-17T05:20:16.278 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[3] 500.000000 - 1000.000000 : 1 2024-09-17T05:20:16.278 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[4] 200.000000 - 500.000000 : 2 2024-09-17T05:20:16.278 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[5] 100.000000 - 200.000000 : 11 2024-09-17T05:20:16.278 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[6] 50.000000 - 100.000000 : 14 2024-09-17T05:20:16.278 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[7] 20.000000 - 50.000000 : 215 2024-09-17T05:20:16.278 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[8] 10.000000 - 20.000000 : 315 2024-09-17T05:20:16.278 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[9] 5.000000 - 10.000000 : 144 2024-09-17T05:20:16.278 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[10] 2.000000 - 5.000000 : 48 2024-09-17T05:20:16.278 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[11] 1.000000 - 2.000000 : 19 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[12] 0.500000 - 1.000000 : 7 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout:[ write] 0.088000 8.554816 93149.023438 1084160 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[0] 5000.000000 - 10000.000000 : 5 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[1] 2000.000000 - 5000.000000 : 60 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[2] 1000.000000 - 2000.000000 : 26 2024-09-17T05:20:16.279 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[3] 500.000000 - 1000.000000 : 24 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[4] 200.000000 - 500.000000 : 42 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[5] 100.000000 - 200.000000 : 101 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[6] 50.000000 - 100.000000 : 3532 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[7] 20.000000 - 50.000000 : 45518 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[8] 10.000000 - 20.000000 : 106648 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[9] 5.000000 - 10.000000 : 196738 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[10] 2.000000 - 5.000000 : 254183 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[11] 1.000000 - 2.000000 : 111917 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[12] 0.500000 - 1.000000 : 50998 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[13] 0.200000 - 0.500000 : 155447 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[14] 0.100000 - 0.200000 : 157409 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[15] 0.050000 - 0.100000 : 1455 2024-09-17T05:20:16.280 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout:[ lseek] 0.000000 0.001023 7.455000 1084160 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-09-17T05:20:16.281 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-09-17T05:20:16.282 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[9] 5.000000 - 10.000000 : 1 2024-09-17T05:20:16.282 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[10] 2.000000 - 5.000000 : 4 2024-09-17T05:20:16.282 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[11] 1.000000 - 2.000000 : 1 2024-09-17T05:20:16.282 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[12] 0.500000 - 1.000000 : 7 2024-09-17T05:20:16.282 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[13] 0.200000 - 0.500000 : 27 2024-09-17T05:20:16.282 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[14] 0.100000 - 0.200000 : 30 2024-09-17T05:20:16.282 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[15] 0.050000 - 0.100000 : 35 2024-09-17T05:20:16.282 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[16] 0.020000 - 0.050000 : 236 2024-09-17T05:20:16.282 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[17] 0.010000 - 0.020000 : 602 2024-09-17T05:20:16.282 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[18] 0.000000 - 0.010000 : 1083217 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout:[ close] 0.020000 2.625140 62.883999 847 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[6] 50.000000 - 100.000000 : 2 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[7] 20.000000 - 50.000000 : 25 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[8] 10.000000 - 20.000000 : 31 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[9] 5.000000 - 10.000000 : 44 2024-09-17T05:20:16.283 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[10] 2.000000 - 5.000000 : 110 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[11] 1.000000 - 2.000000 : 123 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[12] 0.500000 - 1.000000 : 48 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[13] 0.200000 - 0.500000 : 45 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[14] 0.100000 - 0.200000 : 83 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[15] 0.050000 - 0.100000 : 69 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[16] 0.020000 - 0.050000 : 267 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout: 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout:0.3% User Time 2024-09-17T05:20:16.284 INFO:tasks.workunit.client.1.smithi176.stdout:33.8% System Time 2024-09-17T05:20:16.285 INFO:tasks.workunit.client.1.smithi176.stdout:34.0% CPU Utilization 2024-09-17T05:20:16.285 INFO:tasks.workunit.client.1.smithi176.stderr:+ cd .. 2024-09-17T05:20:16.290 INFO:tasks.workunit.client.1.smithi176.stderr:+ rm -r tmp ffsb 2024-09-17T05:20:17.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:17.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.753+0000 7f11d590a640 1 -- 172.21.15.150:0/895384408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d00a7f10 msgr2=0x7f11d00a8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:17.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.753+0000 7f11d590a640 1 --2- 172.21.15.150:0/895384408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d00a7f10 0x7f11d00a8320 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f11d00c9e50 tx=0x7f11c801ccf0 comp rx=0 tx=0).stop 2024-09-17T05:20:17.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.753+0000 7f11d590a640 1 -- 172.21.15.150:0/895384408 shutdown_connections 2024-09-17T05:20:17.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.753+0000 7f11d590a640 1 --2- 172.21.15.150:0/895384408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d00a7f10 0x7f11d00a8320 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:17.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.753+0000 7f11d590a640 1 --2- 172.21.15.150:0/895384408 >> v2:172.21.15.176:3300/0 conn(0x7f11d00c5760 0x7f11d00c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:17.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.753+0000 7f11d590a640 1 --2- 172.21.15.150:0/895384408 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11d01535b0 0x7f11d0157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:17.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.753+0000 7f11d590a640 1 -- 172.21.15.150:0/895384408 >> 172.21.15.150:0/895384408 conn(0x7f11d00aaa80 msgr2=0x7f11d00aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:17.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.753+0000 7f11d590a640 1 -- 172.21.15.150:0/895384408 shutdown_connections 2024-09-17T05:20:17.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.753+0000 7f11d590a640 1 -- 172.21.15.150:0/895384408 wait complete. 2024-09-17T05:20:17.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11d590a640 1 Processor -- start 2024-09-17T05:20:17.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11d590a640 1 -- start start 2024-09-17T05:20:17.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11d590a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11d00a7f10 0x7f11d00c06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:17.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11d590a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d00c5760 0x7f11d00adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:17.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11d590a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11d01535b0 0x7f11d00bf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:17.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11d590a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f11d00718d0 con 0x7f11d00c5760 2024-09-17T05:20:17.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11d590a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f11d0157a60 con 0x7f11d00a7f10 2024-09-17T05:20:17.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11d590a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f11d00c5370 con 0x7f11d01535b0 2024-09-17T05:20:17.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11ce7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d00c5760 0x7f11d00adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:17.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11ce7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d00c5760 0x7f11d00adea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55392/0 (socket says 172.21.15.150:55392) 2024-09-17T05:20:17.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11ce7fc640 1 -- 172.21.15.150:0/2739087844 learned_addr learned my addr 172.21.15.150:0/2739087844 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:17.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11ceffd640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11d00a7f10 0x7f11d00c06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:17.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11cf7fe640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11d01535b0 0x7f11d00bf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:17.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11ce7fc640 1 -- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11d01535b0 msgr2=0x7f11d00bf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:17.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11ce7fc640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11d01535b0 0x7f11d00bf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:17.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11ce7fc640 1 -- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11d00a7f10 msgr2=0x7f11d00c06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:17.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11ce7fc640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11d00a7f10 0x7f11d00c06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:17.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.757+0000 7f11ce7fc640 1 -- 172.21.15.150:0/2739087844 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d0155bb0 con 0x7f11d00c5760 2024-09-17T05:20:17.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.761+0000 7f11ce7fc640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d00c5760 0x7f11d00adea0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f11c000d660 tx=0x7f11c000db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:17.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.761+0000 7f11b7fff640 1 -- 172.21.15.150:0/2739087844 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0017070 con 0x7f11d00c5760 2024-09-17T05:20:17.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.761+0000 7f11b7fff640 1 -- 172.21.15.150:0/2739087844 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11c0009b10 con 0x7f11d00c5760 2024-09-17T05:20:17.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.761+0000 7f11b7fff640 1 -- 172.21.15.150:0/2739087844 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0010460 con 0x7f11d00c5760 2024-09-17T05:20:17.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.761+0000 7f11d590a640 1 -- 172.21.15.150:0/2739087844 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d01ee440 con 0x7f11d00c5760 2024-09-17T05:20:17.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.761+0000 7f11d590a640 1 -- 172.21.15.150:0/2739087844 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d01ee720 con 0x7f11d00c5760 2024-09-17T05:20:17.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.761+0000 7f11d590a640 1 -- 172.21.15.150:0/2739087844 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d0153a10 con 0x7f11d00c5760 2024-09-17T05:20:17.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.761+0000 7f11b7fff640 1 -- 172.21.15.150:0/2739087844 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f11c001a070 con 0x7f11d00c5760 2024-09-17T05:20:17.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.761+0000 7f11b7fff640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f11bc073200 0x7f11bc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:17.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.761+0000 7f11b7fff640 1 -- 172.21.15.150:0/2739087844 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7f11c00953f0 con 0x7f11d00c5760 2024-09-17T05:20:17.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.765+0000 7f11ceffd640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f11bc073200 0x7f11bc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:17.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.765+0000 7f11ceffd640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f11bc073200 0x7f11bc075710 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f11c40058d0 tx=0x7f11c4005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:17.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.769+0000 7f11b7fff640 1 -- 172.21.15.150:0/2739087844 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f11c005f3b0 con 0x7f11d00c5760 2024-09-17T05:20:17.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.977+0000 7f11d590a640 1 -- 172.21.15.150:0/2739087844 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11d00c3070 con 0x7f11d00c5760 2024-09-17T05:20:17.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.977+0000 7f11b7fff640 1 -- 172.21.15.150:0/2739087844 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+6552 (secure 0 0 0) 0x7f11c005ecb0 con 0x7f11d00c5760 2024-09-17T05:20:17.974 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:17.974 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":209,"btime":"2024-09-17T05:20:02:371612+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":7567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":209}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:20:17.975 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 209 2024-09-17T05:20:17.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.981+0000 7f11d590a640 1 -- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f11bc073200 msgr2=0x7f11bc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:17.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.981+0000 7f11d590a640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f11bc073200 0x7f11bc075710 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f11c40058d0 tx=0x7f11c4005860 comp rx=0 tx=0).stop 2024-09-17T05:20:17.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.981+0000 7f11d590a640 1 -- 172.21.15.150:0/2739087844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d00c5760 msgr2=0x7f11d00adea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:17.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.981+0000 7f11d590a640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d00c5760 0x7f11d00adea0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f11c000d660 tx=0x7f11c000db20 comp rx=0 tx=0).stop 2024-09-17T05:20:17.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.981+0000 7f11d590a640 1 -- 172.21.15.150:0/2739087844 shutdown_connections 2024-09-17T05:20:17.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.981+0000 7f11d590a640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f11bc073200 0x7f11bc075710 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:17.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.981+0000 7f11d590a640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f11d01535b0 0x7f11d00bf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:17.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.981+0000 7f11d590a640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11d00c5760 0x7f11d00adea0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:17.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.981+0000 7f11d590a640 1 --2- 172.21.15.150:0/2739087844 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11d00a7f10 0x7f11d00c06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:17.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.981+0000 7f11d590a640 1 -- 172.21.15.150:0/2739087844 >> 172.21.15.150:0/2739087844 conn(0x7f11d00aaa80 msgr2=0x7f11d0151420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:17.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.981+0000 7f11d590a640 1 -- 172.21.15.150:0/2739087844 shutdown_connections 2024-09-17T05:20:17.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:17.981+0000 7f11d590a640 1 -- 172.21.15.150:0/2739087844 wait complete. 2024-09-17T05:20:17.998 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-09-17T05:20:18.009 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T05:20:18.009 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:20:18.025 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T05:20:18.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:18.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.241+0000 7fd43616d640 1 -- 172.21.15.150:0/1577189398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd410004d50 msgr2=0x7fd410005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:18.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.241+0000 7fd43616d640 1 --2- 172.21.15.150:0/1577189398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd410004d50 0x7fd410005160 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fd42800c260 tx=0x7fd42801cf60 comp rx=0 tx=0).stop 2024-09-17T05:20:18.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.241+0000 7fd42ffff640 1 -- 172.21.15.150:0/1577189398 reap_dead start 2024-09-17T05:20:18.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.241+0000 7fd43616d640 1 -- 172.21.15.150:0/1577189398 shutdown_connections 2024-09-17T05:20:18.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.241+0000 7fd43616d640 1 -- 172.21.15.150:0/1577189398 >> 172.21.15.150:0/1577189398 conn(0x7fd4300c4a00 msgr2=0x7fd4300c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:18.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.241+0000 7fd43616d640 1 -- 172.21.15.150:0/1577189398 shutdown_connections 2024-09-17T05:20:18.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.241+0000 7fd43616d640 1 -- 172.21.15.150:0/1577189398 wait complete. 2024-09-17T05:20:18.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.241+0000 7fd43616d640 1 Processor -- start 2024-09-17T05:20:18.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.241+0000 7fd43616d640 1 -- start start 2024-09-17T05:20:18.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd43616d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd410004d50 0x7fd430158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:18.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd43616d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd430147a90 0x7fd4300ac100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:18.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd43616d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4300ac640 0x7fd4301e1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:18.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd43616d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd4300add70 con 0x7fd430147a90 2024-09-17T05:20:18.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd43616d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd4300c3000 con 0x7fd410004d50 2024-09-17T05:20:18.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd43616d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd4300705d0 con 0x7fd4300ac640 2024-09-17T05:20:18.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd42effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd430147a90 0x7fd4300ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd42effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd430147a90 0x7fd4300ac100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55406/0 (socket says 172.21.15.150:55406) 2024-09-17T05:20:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd42effd640 1 -- 172.21.15.150:0/3184235440 learned_addr learned my addr 172.21.15.150:0/3184235440 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd42f7fe640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd410004d50 0x7fd430158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd42ffff640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4300ac640 0x7fd4301e1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd42effd640 1 -- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4300ac640 msgr2=0x7fd4301e1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd42effd640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4300ac640 0x7fd4301e1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd42effd640 1 -- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd410004d50 msgr2=0x7fd430158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd42effd640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd410004d50 0x7fd430158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd42effd640 1 -- 172.21.15.150:0/3184235440 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd430148590 con 0x7fd430147a90 2024-09-17T05:20:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd42f7fe640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd410004d50 0x7fd430158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:20:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd42effd640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd430147a90 0x7fd4300ac100 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fd420011830 tx=0x7fd420011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd427fff640 1 -- 172.21.15.150:0/3184235440 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd420004460 con 0x7fd430147a90 2024-09-17T05:20:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd427fff640 1 -- 172.21.15.150:0/3184235440 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd420016070 con 0x7fd430147a90 2024-09-17T05:20:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd43616d640 1 -- 172.21.15.150:0/3184235440 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4301ee120 con 0x7fd430147a90 2024-09-17T05:20:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd427fff640 1 -- 172.21.15.150:0/3184235440 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42000fb80 con 0x7fd430147a90 2024-09-17T05:20:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.245+0000 7fd43616d640 1 -- 172.21.15.150:0/3184235440 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4300ab520 con 0x7fd430147a90 2024-09-17T05:20:18.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.249+0000 7fd43616d640 1 -- 172.21.15.150:0/3184235440 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4300a75f0 con 0x7fd430147a90 2024-09-17T05:20:18.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.249+0000 7fd427fff640 1 -- 172.21.15.150:0/3184235440 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd420018030 con 0x7fd430147a90 2024-09-17T05:20:18.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.249+0000 7fd427fff640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd414073200 0x7fd414075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:18.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.249+0000 7fd427fff640 1 -- 172.21.15.150:0/3184235440 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7fd420095c00 con 0x7fd430147a90 2024-09-17T05:20:18.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.249+0000 7fd42f7fe640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd414073200 0x7fd414075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:18.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.249+0000 7fd42f7fe640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd414073200 0x7fd414075710 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd4300c2500 tx=0x7fd418009500 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:18.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.253+0000 7fd427fff640 1 -- 172.21.15.150:0/3184235440 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd42009c080 con 0x7fd430147a90 2024-09-17T05:20:18.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.465+0000 7fd43616d640 1 -- 172.21.15.150:0/3184235440 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4301ee620 con 0x7fd430147a90 2024-09-17T05:20:18.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.465+0000 7fd427fff640 1 -- 172.21.15.150:0/3184235440 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+6552 (secure 0 0 0) 0x7fd4301ee620 con 0x7fd430147a90 2024-09-17T05:20:18.462 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:18.462 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":209,"btime":"2024-09-17T05:20:02:371612+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":7567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":209}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:20:18.462 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 209 2024-09-17T05:20:18.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.469+0000 7fd43616d640 1 -- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd414073200 msgr2=0x7fd414075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:18.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.469+0000 7fd43616d640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd414073200 0x7fd414075710 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd4300c2500 tx=0x7fd418009500 comp rx=0 tx=0).stop 2024-09-17T05:20:18.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.469+0000 7fd43616d640 1 -- 172.21.15.150:0/3184235440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd430147a90 msgr2=0x7fd4300ac100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:18.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.469+0000 7fd43616d640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd430147a90 0x7fd4300ac100 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fd420011830 tx=0x7fd420011cf0 comp rx=0 tx=0).stop 2024-09-17T05:20:18.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.469+0000 7fd43616d640 1 -- 172.21.15.150:0/3184235440 shutdown_connections 2024-09-17T05:20:18.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.469+0000 7fd43616d640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd414073200 0x7fd414075710 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:18.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.469+0000 7fd43616d640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd4300ac640 0x7fd4301e1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:18.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.469+0000 7fd43616d640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd430147a90 0x7fd4300ac100 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:18.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.469+0000 7fd43616d640 1 --2- 172.21.15.150:0/3184235440 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd410004d50 0x7fd430158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:18.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.469+0000 7fd43616d640 1 -- 172.21.15.150:0/3184235440 >> 172.21.15.150:0/3184235440 conn(0x7fd4300c4a00 msgr2=0x7fd43006d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:18.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.469+0000 7fd43616d640 1 -- 172.21.15.150:0/3184235440 shutdown_connections 2024-09-17T05:20:18.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:18.469+0000 7fd43616d640 1 -- 172.21.15.150:0/3184235440 wait complete. 2024-09-17T05:20:18.487 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:20:20.489 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:20.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.657+0000 7f6a6b156640 1 -- 172.21.15.150:0/3627238047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a640c41c0 msgr2=0x7f6a6406fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:20.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.657+0000 7f6a6b156640 1 --2- 172.21.15.150:0/3627238047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a640c41c0 0x7f6a6406fab0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f6a64071360 tx=0x7f6a5801c8e0 comp rx=0 tx=0).stop 2024-09-17T05:20:20.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.657+0000 7f6a6b156640 1 -- 172.21.15.150:0/3627238047 shutdown_connections 2024-09-17T05:20:20.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.657+0000 7f6a6b156640 1 --2- 172.21.15.150:0/3627238047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a640c41c0 0x7f6a6406fab0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:20.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.657+0000 7f6a6b156640 1 -- 172.21.15.150:0/3627238047 >> 172.21.15.150:0/3627238047 conn(0x7f6a640c4a00 msgr2=0x7f6a640c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:20.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.657+0000 7f6a6b156640 1 -- 172.21.15.150:0/3627238047 shutdown_connections 2024-09-17T05:20:20.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.657+0000 7f6a6b156640 1 -- 172.21.15.150:0/3627238047 wait complete. 2024-09-17T05:20:20.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a6b156640 1 Processor -- start 2024-09-17T05:20:20.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a6b156640 1 -- start start 2024-09-17T05:20:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a6b156640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a6406fff0 0x7f6a64158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a6b156640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a640c3db0 0x7f6a641f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a6b156640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6a640c41c0 0x7f6a6414dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a6b156640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6a640adb30 con 0x7f6a640c3db0 2024-09-17T05:20:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a6b156640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6a640ad9b0 con 0x7f6a6406fff0 2024-09-17T05:20:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a6b156640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6a640a75f0 con 0x7f6a640c41c0 2024-09-17T05:20:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a63fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a640c3db0 0x7f6a641f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a63fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a640c3db0 0x7f6a641f37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52570/0 (socket says 172.21.15.150:52570) 2024-09-17T05:20:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a63fff640 1 -- 172.21.15.150:0/1868009135 learned_addr learned my addr 172.21.15.150:0/1868009135 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a63fff640 1 -- 172.21.15.150:0/1868009135 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6a640c41c0 msgr2=0x7f6a6414dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a63fff640 1 --2- 172.21.15.150:0/1868009135 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6a640c41c0 0x7f6a6414dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a63fff640 1 -- 172.21.15.150:0/1868009135 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a6406fff0 msgr2=0x7f6a64158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a63fff640 1 --2- 172.21.15.150:0/1868009135 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a6406fff0 0x7f6a64158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a63fff640 1 -- 172.21.15.150:0/1868009135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6414f680 con 0x7f6a640c3db0 2024-09-17T05:20:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a63fff640 1 --2- 172.21.15.150:0/1868009135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a640c3db0 0x7f6a641f37c0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f6a5801a510 tx=0x7f6a580a56b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a617fa640 1 -- 172.21.15.150:0/1868009135 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a580b2040 con 0x7f6a640c3db0 2024-09-17T05:20:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a617fa640 1 -- 172.21.15.150:0/1868009135 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a58004430 con 0x7f6a640c3db0 2024-09-17T05:20:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a6b156640 1 -- 172.21.15.150:0/1868009135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a6414f960 con 0x7f6a640c3db0 2024-09-17T05:20:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a617fa640 1 -- 172.21.15.150:0/1868009135 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58003860 con 0x7f6a640c3db0 2024-09-17T05:20:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a6b156640 1 -- 172.21.15.150:0/1868009135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a641f4190 con 0x7f6a640c3db0 2024-09-17T05:20:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.661+0000 7f6a6b156640 1 -- 172.21.15.150:0/1868009135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a6406fab0 con 0x7f6a640c3db0 2024-09-17T05:20:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.665+0000 7f6a617fa640 1 -- 172.21.15.150:0/1868009135 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6a580bf060 con 0x7f6a640c3db0 2024-09-17T05:20:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.665+0000 7f6a617fa640 1 --2- 172.21.15.150:0/1868009135 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6a50073200 0x7f6a50075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.665+0000 7f6a617fa640 1 -- 172.21.15.150:0/1868009135 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7f6a5812edd0 con 0x7f6a640c3db0 2024-09-17T05:20:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.665+0000 7f6a68ef2640 1 --2- 172.21.15.150:0/1868009135 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6a50073200 0x7f6a50075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.665+0000 7f6a68ef2640 1 --2- 172.21.15.150:0/1868009135 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6a50073200 0x7f6a50075710 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6a640c2500 tx=0x7f6a5c009230 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:20.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.669+0000 7f6a617fa640 1 -- 172.21.15.150:0/1868009135 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6a580164d0 con 0x7f6a640c3db0 2024-09-17T05:20:20.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.877+0000 7f6a6b156640 1 -- 172.21.15.150:0/1868009135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a64073a70 con 0x7f6a640c3db0 2024-09-17T05:20:20.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.877+0000 7f6a617fa640 1 -- 172.21.15.150:0/1868009135 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+6552 (secure 0 0 0) 0x7f6a580f8d90 con 0x7f6a640c3db0 2024-09-17T05:20:20.875 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:20.875 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":209,"btime":"2024-09-17T05:20:02:371612+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":7567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":209}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:20:20.875 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 209 2024-09-17T05:20:20.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.881+0000 7f6a6b156640 1 -- 172.21.15.150:0/1868009135 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6a50073200 msgr2=0x7f6a50075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:20.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.881+0000 7f6a6b156640 1 --2- 172.21.15.150:0/1868009135 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6a50073200 0x7f6a50075710 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6a640c2500 tx=0x7f6a5c009230 comp rx=0 tx=0).stop 2024-09-17T05:20:20.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.881+0000 7f6a6b156640 1 -- 172.21.15.150:0/1868009135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a640c3db0 msgr2=0x7f6a641f37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:20.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.881+0000 7f6a6b156640 1 --2- 172.21.15.150:0/1868009135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a640c3db0 0x7f6a641f37c0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f6a5801a510 tx=0x7f6a580a56b0 comp rx=0 tx=0).stop 2024-09-17T05:20:20.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.881+0000 7f6a6b156640 1 -- 172.21.15.150:0/1868009135 shutdown_connections 2024-09-17T05:20:20.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.881+0000 7f6a6b156640 1 --2- 172.21.15.150:0/1868009135 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6a50073200 0x7f6a50075710 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:20.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.881+0000 7f6a6b156640 1 --2- 172.21.15.150:0/1868009135 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6a640c41c0 0x7f6a6414dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:20.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.881+0000 7f6a6b156640 1 --2- 172.21.15.150:0/1868009135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a640c3db0 0x7f6a641f37c0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:20.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.881+0000 7f6a6b156640 1 --2- 172.21.15.150:0/1868009135 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6a6406fff0 0x7f6a64158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:20.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.881+0000 7f6a6b156640 1 -- 172.21.15.150:0/1868009135 >> 172.21.15.150:0/1868009135 conn(0x7f6a640c4a00 msgr2=0x7f6a64073f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:20.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.881+0000 7f6a6b156640 1 -- 172.21.15.150:0/1868009135 shutdown_connections 2024-09-17T05:20:20.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:20.881+0000 7f6a6b156640 1 -- 172.21.15.150:0/1868009135 wait complete. 2024-09-17T05:20:20.900 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:20:22.901 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:23.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.069+0000 7efda8cd0640 1 -- 172.21.15.150:0/1149244332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd90004d50 msgr2=0x7efd90005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:23.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.069+0000 7efda8cd0640 1 --2- 172.21.15.150:0/1149244332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd90004d50 0x7efd90005160 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7efd9c009eb0 tx=0x7efd9c01d5f0 comp rx=0 tx=0).stop 2024-09-17T05:20:23.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.069+0000 7efda2d9d640 1 -- 172.21.15.150:0/1149244332 reap_dead start 2024-09-17T05:20:23.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.069+0000 7efda8cd0640 1 -- 172.21.15.150:0/1149244332 shutdown_connections 2024-09-17T05:20:23.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.069+0000 7efda8cd0640 1 -- 172.21.15.150:0/1149244332 >> 172.21.15.150:0/1149244332 conn(0x7efda4074750 msgr2=0x7efda4074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:23.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.069+0000 7efda8cd0640 1 -- 172.21.15.150:0/1149244332 shutdown_connections 2024-09-17T05:20:23.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.069+0000 7efda8cd0640 1 -- 172.21.15.150:0/1149244332 wait complete. 2024-09-17T05:20:23.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda8cd0640 1 Processor -- start 2024-09-17T05:20:23.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda8cd0640 1 -- start start 2024-09-17T05:20:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda8cd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd90004d50 0x7efda4158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda8cd0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efda4157bd0 0x7efda4157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda8cd0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efda41e1e60 0x7efda41e2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda8cd0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efda40c4f30 con 0x7efd90004d50 2024-09-17T05:20:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda8cd0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efda41567e0 con 0x7efda41e1e60 2024-09-17T05:20:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda8cd0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7efda40c50b0 con 0x7efda4157bd0 2024-09-17T05:20:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd90004d50 0x7efda4158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd90004d50 0x7efda4158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52588/0 (socket says 172.21.15.150:52588) 2024-09-17T05:20:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda259c640 1 -- 172.21.15.150:0/4211704720 learned_addr learned my addr 172.21.15.150:0/4211704720 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda1d9b640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efda4157bd0 0x7efda4157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda2d9d640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efda41e1e60 0x7efda41e2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda259c640 1 -- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efda4157bd0 msgr2=0x7efda4157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda259c640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efda4157bd0 0x7efda4157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda259c640 1 -- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efda41e1e60 msgr2=0x7efda41e2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:23.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda259c640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efda41e1e60 0x7efda41e2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:23.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda259c640 1 -- 172.21.15.150:0/4211704720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda4158680 con 0x7efd90004d50 2024-09-17T05:20:23.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda259c640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd90004d50 0x7efda4158d50 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7efd94010a40 tx=0x7efd94010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:23.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efd9affd640 1 -- 172.21.15.150:0/4211704720 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9400eda0 con 0x7efd90004d50 2024-09-17T05:20:23.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efd9affd640 1 -- 172.21.15.150:0/4211704720 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd94005020 con 0x7efd90004d50 2024-09-17T05:20:23.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda8cd0640 1 -- 172.21.15.150:0/4211704720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efda4158960 con 0x7efd90004d50 2024-09-17T05:20:23.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efd9affd640 1 -- 172.21.15.150:0/4211704720 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd94005450 con 0x7efd90004d50 2024-09-17T05:20:23.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda8cd0640 1 -- 172.21.15.150:0/4211704720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efda41e27b0 con 0x7efd90004d50 2024-09-17T05:20:23.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.073+0000 7efda8cd0640 1 -- 172.21.15.150:0/4211704720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efda415a880 con 0x7efd90004d50 2024-09-17T05:20:23.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.077+0000 7efd9affd640 1 -- 172.21.15.150:0/4211704720 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7efd940047b0 con 0x7efd90004d50 2024-09-17T05:20:23.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.077+0000 7efd9affd640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efd84073230 0x7efd84075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:23.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.077+0000 7efd9affd640 1 -- 172.21.15.150:0/4211704720 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7efd940965c0 con 0x7efd90004d50 2024-09-17T05:20:23.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.077+0000 7efda1d9b640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efd84073230 0x7efd84075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:23.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.077+0000 7efda1d9b640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efd84073230 0x7efd84075740 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7efda40c9f50 tx=0x7efd8c001280 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:23.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.081+0000 7efd9affd640 1 -- 172.21.15.150:0/4211704720 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7efd94060510 con 0x7efd90004d50 2024-09-17T05:20:23.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.285+0000 7efda8cd0640 1 -- 172.21.15.150:0/4211704720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efda4157330 con 0x7efd90004d50 2024-09-17T05:20:23.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efd9affd640 1 -- 172.21.15.150:0/4211704720 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+6552 (secure 0 0 0) 0x7efd9405fe10 con 0x7efd90004d50 2024-09-17T05:20:23.286 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:23.286 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":209,"btime":"2024-09-17T05:20:02:371612+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":7567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":209}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:20:23.286 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 209 2024-09-17T05:20:23.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efda8cd0640 1 -- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efd84073230 msgr2=0x7efd84075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:23.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efda8cd0640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efd84073230 0x7efd84075740 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7efda40c9f50 tx=0x7efd8c001280 comp rx=0 tx=0).stop 2024-09-17T05:20:23.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efda8cd0640 1 -- 172.21.15.150:0/4211704720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd90004d50 msgr2=0x7efda4158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:23.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efda8cd0640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd90004d50 0x7efda4158d50 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7efd94010a40 tx=0x7efd94010f00 comp rx=0 tx=0).stop 2024-09-17T05:20:23.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efda8cd0640 1 -- 172.21.15.150:0/4211704720 shutdown_connections 2024-09-17T05:20:23.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efda8cd0640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7efd84073230 0x7efd84075740 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:23.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efda8cd0640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efda41e1e60 0x7efda41e2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:23.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efda8cd0640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7efda4157bd0 0x7efda4157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:23.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efda8cd0640 1 --2- 172.21.15.150:0/4211704720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd90004d50 0x7efda4158d50 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:23.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efda8cd0640 1 -- 172.21.15.150:0/4211704720 >> 172.21.15.150:0/4211704720 conn(0x7efda4074750 msgr2=0x7efda41525c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:23.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efda8cd0640 1 -- 172.21.15.150:0/4211704720 shutdown_connections 2024-09-17T05:20:23.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:23.289+0000 7efda8cd0640 1 -- 172.21.15.150:0/4211704720 wait complete. 2024-09-17T05:20:23.308 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:20:25.312 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:25.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.485+0000 7f567629d640 1 -- 172.21.15.150:0/3809221304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5658004090 msgr2=0x7f5658004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:25.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.485+0000 7f567629d640 1 --2- 172.21.15.150:0/3809221304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5658004090 0x7f5658004810 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f566000c2e0 tx=0x7f566001f6e0 comp rx=0 tx=0).stop 2024-09-17T05:20:25.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.485+0000 7f567483a640 1 -- 172.21.15.150:0/3809221304 reap_dead start 2024-09-17T05:20:25.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.485+0000 7f567629d640 1 -- 172.21.15.150:0/3809221304 shutdown_connections 2024-09-17T05:20:25.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.485+0000 7f567629d640 1 -- 172.21.15.150:0/3809221304 >> 172.21.15.150:0/3809221304 conn(0x7f56700c4a00 msgr2=0x7f56700c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:25.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.485+0000 7f567629d640 1 -- 172.21.15.150:0/3809221304 shutdown_connections 2024-09-17T05:20:25.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.485+0000 7f567629d640 1 -- 172.21.15.150:0/3809221304 wait complete. 2024-09-17T05:20:25.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.485+0000 7f567629d640 1 Processor -- start 2024-09-17T05:20:25.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.485+0000 7f567629d640 1 -- start start 2024-09-17T05:20:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f567629d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5658004090 0x7f5670156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f567629d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56700ac920 0x7f5670157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f567629d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56700acd30 0x7f5670148110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f567629d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f56700c3000 con 0x7f5658004090 2024-09-17T05:20:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f567629d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5670155560 con 0x7f56700acd30 2024-09-17T05:20:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f567629d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f56700add70 con 0x7f56700ac920 2024-09-17T05:20:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5658004090 0x7f5670156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:25.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5658004090 0x7f5670156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52604/0 (socket says 172.21.15.150:52604) 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566ffff640 1 -- 172.21.15.150:0/3123061090 learned_addr learned my addr 172.21.15.150:0/3123061090 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f567483a640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56700acd30 0x7f5670148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566f7fe640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56700ac920 0x7f5670157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566ffff640 1 -- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56700ac920 msgr2=0x7f5670157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566ffff640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56700ac920 0x7f5670157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566ffff640 1 -- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56700acd30 msgr2=0x7f5670148110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566ffff640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56700acd30 0x7f5670148110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566ffff640 1 -- 172.21.15.150:0/3123061090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5670156fd0 con 0x7f5658004090 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566f7fe640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56700ac920 0x7f5670157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566ffff640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5658004090 0x7f5670156a90 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f566400e770 tx=0x7f566400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566cff9640 1 -- 172.21.15.150:0/3123061090 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5664016070 con 0x7f5658004090 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566cff9640 1 -- 172.21.15.150:0/3123061090 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f566400ee80 con 0x7f5658004090 2024-09-17T05:20:25.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f567629d640 1 -- 172.21.15.150:0/3123061090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5670152060 con 0x7f5658004090 2024-09-17T05:20:25.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566cff9640 1 -- 172.21.15.150:0/3123061090 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566400ca90 con 0x7f5658004090 2024-09-17T05:20:25.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f567629d640 1 -- 172.21.15.150:0/3123061090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5670156540 con 0x7f5658004090 2024-09-17T05:20:25.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f567629d640 1 -- 172.21.15.150:0/3123061090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56700a75f0 con 0x7f5658004090 2024-09-17T05:20:25.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566cff9640 1 -- 172.21.15.150:0/3123061090 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5664019070 con 0x7f5658004090 2024-09-17T05:20:25.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566cff9640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f565c073200 0x7f565c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:25.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.489+0000 7f566cff9640 1 -- 172.21.15.150:0/3123061090 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7f5664095d90 con 0x7f5658004090 2024-09-17T05:20:25.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.493+0000 7f566f7fe640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f565c073200 0x7f565c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:25.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.493+0000 7f566f7fe640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f565c073200 0x7f565c075710 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f56600b0000 tx=0x7f566001c520 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:25.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.493+0000 7f566cff9640 1 -- 172.21.15.150:0/3123061090 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f566405fdd0 con 0x7f5658004090 2024-09-17T05:20:25.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.701+0000 7f567629d640 1 -- 172.21.15.150:0/3123061090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56701562f0 con 0x7f5658004090 2024-09-17T05:20:25.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.701+0000 7f566cff9640 1 -- 172.21.15.150:0/3123061090 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+6552 (secure 0 0 0) 0x7f566405f6d0 con 0x7f5658004090 2024-09-17T05:20:25.698 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:25.698 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":209,"btime":"2024-09-17T05:20:02:371612+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":7567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":209}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:20:25.698 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 209 2024-09-17T05:20:25.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.705+0000 7f567629d640 1 -- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f565c073200 msgr2=0x7f565c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:25.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.705+0000 7f567629d640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f565c073200 0x7f565c075710 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f56600b0000 tx=0x7f566001c520 comp rx=0 tx=0).stop 2024-09-17T05:20:25.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.705+0000 7f567629d640 1 -- 172.21.15.150:0/3123061090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5658004090 msgr2=0x7f5670156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:25.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.705+0000 7f567629d640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5658004090 0x7f5670156a90 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f566400e770 tx=0x7f566400ec30 comp rx=0 tx=0).stop 2024-09-17T05:20:25.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.705+0000 7f567629d640 1 -- 172.21.15.150:0/3123061090 shutdown_connections 2024-09-17T05:20:25.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.705+0000 7f567629d640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f565c073200 0x7f565c075710 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:25.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.705+0000 7f567629d640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f56700acd30 0x7f5670148110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:25.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.705+0000 7f567629d640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f56700ac920 0x7f5670157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:25.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.705+0000 7f567629d640 1 --2- 172.21.15.150:0/3123061090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5658004090 0x7f5670156a90 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:25.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.705+0000 7f567629d640 1 -- 172.21.15.150:0/3123061090 >> 172.21.15.150:0/3123061090 conn(0x7f56700c4a00 msgr2=0x7f5670154090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:25.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.705+0000 7f567629d640 1 -- 172.21.15.150:0/3123061090 shutdown_connections 2024-09-17T05:20:25.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:25.705+0000 7f567629d640 1 -- 172.21.15.150:0/3123061090 wait complete. 2024-09-17T05:20:25.723 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:20:27.725 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:27.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.896+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/1639735383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b4406e590 msgr2=0x7f4b440719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:27.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.896+0000 7f4b4a2b6640 1 --2- 172.21.15.150:0/1639735383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b4406e590 0x7f4b440719e0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f4b44073a40 tx=0x7f4b3401c7a0 comp rx=0 tx=0).stop 2024-09-17T05:20:27.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.896+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/1639735383 shutdown_connections 2024-09-17T05:20:27.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.896+0000 7f4b4a2b6640 1 --2- 172.21.15.150:0/1639735383 >> v2:172.21.15.176:3300/0 conn(0x7f4b440a98a0 0x7f4b440a9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:27.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.896+0000 7f4b4a2b6640 1 --2- 172.21.15.150:0/1639735383 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b440699b0 0x7f4b44151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:27.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.896+0000 7f4b4a2b6640 1 --2- 172.21.15.150:0/1639735383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b4406e590 0x7f4b440719e0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:27.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.896+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/1639735383 >> 172.21.15.150:0/1639735383 conn(0x7f4b44070b50 msgr2=0x7f4b44070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:27.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.896+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/1639735383 shutdown_connections 2024-09-17T05:20:27.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.896+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/1639735383 wait complete. 2024-09-17T05:20:27.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b4a2b6640 1 Processor -- start 2024-09-17T05:20:27.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b4a2b6640 1 -- start start 2024-09-17T05:20:27.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b4a2b6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b440699b0 0x7f4b44151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:27.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b4a2b6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b4406e590 0x7f4b4414a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b4a2b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b440a98a0 0x7f4b44148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b4a2b6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44158120 con 0x7f4b440a98a0 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b4a2b6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44157fa0 con 0x7f4b440699b0 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b4a2b6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4b441582a0 con 0x7f4b4406e590 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b48853640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b440a98a0 0x7f4b44148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b48853640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b440a98a0 0x7f4b44148bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:52626/0 (socket says 172.21.15.150:52626) 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b48853640 1 -- 172.21.15.150:0/256614094 learned_addr learned my addr 172.21.15.150:0/256614094 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b43fff640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b440699b0 0x7f4b44151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b437fe640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b4406e590 0x7f4b4414a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b48853640 1 -- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b4406e590 msgr2=0x7f4b4414a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b48853640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b4406e590 0x7f4b4414a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b48853640 1 -- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b440699b0 msgr2=0x7f4b44151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:27.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b48853640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b440699b0 0x7f4b44151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:27.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b48853640 1 -- 172.21.15.150:0/256614094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b44149e50 con 0x7f4b440a98a0 2024-09-17T05:20:27.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b48853640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b440a98a0 0x7f4b44148bf0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00caa0 tx=0x7f4b3c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:27.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b40ff9640 1 -- 172.21.15.150:0/256614094 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c00ad90 con 0x7f4b440a98a0 2024-09-17T05:20:27.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b40ff9640 1 -- 172.21.15.150:0/256614094 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b3c003a70 con 0x7f4b440a98a0 2024-09-17T05:20:27.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b40ff9640 1 -- 172.21.15.150:0/256614094 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c004030 con 0x7f4b440a98a0 2024-09-17T05:20:27.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/256614094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b4414c0f0 con 0x7f4b440a98a0 2024-09-17T05:20:27.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.900+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/256614094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b441496d0 con 0x7f4b440a98a0 2024-09-17T05:20:27.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.904+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/256614094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b4414f9b0 con 0x7f4b440a98a0 2024-09-17T05:20:27.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.904+0000 7f4b40ff9640 1 -- 172.21.15.150:0/256614094 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4b3c0041d0 con 0x7f4b440a98a0 2024-09-17T05:20:27.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.904+0000 7f4b40ff9640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4b30073230 0x7f4b30075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:27.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.904+0000 7f4b40ff9640 1 -- 172.21.15.150:0/256614094 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7f4b3c095c40 con 0x7f4b440a98a0 2024-09-17T05:20:27.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.904+0000 7f4b43fff640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4b30073230 0x7f4b30075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:27.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.904+0000 7f4b43fff640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4b30073230 0x7f4b30075740 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f4b340ae000 tx=0x7f4b3401afe0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:27.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:27.908+0000 7f4b40ff9640 1 -- 172.21.15.150:0/256614094 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4b4414f9b0 con 0x7f4b440a98a0 2024-09-17T05:20:28.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.120+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/256614094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b4414f580 con 0x7f4b440a98a0 2024-09-17T05:20:28.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.120+0000 7f4b40ff9640 1 -- 172.21.15.150:0/256614094 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+6552 (secure 0 0 0) 0x7f4b3c014070 con 0x7f4b440a98a0 2024-09-17T05:20:28.119 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:28.119 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":209,"btime":"2024-09-17T05:20:02:371612+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":7567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":209}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:20:28.119 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 209 2024-09-17T05:20:28.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.124+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4b30073230 msgr2=0x7f4b30075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:28.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.124+0000 7f4b4a2b6640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4b30073230 0x7f4b30075740 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f4b340ae000 tx=0x7f4b3401afe0 comp rx=0 tx=0).stop 2024-09-17T05:20:28.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.124+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/256614094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b440a98a0 msgr2=0x7f4b44148bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:28.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.124+0000 7f4b4a2b6640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b440a98a0 0x7f4b44148bf0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00caa0 tx=0x7f4b3c00cf60 comp rx=0 tx=0).stop 2024-09-17T05:20:28.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.124+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/256614094 shutdown_connections 2024-09-17T05:20:28.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.124+0000 7f4b4a2b6640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4b30073230 0x7f4b30075740 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:28.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.124+0000 7f4b4a2b6640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4b440a98a0 0x7f4b44148bf0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:28.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.124+0000 7f4b4a2b6640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4b4406e590 0x7f4b4414a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:28.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.124+0000 7f4b4a2b6640 1 --2- 172.21.15.150:0/256614094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b440699b0 0x7f4b44151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:28.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.124+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/256614094 >> 172.21.15.150:0/256614094 conn(0x7f4b44070b50 msgr2=0x7f4b440c3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:28.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.124+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/256614094 shutdown_connections 2024-09-17T05:20:28.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:28.124+0000 7f4b4a2b6640 1 -- 172.21.15.150:0/256614094 wait complete. 2024-09-17T05:20:28.144 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:20:30.145 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:30.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.312+0000 7f6ac6af4640 1 -- 172.21.15.150:0/757500304 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00c6340 msgr2=0x7f6ac0157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:30.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.312+0000 7f6ac6af4640 1 --2- 172.21.15.150:0/757500304 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00c6340 0x7f6ac0157330 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009840 tx=0x7f6ab401c820 comp rx=0 tx=0).stop 2024-09-17T05:20:30.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.312+0000 7f6ac6af4640 1 -- 172.21.15.150:0/757500304 shutdown_connections 2024-09-17T05:20:30.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.312+0000 7f6ac6af4640 1 --2- 172.21.15.150:0/757500304 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ac00c4b20 0x7f6ac0153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:30.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.312+0000 7f6ac6af4640 1 --2- 172.21.15.150:0/757500304 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00c6340 0x7f6ac0157330 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:30.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.312+0000 7f6ac6af4640 1 -- 172.21.15.150:0/757500304 >> 172.21.15.150:0/757500304 conn(0x7f6ac0074750 msgr2=0x7f6ac0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:30.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.312+0000 7f6ac6af4640 1 -- 172.21.15.150:0/757500304 shutdown_connections 2024-09-17T05:20:30.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.312+0000 7f6ac6af4640 1 -- 172.21.15.150:0/757500304 wait complete. 2024-09-17T05:20:30.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac6af4640 1 Processor -- start 2024-09-17T05:20:30.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac6af4640 1 -- start start 2024-09-17T05:20:30.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac6af4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ac00c4b20 0x7f6ac00c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:30.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac6af4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ac00c5df0 0x7f6ac00cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:30.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac6af4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00c6340 0x7f6ac0158170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:30.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac6af4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac0163d10 con 0x7f6ac00c5df0 2024-09-17T05:20:30.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac6af4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac0163b90 con 0x7f6ac00c6340 2024-09-17T05:20:30.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac6af4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac0163e90 con 0x7f6ac00c4b20 2024-09-17T05:20:30.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6abffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ac00c5df0 0x7f6ac00cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:30.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6abffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ac00c5df0 0x7f6ac00cbfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38430/0 (socket says 172.21.15.150:38430) 2024-09-17T05:20:30.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6abffff640 1 -- 172.21.15.150:0/1097686909 learned_addr learned my addr 172.21.15.150:0/1097686909 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:30.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac4890640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ac00c4b20 0x7f6ac00c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:30.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac5091640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00c6340 0x7f6ac0158170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:30.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6abffff640 1 -- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ac00c4b20 msgr2=0x7f6ac00c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:30.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6abffff640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ac00c4b20 0x7f6ac00c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:30.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6abffff640 1 -- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00c6340 msgr2=0x7f6ac0158170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:30.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6abffff640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00c6340 0x7f6ac0158170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:30.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6abffff640 1 -- 172.21.15.150:0/1097686909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac01f4380 con 0x7f6ac00c5df0 2024-09-17T05:20:30.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac5091640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00c6340 0x7f6ac0158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:20:30.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6abffff640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ac00c5df0 0x7f6ac00cbfb0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f6ab401aed0 tx=0x7f6ab40a5670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:30.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6abd7fa640 1 -- 172.21.15.150:0/1097686909 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab40bb070 con 0x7f6ac00c5df0 2024-09-17T05:20:30.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac6af4640 1 -- 172.21.15.150:0/1097686909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac01f4600 con 0x7f6ac00c5df0 2024-09-17T05:20:30.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6abd7fa640 1 -- 172.21.15.150:0/1097686909 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ab4004470 con 0x7f6ac00c5df0 2024-09-17T05:20:30.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6abd7fa640 1 -- 172.21.15.150:0/1097686909 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab40b2040 con 0x7f6ac00c5df0 2024-09-17T05:20:30.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac6af4640 1 -- 172.21.15.150:0/1097686909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac0159dd0 con 0x7f6ac00c5df0 2024-09-17T05:20:30.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.316+0000 7f6ac6af4640 1 -- 172.21.15.150:0/1097686909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ac0153bc0 con 0x7f6ac00c5df0 2024-09-17T05:20:30.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.320+0000 7f6abd7fa640 1 -- 172.21.15.150:0/1097686909 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f6ab4005290 con 0x7f6ac00c5df0 2024-09-17T05:20:30.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.320+0000 7f6abd7fa640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6aac073230 0x7f6aac075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:30.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.320+0000 7f6abd7fa640 1 -- 172.21.15.150:0/1097686909 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7f6ab412ec50 con 0x7f6ac00c5df0 2024-09-17T05:20:30.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.320+0000 7f6ac4890640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6aac073230 0x7f6aac075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:30.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.320+0000 7f6ac4890640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6aac073230 0x7f6aac075740 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0001e40 tx=0x7f6ab0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:30.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.324+0000 7f6abd7fa640 1 -- 172.21.15.150:0/1097686909 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6ab4134020 con 0x7f6ac00c5df0 2024-09-17T05:20:30.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.520+0000 7f6ac6af4640 1 -- 172.21.15.150:0/1097686909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ac00b1b70 con 0x7f6ac00c5df0 2024-09-17T05:20:30.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.520+0000 7f6abd7fa640 1 -- 172.21.15.150:0/1097686909 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+6552 (secure 0 0 0) 0x7f6ac01f4380 con 0x7f6ac00c5df0 2024-09-17T05:20:30.520 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:30.521 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":209,"btime":"2024-09-17T05:20:02:371612+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":7567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":209}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:20:30.521 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 209 2024-09-17T05:20:30.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.524+0000 7f6ac6af4640 1 -- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6aac073230 msgr2=0x7f6aac075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:30.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.524+0000 7f6ac6af4640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6aac073230 0x7f6aac075740 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0001e40 tx=0x7f6ab0009040 comp rx=0 tx=0).stop 2024-09-17T05:20:30.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.524+0000 7f6ac6af4640 1 -- 172.21.15.150:0/1097686909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ac00c5df0 msgr2=0x7f6ac00cbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:30.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.524+0000 7f6ac6af4640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ac00c5df0 0x7f6ac00cbfb0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f6ab401aed0 tx=0x7f6ab40a5670 comp rx=0 tx=0).stop 2024-09-17T05:20:30.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.524+0000 7f6ac6af4640 1 -- 172.21.15.150:0/1097686909 shutdown_connections 2024-09-17T05:20:30.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.524+0000 7f6ac6af4640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6aac073230 0x7f6aac075740 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:30.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.524+0000 7f6ac6af4640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac00c6340 0x7f6ac0158170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:30.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.524+0000 7f6ac6af4640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ac00c5df0 0x7f6ac00cbfb0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:30.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.524+0000 7f6ac6af4640 1 --2- 172.21.15.150:0/1097686909 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f6ac00c4b20 0x7f6ac00c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:30.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.524+0000 7f6ac6af4640 1 -- 172.21.15.150:0/1097686909 >> 172.21.15.150:0/1097686909 conn(0x7f6ac0074750 msgr2=0x7f6ac00c1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:30.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.524+0000 7f6ac6af4640 1 -- 172.21.15.150:0/1097686909 shutdown_connections 2024-09-17T05:20:30.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:30.524+0000 7f6ac6af4640 1 -- 172.21.15.150:0/1097686909 wait complete. 2024-09-17T05:20:30.543 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:20:32.544 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:32.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.716+0000 7f9f0ce33640 1 -- 172.21.15.150:0/2936295332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f080a7350 msgr2=0x7f9f08150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:32.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.716+0000 7f9f0ce33640 1 --2- 172.21.15.150:0/2936295332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f080a7350 0x7f9f08150060 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009900 tx=0x7f9efc01c8e0 comp rx=0 tx=0).stop 2024-09-17T05:20:32.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.716+0000 7f9f0ce33640 1 -- 172.21.15.150:0/2936295332 shutdown_connections 2024-09-17T05:20:32.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.716+0000 7f9f0ce33640 1 --2- 172.21.15.150:0/2936295332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f080a7350 0x7f9f08150060 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:32.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.716+0000 7f9f0ce33640 1 --2- 172.21.15.150:0/2936295332 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f080c42e0 0x7f9f080c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:32.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.716+0000 7f9f0ce33640 1 -- 172.21.15.150:0/2936295332 >> 172.21.15.150:0/2936295332 conn(0x7f9f080c4a00 msgr2=0x7f9f080c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:32.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.716+0000 7f9f0ce33640 1 -- 172.21.15.150:0/2936295332 shutdown_connections 2024-09-17T05:20:32.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.716+0000 7f9f0ce33640 1 -- 172.21.15.150:0/2936295332 wait complete. 2024-09-17T05:20:32.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.716+0000 7f9f0ce33640 1 Processor -- start 2024-09-17T05:20:32.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.716+0000 7f9f0ce33640 1 -- start start 2024-09-17T05:20:32.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0ce33640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f080a7350 0x7f9f08152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:32.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0ce33640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f080c42e0 0x7f9f080c1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:32.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0ce33640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f081505a0 0x7f9f080acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:32.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0ce33640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9f080c6d50 con 0x7f9f080a7350 2024-09-17T05:20:32.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0ce33640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9f080c6bd0 con 0x7f9f081505a0 2024-09-17T05:20:32.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0ce33640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9f0815fe90 con 0x7f9f080c42e0 2024-09-17T05:20:32.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f080a7350 0x7f9f08152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:32.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f080a7350 0x7f9f08152490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38448/0 (socket says 172.21.15.150:38448) 2024-09-17T05:20:32.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0659c640 1 -- 172.21.15.150:0/801464039 learned_addr learned my addr 172.21.15.150:0/801464039 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:32.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f06d9d640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f081505a0 0x7f9f080acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:32.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f05d9b640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f080c42e0 0x7f9f080c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:32.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0659c640 1 -- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f080c42e0 msgr2=0x7f9f080c1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:32.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0659c640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f080c42e0 0x7f9f080c1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:32.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0659c640 1 -- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f081505a0 msgr2=0x7f9f080acf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:32.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0659c640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f081505a0 0x7f9f080acf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:32.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0659c640 1 -- 172.21.15.150:0/801464039 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f081529d0 con 0x7f9f080a7350 2024-09-17T05:20:32.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f05d9b640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f080c42e0 0x7f9f080c1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:20:32.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0659c640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f080a7350 0x7f9f08152490 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c970 tx=0x7f9ef800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:32.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/801464039 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8004220 con 0x7f9f080a7350 2024-09-17T05:20:32.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/801464039 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ef8007950 con 0x7f9f080a7350 2024-09-17T05:20:32.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0ce33640 1 -- 172.21.15.150:0/801464039 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f081f3de0 con 0x7f9f080a7350 2024-09-17T05:20:32.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/801464039 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef80044b0 con 0x7f9f080a7350 2024-09-17T05:20:32.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0ce33640 1 -- 172.21.15.150:0/801464039 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f08155670 con 0x7f9f080a7350 2024-09-17T05:20:32.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.720+0000 7f9f0ce33640 1 -- 172.21.15.150:0/801464039 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f080c76d0 con 0x7f9f080a7350 2024-09-17T05:20:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.724+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/801464039 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9ef8005020 con 0x7f9f080a7350 2024-09-17T05:20:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.724+0000 7f9ef6ffd640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ee8073200 0x7f9ee8075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.724+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/801464039 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7f9ef8095730 con 0x7f9f080a7350 2024-09-17T05:20:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.724+0000 7f9f05d9b640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ee8073200 0x7f9ee8075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.724+0000 7f9f05d9b640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ee8073200 0x7f9ee8075710 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0af000 tx=0x7f9efc01b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.728+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/801464039 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9ef8014030 con 0x7f9f080a7350 2024-09-17T05:20:32.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.936+0000 7f9f0ce33640 1 -- 172.21.15.150:0/801464039 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f08150060 con 0x7f9f080a7350 2024-09-17T05:20:32.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.936+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/801464039 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+6552 (secure 0 0 0) 0x7f9ef805f810 con 0x7f9f080a7350 2024-09-17T05:20:32.935 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:32.935 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":209,"btime":"2024-09-17T05:20:02:371612+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":7567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":209}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:20:32.935 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 209 2024-09-17T05:20:32.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.940+0000 7f9f0ce33640 1 -- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ee8073200 msgr2=0x7f9ee8075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:32.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.940+0000 7f9f0ce33640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ee8073200 0x7f9ee8075710 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0af000 tx=0x7f9efc01b120 comp rx=0 tx=0).stop 2024-09-17T05:20:32.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.940+0000 7f9f0ce33640 1 -- 172.21.15.150:0/801464039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f080a7350 msgr2=0x7f9f08152490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:32.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.940+0000 7f9f0ce33640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f080a7350 0x7f9f08152490 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c970 tx=0x7f9ef800ce30 comp rx=0 tx=0).stop 2024-09-17T05:20:32.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.940+0000 7f9f0ce33640 1 -- 172.21.15.150:0/801464039 shutdown_connections 2024-09-17T05:20:32.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.940+0000 7f9f0ce33640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9ee8073200 0x7f9ee8075710 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:32.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.940+0000 7f9f0ce33640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9f081505a0 0x7f9f080acf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:32.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.940+0000 7f9f0ce33640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9f080c42e0 0x7f9f080c1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:32.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.940+0000 7f9f0ce33640 1 --2- 172.21.15.150:0/801464039 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f080a7350 0x7f9f08152490 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:32.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.940+0000 7f9f0ce33640 1 -- 172.21.15.150:0/801464039 >> 172.21.15.150:0/801464039 conn(0x7f9f080c4a00 msgr2=0x7f9f0815c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:32.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.940+0000 7f9f0ce33640 1 -- 172.21.15.150:0/801464039 shutdown_connections 2024-09-17T05:20:32.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:32.940+0000 7f9f0ce33640 1 -- 172.21.15.150:0/801464039 wait complete. 2024-09-17T05:20:32.958 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:20:34.959 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:35.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.128+0000 7f878b8cb640 1 -- 172.21.15.150:0/2408627158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8774005360 msgr2=0x7f877400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:35.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.128+0000 7f878b8cb640 1 --2- 172.21.15.150:0/2408627158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8774005360 0x7f877400b9d0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f8770009b30 tx=0x7f877001d270 comp rx=0 tx=0).stop 2024-09-17T05:20:35.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.128+0000 7f878b8cb640 1 -- 172.21.15.150:0/2408627158 shutdown_connections 2024-09-17T05:20:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.128+0000 7f878b8cb640 1 --2- 172.21.15.150:0/2408627158 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8774005360 0x7f877400b9d0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.128+0000 7f878b8cb640 1 -- 172.21.15.150:0/2408627158 >> 172.21.15.150:0/2408627158 conn(0x7f87840c5ae0 msgr2=0x7f8784073c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.128+0000 7f878b8cb640 1 -- 172.21.15.150:0/2408627158 shutdown_connections 2024-09-17T05:20:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.128+0000 7f878b8cb640 1 -- 172.21.15.150:0/2408627158 wait complete. 2024-09-17T05:20:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f878b8cb640 1 Processor -- start 2024-09-17T05:20:35.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f878b8cb640 1 -- start start 2024-09-17T05:20:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f878b8cb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8774005360 0x7f87840ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f878b8cb640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87841ed4b0 0x7f87840c1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f878b8cb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f878414b330 0x7f8784149a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:35.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f878b8cb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8784152390 con 0x7f878414b330 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f878b8cb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f87840c2930 con 0x7f8774005360 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f878b8cb640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8784074ed0 con 0x7f87841ed4b0 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8789e68640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f878414b330 0x7f8784149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8789e68640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f878414b330 0x7f8784149a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38452/0 (socket says 172.21.15.150:38452) 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8789e68640 1 -- 172.21.15.150:0/2555124787 learned_addr learned my addr 172.21.15.150:0/2555124787 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8789667640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8774005360 0x7f87840ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8788e66640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87841ed4b0 0x7f87840c1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8789e68640 1 -- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87841ed4b0 msgr2=0x7f87840c1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8789e68640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87841ed4b0 0x7f87840c1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8789e68640 1 -- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8774005360 msgr2=0x7f87840ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8789e68640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8774005360 0x7f87840ac170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8789e68640 1 -- 172.21.15.150:0/2555124787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87840ace80 con 0x7f878414b330 2024-09-17T05:20:35.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8789667640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8774005360 0x7f87840ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:20:35.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8789e68640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f878414b330 0x7f8784149a20 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f8770009b30 tx=0x7f87700aca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:35.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8781ffb640 1 -- 172.21.15.150:0/2555124787 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8770009730 con 0x7f878414b330 2024-09-17T05:20:35.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8781ffb640 1 -- 172.21.15.150:0/2555124787 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f87700bc070 con 0x7f878414b330 2024-09-17T05:20:35.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f878b8cb640 1 -- 172.21.15.150:0/2555124787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87840ad160 con 0x7f878414b330 2024-09-17T05:20:35.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f8781ffb640 1 -- 172.21.15.150:0/2555124787 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87700acab0 con 0x7f878414b330 2024-09-17T05:20:35.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.132+0000 7f878b8cb640 1 -- 172.21.15.150:0/2555124787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87840ac6b0 con 0x7f878414b330 2024-09-17T05:20:35.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.136+0000 7f878b8cb640 1 -- 172.21.15.150:0/2555124787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87840c3020 con 0x7f878414b330 2024-09-17T05:20:35.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.136+0000 7f8781ffb640 1 -- 172.21.15.150:0/2555124787 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f87700b5040 con 0x7f878414b330 2024-09-17T05:20:35.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.136+0000 7f8781ffb640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8768073200 0x7f8768075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:35.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.136+0000 7f8781ffb640 1 -- 172.21.15.150:0/2555124787 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10106+0+0 (secure 0 0 0) 0x7f87701303d0 con 0x7f878414b330 2024-09-17T05:20:35.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.136+0000 7f8789667640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8768073200 0x7f8768075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:35.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.136+0000 7f8789667640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8768073200 0x7f8768075710 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f877c006fd0 tx=0x7f877c006b70 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:35.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.140+0000 7f8781ffb640 1 -- 172.21.15.150:0/2555124787 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8770016f80 con 0x7f878414b330 2024-09-17T05:20:35.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.340+0000 7f878b8cb640 1 -- 172.21.15.150:0/2555124787 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87841ed970 con 0x7f878414b330 2024-09-17T05:20:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.340+0000 7f8781ffb640 1 -- 172.21.15.150:0/2555124787 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+6552 (secure 0 0 0) 0x7f87700fa300 con 0x7f878414b330 2024-09-17T05:20:35.339 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:35.339 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":209,"btime":"2024-09-17T05:20:02:371612+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":7567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":209}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:18:34.641250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8031},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8031":{"gid":8031,"name":"a","rank":0,"incarnation":199,"state":"up:active","state_seq":26,"addr":"172.21.15.150:6833/2857695220","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":2857695220},{"type":"v1","addr":"172.21.15.150:6833","nonce":2857695220}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8031,"qdb_cluster":[8031]},"id":2}]} 2024-09-17T05:20:35.340 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 209 2024-09-17T05:20:35.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.344+0000 7f878b8cb640 1 -- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8768073200 msgr2=0x7f8768075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.344+0000 7f878b8cb640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8768073200 0x7f8768075710 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f877c006fd0 tx=0x7f877c006b70 comp rx=0 tx=0).stop 2024-09-17T05:20:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.344+0000 7f878b8cb640 1 -- 172.21.15.150:0/2555124787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f878414b330 msgr2=0x7f8784149a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.344+0000 7f878b8cb640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f878414b330 0x7f8784149a20 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f8770009b30 tx=0x7f87700aca40 comp rx=0 tx=0).stop 2024-09-17T05:20:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.344+0000 7f878b8cb640 1 -- 172.21.15.150:0/2555124787 shutdown_connections 2024-09-17T05:20:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.344+0000 7f878b8cb640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8768073200 0x7f8768075710 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.344+0000 7f878b8cb640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f878414b330 0x7f8784149a20 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.344+0000 7f878b8cb640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f87841ed4b0 0x7f87840c1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.344+0000 7f878b8cb640 1 --2- 172.21.15.150:0/2555124787 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8774005360 0x7f87840ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.344+0000 7f878b8cb640 1 -- 172.21.15.150:0/2555124787 >> 172.21.15.150:0/2555124787 conn(0x7f87840c5ae0 msgr2=0x7f8784074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.344+0000 7f878b8cb640 1 -- 172.21.15.150:0/2555124787 shutdown_connections 2024-09-17T05:20:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:35.344+0000 7f878b8cb640 1 -- 172.21.15.150:0/2555124787 wait complete. 2024-09-17T05:20:35.361 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-09-17T05:20:37.362 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:37.497 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:20:37.499 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:20:37.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.532+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/1794097130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e00c97a0 msgr2=0x7fc9e0151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:37.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.532+0000 7fc9e5c8f640 1 --2- 172.21.15.150:0/1794097130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e00c97a0 0x7fc9e0151c70 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009900 tx=0x7fc9d001c8e0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.532+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/1794097130 shutdown_connections 2024-09-17T05:20:37.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.532+0000 7fc9e5c8f640 1 --2- 172.21.15.150:0/1794097130 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9e0072ea0 0x7fc9e00732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.532+0000 7fc9e5c8f640 1 --2- 172.21.15.150:0/1794097130 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e00c97a0 0x7fc9e0151c70 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.532+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/1794097130 >> 172.21.15.150:0/1794097130 conn(0x7fc9e00abb00 msgr2=0x7fc9e00abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:37.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.532+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/1794097130 shutdown_connections 2024-09-17T05:20:37.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.532+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/1794097130 wait complete. 2024-09-17T05:20:37.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.532+0000 7fc9e5c8f640 1 Processor -- start 2024-09-17T05:20:37.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.532+0000 7fc9e5c8f640 1 -- start start 2024-09-17T05:20:37.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9e5c8f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9e0072ea0 0x7fc9e014bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:37.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9e5c8f640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9e00c97a0 0x7fc9e0156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:37.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9e5c8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e0157be0 0x7fc9e014e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:37.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9e5c8f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e0161910 con 0x7fc9e0157be0 2024-09-17T05:20:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9e5c8f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e0161790 con 0x7fc9e0072ea0 2024-09-17T05:20:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9e5c8f640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e0161a90 con 0x7fc9e00c97a0 2024-09-17T05:20:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e0157be0 0x7fc9e014e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e0157be0 0x7fc9e014e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38466/0 (socket says 172.21.15.150:38466) 2024-09-17T05:20:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dffff640 1 -- 172.21.15.150:0/2836355632 learned_addr learned my addr 172.21.15.150:0/2836355632 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9df7fe640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9e0072ea0 0x7fc9e014bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9d7fff640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9e00c97a0 0x7fc9e0156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dffff640 1 -- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9e00c97a0 msgr2=0x7fc9e0156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dffff640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9e00c97a0 0x7fc9e0156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dffff640 1 -- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9e0072ea0 msgr2=0x7fc9e014bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dffff640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9e0072ea0 0x7fc9e014bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dffff640 1 -- 172.21.15.150:0/2836355632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e0151390 con 0x7fc9e0157be0 2024-09-17T05:20:37.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dffff640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e0157be0 0x7fc9e014e740 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800c710 tx=0x7fc9d800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:37.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dcff9640 1 -- 172.21.15.150:0/2836355632 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8003b80 con 0x7fc9e0157be0 2024-09-17T05:20:37.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dcff9640 1 -- 172.21.15.150:0/2836355632 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9d8003d20 con 0x7fc9e0157be0 2024-09-17T05:20:37.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/2836355632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e01510b0 con 0x7fc9e0157be0 2024-09-17T05:20:37.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dcff9640 1 -- 172.21.15.150:0/2836355632 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d800aa50 con 0x7fc9e0157be0 2024-09-17T05:20:37.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/2836355632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e01488e0 con 0x7fc9e0157be0 2024-09-17T05:20:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/2836355632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9e00732b0 con 0x7fc9e0157be0 2024-09-17T05:20:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dcff9640 1 -- 172.21.15.150:0/2836355632 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc9d800abf0 con 0x7fc9e0157be0 2024-09-17T05:20:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dcff9640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc9c0073230 0x7fc9c0075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.536+0000 7fc9dcff9640 1 -- 172.21.15.150:0/2836355632 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7fc9d8094f80 con 0x7fc9e0157be0 2024-09-17T05:20:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.540+0000 7fc9df7fe640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc9c0073230 0x7fc9c0075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:37.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.540+0000 7fc9df7fe640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc9c0073230 0x7fc9c0075740 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc9d00af000 tx=0x7fc9d001b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:37.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.544+0000 7fc9dcff9640 1 -- 172.21.15.150:0/2836355632 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc9d800ce20 con 0x7fc9e0157be0 2024-09-17T05:20:37.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.752+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/2836355632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9e00c40f0 con 0x7fc9e0157be0 2024-09-17T05:20:37.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.752+0000 7fc9dcff9640 1 -- 172.21.15.150:0/2836355632 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) ==== 95+0+5719 (secure 0 0 0) 0x7fc9d805e8a0 con 0x7fc9e0157be0 2024-09-17T05:20:37.750 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:37.750 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":211,"btime":"2024-09-17T05:20:37:499645+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:20:36.887029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7567":{"gid":7567,"name":"b","rank":0,"incarnation":210,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:20:37.750 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 211 2024-09-17T05:20:37.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.756+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc9c0073230 msgr2=0x7fc9c0075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:37.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.756+0000 7fc9e5c8f640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc9c0073230 0x7fc9c0075740 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc9d00af000 tx=0x7fc9d001b120 comp rx=0 tx=0).stop 2024-09-17T05:20:37.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.756+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/2836355632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e0157be0 msgr2=0x7fc9e014e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:37.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.756+0000 7fc9e5c8f640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e0157be0 0x7fc9e014e740 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800c710 tx=0x7fc9d800cbd0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.756+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/2836355632 shutdown_connections 2024-09-17T05:20:37.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.756+0000 7fc9e5c8f640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc9c0073230 0x7fc9c0075740 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.756+0000 7fc9e5c8f640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9e0157be0 0x7fc9e014e740 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.756+0000 7fc9e5c8f640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc9e00c97a0 0x7fc9e0156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.756+0000 7fc9e5c8f640 1 --2- 172.21.15.150:0/2836355632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9e0072ea0 0x7fc9e014bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.756+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/2836355632 >> 172.21.15.150:0/2836355632 conn(0x7fc9e00abb00 msgr2=0x7fc9e00aa030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:37.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.756+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/2836355632 shutdown_connections 2024-09-17T05:20:37.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.756+0000 7fc9e5c8f640 1 -- 172.21.15.150:0/2836355632 wait complete. 2024-09-17T05:20:37.774 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-09-17T05:20:37.775 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-09-17T05:20:37.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:37.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.936+0000 7fcb5a35c640 1 -- 172.21.15.150:0/2487894416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb540c3ac0 msgr2=0x7fcb541501c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.936+0000 7fcb5a35c640 1 --2- 172.21.15.150:0/2487894416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb540c3ac0 0x7fcb541501c0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c009900 tx=0x7fcb3c01c850 comp rx=0 tx=0).stop 2024-09-17T05:20:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.936+0000 7fcb5a35c640 1 -- 172.21.15.150:0/2487894416 shutdown_connections 2024-09-17T05:20:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.936+0000 7fcb5a35c640 1 --2- 172.21.15.150:0/2487894416 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb541528c0 0x7fcb54152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.936+0000 7fcb5a35c640 1 --2- 172.21.15.150:0/2487894416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb540c3ac0 0x7fcb541501c0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.936+0000 7fcb5a35c640 1 -- 172.21.15.150:0/2487894416 >> 172.21.15.150:0/2487894416 conn(0x7fcb540c4a00 msgr2=0x7fcb540c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.936+0000 7fcb5a35c640 1 -- 172.21.15.150:0/2487894416 shutdown_connections 2024-09-17T05:20:37.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.936+0000 7fcb5a35c640 1 -- 172.21.15.150:0/2487894416 wait complete. 2024-09-17T05:20:37.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb5a35c640 1 Processor -- start 2024-09-17T05:20:37.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb5a35c640 1 -- start start 2024-09-17T05:20:37.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb5a35c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb54071310 0x7fcb5414cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:37.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb5a35c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb540c3ac0 0x7fcb54150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:37.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb5a35c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb541528c0 0x7fcb54143250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:37.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb5a35c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcb54071040 con 0x7fcb541528c0 2024-09-17T05:20:37.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb5a35c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcb54070ec0 con 0x7fcb540c3ac0 2024-09-17T05:20:37.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb5a35c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fcb5415d9d0 con 0x7fcb54071310 2024-09-17T05:20:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb588f9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb541528c0 0x7fcb54143250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb588f9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb541528c0 0x7fcb54143250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38492/0 (socket says 172.21.15.150:38492) 2024-09-17T05:20:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb588f9640 1 -- 172.21.15.150:0/3278650850 learned_addr learned my addr 172.21.15.150:0/3278650850 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb53fff640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb54071310 0x7fcb5414cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb537fe640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb540c3ac0 0x7fcb54150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb588f9640 1 -- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb54071310 msgr2=0x7fcb5414cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb588f9640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb54071310 0x7fcb5414cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb588f9640 1 -- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb540c3ac0 msgr2=0x7fcb54150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb588f9640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb540c3ac0 0x7fcb54150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb588f9640 1 -- 172.21.15.150:0/3278650850 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb541464d0 con 0x7fcb541528c0 2024-09-17T05:20:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb537fe640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb540c3ac0 0x7fcb54150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:20:37.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb588f9640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb541528c0 0x7fcb54143250 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00cad0 tx=0x7fcb4c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb50ff9640 1 -- 172.21.15.150:0/3278650850 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c00ae20 con 0x7fcb541528c0 2024-09-17T05:20:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb50ff9640 1 -- 172.21.15.150:0/3278650850 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb4c003a70 con 0x7fcb541528c0 2024-09-17T05:20:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb5a35c640 1 -- 172.21.15.150:0/3278650850 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb54152140 con 0x7fcb541528c0 2024-09-17T05:20:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb50ff9640 1 -- 172.21.15.150:0/3278650850 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c004030 con 0x7fcb541528c0 2024-09-17T05:20:37.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb5a35c640 1 -- 172.21.15.150:0/3278650850 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb540c1fb0 con 0x7fcb541528c0 2024-09-17T05:20:37.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.940+0000 7fcb5a35c640 1 -- 172.21.15.150:0/3278650850 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb540748e0 con 0x7fcb541528c0 2024-09-17T05:20:37.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.944+0000 7fcb50ff9640 1 -- 172.21.15.150:0/3278650850 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fcb4c0041d0 con 0x7fcb541528c0 2024-09-17T05:20:37.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.944+0000 7fcb50ff9640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcb34073230 0x7fcb34075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:37.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.944+0000 7fcb50ff9640 1 -- 172.21.15.150:0/3278650850 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7fcb4c095b50 con 0x7fcb541528c0 2024-09-17T05:20:37.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.944+0000 7fcb53fff640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcb34073230 0x7fcb34075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:37.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.944+0000 7fcb53fff640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcb34073230 0x7fcb34075740 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400b810 tx=0x7fcb44008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:37.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:37.948+0000 7fcb50ff9640 1 -- 172.21.15.150:0/3278650850 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcb540748e0 con 0x7fcb541528c0 2024-09-17T05:20:38.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.148+0000 7fcb5a35c640 1 -- 172.21.15.150:0/3278650850 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb5414dfa0 con 0x7fcb541528c0 2024-09-17T05:20:38.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.148+0000 7fcb50ff9640 1 -- 172.21.15.150:0/3278650850 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) ==== 95+0+5719 (secure 0 0 0) 0x7fcb5414dfa0 con 0x7fcb541528c0 2024-09-17T05:20:38.147 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:38.147 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":211,"btime":"2024-09-17T05:20:37:499645+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:20:36.887029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7567":{"gid":7567,"name":"b","rank":0,"incarnation":210,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-17T05:20:38.147 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 211 2024-09-17T05:20:38.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.152+0000 7fcb5a35c640 1 -- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcb34073230 msgr2=0x7fcb34075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:38.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.152+0000 7fcb5a35c640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcb34073230 0x7fcb34075740 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400b810 tx=0x7fcb44008040 comp rx=0 tx=0).stop 2024-09-17T05:20:38.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.152+0000 7fcb5a35c640 1 -- 172.21.15.150:0/3278650850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb541528c0 msgr2=0x7fcb54143250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:38.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.152+0000 7fcb5a35c640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb541528c0 0x7fcb54143250 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00cad0 tx=0x7fcb4c00cf90 comp rx=0 tx=0).stop 2024-09-17T05:20:38.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.152+0000 7fcb5a35c640 1 -- 172.21.15.150:0/3278650850 shutdown_connections 2024-09-17T05:20:38.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.152+0000 7fcb5a35c640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fcb34073230 0x7fcb34075740 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:38.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.152+0000 7fcb5a35c640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcb540c3ac0 0x7fcb54150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:38.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.152+0000 7fcb5a35c640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcb541528c0 0x7fcb54143250 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:38.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.152+0000 7fcb5a35c640 1 --2- 172.21.15.150:0/3278650850 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fcb54071310 0x7fcb5414cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:38.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.152+0000 7fcb5a35c640 1 -- 172.21.15.150:0/3278650850 >> 172.21.15.150:0/3278650850 conn(0x7fcb540c4a00 msgr2=0x7fcb5415f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:38.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.152+0000 7fcb5a35c640 1 -- 172.21.15.150:0/3278650850 shutdown_connections 2024-09-17T05:20:38.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:38.152+0000 7fcb5a35c640 1 -- 172.21.15.150:0/3278650850 wait complete. 2024-09-17T05:20:38.169 INFO:tasks.mds_thrash.fs.[b]:mds map: { "btime": "2024-09-17T05:20:37:499645+0000", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 211, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:31.253633+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 208, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8199": { "addr": "172.21.15.150:6835/1799698362", "addrs": { "addrvec": [ { "addr": "172.21.15.150:6834", "nonce": 1799698362, "type": "v2" }, { "addr": "172.21.15.150:6835", "nonce": 1799698362, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 8199, "incarnation": 205, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-09-17T05:19:54.337480+0000", "qdb_cluster": [ 8199 ], "qdb_leader": 8199, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8199 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-09-17T04:49:40.378290+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 211, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7567": { "addr": "172.21.15.176:6835/1117422294", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6834", "nonce": 1117422294, "type": "v2" }, { "addr": "172.21.15.176:6835", "nonce": 1117422294, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 7567, "incarnation": 210, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-09-17T05:20:36.887029+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7567 } } } ], "standbys": [ { "addr": "172.21.15.176:6837/2843535555", "addrs": { "addrvec": [ { "addr": "172.21.15.176:6836", "nonce": 2843535555, "type": "v2" }, { "addr": "172.21.15.176:6837", "nonce": 2843535555, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 203, "export_targets": [], "features": 4540719139924082687, "flags": 0, "gid": 6749, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-09-17T05:20:40.170 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:20:40.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.468+0000 7f0a499aa640 1 -- 172.21.15.150:0/3520156929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a30004810 msgr2=0x7f0a30004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:40.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.468+0000 7f0a499aa640 1 --2- 172.21.15.150:0/3520156929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a30004810 0x7f0a30004c20 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f0a44072c10 tx=0x7f0a3401c7d0 comp rx=0 tx=0).stop 2024-09-17T05:20:40.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.468+0000 7f0a437fe640 1 -- 172.21.15.150:0/3520156929 reap_dead start 2024-09-17T05:20:40.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.468+0000 7f0a499aa640 1 -- 172.21.15.150:0/3520156929 shutdown_connections 2024-09-17T05:20:40.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.468+0000 7f0a499aa640 1 -- 172.21.15.150:0/3520156929 >> 172.21.15.150:0/3520156929 conn(0x7f0a440c4e30 msgr2=0x7f0a4406ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:40.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.468+0000 7f0a499aa640 1 -- 172.21.15.150:0/3520156929 shutdown_connections 2024-09-17T05:20:40.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.468+0000 7f0a499aa640 1 -- 172.21.15.150:0/3520156929 wait complete. 2024-09-17T05:20:40.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.468+0000 7f0a499aa640 1 Processor -- start 2024-09-17T05:20:40.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.472+0000 7f0a499aa640 1 -- start start 2024-09-17T05:20:40.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a499aa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a30004810 0x7f0a441ff9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:40.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a499aa640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a4415c520 0x7f0a440c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:40.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a499aa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a4415d190 0x7f0a4415d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:40.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a499aa640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0a44151000 con 0x7f0a30004810 2024-09-17T05:20:40.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a499aa640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0a44071e50 con 0x7f0a4415d190 2024-09-17T05:20:40.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a499aa640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0a44073e50 con 0x7f0a4415c520 2024-09-17T05:20:40.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a437fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a4415d190 0x7f0a4415d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:40.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a437fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a4415d190 0x7f0a4415d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.150:47376/0 (socket says 172.21.15.150:47376) 2024-09-17T05:20:40.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a437fe640 1 -- 172.21.15.150:0/1285129889 learned_addr learned my addr 172.21.15.150:0/1285129889 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:40.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a437fe640 1 -- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a4415c520 msgr2=0x7f0a440c62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:40.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a437fe640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a4415c520 0x7f0a440c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:40.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a437fe640 1 -- 172.21.15.150:0/1285129889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a30004810 msgr2=0x7f0a441ff9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T05:20:40.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a42ffd640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a30004810 0x7f0a441ff9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:40.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a437fe640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a30004810 0x7f0a441ff9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:40.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a437fe640 1 -- 172.21.15.150:0/1285129889 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4415ec90 con 0x7f0a4415d190 2024-09-17T05:20:40.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a42ffd640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a30004810 0x7f0a441ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:20:40.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a437fe640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a4415d190 0x7f0a4415d5a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c000f90 tx=0x7f0a3c0064f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:40.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a40ff9640 1 -- 172.21.15.150:0/1285129889 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c00f980 con 0x7f0a4415d190 2024-09-17T05:20:40.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a40ff9640 1 -- 172.21.15.150:0/1285129889 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a3c00fb60 con 0x7f0a4415d190 2024-09-17T05:20:40.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.476+0000 7f0a499aa640 1 -- 172.21.15.150:0/1285129889 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a4415a880 con 0x7f0a4415d190 2024-09-17T05:20:40.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.480+0000 7f0a40ff9640 1 -- 172.21.15.150:0/1285129889 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c011430 con 0x7f0a4415d190 2024-09-17T05:20:40.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.480+0000 7f0a499aa640 1 -- 172.21.15.150:0/1285129889 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a4415e9b0 con 0x7f0a4415d190 2024-09-17T05:20:40.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.480+0000 7f0a499aa640 1 -- 172.21.15.150:0/1285129889 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a4415a090 con 0x7f0a4415d190 2024-09-17T05:20:40.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.480+0000 7f0a40ff9640 1 -- 172.21.15.150:0/1285129889 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f0a3c0115d0 con 0x7f0a4415d190 2024-09-17T05:20:40.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.480+0000 7f0a40ff9640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a24073230 0x7f0a24075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:40.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.480+0000 7f0a40ff9640 1 -- 172.21.15.150:0/1285129889 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f0a3c095c90 con 0x7f0a4415d190 2024-09-17T05:20:40.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.480+0000 7f0a42ffd640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a24073230 0x7f0a24075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:40.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.484+0000 7f0a42ffd640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a24073230 0x7f0a24075740 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009ac0 tx=0x7f0a2c005750 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:40.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.492+0000 7f0a40ff9640 1 -- 172.21.15.150:0/1285129889 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0a4415a090 con 0x7f0a4415d190 2024-09-17T05:20:40.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.916+0000 7f0a499aa640 1 -- 172.21.15.150:0/1285129889 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a4406d560 con 0x7f0a4415d190 2024-09-17T05:20:40.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.920+0000 7f0a40ff9640 1 -- 172.21.15.150:0/1285129889 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) ==== 95+0+5723 (secure 0 0 0) 0x7f0a3c05fc00 con 0x7f0a4415d190 2024-09-17T05:20:40.918 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:40.918 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":213,"btime":"2024-09-17T05:20:39:523989+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:20:39.523977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7567":{"gid":7567,"name":"b","rank":0,"incarnation":210,"state":"up:active","state_seq":12,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7567,"qdb_cluster":[7567]},"id":2}]} 2024-09-17T05:20:40.918 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 213 2024-09-17T05:20:40.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.924+0000 7f0a499aa640 1 -- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a24073230 msgr2=0x7f0a24075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:40.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.924+0000 7f0a499aa640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a24073230 0x7f0a24075740 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c009ac0 tx=0x7f0a2c005750 comp rx=0 tx=0).stop 2024-09-17T05:20:40.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.924+0000 7f0a499aa640 1 -- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a4415d190 msgr2=0x7f0a4415d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:40.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.924+0000 7f0a499aa640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a4415d190 0x7f0a4415d5a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c000f90 tx=0x7f0a3c0064f0 comp rx=0 tx=0).stop 2024-09-17T05:20:40.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.924+0000 7f0a499aa640 1 -- 172.21.15.150:0/1285129889 shutdown_connections 2024-09-17T05:20:40.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.924+0000 7f0a499aa640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f0a24073230 0x7f0a24075740 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:40.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.924+0000 7f0a499aa640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0a4415d190 0x7f0a4415d5a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:40.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.924+0000 7f0a499aa640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f0a4415c520 0x7f0a440c62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:40.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.924+0000 7f0a499aa640 1 --2- 172.21.15.150:0/1285129889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a30004810 0x7f0a441ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:40.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.924+0000 7f0a499aa640 1 -- 172.21.15.150:0/1285129889 >> 172.21.15.150:0/1285129889 conn(0x7f0a440c4e30 msgr2=0x7f0a44151d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:40.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.924+0000 7f0a499aa640 1 -- 172.21.15.150:0/1285129889 shutdown_connections 2024-09-17T05:20:40.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:40.924+0000 7f0a499aa640 1 -- 172.21.15.150:0/1285129889 wait complete. 2024-09-17T05:20:40.961 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=8031 2024-09-17T05:20:40.962 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.a 2024-09-17T05:20:42.401 INFO:tasks.ceph.mds.b.smithi176.stderr:2024-09-17T05:20:42.397+0000 7f693a4ed640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:20:42.520 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T05:20:42.521 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-09-17T05:20:42.576 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.1... 2024-09-17T05:20:42.576 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-09-17T05:20:43.043 DEBUG:teuthology.parallel:result is None 2024-09-17T05:20:43.044 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T05:20:43.080 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T05:20:43.080 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-09-17T05:20:43.101 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-09-17T05:20:43.101 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-09-17T05:20:43.111 INFO:tasks.mds_thrash:joining mds_thrasher 2024-09-17T05:20:43.111 ERROR:tasks.mds_thrash.fs.[b]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa/tasks/mds_thrash.py", line 121, in _run self.do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa/tasks/mds_thrash.py", line 314, in do_thrash self.sleep_unless_stopped(delay) File "/home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa/tasks/thrasher.py", line 50, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-09-17T05:20:43.112 INFO:tasks.mds_thrash:done joining 2024-09-17T05:20:43.112 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-09-17T05:20:43.119 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-09-17T05:20:43.120 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T05:20:43.120 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T05:20:43.137 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T05:20:43.137 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T05:20:43.203 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T05:20:43.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.464+0000 7f22977b6640 1 -- 172.21.15.150:0/1460175778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22900c5390 msgr2=0x7f2290074960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:43.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.464+0000 7f22977b6640 1 --2- 172.21.15.150:0/1460175778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22900c5390 0x7f2290074960 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f22900c6cd0 tx=0x7f228c01c800 comp rx=0 tx=0).stop 2024-09-17T05:20:43.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.464+0000 7f22977b6640 1 -- 172.21.15.150:0/1460175778 shutdown_connections 2024-09-17T05:20:43.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.464+0000 7f22977b6640 1 --2- 172.21.15.150:0/1460175778 >> v2:172.21.15.176:3300/0 conn(0x7f22901576d0 0x7f22900c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.464+0000 7f22977b6640 1 --2- 172.21.15.150:0/1460175778 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22900c8230 0x7f2290157190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.464+0000 7f22977b6640 1 --2- 172.21.15.150:0/1460175778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22900c5390 0x7f2290074960 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.464+0000 7f22977b6640 1 -- 172.21.15.150:0/1460175778 >> 172.21.15.150:0/1460175778 conn(0x7f229006e360 msgr2=0x7f229006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:43.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.464+0000 7f22977b6640 1 -- 172.21.15.150:0/1460175778 shutdown_connections 2024-09-17T05:20:43.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.464+0000 7f22977b6640 1 -- 172.21.15.150:0/1460175778 wait complete. 2024-09-17T05:20:43.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f22977b6640 1 Processor -- start 2024-09-17T05:20:43.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f22977b6640 1 -- start start 2024-09-17T05:20:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f22977b6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22900c5390 0x7f22900a9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f22977b6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22900c8230 0x7f229015cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f22977b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22901576d0 0x7f2290151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f22977b6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f229006d1e0 con 0x7f22901576d0 2024-09-17T05:20:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f22977b6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f229006d060 con 0x7f22900c5390 2024-09-17T05:20:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f22977b6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f229006d360 con 0x7f22900c8230 2024-09-17T05:20:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2295d53640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22901576d0 0x7f2290151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2295d53640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22901576d0 0x7f2290151130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34488/0 (socket says 172.21.15.150:34488) 2024-09-17T05:20:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2295d53640 1 -- 172.21.15.150:0/4145196084 learned_addr learned my addr 172.21.15.150:0/4145196084 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2294d51640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22900c8230 0x7f229015cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2295552640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22900c5390 0x7f22900a9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2295d53640 1 -- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22900c8230 msgr2=0x7f229015cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2295d53640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22900c8230 0x7f229015cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2295d53640 1 -- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22900c5390 msgr2=0x7f22900a9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:43.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2295d53640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22900c5390 0x7f22900a9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2295d53640 1 -- 172.21.15.150:0/4145196084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2290158d50 con 0x7f22901576d0 2024-09-17T05:20:43.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2295d53640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22901576d0 0x7f2290151130 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f228800e9a0 tx=0x7f228800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:43.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2285ffb640 1 -- 172.21.15.150:0/4145196084 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2288004d30 con 0x7f22901576d0 2024-09-17T05:20:43.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2285ffb640 1 -- 172.21.15.150:0/4145196084 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f228800cb20 con 0x7f22901576d0 2024-09-17T05:20:43.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f2285ffb640 1 -- 172.21.15.150:0/4145196084 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2288005020 con 0x7f22901576d0 2024-09-17T05:20:43.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f22977b6640 1 -- 172.21.15.150:0/4145196084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22901f3fa0 con 0x7f22901576d0 2024-09-17T05:20:43.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.468+0000 7f22977b6640 1 -- 172.21.15.150:0/4145196084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f229015b130 con 0x7f22901576d0 2024-09-17T05:20:43.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.472+0000 7f22977b6640 1 -- 172.21.15.150:0/4145196084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2290157190 con 0x7f22901576d0 2024-09-17T05:20:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.472+0000 7f2285ffb640 1 -- 172.21.15.150:0/4145196084 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f22880051c0 con 0x7f22901576d0 2024-09-17T05:20:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.472+0000 7f2285ffb640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2274073230 0x7f2274075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.472+0000 7f2285ffb640 1 -- 172.21.15.150:0/4145196084 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f2288095bd0 con 0x7f22901576d0 2024-09-17T05:20:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.472+0000 7f2295552640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2274073230 0x7f2274075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.472+0000 7f2295552640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2274073230 0x7f2274075740 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f228c0af000 tx=0x7f228c01b040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:43.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.476+0000 7f2285ffb640 1 -- 172.21.15.150:0/4145196084 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2288016030 con 0x7f22901576d0 2024-09-17T05:20:43.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 -- 172.21.15.150:0/4145196084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2290158f20 con 0x7f22901576d0 2024-09-17T05:20:43.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f2285ffb640 1 -- 172.21.15.150:0/4145196084 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 82 entries v68) ==== 81+0+5048 (secure 0 0 0) 0x7f228805fb40 con 0x7f22901576d0 2024-09-17T05:20:43.646 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2857695220 2024-09-18T05:20:36.328050+0000 2024-09-17T05:20:43.646 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/2857695220 2024-09-18T05:20:36.328050+0000 2024-09-17T05:20:43.646 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6835/1690935336 2024-09-18T05:19:51.316025+0000 2024-09-17T05:20:43.646 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6834/1690935336 2024-09-18T05:19:51.316025+0000 2024-09-17T05:20:43.646 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3297154462 2024-09-18T05:18:31.294755+0000 2024-09-17T05:20:43.646 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/1903363960 2024-09-18T05:18:31.294598+0000 2024-09-17T05:20:43.646 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/1903363960 2024-09-18T05:18:31.294598+0000 2024-09-17T05:20:43.646 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3297154462 2024-09-18T05:18:31.294755+0000 2024-09-17T05:20:43.646 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6835/524846644 2024-09-18T05:16:53.685445+0000 2024-09-17T05:20:43.646 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/949083177 2024-09-18T05:16:51.270750+0000 2024-09-17T05:20:43.646 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/178583496 2024-09-18T05:15:51.246236+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/2338457617 2024-09-18T05:15:08.260160+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/3933379792 2024-09-18T05:15:06.233954+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3179072684 2024-09-18T05:14:26.220879+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/12982302 2024-09-18T05:13:46.203493+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/2962113435 2024-09-18T05:12:21.179693+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/178583496 2024-09-18T05:15:51.246236+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/713081612 2024-09-18T04:59:20.944548+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/2081398022 2024-09-18T05:00:20.961331+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3079958649 2024-09-18T04:58:00.922711+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3079958649 2024-09-18T04:58:00.922711+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/12982302 2024-09-18T05:13:46.203493+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/233311351 2024-09-18T05:04:06.023230+0000 2024-09-17T05:20:43.647 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/949083177 2024-09-18T05:16:51.270750+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/284564893 2024-09-18T04:57:40.911134+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/903190072 2024-09-18T05:04:36.031676+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/2374314256 2024-09-18T04:56:10.885556+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3179072684 2024-09-18T05:14:26.220879+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/3543629780 2024-09-18T04:58:30.930118+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/4286123257 2024-09-18T04:55:40.878553+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/480168496 2024-09-18T04:51:40.821573+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/1045115232 2024-09-18T04:53:50.852056+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/1045115232 2024-09-18T04:53:50.852056+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/285376980 2024-09-18T05:10:51.151028+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/2313391683 2024-09-18T05:00:45.968345+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/2081398022 2024-09-18T05:00:20.961331+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/480168496 2024-09-18T04:51:40.821573+0000 2024-09-17T05:20:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/151500360 2024-09-18T04:54:55.866806+0000 2024-09-17T05:20:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2313391683 2024-09-18T05:00:45.968345+0000 2024-09-17T05:20:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/1275062894 2024-09-18T04:58:50.936080+0000 2024-09-17T05:20:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/1275062894 2024-09-18T04:58:50.936080+0000 2024-09-17T05:20:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/961213876 2024-09-18T04:52:55.838337+0000 2024-09-17T05:20:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/151500360 2024-09-18T04:54:55.866806+0000 2024-09-17T05:20:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/2374314256 2024-09-18T04:56:10.885556+0000 2024-09-17T05:20:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/1237450980 2024-09-18T05:06:31.077239+0000 2024-09-17T05:20:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/2338457617 2024-09-18T05:15:08.260160+0000 2024-09-17T05:20:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/961213876 2024-09-18T04:52:55.838337+0000 2024-09-17T05:20:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/3532647866 2024-09-18T04:55:10.871121+0000 2024-09-17T05:20:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/713081612 2024-09-18T04:59:20.944548+0000 2024-09-17T05:20:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/792092400 2024-09-18T05:09:46.133255+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2613040311 2024-09-18T04:51:25.818032+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/3532647866 2024-09-18T04:55:10.871121+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/2062728083 2024-09-18T05:03:01.005221+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/523364553 2024-09-18T05:07:21.092015+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/233311351 2024-09-18T05:04:06.023230+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/2962113435 2024-09-18T05:12:21.179693+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/4286123257 2024-09-18T04:55:40.878553+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/792092400 2024-09-18T05:09:46.133255+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/702608997 2024-09-18T05:02:30.998209+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/284564893 2024-09-18T04:57:40.911134+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/3622725045 2024-09-18T05:09:06.121144+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/2062728083 2024-09-18T05:03:01.005221+0000 2024-09-17T05:20:43.650 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/903190072 2024-09-18T05:04:36.031676+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/1984876738 2024-09-18T05:05:21.045156+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6834/524846644 2024-09-18T05:16:53.685445+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/1984876738 2024-09-18T05:05:21.045156+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/3933379792 2024-09-18T05:15:06.233954+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3807957687 2024-09-18T05:05:56.055429+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3807957687 2024-09-18T05:05:56.055429+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/3940723333 2024-09-18T05:06:51.083809+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/1237450980 2024-09-18T05:06:31.077239+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/3543629780 2024-09-18T04:58:30.930118+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/702608997 2024-09-18T05:02:30.998209+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/92695211 2024-09-18T05:08:16.109918+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/2613040311 2024-09-18T04:51:25.818032+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/523364553 2024-09-18T05:07:21.092015+0000 2024-09-17T05:20:43.651 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/92695211 2024-09-18T05:08:16.109918+0000 2024-09-17T05:20:43.652 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/3622725045 2024-09-18T05:09:06.121144+0000 2024-09-17T05:20:43.652 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/3940723333 2024-09-18T05:06:51.083809+0000 2024-09-17T05:20:43.652 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/285376980 2024-09-18T05:10:51.151028+0000 2024-09-17T05:20:43.652 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2056467678 2024-09-18T05:12:01.173633+0000 2024-09-17T05:20:43.652 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/2056467678 2024-09-18T05:12:01.173633+0000 2024-09-17T05:20:43.653 INFO:teuthology.orchestra.run.smithi150.stderr:listed 82 entries 2024-09-17T05:20:43.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 -- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2274073230 msgr2=0x7f2274075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:43.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2274073230 0x7f2274075740 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f228c0af000 tx=0x7f228c01b040 comp rx=0 tx=0).stop 2024-09-17T05:20:43.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 -- 172.21.15.150:0/4145196084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22901576d0 msgr2=0x7f2290151130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:43.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22901576d0 0x7f2290151130 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f228800e9a0 tx=0x7f228800ee60 comp rx=0 tx=0).stop 2024-09-17T05:20:43.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 -- 172.21.15.150:0/4145196084 shutdown_connections 2024-09-17T05:20:43.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2274073230 0x7f2274075740 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22901576d0 0x7f2290151130 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f22900c8230 0x7f229015cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 --2- 172.21.15.150:0/4145196084 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f22900c5390 0x7f22900a9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 -- 172.21.15.150:0/4145196084 >> 172.21.15.150:0/4145196084 conn(0x7f229006e360 msgr2=0x7f229006e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:43.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 -- 172.21.15.150:0/4145196084 shutdown_connections 2024-09-17T05:20:43.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.648+0000 7f22977b6640 1 -- 172.21.15.150:0/4145196084 wait complete. 2024-09-17T05:20:43.667 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-09-17T05:20:43.667 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T05:20:43.673 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T05:20:43.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.872+0000 7f7e00847640 1 -- 172.21.15.150:0/4138982897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dfc152cf0 msgr2=0x7f7dfc153100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:43.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.872+0000 7f7e00847640 1 --2- 172.21.15.150:0/4138982897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dfc152cf0 0x7f7dfc153100 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f7df400ad90 tx=0x7f7df401dcb0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.872+0000 7f7e00847640 1 -- 172.21.15.150:0/4138982897 shutdown_connections 2024-09-17T05:20:43.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.872+0000 7f7e00847640 1 --2- 172.21.15.150:0/4138982897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dfc152cf0 0x7f7dfc153100 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.872+0000 7f7e00847640 1 -- 172.21.15.150:0/4138982897 >> 172.21.15.150:0/4138982897 conn(0x7f7dfc0c4a00 msgr2=0x7f7dfc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:43.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.872+0000 7f7e00847640 1 -- 172.21.15.150:0/4138982897 shutdown_connections 2024-09-17T05:20:43.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.872+0000 7f7e00847640 1 -- 172.21.15.150:0/4138982897 wait complete. 2024-09-17T05:20:43.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.872+0000 7f7e00847640 1 Processor -- start 2024-09-17T05:20:43.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.872+0000 7f7e00847640 1 -- start start 2024-09-17T05:20:43.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7e00847640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dfc0a7350 0x7f7dfc1562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:43.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7e00847640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dfc0c42e0 0x7f7dfc1f14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:43.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7e00847640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc152cf0 0x7f7dfc1466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:43.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7e00847640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc150770 con 0x7f7dfc0a7350 2024-09-17T05:20:43.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7e00847640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7dfc1508f0 con 0x7f7dfc152cf0 2024-09-17T05:20:43.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7e00847640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7dfc160690 con 0x7f7dfc0c42e0 2024-09-17T05:20:43.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7dfa59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dfc0a7350 0x7f7dfc1562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:43.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7dfa59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dfc0a7350 0x7f7dfc1562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34510/0 (socket says 172.21.15.150:34510) 2024-09-17T05:20:43.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7dfa59c640 1 -- 172.21.15.150:0/284767568 learned_addr learned my addr 172.21.15.150:0/284767568 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:43.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7dfad9d640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc152cf0 0x7f7dfc1466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:43.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7df1d9b640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dfc0c42e0 0x7f7dfc1f14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:43.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7dfa59c640 1 -- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dfc0c42e0 msgr2=0x7f7dfc1f14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:43.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7dfa59c640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dfc0c42e0 0x7f7dfc1f14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7dfa59c640 1 -- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc152cf0 msgr2=0x7f7dfc1466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:43.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7dfa59c640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc152cf0 0x7f7dfc1466e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:43.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7dfa59c640 1 -- 172.21.15.150:0/284767568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dfc144b00 con 0x7f7dfc0a7350 2024-09-17T05:20:43.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7dfa59c640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dfc0a7350 0x7f7dfc1562f0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00c3c0 tx=0x7f7dec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:43.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7df37fe640 1 -- 172.21.15.150:0/284767568 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec017070 con 0x7f7dfc0a7350 2024-09-17T05:20:43.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7df37fe640 1 -- 172.21.15.150:0/284767568 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7dec0053c0 con 0x7f7dfc0a7350 2024-09-17T05:20:43.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7e00847640 1 -- 172.21.15.150:0/284767568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dfc145e20 con 0x7f7dfc0a7350 2024-09-17T05:20:43.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7df37fe640 1 -- 172.21.15.150:0/284767568 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec003860 con 0x7f7dfc0a7350 2024-09-17T05:20:43.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7e00847640 1 -- 172.21.15.150:0/284767568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dfc0ad990 con 0x7f7dfc0a7350 2024-09-17T05:20:43.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7e00847640 1 -- 172.21.15.150:0/284767568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dfc153100 con 0x7f7dfc0a7350 2024-09-17T05:20:43.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.876+0000 7f7df37fe640 1 -- 172.21.15.150:0/284767568 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7dec01a070 con 0x7f7dfc0a7350 2024-09-17T05:20:43.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.880+0000 7f7df37fe640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7ddc073200 0x7f7ddc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:43.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.880+0000 7f7df37fe640 1 -- 172.21.15.150:0/284767568 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f7dec094550 con 0x7f7dfc0a7350 2024-09-17T05:20:43.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.880+0000 7f7df1d9b640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7ddc073200 0x7f7ddc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:43.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.880+0000 7f7df1d9b640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7ddc073200 0x7f7ddc075710 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc076ec0 tx=0x7f7de4009060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:43.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:43.884+0000 7f7df37fe640 1 -- 172.21.15.150:0/284767568 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7dec05de70 con 0x7f7dfc0a7350 2024-09-17T05:20:44.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.056+0000 7f7e00847640 1 -- 172.21.15.150:0/284767568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7dfc0a7130 con 0x7f7dfc0a7350 2024-09-17T05:20:44.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.056+0000 7f7df37fe640 1 -- 172.21.15.150:0/284767568 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 82 entries v68) ==== 81+0+5048 (secure 0 0 0) 0x7f7dec0628a0 con 0x7f7dfc0a7350 2024-09-17T05:20:44.055 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2857695220 2024-09-18T05:20:36.328050+0000 2024-09-17T05:20:44.055 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/2857695220 2024-09-18T05:20:36.328050+0000 2024-09-17T05:20:44.055 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6835/1690935336 2024-09-18T05:19:51.316025+0000 2024-09-17T05:20:44.055 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6834/1690935336 2024-09-18T05:19:51.316025+0000 2024-09-17T05:20:44.055 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3297154462 2024-09-18T05:18:31.294755+0000 2024-09-17T05:20:44.055 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/1903363960 2024-09-18T05:18:31.294598+0000 2024-09-17T05:20:44.055 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/1903363960 2024-09-18T05:18:31.294598+0000 2024-09-17T05:20:44.055 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3297154462 2024-09-18T05:18:31.294755+0000 2024-09-17T05:20:44.055 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6835/524846644 2024-09-18T05:16:53.685445+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/949083177 2024-09-18T05:16:51.270750+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/178583496 2024-09-18T05:15:51.246236+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/2338457617 2024-09-18T05:15:08.260160+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/3933379792 2024-09-18T05:15:06.233954+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3179072684 2024-09-18T05:14:26.220879+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/12982302 2024-09-18T05:13:46.203493+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/2962113435 2024-09-18T05:12:21.179693+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/178583496 2024-09-18T05:15:51.246236+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/713081612 2024-09-18T04:59:20.944548+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/2081398022 2024-09-18T05:00:20.961331+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3079958649 2024-09-18T04:58:00.922711+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3079958649 2024-09-18T04:58:00.922711+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/12982302 2024-09-18T05:13:46.203493+0000 2024-09-17T05:20:44.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/233311351 2024-09-18T05:04:06.023230+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/949083177 2024-09-18T05:16:51.270750+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/284564893 2024-09-18T04:57:40.911134+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/903190072 2024-09-18T05:04:36.031676+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/2374314256 2024-09-18T04:56:10.885556+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3179072684 2024-09-18T05:14:26.220879+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/3543629780 2024-09-18T04:58:30.930118+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/4286123257 2024-09-18T04:55:40.878553+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/480168496 2024-09-18T04:51:40.821573+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/1045115232 2024-09-18T04:53:50.852056+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/1045115232 2024-09-18T04:53:50.852056+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/285376980 2024-09-18T05:10:51.151028+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/2313391683 2024-09-18T05:00:45.968345+0000 2024-09-17T05:20:44.057 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/2081398022 2024-09-18T05:00:20.961331+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/480168496 2024-09-18T04:51:40.821573+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/151500360 2024-09-18T04:54:55.866806+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2313391683 2024-09-18T05:00:45.968345+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/1275062894 2024-09-18T04:58:50.936080+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/1275062894 2024-09-18T04:58:50.936080+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/961213876 2024-09-18T04:52:55.838337+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/151500360 2024-09-18T04:54:55.866806+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/2374314256 2024-09-18T04:56:10.885556+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/1237450980 2024-09-18T05:06:31.077239+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/2338457617 2024-09-18T05:15:08.260160+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/961213876 2024-09-18T04:52:55.838337+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/3532647866 2024-09-18T04:55:10.871121+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/713081612 2024-09-18T04:59:20.944548+0000 2024-09-17T05:20:44.058 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/792092400 2024-09-18T05:09:46.133255+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2613040311 2024-09-18T04:51:25.818032+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/3532647866 2024-09-18T04:55:10.871121+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/2062728083 2024-09-18T05:03:01.005221+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/523364553 2024-09-18T05:07:21.092015+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/233311351 2024-09-18T05:04:06.023230+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/2962113435 2024-09-18T05:12:21.179693+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/4286123257 2024-09-18T04:55:40.878553+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/792092400 2024-09-18T05:09:46.133255+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/702608997 2024-09-18T05:02:30.998209+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/284564893 2024-09-18T04:57:40.911134+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/3622725045 2024-09-18T05:09:06.121144+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/2062728083 2024-09-18T05:03:01.005221+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/903190072 2024-09-18T05:04:36.031676+0000 2024-09-17T05:20:44.059 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/1984876738 2024-09-18T05:05:21.045156+0000 2024-09-17T05:20:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6834/524846644 2024-09-18T05:16:53.685445+0000 2024-09-17T05:20:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/1984876738 2024-09-18T05:05:21.045156+0000 2024-09-17T05:20:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/3933379792 2024-09-18T05:15:06.233954+0000 2024-09-17T05:20:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3807957687 2024-09-18T05:05:56.055429+0000 2024-09-17T05:20:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3807957687 2024-09-18T05:05:56.055429+0000 2024-09-17T05:20:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/3940723333 2024-09-18T05:06:51.083809+0000 2024-09-17T05:20:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/1237450980 2024-09-18T05:06:31.077239+0000 2024-09-17T05:20:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/3543629780 2024-09-18T04:58:30.930118+0000 2024-09-17T05:20:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/702608997 2024-09-18T05:02:30.998209+0000 2024-09-17T05:20:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/92695211 2024-09-18T05:08:16.109918+0000 2024-09-17T05:20:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/2613040311 2024-09-18T04:51:25.818032+0000 2024-09-17T05:20:44.060 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/523364553 2024-09-18T05:07:21.092015+0000 2024-09-17T05:20:44.061 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/92695211 2024-09-18T05:08:16.109918+0000 2024-09-17T05:20:44.061 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/3622725045 2024-09-18T05:09:06.121144+0000 2024-09-17T05:20:44.061 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/3940723333 2024-09-18T05:06:51.083809+0000 2024-09-17T05:20:44.061 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/285376980 2024-09-18T05:10:51.151028+0000 2024-09-17T05:20:44.061 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2056467678 2024-09-18T05:12:01.173633+0000 2024-09-17T05:20:44.061 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/2056467678 2024-09-18T05:12:01.173633+0000 2024-09-17T05:20:44.062 INFO:teuthology.orchestra.run.smithi150.stderr:listed 82 entries 2024-09-17T05:20:44.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.060+0000 7f7e00847640 1 -- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7ddc073200 msgr2=0x7f7ddc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:44.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.060+0000 7f7e00847640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7ddc073200 0x7f7ddc075710 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc076ec0 tx=0x7f7de4009060 comp rx=0 tx=0).stop 2024-09-17T05:20:44.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.060+0000 7f7e00847640 1 -- 172.21.15.150:0/284767568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dfc0a7350 msgr2=0x7f7dfc1562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:44.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.060+0000 7f7e00847640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dfc0a7350 0x7f7dfc1562f0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00c3c0 tx=0x7f7dec00c880 comp rx=0 tx=0).stop 2024-09-17T05:20:44.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.060+0000 7f7e00847640 1 -- 172.21.15.150:0/284767568 shutdown_connections 2024-09-17T05:20:44.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.060+0000 7f7e00847640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7ddc073200 0x7f7ddc075710 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:44.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.060+0000 7f7e00847640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7dfc152cf0 0x7f7dfc1466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:44.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.060+0000 7f7e00847640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7dfc0c42e0 0x7f7dfc1f14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:44.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.060+0000 7f7e00847640 1 --2- 172.21.15.150:0/284767568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7dfc0a7350 0x7f7dfc1562f0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:44.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.060+0000 7f7e00847640 1 -- 172.21.15.150:0/284767568 >> 172.21.15.150:0/284767568 conn(0x7f7dfc0c4a00 msgr2=0x7f7dfc1523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:44.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.060+0000 7f7e00847640 1 -- 172.21.15.150:0/284767568 shutdown_connections 2024-09-17T05:20:44.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:44.060+0000 7f7e00847640 1 -- 172.21.15.150:0/284767568 wait complete. 2024-09-17T05:20:44.077 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi150.front.sepia.ceph.com... 2024-09-17T05:20:44.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:20:44.077 DEBUG:teuthology.orchestra.run.smithi150:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-17T05:20:44.109 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:20:44.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[37323]: fuse finished with error 0 and tester_r 0 2024-09-17T05:20:50.211 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-09-17T05:20:50.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:20:50.212 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T05:20:50.232 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T05:20:50.232 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T05:20:50.246 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T05:20:50.246 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T05:20:50.292 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T05:20:50.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.464+0000 7f304295a640 1 -- 172.21.15.150:0/3721225262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30240053b0 msgr2=0x7f302400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:50.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.464+0000 7f304295a640 1 --2- 172.21.15.150:0/3721225262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30240053b0 0x7f302400ba60 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f30340016f0 tx=0x7f303401c7b0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.464+0000 7f304295a640 1 -- 172.21.15.150:0/3721225262 shutdown_connections 2024-09-17T05:20:50.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.464+0000 7f304295a640 1 --2- 172.21.15.150:0/3721225262 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30240053b0 0x7f302400ba60 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.464+0000 7f304295a640 1 -- 172.21.15.150:0/3721225262 >> 172.21.15.150:0/3721225262 conn(0x7f303c0c9de0 msgr2=0x7f303c0ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:50.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.464+0000 7f304295a640 1 -- 172.21.15.150:0/3721225262 shutdown_connections 2024-09-17T05:20:50.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.464+0000 7f304295a640 1 -- 172.21.15.150:0/3721225262 wait complete. 2024-09-17T05:20:50.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f304295a640 1 Processor -- start 2024-09-17T05:20:50.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f304295a640 1 -- start start 2024-09-17T05:20:50.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f304295a640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30240053b0 0x7f303c0c32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:50.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f304295a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c1f32b0 0x7f303c146390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:50.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f304295a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f303c1f36c0 0x7f303c1468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:50.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f304295a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f303c0ad210 con 0x7f303c1f36c0 2024-09-17T05:20:50.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f304295a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f303c151e10 con 0x7f303c1f32b0 2024-09-17T05:20:50.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f304295a640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f303c151f90 con 0x7f30240053b0 2024-09-17T05:20:50.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f3040ef7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f303c1f36c0 0x7f303c1468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f3040ef7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f303c1f36c0 0x7f303c1468d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33144/0 (socket says 172.21.15.150:33144) 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f3040ef7640 1 -- 172.21.15.150:0/1117067322 learned_addr learned my addr 172.21.15.150:0/1117067322 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f303bfff640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30240053b0 0x7f303c0c32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f303b7fe640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c1f32b0 0x7f303c146390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f3040ef7640 1 -- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30240053b0 msgr2=0x7f303c0c32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f3040ef7640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30240053b0 0x7f303c0c32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f3040ef7640 1 -- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c1f32b0 msgr2=0x7f303c146390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f3040ef7640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c1f32b0 0x7f303c146390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f3040ef7640 1 -- 172.21.15.150:0/1117067322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303c155f90 con 0x7f303c1f36c0 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f303bfff640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30240053b0 0x7f303c0c32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f3040ef7640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f303c1f36c0 0x7f303c1468d0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f303401cc90 tx=0x7f3034004c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f3038ff9640 1 -- 172.21.15.150:0/1117067322 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340b6070 con 0x7f303c1f36c0 2024-09-17T05:20:50.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f3038ff9640 1 -- 172.21.15.150:0/1117067322 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30340064e0 con 0x7f303c1f36c0 2024-09-17T05:20:50.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f304295a640 1 -- 172.21.15.150:0/1117067322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f303c147560 con 0x7f303c1f36c0 2024-09-17T05:20:50.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f3038ff9640 1 -- 172.21.15.150:0/1117067322 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340ac040 con 0x7f303c1f36c0 2024-09-17T05:20:50.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f304295a640 1 -- 172.21.15.150:0/1117067322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f303c1f2ab0 con 0x7f303c1f36c0 2024-09-17T05:20:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.468+0000 7f304295a640 1 -- 172.21.15.150:0/1117067322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f303c1580b0 con 0x7f303c1f36c0 2024-09-17T05:20:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.472+0000 7f3038ff9640 1 -- 172.21.15.150:0/1117067322 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f30340b9070 con 0x7f303c1f36c0 2024-09-17T05:20:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.472+0000 7f3038ff9640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3028073230 0x7f3028075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.472+0000 7f3038ff9640 1 -- 172.21.15.150:0/1117067322 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f303412e3d0 con 0x7f303c1f36c0 2024-09-17T05:20:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.472+0000 7f303bfff640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3028073230 0x7f3028075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:50.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.472+0000 7f303bfff640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3028073230 0x7f3028075740 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f302c001390 tx=0x7f302c0014e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:50.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.476+0000 7f3038ff9640 1 -- 172.21.15.150:0/1117067322 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3034016510 con 0x7f303c1f36c0 2024-09-17T05:20:50.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.660+0000 7f304295a640 1 -- 172.21.15.150:0/1117067322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f303c155900 con 0x7f303c1f36c0 2024-09-17T05:20:50.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.660+0000 7f3038ff9640 1 -- 172.21.15.150:0/1117067322 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 82 entries v68) ==== 81+0+5048 (secure 0 0 0) 0x7f30340f7d70 con 0x7f303c1f36c0 2024-09-17T05:20:50.660 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2857695220 2024-09-18T05:20:36.328050+0000 2024-09-17T05:20:50.660 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/2857695220 2024-09-18T05:20:36.328050+0000 2024-09-17T05:20:50.660 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6835/1690935336 2024-09-18T05:19:51.316025+0000 2024-09-17T05:20:50.660 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6834/1690935336 2024-09-18T05:19:51.316025+0000 2024-09-17T05:20:50.660 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3297154462 2024-09-18T05:18:31.294755+0000 2024-09-17T05:20:50.660 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/1903363960 2024-09-18T05:18:31.294598+0000 2024-09-17T05:20:50.660 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/1903363960 2024-09-18T05:18:31.294598+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3297154462 2024-09-18T05:18:31.294755+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6835/524846644 2024-09-18T05:16:53.685445+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/949083177 2024-09-18T05:16:51.270750+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/178583496 2024-09-18T05:15:51.246236+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/2338457617 2024-09-18T05:15:08.260160+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/3933379792 2024-09-18T05:15:06.233954+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3179072684 2024-09-18T05:14:26.220879+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/12982302 2024-09-18T05:13:46.203493+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/2962113435 2024-09-18T05:12:21.179693+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/178583496 2024-09-18T05:15:51.246236+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/713081612 2024-09-18T04:59:20.944548+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/2081398022 2024-09-18T05:00:20.961331+0000 2024-09-17T05:20:50.661 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3079958649 2024-09-18T04:58:00.922711+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3079958649 2024-09-18T04:58:00.922711+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/12982302 2024-09-18T05:13:46.203493+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/233311351 2024-09-18T05:04:06.023230+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/949083177 2024-09-18T05:16:51.270750+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/284564893 2024-09-18T04:57:40.911134+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/903190072 2024-09-18T05:04:36.031676+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/2374314256 2024-09-18T04:56:10.885556+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3179072684 2024-09-18T05:14:26.220879+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/3543629780 2024-09-18T04:58:30.930118+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/4286123257 2024-09-18T04:55:40.878553+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/480168496 2024-09-18T04:51:40.821573+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/1045115232 2024-09-18T04:53:50.852056+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/1045115232 2024-09-18T04:53:50.852056+0000 2024-09-17T05:20:50.662 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/285376980 2024-09-18T05:10:51.151028+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/2313391683 2024-09-18T05:00:45.968345+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/2081398022 2024-09-18T05:00:20.961331+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/480168496 2024-09-18T04:51:40.821573+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/151500360 2024-09-18T04:54:55.866806+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2313391683 2024-09-18T05:00:45.968345+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/1275062894 2024-09-18T04:58:50.936080+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/1275062894 2024-09-18T04:58:50.936080+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/961213876 2024-09-18T04:52:55.838337+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/151500360 2024-09-18T04:54:55.866806+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/2374314256 2024-09-18T04:56:10.885556+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/1237450980 2024-09-18T05:06:31.077239+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/2338457617 2024-09-18T05:15:08.260160+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/961213876 2024-09-18T04:52:55.838337+0000 2024-09-17T05:20:50.663 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/3532647866 2024-09-18T04:55:10.871121+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/713081612 2024-09-18T04:59:20.944548+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/792092400 2024-09-18T05:09:46.133255+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2613040311 2024-09-18T04:51:25.818032+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/3532647866 2024-09-18T04:55:10.871121+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/2062728083 2024-09-18T05:03:01.005221+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/523364553 2024-09-18T05:07:21.092015+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/233311351 2024-09-18T05:04:06.023230+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/2962113435 2024-09-18T05:12:21.179693+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/4286123257 2024-09-18T04:55:40.878553+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/792092400 2024-09-18T05:09:46.133255+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/702608997 2024-09-18T05:02:30.998209+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/284564893 2024-09-18T04:57:40.911134+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/3622725045 2024-09-18T05:09:06.121144+0000 2024-09-17T05:20:50.664 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/2062728083 2024-09-18T05:03:01.005221+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/903190072 2024-09-18T05:04:36.031676+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/1984876738 2024-09-18T05:05:21.045156+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6834/524846644 2024-09-18T05:16:53.685445+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/1984876738 2024-09-18T05:05:21.045156+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/3933379792 2024-09-18T05:15:06.233954+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3807957687 2024-09-18T05:05:56.055429+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3807957687 2024-09-18T05:05:56.055429+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/3940723333 2024-09-18T05:06:51.083809+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/1237450980 2024-09-18T05:06:31.077239+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/3543629780 2024-09-18T04:58:30.930118+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/702608997 2024-09-18T05:02:30.998209+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/92695211 2024-09-18T05:08:16.109918+0000 2024-09-17T05:20:50.665 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/2613040311 2024-09-18T04:51:25.818032+0000 2024-09-17T05:20:50.666 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/523364553 2024-09-18T05:07:21.092015+0000 2024-09-17T05:20:50.666 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/92695211 2024-09-18T05:08:16.109918+0000 2024-09-17T05:20:50.666 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/3622725045 2024-09-18T05:09:06.121144+0000 2024-09-17T05:20:50.666 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/3940723333 2024-09-18T05:06:51.083809+0000 2024-09-17T05:20:50.666 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/285376980 2024-09-18T05:10:51.151028+0000 2024-09-17T05:20:50.666 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2056467678 2024-09-18T05:12:01.173633+0000 2024-09-17T05:20:50.666 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/2056467678 2024-09-18T05:12:01.173633+0000 2024-09-17T05:20:50.667 INFO:teuthology.orchestra.run.smithi150.stderr:listed 82 entries 2024-09-17T05:20:50.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.664+0000 7f304295a640 1 -- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3028073230 msgr2=0x7f3028075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:50.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.664+0000 7f304295a640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3028073230 0x7f3028075740 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f302c001390 tx=0x7f302c0014e0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.664+0000 7f304295a640 1 -- 172.21.15.150:0/1117067322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f303c1f36c0 msgr2=0x7f303c1468d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:50.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.664+0000 7f304295a640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f303c1f36c0 0x7f303c1468d0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f303401cc90 tx=0x7f3034004c50 comp rx=0 tx=0).stop 2024-09-17T05:20:50.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.664+0000 7f304295a640 1 -- 172.21.15.150:0/1117067322 shutdown_connections 2024-09-17T05:20:50.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.664+0000 7f304295a640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3028073230 0x7f3028075740 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.664+0000 7f304295a640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c1f32b0 0x7f303c146390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.664+0000 7f304295a640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f303c1f36c0 0x7f303c1468d0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.664+0000 7f304295a640 1 --2- 172.21.15.150:0/1117067322 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f30240053b0 0x7f303c0c32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.664+0000 7f304295a640 1 -- 172.21.15.150:0/1117067322 >> 172.21.15.150:0/1117067322 conn(0x7f303c0c9de0 msgr2=0x7f303c0c4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:50.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.664+0000 7f304295a640 1 -- 172.21.15.150:0/1117067322 shutdown_connections 2024-09-17T05:20:50.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.664+0000 7f304295a640 1 -- 172.21.15.150:0/1117067322 wait complete. 2024-09-17T05:20:50.683 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-17T05:20:50.683 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T05:20:50.690 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T05:20:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.852+0000 7f39be1c9640 1 -- 172.21.15.150:0/3848113804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39b80c42e0 msgr2=0x7f39b80c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.852+0000 7f39be1c9640 1 --2- 172.21.15.150:0/3848113804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39b80c42e0 0x7f39b80c6690 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f39a800acd0 tx=0x7f39a801de60 comp rx=0 tx=0).stop 2024-09-17T05:20:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.852+0000 7f39be1c9640 1 -- 172.21.15.150:0/3848113804 shutdown_connections 2024-09-17T05:20:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.852+0000 7f39be1c9640 1 --2- 172.21.15.150:0/3848113804 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39b8152cf0 0x7f39b8153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.852+0000 7f39be1c9640 1 --2- 172.21.15.150:0/3848113804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39b80c42e0 0x7f39b80c6690 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.852+0000 7f39be1c9640 1 -- 172.21.15.150:0/3848113804 >> 172.21.15.150:0/3848113804 conn(0x7f39b80c4a00 msgr2=0x7f39b80c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.852+0000 7f39be1c9640 1 -- 172.21.15.150:0/3848113804 shutdown_connections 2024-09-17T05:20:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.852+0000 7f39be1c9640 1 -- 172.21.15.150:0/3848113804 wait complete. 2024-09-17T05:20:50.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39be1c9640 1 Processor -- start 2024-09-17T05:20:50.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39be1c9640 1 -- start start 2024-09-17T05:20:50.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39be1c9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39b80a7350 0x7f39b81f1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:50.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39be1c9640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39b80c42e0 0x7f39b8155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:50.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39be1c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39b8152cf0 0x7f39b8154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:50.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39be1c9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f39b80c6d50 con 0x7f39b8152cf0 2024-09-17T05:20:50.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39be1c9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f39b80c6bd0 con 0x7f39b80a7350 2024-09-17T05:20:50.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39be1c9640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f39b8153640 con 0x7f39b80c42e0 2024-09-17T05:20:50.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39b7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39b8152cf0 0x7f39b8154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:50.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39b7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39b8152cf0 0x7f39b8154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33156/0 (socket says 172.21.15.150:33156) 2024-09-17T05:20:50.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39b7fff640 1 -- 172.21.15.150:0/3012740024 learned_addr learned my addr 172.21.15.150:0/3012740024 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39b6ffd640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39b80c42e0 0x7f39b8155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39b77fe640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39b80a7350 0x7f39b81f1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39b7fff640 1 -- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39b80c42e0 msgr2=0x7f39b8155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39b7fff640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39b80c42e0 0x7f39b8155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39b7fff640 1 -- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39b80a7350 msgr2=0x7f39b81f1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39b7fff640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39b80a7350 0x7f39b81f1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39b7fff640 1 -- 172.21.15.150:0/3012740024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b8146cb0 con 0x7f39b8152cf0 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39b7fff640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39b8152cf0 0x7f39b8154a70 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f39b000e970 tx=0x7f39b000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39affff640 1 -- 172.21.15.150:0/3012740024 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b000ccc0 con 0x7f39b8152cf0 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39affff640 1 -- 172.21.15.150:0/3012740024 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39b0004020 con 0x7f39b8152cf0 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39be1c9640 1 -- 172.21.15.150:0/3012740024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b81f1c20 con 0x7f39b8152cf0 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39affff640 1 -- 172.21.15.150:0/3012740024 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b000ce60 con 0x7f39b8152cf0 2024-09-17T05:20:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.856+0000 7f39be1c9640 1 -- 172.21.15.150:0/3012740024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b8146920 con 0x7f39b8152cf0 2024-09-17T05:20:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.860+0000 7f39be1c9640 1 -- 172.21.15.150:0/3012740024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39b80c6690 con 0x7f39b8152cf0 2024-09-17T05:20:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.860+0000 7f39affff640 1 -- 172.21.15.150:0/3012740024 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f39b0003600 con 0x7f39b8152cf0 2024-09-17T05:20:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.860+0000 7f39affff640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3998073200 0x7f3998075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.860+0000 7f39affff640 1 -- 172.21.15.150:0/3012740024 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f39b0095ca0 con 0x7f39b8152cf0 2024-09-17T05:20:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.860+0000 7f39b77fe640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3998073200 0x7f3998075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:50.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.860+0000 7f39b77fe640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3998073200 0x7f3998075710 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f39a80b0000 tx=0x7f39a801c700 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:50.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:50.864+0000 7f39affff640 1 -- 172.21.15.150:0/3012740024 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f39b005fc10 con 0x7f39b8152cf0 2024-09-17T05:20:51.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.044+0000 7f39be1c9640 1 -- 172.21.15.150:0/3012740024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f39b80a7130 con 0x7f39b8152cf0 2024-09-17T05:20:51.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.044+0000 7f39affff640 1 -- 172.21.15.150:0/3012740024 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 82 entries v68) ==== 81+0+5048 (secure 0 0 0) 0x7f39b005f510 con 0x7f39b8152cf0 2024-09-17T05:20:51.044 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2857695220 2024-09-18T05:20:36.328050+0000 2024-09-17T05:20:51.044 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/2857695220 2024-09-18T05:20:36.328050+0000 2024-09-17T05:20:51.044 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6835/1690935336 2024-09-18T05:19:51.316025+0000 2024-09-17T05:20:51.044 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6834/1690935336 2024-09-18T05:19:51.316025+0000 2024-09-17T05:20:51.044 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3297154462 2024-09-18T05:18:31.294755+0000 2024-09-17T05:20:51.044 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/1903363960 2024-09-18T05:18:31.294598+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/1903363960 2024-09-18T05:18:31.294598+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3297154462 2024-09-18T05:18:31.294755+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6835/524846644 2024-09-18T05:16:53.685445+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/949083177 2024-09-18T05:16:51.270750+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/178583496 2024-09-18T05:15:51.246236+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/2338457617 2024-09-18T05:15:08.260160+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/3933379792 2024-09-18T05:15:06.233954+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3179072684 2024-09-18T05:14:26.220879+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/12982302 2024-09-18T05:13:46.203493+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/2962113435 2024-09-18T05:12:21.179693+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/178583496 2024-09-18T05:15:51.246236+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/713081612 2024-09-18T04:59:20.944548+0000 2024-09-17T05:20:51.045 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/2081398022 2024-09-18T05:00:20.961331+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3079958649 2024-09-18T04:58:00.922711+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3079958649 2024-09-18T04:58:00.922711+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/12982302 2024-09-18T05:13:46.203493+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/233311351 2024-09-18T05:04:06.023230+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/949083177 2024-09-18T05:16:51.270750+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/284564893 2024-09-18T04:57:40.911134+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/903190072 2024-09-18T05:04:36.031676+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/2374314256 2024-09-18T04:56:10.885556+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3179072684 2024-09-18T05:14:26.220879+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/3543629780 2024-09-18T04:58:30.930118+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/4286123257 2024-09-18T04:55:40.878553+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/480168496 2024-09-18T04:51:40.821573+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/1045115232 2024-09-18T04:53:50.852056+0000 2024-09-17T05:20:51.046 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/1045115232 2024-09-18T04:53:50.852056+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/285376980 2024-09-18T05:10:51.151028+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/2313391683 2024-09-18T05:00:45.968345+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/2081398022 2024-09-18T05:00:20.961331+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/480168496 2024-09-18T04:51:40.821573+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/151500360 2024-09-18T04:54:55.866806+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2313391683 2024-09-18T05:00:45.968345+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/1275062894 2024-09-18T04:58:50.936080+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/1275062894 2024-09-18T04:58:50.936080+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/961213876 2024-09-18T04:52:55.838337+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/151500360 2024-09-18T04:54:55.866806+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/2374314256 2024-09-18T04:56:10.885556+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/1237450980 2024-09-18T05:06:31.077239+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/2338457617 2024-09-18T05:15:08.260160+0000 2024-09-17T05:20:51.047 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/961213876 2024-09-18T04:52:55.838337+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/3532647866 2024-09-18T04:55:10.871121+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/713081612 2024-09-18T04:59:20.944548+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/792092400 2024-09-18T05:09:46.133255+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2613040311 2024-09-18T04:51:25.818032+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/3532647866 2024-09-18T04:55:10.871121+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/2062728083 2024-09-18T05:03:01.005221+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/523364553 2024-09-18T05:07:21.092015+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/233311351 2024-09-18T05:04:06.023230+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/2962113435 2024-09-18T05:12:21.179693+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/4286123257 2024-09-18T04:55:40.878553+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/792092400 2024-09-18T05:09:46.133255+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/702608997 2024-09-18T05:02:30.998209+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/284564893 2024-09-18T04:57:40.911134+0000 2024-09-17T05:20:51.048 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/3622725045 2024-09-18T05:09:06.121144+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/2062728083 2024-09-18T05:03:01.005221+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/903190072 2024-09-18T05:04:36.031676+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/1984876738 2024-09-18T05:05:21.045156+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6834/524846644 2024-09-18T05:16:53.685445+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/1984876738 2024-09-18T05:05:21.045156+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/3933379792 2024-09-18T05:15:06.233954+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3807957687 2024-09-18T05:05:56.055429+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/3807957687 2024-09-18T05:05:56.055429+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/3940723333 2024-09-18T05:06:51.083809+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/1237450980 2024-09-18T05:06:31.077239+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/3543629780 2024-09-18T04:58:30.930118+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6837/702608997 2024-09-18T05:02:30.998209+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/92695211 2024-09-18T05:08:16.109918+0000 2024-09-17T05:20:51.049 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6835/2613040311 2024-09-18T04:51:25.818032+0000 2024-09-17T05:20:51.050 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/523364553 2024-09-18T05:07:21.092015+0000 2024-09-17T05:20:51.050 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/92695211 2024-09-18T05:08:16.109918+0000 2024-09-17T05:20:51.050 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/3622725045 2024-09-18T05:09:06.121144+0000 2024-09-17T05:20:51.050 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.176:6836/3940723333 2024-09-18T05:06:51.083809+0000 2024-09-17T05:20:51.050 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/285376980 2024-09-18T05:10:51.151028+0000 2024-09-17T05:20:51.050 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6833/2056467678 2024-09-18T05:12:01.173633+0000 2024-09-17T05:20:51.050 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6832/2056467678 2024-09-18T05:12:01.173633+0000 2024-09-17T05:20:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:listed 82 entries 2024-09-17T05:20:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.048+0000 7f39be1c9640 1 -- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3998073200 msgr2=0x7f3998075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.048+0000 7f39be1c9640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3998073200 0x7f3998075710 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f39a80b0000 tx=0x7f39a801c700 comp rx=0 tx=0).stop 2024-09-17T05:20:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.048+0000 7f39be1c9640 1 -- 172.21.15.150:0/3012740024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39b8152cf0 msgr2=0x7f39b8154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.048+0000 7f39be1c9640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39b8152cf0 0x7f39b8154a70 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f39b000e970 tx=0x7f39b000ee30 comp rx=0 tx=0).stop 2024-09-17T05:20:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.048+0000 7f39be1c9640 1 -- 172.21.15.150:0/3012740024 shutdown_connections 2024-09-17T05:20:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.048+0000 7f39be1c9640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f3998073200 0x7f3998075710 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.048+0000 7f39be1c9640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f39b8152cf0 0x7f39b8154a70 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:51.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.048+0000 7f39be1c9640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f39b80c42e0 0x7f39b8155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:51.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.048+0000 7f39be1c9640 1 --2- 172.21.15.150:0/3012740024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39b80a7350 0x7f39b81f1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:51.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.048+0000 7f39be1c9640 1 -- 172.21.15.150:0/3012740024 >> 172.21.15.150:0/3012740024 conn(0x7f39b80c4a00 msgr2=0x7f39b80746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:51.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.048+0000 7f39be1c9640 1 -- 172.21.15.150:0/3012740024 shutdown_connections 2024-09-17T05:20:51.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:51.048+0000 7f39be1c9640 1 -- 172.21.15.150:0/3012740024 wait complete. 2024-09-17T05:20:51.067 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi176.front.sepia.ceph.com... 2024-09-17T05:20:51.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:20:51.067 DEBUG:teuthology.orchestra.run.smithi176:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-17T05:20:51.098 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:20:51.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi176.stderr:ceph-fuse[34616]: fuse finished with error 0 and tester_r 0 2024-09-17T05:20:57.200 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi176.front.sepia.ceph.com 2024-09-17T05:20:57.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:20:57.200 DEBUG:teuthology.orchestra.run.smithi176:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-17T05:20:57.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:20:57.215 DEBUG:teuthology.orchestra.run.smithi176:> ip netns list 2024-09-17T05:20:57.319 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-17T05:20:57.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:20:57.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-17T05:20:57.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:20:57.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo ip link delete ceph-brx 2024-09-17T05:20:57.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:20:57.565 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-09-17T05:20:57.691 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T05:20:57.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:20:57.691 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T05:20:57.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T05:20:57.706 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-09-17T05:20:57.898 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-17T05:20:57.909 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T05:20:57.910 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:20:58.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.076+0000 7fa45f545640 1 -- 172.21.15.150:0/419817554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa444004810 msgr2=0x7fa444004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.076+0000 7fa45f545640 1 --2- 172.21.15.150:0/419817554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa444004810 0x7fa444004c20 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fa454009e30 tx=0x7fa45401d190 comp rx=0 tx=0).stop 2024-09-17T05:20:58.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.076+0000 7fa45dae2640 1 -- 172.21.15.150:0/419817554 reap_dead start 2024-09-17T05:20:58.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.076+0000 7fa45f545640 1 -- 172.21.15.150:0/419817554 shutdown_connections 2024-09-17T05:20:58.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.076+0000 7fa45f545640 1 -- 172.21.15.150:0/419817554 >> 172.21.15.150:0/419817554 conn(0x7fa4580c4a00 msgr2=0x7fa4580c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:58.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.076+0000 7fa45f545640 1 -- 172.21.15.150:0/419817554 shutdown_connections 2024-09-17T05:20:58.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.076+0000 7fa45f545640 1 -- 172.21.15.150:0/419817554 wait complete. 2024-09-17T05:20:58.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45f545640 1 Processor -- start 2024-09-17T05:20:58.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45f545640 1 -- start start 2024-09-17T05:20:58.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45f545640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa45814feb0 0x7fa458157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:58.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45f545640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa45814d460 0x7fa4581517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:58.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45f545640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45814d870 0x7fa458156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:58.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45f545640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa4580add70 con 0x7fa45814d870 2024-09-17T05:20:58.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45f545640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa4580c3000 con 0x7fa45814d460 2024-09-17T05:20:58.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45f545640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fa4580ad9b0 con 0x7fa45814feb0 2024-09-17T05:20:58.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45dae2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45814d870 0x7fa458156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:58.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45dae2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45814d870 0x7fa458156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33186/0 (socket says 172.21.15.150:33186) 2024-09-17T05:20:58.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45dae2640 1 -- 172.21.15.150:0/854044457 learned_addr learned my addr 172.21.15.150:0/854044457 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45d2e1640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa45814feb0 0x7fa458157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45cae0640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa45814d460 0x7fa4581517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45dae2640 1 -- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa45814feb0 msgr2=0x7fa458157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45dae2640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa45814feb0 0x7fa458157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45dae2640 1 -- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa45814d460 msgr2=0x7fa4581517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45dae2640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa45814d460 0x7fa4581517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45dae2640 1 -- 172.21.15.150:0/854044457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa458150e40 con 0x7fa45814d870 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45cae0640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa45814d460 0x7fa4581517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45dae2640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45814d870 0x7fa458156d10 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fa458072820 tx=0x7fa45000e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa449ffb640 1 -- 172.21.15.150:0/854044457 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa450015030 con 0x7fa45814d870 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa449ffb640 1 -- 172.21.15.150:0/854044457 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa450004350 con 0x7fa45814d870 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45f545640 1 -- 172.21.15.150:0/854044457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa458150530 con 0x7fa45814d870 2024-09-17T05:20:58.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa449ffb640 1 -- 172.21.15.150:0/854044457 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa450005280 con 0x7fa45814d870 2024-09-17T05:20:58.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45f545640 1 -- 172.21.15.150:0/854044457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa458151d00 con 0x7fa45814d870 2024-09-17T05:20:58.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.080+0000 7fa45f545640 1 -- 172.21.15.150:0/854044457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa458150350 con 0x7fa45814d870 2024-09-17T05:20:58.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.084+0000 7fa449ffb640 1 -- 172.21.15.150:0/854044457 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fa450018070 con 0x7fa45814d870 2024-09-17T05:20:58.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.084+0000 7fa449ffb640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa43c073230 0x7fa43c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:58.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.084+0000 7fa449ffb640 1 -- 172.21.15.150:0/854044457 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7fa450095040 con 0x7fa45814d870 2024-09-17T05:20:58.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.084+0000 7fa45d2e1640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa43c073230 0x7fa43c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:58.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.084+0000 7fa45d2e1640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa43c073230 0x7fa43c075740 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fa45806ff70 tx=0x7fa44c001770 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:58.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.088+0000 7fa449ffb640 1 -- 172.21.15.150:0/854044457 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa458150350 con 0x7fa45814d870 2024-09-17T05:20:58.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.248+0000 7fa45f545640 1 -- 172.21.15.150:0/854044457 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa4580a75f0 con 0x7fa43c073230 2024-09-17T05:20:58.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.272+0000 7fa449ffb640 1 -- 172.21.15.150:0/854044457 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493818 (secure 0 0 0) 0x7fa4580a75f0 con 0x7fa43c073230 2024-09-17T05:20:58.276 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:58.281 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T05:20:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.280+0000 7fa45f545640 1 -- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa43c073230 msgr2=0x7fa43c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.280+0000 7fa45f545640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa43c073230 0x7fa43c075740 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fa45806ff70 tx=0x7fa44c001770 comp rx=0 tx=0).stop 2024-09-17T05:20:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.280+0000 7fa45f545640 1 -- 172.21.15.150:0/854044457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45814d870 msgr2=0x7fa458156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.280+0000 7fa45f545640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45814d870 0x7fa458156d10 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fa458072820 tx=0x7fa45000e8f0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.280+0000 7fa45f545640 1 -- 172.21.15.150:0/854044457 shutdown_connections 2024-09-17T05:20:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.280+0000 7fa45f545640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fa43c073230 0x7fa43c075740 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.280+0000 7fa45f545640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa45814d460 0x7fa4581517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.280+0000 7fa45f545640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa45814d870 0x7fa458156d10 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.280+0000 7fa45f545640 1 --2- 172.21.15.150:0/854044457 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fa45814feb0 0x7fa458157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.280+0000 7fa45f545640 1 -- 172.21.15.150:0/854044457 >> 172.21.15.150:0/854044457 conn(0x7fa4580c4a00 msgr2=0x7fa4580a8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:58.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.280+0000 7fa45f545640 1 -- 172.21.15.150:0/854044457 shutdown_connections 2024-09-17T05:20:58.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.280+0000 7fa45f545640 1 -- 172.21.15.150:0/854044457 wait complete. 2024-09-17T05:20:58.299 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":1010,"stamp":"2024-09-17T05:20:57.882067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37864050,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3107,"num_read_kb":438866,"num_write":2440075,"num_write_kb":36007256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":458354,"ondisk_log_size":458354,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5828384,"kb_used_data":80928,"kb_used_omap":12,"kb_used_meta":5747315,"kb_avail":749146336,"statfs":{"total":773094113280,"available":767125848064,"internally_reserved":0,"allocated":82870272,"data_stored":79467812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":5885251122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7484347531,"num_objects":-1955,"num_object_clones":0,"num_object_copies":-3910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1955,"num_whiteouts":0,"num_read":10,"num_read_kb":30,"num_write":2076,"num_write_kb":727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006566"},"pg_stats":[{"pgid":"5.18","version":"68'3479","reported_seq":3591,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.418967+0000","last_change":"2024-09-17T04:49:41.025271+0000","last_active":"2024-09-17T05:20:44.418967+0000","last_peered":"2024-09-17T05:20:44.418967+0000","last_clean":"2024-09-17T05:20:44.418967+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T05:20:44.418967+0000","last_undegraded":"2024-09-17T05:20:44.418967+0000","last_fullsized":"2024-09-17T05:20:44.418967+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3479,"log_dups_size":0,"ondisk_log_size":3479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:39.715602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":12,"num_write":18686,"num_write_kb":287786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"65'1001","reported_seq":1175,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340557+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T05:20:36.340557+0000","last_peered":"2024-09-17T05:20:36.340557+0000","last_clean":"2024-09-17T05:20:36.340557+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T05:20:36.340557+0000","last_undegraded":"2024-09-17T05:20:36.340557+0000","last_fullsized":"2024-09-17T05:20:36.340557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":20774,"num_write":978,"num_write_kb":4679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'4064","reported_seq":4184,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.351711+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T05:20:36.351711+0000","last_peered":"2024-09-17T05:20:36.351711+0000","last_clean":"2024-09-17T05:20:36.351711+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T05:20:36.351711+0000","last_undegraded":"2024-09-17T05:20:36.351711+0000","last_fullsized":"2024-09-17T05:20:36.351711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22217,"num_write_kb":281701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"68'3","reported_seq":114,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402753+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T05:20:43.402753+0000","last_peered":"2024-09-17T05:20:43.402753+0000","last_clean":"2024-09-17T05:20:43.402753+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T05:20:43.402753+0000","last_undegraded":"2024-09-17T05:20:43.402753+0000","last_fullsized":"2024-09-17T05:20:43.402753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:38.417840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"68'3721","reported_seq":3833,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.413711+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T05:20:43.413711+0000","last_peered":"2024-09-17T05:20:43.413711+0000","last_clean":"2024-09-17T05:20:43.413711+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T05:20:43.413711+0000","last_undegraded":"2024-09-17T05:20:43.413711+0000","last_fullsized":"2024-09-17T05:20:43.413711+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3721,"log_dups_size":0,"ondisk_log_size":3721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":22,"num_write":19557,"num_write_kb":316044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"67'26","reported_seq":148,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352348+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T05:20:36.352348+0000","last_peered":"2024-09-17T05:20:36.352348+0000","last_clean":"2024-09-17T05:20:36.352348+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T05:20:36.352348+0000","last_undegraded":"2024-09-17T05:20:36.352348+0000","last_fullsized":"2024-09-17T05:20:36.352348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4998","reported_seq":5118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.353392+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T05:20:36.353392+0000","last_peered":"2024-09-17T05:20:36.353392+0000","last_clean":"2024-09-17T05:20:36.353392+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T05:20:36.353392+0000","last_undegraded":"2024-09-17T05:20:36.353392+0000","last_fullsized":"2024-09-17T05:20:36.353392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27336,"num_write_kb":388205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'2","reported_seq":113,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362495+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T05:20:36.362495+0000","last_peered":"2024-09-17T05:20:36.362495+0000","last_clean":"2024-09-17T05:20:36.362495+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T05:20:36.362495+0000","last_undegraded":"2024-09-17T05:20:36.362495+0000","last_fullsized":"2024-09-17T05:20:36.362495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"68'2765","reported_seq":2876,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.349402+0000","last_change":"2024-09-17T04:49:41.067317+0000","last_active":"2024-09-17T05:20:43.349402+0000","last_peered":"2024-09-17T05:20:43.349402+0000","last_clean":"2024-09-17T05:20:43.349402+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T05:20:43.349402+0000","last_undegraded":"2024-09-17T05:20:43.349402+0000","last_fullsized":"2024-09-17T05:20:43.349402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:37.637477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":14571,"num_write_kb":258996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"68'46","reported_seq":207,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558188+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T05:20:36.558188+0000","last_peered":"2024-09-17T05:20:36.558188+0000","last_clean":"2024-09-17T05:20:36.558188+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T05:20:36.558188+0000","last_undegraded":"2024-09-17T05:20:36.558188+0000","last_fullsized":"2024-09-17T05:20:36.558188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"65'3140","reported_seq":3259,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.346435+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T05:20:36.346435+0000","last_peered":"2024-09-17T05:20:36.346435+0000","last_clean":"2024-09-17T05:20:36.346435+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T05:20:36.346435+0000","last_undegraded":"2024-09-17T05:20:36.346435+0000","last_fullsized":"2024-09-17T05:20:36.346435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3140,"log_dups_size":0,"ondisk_log_size":3140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16962,"num_write_kb":252282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"68'751","reported_seq":888,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.388359+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T05:20:43.388359+0000","last_peered":"2024-09-17T05:20:43.388359+0000","last_clean":"2024-09-17T05:20:43.388359+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T05:20:43.388359+0000","last_undegraded":"2024-09-17T05:20:43.388359+0000","last_fullsized":"2024-09-17T05:20:43.388359+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":75812,"num_write":739,"num_write_kb":4324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"68'2700","reported_seq":2810,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.180944+0000","last_change":"2024-09-17T04:49:41.043531+0000","last_active":"2024-09-17T05:20:43.180944+0000","last_peered":"2024-09-17T05:20:43.180944+0000","last_clean":"2024-09-17T05:20:43.180944+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T05:20:43.180944+0000","last_undegraded":"2024-09-17T05:20:43.180944+0000","last_fullsized":"2024-09-17T05:20:43.180944+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2700,"log_dups_size":0,"ondisk_log_size":2700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":5,"num_write":13438,"num_write_kb":225755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"65'5","reported_seq":166,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559425+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T05:20:36.559425+0000","last_peered":"2024-09-17T05:20:36.559425+0000","last_clean":"2024-09-17T05:20:36.559425+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T05:20:36.559425+0000","last_undegraded":"2024-09-17T05:20:36.559425+0000","last_fullsized":"2024-09-17T05:20:36.559425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"65'4481","reported_seq":4598,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.373776+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T05:20:36.373776+0000","last_peered":"2024-09-17T05:20:36.373776+0000","last_clean":"2024-09-17T05:20:36.373776+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T05:20:36.373776+0000","last_undegraded":"2024-09-17T05:20:36.373776+0000","last_fullsized":"2024-09-17T05:20:36.373776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24963,"num_write_kb":346034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"68'143","reported_seq":256,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:51.881072+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T05:20:51.881072+0000","last_peered":"2024-09-17T05:20:51.881072+0000","last_clean":"2024-09-17T05:20:51.881072+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T05:20:51.881072+0000","last_undegraded":"2024-09-17T05:20:51.881072+0000","last_fullsized":"2024-09-17T05:20:51.881072+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:16.854365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1447387,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":383,"num_write":117,"num_write_kb":1479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"68'2942","reported_seq":3054,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.409983+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T05:20:44.409983+0000","last_peered":"2024-09-17T05:20:44.409983+0000","last_clean":"2024-09-17T05:20:44.409983+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T05:20:44.409983+0000","last_undegraded":"2024-09-17T05:20:44.409983+0000","last_fullsized":"2024-09-17T05:20:44.409983+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":16254,"num_write_kb":278162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"65'3968","reported_seq":4087,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362298+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T05:20:36.362298+0000","last_peered":"2024-09-17T05:20:36.362298+0000","last_clean":"2024-09-17T05:20:36.362298+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T05:20:36.362298+0000","last_undegraded":"2024-09-17T05:20:36.362298+0000","last_fullsized":"2024-09-17T05:20:36.362298+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3968,"log_dups_size":0,"ondisk_log_size":3968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21939,"num_write_kb":297018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"68'1467","reported_seq":1636,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366269+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T05:20:36.366269+0000","last_peered":"2024-09-17T05:20:36.366269+0000","last_clean":"2024-09-17T05:20:36.366269+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T05:20:36.366269+0000","last_undegraded":"2024-09-17T05:20:36.366269+0000","last_fullsized":"2024-09-17T05:20:36.366269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1467,"log_dups_size":0,"ondisk_log_size":1467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":43333,"num_write":1444,"num_write_kb":4521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'6","reported_seq":138,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.115470+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T05:20:43.115470+0000","last_peered":"2024-09-17T05:20:43.115470+0000","last_clean":"2024-09-17T05:20:43.115470+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T05:20:43.115470+0000","last_undegraded":"2024-09-17T05:20:43.115470+0000","last_fullsized":"2024-09-17T05:20:43.115470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:28.365399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'3236","reported_seq":3349,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.400322+0000","last_change":"2024-09-17T04:49:41.038415+0000","last_active":"2024-09-17T05:20:44.400322+0000","last_peered":"2024-09-17T05:20:44.400322+0000","last_clean":"2024-09-17T05:20:44.400322+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T05:20:44.400322+0000","last_undegraded":"2024-09-17T05:20:44.400322+0000","last_fullsized":"2024-09-17T05:20:44.400322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3236,"log_dups_size":0,"ondisk_log_size":3236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:17.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":17542,"num_write_kb":271055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"65'3303","reported_seq":3421,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.358141+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T05:20:36.358141+0000","last_peered":"2024-09-17T05:20:36.358141+0000","last_clean":"2024-09-17T05:20:36.358141+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T05:20:36.358141+0000","last_undegraded":"2024-09-17T05:20:36.358141+0000","last_fullsized":"2024-09-17T05:20:36.358141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3303,"log_dups_size":0,"ondisk_log_size":3303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17463,"num_write_kb":238014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"65'3","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.344536+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T05:20:36.344536+0000","last_peered":"2024-09-17T05:20:36.344536+0000","last_clean":"2024-09-17T05:20:36.344536+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T05:20:36.344536+0000","last_undegraded":"2024-09-17T05:20:36.344536+0000","last_fullsized":"2024-09-17T05:20:36.344536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"68'5","reported_seq":117,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.405924+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T05:20:43.405924+0000","last_peered":"2024-09-17T05:20:43.405924+0000","last_clean":"2024-09-17T05:20:43.405924+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T05:20:43.405924+0000","last_undegraded":"2024-09-17T05:20:43.405924+0000","last_fullsized":"2024-09-17T05:20:43.405924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:17.424265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"68'2691","reported_seq":2802,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.414897+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T05:20:43.414897+0000","last_peered":"2024-09-17T05:20:43.414897+0000","last_clean":"2024-09-17T05:20:43.414897+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T05:20:43.414897+0000","last_undegraded":"2024-09-17T05:20:43.414897+0000","last_fullsized":"2024-09-17T05:20:43.414897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:06.516432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":14365,"num_write_kb":225564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"65'3640","reported_seq":3759,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.337791+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T05:20:36.337791+0000","last_peered":"2024-09-17T05:20:36.337791+0000","last_clean":"2024-09-17T05:20:36.337791+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T05:20:36.337791+0000","last_undegraded":"2024-09-17T05:20:36.337791+0000","last_fullsized":"2024-09-17T05:20:36.337791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3640,"log_dups_size":0,"ondisk_log_size":3640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20183,"num_write_kb":277678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"65'2","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340129+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T05:20:36.340129+0000","last_peered":"2024-09-17T05:20:36.340129+0000","last_clean":"2024-09-17T05:20:36.340129+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T05:20:36.340129+0000","last_undegraded":"2024-09-17T05:20:36.340129+0000","last_fullsized":"2024-09-17T05:20:36.340129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"68'4","reported_seq":136,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.390191+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T05:20:43.390191+0000","last_peered":"2024-09-17T05:20:43.390191+0000","last_clean":"2024-09-17T05:20:43.390191+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T05:20:43.390191+0000","last_undegraded":"2024-09-17T05:20:43.390191+0000","last_fullsized":"2024-09-17T05:20:43.390191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:00.010365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"68'3943","reported_seq":4053,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.367267+0000","last_change":"2024-09-17T04:49:41.036658+0000","last_active":"2024-09-17T05:20:43.367267+0000","last_peered":"2024-09-17T05:20:43.367267+0000","last_clean":"2024-09-17T05:20:43.367267+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T05:20:43.367267+0000","last_undegraded":"2024-09-17T05:20:43.367267+0000","last_fullsized":"2024-09-17T05:20:43.367267+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21088,"num_write_kb":359062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"65'3699","reported_seq":3818,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364687+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T05:20:36.364687+0000","last_peered":"2024-09-17T05:20:36.364687+0000","last_clean":"2024-09-17T05:20:36.364687+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T05:20:36.364687+0000","last_undegraded":"2024-09-17T05:20:36.364687+0000","last_fullsized":"2024-09-17T05:20:36.364687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19661,"num_write_kb":271466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"65'5","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.555326+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T05:20:36.555326+0000","last_peered":"2024-09-17T05:20:36.555326+0000","last_clean":"2024-09-17T05:20:36.555326+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T05:20:36.555326+0000","last_undegraded":"2024-09-17T05:20:36.555326+0000","last_fullsized":"2024-09-17T05:20:36.555326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"68'7","reported_seq":159,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.387118+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T05:20:43.387118+0000","last_peered":"2024-09-17T05:20:43.387118+0000","last_clean":"2024-09-17T05:20:43.387118+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T05:20:43.387118+0000","last_undegraded":"2024-09-17T05:20:43.387118+0000","last_fullsized":"2024-09-17T05:20:43.387118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"68'2532","reported_seq":2644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.411846+0000","last_change":"2024-09-17T04:49:41.050170+0000","last_active":"2024-09-17T05:20:43.411846+0000","last_peered":"2024-09-17T05:20:43.411846+0000","last_clean":"2024-09-17T05:20:43.411846+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T05:20:43.411846+0000","last_undegraded":"2024-09-17T05:20:43.411846+0000","last_fullsized":"2024-09-17T05:20:43.411846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2532,"log_dups_size":0,"ondisk_log_size":2532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13954,"num_write_kb":219876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"65'4598","reported_seq":4718,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339224+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T05:20:36.339224+0000","last_peered":"2024-09-17T05:20:36.339224+0000","last_clean":"2024-09-17T05:20:36.339224+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T05:20:36.339224+0000","last_undegraded":"2024-09-17T05:20:36.339224+0000","last_fullsized":"2024-09-17T05:20:36.339224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24313,"num_write_kb":335123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"65'9","reported_seq":172,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.367024+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T05:20:36.367024+0000","last_peered":"2024-09-17T05:20:36.367024+0000","last_clean":"2024-09-17T05:20:36.367024+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T05:20:36.367024+0000","last_undegraded":"2024-09-17T05:20:36.367024+0000","last_fullsized":"2024-09-17T05:20:36.367024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'50","reported_seq":229,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:48.374529+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T05:20:48.374529+0000","last_peered":"2024-09-17T05:20:48.374529+0000","last_clean":"2024-09-17T05:20:48.374529+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T05:20:48.374529+0000","last_undegraded":"2024-09-17T05:20:48.374529+0000","last_fullsized":"2024-09-17T05:20:48.374529+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":65,"num_read_kb":114,"num_write":45,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"68'2531","reported_seq":2644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.376480+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T05:20:43.376480+0000","last_peered":"2024-09-17T05:20:43.376480+0000","last_clean":"2024-09-17T05:20:43.376480+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T05:20:43.376480+0000","last_undegraded":"2024-09-17T05:20:43.376480+0000","last_fullsized":"2024-09-17T05:20:43.376480+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2531,"log_dups_size":0,"ondisk_log_size":2531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13881,"num_write_kb":219504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"65'3884","reported_seq":4003,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340033+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T05:20:36.340033+0000","last_peered":"2024-09-17T05:20:36.340033+0000","last_clean":"2024-09-17T05:20:36.340033+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T05:20:36.340033+0000","last_undegraded":"2024-09-17T05:20:36.340033+0000","last_fullsized":"2024-09-17T05:20:36.340033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21342,"num_write_kb":257935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"68'473","reported_seq":615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.328144+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T05:20:44.328144+0000","last_peered":"2024-09-17T05:20:44.328144+0000","last_clean":"2024-09-17T05:20:44.328144+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T05:20:44.328144+0000","last_undegraded":"2024-09-17T05:20:44.328144+0000","last_fullsized":"2024-09-17T05:20:44.328144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":57,"num_write":1105,"num_write_kb":359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'11","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410351+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T05:20:43.410351+0000","last_peered":"2024-09-17T05:20:43.410351+0000","last_clean":"2024-09-17T05:20:43.410351+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T05:20:43.410351+0000","last_undegraded":"2024-09-17T05:20:43.410351+0000","last_fullsized":"2024-09-17T05:20:43.410351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"68'2691","reported_seq":2802,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.217861+0000","last_change":"2024-09-17T04:49:41.042314+0000","last_active":"2024-09-17T05:20:43.217861+0000","last_peered":"2024-09-17T05:20:43.217861+0000","last_clean":"2024-09-17T05:20:43.217861+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T05:20:43.217861+0000","last_undegraded":"2024-09-17T05:20:43.217861+0000","last_fullsized":"2024-09-17T05:20:43.217861+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14724,"num_write_kb":243278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"65'4637","reported_seq":4757,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341853+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T05:20:36.341853+0000","last_peered":"2024-09-17T05:20:36.341853+0000","last_clean":"2024-09-17T05:20:36.341853+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T05:20:36.341853+0000","last_undegraded":"2024-09-17T05:20:36.341853+0000","last_fullsized":"2024-09-17T05:20:36.341853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25111,"num_write_kb":342484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"65'2","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.375220+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T05:20:36.375220+0000","last_peered":"2024-09-17T05:20:36.375220+0000","last_clean":"2024-09-17T05:20:36.375220+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T05:20:36.375220+0000","last_undegraded":"2024-09-17T05:20:36.375220+0000","last_fullsized":"2024-09-17T05:20:36.375220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'3","reported_seq":117,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393956+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T05:20:43.393956+0000","last_peered":"2024-09-17T05:20:43.393956+0000","last_clean":"2024-09-17T05:20:43.393956+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T05:20:43.393956+0000","last_undegraded":"2024-09-17T05:20:43.393956+0000","last_fullsized":"2024-09-17T05:20:43.393956+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"68'3511","reported_seq":3623,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.341640+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T05:20:43.341640+0000","last_peered":"2024-09-17T05:20:43.341640+0000","last_clean":"2024-09-17T05:20:43.341640+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T05:20:43.341640+0000","last_undegraded":"2024-09-17T05:20:43.341640+0000","last_fullsized":"2024-09-17T05:20:43.341640+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3511,"log_dups_size":0,"ondisk_log_size":3511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19215,"num_write_kb":314590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"65'3495","reported_seq":3615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362299+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T05:20:36.362299+0000","last_peered":"2024-09-17T05:20:36.362299+0000","last_clean":"2024-09-17T05:20:36.362299+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T05:20:36.362299+0000","last_undegraded":"2024-09-17T05:20:36.362299+0000","last_fullsized":"2024-09-17T05:20:36.362299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18740,"num_write_kb":251346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"65'2","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.351220+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T05:20:36.351220+0000","last_peered":"2024-09-17T05:20:36.351220+0000","last_clean":"2024-09-17T05:20:36.351220+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T05:20:36.351220+0000","last_undegraded":"2024-09-17T05:20:36.351220+0000","last_fullsized":"2024-09-17T05:20:36.351220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'4","reported_seq":118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.112153+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T05:20:43.112153+0000","last_peered":"2024-09-17T05:20:43.112153+0000","last_clean":"2024-09-17T05:20:43.112153+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T05:20:43.112153+0000","last_undegraded":"2024-09-17T05:20:43.112153+0000","last_fullsized":"2024-09-17T05:20:43.112153+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"68'2923","reported_seq":3035,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.395059+0000","last_change":"2024-09-17T04:49:41.069177+0000","last_active":"2024-09-17T05:20:43.395059+0000","last_peered":"2024-09-17T05:20:43.395059+0000","last_clean":"2024-09-17T05:20:43.395059+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T05:20:43.395059+0000","last_undegraded":"2024-09-17T05:20:43.395059+0000","last_fullsized":"2024-09-17T05:20:43.395059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:16.855206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":7,"num_write":15908,"num_write_kb":260386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"65'4495","reported_seq":4615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559390+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T05:20:36.559390+0000","last_peered":"2024-09-17T05:20:36.559390+0000","last_clean":"2024-09-17T05:20:36.559390+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T05:20:36.559390+0000","last_undegraded":"2024-09-17T05:20:36.559390+0000","last_fullsized":"2024-09-17T05:20:36.559390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24287,"num_write_kb":330898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"65'4","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.365137+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T05:20:36.365137+0000","last_peered":"2024-09-17T05:20:36.365137+0000","last_clean":"2024-09-17T05:20:36.365137+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T05:20:36.365137+0000","last_undegraded":"2024-09-17T05:20:36.365137+0000","last_fullsized":"2024-09-17T05:20:36.365137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'2","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.401317+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T05:20:43.401317+0000","last_peered":"2024-09-17T05:20:43.401317+0000","last_clean":"2024-09-17T05:20:43.401317+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T05:20:43.401317+0000","last_undegraded":"2024-09-17T05:20:43.401317+0000","last_fullsized":"2024-09-17T05:20:43.401317+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"68'3959","reported_seq":4074,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.362853+0000","last_change":"2024-09-17T04:49:41.068537+0000","last_active":"2024-09-17T05:20:43.362853+0000","last_peered":"2024-09-17T05:20:43.362853+0000","last_clean":"2024-09-17T05:20:43.362853+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T05:20:43.362853+0000","last_undegraded":"2024-09-17T05:20:43.362853+0000","last_fullsized":"2024-09-17T05:20:43.362853+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3959,"log_dups_size":0,"ondisk_log_size":3959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":13,"num_write":21335,"num_write_kb":323115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'3351","reported_seq":3471,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.348480+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T05:20:36.348480+0000","last_peered":"2024-09-17T05:20:36.348480+0000","last_clean":"2024-09-17T05:20:36.348480+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T05:20:36.348480+0000","last_undegraded":"2024-09-17T05:20:36.348480+0000","last_fullsized":"2024-09-17T05:20:36.348480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18427,"num_write_kb":241864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"68'53","reported_seq":243,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.366344+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T05:20:44.366344+0000","last_peered":"2024-09-17T05:20:44.366344+0000","last_clean":"2024-09-17T05:20:44.366344+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T05:20:44.366344+0000","last_undegraded":"2024-09-17T05:20:44.366344+0000","last_fullsized":"2024-09-17T05:20:44.366344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":116,"num_write":48,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'9","reported_seq":162,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.386330+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T05:20:44.386330+0000","last_peered":"2024-09-17T05:20:44.386330+0000","last_clean":"2024-09-17T05:20:44.386330+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T05:20:44.386330+0000","last_undegraded":"2024-09-17T05:20:44.386330+0000","last_fullsized":"2024-09-17T05:20:44.386330+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"68'2708","reported_seq":2819,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403990+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T05:20:44.403990+0000","last_peered":"2024-09-17T05:20:44.403990+0000","last_clean":"2024-09-17T05:20:44.403990+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T05:20:44.403990+0000","last_undegraded":"2024-09-17T05:20:44.403990+0000","last_fullsized":"2024-09-17T05:20:44.403990+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14599,"num_write_kb":234639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"65'3456","reported_seq":3576,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339164+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T05:20:36.339164+0000","last_peered":"2024-09-17T05:20:36.339164+0000","last_clean":"2024-09-17T05:20:36.339164+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T05:20:36.339164+0000","last_undegraded":"2024-09-17T05:20:36.339164+0000","last_fullsized":"2024-09-17T05:20:36.339164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18921,"num_write_kb":260292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"67'13","reported_seq":135,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366577+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T05:20:36.366577+0000","last_peered":"2024-09-17T05:20:36.366577+0000","last_clean":"2024-09-17T05:20:36.366577+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T05:20:36.366577+0000","last_undegraded":"2024-09-17T05:20:36.366577+0000","last_fullsized":"2024-09-17T05:20:36.366577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"68'473","reported_seq":606,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:47.345918+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T05:20:47.345918+0000","last_peered":"2024-09-17T05:20:47.345918+0000","last_clean":"2024-09-17T05:20:47.345918+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T05:20:47.345918+0000","last_undegraded":"2024-09-17T05:20:47.345918+0000","last_fullsized":"2024-09-17T05:20:47.345918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":54,"num_write":1100,"num_write_kb":354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"68'3699","reported_seq":3811,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.383257+0000","last_change":"2024-09-17T04:49:41.052564+0000","last_active":"2024-09-17T05:20:43.383257+0000","last_peered":"2024-09-17T05:20:43.383257+0000","last_clean":"2024-09-17T05:20:43.383257+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T05:20:43.383257+0000","last_undegraded":"2024-09-17T05:20:43.383257+0000","last_fullsized":"2024-09-17T05:20:43.383257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":20021,"num_write_kb":309708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"65'3651","reported_seq":3771,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341142+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T05:20:36.341142+0000","last_peered":"2024-09-17T05:20:36.341142+0000","last_clean":"2024-09-17T05:20:36.341142+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T05:20:36.341142+0000","last_undegraded":"2024-09-17T05:20:36.341142+0000","last_fullsized":"2024-09-17T05:20:36.341142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19517,"num_write_kb":274922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"65'3","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366726+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T05:20:36.366726+0000","last_peered":"2024-09-17T05:20:36.366726+0000","last_clean":"2024-09-17T05:20:36.366726+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T05:20:36.366726+0000","last_undegraded":"2024-09-17T05:20:36.366726+0000","last_fullsized":"2024-09-17T05:20:36.366726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":218,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.343214+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T05:20:36.343214+0000","last_peered":"2024-09-17T05:20:36.343214+0000","last_clean":"2024-09-17T05:20:36.343214+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T05:20:36.343214+0000","last_undegraded":"2024-09-17T05:20:36.343214+0000","last_fullsized":"2024-09-17T05:20:36.343214+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'2","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374244+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T05:20:36.374244+0000","last_peered":"2024-09-17T05:20:36.374244+0000","last_clean":"2024-09-17T05:20:36.374244+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T05:20:36.374244+0000","last_undegraded":"2024-09-17T05:20:36.374244+0000","last_fullsized":"2024-09-17T05:20:36.374244+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:21.202398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"68'2130","reported_seq":2241,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404345+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T05:20:44.404345+0000","last_peered":"2024-09-17T05:20:44.404345+0000","last_clean":"2024-09-17T05:20:44.404345+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T05:20:44.404345+0000","last_undegraded":"2024-09-17T05:20:44.404345+0000","last_fullsized":"2024-09-17T05:20:44.404345+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2130,"log_dups_size":0,"ondisk_log_size":2130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:48.128609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":11413,"num_write_kb":185062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"65'4183","reported_seq":4301,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374398+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T05:20:36.374398+0000","last_peered":"2024-09-17T05:20:36.374398+0000","last_clean":"2024-09-17T05:20:36.374398+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T05:20:36.374398+0000","last_undegraded":"2024-09-17T05:20:36.374398+0000","last_fullsized":"2024-09-17T05:20:36.374398+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22326,"num_write_kb":302603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"65'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.342324+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T05:20:36.342324+0000","last_peered":"2024-09-17T05:20:36.342324+0000","last_clean":"2024-09-17T05:20:36.342324+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T05:20:36.342324+0000","last_undegraded":"2024-09-17T05:20:36.342324+0000","last_fullsized":"2024-09-17T05:20:36.342324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"68'3","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.376325+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T05:20:43.376325+0000","last_peered":"2024-09-17T05:20:43.376325+0000","last_clean":"2024-09-17T05:20:43.376325+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T05:20:43.376325+0000","last_undegraded":"2024-09-17T05:20:43.376325+0000","last_fullsized":"2024-09-17T05:20:43.376325+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"68'3285","reported_seq":3397,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.413822+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T05:20:43.413822+0000","last_peered":"2024-09-17T05:20:43.413822+0000","last_clean":"2024-09-17T05:20:43.413822+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T05:20:43.413822+0000","last_undegraded":"2024-09-17T05:20:43.413822+0000","last_fullsized":"2024-09-17T05:20:43.413822+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3285,"log_dups_size":0,"ondisk_log_size":3285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18245,"num_write_kb":282158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"65'4407","reported_seq":4525,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558055+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T05:20:36.558055+0000","last_peered":"2024-09-17T05:20:36.558055+0000","last_clean":"2024-09-17T05:20:36.558055+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T05:20:36.558055+0000","last_undegraded":"2024-09-17T05:20:36.558055+0000","last_fullsized":"2024-09-17T05:20:36.558055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24558,"num_write_kb":315810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"65'6","reported_seq":147,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352587+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T05:20:36.352587+0000","last_peered":"2024-09-17T05:20:36.352587+0000","last_clean":"2024-09-17T05:20:36.352587+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T05:20:36.352587+0000","last_undegraded":"2024-09-17T05:20:36.352587+0000","last_fullsized":"2024-09-17T05:20:36.352587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'1445","reported_seq":1601,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410628+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T05:20:43.410628+0000","last_peered":"2024-09-17T05:20:43.410628+0000","last_clean":"2024-09-17T05:20:43.410628+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T05:20:43.410628+0000","last_undegraded":"2024-09-17T05:20:43.410628+0000","last_fullsized":"2024-09-17T05:20:43.410628+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:18.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":38239,"num_write":1421,"num_write_kb":4489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'2587","reported_seq":2698,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.177381+0000","last_change":"2024-09-17T04:49:41.031036+0000","last_active":"2024-09-17T05:20:43.177381+0000","last_peered":"2024-09-17T05:20:43.177381+0000","last_clean":"2024-09-17T05:20:43.177381+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T05:20:43.177381+0000","last_undegraded":"2024-09-17T05:20:43.177381+0000","last_fullsized":"2024-09-17T05:20:43.177381+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2587,"log_dups_size":0,"ondisk_log_size":2587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:48.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13558,"num_write_kb":238553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"65'4762","reported_seq":4880,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559085+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T05:20:36.559085+0000","last_peered":"2024-09-17T05:20:36.559085+0000","last_clean":"2024-09-17T05:20:36.559085+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T05:20:36.559085+0000","last_undegraded":"2024-09-17T05:20:36.559085+0000","last_fullsized":"2024-09-17T05:20:36.559085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25049,"num_write_kb":354398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"65'7","reported_seq":169,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340155+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T05:20:36.340155+0000","last_peered":"2024-09-17T05:20:36.340155+0000","last_clean":"2024-09-17T05:20:36.340155+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T05:20:36.340155+0000","last_undegraded":"2024-09-17T05:20:36.340155+0000","last_fullsized":"2024-09-17T05:20:36.340155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"68'5","reported_seq":118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.116235+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T05:20:43.116235+0000","last_peered":"2024-09-17T05:20:43.116235+0000","last_clean":"2024-09-17T05:20:43.116235+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T05:20:43.116235+0000","last_undegraded":"2024-09-17T05:20:43.116235+0000","last_fullsized":"2024-09-17T05:20:43.116235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"68'2327","reported_seq":2438,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.108349+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T05:20:43.108349+0000","last_peered":"2024-09-17T05:20:43.108349+0000","last_clean":"2024-09-17T05:20:43.108349+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T05:20:43.108349+0000","last_undegraded":"2024-09-17T05:20:43.108349+0000","last_fullsized":"2024-09-17T05:20:43.108349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2327,"log_dups_size":0,"ondisk_log_size":2327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12892,"num_write_kb":211370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"65'3722","reported_seq":3840,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366989+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T05:20:36.366989+0000","last_peered":"2024-09-17T05:20:36.366989+0000","last_clean":"2024-09-17T05:20:36.366989+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T05:20:36.366989+0000","last_undegraded":"2024-09-17T05:20:36.366989+0000","last_fullsized":"2024-09-17T05:20:36.366989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3722,"log_dups_size":0,"ondisk_log_size":3722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19724,"num_write_kb":263684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"65'4","reported_seq":145,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.560132+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T05:20:36.560132+0000","last_peered":"2024-09-17T05:20:36.560132+0000","last_clean":"2024-09-17T05:20:36.560132+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T05:20:36.560132+0000","last_undegraded":"2024-09-17T05:20:36.560132+0000","last_fullsized":"2024-09-17T05:20:36.560132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"68'2","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.385249+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T05:20:43.385249+0000","last_peered":"2024-09-17T05:20:43.385249+0000","last_clean":"2024-09-17T05:20:43.385249+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T05:20:43.385249+0000","last_undegraded":"2024-09-17T05:20:43.385249+0000","last_fullsized":"2024-09-17T05:20:43.385249+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:09.712272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"68'3186","reported_seq":3295,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.381181+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T05:20:43.381181+0000","last_peered":"2024-09-17T05:20:43.381181+0000","last_clean":"2024-09-17T05:20:43.381181+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T05:20:43.381181+0000","last_undegraded":"2024-09-17T05:20:43.381181+0000","last_fullsized":"2024-09-17T05:20:43.381181+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:43.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16763,"num_write_kb":264037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"65'3694","reported_seq":3814,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362961+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T05:20:36.362961+0000","last_peered":"2024-09-17T05:20:36.362961+0000","last_clean":"2024-09-17T05:20:36.362961+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T05:20:36.362961+0000","last_undegraded":"2024-09-17T05:20:36.362961+0000","last_fullsized":"2024-09-17T05:20:36.362961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20762,"num_write_kb":284135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"68'15","reported_seq":223,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.342372+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T05:20:36.342372+0000","last_peered":"2024-09-17T05:20:36.342372+0000","last_clean":"2024-09-17T05:20:36.342372+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T05:20:36.342372+0000","last_undegraded":"2024-09-17T05:20:36.342372+0000","last_fullsized":"2024-09-17T05:20:36.342372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":264,"num_read_kb":264,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"68'3","reported_seq":114,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.394770+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T05:20:43.394770+0000","last_peered":"2024-09-17T05:20:43.394770+0000","last_clean":"2024-09-17T05:20:43.394770+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T05:20:43.394770+0000","last_undegraded":"2024-09-17T05:20:43.394770+0000","last_fullsized":"2024-09-17T05:20:43.394770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:52.364568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"68'3074","reported_seq":3186,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.395538+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T05:20:44.395538+0000","last_peered":"2024-09-17T05:20:44.395538+0000","last_clean":"2024-09-17T05:20:44.395538+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T05:20:44.395538+0000","last_undegraded":"2024-09-17T05:20:44.395538+0000","last_fullsized":"2024-09-17T05:20:44.395538+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:03.537392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":16509,"num_write_kb":287321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'2828","reported_seq":2946,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.342437+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T05:20:36.342437+0000","last_peered":"2024-09-17T05:20:36.342437+0000","last_clean":"2024-09-17T05:20:36.342437+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T05:20:36.342437+0000","last_undegraded":"2024-09-17T05:20:36.342437+0000","last_fullsized":"2024-09-17T05:20:36.342437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14821,"num_write_kb":206837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"65'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557382+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T05:20:36.557382+0000","last_peered":"2024-09-17T05:20:36.557382+0000","last_clean":"2024-09-17T05:20:36.557382+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T05:20:36.557382+0000","last_undegraded":"2024-09-17T05:20:36.557382+0000","last_fullsized":"2024-09-17T05:20:36.557382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"68'5","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.388852+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T05:20:43.388852+0000","last_peered":"2024-09-17T05:20:43.388852+0000","last_clean":"2024-09-17T05:20:43.388852+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T05:20:43.388852+0000","last_undegraded":"2024-09-17T05:20:43.388852+0000","last_fullsized":"2024-09-17T05:20:43.388852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:52.120543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"68'2590","reported_seq":2699,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.324869+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T05:20:43.324869+0000","last_peered":"2024-09-17T05:20:43.324869+0000","last_clean":"2024-09-17T05:20:43.324869+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T05:20:43.324869+0000","last_undegraded":"2024-09-17T05:20:43.324869+0000","last_fullsized":"2024-09-17T05:20:43.324869+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:56.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14722,"num_write_kb":243326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'4461","reported_seq":4579,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364760+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T05:20:36.364760+0000","last_peered":"2024-09-17T05:20:36.364760+0000","last_clean":"2024-09-17T05:20:36.364760+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T05:20:36.364760+0000","last_undegraded":"2024-09-17T05:20:36.364760+0000","last_fullsized":"2024-09-17T05:20:36.364760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4461,"log_dups_size":0,"ondisk_log_size":4461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24667,"num_write_kb":332445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"65'3","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364408+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T05:20:36.364408+0000","last_peered":"2024-09-17T05:20:36.364408+0000","last_clean":"2024-09-17T05:20:36.364408+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T05:20:36.364408+0000","last_undegraded":"2024-09-17T05:20:36.364408+0000","last_fullsized":"2024-09-17T05:20:36.364408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"68'367","reported_seq":579,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:39.638931+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T05:20:39.638931+0000","last_peered":"2024-09-17T05:20:39.638931+0000","last_clean":"2024-09-17T05:20:39.638931+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T05:20:39.638931+0000","last_undegraded":"2024-09-17T05:20:39.638931+0000","last_fullsized":"2024-09-17T05:20:39.638931+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":180,"num_read_kb":82080,"num_write":364,"num_write_kb":4295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"68'3856","reported_seq":3961,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403743+0000","last_change":"2024-09-17T04:50:12.067788+0000","last_active":"2024-09-17T05:20:44.403743+0000","last_peered":"2024-09-17T05:20:44.403743+0000","last_clean":"2024-09-17T05:20:44.403743+0000","last_became_active":"2024-09-17T04:50:12.066741+0000","last_became_peered":"2024-09-17T04:50:12.066741+0000","last_unstale":"2024-09-17T05:20:44.403743+0000","last_undegraded":"2024-09-17T05:20:44.403743+0000","last_fullsized":"2024-09-17T05:20:44.403743+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3856,"log_dups_size":0,"ondisk_log_size":3856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":20645,"num_write_kb":344954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"65'3521","reported_seq":3639,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.344504+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T05:20:36.344504+0000","last_peered":"2024-09-17T05:20:36.344504+0000","last_clean":"2024-09-17T05:20:36.344504+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T05:20:36.344504+0000","last_undegraded":"2024-09-17T05:20:36.344504+0000","last_fullsized":"2024-09-17T05:20:36.344504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3521,"log_dups_size":0,"ondisk_log_size":3521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19198,"num_write_kb":253218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"67'10","reported_seq":172,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.358352+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T05:20:36.358352+0000","last_peered":"2024-09-17T05:20:36.358352+0000","last_clean":"2024-09-17T05:20:36.358352+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T05:20:36.358352+0000","last_undegraded":"2024-09-17T05:20:36.358352+0000","last_fullsized":"2024-09-17T05:20:36.358352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":168,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"68'30","reported_seq":181,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:45.378073+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T05:20:45.378073+0000","last_peered":"2024-09-17T05:20:45.378073+0000","last_clean":"2024-09-17T05:20:45.378073+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T05:20:45.378073+0000","last_undegraded":"2024-09-17T05:20:45.378073+0000","last_fullsized":"2024-09-17T05:20:45.378073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:07.945355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":31,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"68'3085","reported_seq":3195,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.412893+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T05:20:43.412893+0000","last_peered":"2024-09-17T05:20:43.412893+0000","last_clean":"2024-09-17T05:20:43.412893+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T05:20:43.412893+0000","last_undegraded":"2024-09-17T05:20:43.412893+0000","last_fullsized":"2024-09-17T05:20:43.412893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16824,"num_write_kb":288541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"65'3611","reported_seq":3729,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.556980+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T05:20:36.556980+0000","last_peered":"2024-09-17T05:20:36.556980+0000","last_clean":"2024-09-17T05:20:36.556980+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T05:20:36.556980+0000","last_undegraded":"2024-09-17T05:20:36.556980+0000","last_fullsized":"2024-09-17T05:20:36.556980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3611,"log_dups_size":0,"ondisk_log_size":3611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19975,"num_write_kb":264442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"68'349","reported_seq":575,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.121847+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T05:20:44.121847+0000","last_peered":"2024-09-17T05:20:44.121847+0000","last_clean":"2024-09-17T05:20:44.121847+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T05:20:44.121847+0000","last_undegraded":"2024-09-17T05:20:44.121847+0000","last_fullsized":"2024-09-17T05:20:44.121847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":190,"num_read_kb":82094,"num_write":345,"num_write_kb":4278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"68'3","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.396510+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T05:20:44.396510+0000","last_peered":"2024-09-17T05:20:44.396510+0000","last_clean":"2024-09-17T05:20:44.396510+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T05:20:44.396510+0000","last_undegraded":"2024-09-17T05:20:44.396510+0000","last_fullsized":"2024-09-17T05:20:44.396510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:16.447835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"68'2950","reported_seq":3061,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.342183+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T05:20:43.342183+0000","last_peered":"2024-09-17T05:20:43.342183+0000","last_clean":"2024-09-17T05:20:43.342183+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T05:20:43.342183+0000","last_undegraded":"2024-09-17T05:20:43.342183+0000","last_fullsized":"2024-09-17T05:20:43.342183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":16110,"num_write_kb":249022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"65'4529","reported_seq":4647,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.361047+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T05:20:36.361047+0000","last_peered":"2024-09-17T05:20:36.361047+0000","last_clean":"2024-09-17T05:20:36.361047+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T05:20:36.361047+0000","last_undegraded":"2024-09-17T05:20:36.361047+0000","last_fullsized":"2024-09-17T05:20:36.361047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23581,"num_write_kb":327974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"68'31","reported_seq":214,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.399687+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T05:20:44.399687+0000","last_peered":"2024-09-17T05:20:44.399687+0000","last_clean":"2024-09-17T05:20:44.399687+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T05:20:44.399687+0000","last_undegraded":"2024-09-17T05:20:44.399687+0000","last_fullsized":"2024-09-17T05:20:44.399687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'8","reported_seq":160,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.415231+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T05:20:43.415231+0000","last_peered":"2024-09-17T05:20:43.415231+0000","last_clean":"2024-09-17T05:20:43.415231+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T05:20:43.415231+0000","last_undegraded":"2024-09-17T05:20:43.415231+0000","last_fullsized":"2024-09-17T05:20:43.415231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"68'2811","reported_seq":2924,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.378555+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T05:20:43.378555+0000","last_peered":"2024-09-17T05:20:43.378555+0000","last_clean":"2024-09-17T05:20:43.378555+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T05:20:43.378555+0000","last_undegraded":"2024-09-17T05:20:43.378555+0000","last_fullsized":"2024-09-17T05:20:43.378555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2811,"log_dups_size":0,"ondisk_log_size":2811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15712,"num_write_kb":243392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"65'4109","reported_seq":4226,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.353428+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T05:20:36.353428+0000","last_peered":"2024-09-17T05:20:36.353428+0000","last_clean":"2024-09-17T05:20:36.353428+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T05:20:36.353428+0000","last_undegraded":"2024-09-17T05:20:36.353428+0000","last_fullsized":"2024-09-17T05:20:36.353428+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21556,"num_write_kb":307833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"65'3","reported_seq":122,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558714+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T05:20:36.558714+0000","last_peered":"2024-09-17T05:20:36.558714+0000","last_clean":"2024-09-17T05:20:36.558714+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T05:20:36.558714+0000","last_undegraded":"2024-09-17T05:20:36.558714+0000","last_fullsized":"2024-09-17T05:20:36.558714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'15","reported_seq":209,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410016+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T05:20:43.410016+0000","last_peered":"2024-09-17T05:20:43.410016+0000","last_clean":"2024-09-17T05:20:43.410016+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T05:20:43.410016+0000","last_undegraded":"2024-09-17T05:20:43.410016+0000","last_fullsized":"2024-09-17T05:20:43.410016+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:40.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":246,"num_read_kb":246,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"68'2542","reported_seq":2652,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:42.865170+0000","last_change":"2024-09-17T04:49:41.040278+0000","last_active":"2024-09-17T05:20:42.865170+0000","last_peered":"2024-09-17T05:20:42.865170+0000","last_clean":"2024-09-17T05:20:42.865170+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T05:20:42.865170+0000","last_undegraded":"2024-09-17T05:20:42.865170+0000","last_fullsized":"2024-09-17T05:20:42.865170+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2542,"log_dups_size":0,"ondisk_log_size":2542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.895223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13327,"num_write_kb":225533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"65'3736","reported_seq":3854,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.365811+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T05:20:36.365811+0000","last_peered":"2024-09-17T05:20:36.365811+0000","last_clean":"2024-09-17T05:20:36.365811+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T05:20:36.365811+0000","last_undegraded":"2024-09-17T05:20:36.365811+0000","last_fullsized":"2024-09-17T05:20:36.365811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20513,"num_write_kb":287089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"68'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.363961+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T05:20:44.363961+0000","last_peered":"2024-09-17T05:20:44.363961+0000","last_clean":"2024-09-17T05:20:44.363961+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T05:20:44.363961+0000","last_undegraded":"2024-09-17T05:20:44.363961+0000","last_fullsized":"2024-09-17T05:20:44.363961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"68'5","reported_seq":117,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.409527+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T05:20:43.409527+0000","last_peered":"2024-09-17T05:20:43.409527+0000","last_clean":"2024-09-17T05:20:43.409527+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T05:20:43.409527+0000","last_undegraded":"2024-09-17T05:20:43.409527+0000","last_fullsized":"2024-09-17T05:20:43.409527+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:15.845612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"68'4175","reported_seq":4291,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.426185+0000","last_change":"2024-09-17T04:49:41.050551+0000","last_active":"2024-09-17T05:20:44.426185+0000","last_peered":"2024-09-17T05:20:44.426185+0000","last_clean":"2024-09-17T05:20:44.426185+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T05:20:44.426185+0000","last_undegraded":"2024-09-17T05:20:44.426185+0000","last_fullsized":"2024-09-17T05:20:44.426185+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:11.088230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":40,"num_write":23028,"num_write_kb":362813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"68'25","reported_seq":139,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.384829+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T05:20:43.384829+0000","last_peered":"2024-09-17T05:20:43.384829+0000","last_clean":"2024-09-17T05:20:43.384829+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T05:20:43.384829+0000","last_undegraded":"2024-09-17T05:20:43.384829+0000","last_fullsized":"2024-09-17T05:20:43.384829+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:37.778756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"65'4339","reported_seq":4456,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.338640+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T05:20:36.338640+0000","last_peered":"2024-09-17T05:20:36.338640+0000","last_clean":"2024-09-17T05:20:36.338640+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T05:20:36.338640+0000","last_undegraded":"2024-09-17T05:20:36.338640+0000","last_fullsized":"2024-09-17T05:20:36.338640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23134,"num_write_kb":338604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"65'2","reported_seq":121,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366244+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T05:20:36.366244+0000","last_peered":"2024-09-17T05:20:36.366244+0000","last_clean":"2024-09-17T05:20:36.366244+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T05:20:36.366244+0000","last_undegraded":"2024-09-17T05:20:36.366244+0000","last_fullsized":"2024-09-17T05:20:36.366244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"68'2877","reported_seq":2989,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.395024+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T05:20:43.395024+0000","last_peered":"2024-09-17T05:20:43.395024+0000","last_clean":"2024-09-17T05:20:43.395024+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T05:20:43.395024+0000","last_undegraded":"2024-09-17T05:20:43.395024+0000","last_fullsized":"2024-09-17T05:20:43.395024+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2877,"log_dups_size":0,"ondisk_log_size":2877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15865,"num_write_kb":272557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"68'1000","reported_seq":1162,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.429369+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T05:20:44.429369+0000","last_peered":"2024-09-17T05:20:44.429369+0000","last_clean":"2024-09-17T05:20:44.429369+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T05:20:44.429369+0000","last_undegraded":"2024-09-17T05:20:44.429369+0000","last_fullsized":"2024-09-17T05:20:44.429369+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":9969,"num_write":977,"num_write_kb":4635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"65'4230","reported_seq":4347,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.349109+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T05:20:36.349109+0000","last_peered":"2024-09-17T05:20:36.349109+0000","last_clean":"2024-09-17T05:20:36.349109+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T05:20:36.349109+0000","last_undegraded":"2024-09-17T05:20:36.349109+0000","last_fullsized":"2024-09-17T05:20:36.349109+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22609,"num_write_kb":319990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"65'3","reported_seq":122,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339504+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T05:20:36.339504+0000","last_peered":"2024-09-17T05:20:36.339504+0000","last_clean":"2024-09-17T05:20:36.339504+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T05:20:36.339504+0000","last_undegraded":"2024-09-17T05:20:36.339504+0000","last_fullsized":"2024-09-17T05:20:36.339504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"68'3526","reported_seq":3630,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393790+0000","last_change":"2024-09-17T04:50:12.072014+0000","last_active":"2024-09-17T05:20:43.393790+0000","last_peered":"2024-09-17T05:20:43.393790+0000","last_clean":"2024-09-17T05:20:43.393790+0000","last_became_active":"2024-09-17T04:50:12.071053+0000","last_became_peered":"2024-09-17T04:50:12.071053+0000","last_unstale":"2024-09-17T05:20:43.393790+0000","last_undegraded":"2024-09-17T05:20:43.393790+0000","last_fullsized":"2024-09-17T05:20:43.393790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20183,"num_write_kb":298078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"68'284","reported_seq":409,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.123738+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T05:20:44.123738+0000","last_peered":"2024-09-17T05:20:44.123738+0000","last_clean":"2024-09-17T05:20:44.123738+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T05:20:44.123738+0000","last_undegraded":"2024-09-17T05:20:44.123738+0000","last_fullsized":"2024-09-17T05:20:44.123738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2099732,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":4077,"num_write":257,"num_write_kb":2207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'3509","reported_seq":3626,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339983+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T05:20:36.339983+0000","last_peered":"2024-09-17T05:20:36.339983+0000","last_clean":"2024-09-17T05:20:36.339983+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T05:20:36.339983+0000","last_undegraded":"2024-09-17T05:20:36.339983+0000","last_fullsized":"2024-09-17T05:20:36.339983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3509,"log_dups_size":0,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18360,"num_write_kb":249727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'5","reported_seq":156,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:39.659577+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T05:20:39.659577+0000","last_peered":"2024-09-17T05:20:39.659577+0000","last_clean":"2024-09-17T05:20:39.659577+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T05:20:39.659577+0000","last_undegraded":"2024-09-17T05:20:39.659577+0000","last_fullsized":"2024-09-17T05:20:39.659577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:27.230604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"68'3113","reported_seq":3225,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.330899+0000","last_change":"2024-09-17T04:49:41.055263+0000","last_active":"2024-09-17T05:20:43.330899+0000","last_peered":"2024-09-17T05:20:43.330899+0000","last_clean":"2024-09-17T05:20:43.330899+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T05:20:43.330899+0000","last_undegraded":"2024-09-17T05:20:43.330899+0000","last_fullsized":"2024-09-17T05:20:43.330899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":16978,"num_write_kb":256690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"65'657","reported_seq":800,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.349177+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T05:20:36.349177+0000","last_peered":"2024-09-17T05:20:36.349177+0000","last_clean":"2024-09-17T05:20:36.349177+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T05:20:36.349177+0000","last_undegraded":"2024-09-17T05:20:36.349177+0000","last_fullsized":"2024-09-17T05:20:36.349177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":79257,"num_write":647,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'3199","reported_seq":3317,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.343729+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T05:20:36.343729+0000","last_peered":"2024-09-17T05:20:36.343729+0000","last_clean":"2024-09-17T05:20:36.343729+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T05:20:36.343729+0000","last_undegraded":"2024-09-17T05:20:36.343729+0000","last_fullsized":"2024-09-17T05:20:36.343729+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16756,"num_write_kb":239128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"68'43","reported_seq":194,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.403635+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T05:20:43.403635+0000","last_peered":"2024-09-17T05:20:43.403635+0000","last_clean":"2024-09-17T05:20:43.403635+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T05:20:43.403635+0000","last_undegraded":"2024-09-17T05:20:43.403635+0000","last_fullsized":"2024-09-17T05:20:43.403635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:24.586834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"68'3014","reported_seq":3125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402775+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T05:20:43.402775+0000","last_peered":"2024-09-17T05:20:43.402775+0000","last_clean":"2024-09-17T05:20:43.402775+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T05:20:43.402775+0000","last_undegraded":"2024-09-17T05:20:43.402775+0000","last_fullsized":"2024-09-17T05:20:43.402775+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3014,"log_dups_size":0,"ondisk_log_size":3014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16762,"num_write_kb":251577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"65'3751","reported_seq":3871,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364101+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T05:20:36.364101+0000","last_peered":"2024-09-17T05:20:36.364101+0000","last_clean":"2024-09-17T05:20:36.364101+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T05:20:36.364101+0000","last_undegraded":"2024-09-17T05:20:36.364101+0000","last_fullsized":"2024-09-17T05:20:36.364101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20040,"num_write_kb":277001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"68'3862","reported_seq":3974,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.271131+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T05:20:43.271131+0000","last_peered":"2024-09-17T05:20:43.271131+0000","last_clean":"2024-09-17T05:20:43.271131+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T05:20:43.271131+0000","last_undegraded":"2024-09-17T05:20:43.271131+0000","last_fullsized":"2024-09-17T05:20:43.271131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20235,"num_write_kb":310497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"65'4193","reported_seq":4314,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.356835+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T05:20:36.356835+0000","last_peered":"2024-09-17T05:20:36.356835+0000","last_clean":"2024-09-17T05:20:36.356835+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T05:20:36.356835+0000","last_undegraded":"2024-09-17T05:20:36.356835+0000","last_fullsized":"2024-09-17T05:20:36.356835+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4193,"log_dups_size":0,"ondisk_log_size":4193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22222,"num_write_kb":306243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"68'3519","reported_seq":3632,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.398772+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T05:20:43.398772+0000","last_peered":"2024-09-17T05:20:43.398772+0000","last_clean":"2024-09-17T05:20:43.398772+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T05:20:43.398772+0000","last_undegraded":"2024-09-17T05:20:43.398772+0000","last_fullsized":"2024-09-17T05:20:43.398772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3519,"log_dups_size":0,"ondisk_log_size":3519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":19548,"num_write_kb":295624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"65'3872","reported_seq":3994,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.360995+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T05:20:36.360995+0000","last_peered":"2024-09-17T05:20:36.360995+0000","last_clean":"2024-09-17T05:20:36.360995+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T05:20:36.360995+0000","last_undegraded":"2024-09-17T05:20:36.360995+0000","last_fullsized":"2024-09-17T05:20:36.360995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3872,"log_dups_size":0,"ondisk_log_size":3872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20576,"num_write_kb":279241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"68'2938","reported_seq":3050,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.380635+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T05:20:43.380635+0000","last_peered":"2024-09-17T05:20:43.380635+0000","last_clean":"2024-09-17T05:20:43.380635+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T05:20:43.380635+0000","last_undegraded":"2024-09-17T05:20:43.380635+0000","last_fullsized":"2024-09-17T05:20:43.380635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2938,"log_dups_size":0,"ondisk_log_size":2938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":16647,"num_write_kb":271506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"65'3500","reported_seq":3620,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.348982+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T05:20:36.348982+0000","last_peered":"2024-09-17T05:20:36.348982+0000","last_clean":"2024-09-17T05:20:36.348982+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T05:20:36.348982+0000","last_undegraded":"2024-09-17T05:20:36.348982+0000","last_fullsized":"2024-09-17T05:20:36.348982+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18562,"num_write_kb":258704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"68'3339","reported_seq":3452,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.406315+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T05:20:43.406315+0000","last_peered":"2024-09-17T05:20:43.406315+0000","last_clean":"2024-09-17T05:20:43.406315+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T05:20:43.406315+0000","last_undegraded":"2024-09-17T05:20:43.406315+0000","last_fullsized":"2024-09-17T05:20:43.406315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3339,"log_dups_size":0,"ondisk_log_size":3339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17940,"num_write_kb":305481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"65'3819","reported_seq":3939,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341730+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T05:20:36.341730+0000","last_peered":"2024-09-17T05:20:36.341730+0000","last_clean":"2024-09-17T05:20:36.341730+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T05:20:36.341730+0000","last_undegraded":"2024-09-17T05:20:36.341730+0000","last_fullsized":"2024-09-17T05:20:36.341730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21007,"num_write_kb":273012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"68'3374","reported_seq":3492,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.366480+0000","last_change":"2024-09-17T04:50:12.076134+0000","last_active":"2024-09-17T05:20:43.366480+0000","last_peered":"2024-09-17T05:20:43.366480+0000","last_clean":"2024-09-17T05:20:43.366480+0000","last_became_active":"2024-09-17T04:50:12.075350+0000","last_became_peered":"2024-09-17T04:50:12.075350+0000","last_unstale":"2024-09-17T05:20:43.366480+0000","last_undegraded":"2024-09-17T05:20:43.366480+0000","last_fullsized":"2024-09-17T05:20:43.366480+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3374,"log_dups_size":0,"ondisk_log_size":3374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18297,"num_write_kb":270032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"65'4233","reported_seq":4352,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341464+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T05:20:36.341464+0000","last_peered":"2024-09-17T05:20:36.341464+0000","last_clean":"2024-09-17T05:20:36.341464+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T05:20:36.341464+0000","last_undegraded":"2024-09-17T05:20:36.341464+0000","last_fullsized":"2024-09-17T05:20:36.341464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23577,"num_write_kb":311694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"68'3382","reported_seq":3494,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.416880+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T05:20:43.416880+0000","last_peered":"2024-09-17T05:20:43.416880+0000","last_clean":"2024-09-17T05:20:43.416880+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T05:20:43.416880+0000","last_undegraded":"2024-09-17T05:20:43.416880+0000","last_fullsized":"2024-09-17T05:20:43.416880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18178,"num_write_kb":293977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"65'3778","reported_seq":3897,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364369+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T05:20:36.364369+0000","last_peered":"2024-09-17T05:20:36.364369+0000","last_clean":"2024-09-17T05:20:36.364369+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T05:20:36.364369+0000","last_undegraded":"2024-09-17T05:20:36.364369+0000","last_fullsized":"2024-09-17T05:20:36.364369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21044,"num_write_kb":281017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"68'2956","reported_seq":3071,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404554+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T05:20:44.404554+0000","last_peered":"2024-09-17T05:20:44.404554+0000","last_clean":"2024-09-17T05:20:44.404554+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T05:20:44.404554+0000","last_undegraded":"2024-09-17T05:20:44.404554+0000","last_fullsized":"2024-09-17T05:20:44.404554+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2956,"log_dups_size":0,"ondisk_log_size":2956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15875,"num_write_kb":264255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"65'4647","reported_seq":4767,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352380+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T05:20:36.352380+0000","last_peered":"2024-09-17T05:20:36.352380+0000","last_clean":"2024-09-17T05:20:36.352380+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T05:20:36.352380+0000","last_undegraded":"2024-09-17T05:20:36.352380+0000","last_fullsized":"2024-09-17T05:20:36.352380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24699,"num_write_kb":336693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"68'3095","reported_seq":3210,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.396708+0000","last_change":"2024-09-17T04:49:41.045502+0000","last_active":"2024-09-17T05:20:43.396708+0000","last_peered":"2024-09-17T05:20:43.396708+0000","last_clean":"2024-09-17T05:20:43.396708+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T05:20:43.396708+0000","last_undegraded":"2024-09-17T05:20:43.396708+0000","last_fullsized":"2024-09-17T05:20:43.396708+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:19.746356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":36,"num_write":16750,"num_write_kb":272041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"65'4070","reported_seq":4190,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.361587+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T05:20:36.361587+0000","last_peered":"2024-09-17T05:20:36.361587+0000","last_clean":"2024-09-17T05:20:36.361587+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T05:20:36.361587+0000","last_undegraded":"2024-09-17T05:20:36.361587+0000","last_fullsized":"2024-09-17T05:20:36.361587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4070,"log_dups_size":0,"ondisk_log_size":4070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21906,"num_write_kb":288644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"68'4276","reported_seq":4387,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404966+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T05:20:44.404966+0000","last_peered":"2024-09-17T05:20:44.404966+0000","last_clean":"2024-09-17T05:20:44.404966+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T05:20:44.404966+0000","last_undegraded":"2024-09-17T05:20:44.404966+0000","last_fullsized":"2024-09-17T05:20:44.404966+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22732,"num_write_kb":365385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"65'4174","reported_seq":4292,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.348275+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T05:20:36.348275+0000","last_peered":"2024-09-17T05:20:36.348275+0000","last_clean":"2024-09-17T05:20:36.348275+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T05:20:36.348275+0000","last_undegraded":"2024-09-17T05:20:36.348275+0000","last_fullsized":"2024-09-17T05:20:36.348275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22870,"num_write_kb":296621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"68'2821","reported_seq":2931,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393189+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T05:20:43.393189+0000","last_peered":"2024-09-17T05:20:43.393189+0000","last_clean":"2024-09-17T05:20:43.393189+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T05:20:43.393189+0000","last_undegraded":"2024-09-17T05:20:43.393189+0000","last_fullsized":"2024-09-17T05:20:43.393189+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2821,"log_dups_size":0,"ondisk_log_size":2821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15110,"num_write_kb":232606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"65'4280","reported_seq":4398,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.343678+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T05:20:36.343678+0000","last_peered":"2024-09-17T05:20:36.343678+0000","last_clean":"2024-09-17T05:20:36.343678+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T05:20:36.343678+0000","last_undegraded":"2024-09-17T05:20:36.343678+0000","last_fullsized":"2024-09-17T05:20:36.343678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4280,"log_dups_size":0,"ondisk_log_size":4280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23007,"num_write_kb":313406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"68'3452","reported_seq":3562,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.303318+0000","last_change":"2024-09-17T04:49:41.038265+0000","last_active":"2024-09-17T05:20:43.303318+0000","last_peered":"2024-09-17T05:20:43.303318+0000","last_clean":"2024-09-17T05:20:43.303318+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T05:20:43.303318+0000","last_undegraded":"2024-09-17T05:20:43.303318+0000","last_fullsized":"2024-09-17T05:20:43.303318+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3452,"log_dups_size":0,"ondisk_log_size":3452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:41.638403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19160,"num_write_kb":306156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"65'4384","reported_seq":4504,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339211+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T05:20:36.339211+0000","last_peered":"2024-09-17T05:20:36.339211+0000","last_clean":"2024-09-17T05:20:36.339211+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T05:20:36.339211+0000","last_undegraded":"2024-09-17T05:20:36.339211+0000","last_fullsized":"2024-09-17T05:20:36.339211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23582,"num_write_kb":336581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"68'3472","reported_seq":3589,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.255838+0000","last_change":"2024-09-17T04:50:12.068348+0000","last_active":"2024-09-17T05:20:43.255838+0000","last_peered":"2024-09-17T05:20:43.255838+0000","last_clean":"2024-09-17T05:20:43.255838+0000","last_became_active":"2024-09-17T04:50:12.067942+0000","last_became_peered":"2024-09-17T04:50:12.067942+0000","last_unstale":"2024-09-17T05:20:43.255838+0000","last_undegraded":"2024-09-17T05:20:43.255838+0000","last_fullsized":"2024-09-17T05:20:43.255838+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3472,"log_dups_size":0,"ondisk_log_size":3472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:33.954020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18777,"num_write_kb":292458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"65'3688","reported_seq":3806,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557618+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T05:20:36.557618+0000","last_peered":"2024-09-17T05:20:36.557618+0000","last_clean":"2024-09-17T05:20:36.557618+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T05:20:36.557618+0000","last_undegraded":"2024-09-17T05:20:36.557618+0000","last_fullsized":"2024-09-17T05:20:36.557618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20358,"num_write_kb":257774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"68'3214","reported_seq":3329,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.416523+0000","last_change":"2024-09-17T04:49:41.038072+0000","last_active":"2024-09-17T05:20:43.416523+0000","last_peered":"2024-09-17T05:20:43.416523+0000","last_clean":"2024-09-17T05:20:43.416523+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T05:20:43.416523+0000","last_undegraded":"2024-09-17T05:20:43.416523+0000","last_fullsized":"2024-09-17T05:20:43.416523+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":17543,"num_write_kb":304029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"65'4068","reported_seq":4186,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374459+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T05:20:36.374459+0000","last_peered":"2024-09-17T05:20:36.374459+0000","last_clean":"2024-09-17T05:20:36.374459+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T05:20:36.374459+0000","last_undegraded":"2024-09-17T05:20:36.374459+0000","last_fullsized":"2024-09-17T05:20:36.374459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21759,"num_write_kb":308395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"68'2551","reported_seq":2663,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402052+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T05:20:43.402052+0000","last_peered":"2024-09-17T05:20:43.402052+0000","last_clean":"2024-09-17T05:20:43.402052+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T05:20:43.402052+0000","last_undegraded":"2024-09-17T05:20:43.402052+0000","last_fullsized":"2024-09-17T05:20:43.402052+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2551,"log_dups_size":0,"ondisk_log_size":2551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13664,"num_write_kb":247232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"65'4369","reported_seq":4488,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.359366+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T05:20:36.359366+0000","last_peered":"2024-09-17T05:20:36.359366+0000","last_clean":"2024-09-17T05:20:36.359366+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T05:20:36.359366+0000","last_undegraded":"2024-09-17T05:20:36.359366+0000","last_fullsized":"2024-09-17T05:20:36.359366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23171,"num_write_kb":318257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"68'3277","reported_seq":3388,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393579+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T05:20:43.393579+0000","last_peered":"2024-09-17T05:20:43.393579+0000","last_clean":"2024-09-17T05:20:43.393579+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T05:20:43.393579+0000","last_undegraded":"2024-09-17T05:20:43.393579+0000","last_fullsized":"2024-09-17T05:20:43.393579+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18007,"num_write_kb":264733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"65'5432","reported_seq":5550,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366356+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T05:20:36.366356+0000","last_peered":"2024-09-17T05:20:36.366356+0000","last_clean":"2024-09-17T05:20:36.366356+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T05:20:36.366356+0000","last_undegraded":"2024-09-17T05:20:36.366356+0000","last_fullsized":"2024-09-17T05:20:36.366356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29730,"num_write_kb":389609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"68'3135","reported_seq":3246,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.377491+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T05:20:43.377491+0000","last_peered":"2024-09-17T05:20:43.377491+0000","last_clean":"2024-09-17T05:20:43.377491+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T05:20:43.377491+0000","last_undegraded":"2024-09-17T05:20:43.377491+0000","last_fullsized":"2024-09-17T05:20:43.377491+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":16884,"num_write_kb":284681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"65'3941","reported_seq":4059,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.360280+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T05:20:36.360280+0000","last_peered":"2024-09-17T05:20:36.360280+0000","last_clean":"2024-09-17T05:20:36.360280+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T05:20:36.360280+0000","last_undegraded":"2024-09-17T05:20:36.360280+0000","last_fullsized":"2024-09-17T05:20:36.360280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21845,"num_write_kb":304681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"68'3913","reported_seq":4026,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.330331+0000","last_change":"2024-09-17T04:49:41.067207+0000","last_active":"2024-09-17T05:20:43.330331+0000","last_peered":"2024-09-17T05:20:43.330331+0000","last_clean":"2024-09-17T05:20:43.330331+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T05:20:43.330331+0000","last_undegraded":"2024-09-17T05:20:43.330331+0000","last_fullsized":"2024-09-17T05:20:43.330331+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3913,"log_dups_size":0,"ondisk_log_size":3913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:08.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":21530,"num_write_kb":364884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"65'3351","reported_seq":3475,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.347528+0000","last_change":"2024-09-17T04:50:12.469782+0000","last_active":"2024-09-17T05:20:36.347528+0000","last_peered":"2024-09-17T05:20:36.347528+0000","last_clean":"2024-09-17T05:20:36.347528+0000","last_became_active":"2024-09-17T04:50:12.469425+0000","last_became_peered":"2024-09-17T04:50:12.469425+0000","last_unstale":"2024-09-17T05:20:36.347528+0000","last_undegraded":"2024-09-17T05:20:36.347528+0000","last_fullsized":"2024-09-17T05:20:36.347528+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:04.098943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18172,"num_write_kb":265354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"68'1954","reported_seq":2066,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.346900+0000","last_change":"2024-09-17T04:49:41.046170+0000","last_active":"2024-09-17T05:20:43.346900+0000","last_peered":"2024-09-17T05:20:43.346900+0000","last_clean":"2024-09-17T05:20:43.346900+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T05:20:43.346900+0000","last_undegraded":"2024-09-17T05:20:43.346900+0000","last_fullsized":"2024-09-17T05:20:43.346900+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":1954,"log_dups_size":0,"ondisk_log_size":1954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:01.683826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":34,"num_write":10478,"num_write_kb":181241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"65'3661","reported_seq":3779,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.344111+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T05:20:36.344111+0000","last_peered":"2024-09-17T05:20:36.344111+0000","last_clean":"2024-09-17T05:20:36.344111+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T05:20:36.344111+0000","last_undegraded":"2024-09-17T05:20:36.344111+0000","last_fullsized":"2024-09-17T05:20:36.344111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19087,"num_write_kb":273280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"68'2981","reported_seq":3093,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.203029+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T05:20:43.203029+0000","last_peered":"2024-09-17T05:20:43.203029+0000","last_clean":"2024-09-17T05:20:43.203029+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T05:20:43.203029+0000","last_undegraded":"2024-09-17T05:20:43.203029+0000","last_fullsized":"2024-09-17T05:20:43.203029+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:25.574682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":15573,"num_write_kb":271248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"65'3297","reported_seq":3414,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340729+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T05:20:36.340729+0000","last_peered":"2024-09-17T05:20:36.340729+0000","last_clean":"2024-09-17T05:20:36.340729+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T05:20:36.340729+0000","last_undegraded":"2024-09-17T05:20:36.340729+0000","last_fullsized":"2024-09-17T05:20:36.340729+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3297,"log_dups_size":0,"ondisk_log_size":3297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17851,"num_write_kb":243868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"68'2872","reported_seq":2983,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403678+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T05:20:44.403678+0000","last_peered":"2024-09-17T05:20:44.403678+0000","last_clean":"2024-09-17T05:20:44.403678+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T05:20:44.403678+0000","last_undegraded":"2024-09-17T05:20:44.403678+0000","last_fullsized":"2024-09-17T05:20:44.403678+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2872,"log_dups_size":0,"ondisk_log_size":2872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15784,"num_write_kb":259984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"65'2574","reported_seq":2691,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.556549+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T05:20:36.556549+0000","last_peered":"2024-09-17T05:20:36.556549+0000","last_clean":"2024-09-17T05:20:36.556549+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T05:20:36.556549+0000","last_undegraded":"2024-09-17T05:20:36.556549+0000","last_fullsized":"2024-09-17T05:20:36.556549+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":2574,"log_dups_size":0,"ondisk_log_size":2574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14007,"num_write_kb":203729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"68'2459","reported_seq":2572,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393891+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T05:20:43.393891+0000","last_peered":"2024-09-17T05:20:43.393891+0000","last_clean":"2024-09-17T05:20:43.393891+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T05:20:43.393891+0000","last_undegraded":"2024-09-17T05:20:43.393891+0000","last_fullsized":"2024-09-17T05:20:43.393891+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2459,"log_dups_size":0,"ondisk_log_size":2459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13572,"num_write_kb":222508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"65'3245","reported_seq":3362,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.556406+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T05:20:36.556406+0000","last_peered":"2024-09-17T05:20:36.556406+0000","last_clean":"2024-09-17T05:20:36.556406+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T05:20:36.556406+0000","last_undegraded":"2024-09-17T05:20:36.556406+0000","last_fullsized":"2024-09-17T05:20:36.556406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3245,"log_dups_size":0,"ondisk_log_size":3245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16903,"num_write_kb":256080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"68'3074","reported_seq":3184,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393584+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T05:20:43.393584+0000","last_peered":"2024-09-17T05:20:43.393584+0000","last_clean":"2024-09-17T05:20:43.393584+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T05:20:43.393584+0000","last_undegraded":"2024-09-17T05:20:43.393584+0000","last_fullsized":"2024-09-17T05:20:43.393584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16752,"num_write_kb":271957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"65'4387","reported_seq":4504,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557565+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T05:20:36.557565+0000","last_peered":"2024-09-17T05:20:36.557565+0000","last_clean":"2024-09-17T05:20:36.557565+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T05:20:36.557565+0000","last_undegraded":"2024-09-17T05:20:36.557565+0000","last_fullsized":"2024-09-17T05:20:36.557565+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24001,"num_write_kb":317505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"68'2713","reported_seq":2824,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.417489+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T05:20:43.417489+0000","last_peered":"2024-09-17T05:20:43.417489+0000","last_clean":"2024-09-17T05:20:43.417489+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T05:20:43.417489+0000","last_undegraded":"2024-09-17T05:20:43.417489+0000","last_fullsized":"2024-09-17T05:20:43.417489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2713,"log_dups_size":0,"ondisk_log_size":2713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14521,"num_write_kb":262890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"65'4656","reported_seq":4775,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.347514+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T05:20:36.347514+0000","last_peered":"2024-09-17T05:20:36.347514+0000","last_clean":"2024-09-17T05:20:36.347514+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T05:20:36.347514+0000","last_undegraded":"2024-09-17T05:20:36.347514+0000","last_fullsized":"2024-09-17T05:20:36.347514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25090,"num_write_kb":362900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"68'3588","reported_seq":3702,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.402501+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T05:20:44.402501+0000","last_peered":"2024-09-17T05:20:44.402501+0000","last_clean":"2024-09-17T05:20:44.402501+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T05:20:44.402501+0000","last_undegraded":"2024-09-17T05:20:44.402501+0000","last_fullsized":"2024-09-17T05:20:44.402501+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3588,"log_dups_size":0,"ondisk_log_size":3588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":19474,"num_write_kb":319707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"65'4149","reported_seq":4266,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.347526+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T05:20:36.347526+0000","last_peered":"2024-09-17T05:20:36.347526+0000","last_clean":"2024-09-17T05:20:36.347526+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T05:20:36.347526+0000","last_undegraded":"2024-09-17T05:20:36.347526+0000","last_fullsized":"2024-09-17T05:20:36.347526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22453,"num_write_kb":298991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"68'3299","reported_seq":3415,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.422884+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T05:20:44.422884+0000","last_peered":"2024-09-17T05:20:44.422884+0000","last_clean":"2024-09-17T05:20:44.422884+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T05:20:44.422884+0000","last_undegraded":"2024-09-17T05:20:44.422884+0000","last_fullsized":"2024-09-17T05:20:44.422884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3299,"log_dups_size":0,"ondisk_log_size":3299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":17863,"num_write_kb":288721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"65'3910","reported_seq":4027,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.555930+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T05:20:36.555930+0000","last_peered":"2024-09-17T05:20:36.555930+0000","last_clean":"2024-09-17T05:20:36.555930+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T05:20:36.555930+0000","last_undegraded":"2024-09-17T05:20:36.555930+0000","last_fullsized":"2024-09-17T05:20:36.555930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21371,"num_write_kb":294687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"68'2856","reported_seq":2969,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.325969+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T05:20:43.325969+0000","last_peered":"2024-09-17T05:20:43.325969+0000","last_clean":"2024-09-17T05:20:43.325969+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T05:20:43.325969+0000","last_undegraded":"2024-09-17T05:20:43.325969+0000","last_fullsized":"2024-09-17T05:20:43.325969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15916,"num_write_kb":235941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"65'3148","reported_seq":3265,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.373270+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T05:20:36.373270+0000","last_peered":"2024-09-17T05:20:36.373270+0000","last_clean":"2024-09-17T05:20:36.373270+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T05:20:36.373270+0000","last_undegraded":"2024-09-17T05:20:36.373270+0000","last_fullsized":"2024-09-17T05:20:36.373270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16748,"num_write_kb":239238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"68'3516","reported_seq":3630,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.409318+0000","last_change":"2024-09-17T04:49:41.041201+0000","last_active":"2024-09-17T05:20:43.409318+0000","last_peered":"2024-09-17T05:20:43.409318+0000","last_clean":"2024-09-17T05:20:43.409318+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T05:20:43.409318+0000","last_undegraded":"2024-09-17T05:20:43.409318+0000","last_fullsized":"2024-09-17T05:20:43.409318+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:02.588786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18785,"num_write_kb":280186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"65'3294","reported_seq":3411,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558447+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T05:20:36.558447+0000","last_peered":"2024-09-17T05:20:36.558447+0000","last_clean":"2024-09-17T05:20:36.558447+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T05:20:36.558447+0000","last_undegraded":"2024-09-17T05:20:36.558447+0000","last_fullsized":"2024-09-17T05:20:36.558447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16934,"num_write_kb":260475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"68'2875","reported_seq":2985,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.359568+0000","last_change":"2024-09-17T04:49:41.061092+0000","last_active":"2024-09-17T05:20:43.359568+0000","last_peered":"2024-09-17T05:20:43.359568+0000","last_clean":"2024-09-17T05:20:43.359568+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T05:20:43.359568+0000","last_undegraded":"2024-09-17T05:20:43.359568+0000","last_fullsized":"2024-09-17T05:20:43.359568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2875,"log_dups_size":0,"ondisk_log_size":2875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:52.714614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15268,"num_write_kb":241386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"65'3071","reported_seq":3192,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.370583+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T05:20:36.370583+0000","last_peered":"2024-09-17T05:20:36.370583+0000","last_clean":"2024-09-17T05:20:36.370583+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T05:20:36.370583+0000","last_undegraded":"2024-09-17T05:20:36.370583+0000","last_fullsized":"2024-09-17T05:20:36.370583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3071,"log_dups_size":0,"ondisk_log_size":3071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16593,"num_write_kb":218157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"68'2617","reported_seq":2728,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.405858+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T05:20:43.405858+0000","last_peered":"2024-09-17T05:20:43.405858+0000","last_clean":"2024-09-17T05:20:43.405858+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T05:20:43.405858+0000","last_undegraded":"2024-09-17T05:20:43.405858+0000","last_fullsized":"2024-09-17T05:20:43.405858+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3,"num_write":13428,"num_write_kb":250535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"65'3866","reported_seq":3987,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352182+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T05:20:36.352182+0000","last_peered":"2024-09-17T05:20:36.352182+0000","last_clean":"2024-09-17T05:20:36.352182+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T05:20:36.352182+0000","last_undegraded":"2024-09-17T05:20:36.352182+0000","last_fullsized":"2024-09-17T05:20:36.352182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3866,"log_dups_size":0,"ondisk_log_size":3866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20832,"num_write_kb":272078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"68'3056","reported_seq":3165,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.326456+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T05:20:43.326456+0000","last_peered":"2024-09-17T05:20:43.326456+0000","last_clean":"2024-09-17T05:20:43.326456+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T05:20:43.326456+0000","last_undegraded":"2024-09-17T05:20:43.326456+0000","last_fullsized":"2024-09-17T05:20:43.326456+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3056,"log_dups_size":0,"ondisk_log_size":3056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16712,"num_write_kb":275987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"65'4485","reported_seq":4605,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.351407+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T05:20:36.351407+0000","last_peered":"2024-09-17T05:20:36.351407+0000","last_clean":"2024-09-17T05:20:36.351407+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T05:20:36.351407+0000","last_undegraded":"2024-09-17T05:20:36.351407+0000","last_fullsized":"2024-09-17T05:20:36.351407+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23824,"num_write_kb":337279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"68'3229","reported_seq":3338,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.420569+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T05:20:44.420569+0000","last_peered":"2024-09-17T05:20:44.420569+0000","last_clean":"2024-09-17T05:20:44.420569+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T05:20:44.420569+0000","last_undegraded":"2024-09-17T05:20:44.420569+0000","last_fullsized":"2024-09-17T05:20:44.420569+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3229,"log_dups_size":0,"ondisk_log_size":3229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17855,"num_write_kb":268347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"65'3523","reported_seq":3644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559155+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T05:20:36.559155+0000","last_peered":"2024-09-17T05:20:36.559155+0000","last_clean":"2024-09-17T05:20:36.559155+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T05:20:36.559155+0000","last_undegraded":"2024-09-17T05:20:36.559155+0000","last_fullsized":"2024-09-17T05:20:36.559155+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3523,"log_dups_size":0,"ondisk_log_size":3523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19890,"num_write_kb":260121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":321,"num_write":1077532,"num_write_kb":17428719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":198824,"ondisk_log_size":198824,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":18310618,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1391,"num_read_kb":207657,"num_write":4849,"num_write_kb":19832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36683776,"data_stored":36621236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4439,"ondisk_log_size":4439,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1352682,"num_write_kb":18536494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250516,"ondisk_log_size":250516,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":18963064,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1535,"num_read_kb":230833,"num_write":4911,"num_write_kb":20583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37986304,"data_stored":37926128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4503,"ondisk_log_size":4503,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509823,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533064,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":532094,"kb_avail":93838776,"statfs":{"total":96636764160,"available":96090906624,"internally_reserved":0,"allocated":974848,"data_stored":552256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":544864714},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509823,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762124,"kb_used_data":17916,"kb_used_omap":1,"kb_used_meta":744190,"kb_avail":93609716,"statfs":{"total":96636764160,"available":95856349184,"internally_reserved":0,"allocated":18345984,"data_stored":17920368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":762051015},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509823,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756036,"kb_used_data":17268,"kb_used_omap":1,"kb_used_meta":738750,"kb_avail":93615804,"statfs":{"total":96636764160,"available":95862583296,"internally_reserved":0,"allocated":17682432,"data_stored":17246718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756480453},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509823,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755412,"kb_used_data":15300,"kb_used_omap":1,"kb_used_meta":740094,"kb_avail":93616428,"statfs":{"total":96636764160,"available":95863222272,"internally_reserved":0,"allocated":15667200,"data_stored":15236011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":757856711},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509823,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877392,"kb_used_data":14656,"kb_used_omap":1,"kb_used_meta":862718,"kb_avail":93494448,"statfs":{"total":96636764160,"available":95738314752,"internally_reserved":0,"allocated":15007744,"data_stored":14583014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":883423685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509824,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630324,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":628862,"kb_avail":93741516,"statfs":{"total":96636764160,"available":95991312384,"internally_reserved":0,"allocated":1478656,"data_stored":1057972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":643955141},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509823,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":819832,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":816894,"kb_avail":93552008,"statfs":{"total":96636764160,"available":95797256192,"internally_reserved":0,"allocated":2990080,"data_stored":2567342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":836499909},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542527,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694200,"kb_used_data":10472,"kb_used_omap":1,"kb_used_meta":683710,"kb_avail":93677640,"statfs":{"total":96636764160,"available":95925903360,"internally_reserved":0,"allocated":10723328,"data_stored":10304131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":700119494},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2099788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10493952,"data_stored":10488922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4279042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":84770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5726270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4279625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5652480,"data_stored":5642363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:20:58.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:20:58.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.500+0000 7fabc5c66640 1 -- 172.21.15.150:0/3770479139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c42e0 msgr2=0x7fabc00c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.500+0000 7fabc5c66640 1 --2- 172.21.15.150:0/3770479139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c42e0 0x7fabc00c6690 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009900 tx=0x7fabb001c850 comp rx=0 tx=0).stop 2024-09-17T05:20:58.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.500+0000 7fabc5c66640 1 -- 172.21.15.150:0/3770479139 shutdown_connections 2024-09-17T05:20:58.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.500+0000 7fabc5c66640 1 --2- 172.21.15.150:0/3770479139 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc0152cf0 0x7fabc0153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.500+0000 7fabc5c66640 1 --2- 172.21.15.150:0/3770479139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c42e0 0x7fabc00c6690 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.500+0000 7fabc5c66640 1 -- 172.21.15.150:0/3770479139 >> 172.21.15.150:0/3770479139 conn(0x7fabc00c4a00 msgr2=0x7fabc00c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:58.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.500+0000 7fabc5c66640 1 -- 172.21.15.150:0/3770479139 shutdown_connections 2024-09-17T05:20:58.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.500+0000 7fabc5c66640 1 -- 172.21.15.150:0/3770479139 wait complete. 2024-09-17T05:20:58.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.500+0000 7fabc5c66640 1 Processor -- start 2024-09-17T05:20:58.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.500+0000 7fabc5c66640 1 -- start start 2024-09-17T05:20:58.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabc5c66640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc00a7350 0x7fabc0154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:58.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabc5c66640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c42e0 0x7fabc00bd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:58.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabc5c66640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc0152cf0 0x7fabc01f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:58.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabc5c66640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fabc00c6d50 con 0x7fabc00c42e0 2024-09-17T05:20:58.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabc5c66640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fabc00c6bd0 con 0x7fabc00a7350 2024-09-17T05:20:58.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabc5c66640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fabc0153640 con 0x7fabc0152cf0 2024-09-17T05:20:58.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabbeffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c42e0 0x7fabc00bd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:58.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabbeffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c42e0 0x7fabc00bd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33206/0 (socket says 172.21.15.150:33206) 2024-09-17T05:20:58.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabbeffd640 1 -- 172.21.15.150:0/3455094819 learned_addr learned my addr 172.21.15.150:0/3455094819 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:58.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabbffff640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc0152cf0 0x7fabc01f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:58.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabbf7fe640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc00a7350 0x7fabc0154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:58.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabbeffd640 1 -- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc0152cf0 msgr2=0x7fabc01f0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabbeffd640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc0152cf0 0x7fabc01f0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabbeffd640 1 -- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc00a7350 msgr2=0x7fabc0154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabbeffd640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc00a7350 0x7fabc0154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabbeffd640 1 -- 172.21.15.150:0/3455094819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc01f1a30 con 0x7fabc00c42e0 2024-09-17T05:20:58.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabbeffd640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c42e0 0x7fabc00bd0f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fabb400b410 tx=0x7fabb400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:58.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7faba7fff640 1 -- 172.21.15.150:0/3455094819 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb4015020 con 0x7fabc00c42e0 2024-09-17T05:20:58.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7faba7fff640 1 -- 172.21.15.150:0/3455094819 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabb4011920 con 0x7fabc00c42e0 2024-09-17T05:20:58.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabc5c66640 1 -- 172.21.15.150:0/3455094819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc0154030 con 0x7fabc00c42e0 2024-09-17T05:20:58.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7faba7fff640 1 -- 172.21.15.150:0/3455094819 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb4011d70 con 0x7fabc00c42e0 2024-09-17T05:20:58.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabc5c66640 1 -- 172.21.15.150:0/3455094819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc00acc80 con 0x7fabc00c42e0 2024-09-17T05:20:58.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabc5c66640 1 -- 172.21.15.150:0/3455094819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc00c6690 con 0x7fabc00c42e0 2024-09-17T05:20:58.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7faba7fff640 1 -- 172.21.15.150:0/3455094819 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fabb4018020 con 0x7fabc00c42e0 2024-09-17T05:20:58.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7faba7fff640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabac073200 0x7fabac075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:58.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7faba7fff640 1 -- 172.21.15.150:0/3455094819 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7fabb4095300 con 0x7fabc00c42e0 2024-09-17T05:20:58.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.504+0000 7fabbf7fe640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabac073200 0x7fabac075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:58.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.508+0000 7fabbf7fe640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabac073200 0x7fabac075710 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fabb00af000 tx=0x7fabb001b0f0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:58.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.512+0000 7faba7fff640 1 -- 172.21.15.150:0/3455094819 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fabb401f020 con 0x7fabc00c42e0 2024-09-17T05:20:58.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.680+0000 7fabc5c66640 1 -- 172.21.15.150:0/3455094819 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fabc00a7130 con 0x7fabac073200 2024-09-17T05:20:58.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.700+0000 7faba7fff640 1 -- 172.21.15.150:0/3455094819 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493818 (secure 0 0 0) 0x7fabc00a7130 con 0x7fabac073200 2024-09-17T05:20:58.701 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:58.707 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T05:20:58.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.708+0000 7fabc5c66640 1 -- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabac073200 msgr2=0x7fabac075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.708+0000 7fabc5c66640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabac073200 0x7fabac075710 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fabb00af000 tx=0x7fabb001b0f0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.708+0000 7fabc5c66640 1 -- 172.21.15.150:0/3455094819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c42e0 msgr2=0x7fabc00bd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.708+0000 7fabc5c66640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c42e0 0x7fabc00bd0f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fabb400b410 tx=0x7fabb400b8d0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.708+0000 7fabc5c66640 1 -- 172.21.15.150:0/3455094819 shutdown_connections 2024-09-17T05:20:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.708+0000 7fabc5c66640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabac073200 0x7fabac075710 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.708+0000 7fabc5c66640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabc0152cf0 0x7fabc01f0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.708+0000 7fabc5c66640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc00c42e0 0x7fabc00bd0f0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.708+0000 7fabc5c66640 1 --2- 172.21.15.150:0/3455094819 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabc00a7350 0x7fabc0154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.708+0000 7fabc5c66640 1 -- 172.21.15.150:0/3455094819 >> 172.21.15.150:0/3455094819 conn(0x7fabc00c4a00 msgr2=0x7fabc00744e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.708+0000 7fabc5c66640 1 -- 172.21.15.150:0/3455094819 shutdown_connections 2024-09-17T05:20:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.708+0000 7fabc5c66640 1 -- 172.21.15.150:0/3455094819 wait complete. 2024-09-17T05:20:58.729 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":1010,"stamp":"2024-09-17T05:20:57.882067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37864050,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3107,"num_read_kb":438866,"num_write":2440075,"num_write_kb":36007256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":458354,"ondisk_log_size":458354,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5828384,"kb_used_data":80928,"kb_used_omap":12,"kb_used_meta":5747315,"kb_avail":749146336,"statfs":{"total":773094113280,"available":767125848064,"internally_reserved":0,"allocated":82870272,"data_stored":79467812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":5885251122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7484347531,"num_objects":-1955,"num_object_clones":0,"num_object_copies":-3910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1955,"num_whiteouts":0,"num_read":10,"num_read_kb":30,"num_write":2076,"num_write_kb":727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006566"},"pg_stats":[{"pgid":"5.18","version":"68'3479","reported_seq":3591,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.418967+0000","last_change":"2024-09-17T04:49:41.025271+0000","last_active":"2024-09-17T05:20:44.418967+0000","last_peered":"2024-09-17T05:20:44.418967+0000","last_clean":"2024-09-17T05:20:44.418967+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T05:20:44.418967+0000","last_undegraded":"2024-09-17T05:20:44.418967+0000","last_fullsized":"2024-09-17T05:20:44.418967+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3479,"log_dups_size":0,"ondisk_log_size":3479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:39.715602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":12,"num_write":18686,"num_write_kb":287786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"65'1001","reported_seq":1175,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340557+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T05:20:36.340557+0000","last_peered":"2024-09-17T05:20:36.340557+0000","last_clean":"2024-09-17T05:20:36.340557+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T05:20:36.340557+0000","last_undegraded":"2024-09-17T05:20:36.340557+0000","last_fullsized":"2024-09-17T05:20:36.340557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":20774,"num_write":978,"num_write_kb":4679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'4064","reported_seq":4184,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.351711+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T05:20:36.351711+0000","last_peered":"2024-09-17T05:20:36.351711+0000","last_clean":"2024-09-17T05:20:36.351711+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T05:20:36.351711+0000","last_undegraded":"2024-09-17T05:20:36.351711+0000","last_fullsized":"2024-09-17T05:20:36.351711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22217,"num_write_kb":281701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"68'3","reported_seq":114,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402753+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T05:20:43.402753+0000","last_peered":"2024-09-17T05:20:43.402753+0000","last_clean":"2024-09-17T05:20:43.402753+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T05:20:43.402753+0000","last_undegraded":"2024-09-17T05:20:43.402753+0000","last_fullsized":"2024-09-17T05:20:43.402753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:38.417840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"68'3721","reported_seq":3833,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.413711+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T05:20:43.413711+0000","last_peered":"2024-09-17T05:20:43.413711+0000","last_clean":"2024-09-17T05:20:43.413711+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T05:20:43.413711+0000","last_undegraded":"2024-09-17T05:20:43.413711+0000","last_fullsized":"2024-09-17T05:20:43.413711+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3721,"log_dups_size":0,"ondisk_log_size":3721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":22,"num_write":19557,"num_write_kb":316044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"67'26","reported_seq":148,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352348+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T05:20:36.352348+0000","last_peered":"2024-09-17T05:20:36.352348+0000","last_clean":"2024-09-17T05:20:36.352348+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T05:20:36.352348+0000","last_undegraded":"2024-09-17T05:20:36.352348+0000","last_fullsized":"2024-09-17T05:20:36.352348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4998","reported_seq":5118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.353392+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T05:20:36.353392+0000","last_peered":"2024-09-17T05:20:36.353392+0000","last_clean":"2024-09-17T05:20:36.353392+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T05:20:36.353392+0000","last_undegraded":"2024-09-17T05:20:36.353392+0000","last_fullsized":"2024-09-17T05:20:36.353392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27336,"num_write_kb":388205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'2","reported_seq":113,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362495+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T05:20:36.362495+0000","last_peered":"2024-09-17T05:20:36.362495+0000","last_clean":"2024-09-17T05:20:36.362495+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T05:20:36.362495+0000","last_undegraded":"2024-09-17T05:20:36.362495+0000","last_fullsized":"2024-09-17T05:20:36.362495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"68'2765","reported_seq":2876,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.349402+0000","last_change":"2024-09-17T04:49:41.067317+0000","last_active":"2024-09-17T05:20:43.349402+0000","last_peered":"2024-09-17T05:20:43.349402+0000","last_clean":"2024-09-17T05:20:43.349402+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T05:20:43.349402+0000","last_undegraded":"2024-09-17T05:20:43.349402+0000","last_fullsized":"2024-09-17T05:20:43.349402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:37.637477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":14571,"num_write_kb":258996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"68'46","reported_seq":207,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558188+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T05:20:36.558188+0000","last_peered":"2024-09-17T05:20:36.558188+0000","last_clean":"2024-09-17T05:20:36.558188+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T05:20:36.558188+0000","last_undegraded":"2024-09-17T05:20:36.558188+0000","last_fullsized":"2024-09-17T05:20:36.558188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"65'3140","reported_seq":3259,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.346435+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T05:20:36.346435+0000","last_peered":"2024-09-17T05:20:36.346435+0000","last_clean":"2024-09-17T05:20:36.346435+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T05:20:36.346435+0000","last_undegraded":"2024-09-17T05:20:36.346435+0000","last_fullsized":"2024-09-17T05:20:36.346435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3140,"log_dups_size":0,"ondisk_log_size":3140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16962,"num_write_kb":252282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"68'751","reported_seq":888,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.388359+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T05:20:43.388359+0000","last_peered":"2024-09-17T05:20:43.388359+0000","last_clean":"2024-09-17T05:20:43.388359+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T05:20:43.388359+0000","last_undegraded":"2024-09-17T05:20:43.388359+0000","last_fullsized":"2024-09-17T05:20:43.388359+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":75812,"num_write":739,"num_write_kb":4324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"68'2700","reported_seq":2810,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.180944+0000","last_change":"2024-09-17T04:49:41.043531+0000","last_active":"2024-09-17T05:20:43.180944+0000","last_peered":"2024-09-17T05:20:43.180944+0000","last_clean":"2024-09-17T05:20:43.180944+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T05:20:43.180944+0000","last_undegraded":"2024-09-17T05:20:43.180944+0000","last_fullsized":"2024-09-17T05:20:43.180944+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2700,"log_dups_size":0,"ondisk_log_size":2700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":5,"num_write":13438,"num_write_kb":225755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"65'5","reported_seq":166,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559425+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T05:20:36.559425+0000","last_peered":"2024-09-17T05:20:36.559425+0000","last_clean":"2024-09-17T05:20:36.559425+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T05:20:36.559425+0000","last_undegraded":"2024-09-17T05:20:36.559425+0000","last_fullsized":"2024-09-17T05:20:36.559425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"65'4481","reported_seq":4598,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.373776+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T05:20:36.373776+0000","last_peered":"2024-09-17T05:20:36.373776+0000","last_clean":"2024-09-17T05:20:36.373776+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T05:20:36.373776+0000","last_undegraded":"2024-09-17T05:20:36.373776+0000","last_fullsized":"2024-09-17T05:20:36.373776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24963,"num_write_kb":346034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"68'143","reported_seq":256,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:51.881072+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T05:20:51.881072+0000","last_peered":"2024-09-17T05:20:51.881072+0000","last_clean":"2024-09-17T05:20:51.881072+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T05:20:51.881072+0000","last_undegraded":"2024-09-17T05:20:51.881072+0000","last_fullsized":"2024-09-17T05:20:51.881072+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:16.854365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1447387,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":383,"num_write":117,"num_write_kb":1479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"68'2942","reported_seq":3054,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.409983+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T05:20:44.409983+0000","last_peered":"2024-09-17T05:20:44.409983+0000","last_clean":"2024-09-17T05:20:44.409983+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T05:20:44.409983+0000","last_undegraded":"2024-09-17T05:20:44.409983+0000","last_fullsized":"2024-09-17T05:20:44.409983+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":16254,"num_write_kb":278162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"65'3968","reported_seq":4087,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362298+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T05:20:36.362298+0000","last_peered":"2024-09-17T05:20:36.362298+0000","last_clean":"2024-09-17T05:20:36.362298+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T05:20:36.362298+0000","last_undegraded":"2024-09-17T05:20:36.362298+0000","last_fullsized":"2024-09-17T05:20:36.362298+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3968,"log_dups_size":0,"ondisk_log_size":3968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21939,"num_write_kb":297018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"68'1467","reported_seq":1636,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366269+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T05:20:36.366269+0000","last_peered":"2024-09-17T05:20:36.366269+0000","last_clean":"2024-09-17T05:20:36.366269+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T05:20:36.366269+0000","last_undegraded":"2024-09-17T05:20:36.366269+0000","last_fullsized":"2024-09-17T05:20:36.366269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1467,"log_dups_size":0,"ondisk_log_size":1467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":43333,"num_write":1444,"num_write_kb":4521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'6","reported_seq":138,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.115470+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T05:20:43.115470+0000","last_peered":"2024-09-17T05:20:43.115470+0000","last_clean":"2024-09-17T05:20:43.115470+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T05:20:43.115470+0000","last_undegraded":"2024-09-17T05:20:43.115470+0000","last_fullsized":"2024-09-17T05:20:43.115470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:28.365399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'3236","reported_seq":3349,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.400322+0000","last_change":"2024-09-17T04:49:41.038415+0000","last_active":"2024-09-17T05:20:44.400322+0000","last_peered":"2024-09-17T05:20:44.400322+0000","last_clean":"2024-09-17T05:20:44.400322+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T05:20:44.400322+0000","last_undegraded":"2024-09-17T05:20:44.400322+0000","last_fullsized":"2024-09-17T05:20:44.400322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3236,"log_dups_size":0,"ondisk_log_size":3236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:17.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":17542,"num_write_kb":271055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"65'3303","reported_seq":3421,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.358141+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T05:20:36.358141+0000","last_peered":"2024-09-17T05:20:36.358141+0000","last_clean":"2024-09-17T05:20:36.358141+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T05:20:36.358141+0000","last_undegraded":"2024-09-17T05:20:36.358141+0000","last_fullsized":"2024-09-17T05:20:36.358141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3303,"log_dups_size":0,"ondisk_log_size":3303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17463,"num_write_kb":238014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"65'3","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.344536+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T05:20:36.344536+0000","last_peered":"2024-09-17T05:20:36.344536+0000","last_clean":"2024-09-17T05:20:36.344536+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T05:20:36.344536+0000","last_undegraded":"2024-09-17T05:20:36.344536+0000","last_fullsized":"2024-09-17T05:20:36.344536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"68'5","reported_seq":117,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.405924+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T05:20:43.405924+0000","last_peered":"2024-09-17T05:20:43.405924+0000","last_clean":"2024-09-17T05:20:43.405924+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T05:20:43.405924+0000","last_undegraded":"2024-09-17T05:20:43.405924+0000","last_fullsized":"2024-09-17T05:20:43.405924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:17.424265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"68'2691","reported_seq":2802,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.414897+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T05:20:43.414897+0000","last_peered":"2024-09-17T05:20:43.414897+0000","last_clean":"2024-09-17T05:20:43.414897+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T05:20:43.414897+0000","last_undegraded":"2024-09-17T05:20:43.414897+0000","last_fullsized":"2024-09-17T05:20:43.414897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:06.516432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":14365,"num_write_kb":225564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"65'3640","reported_seq":3759,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.337791+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T05:20:36.337791+0000","last_peered":"2024-09-17T05:20:36.337791+0000","last_clean":"2024-09-17T05:20:36.337791+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T05:20:36.337791+0000","last_undegraded":"2024-09-17T05:20:36.337791+0000","last_fullsized":"2024-09-17T05:20:36.337791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3640,"log_dups_size":0,"ondisk_log_size":3640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20183,"num_write_kb":277678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"65'2","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340129+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T05:20:36.340129+0000","last_peered":"2024-09-17T05:20:36.340129+0000","last_clean":"2024-09-17T05:20:36.340129+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T05:20:36.340129+0000","last_undegraded":"2024-09-17T05:20:36.340129+0000","last_fullsized":"2024-09-17T05:20:36.340129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"68'4","reported_seq":136,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.390191+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T05:20:43.390191+0000","last_peered":"2024-09-17T05:20:43.390191+0000","last_clean":"2024-09-17T05:20:43.390191+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T05:20:43.390191+0000","last_undegraded":"2024-09-17T05:20:43.390191+0000","last_fullsized":"2024-09-17T05:20:43.390191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:00.010365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"68'3943","reported_seq":4053,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.367267+0000","last_change":"2024-09-17T04:49:41.036658+0000","last_active":"2024-09-17T05:20:43.367267+0000","last_peered":"2024-09-17T05:20:43.367267+0000","last_clean":"2024-09-17T05:20:43.367267+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T05:20:43.367267+0000","last_undegraded":"2024-09-17T05:20:43.367267+0000","last_fullsized":"2024-09-17T05:20:43.367267+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21088,"num_write_kb":359062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"65'3699","reported_seq":3818,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364687+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T05:20:36.364687+0000","last_peered":"2024-09-17T05:20:36.364687+0000","last_clean":"2024-09-17T05:20:36.364687+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T05:20:36.364687+0000","last_undegraded":"2024-09-17T05:20:36.364687+0000","last_fullsized":"2024-09-17T05:20:36.364687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19661,"num_write_kb":271466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"65'5","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.555326+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T05:20:36.555326+0000","last_peered":"2024-09-17T05:20:36.555326+0000","last_clean":"2024-09-17T05:20:36.555326+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T05:20:36.555326+0000","last_undegraded":"2024-09-17T05:20:36.555326+0000","last_fullsized":"2024-09-17T05:20:36.555326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"68'7","reported_seq":159,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.387118+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T05:20:43.387118+0000","last_peered":"2024-09-17T05:20:43.387118+0000","last_clean":"2024-09-17T05:20:43.387118+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T05:20:43.387118+0000","last_undegraded":"2024-09-17T05:20:43.387118+0000","last_fullsized":"2024-09-17T05:20:43.387118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"68'2532","reported_seq":2644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.411846+0000","last_change":"2024-09-17T04:49:41.050170+0000","last_active":"2024-09-17T05:20:43.411846+0000","last_peered":"2024-09-17T05:20:43.411846+0000","last_clean":"2024-09-17T05:20:43.411846+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T05:20:43.411846+0000","last_undegraded":"2024-09-17T05:20:43.411846+0000","last_fullsized":"2024-09-17T05:20:43.411846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2532,"log_dups_size":0,"ondisk_log_size":2532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13954,"num_write_kb":219876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"65'4598","reported_seq":4718,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339224+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T05:20:36.339224+0000","last_peered":"2024-09-17T05:20:36.339224+0000","last_clean":"2024-09-17T05:20:36.339224+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T05:20:36.339224+0000","last_undegraded":"2024-09-17T05:20:36.339224+0000","last_fullsized":"2024-09-17T05:20:36.339224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24313,"num_write_kb":335123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"65'9","reported_seq":172,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.367024+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T05:20:36.367024+0000","last_peered":"2024-09-17T05:20:36.367024+0000","last_clean":"2024-09-17T05:20:36.367024+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T05:20:36.367024+0000","last_undegraded":"2024-09-17T05:20:36.367024+0000","last_fullsized":"2024-09-17T05:20:36.367024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'50","reported_seq":229,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:48.374529+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T05:20:48.374529+0000","last_peered":"2024-09-17T05:20:48.374529+0000","last_clean":"2024-09-17T05:20:48.374529+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T05:20:48.374529+0000","last_undegraded":"2024-09-17T05:20:48.374529+0000","last_fullsized":"2024-09-17T05:20:48.374529+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":65,"num_read_kb":114,"num_write":45,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"68'2531","reported_seq":2644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.376480+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T05:20:43.376480+0000","last_peered":"2024-09-17T05:20:43.376480+0000","last_clean":"2024-09-17T05:20:43.376480+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T05:20:43.376480+0000","last_undegraded":"2024-09-17T05:20:43.376480+0000","last_fullsized":"2024-09-17T05:20:43.376480+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2531,"log_dups_size":0,"ondisk_log_size":2531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13881,"num_write_kb":219504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"65'3884","reported_seq":4003,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340033+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T05:20:36.340033+0000","last_peered":"2024-09-17T05:20:36.340033+0000","last_clean":"2024-09-17T05:20:36.340033+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T05:20:36.340033+0000","last_undegraded":"2024-09-17T05:20:36.340033+0000","last_fullsized":"2024-09-17T05:20:36.340033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21342,"num_write_kb":257935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"68'473","reported_seq":615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.328144+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T05:20:44.328144+0000","last_peered":"2024-09-17T05:20:44.328144+0000","last_clean":"2024-09-17T05:20:44.328144+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T05:20:44.328144+0000","last_undegraded":"2024-09-17T05:20:44.328144+0000","last_fullsized":"2024-09-17T05:20:44.328144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":57,"num_write":1105,"num_write_kb":359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'11","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410351+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T05:20:43.410351+0000","last_peered":"2024-09-17T05:20:43.410351+0000","last_clean":"2024-09-17T05:20:43.410351+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T05:20:43.410351+0000","last_undegraded":"2024-09-17T05:20:43.410351+0000","last_fullsized":"2024-09-17T05:20:43.410351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"68'2691","reported_seq":2802,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.217861+0000","last_change":"2024-09-17T04:49:41.042314+0000","last_active":"2024-09-17T05:20:43.217861+0000","last_peered":"2024-09-17T05:20:43.217861+0000","last_clean":"2024-09-17T05:20:43.217861+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T05:20:43.217861+0000","last_undegraded":"2024-09-17T05:20:43.217861+0000","last_fullsized":"2024-09-17T05:20:43.217861+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14724,"num_write_kb":243278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"65'4637","reported_seq":4757,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341853+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T05:20:36.341853+0000","last_peered":"2024-09-17T05:20:36.341853+0000","last_clean":"2024-09-17T05:20:36.341853+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T05:20:36.341853+0000","last_undegraded":"2024-09-17T05:20:36.341853+0000","last_fullsized":"2024-09-17T05:20:36.341853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25111,"num_write_kb":342484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"65'2","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.375220+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T05:20:36.375220+0000","last_peered":"2024-09-17T05:20:36.375220+0000","last_clean":"2024-09-17T05:20:36.375220+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T05:20:36.375220+0000","last_undegraded":"2024-09-17T05:20:36.375220+0000","last_fullsized":"2024-09-17T05:20:36.375220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'3","reported_seq":117,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393956+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T05:20:43.393956+0000","last_peered":"2024-09-17T05:20:43.393956+0000","last_clean":"2024-09-17T05:20:43.393956+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T05:20:43.393956+0000","last_undegraded":"2024-09-17T05:20:43.393956+0000","last_fullsized":"2024-09-17T05:20:43.393956+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"68'3511","reported_seq":3623,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.341640+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T05:20:43.341640+0000","last_peered":"2024-09-17T05:20:43.341640+0000","last_clean":"2024-09-17T05:20:43.341640+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T05:20:43.341640+0000","last_undegraded":"2024-09-17T05:20:43.341640+0000","last_fullsized":"2024-09-17T05:20:43.341640+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3511,"log_dups_size":0,"ondisk_log_size":3511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19215,"num_write_kb":314590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"65'3495","reported_seq":3615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362299+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T05:20:36.362299+0000","last_peered":"2024-09-17T05:20:36.362299+0000","last_clean":"2024-09-17T05:20:36.362299+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T05:20:36.362299+0000","last_undegraded":"2024-09-17T05:20:36.362299+0000","last_fullsized":"2024-09-17T05:20:36.362299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18740,"num_write_kb":251346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"65'2","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.351220+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T05:20:36.351220+0000","last_peered":"2024-09-17T05:20:36.351220+0000","last_clean":"2024-09-17T05:20:36.351220+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T05:20:36.351220+0000","last_undegraded":"2024-09-17T05:20:36.351220+0000","last_fullsized":"2024-09-17T05:20:36.351220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'4","reported_seq":118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.112153+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T05:20:43.112153+0000","last_peered":"2024-09-17T05:20:43.112153+0000","last_clean":"2024-09-17T05:20:43.112153+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T05:20:43.112153+0000","last_undegraded":"2024-09-17T05:20:43.112153+0000","last_fullsized":"2024-09-17T05:20:43.112153+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"68'2923","reported_seq":3035,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.395059+0000","last_change":"2024-09-17T04:49:41.069177+0000","last_active":"2024-09-17T05:20:43.395059+0000","last_peered":"2024-09-17T05:20:43.395059+0000","last_clean":"2024-09-17T05:20:43.395059+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T05:20:43.395059+0000","last_undegraded":"2024-09-17T05:20:43.395059+0000","last_fullsized":"2024-09-17T05:20:43.395059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:16.855206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":7,"num_write":15908,"num_write_kb":260386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"65'4495","reported_seq":4615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559390+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T05:20:36.559390+0000","last_peered":"2024-09-17T05:20:36.559390+0000","last_clean":"2024-09-17T05:20:36.559390+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T05:20:36.559390+0000","last_undegraded":"2024-09-17T05:20:36.559390+0000","last_fullsized":"2024-09-17T05:20:36.559390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24287,"num_write_kb":330898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"65'4","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.365137+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T05:20:36.365137+0000","last_peered":"2024-09-17T05:20:36.365137+0000","last_clean":"2024-09-17T05:20:36.365137+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T05:20:36.365137+0000","last_undegraded":"2024-09-17T05:20:36.365137+0000","last_fullsized":"2024-09-17T05:20:36.365137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'2","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.401317+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T05:20:43.401317+0000","last_peered":"2024-09-17T05:20:43.401317+0000","last_clean":"2024-09-17T05:20:43.401317+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T05:20:43.401317+0000","last_undegraded":"2024-09-17T05:20:43.401317+0000","last_fullsized":"2024-09-17T05:20:43.401317+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"68'3959","reported_seq":4074,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.362853+0000","last_change":"2024-09-17T04:49:41.068537+0000","last_active":"2024-09-17T05:20:43.362853+0000","last_peered":"2024-09-17T05:20:43.362853+0000","last_clean":"2024-09-17T05:20:43.362853+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T05:20:43.362853+0000","last_undegraded":"2024-09-17T05:20:43.362853+0000","last_fullsized":"2024-09-17T05:20:43.362853+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3959,"log_dups_size":0,"ondisk_log_size":3959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":13,"num_write":21335,"num_write_kb":323115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'3351","reported_seq":3471,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.348480+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T05:20:36.348480+0000","last_peered":"2024-09-17T05:20:36.348480+0000","last_clean":"2024-09-17T05:20:36.348480+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T05:20:36.348480+0000","last_undegraded":"2024-09-17T05:20:36.348480+0000","last_fullsized":"2024-09-17T05:20:36.348480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18427,"num_write_kb":241864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"68'53","reported_seq":243,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.366344+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T05:20:44.366344+0000","last_peered":"2024-09-17T05:20:44.366344+0000","last_clean":"2024-09-17T05:20:44.366344+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T05:20:44.366344+0000","last_undegraded":"2024-09-17T05:20:44.366344+0000","last_fullsized":"2024-09-17T05:20:44.366344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":116,"num_write":48,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'9","reported_seq":162,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.386330+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T05:20:44.386330+0000","last_peered":"2024-09-17T05:20:44.386330+0000","last_clean":"2024-09-17T05:20:44.386330+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T05:20:44.386330+0000","last_undegraded":"2024-09-17T05:20:44.386330+0000","last_fullsized":"2024-09-17T05:20:44.386330+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"68'2708","reported_seq":2819,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403990+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T05:20:44.403990+0000","last_peered":"2024-09-17T05:20:44.403990+0000","last_clean":"2024-09-17T05:20:44.403990+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T05:20:44.403990+0000","last_undegraded":"2024-09-17T05:20:44.403990+0000","last_fullsized":"2024-09-17T05:20:44.403990+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14599,"num_write_kb":234639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"65'3456","reported_seq":3576,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339164+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T05:20:36.339164+0000","last_peered":"2024-09-17T05:20:36.339164+0000","last_clean":"2024-09-17T05:20:36.339164+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T05:20:36.339164+0000","last_undegraded":"2024-09-17T05:20:36.339164+0000","last_fullsized":"2024-09-17T05:20:36.339164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18921,"num_write_kb":260292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"67'13","reported_seq":135,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366577+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T05:20:36.366577+0000","last_peered":"2024-09-17T05:20:36.366577+0000","last_clean":"2024-09-17T05:20:36.366577+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T05:20:36.366577+0000","last_undegraded":"2024-09-17T05:20:36.366577+0000","last_fullsized":"2024-09-17T05:20:36.366577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"68'473","reported_seq":606,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:47.345918+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T05:20:47.345918+0000","last_peered":"2024-09-17T05:20:47.345918+0000","last_clean":"2024-09-17T05:20:47.345918+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T05:20:47.345918+0000","last_undegraded":"2024-09-17T05:20:47.345918+0000","last_fullsized":"2024-09-17T05:20:47.345918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":54,"num_write":1100,"num_write_kb":354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"68'3699","reported_seq":3811,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.383257+0000","last_change":"2024-09-17T04:49:41.052564+0000","last_active":"2024-09-17T05:20:43.383257+0000","last_peered":"2024-09-17T05:20:43.383257+0000","last_clean":"2024-09-17T05:20:43.383257+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T05:20:43.383257+0000","last_undegraded":"2024-09-17T05:20:43.383257+0000","last_fullsized":"2024-09-17T05:20:43.383257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":20021,"num_write_kb":309708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"65'3651","reported_seq":3771,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341142+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T05:20:36.341142+0000","last_peered":"2024-09-17T05:20:36.341142+0000","last_clean":"2024-09-17T05:20:36.341142+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T05:20:36.341142+0000","last_undegraded":"2024-09-17T05:20:36.341142+0000","last_fullsized":"2024-09-17T05:20:36.341142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19517,"num_write_kb":274922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"65'3","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366726+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T05:20:36.366726+0000","last_peered":"2024-09-17T05:20:36.366726+0000","last_clean":"2024-09-17T05:20:36.366726+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T05:20:36.366726+0000","last_undegraded":"2024-09-17T05:20:36.366726+0000","last_fullsized":"2024-09-17T05:20:36.366726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":218,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.343214+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T05:20:36.343214+0000","last_peered":"2024-09-17T05:20:36.343214+0000","last_clean":"2024-09-17T05:20:36.343214+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T05:20:36.343214+0000","last_undegraded":"2024-09-17T05:20:36.343214+0000","last_fullsized":"2024-09-17T05:20:36.343214+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'2","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374244+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T05:20:36.374244+0000","last_peered":"2024-09-17T05:20:36.374244+0000","last_clean":"2024-09-17T05:20:36.374244+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T05:20:36.374244+0000","last_undegraded":"2024-09-17T05:20:36.374244+0000","last_fullsized":"2024-09-17T05:20:36.374244+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:21.202398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"68'2130","reported_seq":2241,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404345+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T05:20:44.404345+0000","last_peered":"2024-09-17T05:20:44.404345+0000","last_clean":"2024-09-17T05:20:44.404345+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T05:20:44.404345+0000","last_undegraded":"2024-09-17T05:20:44.404345+0000","last_fullsized":"2024-09-17T05:20:44.404345+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2130,"log_dups_size":0,"ondisk_log_size":2130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:48.128609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":11413,"num_write_kb":185062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"65'4183","reported_seq":4301,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374398+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T05:20:36.374398+0000","last_peered":"2024-09-17T05:20:36.374398+0000","last_clean":"2024-09-17T05:20:36.374398+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T05:20:36.374398+0000","last_undegraded":"2024-09-17T05:20:36.374398+0000","last_fullsized":"2024-09-17T05:20:36.374398+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22326,"num_write_kb":302603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"65'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.342324+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T05:20:36.342324+0000","last_peered":"2024-09-17T05:20:36.342324+0000","last_clean":"2024-09-17T05:20:36.342324+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T05:20:36.342324+0000","last_undegraded":"2024-09-17T05:20:36.342324+0000","last_fullsized":"2024-09-17T05:20:36.342324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"68'3","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.376325+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T05:20:43.376325+0000","last_peered":"2024-09-17T05:20:43.376325+0000","last_clean":"2024-09-17T05:20:43.376325+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T05:20:43.376325+0000","last_undegraded":"2024-09-17T05:20:43.376325+0000","last_fullsized":"2024-09-17T05:20:43.376325+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"68'3285","reported_seq":3397,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.413822+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T05:20:43.413822+0000","last_peered":"2024-09-17T05:20:43.413822+0000","last_clean":"2024-09-17T05:20:43.413822+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T05:20:43.413822+0000","last_undegraded":"2024-09-17T05:20:43.413822+0000","last_fullsized":"2024-09-17T05:20:43.413822+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3285,"log_dups_size":0,"ondisk_log_size":3285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18245,"num_write_kb":282158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"65'4407","reported_seq":4525,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558055+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T05:20:36.558055+0000","last_peered":"2024-09-17T05:20:36.558055+0000","last_clean":"2024-09-17T05:20:36.558055+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T05:20:36.558055+0000","last_undegraded":"2024-09-17T05:20:36.558055+0000","last_fullsized":"2024-09-17T05:20:36.558055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24558,"num_write_kb":315810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"65'6","reported_seq":147,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352587+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T05:20:36.352587+0000","last_peered":"2024-09-17T05:20:36.352587+0000","last_clean":"2024-09-17T05:20:36.352587+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T05:20:36.352587+0000","last_undegraded":"2024-09-17T05:20:36.352587+0000","last_fullsized":"2024-09-17T05:20:36.352587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'1445","reported_seq":1601,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410628+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T05:20:43.410628+0000","last_peered":"2024-09-17T05:20:43.410628+0000","last_clean":"2024-09-17T05:20:43.410628+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T05:20:43.410628+0000","last_undegraded":"2024-09-17T05:20:43.410628+0000","last_fullsized":"2024-09-17T05:20:43.410628+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:18.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":38239,"num_write":1421,"num_write_kb":4489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'2587","reported_seq":2698,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.177381+0000","last_change":"2024-09-17T04:49:41.031036+0000","last_active":"2024-09-17T05:20:43.177381+0000","last_peered":"2024-09-17T05:20:43.177381+0000","last_clean":"2024-09-17T05:20:43.177381+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T05:20:43.177381+0000","last_undegraded":"2024-09-17T05:20:43.177381+0000","last_fullsized":"2024-09-17T05:20:43.177381+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2587,"log_dups_size":0,"ondisk_log_size":2587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:48.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13558,"num_write_kb":238553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"65'4762","reported_seq":4880,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559085+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T05:20:36.559085+0000","last_peered":"2024-09-17T05:20:36.559085+0000","last_clean":"2024-09-17T05:20:36.559085+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T05:20:36.559085+0000","last_undegraded":"2024-09-17T05:20:36.559085+0000","last_fullsized":"2024-09-17T05:20:36.559085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25049,"num_write_kb":354398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"65'7","reported_seq":169,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340155+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T05:20:36.340155+0000","last_peered":"2024-09-17T05:20:36.340155+0000","last_clean":"2024-09-17T05:20:36.340155+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T05:20:36.340155+0000","last_undegraded":"2024-09-17T05:20:36.340155+0000","last_fullsized":"2024-09-17T05:20:36.340155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"68'5","reported_seq":118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.116235+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T05:20:43.116235+0000","last_peered":"2024-09-17T05:20:43.116235+0000","last_clean":"2024-09-17T05:20:43.116235+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T05:20:43.116235+0000","last_undegraded":"2024-09-17T05:20:43.116235+0000","last_fullsized":"2024-09-17T05:20:43.116235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"68'2327","reported_seq":2438,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.108349+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T05:20:43.108349+0000","last_peered":"2024-09-17T05:20:43.108349+0000","last_clean":"2024-09-17T05:20:43.108349+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T05:20:43.108349+0000","last_undegraded":"2024-09-17T05:20:43.108349+0000","last_fullsized":"2024-09-17T05:20:43.108349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2327,"log_dups_size":0,"ondisk_log_size":2327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12892,"num_write_kb":211370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"65'3722","reported_seq":3840,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366989+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T05:20:36.366989+0000","last_peered":"2024-09-17T05:20:36.366989+0000","last_clean":"2024-09-17T05:20:36.366989+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T05:20:36.366989+0000","last_undegraded":"2024-09-17T05:20:36.366989+0000","last_fullsized":"2024-09-17T05:20:36.366989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3722,"log_dups_size":0,"ondisk_log_size":3722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19724,"num_write_kb":263684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"65'4","reported_seq":145,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.560132+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T05:20:36.560132+0000","last_peered":"2024-09-17T05:20:36.560132+0000","last_clean":"2024-09-17T05:20:36.560132+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T05:20:36.560132+0000","last_undegraded":"2024-09-17T05:20:36.560132+0000","last_fullsized":"2024-09-17T05:20:36.560132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"68'2","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.385249+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T05:20:43.385249+0000","last_peered":"2024-09-17T05:20:43.385249+0000","last_clean":"2024-09-17T05:20:43.385249+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T05:20:43.385249+0000","last_undegraded":"2024-09-17T05:20:43.385249+0000","last_fullsized":"2024-09-17T05:20:43.385249+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:09.712272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"68'3186","reported_seq":3295,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.381181+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T05:20:43.381181+0000","last_peered":"2024-09-17T05:20:43.381181+0000","last_clean":"2024-09-17T05:20:43.381181+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T05:20:43.381181+0000","last_undegraded":"2024-09-17T05:20:43.381181+0000","last_fullsized":"2024-09-17T05:20:43.381181+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:43.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16763,"num_write_kb":264037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"65'3694","reported_seq":3814,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362961+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T05:20:36.362961+0000","last_peered":"2024-09-17T05:20:36.362961+0000","last_clean":"2024-09-17T05:20:36.362961+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T05:20:36.362961+0000","last_undegraded":"2024-09-17T05:20:36.362961+0000","last_fullsized":"2024-09-17T05:20:36.362961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20762,"num_write_kb":284135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"68'15","reported_seq":223,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.342372+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T05:20:36.342372+0000","last_peered":"2024-09-17T05:20:36.342372+0000","last_clean":"2024-09-17T05:20:36.342372+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T05:20:36.342372+0000","last_undegraded":"2024-09-17T05:20:36.342372+0000","last_fullsized":"2024-09-17T05:20:36.342372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":264,"num_read_kb":264,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"68'3","reported_seq":114,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.394770+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T05:20:43.394770+0000","last_peered":"2024-09-17T05:20:43.394770+0000","last_clean":"2024-09-17T05:20:43.394770+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T05:20:43.394770+0000","last_undegraded":"2024-09-17T05:20:43.394770+0000","last_fullsized":"2024-09-17T05:20:43.394770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:52.364568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"68'3074","reported_seq":3186,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.395538+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T05:20:44.395538+0000","last_peered":"2024-09-17T05:20:44.395538+0000","last_clean":"2024-09-17T05:20:44.395538+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T05:20:44.395538+0000","last_undegraded":"2024-09-17T05:20:44.395538+0000","last_fullsized":"2024-09-17T05:20:44.395538+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:03.537392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":16509,"num_write_kb":287321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'2828","reported_seq":2946,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.342437+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T05:20:36.342437+0000","last_peered":"2024-09-17T05:20:36.342437+0000","last_clean":"2024-09-17T05:20:36.342437+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T05:20:36.342437+0000","last_undegraded":"2024-09-17T05:20:36.342437+0000","last_fullsized":"2024-09-17T05:20:36.342437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14821,"num_write_kb":206837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"65'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557382+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T05:20:36.557382+0000","last_peered":"2024-09-17T05:20:36.557382+0000","last_clean":"2024-09-17T05:20:36.557382+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T05:20:36.557382+0000","last_undegraded":"2024-09-17T05:20:36.557382+0000","last_fullsized":"2024-09-17T05:20:36.557382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"68'5","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.388852+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T05:20:43.388852+0000","last_peered":"2024-09-17T05:20:43.388852+0000","last_clean":"2024-09-17T05:20:43.388852+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T05:20:43.388852+0000","last_undegraded":"2024-09-17T05:20:43.388852+0000","last_fullsized":"2024-09-17T05:20:43.388852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:52.120543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"68'2590","reported_seq":2699,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.324869+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T05:20:43.324869+0000","last_peered":"2024-09-17T05:20:43.324869+0000","last_clean":"2024-09-17T05:20:43.324869+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T05:20:43.324869+0000","last_undegraded":"2024-09-17T05:20:43.324869+0000","last_fullsized":"2024-09-17T05:20:43.324869+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:56.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14722,"num_write_kb":243326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'4461","reported_seq":4579,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364760+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T05:20:36.364760+0000","last_peered":"2024-09-17T05:20:36.364760+0000","last_clean":"2024-09-17T05:20:36.364760+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T05:20:36.364760+0000","last_undegraded":"2024-09-17T05:20:36.364760+0000","last_fullsized":"2024-09-17T05:20:36.364760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4461,"log_dups_size":0,"ondisk_log_size":4461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24667,"num_write_kb":332445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"65'3","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364408+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T05:20:36.364408+0000","last_peered":"2024-09-17T05:20:36.364408+0000","last_clean":"2024-09-17T05:20:36.364408+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T05:20:36.364408+0000","last_undegraded":"2024-09-17T05:20:36.364408+0000","last_fullsized":"2024-09-17T05:20:36.364408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"68'367","reported_seq":579,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:39.638931+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T05:20:39.638931+0000","last_peered":"2024-09-17T05:20:39.638931+0000","last_clean":"2024-09-17T05:20:39.638931+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T05:20:39.638931+0000","last_undegraded":"2024-09-17T05:20:39.638931+0000","last_fullsized":"2024-09-17T05:20:39.638931+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":180,"num_read_kb":82080,"num_write":364,"num_write_kb":4295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"68'3856","reported_seq":3961,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403743+0000","last_change":"2024-09-17T04:50:12.067788+0000","last_active":"2024-09-17T05:20:44.403743+0000","last_peered":"2024-09-17T05:20:44.403743+0000","last_clean":"2024-09-17T05:20:44.403743+0000","last_became_active":"2024-09-17T04:50:12.066741+0000","last_became_peered":"2024-09-17T04:50:12.066741+0000","last_unstale":"2024-09-17T05:20:44.403743+0000","last_undegraded":"2024-09-17T05:20:44.403743+0000","last_fullsized":"2024-09-17T05:20:44.403743+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3856,"log_dups_size":0,"ondisk_log_size":3856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":20645,"num_write_kb":344954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"65'3521","reported_seq":3639,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.344504+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T05:20:36.344504+0000","last_peered":"2024-09-17T05:20:36.344504+0000","last_clean":"2024-09-17T05:20:36.344504+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T05:20:36.344504+0000","last_undegraded":"2024-09-17T05:20:36.344504+0000","last_fullsized":"2024-09-17T05:20:36.344504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3521,"log_dups_size":0,"ondisk_log_size":3521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19198,"num_write_kb":253218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"67'10","reported_seq":172,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.358352+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T05:20:36.358352+0000","last_peered":"2024-09-17T05:20:36.358352+0000","last_clean":"2024-09-17T05:20:36.358352+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T05:20:36.358352+0000","last_undegraded":"2024-09-17T05:20:36.358352+0000","last_fullsized":"2024-09-17T05:20:36.358352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":168,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"68'30","reported_seq":181,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:45.378073+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T05:20:45.378073+0000","last_peered":"2024-09-17T05:20:45.378073+0000","last_clean":"2024-09-17T05:20:45.378073+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T05:20:45.378073+0000","last_undegraded":"2024-09-17T05:20:45.378073+0000","last_fullsized":"2024-09-17T05:20:45.378073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:07.945355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":31,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"68'3085","reported_seq":3195,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.412893+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T05:20:43.412893+0000","last_peered":"2024-09-17T05:20:43.412893+0000","last_clean":"2024-09-17T05:20:43.412893+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T05:20:43.412893+0000","last_undegraded":"2024-09-17T05:20:43.412893+0000","last_fullsized":"2024-09-17T05:20:43.412893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16824,"num_write_kb":288541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"65'3611","reported_seq":3729,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.556980+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T05:20:36.556980+0000","last_peered":"2024-09-17T05:20:36.556980+0000","last_clean":"2024-09-17T05:20:36.556980+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T05:20:36.556980+0000","last_undegraded":"2024-09-17T05:20:36.556980+0000","last_fullsized":"2024-09-17T05:20:36.556980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3611,"log_dups_size":0,"ondisk_log_size":3611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19975,"num_write_kb":264442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"68'349","reported_seq":575,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.121847+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T05:20:44.121847+0000","last_peered":"2024-09-17T05:20:44.121847+0000","last_clean":"2024-09-17T05:20:44.121847+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T05:20:44.121847+0000","last_undegraded":"2024-09-17T05:20:44.121847+0000","last_fullsized":"2024-09-17T05:20:44.121847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":190,"num_read_kb":82094,"num_write":345,"num_write_kb":4278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"68'3","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.396510+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T05:20:44.396510+0000","last_peered":"2024-09-17T05:20:44.396510+0000","last_clean":"2024-09-17T05:20:44.396510+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T05:20:44.396510+0000","last_undegraded":"2024-09-17T05:20:44.396510+0000","last_fullsized":"2024-09-17T05:20:44.396510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:16.447835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"68'2950","reported_seq":3061,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.342183+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T05:20:43.342183+0000","last_peered":"2024-09-17T05:20:43.342183+0000","last_clean":"2024-09-17T05:20:43.342183+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T05:20:43.342183+0000","last_undegraded":"2024-09-17T05:20:43.342183+0000","last_fullsized":"2024-09-17T05:20:43.342183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":16110,"num_write_kb":249022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"65'4529","reported_seq":4647,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.361047+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T05:20:36.361047+0000","last_peered":"2024-09-17T05:20:36.361047+0000","last_clean":"2024-09-17T05:20:36.361047+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T05:20:36.361047+0000","last_undegraded":"2024-09-17T05:20:36.361047+0000","last_fullsized":"2024-09-17T05:20:36.361047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23581,"num_write_kb":327974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"68'31","reported_seq":214,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.399687+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T05:20:44.399687+0000","last_peered":"2024-09-17T05:20:44.399687+0000","last_clean":"2024-09-17T05:20:44.399687+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T05:20:44.399687+0000","last_undegraded":"2024-09-17T05:20:44.399687+0000","last_fullsized":"2024-09-17T05:20:44.399687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'8","reported_seq":160,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.415231+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T05:20:43.415231+0000","last_peered":"2024-09-17T05:20:43.415231+0000","last_clean":"2024-09-17T05:20:43.415231+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T05:20:43.415231+0000","last_undegraded":"2024-09-17T05:20:43.415231+0000","last_fullsized":"2024-09-17T05:20:43.415231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"68'2811","reported_seq":2924,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.378555+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T05:20:43.378555+0000","last_peered":"2024-09-17T05:20:43.378555+0000","last_clean":"2024-09-17T05:20:43.378555+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T05:20:43.378555+0000","last_undegraded":"2024-09-17T05:20:43.378555+0000","last_fullsized":"2024-09-17T05:20:43.378555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2811,"log_dups_size":0,"ondisk_log_size":2811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15712,"num_write_kb":243392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"65'4109","reported_seq":4226,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.353428+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T05:20:36.353428+0000","last_peered":"2024-09-17T05:20:36.353428+0000","last_clean":"2024-09-17T05:20:36.353428+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T05:20:36.353428+0000","last_undegraded":"2024-09-17T05:20:36.353428+0000","last_fullsized":"2024-09-17T05:20:36.353428+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21556,"num_write_kb":307833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"65'3","reported_seq":122,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558714+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T05:20:36.558714+0000","last_peered":"2024-09-17T05:20:36.558714+0000","last_clean":"2024-09-17T05:20:36.558714+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T05:20:36.558714+0000","last_undegraded":"2024-09-17T05:20:36.558714+0000","last_fullsized":"2024-09-17T05:20:36.558714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'15","reported_seq":209,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410016+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T05:20:43.410016+0000","last_peered":"2024-09-17T05:20:43.410016+0000","last_clean":"2024-09-17T05:20:43.410016+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T05:20:43.410016+0000","last_undegraded":"2024-09-17T05:20:43.410016+0000","last_fullsized":"2024-09-17T05:20:43.410016+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:40.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":246,"num_read_kb":246,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"68'2542","reported_seq":2652,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:42.865170+0000","last_change":"2024-09-17T04:49:41.040278+0000","last_active":"2024-09-17T05:20:42.865170+0000","last_peered":"2024-09-17T05:20:42.865170+0000","last_clean":"2024-09-17T05:20:42.865170+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T05:20:42.865170+0000","last_undegraded":"2024-09-17T05:20:42.865170+0000","last_fullsized":"2024-09-17T05:20:42.865170+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2542,"log_dups_size":0,"ondisk_log_size":2542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.895223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13327,"num_write_kb":225533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"65'3736","reported_seq":3854,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.365811+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T05:20:36.365811+0000","last_peered":"2024-09-17T05:20:36.365811+0000","last_clean":"2024-09-17T05:20:36.365811+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T05:20:36.365811+0000","last_undegraded":"2024-09-17T05:20:36.365811+0000","last_fullsized":"2024-09-17T05:20:36.365811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20513,"num_write_kb":287089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"68'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.363961+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T05:20:44.363961+0000","last_peered":"2024-09-17T05:20:44.363961+0000","last_clean":"2024-09-17T05:20:44.363961+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T05:20:44.363961+0000","last_undegraded":"2024-09-17T05:20:44.363961+0000","last_fullsized":"2024-09-17T05:20:44.363961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"68'5","reported_seq":117,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.409527+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T05:20:43.409527+0000","last_peered":"2024-09-17T05:20:43.409527+0000","last_clean":"2024-09-17T05:20:43.409527+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T05:20:43.409527+0000","last_undegraded":"2024-09-17T05:20:43.409527+0000","last_fullsized":"2024-09-17T05:20:43.409527+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:15.845612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"68'4175","reported_seq":4291,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.426185+0000","last_change":"2024-09-17T04:49:41.050551+0000","last_active":"2024-09-17T05:20:44.426185+0000","last_peered":"2024-09-17T05:20:44.426185+0000","last_clean":"2024-09-17T05:20:44.426185+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T05:20:44.426185+0000","last_undegraded":"2024-09-17T05:20:44.426185+0000","last_fullsized":"2024-09-17T05:20:44.426185+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:11.088230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":40,"num_write":23028,"num_write_kb":362813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"68'25","reported_seq":139,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.384829+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T05:20:43.384829+0000","last_peered":"2024-09-17T05:20:43.384829+0000","last_clean":"2024-09-17T05:20:43.384829+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T05:20:43.384829+0000","last_undegraded":"2024-09-17T05:20:43.384829+0000","last_fullsized":"2024-09-17T05:20:43.384829+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:37.778756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"65'4339","reported_seq":4456,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.338640+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T05:20:36.338640+0000","last_peered":"2024-09-17T05:20:36.338640+0000","last_clean":"2024-09-17T05:20:36.338640+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T05:20:36.338640+0000","last_undegraded":"2024-09-17T05:20:36.338640+0000","last_fullsized":"2024-09-17T05:20:36.338640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23134,"num_write_kb":338604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"65'2","reported_seq":121,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366244+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T05:20:36.366244+0000","last_peered":"2024-09-17T05:20:36.366244+0000","last_clean":"2024-09-17T05:20:36.366244+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T05:20:36.366244+0000","last_undegraded":"2024-09-17T05:20:36.366244+0000","last_fullsized":"2024-09-17T05:20:36.366244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"68'2877","reported_seq":2989,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.395024+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T05:20:43.395024+0000","last_peered":"2024-09-17T05:20:43.395024+0000","last_clean":"2024-09-17T05:20:43.395024+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T05:20:43.395024+0000","last_undegraded":"2024-09-17T05:20:43.395024+0000","last_fullsized":"2024-09-17T05:20:43.395024+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2877,"log_dups_size":0,"ondisk_log_size":2877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15865,"num_write_kb":272557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"68'1000","reported_seq":1162,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.429369+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T05:20:44.429369+0000","last_peered":"2024-09-17T05:20:44.429369+0000","last_clean":"2024-09-17T05:20:44.429369+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T05:20:44.429369+0000","last_undegraded":"2024-09-17T05:20:44.429369+0000","last_fullsized":"2024-09-17T05:20:44.429369+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":9969,"num_write":977,"num_write_kb":4635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"65'4230","reported_seq":4347,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.349109+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T05:20:36.349109+0000","last_peered":"2024-09-17T05:20:36.349109+0000","last_clean":"2024-09-17T05:20:36.349109+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T05:20:36.349109+0000","last_undegraded":"2024-09-17T05:20:36.349109+0000","last_fullsized":"2024-09-17T05:20:36.349109+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22609,"num_write_kb":319990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"65'3","reported_seq":122,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339504+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T05:20:36.339504+0000","last_peered":"2024-09-17T05:20:36.339504+0000","last_clean":"2024-09-17T05:20:36.339504+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T05:20:36.339504+0000","last_undegraded":"2024-09-17T05:20:36.339504+0000","last_fullsized":"2024-09-17T05:20:36.339504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"68'3526","reported_seq":3630,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393790+0000","last_change":"2024-09-17T04:50:12.072014+0000","last_active":"2024-09-17T05:20:43.393790+0000","last_peered":"2024-09-17T05:20:43.393790+0000","last_clean":"2024-09-17T05:20:43.393790+0000","last_became_active":"2024-09-17T04:50:12.071053+0000","last_became_peered":"2024-09-17T04:50:12.071053+0000","last_unstale":"2024-09-17T05:20:43.393790+0000","last_undegraded":"2024-09-17T05:20:43.393790+0000","last_fullsized":"2024-09-17T05:20:43.393790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20183,"num_write_kb":298078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"68'284","reported_seq":409,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.123738+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T05:20:44.123738+0000","last_peered":"2024-09-17T05:20:44.123738+0000","last_clean":"2024-09-17T05:20:44.123738+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T05:20:44.123738+0000","last_undegraded":"2024-09-17T05:20:44.123738+0000","last_fullsized":"2024-09-17T05:20:44.123738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2099732,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":4077,"num_write":257,"num_write_kb":2207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'3509","reported_seq":3626,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339983+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T05:20:36.339983+0000","last_peered":"2024-09-17T05:20:36.339983+0000","last_clean":"2024-09-17T05:20:36.339983+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T05:20:36.339983+0000","last_undegraded":"2024-09-17T05:20:36.339983+0000","last_fullsized":"2024-09-17T05:20:36.339983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3509,"log_dups_size":0,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18360,"num_write_kb":249727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'5","reported_seq":156,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:39.659577+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T05:20:39.659577+0000","last_peered":"2024-09-17T05:20:39.659577+0000","last_clean":"2024-09-17T05:20:39.659577+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T05:20:39.659577+0000","last_undegraded":"2024-09-17T05:20:39.659577+0000","last_fullsized":"2024-09-17T05:20:39.659577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:27.230604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"68'3113","reported_seq":3225,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.330899+0000","last_change":"2024-09-17T04:49:41.055263+0000","last_active":"2024-09-17T05:20:43.330899+0000","last_peered":"2024-09-17T05:20:43.330899+0000","last_clean":"2024-09-17T05:20:43.330899+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T05:20:43.330899+0000","last_undegraded":"2024-09-17T05:20:43.330899+0000","last_fullsized":"2024-09-17T05:20:43.330899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":16978,"num_write_kb":256690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"65'657","reported_seq":800,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.349177+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T05:20:36.349177+0000","last_peered":"2024-09-17T05:20:36.349177+0000","last_clean":"2024-09-17T05:20:36.349177+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T05:20:36.349177+0000","last_undegraded":"2024-09-17T05:20:36.349177+0000","last_fullsized":"2024-09-17T05:20:36.349177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":79257,"num_write":647,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'3199","reported_seq":3317,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.343729+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T05:20:36.343729+0000","last_peered":"2024-09-17T05:20:36.343729+0000","last_clean":"2024-09-17T05:20:36.343729+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T05:20:36.343729+0000","last_undegraded":"2024-09-17T05:20:36.343729+0000","last_fullsized":"2024-09-17T05:20:36.343729+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16756,"num_write_kb":239128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"68'43","reported_seq":194,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.403635+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T05:20:43.403635+0000","last_peered":"2024-09-17T05:20:43.403635+0000","last_clean":"2024-09-17T05:20:43.403635+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T05:20:43.403635+0000","last_undegraded":"2024-09-17T05:20:43.403635+0000","last_fullsized":"2024-09-17T05:20:43.403635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:24.586834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"68'3014","reported_seq":3125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402775+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T05:20:43.402775+0000","last_peered":"2024-09-17T05:20:43.402775+0000","last_clean":"2024-09-17T05:20:43.402775+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T05:20:43.402775+0000","last_undegraded":"2024-09-17T05:20:43.402775+0000","last_fullsized":"2024-09-17T05:20:43.402775+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3014,"log_dups_size":0,"ondisk_log_size":3014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16762,"num_write_kb":251577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"65'3751","reported_seq":3871,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364101+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T05:20:36.364101+0000","last_peered":"2024-09-17T05:20:36.364101+0000","last_clean":"2024-09-17T05:20:36.364101+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T05:20:36.364101+0000","last_undegraded":"2024-09-17T05:20:36.364101+0000","last_fullsized":"2024-09-17T05:20:36.364101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20040,"num_write_kb":277001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"68'3862","reported_seq":3974,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.271131+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T05:20:43.271131+0000","last_peered":"2024-09-17T05:20:43.271131+0000","last_clean":"2024-09-17T05:20:43.271131+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T05:20:43.271131+0000","last_undegraded":"2024-09-17T05:20:43.271131+0000","last_fullsized":"2024-09-17T05:20:43.271131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20235,"num_write_kb":310497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"65'4193","reported_seq":4314,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.356835+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T05:20:36.356835+0000","last_peered":"2024-09-17T05:20:36.356835+0000","last_clean":"2024-09-17T05:20:36.356835+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T05:20:36.356835+0000","last_undegraded":"2024-09-17T05:20:36.356835+0000","last_fullsized":"2024-09-17T05:20:36.356835+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4193,"log_dups_size":0,"ondisk_log_size":4193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22222,"num_write_kb":306243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"68'3519","reported_seq":3632,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.398772+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T05:20:43.398772+0000","last_peered":"2024-09-17T05:20:43.398772+0000","last_clean":"2024-09-17T05:20:43.398772+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T05:20:43.398772+0000","last_undegraded":"2024-09-17T05:20:43.398772+0000","last_fullsized":"2024-09-17T05:20:43.398772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3519,"log_dups_size":0,"ondisk_log_size":3519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":19548,"num_write_kb":295624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"65'3872","reported_seq":3994,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.360995+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T05:20:36.360995+0000","last_peered":"2024-09-17T05:20:36.360995+0000","last_clean":"2024-09-17T05:20:36.360995+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T05:20:36.360995+0000","last_undegraded":"2024-09-17T05:20:36.360995+0000","last_fullsized":"2024-09-17T05:20:36.360995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3872,"log_dups_size":0,"ondisk_log_size":3872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20576,"num_write_kb":279241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"68'2938","reported_seq":3050,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.380635+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T05:20:43.380635+0000","last_peered":"2024-09-17T05:20:43.380635+0000","last_clean":"2024-09-17T05:20:43.380635+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T05:20:43.380635+0000","last_undegraded":"2024-09-17T05:20:43.380635+0000","last_fullsized":"2024-09-17T05:20:43.380635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2938,"log_dups_size":0,"ondisk_log_size":2938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":16647,"num_write_kb":271506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"65'3500","reported_seq":3620,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.348982+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T05:20:36.348982+0000","last_peered":"2024-09-17T05:20:36.348982+0000","last_clean":"2024-09-17T05:20:36.348982+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T05:20:36.348982+0000","last_undegraded":"2024-09-17T05:20:36.348982+0000","last_fullsized":"2024-09-17T05:20:36.348982+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18562,"num_write_kb":258704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"68'3339","reported_seq":3452,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.406315+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T05:20:43.406315+0000","last_peered":"2024-09-17T05:20:43.406315+0000","last_clean":"2024-09-17T05:20:43.406315+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T05:20:43.406315+0000","last_undegraded":"2024-09-17T05:20:43.406315+0000","last_fullsized":"2024-09-17T05:20:43.406315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3339,"log_dups_size":0,"ondisk_log_size":3339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17940,"num_write_kb":305481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"65'3819","reported_seq":3939,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341730+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T05:20:36.341730+0000","last_peered":"2024-09-17T05:20:36.341730+0000","last_clean":"2024-09-17T05:20:36.341730+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T05:20:36.341730+0000","last_undegraded":"2024-09-17T05:20:36.341730+0000","last_fullsized":"2024-09-17T05:20:36.341730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21007,"num_write_kb":273012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"68'3374","reported_seq":3492,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.366480+0000","last_change":"2024-09-17T04:50:12.076134+0000","last_active":"2024-09-17T05:20:43.366480+0000","last_peered":"2024-09-17T05:20:43.366480+0000","last_clean":"2024-09-17T05:20:43.366480+0000","last_became_active":"2024-09-17T04:50:12.075350+0000","last_became_peered":"2024-09-17T04:50:12.075350+0000","last_unstale":"2024-09-17T05:20:43.366480+0000","last_undegraded":"2024-09-17T05:20:43.366480+0000","last_fullsized":"2024-09-17T05:20:43.366480+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3374,"log_dups_size":0,"ondisk_log_size":3374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18297,"num_write_kb":270032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"65'4233","reported_seq":4352,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341464+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T05:20:36.341464+0000","last_peered":"2024-09-17T05:20:36.341464+0000","last_clean":"2024-09-17T05:20:36.341464+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T05:20:36.341464+0000","last_undegraded":"2024-09-17T05:20:36.341464+0000","last_fullsized":"2024-09-17T05:20:36.341464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23577,"num_write_kb":311694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"68'3382","reported_seq":3494,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.416880+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T05:20:43.416880+0000","last_peered":"2024-09-17T05:20:43.416880+0000","last_clean":"2024-09-17T05:20:43.416880+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T05:20:43.416880+0000","last_undegraded":"2024-09-17T05:20:43.416880+0000","last_fullsized":"2024-09-17T05:20:43.416880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18178,"num_write_kb":293977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"65'3778","reported_seq":3897,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364369+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T05:20:36.364369+0000","last_peered":"2024-09-17T05:20:36.364369+0000","last_clean":"2024-09-17T05:20:36.364369+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T05:20:36.364369+0000","last_undegraded":"2024-09-17T05:20:36.364369+0000","last_fullsized":"2024-09-17T05:20:36.364369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21044,"num_write_kb":281017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"68'2956","reported_seq":3071,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404554+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T05:20:44.404554+0000","last_peered":"2024-09-17T05:20:44.404554+0000","last_clean":"2024-09-17T05:20:44.404554+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T05:20:44.404554+0000","last_undegraded":"2024-09-17T05:20:44.404554+0000","last_fullsized":"2024-09-17T05:20:44.404554+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2956,"log_dups_size":0,"ondisk_log_size":2956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15875,"num_write_kb":264255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"65'4647","reported_seq":4767,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352380+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T05:20:36.352380+0000","last_peered":"2024-09-17T05:20:36.352380+0000","last_clean":"2024-09-17T05:20:36.352380+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T05:20:36.352380+0000","last_undegraded":"2024-09-17T05:20:36.352380+0000","last_fullsized":"2024-09-17T05:20:36.352380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24699,"num_write_kb":336693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"68'3095","reported_seq":3210,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.396708+0000","last_change":"2024-09-17T04:49:41.045502+0000","last_active":"2024-09-17T05:20:43.396708+0000","last_peered":"2024-09-17T05:20:43.396708+0000","last_clean":"2024-09-17T05:20:43.396708+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T05:20:43.396708+0000","last_undegraded":"2024-09-17T05:20:43.396708+0000","last_fullsized":"2024-09-17T05:20:43.396708+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:19.746356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":36,"num_write":16750,"num_write_kb":272041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"65'4070","reported_seq":4190,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.361587+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T05:20:36.361587+0000","last_peered":"2024-09-17T05:20:36.361587+0000","last_clean":"2024-09-17T05:20:36.361587+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T05:20:36.361587+0000","last_undegraded":"2024-09-17T05:20:36.361587+0000","last_fullsized":"2024-09-17T05:20:36.361587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4070,"log_dups_size":0,"ondisk_log_size":4070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21906,"num_write_kb":288644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"68'4276","reported_seq":4387,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404966+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T05:20:44.404966+0000","last_peered":"2024-09-17T05:20:44.404966+0000","last_clean":"2024-09-17T05:20:44.404966+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T05:20:44.404966+0000","last_undegraded":"2024-09-17T05:20:44.404966+0000","last_fullsized":"2024-09-17T05:20:44.404966+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22732,"num_write_kb":365385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"65'4174","reported_seq":4292,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.348275+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T05:20:36.348275+0000","last_peered":"2024-09-17T05:20:36.348275+0000","last_clean":"2024-09-17T05:20:36.348275+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T05:20:36.348275+0000","last_undegraded":"2024-09-17T05:20:36.348275+0000","last_fullsized":"2024-09-17T05:20:36.348275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22870,"num_write_kb":296621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"68'2821","reported_seq":2931,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393189+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T05:20:43.393189+0000","last_peered":"2024-09-17T05:20:43.393189+0000","last_clean":"2024-09-17T05:20:43.393189+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T05:20:43.393189+0000","last_undegraded":"2024-09-17T05:20:43.393189+0000","last_fullsized":"2024-09-17T05:20:43.393189+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2821,"log_dups_size":0,"ondisk_log_size":2821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15110,"num_write_kb":232606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"65'4280","reported_seq":4398,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.343678+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T05:20:36.343678+0000","last_peered":"2024-09-17T05:20:36.343678+0000","last_clean":"2024-09-17T05:20:36.343678+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T05:20:36.343678+0000","last_undegraded":"2024-09-17T05:20:36.343678+0000","last_fullsized":"2024-09-17T05:20:36.343678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4280,"log_dups_size":0,"ondisk_log_size":4280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23007,"num_write_kb":313406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"68'3452","reported_seq":3562,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.303318+0000","last_change":"2024-09-17T04:49:41.038265+0000","last_active":"2024-09-17T05:20:43.303318+0000","last_peered":"2024-09-17T05:20:43.303318+0000","last_clean":"2024-09-17T05:20:43.303318+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T05:20:43.303318+0000","last_undegraded":"2024-09-17T05:20:43.303318+0000","last_fullsized":"2024-09-17T05:20:43.303318+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3452,"log_dups_size":0,"ondisk_log_size":3452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:41.638403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19160,"num_write_kb":306156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"65'4384","reported_seq":4504,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339211+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T05:20:36.339211+0000","last_peered":"2024-09-17T05:20:36.339211+0000","last_clean":"2024-09-17T05:20:36.339211+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T05:20:36.339211+0000","last_undegraded":"2024-09-17T05:20:36.339211+0000","last_fullsized":"2024-09-17T05:20:36.339211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23582,"num_write_kb":336581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"68'3472","reported_seq":3589,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.255838+0000","last_change":"2024-09-17T04:50:12.068348+0000","last_active":"2024-09-17T05:20:43.255838+0000","last_peered":"2024-09-17T05:20:43.255838+0000","last_clean":"2024-09-17T05:20:43.255838+0000","last_became_active":"2024-09-17T04:50:12.067942+0000","last_became_peered":"2024-09-17T04:50:12.067942+0000","last_unstale":"2024-09-17T05:20:43.255838+0000","last_undegraded":"2024-09-17T05:20:43.255838+0000","last_fullsized":"2024-09-17T05:20:43.255838+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3472,"log_dups_size":0,"ondisk_log_size":3472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:33.954020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18777,"num_write_kb":292458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"65'3688","reported_seq":3806,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557618+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T05:20:36.557618+0000","last_peered":"2024-09-17T05:20:36.557618+0000","last_clean":"2024-09-17T05:20:36.557618+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T05:20:36.557618+0000","last_undegraded":"2024-09-17T05:20:36.557618+0000","last_fullsized":"2024-09-17T05:20:36.557618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20358,"num_write_kb":257774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"68'3214","reported_seq":3329,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.416523+0000","last_change":"2024-09-17T04:49:41.038072+0000","last_active":"2024-09-17T05:20:43.416523+0000","last_peered":"2024-09-17T05:20:43.416523+0000","last_clean":"2024-09-17T05:20:43.416523+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T05:20:43.416523+0000","last_undegraded":"2024-09-17T05:20:43.416523+0000","last_fullsized":"2024-09-17T05:20:43.416523+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":17543,"num_write_kb":304029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"65'4068","reported_seq":4186,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374459+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T05:20:36.374459+0000","last_peered":"2024-09-17T05:20:36.374459+0000","last_clean":"2024-09-17T05:20:36.374459+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T05:20:36.374459+0000","last_undegraded":"2024-09-17T05:20:36.374459+0000","last_fullsized":"2024-09-17T05:20:36.374459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21759,"num_write_kb":308395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"68'2551","reported_seq":2663,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402052+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T05:20:43.402052+0000","last_peered":"2024-09-17T05:20:43.402052+0000","last_clean":"2024-09-17T05:20:43.402052+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T05:20:43.402052+0000","last_undegraded":"2024-09-17T05:20:43.402052+0000","last_fullsized":"2024-09-17T05:20:43.402052+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2551,"log_dups_size":0,"ondisk_log_size":2551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13664,"num_write_kb":247232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"65'4369","reported_seq":4488,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.359366+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T05:20:36.359366+0000","last_peered":"2024-09-17T05:20:36.359366+0000","last_clean":"2024-09-17T05:20:36.359366+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T05:20:36.359366+0000","last_undegraded":"2024-09-17T05:20:36.359366+0000","last_fullsized":"2024-09-17T05:20:36.359366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23171,"num_write_kb":318257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"68'3277","reported_seq":3388,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393579+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T05:20:43.393579+0000","last_peered":"2024-09-17T05:20:43.393579+0000","last_clean":"2024-09-17T05:20:43.393579+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T05:20:43.393579+0000","last_undegraded":"2024-09-17T05:20:43.393579+0000","last_fullsized":"2024-09-17T05:20:43.393579+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18007,"num_write_kb":264733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"65'5432","reported_seq":5550,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366356+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T05:20:36.366356+0000","last_peered":"2024-09-17T05:20:36.366356+0000","last_clean":"2024-09-17T05:20:36.366356+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T05:20:36.366356+0000","last_undegraded":"2024-09-17T05:20:36.366356+0000","last_fullsized":"2024-09-17T05:20:36.366356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29730,"num_write_kb":389609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"68'3135","reported_seq":3246,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.377491+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T05:20:43.377491+0000","last_peered":"2024-09-17T05:20:43.377491+0000","last_clean":"2024-09-17T05:20:43.377491+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T05:20:43.377491+0000","last_undegraded":"2024-09-17T05:20:43.377491+0000","last_fullsized":"2024-09-17T05:20:43.377491+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":16884,"num_write_kb":284681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"65'3941","reported_seq":4059,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.360280+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T05:20:36.360280+0000","last_peered":"2024-09-17T05:20:36.360280+0000","last_clean":"2024-09-17T05:20:36.360280+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T05:20:36.360280+0000","last_undegraded":"2024-09-17T05:20:36.360280+0000","last_fullsized":"2024-09-17T05:20:36.360280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21845,"num_write_kb":304681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"68'3913","reported_seq":4026,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.330331+0000","last_change":"2024-09-17T04:49:41.067207+0000","last_active":"2024-09-17T05:20:43.330331+0000","last_peered":"2024-09-17T05:20:43.330331+0000","last_clean":"2024-09-17T05:20:43.330331+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T05:20:43.330331+0000","last_undegraded":"2024-09-17T05:20:43.330331+0000","last_fullsized":"2024-09-17T05:20:43.330331+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3913,"log_dups_size":0,"ondisk_log_size":3913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:08.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":21530,"num_write_kb":364884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"65'3351","reported_seq":3475,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.347528+0000","last_change":"2024-09-17T04:50:12.469782+0000","last_active":"2024-09-17T05:20:36.347528+0000","last_peered":"2024-09-17T05:20:36.347528+0000","last_clean":"2024-09-17T05:20:36.347528+0000","last_became_active":"2024-09-17T04:50:12.469425+0000","last_became_peered":"2024-09-17T04:50:12.469425+0000","last_unstale":"2024-09-17T05:20:36.347528+0000","last_undegraded":"2024-09-17T05:20:36.347528+0000","last_fullsized":"2024-09-17T05:20:36.347528+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:04.098943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18172,"num_write_kb":265354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"68'1954","reported_seq":2066,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.346900+0000","last_change":"2024-09-17T04:49:41.046170+0000","last_active":"2024-09-17T05:20:43.346900+0000","last_peered":"2024-09-17T05:20:43.346900+0000","last_clean":"2024-09-17T05:20:43.346900+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T05:20:43.346900+0000","last_undegraded":"2024-09-17T05:20:43.346900+0000","last_fullsized":"2024-09-17T05:20:43.346900+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":1954,"log_dups_size":0,"ondisk_log_size":1954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:01.683826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":34,"num_write":10478,"num_write_kb":181241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"65'3661","reported_seq":3779,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.344111+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T05:20:36.344111+0000","last_peered":"2024-09-17T05:20:36.344111+0000","last_clean":"2024-09-17T05:20:36.344111+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T05:20:36.344111+0000","last_undegraded":"2024-09-17T05:20:36.344111+0000","last_fullsized":"2024-09-17T05:20:36.344111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19087,"num_write_kb":273280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"68'2981","reported_seq":3093,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.203029+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T05:20:43.203029+0000","last_peered":"2024-09-17T05:20:43.203029+0000","last_clean":"2024-09-17T05:20:43.203029+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T05:20:43.203029+0000","last_undegraded":"2024-09-17T05:20:43.203029+0000","last_fullsized":"2024-09-17T05:20:43.203029+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:25.574682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":15573,"num_write_kb":271248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"65'3297","reported_seq":3414,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340729+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T05:20:36.340729+0000","last_peered":"2024-09-17T05:20:36.340729+0000","last_clean":"2024-09-17T05:20:36.340729+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T05:20:36.340729+0000","last_undegraded":"2024-09-17T05:20:36.340729+0000","last_fullsized":"2024-09-17T05:20:36.340729+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3297,"log_dups_size":0,"ondisk_log_size":3297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17851,"num_write_kb":243868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"68'2872","reported_seq":2983,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403678+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T05:20:44.403678+0000","last_peered":"2024-09-17T05:20:44.403678+0000","last_clean":"2024-09-17T05:20:44.403678+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T05:20:44.403678+0000","last_undegraded":"2024-09-17T05:20:44.403678+0000","last_fullsized":"2024-09-17T05:20:44.403678+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2872,"log_dups_size":0,"ondisk_log_size":2872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15784,"num_write_kb":259984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"65'2574","reported_seq":2691,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.556549+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T05:20:36.556549+0000","last_peered":"2024-09-17T05:20:36.556549+0000","last_clean":"2024-09-17T05:20:36.556549+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T05:20:36.556549+0000","last_undegraded":"2024-09-17T05:20:36.556549+0000","last_fullsized":"2024-09-17T05:20:36.556549+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":2574,"log_dups_size":0,"ondisk_log_size":2574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14007,"num_write_kb":203729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"68'2459","reported_seq":2572,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393891+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T05:20:43.393891+0000","last_peered":"2024-09-17T05:20:43.393891+0000","last_clean":"2024-09-17T05:20:43.393891+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T05:20:43.393891+0000","last_undegraded":"2024-09-17T05:20:43.393891+0000","last_fullsized":"2024-09-17T05:20:43.393891+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2459,"log_dups_size":0,"ondisk_log_size":2459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13572,"num_write_kb":222508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"65'3245","reported_seq":3362,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.556406+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T05:20:36.556406+0000","last_peered":"2024-09-17T05:20:36.556406+0000","last_clean":"2024-09-17T05:20:36.556406+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T05:20:36.556406+0000","last_undegraded":"2024-09-17T05:20:36.556406+0000","last_fullsized":"2024-09-17T05:20:36.556406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3245,"log_dups_size":0,"ondisk_log_size":3245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16903,"num_write_kb":256080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"68'3074","reported_seq":3184,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393584+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T05:20:43.393584+0000","last_peered":"2024-09-17T05:20:43.393584+0000","last_clean":"2024-09-17T05:20:43.393584+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T05:20:43.393584+0000","last_undegraded":"2024-09-17T05:20:43.393584+0000","last_fullsized":"2024-09-17T05:20:43.393584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16752,"num_write_kb":271957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"65'4387","reported_seq":4504,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557565+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T05:20:36.557565+0000","last_peered":"2024-09-17T05:20:36.557565+0000","last_clean":"2024-09-17T05:20:36.557565+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T05:20:36.557565+0000","last_undegraded":"2024-09-17T05:20:36.557565+0000","last_fullsized":"2024-09-17T05:20:36.557565+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24001,"num_write_kb":317505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"68'2713","reported_seq":2824,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.417489+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T05:20:43.417489+0000","last_peered":"2024-09-17T05:20:43.417489+0000","last_clean":"2024-09-17T05:20:43.417489+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T05:20:43.417489+0000","last_undegraded":"2024-09-17T05:20:43.417489+0000","last_fullsized":"2024-09-17T05:20:43.417489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2713,"log_dups_size":0,"ondisk_log_size":2713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14521,"num_write_kb":262890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"65'4656","reported_seq":4775,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.347514+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T05:20:36.347514+0000","last_peered":"2024-09-17T05:20:36.347514+0000","last_clean":"2024-09-17T05:20:36.347514+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T05:20:36.347514+0000","last_undegraded":"2024-09-17T05:20:36.347514+0000","last_fullsized":"2024-09-17T05:20:36.347514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25090,"num_write_kb":362900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"68'3588","reported_seq":3702,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.402501+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T05:20:44.402501+0000","last_peered":"2024-09-17T05:20:44.402501+0000","last_clean":"2024-09-17T05:20:44.402501+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T05:20:44.402501+0000","last_undegraded":"2024-09-17T05:20:44.402501+0000","last_fullsized":"2024-09-17T05:20:44.402501+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3588,"log_dups_size":0,"ondisk_log_size":3588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":19474,"num_write_kb":319707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"65'4149","reported_seq":4266,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.347526+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T05:20:36.347526+0000","last_peered":"2024-09-17T05:20:36.347526+0000","last_clean":"2024-09-17T05:20:36.347526+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T05:20:36.347526+0000","last_undegraded":"2024-09-17T05:20:36.347526+0000","last_fullsized":"2024-09-17T05:20:36.347526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22453,"num_write_kb":298991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"68'3299","reported_seq":3415,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.422884+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T05:20:44.422884+0000","last_peered":"2024-09-17T05:20:44.422884+0000","last_clean":"2024-09-17T05:20:44.422884+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T05:20:44.422884+0000","last_undegraded":"2024-09-17T05:20:44.422884+0000","last_fullsized":"2024-09-17T05:20:44.422884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3299,"log_dups_size":0,"ondisk_log_size":3299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":17863,"num_write_kb":288721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"65'3910","reported_seq":4027,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.555930+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T05:20:36.555930+0000","last_peered":"2024-09-17T05:20:36.555930+0000","last_clean":"2024-09-17T05:20:36.555930+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T05:20:36.555930+0000","last_undegraded":"2024-09-17T05:20:36.555930+0000","last_fullsized":"2024-09-17T05:20:36.555930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21371,"num_write_kb":294687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"68'2856","reported_seq":2969,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.325969+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T05:20:43.325969+0000","last_peered":"2024-09-17T05:20:43.325969+0000","last_clean":"2024-09-17T05:20:43.325969+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T05:20:43.325969+0000","last_undegraded":"2024-09-17T05:20:43.325969+0000","last_fullsized":"2024-09-17T05:20:43.325969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15916,"num_write_kb":235941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"65'3148","reported_seq":3265,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.373270+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T05:20:36.373270+0000","last_peered":"2024-09-17T05:20:36.373270+0000","last_clean":"2024-09-17T05:20:36.373270+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T05:20:36.373270+0000","last_undegraded":"2024-09-17T05:20:36.373270+0000","last_fullsized":"2024-09-17T05:20:36.373270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16748,"num_write_kb":239238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"68'3516","reported_seq":3630,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.409318+0000","last_change":"2024-09-17T04:49:41.041201+0000","last_active":"2024-09-17T05:20:43.409318+0000","last_peered":"2024-09-17T05:20:43.409318+0000","last_clean":"2024-09-17T05:20:43.409318+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T05:20:43.409318+0000","last_undegraded":"2024-09-17T05:20:43.409318+0000","last_fullsized":"2024-09-17T05:20:43.409318+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:02.588786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18785,"num_write_kb":280186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"65'3294","reported_seq":3411,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558447+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T05:20:36.558447+0000","last_peered":"2024-09-17T05:20:36.558447+0000","last_clean":"2024-09-17T05:20:36.558447+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T05:20:36.558447+0000","last_undegraded":"2024-09-17T05:20:36.558447+0000","last_fullsized":"2024-09-17T05:20:36.558447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16934,"num_write_kb":260475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"68'2875","reported_seq":2985,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.359568+0000","last_change":"2024-09-17T04:49:41.061092+0000","last_active":"2024-09-17T05:20:43.359568+0000","last_peered":"2024-09-17T05:20:43.359568+0000","last_clean":"2024-09-17T05:20:43.359568+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T05:20:43.359568+0000","last_undegraded":"2024-09-17T05:20:43.359568+0000","last_fullsized":"2024-09-17T05:20:43.359568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2875,"log_dups_size":0,"ondisk_log_size":2875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:52.714614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15268,"num_write_kb":241386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"65'3071","reported_seq":3192,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.370583+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T05:20:36.370583+0000","last_peered":"2024-09-17T05:20:36.370583+0000","last_clean":"2024-09-17T05:20:36.370583+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T05:20:36.370583+0000","last_undegraded":"2024-09-17T05:20:36.370583+0000","last_fullsized":"2024-09-17T05:20:36.370583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3071,"log_dups_size":0,"ondisk_log_size":3071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16593,"num_write_kb":218157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"68'2617","reported_seq":2728,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.405858+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T05:20:43.405858+0000","last_peered":"2024-09-17T05:20:43.405858+0000","last_clean":"2024-09-17T05:20:43.405858+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T05:20:43.405858+0000","last_undegraded":"2024-09-17T05:20:43.405858+0000","last_fullsized":"2024-09-17T05:20:43.405858+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3,"num_write":13428,"num_write_kb":250535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"65'3866","reported_seq":3987,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352182+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T05:20:36.352182+0000","last_peered":"2024-09-17T05:20:36.352182+0000","last_clean":"2024-09-17T05:20:36.352182+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T05:20:36.352182+0000","last_undegraded":"2024-09-17T05:20:36.352182+0000","last_fullsized":"2024-09-17T05:20:36.352182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3866,"log_dups_size":0,"ondisk_log_size":3866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20832,"num_write_kb":272078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"68'3056","reported_seq":3165,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.326456+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T05:20:43.326456+0000","last_peered":"2024-09-17T05:20:43.326456+0000","last_clean":"2024-09-17T05:20:43.326456+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T05:20:43.326456+0000","last_undegraded":"2024-09-17T05:20:43.326456+0000","last_fullsized":"2024-09-17T05:20:43.326456+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3056,"log_dups_size":0,"ondisk_log_size":3056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16712,"num_write_kb":275987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"65'4485","reported_seq":4605,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.351407+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T05:20:36.351407+0000","last_peered":"2024-09-17T05:20:36.351407+0000","last_clean":"2024-09-17T05:20:36.351407+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T05:20:36.351407+0000","last_undegraded":"2024-09-17T05:20:36.351407+0000","last_fullsized":"2024-09-17T05:20:36.351407+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23824,"num_write_kb":337279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"68'3229","reported_seq":3338,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.420569+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T05:20:44.420569+0000","last_peered":"2024-09-17T05:20:44.420569+0000","last_clean":"2024-09-17T05:20:44.420569+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T05:20:44.420569+0000","last_undegraded":"2024-09-17T05:20:44.420569+0000","last_fullsized":"2024-09-17T05:20:44.420569+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3229,"log_dups_size":0,"ondisk_log_size":3229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17855,"num_write_kb":268347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"65'3523","reported_seq":3644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559155+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T05:20:36.559155+0000","last_peered":"2024-09-17T05:20:36.559155+0000","last_clean":"2024-09-17T05:20:36.559155+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T05:20:36.559155+0000","last_undegraded":"2024-09-17T05:20:36.559155+0000","last_fullsized":"2024-09-17T05:20:36.559155+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3523,"log_dups_size":0,"ondisk_log_size":3523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19890,"num_write_kb":260121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":321,"num_write":1077532,"num_write_kb":17428719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":198824,"ondisk_log_size":198824,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":18310618,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1391,"num_read_kb":207657,"num_write":4849,"num_write_kb":19832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36683776,"data_stored":36621236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4439,"ondisk_log_size":4439,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1352682,"num_write_kb":18536494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250516,"ondisk_log_size":250516,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":18963064,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1535,"num_read_kb":230833,"num_write":4911,"num_write_kb":20583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37986304,"data_stored":37926128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4503,"ondisk_log_size":4503,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509823,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533064,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":532094,"kb_avail":93838776,"statfs":{"total":96636764160,"available":96090906624,"internally_reserved":0,"allocated":974848,"data_stored":552256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":544864714},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509823,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762124,"kb_used_data":17916,"kb_used_omap":1,"kb_used_meta":744190,"kb_avail":93609716,"statfs":{"total":96636764160,"available":95856349184,"internally_reserved":0,"allocated":18345984,"data_stored":17920368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":762051015},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509823,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756036,"kb_used_data":17268,"kb_used_omap":1,"kb_used_meta":738750,"kb_avail":93615804,"statfs":{"total":96636764160,"available":95862583296,"internally_reserved":0,"allocated":17682432,"data_stored":17246718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756480453},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509823,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755412,"kb_used_data":15300,"kb_used_omap":1,"kb_used_meta":740094,"kb_avail":93616428,"statfs":{"total":96636764160,"available":95863222272,"internally_reserved":0,"allocated":15667200,"data_stored":15236011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":757856711},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509823,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877392,"kb_used_data":14656,"kb_used_omap":1,"kb_used_meta":862718,"kb_avail":93494448,"statfs":{"total":96636764160,"available":95738314752,"internally_reserved":0,"allocated":15007744,"data_stored":14583014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":883423685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509824,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630324,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":628862,"kb_avail":93741516,"statfs":{"total":96636764160,"available":95991312384,"internally_reserved":0,"allocated":1478656,"data_stored":1057972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":643955141},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509823,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":819832,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":816894,"kb_avail":93552008,"statfs":{"total":96636764160,"available":95797256192,"internally_reserved":0,"allocated":2990080,"data_stored":2567342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":836499909},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542527,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694200,"kb_used_data":10472,"kb_used_omap":1,"kb_used_meta":683710,"kb_avail":93677640,"statfs":{"total":96636764160,"available":95925903360,"internally_reserved":0,"allocated":10723328,"data_stored":10304131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":700119494},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2099788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10493952,"data_stored":10488922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4279042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":84770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5726270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4279625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5652480,"data_stored":5642363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:20:58.737 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T05:20:58.737 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:20:58.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.932+0000 7f424e2f5640 1 -- 172.21.15.150:0/2642165705 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3db0 msgr2=0x7f42480ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.932+0000 7f424e2f5640 1 --2- 172.21.15.150:0/2642165705 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3db0 0x7f42480ad2a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f424806f7a0 tx=0x7f423801c8e0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.932+0000 7f424e2f5640 1 -- 172.21.15.150:0/2642165705 shutdown_connections 2024-09-17T05:20:58.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.932+0000 7f424e2f5640 1 --2- 172.21.15.150:0/2642165705 >> v2:172.21.15.176:3300/0 conn(0x7f42480c41c0 0x7f42480ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.932+0000 7f424e2f5640 1 --2- 172.21.15.150:0/2642165705 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3db0 0x7f42480ad2a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.932+0000 7f424e2f5640 1 -- 172.21.15.150:0/2642165705 >> 172.21.15.150:0/2642165705 conn(0x7f42480c4a00 msgr2=0x7f42480c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:58.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.932+0000 7f424e2f5640 1 -- 172.21.15.150:0/2642165705 shutdown_connections 2024-09-17T05:20:58.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.932+0000 7f424e2f5640 1 -- 172.21.15.150:0/2642165705 wait complete. 2024-09-17T05:20:58.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424e2f5640 1 Processor -- start 2024-09-17T05:20:58.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424e2f5640 1 -- start start 2024-09-17T05:20:58.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424e2f5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f424806fab0 0x7f42480bdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:58.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424e2f5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3db0 0x7f42480bcf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:58.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424e2f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42480c41c0 0x7f424814d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:58.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424e2f5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4248070750 con 0x7f42480c41c0 2024-09-17T05:20:58.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424e2f5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f42480705d0 con 0x7f424806fab0 2024-09-17T05:20:58.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424e2f5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f42480a75f0 con 0x7f42480c3db0 2024-09-17T05:20:58.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424c892640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42480c41c0 0x7f424814d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:58.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424c892640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42480c41c0 0x7f424814d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33208/0 (socket says 172.21.15.150:33208) 2024-09-17T05:20:58.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424c892640 1 -- 172.21.15.150:0/3669310409 learned_addr learned my addr 172.21.15.150:0/3669310409 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:58.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f4247fff640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f424806fab0 0x7f42480bdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:58.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f42477fe640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3db0 0x7f42480bcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:58.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424c892640 1 -- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3db0 msgr2=0x7f42480bcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424c892640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3db0 0x7f42480bcf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424c892640 1 -- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f424806fab0 msgr2=0x7f42480bdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:58.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424c892640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f424806fab0 0x7f42480bdd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:58.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424c892640 1 -- 172.21.15.150:0/3669310409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424814e000 con 0x7f42480c41c0 2024-09-17T05:20:58.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.936+0000 7f424c892640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42480c41c0 0x7f424814d480 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f424000c790 tx=0x7f424000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:58.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.940+0000 7f4244ff9640 1 -- 172.21.15.150:0/3669310409 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4240003b80 con 0x7f42480c41c0 2024-09-17T05:20:58.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.940+0000 7f4244ff9640 1 -- 172.21.15.150:0/3669310409 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4240003d20 con 0x7f42480c41c0 2024-09-17T05:20:58.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.940+0000 7f4244ff9640 1 -- 172.21.15.150:0/3669310409 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424000aab0 con 0x7f42480c41c0 2024-09-17T05:20:58.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.940+0000 7f424e2f5640 1 -- 172.21.15.150:0/3669310409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4248151e80 con 0x7f42480c41c0 2024-09-17T05:20:58.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.940+0000 7f424e2f5640 1 -- 172.21.15.150:0/3669310409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42480a9680 con 0x7f42480c41c0 2024-09-17T05:20:58.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.940+0000 7f424e2f5640 1 -- 172.21.15.150:0/3669310409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4248151ca0 con 0x7f42480c41c0 2024-09-17T05:20:58.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.940+0000 7f4244ff9640 1 -- 172.21.15.150:0/3669310409 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f424000ac50 con 0x7f42480c41c0 2024-09-17T05:20:58.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.940+0000 7f4244ff9640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4234073200 0x7f4234075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:58.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.940+0000 7f4244ff9640 1 -- 172.21.15.150:0/3669310409 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f4240094fc0 con 0x7f42480c41c0 2024-09-17T05:20:58.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.940+0000 7f4247fff640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4234073200 0x7f4234075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:58.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.940+0000 7f4247fff640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4234073200 0x7f4234075710 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f42380af000 tx=0x7f423801b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:58.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:58.944+0000 7f4244ff9640 1 -- 172.21.15.150:0/3669310409 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4248151ca0 con 0x7f42480c41c0 2024-09-17T05:20:59.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.112+0000 7f424e2f5640 1 -- 172.21.15.150:0/3669310409 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f424806fec0 con 0x7f4234073200 2024-09-17T05:20:59.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.136+0000 7f4244ff9640 1 -- 172.21.15.150:0/3669310409 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493818 (secure 0 0 0) 0x7f424806fec0 con 0x7f4234073200 2024-09-17T05:20:59.139 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:59.143 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T05:20:59.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.144+0000 7f424e2f5640 1 -- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4234073200 msgr2=0x7f4234075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:59.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.144+0000 7f424e2f5640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4234073200 0x7f4234075710 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f42380af000 tx=0x7f423801b120 comp rx=0 tx=0).stop 2024-09-17T05:20:59.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.144+0000 7f424e2f5640 1 -- 172.21.15.150:0/3669310409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42480c41c0 msgr2=0x7f424814d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:59.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.144+0000 7f424e2f5640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42480c41c0 0x7f424814d480 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f424000c790 tx=0x7f424000cc50 comp rx=0 tx=0).stop 2024-09-17T05:20:59.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.144+0000 7f424e2f5640 1 -- 172.21.15.150:0/3669310409 shutdown_connections 2024-09-17T05:20:59.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.144+0000 7f424e2f5640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4234073200 0x7f4234075710 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.144+0000 7f424e2f5640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f42480c41c0 0x7f424814d480 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.144+0000 7f424e2f5640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f42480c3db0 0x7f42480bcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.144+0000 7f424e2f5640 1 --2- 172.21.15.150:0/3669310409 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f424806fab0 0x7f42480bdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.144+0000 7f424e2f5640 1 -- 172.21.15.150:0/3669310409 >> 172.21.15.150:0/3669310409 conn(0x7f42480c4a00 msgr2=0x7f42480c69c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:59.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.144+0000 7f424e2f5640 1 -- 172.21.15.150:0/3669310409 shutdown_connections 2024-09-17T05:20:59.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.144+0000 7f424e2f5640 1 -- 172.21.15.150:0/3669310409 wait complete. 2024-09-17T05:20:59.163 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":1010,"stamp":"2024-09-17T05:20:57.882067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37864050,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3107,"num_read_kb":438866,"num_write":2440075,"num_write_kb":36007256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":458354,"ondisk_log_size":458354,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5828384,"kb_used_data":80928,"kb_used_omap":12,"kb_used_meta":5747315,"kb_avail":749146336,"statfs":{"total":773094113280,"available":767125848064,"internally_reserved":0,"allocated":82870272,"data_stored":79467812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":5885251122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7484347531,"num_objects":-1955,"num_object_clones":0,"num_object_copies":-3910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1955,"num_whiteouts":0,"num_read":10,"num_read_kb":30,"num_write":2076,"num_write_kb":727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006566"},"pg_stats":[{"pgid":"5.18","version":"68'3479","reported_seq":3591,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.418967+0000","last_change":"2024-09-17T04:49:41.025271+0000","last_active":"2024-09-17T05:20:44.418967+0000","last_peered":"2024-09-17T05:20:44.418967+0000","last_clean":"2024-09-17T05:20:44.418967+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T05:20:44.418967+0000","last_undegraded":"2024-09-17T05:20:44.418967+0000","last_fullsized":"2024-09-17T05:20:44.418967+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3479,"log_dups_size":0,"ondisk_log_size":3479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:39.715602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":12,"num_write":18686,"num_write_kb":287786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"65'1001","reported_seq":1175,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340557+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T05:20:36.340557+0000","last_peered":"2024-09-17T05:20:36.340557+0000","last_clean":"2024-09-17T05:20:36.340557+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T05:20:36.340557+0000","last_undegraded":"2024-09-17T05:20:36.340557+0000","last_fullsized":"2024-09-17T05:20:36.340557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":20774,"num_write":978,"num_write_kb":4679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'4064","reported_seq":4184,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.351711+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T05:20:36.351711+0000","last_peered":"2024-09-17T05:20:36.351711+0000","last_clean":"2024-09-17T05:20:36.351711+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T05:20:36.351711+0000","last_undegraded":"2024-09-17T05:20:36.351711+0000","last_fullsized":"2024-09-17T05:20:36.351711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22217,"num_write_kb":281701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"68'3","reported_seq":114,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402753+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T05:20:43.402753+0000","last_peered":"2024-09-17T05:20:43.402753+0000","last_clean":"2024-09-17T05:20:43.402753+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T05:20:43.402753+0000","last_undegraded":"2024-09-17T05:20:43.402753+0000","last_fullsized":"2024-09-17T05:20:43.402753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:38.417840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"68'3721","reported_seq":3833,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.413711+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T05:20:43.413711+0000","last_peered":"2024-09-17T05:20:43.413711+0000","last_clean":"2024-09-17T05:20:43.413711+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T05:20:43.413711+0000","last_undegraded":"2024-09-17T05:20:43.413711+0000","last_fullsized":"2024-09-17T05:20:43.413711+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3721,"log_dups_size":0,"ondisk_log_size":3721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":22,"num_write":19557,"num_write_kb":316044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"67'26","reported_seq":148,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352348+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T05:20:36.352348+0000","last_peered":"2024-09-17T05:20:36.352348+0000","last_clean":"2024-09-17T05:20:36.352348+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T05:20:36.352348+0000","last_undegraded":"2024-09-17T05:20:36.352348+0000","last_fullsized":"2024-09-17T05:20:36.352348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4998","reported_seq":5118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.353392+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T05:20:36.353392+0000","last_peered":"2024-09-17T05:20:36.353392+0000","last_clean":"2024-09-17T05:20:36.353392+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T05:20:36.353392+0000","last_undegraded":"2024-09-17T05:20:36.353392+0000","last_fullsized":"2024-09-17T05:20:36.353392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27336,"num_write_kb":388205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'2","reported_seq":113,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362495+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T05:20:36.362495+0000","last_peered":"2024-09-17T05:20:36.362495+0000","last_clean":"2024-09-17T05:20:36.362495+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T05:20:36.362495+0000","last_undegraded":"2024-09-17T05:20:36.362495+0000","last_fullsized":"2024-09-17T05:20:36.362495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"68'2765","reported_seq":2876,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.349402+0000","last_change":"2024-09-17T04:49:41.067317+0000","last_active":"2024-09-17T05:20:43.349402+0000","last_peered":"2024-09-17T05:20:43.349402+0000","last_clean":"2024-09-17T05:20:43.349402+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T05:20:43.349402+0000","last_undegraded":"2024-09-17T05:20:43.349402+0000","last_fullsized":"2024-09-17T05:20:43.349402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:37.637477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":14571,"num_write_kb":258996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"68'46","reported_seq":207,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558188+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T05:20:36.558188+0000","last_peered":"2024-09-17T05:20:36.558188+0000","last_clean":"2024-09-17T05:20:36.558188+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T05:20:36.558188+0000","last_undegraded":"2024-09-17T05:20:36.558188+0000","last_fullsized":"2024-09-17T05:20:36.558188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"65'3140","reported_seq":3259,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.346435+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T05:20:36.346435+0000","last_peered":"2024-09-17T05:20:36.346435+0000","last_clean":"2024-09-17T05:20:36.346435+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T05:20:36.346435+0000","last_undegraded":"2024-09-17T05:20:36.346435+0000","last_fullsized":"2024-09-17T05:20:36.346435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3140,"log_dups_size":0,"ondisk_log_size":3140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16962,"num_write_kb":252282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"68'751","reported_seq":888,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.388359+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T05:20:43.388359+0000","last_peered":"2024-09-17T05:20:43.388359+0000","last_clean":"2024-09-17T05:20:43.388359+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T05:20:43.388359+0000","last_undegraded":"2024-09-17T05:20:43.388359+0000","last_fullsized":"2024-09-17T05:20:43.388359+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":75812,"num_write":739,"num_write_kb":4324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"68'2700","reported_seq":2810,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.180944+0000","last_change":"2024-09-17T04:49:41.043531+0000","last_active":"2024-09-17T05:20:43.180944+0000","last_peered":"2024-09-17T05:20:43.180944+0000","last_clean":"2024-09-17T05:20:43.180944+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T05:20:43.180944+0000","last_undegraded":"2024-09-17T05:20:43.180944+0000","last_fullsized":"2024-09-17T05:20:43.180944+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2700,"log_dups_size":0,"ondisk_log_size":2700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":5,"num_write":13438,"num_write_kb":225755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"65'5","reported_seq":166,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559425+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T05:20:36.559425+0000","last_peered":"2024-09-17T05:20:36.559425+0000","last_clean":"2024-09-17T05:20:36.559425+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T05:20:36.559425+0000","last_undegraded":"2024-09-17T05:20:36.559425+0000","last_fullsized":"2024-09-17T05:20:36.559425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"65'4481","reported_seq":4598,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.373776+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T05:20:36.373776+0000","last_peered":"2024-09-17T05:20:36.373776+0000","last_clean":"2024-09-17T05:20:36.373776+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T05:20:36.373776+0000","last_undegraded":"2024-09-17T05:20:36.373776+0000","last_fullsized":"2024-09-17T05:20:36.373776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24963,"num_write_kb":346034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"68'143","reported_seq":256,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:51.881072+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T05:20:51.881072+0000","last_peered":"2024-09-17T05:20:51.881072+0000","last_clean":"2024-09-17T05:20:51.881072+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T05:20:51.881072+0000","last_undegraded":"2024-09-17T05:20:51.881072+0000","last_fullsized":"2024-09-17T05:20:51.881072+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:16.854365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1447387,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":383,"num_write":117,"num_write_kb":1479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"68'2942","reported_seq":3054,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.409983+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T05:20:44.409983+0000","last_peered":"2024-09-17T05:20:44.409983+0000","last_clean":"2024-09-17T05:20:44.409983+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T05:20:44.409983+0000","last_undegraded":"2024-09-17T05:20:44.409983+0000","last_fullsized":"2024-09-17T05:20:44.409983+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":16254,"num_write_kb":278162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"65'3968","reported_seq":4087,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362298+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T05:20:36.362298+0000","last_peered":"2024-09-17T05:20:36.362298+0000","last_clean":"2024-09-17T05:20:36.362298+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T05:20:36.362298+0000","last_undegraded":"2024-09-17T05:20:36.362298+0000","last_fullsized":"2024-09-17T05:20:36.362298+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3968,"log_dups_size":0,"ondisk_log_size":3968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21939,"num_write_kb":297018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"68'1467","reported_seq":1636,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366269+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T05:20:36.366269+0000","last_peered":"2024-09-17T05:20:36.366269+0000","last_clean":"2024-09-17T05:20:36.366269+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T05:20:36.366269+0000","last_undegraded":"2024-09-17T05:20:36.366269+0000","last_fullsized":"2024-09-17T05:20:36.366269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1467,"log_dups_size":0,"ondisk_log_size":1467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":43333,"num_write":1444,"num_write_kb":4521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'6","reported_seq":138,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.115470+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T05:20:43.115470+0000","last_peered":"2024-09-17T05:20:43.115470+0000","last_clean":"2024-09-17T05:20:43.115470+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T05:20:43.115470+0000","last_undegraded":"2024-09-17T05:20:43.115470+0000","last_fullsized":"2024-09-17T05:20:43.115470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:28.365399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'3236","reported_seq":3349,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.400322+0000","last_change":"2024-09-17T04:49:41.038415+0000","last_active":"2024-09-17T05:20:44.400322+0000","last_peered":"2024-09-17T05:20:44.400322+0000","last_clean":"2024-09-17T05:20:44.400322+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T05:20:44.400322+0000","last_undegraded":"2024-09-17T05:20:44.400322+0000","last_fullsized":"2024-09-17T05:20:44.400322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3236,"log_dups_size":0,"ondisk_log_size":3236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:17.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":17542,"num_write_kb":271055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"65'3303","reported_seq":3421,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.358141+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T05:20:36.358141+0000","last_peered":"2024-09-17T05:20:36.358141+0000","last_clean":"2024-09-17T05:20:36.358141+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T05:20:36.358141+0000","last_undegraded":"2024-09-17T05:20:36.358141+0000","last_fullsized":"2024-09-17T05:20:36.358141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3303,"log_dups_size":0,"ondisk_log_size":3303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17463,"num_write_kb":238014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"65'3","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.344536+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T05:20:36.344536+0000","last_peered":"2024-09-17T05:20:36.344536+0000","last_clean":"2024-09-17T05:20:36.344536+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T05:20:36.344536+0000","last_undegraded":"2024-09-17T05:20:36.344536+0000","last_fullsized":"2024-09-17T05:20:36.344536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"68'5","reported_seq":117,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.405924+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T05:20:43.405924+0000","last_peered":"2024-09-17T05:20:43.405924+0000","last_clean":"2024-09-17T05:20:43.405924+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T05:20:43.405924+0000","last_undegraded":"2024-09-17T05:20:43.405924+0000","last_fullsized":"2024-09-17T05:20:43.405924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:17.424265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"68'2691","reported_seq":2802,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.414897+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T05:20:43.414897+0000","last_peered":"2024-09-17T05:20:43.414897+0000","last_clean":"2024-09-17T05:20:43.414897+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T05:20:43.414897+0000","last_undegraded":"2024-09-17T05:20:43.414897+0000","last_fullsized":"2024-09-17T05:20:43.414897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:06.516432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":14365,"num_write_kb":225564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"65'3640","reported_seq":3759,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.337791+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T05:20:36.337791+0000","last_peered":"2024-09-17T05:20:36.337791+0000","last_clean":"2024-09-17T05:20:36.337791+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T05:20:36.337791+0000","last_undegraded":"2024-09-17T05:20:36.337791+0000","last_fullsized":"2024-09-17T05:20:36.337791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3640,"log_dups_size":0,"ondisk_log_size":3640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20183,"num_write_kb":277678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"65'2","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340129+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T05:20:36.340129+0000","last_peered":"2024-09-17T05:20:36.340129+0000","last_clean":"2024-09-17T05:20:36.340129+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T05:20:36.340129+0000","last_undegraded":"2024-09-17T05:20:36.340129+0000","last_fullsized":"2024-09-17T05:20:36.340129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"68'4","reported_seq":136,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.390191+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T05:20:43.390191+0000","last_peered":"2024-09-17T05:20:43.390191+0000","last_clean":"2024-09-17T05:20:43.390191+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T05:20:43.390191+0000","last_undegraded":"2024-09-17T05:20:43.390191+0000","last_fullsized":"2024-09-17T05:20:43.390191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:00.010365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"68'3943","reported_seq":4053,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.367267+0000","last_change":"2024-09-17T04:49:41.036658+0000","last_active":"2024-09-17T05:20:43.367267+0000","last_peered":"2024-09-17T05:20:43.367267+0000","last_clean":"2024-09-17T05:20:43.367267+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T05:20:43.367267+0000","last_undegraded":"2024-09-17T05:20:43.367267+0000","last_fullsized":"2024-09-17T05:20:43.367267+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21088,"num_write_kb":359062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"65'3699","reported_seq":3818,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364687+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T05:20:36.364687+0000","last_peered":"2024-09-17T05:20:36.364687+0000","last_clean":"2024-09-17T05:20:36.364687+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T05:20:36.364687+0000","last_undegraded":"2024-09-17T05:20:36.364687+0000","last_fullsized":"2024-09-17T05:20:36.364687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19661,"num_write_kb":271466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"65'5","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.555326+0000","last_change":"2024-09-17T04:49:30.891069+0000","last_active":"2024-09-17T05:20:36.555326+0000","last_peered":"2024-09-17T05:20:36.555326+0000","last_clean":"2024-09-17T05:20:36.555326+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T05:20:36.555326+0000","last_undegraded":"2024-09-17T05:20:36.555326+0000","last_fullsized":"2024-09-17T05:20:36.555326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:15.915206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"68'7","reported_seq":159,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.387118+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T05:20:43.387118+0000","last_peered":"2024-09-17T05:20:43.387118+0000","last_clean":"2024-09-17T05:20:43.387118+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T05:20:43.387118+0000","last_undegraded":"2024-09-17T05:20:43.387118+0000","last_fullsized":"2024-09-17T05:20:43.387118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"68'2532","reported_seq":2644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.411846+0000","last_change":"2024-09-17T04:49:41.050170+0000","last_active":"2024-09-17T05:20:43.411846+0000","last_peered":"2024-09-17T05:20:43.411846+0000","last_clean":"2024-09-17T05:20:43.411846+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T05:20:43.411846+0000","last_undegraded":"2024-09-17T05:20:43.411846+0000","last_fullsized":"2024-09-17T05:20:43.411846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2532,"log_dups_size":0,"ondisk_log_size":2532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13954,"num_write_kb":219876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"65'4598","reported_seq":4718,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339224+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T05:20:36.339224+0000","last_peered":"2024-09-17T05:20:36.339224+0000","last_clean":"2024-09-17T05:20:36.339224+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T05:20:36.339224+0000","last_undegraded":"2024-09-17T05:20:36.339224+0000","last_fullsized":"2024-09-17T05:20:36.339224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24313,"num_write_kb":335123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"65'9","reported_seq":172,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.367024+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T05:20:36.367024+0000","last_peered":"2024-09-17T05:20:36.367024+0000","last_clean":"2024-09-17T05:20:36.367024+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T05:20:36.367024+0000","last_undegraded":"2024-09-17T05:20:36.367024+0000","last_fullsized":"2024-09-17T05:20:36.367024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'50","reported_seq":229,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:48.374529+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T05:20:48.374529+0000","last_peered":"2024-09-17T05:20:48.374529+0000","last_clean":"2024-09-17T05:20:48.374529+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T05:20:48.374529+0000","last_undegraded":"2024-09-17T05:20:48.374529+0000","last_fullsized":"2024-09-17T05:20:48.374529+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":65,"num_read_kb":114,"num_write":45,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"68'2531","reported_seq":2644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.376480+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T05:20:43.376480+0000","last_peered":"2024-09-17T05:20:43.376480+0000","last_clean":"2024-09-17T05:20:43.376480+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T05:20:43.376480+0000","last_undegraded":"2024-09-17T05:20:43.376480+0000","last_fullsized":"2024-09-17T05:20:43.376480+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2531,"log_dups_size":0,"ondisk_log_size":2531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13881,"num_write_kb":219504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"65'3884","reported_seq":4003,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340033+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T05:20:36.340033+0000","last_peered":"2024-09-17T05:20:36.340033+0000","last_clean":"2024-09-17T05:20:36.340033+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T05:20:36.340033+0000","last_undegraded":"2024-09-17T05:20:36.340033+0000","last_fullsized":"2024-09-17T05:20:36.340033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21342,"num_write_kb":257935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"68'473","reported_seq":615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.328144+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T05:20:44.328144+0000","last_peered":"2024-09-17T05:20:44.328144+0000","last_clean":"2024-09-17T05:20:44.328144+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T05:20:44.328144+0000","last_undegraded":"2024-09-17T05:20:44.328144+0000","last_fullsized":"2024-09-17T05:20:44.328144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":57,"num_write":1105,"num_write_kb":359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'11","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410351+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T05:20:43.410351+0000","last_peered":"2024-09-17T05:20:43.410351+0000","last_clean":"2024-09-17T05:20:43.410351+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T05:20:43.410351+0000","last_undegraded":"2024-09-17T05:20:43.410351+0000","last_fullsized":"2024-09-17T05:20:43.410351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"68'2691","reported_seq":2802,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.217861+0000","last_change":"2024-09-17T04:49:41.042314+0000","last_active":"2024-09-17T05:20:43.217861+0000","last_peered":"2024-09-17T05:20:43.217861+0000","last_clean":"2024-09-17T05:20:43.217861+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T05:20:43.217861+0000","last_undegraded":"2024-09-17T05:20:43.217861+0000","last_fullsized":"2024-09-17T05:20:43.217861+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14724,"num_write_kb":243278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"65'4637","reported_seq":4757,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341853+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T05:20:36.341853+0000","last_peered":"2024-09-17T05:20:36.341853+0000","last_clean":"2024-09-17T05:20:36.341853+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T05:20:36.341853+0000","last_undegraded":"2024-09-17T05:20:36.341853+0000","last_fullsized":"2024-09-17T05:20:36.341853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25111,"num_write_kb":342484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"65'2","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.375220+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T05:20:36.375220+0000","last_peered":"2024-09-17T05:20:36.375220+0000","last_clean":"2024-09-17T05:20:36.375220+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T05:20:36.375220+0000","last_undegraded":"2024-09-17T05:20:36.375220+0000","last_fullsized":"2024-09-17T05:20:36.375220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'3","reported_seq":117,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393956+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T05:20:43.393956+0000","last_peered":"2024-09-17T05:20:43.393956+0000","last_clean":"2024-09-17T05:20:43.393956+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T05:20:43.393956+0000","last_undegraded":"2024-09-17T05:20:43.393956+0000","last_fullsized":"2024-09-17T05:20:43.393956+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"68'3511","reported_seq":3623,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.341640+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T05:20:43.341640+0000","last_peered":"2024-09-17T05:20:43.341640+0000","last_clean":"2024-09-17T05:20:43.341640+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T05:20:43.341640+0000","last_undegraded":"2024-09-17T05:20:43.341640+0000","last_fullsized":"2024-09-17T05:20:43.341640+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3511,"log_dups_size":0,"ondisk_log_size":3511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19215,"num_write_kb":314590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"65'3495","reported_seq":3615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362299+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T05:20:36.362299+0000","last_peered":"2024-09-17T05:20:36.362299+0000","last_clean":"2024-09-17T05:20:36.362299+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T05:20:36.362299+0000","last_undegraded":"2024-09-17T05:20:36.362299+0000","last_fullsized":"2024-09-17T05:20:36.362299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18740,"num_write_kb":251346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"65'2","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.351220+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T05:20:36.351220+0000","last_peered":"2024-09-17T05:20:36.351220+0000","last_clean":"2024-09-17T05:20:36.351220+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T05:20:36.351220+0000","last_undegraded":"2024-09-17T05:20:36.351220+0000","last_fullsized":"2024-09-17T05:20:36.351220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'4","reported_seq":118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.112153+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T05:20:43.112153+0000","last_peered":"2024-09-17T05:20:43.112153+0000","last_clean":"2024-09-17T05:20:43.112153+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T05:20:43.112153+0000","last_undegraded":"2024-09-17T05:20:43.112153+0000","last_fullsized":"2024-09-17T05:20:43.112153+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"68'2923","reported_seq":3035,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.395059+0000","last_change":"2024-09-17T04:49:41.069177+0000","last_active":"2024-09-17T05:20:43.395059+0000","last_peered":"2024-09-17T05:20:43.395059+0000","last_clean":"2024-09-17T05:20:43.395059+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T05:20:43.395059+0000","last_undegraded":"2024-09-17T05:20:43.395059+0000","last_fullsized":"2024-09-17T05:20:43.395059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:16.855206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":7,"num_write":15908,"num_write_kb":260386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"65'4495","reported_seq":4615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559390+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T05:20:36.559390+0000","last_peered":"2024-09-17T05:20:36.559390+0000","last_clean":"2024-09-17T05:20:36.559390+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T05:20:36.559390+0000","last_undegraded":"2024-09-17T05:20:36.559390+0000","last_fullsized":"2024-09-17T05:20:36.559390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24287,"num_write_kb":330898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"65'4","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.365137+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T05:20:36.365137+0000","last_peered":"2024-09-17T05:20:36.365137+0000","last_clean":"2024-09-17T05:20:36.365137+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T05:20:36.365137+0000","last_undegraded":"2024-09-17T05:20:36.365137+0000","last_fullsized":"2024-09-17T05:20:36.365137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'2","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.401317+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T05:20:43.401317+0000","last_peered":"2024-09-17T05:20:43.401317+0000","last_clean":"2024-09-17T05:20:43.401317+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T05:20:43.401317+0000","last_undegraded":"2024-09-17T05:20:43.401317+0000","last_fullsized":"2024-09-17T05:20:43.401317+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"68'3959","reported_seq":4074,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.362853+0000","last_change":"2024-09-17T04:49:41.068537+0000","last_active":"2024-09-17T05:20:43.362853+0000","last_peered":"2024-09-17T05:20:43.362853+0000","last_clean":"2024-09-17T05:20:43.362853+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T05:20:43.362853+0000","last_undegraded":"2024-09-17T05:20:43.362853+0000","last_fullsized":"2024-09-17T05:20:43.362853+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3959,"log_dups_size":0,"ondisk_log_size":3959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":13,"num_write":21335,"num_write_kb":323115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'3351","reported_seq":3471,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.348480+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T05:20:36.348480+0000","last_peered":"2024-09-17T05:20:36.348480+0000","last_clean":"2024-09-17T05:20:36.348480+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T05:20:36.348480+0000","last_undegraded":"2024-09-17T05:20:36.348480+0000","last_fullsized":"2024-09-17T05:20:36.348480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18427,"num_write_kb":241864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"68'53","reported_seq":243,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.366344+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T05:20:44.366344+0000","last_peered":"2024-09-17T05:20:44.366344+0000","last_clean":"2024-09-17T05:20:44.366344+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T05:20:44.366344+0000","last_undegraded":"2024-09-17T05:20:44.366344+0000","last_fullsized":"2024-09-17T05:20:44.366344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":116,"num_write":48,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'9","reported_seq":162,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.386330+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T05:20:44.386330+0000","last_peered":"2024-09-17T05:20:44.386330+0000","last_clean":"2024-09-17T05:20:44.386330+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T05:20:44.386330+0000","last_undegraded":"2024-09-17T05:20:44.386330+0000","last_fullsized":"2024-09-17T05:20:44.386330+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"68'2708","reported_seq":2819,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403990+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T05:20:44.403990+0000","last_peered":"2024-09-17T05:20:44.403990+0000","last_clean":"2024-09-17T05:20:44.403990+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T05:20:44.403990+0000","last_undegraded":"2024-09-17T05:20:44.403990+0000","last_fullsized":"2024-09-17T05:20:44.403990+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14599,"num_write_kb":234639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"65'3456","reported_seq":3576,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339164+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T05:20:36.339164+0000","last_peered":"2024-09-17T05:20:36.339164+0000","last_clean":"2024-09-17T05:20:36.339164+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T05:20:36.339164+0000","last_undegraded":"2024-09-17T05:20:36.339164+0000","last_fullsized":"2024-09-17T05:20:36.339164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18921,"num_write_kb":260292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"67'13","reported_seq":135,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366577+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T05:20:36.366577+0000","last_peered":"2024-09-17T05:20:36.366577+0000","last_clean":"2024-09-17T05:20:36.366577+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T05:20:36.366577+0000","last_undegraded":"2024-09-17T05:20:36.366577+0000","last_fullsized":"2024-09-17T05:20:36.366577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"68'473","reported_seq":606,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:47.345918+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T05:20:47.345918+0000","last_peered":"2024-09-17T05:20:47.345918+0000","last_clean":"2024-09-17T05:20:47.345918+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T05:20:47.345918+0000","last_undegraded":"2024-09-17T05:20:47.345918+0000","last_fullsized":"2024-09-17T05:20:47.345918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":54,"num_write":1100,"num_write_kb":354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"68'3699","reported_seq":3811,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.383257+0000","last_change":"2024-09-17T04:49:41.052564+0000","last_active":"2024-09-17T05:20:43.383257+0000","last_peered":"2024-09-17T05:20:43.383257+0000","last_clean":"2024-09-17T05:20:43.383257+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T05:20:43.383257+0000","last_undegraded":"2024-09-17T05:20:43.383257+0000","last_fullsized":"2024-09-17T05:20:43.383257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":20021,"num_write_kb":309708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"65'3651","reported_seq":3771,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341142+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T05:20:36.341142+0000","last_peered":"2024-09-17T05:20:36.341142+0000","last_clean":"2024-09-17T05:20:36.341142+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T05:20:36.341142+0000","last_undegraded":"2024-09-17T05:20:36.341142+0000","last_fullsized":"2024-09-17T05:20:36.341142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19517,"num_write_kb":274922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"65'3","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366726+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T05:20:36.366726+0000","last_peered":"2024-09-17T05:20:36.366726+0000","last_clean":"2024-09-17T05:20:36.366726+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T05:20:36.366726+0000","last_undegraded":"2024-09-17T05:20:36.366726+0000","last_fullsized":"2024-09-17T05:20:36.366726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":218,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.343214+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T05:20:36.343214+0000","last_peered":"2024-09-17T05:20:36.343214+0000","last_clean":"2024-09-17T05:20:36.343214+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T05:20:36.343214+0000","last_undegraded":"2024-09-17T05:20:36.343214+0000","last_fullsized":"2024-09-17T05:20:36.343214+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'2","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374244+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T05:20:36.374244+0000","last_peered":"2024-09-17T05:20:36.374244+0000","last_clean":"2024-09-17T05:20:36.374244+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T05:20:36.374244+0000","last_undegraded":"2024-09-17T05:20:36.374244+0000","last_fullsized":"2024-09-17T05:20:36.374244+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:21.202398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"68'2130","reported_seq":2241,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404345+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T05:20:44.404345+0000","last_peered":"2024-09-17T05:20:44.404345+0000","last_clean":"2024-09-17T05:20:44.404345+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T05:20:44.404345+0000","last_undegraded":"2024-09-17T05:20:44.404345+0000","last_fullsized":"2024-09-17T05:20:44.404345+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2130,"log_dups_size":0,"ondisk_log_size":2130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:48.128609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":11413,"num_write_kb":185062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"65'4183","reported_seq":4301,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374398+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T05:20:36.374398+0000","last_peered":"2024-09-17T05:20:36.374398+0000","last_clean":"2024-09-17T05:20:36.374398+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T05:20:36.374398+0000","last_undegraded":"2024-09-17T05:20:36.374398+0000","last_fullsized":"2024-09-17T05:20:36.374398+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22326,"num_write_kb":302603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"65'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.342324+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T05:20:36.342324+0000","last_peered":"2024-09-17T05:20:36.342324+0000","last_clean":"2024-09-17T05:20:36.342324+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T05:20:36.342324+0000","last_undegraded":"2024-09-17T05:20:36.342324+0000","last_fullsized":"2024-09-17T05:20:36.342324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"68'3","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.376325+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T05:20:43.376325+0000","last_peered":"2024-09-17T05:20:43.376325+0000","last_clean":"2024-09-17T05:20:43.376325+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T05:20:43.376325+0000","last_undegraded":"2024-09-17T05:20:43.376325+0000","last_fullsized":"2024-09-17T05:20:43.376325+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"68'3285","reported_seq":3397,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.413822+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T05:20:43.413822+0000","last_peered":"2024-09-17T05:20:43.413822+0000","last_clean":"2024-09-17T05:20:43.413822+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T05:20:43.413822+0000","last_undegraded":"2024-09-17T05:20:43.413822+0000","last_fullsized":"2024-09-17T05:20:43.413822+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3285,"log_dups_size":0,"ondisk_log_size":3285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18245,"num_write_kb":282158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"65'4407","reported_seq":4525,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558055+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T05:20:36.558055+0000","last_peered":"2024-09-17T05:20:36.558055+0000","last_clean":"2024-09-17T05:20:36.558055+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T05:20:36.558055+0000","last_undegraded":"2024-09-17T05:20:36.558055+0000","last_fullsized":"2024-09-17T05:20:36.558055+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24558,"num_write_kb":315810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"65'6","reported_seq":147,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352587+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T05:20:36.352587+0000","last_peered":"2024-09-17T05:20:36.352587+0000","last_clean":"2024-09-17T05:20:36.352587+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T05:20:36.352587+0000","last_undegraded":"2024-09-17T05:20:36.352587+0000","last_fullsized":"2024-09-17T05:20:36.352587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'1445","reported_seq":1601,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410628+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T05:20:43.410628+0000","last_peered":"2024-09-17T05:20:43.410628+0000","last_clean":"2024-09-17T05:20:43.410628+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T05:20:43.410628+0000","last_undegraded":"2024-09-17T05:20:43.410628+0000","last_fullsized":"2024-09-17T05:20:43.410628+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:18.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":38239,"num_write":1421,"num_write_kb":4489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'2587","reported_seq":2698,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.177381+0000","last_change":"2024-09-17T04:49:41.031036+0000","last_active":"2024-09-17T05:20:43.177381+0000","last_peered":"2024-09-17T05:20:43.177381+0000","last_clean":"2024-09-17T05:20:43.177381+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T05:20:43.177381+0000","last_undegraded":"2024-09-17T05:20:43.177381+0000","last_fullsized":"2024-09-17T05:20:43.177381+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2587,"log_dups_size":0,"ondisk_log_size":2587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:48.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13558,"num_write_kb":238553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"65'4762","reported_seq":4880,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559085+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T05:20:36.559085+0000","last_peered":"2024-09-17T05:20:36.559085+0000","last_clean":"2024-09-17T05:20:36.559085+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T05:20:36.559085+0000","last_undegraded":"2024-09-17T05:20:36.559085+0000","last_fullsized":"2024-09-17T05:20:36.559085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25049,"num_write_kb":354398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"65'7","reported_seq":169,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340155+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T05:20:36.340155+0000","last_peered":"2024-09-17T05:20:36.340155+0000","last_clean":"2024-09-17T05:20:36.340155+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T05:20:36.340155+0000","last_undegraded":"2024-09-17T05:20:36.340155+0000","last_fullsized":"2024-09-17T05:20:36.340155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"68'5","reported_seq":118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.116235+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T05:20:43.116235+0000","last_peered":"2024-09-17T05:20:43.116235+0000","last_clean":"2024-09-17T05:20:43.116235+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T05:20:43.116235+0000","last_undegraded":"2024-09-17T05:20:43.116235+0000","last_fullsized":"2024-09-17T05:20:43.116235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"68'2327","reported_seq":2438,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.108349+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T05:20:43.108349+0000","last_peered":"2024-09-17T05:20:43.108349+0000","last_clean":"2024-09-17T05:20:43.108349+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T05:20:43.108349+0000","last_undegraded":"2024-09-17T05:20:43.108349+0000","last_fullsized":"2024-09-17T05:20:43.108349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2327,"log_dups_size":0,"ondisk_log_size":2327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12892,"num_write_kb":211370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"65'3722","reported_seq":3840,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366989+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T05:20:36.366989+0000","last_peered":"2024-09-17T05:20:36.366989+0000","last_clean":"2024-09-17T05:20:36.366989+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T05:20:36.366989+0000","last_undegraded":"2024-09-17T05:20:36.366989+0000","last_fullsized":"2024-09-17T05:20:36.366989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3722,"log_dups_size":0,"ondisk_log_size":3722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19724,"num_write_kb":263684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"65'4","reported_seq":145,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.560132+0000","last_change":"2024-09-17T04:49:30.902357+0000","last_active":"2024-09-17T05:20:36.560132+0000","last_peered":"2024-09-17T05:20:36.560132+0000","last_clean":"2024-09-17T05:20:36.560132+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T05:20:36.560132+0000","last_undegraded":"2024-09-17T05:20:36.560132+0000","last_fullsized":"2024-09-17T05:20:36.560132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"68'2","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.385249+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T05:20:43.385249+0000","last_peered":"2024-09-17T05:20:43.385249+0000","last_clean":"2024-09-17T05:20:43.385249+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T05:20:43.385249+0000","last_undegraded":"2024-09-17T05:20:43.385249+0000","last_fullsized":"2024-09-17T05:20:43.385249+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:09.712272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"68'3186","reported_seq":3295,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.381181+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T05:20:43.381181+0000","last_peered":"2024-09-17T05:20:43.381181+0000","last_clean":"2024-09-17T05:20:43.381181+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T05:20:43.381181+0000","last_undegraded":"2024-09-17T05:20:43.381181+0000","last_fullsized":"2024-09-17T05:20:43.381181+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:43.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16763,"num_write_kb":264037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"65'3694","reported_seq":3814,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362961+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T05:20:36.362961+0000","last_peered":"2024-09-17T05:20:36.362961+0000","last_clean":"2024-09-17T05:20:36.362961+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T05:20:36.362961+0000","last_undegraded":"2024-09-17T05:20:36.362961+0000","last_fullsized":"2024-09-17T05:20:36.362961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20762,"num_write_kb":284135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"68'15","reported_seq":223,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.342372+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T05:20:36.342372+0000","last_peered":"2024-09-17T05:20:36.342372+0000","last_clean":"2024-09-17T05:20:36.342372+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T05:20:36.342372+0000","last_undegraded":"2024-09-17T05:20:36.342372+0000","last_fullsized":"2024-09-17T05:20:36.342372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":264,"num_read_kb":264,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"68'3","reported_seq":114,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.394770+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T05:20:43.394770+0000","last_peered":"2024-09-17T05:20:43.394770+0000","last_clean":"2024-09-17T05:20:43.394770+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T05:20:43.394770+0000","last_undegraded":"2024-09-17T05:20:43.394770+0000","last_fullsized":"2024-09-17T05:20:43.394770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:52.364568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"68'3074","reported_seq":3186,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.395538+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T05:20:44.395538+0000","last_peered":"2024-09-17T05:20:44.395538+0000","last_clean":"2024-09-17T05:20:44.395538+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T05:20:44.395538+0000","last_undegraded":"2024-09-17T05:20:44.395538+0000","last_fullsized":"2024-09-17T05:20:44.395538+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:03.537392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":16509,"num_write_kb":287321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'2828","reported_seq":2946,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.342437+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T05:20:36.342437+0000","last_peered":"2024-09-17T05:20:36.342437+0000","last_clean":"2024-09-17T05:20:36.342437+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T05:20:36.342437+0000","last_undegraded":"2024-09-17T05:20:36.342437+0000","last_fullsized":"2024-09-17T05:20:36.342437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14821,"num_write_kb":206837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"65'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557382+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T05:20:36.557382+0000","last_peered":"2024-09-17T05:20:36.557382+0000","last_clean":"2024-09-17T05:20:36.557382+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T05:20:36.557382+0000","last_undegraded":"2024-09-17T05:20:36.557382+0000","last_fullsized":"2024-09-17T05:20:36.557382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"68'5","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.388852+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T05:20:43.388852+0000","last_peered":"2024-09-17T05:20:43.388852+0000","last_clean":"2024-09-17T05:20:43.388852+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T05:20:43.388852+0000","last_undegraded":"2024-09-17T05:20:43.388852+0000","last_fullsized":"2024-09-17T05:20:43.388852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:52.120543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"68'2590","reported_seq":2699,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.324869+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T05:20:43.324869+0000","last_peered":"2024-09-17T05:20:43.324869+0000","last_clean":"2024-09-17T05:20:43.324869+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T05:20:43.324869+0000","last_undegraded":"2024-09-17T05:20:43.324869+0000","last_fullsized":"2024-09-17T05:20:43.324869+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:56.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14722,"num_write_kb":243326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'4461","reported_seq":4579,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364760+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T05:20:36.364760+0000","last_peered":"2024-09-17T05:20:36.364760+0000","last_clean":"2024-09-17T05:20:36.364760+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T05:20:36.364760+0000","last_undegraded":"2024-09-17T05:20:36.364760+0000","last_fullsized":"2024-09-17T05:20:36.364760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4461,"log_dups_size":0,"ondisk_log_size":4461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24667,"num_write_kb":332445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"65'3","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364408+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T05:20:36.364408+0000","last_peered":"2024-09-17T05:20:36.364408+0000","last_clean":"2024-09-17T05:20:36.364408+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T05:20:36.364408+0000","last_undegraded":"2024-09-17T05:20:36.364408+0000","last_fullsized":"2024-09-17T05:20:36.364408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"68'367","reported_seq":579,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:39.638931+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T05:20:39.638931+0000","last_peered":"2024-09-17T05:20:39.638931+0000","last_clean":"2024-09-17T05:20:39.638931+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T05:20:39.638931+0000","last_undegraded":"2024-09-17T05:20:39.638931+0000","last_fullsized":"2024-09-17T05:20:39.638931+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":180,"num_read_kb":82080,"num_write":364,"num_write_kb":4295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"68'3856","reported_seq":3961,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403743+0000","last_change":"2024-09-17T04:50:12.067788+0000","last_active":"2024-09-17T05:20:44.403743+0000","last_peered":"2024-09-17T05:20:44.403743+0000","last_clean":"2024-09-17T05:20:44.403743+0000","last_became_active":"2024-09-17T04:50:12.066741+0000","last_became_peered":"2024-09-17T04:50:12.066741+0000","last_unstale":"2024-09-17T05:20:44.403743+0000","last_undegraded":"2024-09-17T05:20:44.403743+0000","last_fullsized":"2024-09-17T05:20:44.403743+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3856,"log_dups_size":0,"ondisk_log_size":3856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":20645,"num_write_kb":344954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"65'3521","reported_seq":3639,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.344504+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T05:20:36.344504+0000","last_peered":"2024-09-17T05:20:36.344504+0000","last_clean":"2024-09-17T05:20:36.344504+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T05:20:36.344504+0000","last_undegraded":"2024-09-17T05:20:36.344504+0000","last_fullsized":"2024-09-17T05:20:36.344504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3521,"log_dups_size":0,"ondisk_log_size":3521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19198,"num_write_kb":253218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"67'10","reported_seq":172,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.358352+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T05:20:36.358352+0000","last_peered":"2024-09-17T05:20:36.358352+0000","last_clean":"2024-09-17T05:20:36.358352+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T05:20:36.358352+0000","last_undegraded":"2024-09-17T05:20:36.358352+0000","last_fullsized":"2024-09-17T05:20:36.358352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":168,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"68'30","reported_seq":181,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:45.378073+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T05:20:45.378073+0000","last_peered":"2024-09-17T05:20:45.378073+0000","last_clean":"2024-09-17T05:20:45.378073+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T05:20:45.378073+0000","last_undegraded":"2024-09-17T05:20:45.378073+0000","last_fullsized":"2024-09-17T05:20:45.378073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:07.945355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":31,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"68'3085","reported_seq":3195,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.412893+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T05:20:43.412893+0000","last_peered":"2024-09-17T05:20:43.412893+0000","last_clean":"2024-09-17T05:20:43.412893+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T05:20:43.412893+0000","last_undegraded":"2024-09-17T05:20:43.412893+0000","last_fullsized":"2024-09-17T05:20:43.412893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16824,"num_write_kb":288541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"65'3611","reported_seq":3729,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.556980+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T05:20:36.556980+0000","last_peered":"2024-09-17T05:20:36.556980+0000","last_clean":"2024-09-17T05:20:36.556980+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T05:20:36.556980+0000","last_undegraded":"2024-09-17T05:20:36.556980+0000","last_fullsized":"2024-09-17T05:20:36.556980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3611,"log_dups_size":0,"ondisk_log_size":3611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19975,"num_write_kb":264442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"68'349","reported_seq":575,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.121847+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T05:20:44.121847+0000","last_peered":"2024-09-17T05:20:44.121847+0000","last_clean":"2024-09-17T05:20:44.121847+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T05:20:44.121847+0000","last_undegraded":"2024-09-17T05:20:44.121847+0000","last_fullsized":"2024-09-17T05:20:44.121847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":190,"num_read_kb":82094,"num_write":345,"num_write_kb":4278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"68'3","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.396510+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T05:20:44.396510+0000","last_peered":"2024-09-17T05:20:44.396510+0000","last_clean":"2024-09-17T05:20:44.396510+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T05:20:44.396510+0000","last_undegraded":"2024-09-17T05:20:44.396510+0000","last_fullsized":"2024-09-17T05:20:44.396510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:16.447835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"68'2950","reported_seq":3061,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.342183+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T05:20:43.342183+0000","last_peered":"2024-09-17T05:20:43.342183+0000","last_clean":"2024-09-17T05:20:43.342183+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T05:20:43.342183+0000","last_undegraded":"2024-09-17T05:20:43.342183+0000","last_fullsized":"2024-09-17T05:20:43.342183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":16110,"num_write_kb":249022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"65'4529","reported_seq":4647,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.361047+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T05:20:36.361047+0000","last_peered":"2024-09-17T05:20:36.361047+0000","last_clean":"2024-09-17T05:20:36.361047+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T05:20:36.361047+0000","last_undegraded":"2024-09-17T05:20:36.361047+0000","last_fullsized":"2024-09-17T05:20:36.361047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23581,"num_write_kb":327974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"68'31","reported_seq":214,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.399687+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T05:20:44.399687+0000","last_peered":"2024-09-17T05:20:44.399687+0000","last_clean":"2024-09-17T05:20:44.399687+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T05:20:44.399687+0000","last_undegraded":"2024-09-17T05:20:44.399687+0000","last_fullsized":"2024-09-17T05:20:44.399687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'8","reported_seq":160,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.415231+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T05:20:43.415231+0000","last_peered":"2024-09-17T05:20:43.415231+0000","last_clean":"2024-09-17T05:20:43.415231+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T05:20:43.415231+0000","last_undegraded":"2024-09-17T05:20:43.415231+0000","last_fullsized":"2024-09-17T05:20:43.415231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"68'2811","reported_seq":2924,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.378555+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T05:20:43.378555+0000","last_peered":"2024-09-17T05:20:43.378555+0000","last_clean":"2024-09-17T05:20:43.378555+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T05:20:43.378555+0000","last_undegraded":"2024-09-17T05:20:43.378555+0000","last_fullsized":"2024-09-17T05:20:43.378555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2811,"log_dups_size":0,"ondisk_log_size":2811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15712,"num_write_kb":243392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"65'4109","reported_seq":4226,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.353428+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T05:20:36.353428+0000","last_peered":"2024-09-17T05:20:36.353428+0000","last_clean":"2024-09-17T05:20:36.353428+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T05:20:36.353428+0000","last_undegraded":"2024-09-17T05:20:36.353428+0000","last_fullsized":"2024-09-17T05:20:36.353428+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21556,"num_write_kb":307833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"65'3","reported_seq":122,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558714+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T05:20:36.558714+0000","last_peered":"2024-09-17T05:20:36.558714+0000","last_clean":"2024-09-17T05:20:36.558714+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T05:20:36.558714+0000","last_undegraded":"2024-09-17T05:20:36.558714+0000","last_fullsized":"2024-09-17T05:20:36.558714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'15","reported_seq":209,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410016+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T05:20:43.410016+0000","last_peered":"2024-09-17T05:20:43.410016+0000","last_clean":"2024-09-17T05:20:43.410016+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T05:20:43.410016+0000","last_undegraded":"2024-09-17T05:20:43.410016+0000","last_fullsized":"2024-09-17T05:20:43.410016+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:40.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":246,"num_read_kb":246,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"68'2542","reported_seq":2652,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:42.865170+0000","last_change":"2024-09-17T04:49:41.040278+0000","last_active":"2024-09-17T05:20:42.865170+0000","last_peered":"2024-09-17T05:20:42.865170+0000","last_clean":"2024-09-17T05:20:42.865170+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T05:20:42.865170+0000","last_undegraded":"2024-09-17T05:20:42.865170+0000","last_fullsized":"2024-09-17T05:20:42.865170+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2542,"log_dups_size":0,"ondisk_log_size":2542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.895223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13327,"num_write_kb":225533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"65'3736","reported_seq":3854,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.365811+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T05:20:36.365811+0000","last_peered":"2024-09-17T05:20:36.365811+0000","last_clean":"2024-09-17T05:20:36.365811+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T05:20:36.365811+0000","last_undegraded":"2024-09-17T05:20:36.365811+0000","last_fullsized":"2024-09-17T05:20:36.365811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20513,"num_write_kb":287089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"68'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.363961+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T05:20:44.363961+0000","last_peered":"2024-09-17T05:20:44.363961+0000","last_clean":"2024-09-17T05:20:44.363961+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T05:20:44.363961+0000","last_undegraded":"2024-09-17T05:20:44.363961+0000","last_fullsized":"2024-09-17T05:20:44.363961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"68'5","reported_seq":117,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.409527+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T05:20:43.409527+0000","last_peered":"2024-09-17T05:20:43.409527+0000","last_clean":"2024-09-17T05:20:43.409527+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T05:20:43.409527+0000","last_undegraded":"2024-09-17T05:20:43.409527+0000","last_fullsized":"2024-09-17T05:20:43.409527+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:15.845612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"68'4175","reported_seq":4291,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.426185+0000","last_change":"2024-09-17T04:49:41.050551+0000","last_active":"2024-09-17T05:20:44.426185+0000","last_peered":"2024-09-17T05:20:44.426185+0000","last_clean":"2024-09-17T05:20:44.426185+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T05:20:44.426185+0000","last_undegraded":"2024-09-17T05:20:44.426185+0000","last_fullsized":"2024-09-17T05:20:44.426185+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:11.088230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":40,"num_write":23028,"num_write_kb":362813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"68'25","reported_seq":139,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.384829+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T05:20:43.384829+0000","last_peered":"2024-09-17T05:20:43.384829+0000","last_clean":"2024-09-17T05:20:43.384829+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T05:20:43.384829+0000","last_undegraded":"2024-09-17T05:20:43.384829+0000","last_fullsized":"2024-09-17T05:20:43.384829+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:37.778756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"65'4339","reported_seq":4456,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.338640+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T05:20:36.338640+0000","last_peered":"2024-09-17T05:20:36.338640+0000","last_clean":"2024-09-17T05:20:36.338640+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T05:20:36.338640+0000","last_undegraded":"2024-09-17T05:20:36.338640+0000","last_fullsized":"2024-09-17T05:20:36.338640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23134,"num_write_kb":338604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"65'2","reported_seq":121,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366244+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T05:20:36.366244+0000","last_peered":"2024-09-17T05:20:36.366244+0000","last_clean":"2024-09-17T05:20:36.366244+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T05:20:36.366244+0000","last_undegraded":"2024-09-17T05:20:36.366244+0000","last_fullsized":"2024-09-17T05:20:36.366244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"68'2877","reported_seq":2989,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.395024+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T05:20:43.395024+0000","last_peered":"2024-09-17T05:20:43.395024+0000","last_clean":"2024-09-17T05:20:43.395024+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T05:20:43.395024+0000","last_undegraded":"2024-09-17T05:20:43.395024+0000","last_fullsized":"2024-09-17T05:20:43.395024+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2877,"log_dups_size":0,"ondisk_log_size":2877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15865,"num_write_kb":272557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"68'1000","reported_seq":1162,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.429369+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T05:20:44.429369+0000","last_peered":"2024-09-17T05:20:44.429369+0000","last_clean":"2024-09-17T05:20:44.429369+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T05:20:44.429369+0000","last_undegraded":"2024-09-17T05:20:44.429369+0000","last_fullsized":"2024-09-17T05:20:44.429369+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":9969,"num_write":977,"num_write_kb":4635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"65'4230","reported_seq":4347,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.349109+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T05:20:36.349109+0000","last_peered":"2024-09-17T05:20:36.349109+0000","last_clean":"2024-09-17T05:20:36.349109+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T05:20:36.349109+0000","last_undegraded":"2024-09-17T05:20:36.349109+0000","last_fullsized":"2024-09-17T05:20:36.349109+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22609,"num_write_kb":319990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"65'3","reported_seq":122,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339504+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T05:20:36.339504+0000","last_peered":"2024-09-17T05:20:36.339504+0000","last_clean":"2024-09-17T05:20:36.339504+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T05:20:36.339504+0000","last_undegraded":"2024-09-17T05:20:36.339504+0000","last_fullsized":"2024-09-17T05:20:36.339504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"68'3526","reported_seq":3630,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393790+0000","last_change":"2024-09-17T04:50:12.072014+0000","last_active":"2024-09-17T05:20:43.393790+0000","last_peered":"2024-09-17T05:20:43.393790+0000","last_clean":"2024-09-17T05:20:43.393790+0000","last_became_active":"2024-09-17T04:50:12.071053+0000","last_became_peered":"2024-09-17T04:50:12.071053+0000","last_unstale":"2024-09-17T05:20:43.393790+0000","last_undegraded":"2024-09-17T05:20:43.393790+0000","last_fullsized":"2024-09-17T05:20:43.393790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20183,"num_write_kb":298078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"68'284","reported_seq":409,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.123738+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T05:20:44.123738+0000","last_peered":"2024-09-17T05:20:44.123738+0000","last_clean":"2024-09-17T05:20:44.123738+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T05:20:44.123738+0000","last_undegraded":"2024-09-17T05:20:44.123738+0000","last_fullsized":"2024-09-17T05:20:44.123738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2099732,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":4077,"num_write":257,"num_write_kb":2207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'3509","reported_seq":3626,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339983+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T05:20:36.339983+0000","last_peered":"2024-09-17T05:20:36.339983+0000","last_clean":"2024-09-17T05:20:36.339983+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T05:20:36.339983+0000","last_undegraded":"2024-09-17T05:20:36.339983+0000","last_fullsized":"2024-09-17T05:20:36.339983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3509,"log_dups_size":0,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18360,"num_write_kb":249727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'5","reported_seq":156,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:39.659577+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T05:20:39.659577+0000","last_peered":"2024-09-17T05:20:39.659577+0000","last_clean":"2024-09-17T05:20:39.659577+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T05:20:39.659577+0000","last_undegraded":"2024-09-17T05:20:39.659577+0000","last_fullsized":"2024-09-17T05:20:39.659577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:27.230604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"68'3113","reported_seq":3225,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.330899+0000","last_change":"2024-09-17T04:49:41.055263+0000","last_active":"2024-09-17T05:20:43.330899+0000","last_peered":"2024-09-17T05:20:43.330899+0000","last_clean":"2024-09-17T05:20:43.330899+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T05:20:43.330899+0000","last_undegraded":"2024-09-17T05:20:43.330899+0000","last_fullsized":"2024-09-17T05:20:43.330899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":16978,"num_write_kb":256690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"65'657","reported_seq":800,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.349177+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T05:20:36.349177+0000","last_peered":"2024-09-17T05:20:36.349177+0000","last_clean":"2024-09-17T05:20:36.349177+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T05:20:36.349177+0000","last_undegraded":"2024-09-17T05:20:36.349177+0000","last_fullsized":"2024-09-17T05:20:36.349177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":79257,"num_write":647,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'3199","reported_seq":3317,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.343729+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T05:20:36.343729+0000","last_peered":"2024-09-17T05:20:36.343729+0000","last_clean":"2024-09-17T05:20:36.343729+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T05:20:36.343729+0000","last_undegraded":"2024-09-17T05:20:36.343729+0000","last_fullsized":"2024-09-17T05:20:36.343729+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16756,"num_write_kb":239128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"68'43","reported_seq":194,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.403635+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T05:20:43.403635+0000","last_peered":"2024-09-17T05:20:43.403635+0000","last_clean":"2024-09-17T05:20:43.403635+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T05:20:43.403635+0000","last_undegraded":"2024-09-17T05:20:43.403635+0000","last_fullsized":"2024-09-17T05:20:43.403635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:24.586834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"68'3014","reported_seq":3125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402775+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T05:20:43.402775+0000","last_peered":"2024-09-17T05:20:43.402775+0000","last_clean":"2024-09-17T05:20:43.402775+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T05:20:43.402775+0000","last_undegraded":"2024-09-17T05:20:43.402775+0000","last_fullsized":"2024-09-17T05:20:43.402775+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3014,"log_dups_size":0,"ondisk_log_size":3014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16762,"num_write_kb":251577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"65'3751","reported_seq":3871,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364101+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T05:20:36.364101+0000","last_peered":"2024-09-17T05:20:36.364101+0000","last_clean":"2024-09-17T05:20:36.364101+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T05:20:36.364101+0000","last_undegraded":"2024-09-17T05:20:36.364101+0000","last_fullsized":"2024-09-17T05:20:36.364101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20040,"num_write_kb":277001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"68'3862","reported_seq":3974,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.271131+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T05:20:43.271131+0000","last_peered":"2024-09-17T05:20:43.271131+0000","last_clean":"2024-09-17T05:20:43.271131+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T05:20:43.271131+0000","last_undegraded":"2024-09-17T05:20:43.271131+0000","last_fullsized":"2024-09-17T05:20:43.271131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20235,"num_write_kb":310497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"65'4193","reported_seq":4314,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.356835+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T05:20:36.356835+0000","last_peered":"2024-09-17T05:20:36.356835+0000","last_clean":"2024-09-17T05:20:36.356835+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T05:20:36.356835+0000","last_undegraded":"2024-09-17T05:20:36.356835+0000","last_fullsized":"2024-09-17T05:20:36.356835+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4193,"log_dups_size":0,"ondisk_log_size":4193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22222,"num_write_kb":306243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"68'3519","reported_seq":3632,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.398772+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T05:20:43.398772+0000","last_peered":"2024-09-17T05:20:43.398772+0000","last_clean":"2024-09-17T05:20:43.398772+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T05:20:43.398772+0000","last_undegraded":"2024-09-17T05:20:43.398772+0000","last_fullsized":"2024-09-17T05:20:43.398772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3519,"log_dups_size":0,"ondisk_log_size":3519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":19548,"num_write_kb":295624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"65'3872","reported_seq":3994,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.360995+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T05:20:36.360995+0000","last_peered":"2024-09-17T05:20:36.360995+0000","last_clean":"2024-09-17T05:20:36.360995+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T05:20:36.360995+0000","last_undegraded":"2024-09-17T05:20:36.360995+0000","last_fullsized":"2024-09-17T05:20:36.360995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3872,"log_dups_size":0,"ondisk_log_size":3872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20576,"num_write_kb":279241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"68'2938","reported_seq":3050,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.380635+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T05:20:43.380635+0000","last_peered":"2024-09-17T05:20:43.380635+0000","last_clean":"2024-09-17T05:20:43.380635+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T05:20:43.380635+0000","last_undegraded":"2024-09-17T05:20:43.380635+0000","last_fullsized":"2024-09-17T05:20:43.380635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2938,"log_dups_size":0,"ondisk_log_size":2938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":16647,"num_write_kb":271506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"65'3500","reported_seq":3620,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.348982+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T05:20:36.348982+0000","last_peered":"2024-09-17T05:20:36.348982+0000","last_clean":"2024-09-17T05:20:36.348982+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T05:20:36.348982+0000","last_undegraded":"2024-09-17T05:20:36.348982+0000","last_fullsized":"2024-09-17T05:20:36.348982+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18562,"num_write_kb":258704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"68'3339","reported_seq":3452,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.406315+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T05:20:43.406315+0000","last_peered":"2024-09-17T05:20:43.406315+0000","last_clean":"2024-09-17T05:20:43.406315+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T05:20:43.406315+0000","last_undegraded":"2024-09-17T05:20:43.406315+0000","last_fullsized":"2024-09-17T05:20:43.406315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3339,"log_dups_size":0,"ondisk_log_size":3339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17940,"num_write_kb":305481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"65'3819","reported_seq":3939,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341730+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T05:20:36.341730+0000","last_peered":"2024-09-17T05:20:36.341730+0000","last_clean":"2024-09-17T05:20:36.341730+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T05:20:36.341730+0000","last_undegraded":"2024-09-17T05:20:36.341730+0000","last_fullsized":"2024-09-17T05:20:36.341730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21007,"num_write_kb":273012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"68'3374","reported_seq":3492,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.366480+0000","last_change":"2024-09-17T04:50:12.076134+0000","last_active":"2024-09-17T05:20:43.366480+0000","last_peered":"2024-09-17T05:20:43.366480+0000","last_clean":"2024-09-17T05:20:43.366480+0000","last_became_active":"2024-09-17T04:50:12.075350+0000","last_became_peered":"2024-09-17T04:50:12.075350+0000","last_unstale":"2024-09-17T05:20:43.366480+0000","last_undegraded":"2024-09-17T05:20:43.366480+0000","last_fullsized":"2024-09-17T05:20:43.366480+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3374,"log_dups_size":0,"ondisk_log_size":3374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18297,"num_write_kb":270032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"65'4233","reported_seq":4352,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341464+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T05:20:36.341464+0000","last_peered":"2024-09-17T05:20:36.341464+0000","last_clean":"2024-09-17T05:20:36.341464+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T05:20:36.341464+0000","last_undegraded":"2024-09-17T05:20:36.341464+0000","last_fullsized":"2024-09-17T05:20:36.341464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23577,"num_write_kb":311694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"68'3382","reported_seq":3494,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.416880+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T05:20:43.416880+0000","last_peered":"2024-09-17T05:20:43.416880+0000","last_clean":"2024-09-17T05:20:43.416880+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T05:20:43.416880+0000","last_undegraded":"2024-09-17T05:20:43.416880+0000","last_fullsized":"2024-09-17T05:20:43.416880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18178,"num_write_kb":293977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"65'3778","reported_seq":3897,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364369+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T05:20:36.364369+0000","last_peered":"2024-09-17T05:20:36.364369+0000","last_clean":"2024-09-17T05:20:36.364369+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T05:20:36.364369+0000","last_undegraded":"2024-09-17T05:20:36.364369+0000","last_fullsized":"2024-09-17T05:20:36.364369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21044,"num_write_kb":281017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"68'2956","reported_seq":3071,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404554+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T05:20:44.404554+0000","last_peered":"2024-09-17T05:20:44.404554+0000","last_clean":"2024-09-17T05:20:44.404554+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T05:20:44.404554+0000","last_undegraded":"2024-09-17T05:20:44.404554+0000","last_fullsized":"2024-09-17T05:20:44.404554+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2956,"log_dups_size":0,"ondisk_log_size":2956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15875,"num_write_kb":264255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"65'4647","reported_seq":4767,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352380+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T05:20:36.352380+0000","last_peered":"2024-09-17T05:20:36.352380+0000","last_clean":"2024-09-17T05:20:36.352380+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T05:20:36.352380+0000","last_undegraded":"2024-09-17T05:20:36.352380+0000","last_fullsized":"2024-09-17T05:20:36.352380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24699,"num_write_kb":336693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"68'3095","reported_seq":3210,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.396708+0000","last_change":"2024-09-17T04:49:41.045502+0000","last_active":"2024-09-17T05:20:43.396708+0000","last_peered":"2024-09-17T05:20:43.396708+0000","last_clean":"2024-09-17T05:20:43.396708+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T05:20:43.396708+0000","last_undegraded":"2024-09-17T05:20:43.396708+0000","last_fullsized":"2024-09-17T05:20:43.396708+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:19.746356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":36,"num_write":16750,"num_write_kb":272041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"65'4070","reported_seq":4190,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.361587+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T05:20:36.361587+0000","last_peered":"2024-09-17T05:20:36.361587+0000","last_clean":"2024-09-17T05:20:36.361587+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T05:20:36.361587+0000","last_undegraded":"2024-09-17T05:20:36.361587+0000","last_fullsized":"2024-09-17T05:20:36.361587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4070,"log_dups_size":0,"ondisk_log_size":4070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21906,"num_write_kb":288644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"68'4276","reported_seq":4387,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404966+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T05:20:44.404966+0000","last_peered":"2024-09-17T05:20:44.404966+0000","last_clean":"2024-09-17T05:20:44.404966+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T05:20:44.404966+0000","last_undegraded":"2024-09-17T05:20:44.404966+0000","last_fullsized":"2024-09-17T05:20:44.404966+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22732,"num_write_kb":365385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"65'4174","reported_seq":4292,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.348275+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T05:20:36.348275+0000","last_peered":"2024-09-17T05:20:36.348275+0000","last_clean":"2024-09-17T05:20:36.348275+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T05:20:36.348275+0000","last_undegraded":"2024-09-17T05:20:36.348275+0000","last_fullsized":"2024-09-17T05:20:36.348275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22870,"num_write_kb":296621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"68'2821","reported_seq":2931,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393189+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T05:20:43.393189+0000","last_peered":"2024-09-17T05:20:43.393189+0000","last_clean":"2024-09-17T05:20:43.393189+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T05:20:43.393189+0000","last_undegraded":"2024-09-17T05:20:43.393189+0000","last_fullsized":"2024-09-17T05:20:43.393189+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2821,"log_dups_size":0,"ondisk_log_size":2821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15110,"num_write_kb":232606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"65'4280","reported_seq":4398,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.343678+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T05:20:36.343678+0000","last_peered":"2024-09-17T05:20:36.343678+0000","last_clean":"2024-09-17T05:20:36.343678+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T05:20:36.343678+0000","last_undegraded":"2024-09-17T05:20:36.343678+0000","last_fullsized":"2024-09-17T05:20:36.343678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4280,"log_dups_size":0,"ondisk_log_size":4280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23007,"num_write_kb":313406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"68'3452","reported_seq":3562,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.303318+0000","last_change":"2024-09-17T04:49:41.038265+0000","last_active":"2024-09-17T05:20:43.303318+0000","last_peered":"2024-09-17T05:20:43.303318+0000","last_clean":"2024-09-17T05:20:43.303318+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T05:20:43.303318+0000","last_undegraded":"2024-09-17T05:20:43.303318+0000","last_fullsized":"2024-09-17T05:20:43.303318+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3452,"log_dups_size":0,"ondisk_log_size":3452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:41.638403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19160,"num_write_kb":306156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"65'4384","reported_seq":4504,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339211+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T05:20:36.339211+0000","last_peered":"2024-09-17T05:20:36.339211+0000","last_clean":"2024-09-17T05:20:36.339211+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T05:20:36.339211+0000","last_undegraded":"2024-09-17T05:20:36.339211+0000","last_fullsized":"2024-09-17T05:20:36.339211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23582,"num_write_kb":336581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"68'3472","reported_seq":3589,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.255838+0000","last_change":"2024-09-17T04:50:12.068348+0000","last_active":"2024-09-17T05:20:43.255838+0000","last_peered":"2024-09-17T05:20:43.255838+0000","last_clean":"2024-09-17T05:20:43.255838+0000","last_became_active":"2024-09-17T04:50:12.067942+0000","last_became_peered":"2024-09-17T04:50:12.067942+0000","last_unstale":"2024-09-17T05:20:43.255838+0000","last_undegraded":"2024-09-17T05:20:43.255838+0000","last_fullsized":"2024-09-17T05:20:43.255838+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3472,"log_dups_size":0,"ondisk_log_size":3472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:33.954020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18777,"num_write_kb":292458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"65'3688","reported_seq":3806,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557618+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T05:20:36.557618+0000","last_peered":"2024-09-17T05:20:36.557618+0000","last_clean":"2024-09-17T05:20:36.557618+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T05:20:36.557618+0000","last_undegraded":"2024-09-17T05:20:36.557618+0000","last_fullsized":"2024-09-17T05:20:36.557618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20358,"num_write_kb":257774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"68'3214","reported_seq":3329,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.416523+0000","last_change":"2024-09-17T04:49:41.038072+0000","last_active":"2024-09-17T05:20:43.416523+0000","last_peered":"2024-09-17T05:20:43.416523+0000","last_clean":"2024-09-17T05:20:43.416523+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T05:20:43.416523+0000","last_undegraded":"2024-09-17T05:20:43.416523+0000","last_fullsized":"2024-09-17T05:20:43.416523+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":17543,"num_write_kb":304029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"65'4068","reported_seq":4186,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374459+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T05:20:36.374459+0000","last_peered":"2024-09-17T05:20:36.374459+0000","last_clean":"2024-09-17T05:20:36.374459+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T05:20:36.374459+0000","last_undegraded":"2024-09-17T05:20:36.374459+0000","last_fullsized":"2024-09-17T05:20:36.374459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21759,"num_write_kb":308395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"68'2551","reported_seq":2663,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402052+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T05:20:43.402052+0000","last_peered":"2024-09-17T05:20:43.402052+0000","last_clean":"2024-09-17T05:20:43.402052+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T05:20:43.402052+0000","last_undegraded":"2024-09-17T05:20:43.402052+0000","last_fullsized":"2024-09-17T05:20:43.402052+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2551,"log_dups_size":0,"ondisk_log_size":2551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13664,"num_write_kb":247232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"65'4369","reported_seq":4488,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.359366+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T05:20:36.359366+0000","last_peered":"2024-09-17T05:20:36.359366+0000","last_clean":"2024-09-17T05:20:36.359366+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T05:20:36.359366+0000","last_undegraded":"2024-09-17T05:20:36.359366+0000","last_fullsized":"2024-09-17T05:20:36.359366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23171,"num_write_kb":318257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"68'3277","reported_seq":3388,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393579+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T05:20:43.393579+0000","last_peered":"2024-09-17T05:20:43.393579+0000","last_clean":"2024-09-17T05:20:43.393579+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T05:20:43.393579+0000","last_undegraded":"2024-09-17T05:20:43.393579+0000","last_fullsized":"2024-09-17T05:20:43.393579+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18007,"num_write_kb":264733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"65'5432","reported_seq":5550,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366356+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T05:20:36.366356+0000","last_peered":"2024-09-17T05:20:36.366356+0000","last_clean":"2024-09-17T05:20:36.366356+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T05:20:36.366356+0000","last_undegraded":"2024-09-17T05:20:36.366356+0000","last_fullsized":"2024-09-17T05:20:36.366356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29730,"num_write_kb":389609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"68'3135","reported_seq":3246,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.377491+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T05:20:43.377491+0000","last_peered":"2024-09-17T05:20:43.377491+0000","last_clean":"2024-09-17T05:20:43.377491+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T05:20:43.377491+0000","last_undegraded":"2024-09-17T05:20:43.377491+0000","last_fullsized":"2024-09-17T05:20:43.377491+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":16884,"num_write_kb":284681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"65'3941","reported_seq":4059,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.360280+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T05:20:36.360280+0000","last_peered":"2024-09-17T05:20:36.360280+0000","last_clean":"2024-09-17T05:20:36.360280+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T05:20:36.360280+0000","last_undegraded":"2024-09-17T05:20:36.360280+0000","last_fullsized":"2024-09-17T05:20:36.360280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21845,"num_write_kb":304681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"68'3913","reported_seq":4026,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.330331+0000","last_change":"2024-09-17T04:49:41.067207+0000","last_active":"2024-09-17T05:20:43.330331+0000","last_peered":"2024-09-17T05:20:43.330331+0000","last_clean":"2024-09-17T05:20:43.330331+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T05:20:43.330331+0000","last_undegraded":"2024-09-17T05:20:43.330331+0000","last_fullsized":"2024-09-17T05:20:43.330331+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3913,"log_dups_size":0,"ondisk_log_size":3913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:08.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":21530,"num_write_kb":364884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"65'3351","reported_seq":3475,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.347528+0000","last_change":"2024-09-17T04:50:12.469782+0000","last_active":"2024-09-17T05:20:36.347528+0000","last_peered":"2024-09-17T05:20:36.347528+0000","last_clean":"2024-09-17T05:20:36.347528+0000","last_became_active":"2024-09-17T04:50:12.469425+0000","last_became_peered":"2024-09-17T04:50:12.469425+0000","last_unstale":"2024-09-17T05:20:36.347528+0000","last_undegraded":"2024-09-17T05:20:36.347528+0000","last_fullsized":"2024-09-17T05:20:36.347528+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:04.098943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18172,"num_write_kb":265354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"68'1954","reported_seq":2066,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.346900+0000","last_change":"2024-09-17T04:49:41.046170+0000","last_active":"2024-09-17T05:20:43.346900+0000","last_peered":"2024-09-17T05:20:43.346900+0000","last_clean":"2024-09-17T05:20:43.346900+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T05:20:43.346900+0000","last_undegraded":"2024-09-17T05:20:43.346900+0000","last_fullsized":"2024-09-17T05:20:43.346900+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":1954,"log_dups_size":0,"ondisk_log_size":1954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:01.683826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":34,"num_write":10478,"num_write_kb":181241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"65'3661","reported_seq":3779,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.344111+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T05:20:36.344111+0000","last_peered":"2024-09-17T05:20:36.344111+0000","last_clean":"2024-09-17T05:20:36.344111+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T05:20:36.344111+0000","last_undegraded":"2024-09-17T05:20:36.344111+0000","last_fullsized":"2024-09-17T05:20:36.344111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19087,"num_write_kb":273280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"68'2981","reported_seq":3093,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.203029+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T05:20:43.203029+0000","last_peered":"2024-09-17T05:20:43.203029+0000","last_clean":"2024-09-17T05:20:43.203029+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T05:20:43.203029+0000","last_undegraded":"2024-09-17T05:20:43.203029+0000","last_fullsized":"2024-09-17T05:20:43.203029+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:25.574682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":15573,"num_write_kb":271248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"65'3297","reported_seq":3414,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340729+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T05:20:36.340729+0000","last_peered":"2024-09-17T05:20:36.340729+0000","last_clean":"2024-09-17T05:20:36.340729+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T05:20:36.340729+0000","last_undegraded":"2024-09-17T05:20:36.340729+0000","last_fullsized":"2024-09-17T05:20:36.340729+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3297,"log_dups_size":0,"ondisk_log_size":3297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17851,"num_write_kb":243868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"68'2872","reported_seq":2983,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403678+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T05:20:44.403678+0000","last_peered":"2024-09-17T05:20:44.403678+0000","last_clean":"2024-09-17T05:20:44.403678+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T05:20:44.403678+0000","last_undegraded":"2024-09-17T05:20:44.403678+0000","last_fullsized":"2024-09-17T05:20:44.403678+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2872,"log_dups_size":0,"ondisk_log_size":2872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15784,"num_write_kb":259984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"65'2574","reported_seq":2691,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.556549+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T05:20:36.556549+0000","last_peered":"2024-09-17T05:20:36.556549+0000","last_clean":"2024-09-17T05:20:36.556549+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T05:20:36.556549+0000","last_undegraded":"2024-09-17T05:20:36.556549+0000","last_fullsized":"2024-09-17T05:20:36.556549+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":2574,"log_dups_size":0,"ondisk_log_size":2574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14007,"num_write_kb":203729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"68'2459","reported_seq":2572,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393891+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T05:20:43.393891+0000","last_peered":"2024-09-17T05:20:43.393891+0000","last_clean":"2024-09-17T05:20:43.393891+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T05:20:43.393891+0000","last_undegraded":"2024-09-17T05:20:43.393891+0000","last_fullsized":"2024-09-17T05:20:43.393891+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2459,"log_dups_size":0,"ondisk_log_size":2459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13572,"num_write_kb":222508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"65'3245","reported_seq":3362,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.556406+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T05:20:36.556406+0000","last_peered":"2024-09-17T05:20:36.556406+0000","last_clean":"2024-09-17T05:20:36.556406+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T05:20:36.556406+0000","last_undegraded":"2024-09-17T05:20:36.556406+0000","last_fullsized":"2024-09-17T05:20:36.556406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3245,"log_dups_size":0,"ondisk_log_size":3245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16903,"num_write_kb":256080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"68'3074","reported_seq":3184,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393584+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T05:20:43.393584+0000","last_peered":"2024-09-17T05:20:43.393584+0000","last_clean":"2024-09-17T05:20:43.393584+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T05:20:43.393584+0000","last_undegraded":"2024-09-17T05:20:43.393584+0000","last_fullsized":"2024-09-17T05:20:43.393584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16752,"num_write_kb":271957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"65'4387","reported_seq":4504,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557565+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T05:20:36.557565+0000","last_peered":"2024-09-17T05:20:36.557565+0000","last_clean":"2024-09-17T05:20:36.557565+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T05:20:36.557565+0000","last_undegraded":"2024-09-17T05:20:36.557565+0000","last_fullsized":"2024-09-17T05:20:36.557565+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24001,"num_write_kb":317505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"68'2713","reported_seq":2824,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.417489+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T05:20:43.417489+0000","last_peered":"2024-09-17T05:20:43.417489+0000","last_clean":"2024-09-17T05:20:43.417489+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T05:20:43.417489+0000","last_undegraded":"2024-09-17T05:20:43.417489+0000","last_fullsized":"2024-09-17T05:20:43.417489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2713,"log_dups_size":0,"ondisk_log_size":2713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14521,"num_write_kb":262890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"65'4656","reported_seq":4775,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.347514+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T05:20:36.347514+0000","last_peered":"2024-09-17T05:20:36.347514+0000","last_clean":"2024-09-17T05:20:36.347514+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T05:20:36.347514+0000","last_undegraded":"2024-09-17T05:20:36.347514+0000","last_fullsized":"2024-09-17T05:20:36.347514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25090,"num_write_kb":362900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"68'3588","reported_seq":3702,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.402501+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T05:20:44.402501+0000","last_peered":"2024-09-17T05:20:44.402501+0000","last_clean":"2024-09-17T05:20:44.402501+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T05:20:44.402501+0000","last_undegraded":"2024-09-17T05:20:44.402501+0000","last_fullsized":"2024-09-17T05:20:44.402501+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3588,"log_dups_size":0,"ondisk_log_size":3588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":19474,"num_write_kb":319707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"65'4149","reported_seq":4266,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.347526+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T05:20:36.347526+0000","last_peered":"2024-09-17T05:20:36.347526+0000","last_clean":"2024-09-17T05:20:36.347526+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T05:20:36.347526+0000","last_undegraded":"2024-09-17T05:20:36.347526+0000","last_fullsized":"2024-09-17T05:20:36.347526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22453,"num_write_kb":298991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"68'3299","reported_seq":3415,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.422884+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T05:20:44.422884+0000","last_peered":"2024-09-17T05:20:44.422884+0000","last_clean":"2024-09-17T05:20:44.422884+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T05:20:44.422884+0000","last_undegraded":"2024-09-17T05:20:44.422884+0000","last_fullsized":"2024-09-17T05:20:44.422884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3299,"log_dups_size":0,"ondisk_log_size":3299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":17863,"num_write_kb":288721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"65'3910","reported_seq":4027,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.555930+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T05:20:36.555930+0000","last_peered":"2024-09-17T05:20:36.555930+0000","last_clean":"2024-09-17T05:20:36.555930+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T05:20:36.555930+0000","last_undegraded":"2024-09-17T05:20:36.555930+0000","last_fullsized":"2024-09-17T05:20:36.555930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21371,"num_write_kb":294687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"68'2856","reported_seq":2969,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.325969+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T05:20:43.325969+0000","last_peered":"2024-09-17T05:20:43.325969+0000","last_clean":"2024-09-17T05:20:43.325969+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T05:20:43.325969+0000","last_undegraded":"2024-09-17T05:20:43.325969+0000","last_fullsized":"2024-09-17T05:20:43.325969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15916,"num_write_kb":235941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"65'3148","reported_seq":3265,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.373270+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T05:20:36.373270+0000","last_peered":"2024-09-17T05:20:36.373270+0000","last_clean":"2024-09-17T05:20:36.373270+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T05:20:36.373270+0000","last_undegraded":"2024-09-17T05:20:36.373270+0000","last_fullsized":"2024-09-17T05:20:36.373270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16748,"num_write_kb":239238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"68'3516","reported_seq":3630,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.409318+0000","last_change":"2024-09-17T04:49:41.041201+0000","last_active":"2024-09-17T05:20:43.409318+0000","last_peered":"2024-09-17T05:20:43.409318+0000","last_clean":"2024-09-17T05:20:43.409318+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T05:20:43.409318+0000","last_undegraded":"2024-09-17T05:20:43.409318+0000","last_fullsized":"2024-09-17T05:20:43.409318+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:02.588786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18785,"num_write_kb":280186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"65'3294","reported_seq":3411,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558447+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T05:20:36.558447+0000","last_peered":"2024-09-17T05:20:36.558447+0000","last_clean":"2024-09-17T05:20:36.558447+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T05:20:36.558447+0000","last_undegraded":"2024-09-17T05:20:36.558447+0000","last_fullsized":"2024-09-17T05:20:36.558447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16934,"num_write_kb":260475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"68'2875","reported_seq":2985,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.359568+0000","last_change":"2024-09-17T04:49:41.061092+0000","last_active":"2024-09-17T05:20:43.359568+0000","last_peered":"2024-09-17T05:20:43.359568+0000","last_clean":"2024-09-17T05:20:43.359568+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T05:20:43.359568+0000","last_undegraded":"2024-09-17T05:20:43.359568+0000","last_fullsized":"2024-09-17T05:20:43.359568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2875,"log_dups_size":0,"ondisk_log_size":2875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:52.714614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15268,"num_write_kb":241386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"65'3071","reported_seq":3192,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.370583+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T05:20:36.370583+0000","last_peered":"2024-09-17T05:20:36.370583+0000","last_clean":"2024-09-17T05:20:36.370583+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T05:20:36.370583+0000","last_undegraded":"2024-09-17T05:20:36.370583+0000","last_fullsized":"2024-09-17T05:20:36.370583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3071,"log_dups_size":0,"ondisk_log_size":3071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16593,"num_write_kb":218157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"68'2617","reported_seq":2728,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.405858+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T05:20:43.405858+0000","last_peered":"2024-09-17T05:20:43.405858+0000","last_clean":"2024-09-17T05:20:43.405858+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T05:20:43.405858+0000","last_undegraded":"2024-09-17T05:20:43.405858+0000","last_fullsized":"2024-09-17T05:20:43.405858+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3,"num_write":13428,"num_write_kb":250535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"65'3866","reported_seq":3987,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352182+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T05:20:36.352182+0000","last_peered":"2024-09-17T05:20:36.352182+0000","last_clean":"2024-09-17T05:20:36.352182+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T05:20:36.352182+0000","last_undegraded":"2024-09-17T05:20:36.352182+0000","last_fullsized":"2024-09-17T05:20:36.352182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3866,"log_dups_size":0,"ondisk_log_size":3866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20832,"num_write_kb":272078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"68'3056","reported_seq":3165,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.326456+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T05:20:43.326456+0000","last_peered":"2024-09-17T05:20:43.326456+0000","last_clean":"2024-09-17T05:20:43.326456+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T05:20:43.326456+0000","last_undegraded":"2024-09-17T05:20:43.326456+0000","last_fullsized":"2024-09-17T05:20:43.326456+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3056,"log_dups_size":0,"ondisk_log_size":3056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16712,"num_write_kb":275987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"65'4485","reported_seq":4605,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.351407+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T05:20:36.351407+0000","last_peered":"2024-09-17T05:20:36.351407+0000","last_clean":"2024-09-17T05:20:36.351407+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T05:20:36.351407+0000","last_undegraded":"2024-09-17T05:20:36.351407+0000","last_fullsized":"2024-09-17T05:20:36.351407+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23824,"num_write_kb":337279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"68'3229","reported_seq":3338,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.420569+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T05:20:44.420569+0000","last_peered":"2024-09-17T05:20:44.420569+0000","last_clean":"2024-09-17T05:20:44.420569+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T05:20:44.420569+0000","last_undegraded":"2024-09-17T05:20:44.420569+0000","last_fullsized":"2024-09-17T05:20:44.420569+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3229,"log_dups_size":0,"ondisk_log_size":3229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17855,"num_write_kb":268347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"65'3523","reported_seq":3644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559155+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T05:20:36.559155+0000","last_peered":"2024-09-17T05:20:36.559155+0000","last_clean":"2024-09-17T05:20:36.559155+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T05:20:36.559155+0000","last_undegraded":"2024-09-17T05:20:36.559155+0000","last_fullsized":"2024-09-17T05:20:36.559155+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3523,"log_dups_size":0,"ondisk_log_size":3523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19890,"num_write_kb":260121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":321,"num_write":1077532,"num_write_kb":17428719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":198824,"ondisk_log_size":198824,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":18310618,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1391,"num_read_kb":207657,"num_write":4849,"num_write_kb":19832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36683776,"data_stored":36621236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4439,"ondisk_log_size":4439,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1352682,"num_write_kb":18536494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250516,"ondisk_log_size":250516,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":18963064,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1535,"num_read_kb":230833,"num_write":4911,"num_write_kb":20583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37986304,"data_stored":37926128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4503,"ondisk_log_size":4503,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509823,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533064,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":532094,"kb_avail":93838776,"statfs":{"total":96636764160,"available":96090906624,"internally_reserved":0,"allocated":974848,"data_stored":552256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":544864714},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509823,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762124,"kb_used_data":17916,"kb_used_omap":1,"kb_used_meta":744190,"kb_avail":93609716,"statfs":{"total":96636764160,"available":95856349184,"internally_reserved":0,"allocated":18345984,"data_stored":17920368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":762051015},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509823,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756036,"kb_used_data":17268,"kb_used_omap":1,"kb_used_meta":738750,"kb_avail":93615804,"statfs":{"total":96636764160,"available":95862583296,"internally_reserved":0,"allocated":17682432,"data_stored":17246718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756480453},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509823,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755412,"kb_used_data":15300,"kb_used_omap":1,"kb_used_meta":740094,"kb_avail":93616428,"statfs":{"total":96636764160,"available":95863222272,"internally_reserved":0,"allocated":15667200,"data_stored":15236011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":757856711},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509823,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877392,"kb_used_data":14656,"kb_used_omap":1,"kb_used_meta":862718,"kb_avail":93494448,"statfs":{"total":96636764160,"available":95738314752,"internally_reserved":0,"allocated":15007744,"data_stored":14583014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":883423685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509824,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630324,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":628862,"kb_avail":93741516,"statfs":{"total":96636764160,"available":95991312384,"internally_reserved":0,"allocated":1478656,"data_stored":1057972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":643955141},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509823,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":819832,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":816894,"kb_avail":93552008,"statfs":{"total":96636764160,"available":95797256192,"internally_reserved":0,"allocated":2990080,"data_stored":2567342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":836499909},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542527,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694200,"kb_used_data":10472,"kb_used_omap":1,"kb_used_meta":683710,"kb_avail":93677640,"statfs":{"total":96636764160,"available":95925903360,"internally_reserved":0,"allocated":10723328,"data_stored":10304131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":700119494},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2099788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10493952,"data_stored":10488922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4279042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":84770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5726270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4279625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5652480,"data_stored":5642363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:20:59.170 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:20:59.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 -- 172.21.15.150:0/4183557637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5300c41c0 msgr2=0x7fe53006fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:59.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 --2- 172.21.15.150:0/4183557637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5300c41c0 0x7fe53006fab0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fe530071360 tx=0x7fe52401c8e0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 -- 172.21.15.150:0/4183557637 shutdown_connections 2024-09-17T05:20:59.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 --2- 172.21.15.150:0/4183557637 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5300c41c0 0x7fe53006fab0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 -- 172.21.15.150:0/4183557637 >> 172.21.15.150:0/4183557637 conn(0x7fe5300c4a00 msgr2=0x7fe5300c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:59.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 -- 172.21.15.150:0/4183557637 shutdown_connections 2024-09-17T05:20:59.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 -- 172.21.15.150:0/4183557637 wait complete. 2024-09-17T05:20:59.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 Processor -- start 2024-09-17T05:20:59.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 -- start start 2024-09-17T05:20:59.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53006fff0 0x7fe53014cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:59.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5300c3db0 0x7fe530157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:59.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5300c41c0 0x7fe53014b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:59.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe5300adb30 con 0x7fe53006fff0 2024-09-17T05:20:59.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe5300ad9b0 con 0x7fe5300c3db0 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe5365b5640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fe5300a75f0 con 0x7fe5300c41c0 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe52ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53006fff0 0x7fe53014cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe52ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53006fff0 0x7fe53014cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41600/0 (socket says 172.21.15.150:41600) 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.360+0000 7fe52ffff640 1 -- 172.21.15.150:0/390915653 learned_addr learned my addr 172.21.15.150:0/390915653 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52f7fe640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5300c3db0 0x7fe530157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe534b52640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5300c41c0 0x7fe53014b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52ffff640 1 -- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5300c41c0 msgr2=0x7fe53014b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52ffff640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5300c41c0 0x7fe53014b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52ffff640 1 -- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5300c3db0 msgr2=0x7fe530157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52ffff640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5300c3db0 0x7fe530157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52ffff640 1 -- 172.21.15.150:0/390915653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe530158460 con 0x7fe53006fff0 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52f7fe640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5300c3db0 0x7fe530157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52ffff640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53006fff0 0x7fe53014cff0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fe52000b3e0 tx=0x7fe52000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:59.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52cff9640 1 -- 172.21.15.150:0/390915653 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe520015020 con 0x7fe53006fff0 2024-09-17T05:20:59.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52cff9640 1 -- 172.21.15.150:0/390915653 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe520011920 con 0x7fe53006fff0 2024-09-17T05:20:59.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe5365b5640 1 -- 172.21.15.150:0/390915653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe530158870 con 0x7fe53006fff0 2024-09-17T05:20:59.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52cff9640 1 -- 172.21.15.150:0/390915653 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe520011db0 con 0x7fe53006fff0 2024-09-17T05:20:59.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe5365b5640 1 -- 172.21.15.150:0/390915653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe53014d530 con 0x7fe53006fff0 2024-09-17T05:20:59.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe5365b5640 1 -- 172.21.15.150:0/390915653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe53006fab0 con 0x7fe53006fff0 2024-09-17T05:20:59.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52cff9640 1 -- 172.21.15.150:0/390915653 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fe520018020 con 0x7fe53006fff0 2024-09-17T05:20:59.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52cff9640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe51c073230 0x7fe51c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:20:59.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52cff9640 1 -- 172.21.15.150:0/390915653 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7fe520094f20 con 0x7fe53006fff0 2024-09-17T05:20:59.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.364+0000 7fe52f7fe640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe51c073230 0x7fe51c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:20:59.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.368+0000 7fe52f7fe640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe51c073230 0x7fe51c075740 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe5240af000 tx=0x7fe52401b120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:20:59.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.368+0000 7fe52cff9640 1 -- 172.21.15.150:0/390915653 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe52001f030 con 0x7fe53006fff0 2024-09-17T05:20:59.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.532+0000 7fe5365b5640 1 -- 172.21.15.150:0/390915653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5301ee0f0 con 0x7fe53006fff0 2024-09-17T05:20:59.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.536+0000 7fe52cff9640 1 -- 172.21.15.150:0/390915653 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) ==== 74+0+26103 (secure 0 0 0) 0x7fe52005ee90 con 0x7fe53006fff0 2024-09-17T05:20:59.534 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:20:59.534 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":68,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T05:20:36.328166+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.31","mappings":[{"from":0,"to":7}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]},{"pgid":"5.2a","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6833/2857695220":"2024-09-18T05:20:36.328050+0000","172.21.15.150:6832/2857695220":"2024-09-18T05:20:36.328050+0000","172.21.15.176:6835/1690935336":"2024-09-18T05:19:51.316025+0000","172.21.15.176:6834/1690935336":"2024-09-18T05:19:51.316025+0000","172.21.15.150:6834/3297154462":"2024-09-18T05:18:31.294755+0000","172.21.15.176:6837/1903363960":"2024-09-18T05:18:31.294598+0000","172.21.15.176:6836/1903363960":"2024-09-18T05:18:31.294598+0000","172.21.15.150:6835/3297154462":"2024-09-18T05:18:31.294755+0000","172.21.15.176:6835/524846644":"2024-09-18T05:16:53.685445+0000","172.21.15.150:6833/949083177":"2024-09-18T05:16:51.270750+0000","172.21.15.150:6834/178583496":"2024-09-18T05:15:51.246236+0000","172.21.15.176:6836/2338457617":"2024-09-18T05:15:08.260160+0000","172.21.15.150:6833/3933379792":"2024-09-18T05:15:06.233954+0000","172.21.15.150:6835/3179072684":"2024-09-18T05:14:26.220879+0000","172.21.15.150:6833/12982302":"2024-09-18T05:13:46.203493+0000","172.21.15.176:6837/2962113435":"2024-09-18T05:12:21.179693+0000","172.21.15.150:6835/178583496":"2024-09-18T05:15:51.246236+0000","172.21.15.176:6837/713081612":"2024-09-18T04:59:20.944548+0000","172.21.15.150:6834/2081398022":"2024-09-18T05:00:20.961331+0000","172.21.15.150:6835/3079958649":"2024-09-18T04:58:00.922711+0000","172.21.15.150:6834/3079958649":"2024-09-18T04:58:00.922711+0000","172.21.15.150:6832/12982302":"2024-09-18T05:13:46.203493+0000","172.21.15.150:6832/233311351":"2024-09-18T05:04:06.023230+0000","172.21.15.150:6832/949083177":"2024-09-18T05:16:51.270750+0000","172.21.15.150:6833/284564893":"2024-09-18T04:57:40.911134+0000","172.21.15.176:6837/903190072":"2024-09-18T05:04:36.031676+0000","172.21.15.176:6837/2374314256":"2024-09-18T04:56:10.885556+0000","172.21.15.150:6834/3179072684":"2024-09-18T05:14:26.220879+0000","172.21.15.176:6836/3543629780":"2024-09-18T04:58:30.930118+0000","172.21.15.150:6833/4286123257":"2024-09-18T04:55:40.878553+0000","172.21.15.150:6834/480168496":"2024-09-18T04:51:40.821573+0000","172.21.15.176:6836/1045115232":"2024-09-18T04:53:50.852056+0000","172.21.15.176:6837/1045115232":"2024-09-18T04:53:50.852056+0000","172.21.15.150:6835/285376980":"2024-09-18T05:10:51.151028+0000","172.21.15.150:6832/2313391683":"2024-09-18T05:00:45.968345+0000","172.21.15.150:6835/2081398022":"2024-09-18T05:00:20.961331+0000","172.21.15.150:6832/480168496":"2024-09-18T04:51:40.821573+0000","172.21.15.150:6834/151500360":"2024-09-18T04:54:55.866806+0000","172.21.15.150:6833/2313391683":"2024-09-18T05:00:45.968345+0000","172.21.15.150:6833/1275062894":"2024-09-18T04:58:50.936080+0000","172.21.15.150:6832/1275062894":"2024-09-18T04:58:50.936080+0000","172.21.15.150:6832/961213876":"2024-09-18T04:52:55.838337+0000","172.21.15.150:6835/151500360":"2024-09-18T04:54:55.866806+0000","172.21.15.176:6836/2374314256":"2024-09-18T04:56:10.885556+0000","172.21.15.150:6832/1237450980":"2024-09-18T05:06:31.077239+0000","172.21.15.176:6837/2338457617":"2024-09-18T05:15:08.260160+0000","172.21.15.150:6833/961213876":"2024-09-18T04:52:55.838337+0000","172.21.15.150:6832/3532647866":"2024-09-18T04:55:10.871121+0000","172.21.15.176:6836/713081612":"2024-09-18T04:59:20.944548+0000","172.21.15.176:6837/792092400":"2024-09-18T05:09:46.133255+0000","172.21.15.150:6833/2613040311":"2024-09-18T04:51:25.818032+0000","172.21.15.150:6833/3532647866":"2024-09-18T04:55:10.871121+0000","172.21.15.150:6834/2062728083":"2024-09-18T05:03:01.005221+0000","172.21.15.150:6832/523364553":"2024-09-18T05:07:21.092015+0000","172.21.15.150:6833/233311351":"2024-09-18T05:04:06.023230+0000","172.21.15.176:6836/2962113435":"2024-09-18T05:12:21.179693+0000","172.21.15.150:6832/4286123257":"2024-09-18T04:55:40.878553+0000","172.21.15.176:6836/792092400":"2024-09-18T05:09:46.133255+0000","172.21.15.176:6836/702608997":"2024-09-18T05:02:30.998209+0000","172.21.15.150:6832/284564893":"2024-09-18T04:57:40.911134+0000","172.21.15.150:6833/3622725045":"2024-09-18T05:09:06.121144+0000","172.21.15.150:6835/2062728083":"2024-09-18T05:03:01.005221+0000","172.21.15.176:6836/903190072":"2024-09-18T05:04:36.031676+0000","172.21.15.150:6832/1984876738":"2024-09-18T05:05:21.045156+0000","172.21.15.176:6834/524846644":"2024-09-18T05:16:53.685445+0000","172.21.15.150:6833/1984876738":"2024-09-18T05:05:21.045156+0000","172.21.15.150:6832/3933379792":"2024-09-18T05:15:06.233954+0000","172.21.15.150:6834/3807957687":"2024-09-18T05:05:56.055429+0000","172.21.15.150:6835/3807957687":"2024-09-18T05:05:56.055429+0000","172.21.15.176:6837/3940723333":"2024-09-18T05:06:51.083809+0000","172.21.15.150:6833/1237450980":"2024-09-18T05:06:31.077239+0000","172.21.15.176:6837/3543629780":"2024-09-18T04:58:30.930118+0000","172.21.15.176:6837/702608997":"2024-09-18T05:02:30.998209+0000","172.21.15.150:6835/92695211":"2024-09-18T05:08:16.109918+0000","172.21.15.150:6835/2613040311":"2024-09-18T04:51:25.818032+0000","172.21.15.150:6833/523364553":"2024-09-18T05:07:21.092015+0000","172.21.15.150:6834/92695211":"2024-09-18T05:08:16.109918+0000","172.21.15.150:6832/3622725045":"2024-09-18T05:09:06.121144+0000","172.21.15.176:6836/3940723333":"2024-09-18T05:06:51.083809+0000","172.21.15.150:6834/285376980":"2024-09-18T05:10:51.151028+0000","172.21.15.150:6833/2056467678":"2024-09-18T05:12:01.173633+0000","172.21.15.150:6832/2056467678":"2024-09-18T05:12:01.173633+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:20:59.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.540+0000 7fe5365b5640 1 -- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe51c073230 msgr2=0x7fe51c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:59.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.540+0000 7fe5365b5640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe51c073230 0x7fe51c075740 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe5240af000 tx=0x7fe52401b120 comp rx=0 tx=0).stop 2024-09-17T05:20:59.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.540+0000 7fe5365b5640 1 -- 172.21.15.150:0/390915653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53006fff0 msgr2=0x7fe53014cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:20:59.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.540+0000 7fe5365b5640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53006fff0 0x7fe53014cff0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fe52000b3e0 tx=0x7fe52000b8a0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.540+0000 7fe5365b5640 1 -- 172.21.15.150:0/390915653 shutdown_connections 2024-09-17T05:20:59.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.540+0000 7fe5365b5640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fe51c073230 0x7fe51c075740 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.540+0000 7fe5365b5640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe5300c3db0 0x7fe530157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.540+0000 7fe5365b5640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fe5300c41c0 0x7fe53014b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.540+0000 7fe5365b5640 1 --2- 172.21.15.150:0/390915653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe53006fff0 0x7fe53014cff0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:20:59.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.540+0000 7fe5365b5640 1 -- 172.21.15.150:0/390915653 >> 172.21.15.150:0/390915653 conn(0x7fe5300c4a00 msgr2=0x7fe530152540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:20:59.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.540+0000 7fe5365b5640 1 -- 172.21.15.150:0/390915653 shutdown_connections 2024-09-17T05:20:59.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:20:59.540+0000 7fe5365b5640 1 -- 172.21.15.150:0/390915653 wait complete. 2024-09-17T05:21:00.560 INFO:tasks.ceph:Scrubbing osd.0 2024-09-17T05:21:00.561 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:21:00.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.736+0000 7f2939ce0640 1 -- 172.21.15.150:0/3524714563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293414e740 msgr2=0x7f2934157bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:00.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.736+0000 7f2939ce0640 1 --2- 172.21.15.150:0/3524714563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293414e740 0x7f2934157bf0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f29340c4700 tx=0x7f292c01caf0 comp rx=0 tx=0).stop 2024-09-17T05:21:00.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.736+0000 7f2939ce0640 1 -- 172.21.15.150:0/3524714563 shutdown_connections 2024-09-17T05:21:00.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.736+0000 7f2939ce0640 1 --2- 172.21.15.150:0/3524714563 >> v2:172.21.15.176:3300/0 conn(0x7f29340c62d0 0x7f29340ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:00.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.736+0000 7f2939ce0640 1 --2- 172.21.15.150:0/3524714563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f293414e740 0x7f2934157bf0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:00.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.736+0000 7f2939ce0640 1 -- 172.21.15.150:0/3524714563 >> 172.21.15.150:0/3524714563 conn(0x7f29340c9eb0 msgr2=0x7f29340ca2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:00.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.736+0000 7f2939ce0640 1 -- 172.21.15.150:0/3524714563 shutdown_connections 2024-09-17T05:21:00.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.736+0000 7f2939ce0640 1 -- 172.21.15.150:0/3524714563 wait complete. 2024-09-17T05:21:00.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2939ce0640 1 Processor -- start 2024-09-17T05:21:00.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2939ce0640 1 -- start start 2024-09-17T05:21:00.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2939ce0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29340acea0 0x7f2934153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:00.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2939ce0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29340c62d0 0x7f2934157840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:00.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2939ce0640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f293414e740 0x7f2934152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:00.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2939ce0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2934161970 con 0x7f29340acea0 2024-09-17T05:21:00.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2939ce0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f29341617f0 con 0x7f29340c62d0 2024-09-17T05:21:00.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2939ce0640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f2934161af0 con 0x7f293414e740 2024-09-17T05:21:00.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f29337fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29340acea0 0x7f2934153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:00.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f29337fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29340acea0 0x7f2934153ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41626/0 (socket says 172.21.15.150:41626) 2024-09-17T05:21:00.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f29337fe640 1 -- 172.21.15.150:0/3622935717 learned_addr learned my addr 172.21.15.150:0/3622935717 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:00.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2932ffd640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29340c62d0 0x7f2934157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:00.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2933fff640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f293414e740 0x7f2934152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:00.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f29337fe640 1 -- 172.21.15.150:0/3622935717 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f293414e740 msgr2=0x7f2934152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:00.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f29337fe640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f293414e740 0x7f2934152690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:00.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f29337fe640 1 -- 172.21.15.150:0/3622935717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29340c62d0 msgr2=0x7f2934157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:00.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f29337fe640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29340c62d0 0x7f2934157840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:00.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f29337fe640 1 -- 172.21.15.150:0/3622935717 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2934156730 con 0x7f29340acea0 2024-09-17T05:21:00.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2932ffd640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f29340c62d0 0x7f2934157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:00.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f29337fe640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29340acea0 0x7f2934153ae0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f292400c430 tx=0x7f292400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:00.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f291bfff640 1 -- 172.21.15.150:0/3622935717 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2924017070 con 0x7f29340acea0 2024-09-17T05:21:00.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f291bfff640 1 -- 172.21.15.150:0/3622935717 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2924003600 con 0x7f29340acea0 2024-09-17T05:21:00.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2939ce0640 1 -- 172.21.15.150:0/3622935717 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2934157d80 con 0x7f29340acea0 2024-09-17T05:21:00.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f291bfff640 1 -- 172.21.15.150:0/3622935717 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2924003a50 con 0x7f29340acea0 2024-09-17T05:21:00.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2939ce0640 1 -- 172.21.15.150:0/3622935717 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29341ee970 con 0x7f29340acea0 2024-09-17T05:21:00.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.740+0000 7f2939ce0640 1 -- 172.21.15.150:0/3622935717 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f29340ad8b0 con 0x7f29340acea0 2024-09-17T05:21:00.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.744+0000 7f291bfff640 1 -- 172.21.15.150:0/3622935717 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f292401a070 con 0x7f29340acea0 2024-09-17T05:21:00.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.744+0000 7f291bfff640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2920073230 0x7f2920075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:00.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.744+0000 7f291bfff640 1 -- 172.21.15.150:0/3622935717 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f2924094cf0 con 0x7f29340acea0 2024-09-17T05:21:00.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.744+0000 7f291bfff640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] conn(0x7f292007fd50 0x7f2920082240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:00.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.744+0000 7f2932ffd640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2920073230 0x7f2920075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:00.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.744+0000 7f291bfff640 1 -- 172.21.15.150:0/3622935717 --> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2920082910 con 0x7f292007fd50 2024-09-17T05:21:00.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.744+0000 7f291bfff640 1 -- 172.21.15.150:0/3622935717 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f29240950e0 con 0x7f29340acea0 2024-09-17T05:21:00.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.744+0000 7f2933fff640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] conn(0x7f292007fd50 0x7f2920082240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:00.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.744+0000 7f2933fff640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] conn(0x7f292007fd50 0x7f2920082240 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:00.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.744+0000 7f2932ffd640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2920073230 0x7f2920075740 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f29340c76d0 tx=0x7f2928005d20 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:00.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.744+0000 7f291bfff640 1 -- 172.21.15.150:0/3622935717 <== osd.0 v2:172.21.15.150:6800/437062879 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2920082910 con 0x7f292007fd50 2024-09-17T05:21:00.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.756+0000 7f2939ce0640 1 -- 172.21.15.150:0/3622935717 --> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f29340c49e0 con 0x7f292007fd50 2024-09-17T05:21:00.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.760+0000 7f291bfff640 1 -- 172.21.15.150:0/3622935717 <== osd.0 v2:172.21.15.150:6800/437062879 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f29340c49e0 con 0x7f292007fd50 2024-09-17T05:21:00.756 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-17T05:21:00.757 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:21:00.757 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-17T05:21:00.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.760+0000 7f2939ce0640 1 -- 172.21.15.150:0/3622935717 >> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] conn(0x7f292007fd50 msgr2=0x7f2920082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:00.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.760+0000 7f2939ce0640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.150:6800/437062879,v1:172.21.15.150:6801/437062879] conn(0x7f292007fd50 0x7f2920082240 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:00.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.760+0000 7f2939ce0640 1 -- 172.21.15.150:0/3622935717 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2920073230 msgr2=0x7f2920075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:00.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.760+0000 7f2939ce0640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f2920073230 0x7f2920075740 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f29340c76d0 tx=0x7f2928005d20 comp rx=0 tx=0).stop 2024-09-17T05:21:00.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.760+0000 7f2939ce0640 1 -- 172.21.15.150:0/3622935717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29340acea0 msgr2=0x7f2934153ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:00.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.760+0000 7f2939ce0640 1 --2- 172.21.15.150:0/3622935717 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29340acea0 0x7f2934153ae0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f292400c430 tx=0x7f292400c8f0 comp rx=0 tx=0).stop 2024-09-17T05:21:00.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.760+0000 7f2933fff640 1 -- 172.21.15.150:0/3622935717 reap_dead start 2024-09-17T05:21:00.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.760+0000 7f2939ce0640 1 -- 172.21.15.150:0/3622935717 shutdown_connections 2024-09-17T05:21:00.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.760+0000 7f2939ce0640 1 -- 172.21.15.150:0/3622935717 >> 172.21.15.150:0/3622935717 conn(0x7f29340c9eb0 msgr2=0x7f293414bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:00.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.760+0000 7f2939ce0640 1 -- 172.21.15.150:0/3622935717 shutdown_connections 2024-09-17T05:21:00.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.760+0000 7f2939ce0640 1 -- 172.21.15.150:0/3622935717 wait complete. 2024-09-17T05:21:00.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-17T05:21:00.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.928+0000 7f45f76e1640 1 -- 172.21.15.150:0/4042141229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f00c37b0 msgr2=0x7f45f01535e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:00.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.928+0000 7f45f76e1640 1 --2- 172.21.15.150:0/4042141229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f00c37b0 0x7f45f01535e0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f45f014dc20 tx=0x7f45e001c8c0 comp rx=0 tx=0).stop 2024-09-17T05:21:00.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.928+0000 7f45f76e1640 1 -- 172.21.15.150:0/4042141229 shutdown_connections 2024-09-17T05:21:00.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.928+0000 7f45f76e1640 1 --2- 172.21.15.150:0/4042141229 >> v2:172.21.15.176:3300/0 conn(0x7f45f0157810 0x7f45f0153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:00.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.928+0000 7f45f76e1640 1 --2- 172.21.15.150:0/4042141229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f00c37b0 0x7f45f01535e0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:00.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.928+0000 7f45f76e1640 1 -- 172.21.15.150:0/4042141229 >> 172.21.15.150:0/4042141229 conn(0x7f45f00aab60 msgr2=0x7f45f00aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:00.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.928+0000 7f45f76e1640 1 -- 172.21.15.150:0/4042141229 shutdown_connections 2024-09-17T05:21:00.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.928+0000 7f45f76e1640 1 -- 172.21.15.150:0/4042141229 wait complete. 2024-09-17T05:21:00.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f76e1640 1 Processor -- start 2024-09-17T05:21:00.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f76e1640 1 -- start start 2024-09-17T05:21:00.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f76e1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f00a8120 0x7f45f0153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:00.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f76e1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f00c37b0 0x7f45f01ed0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:00.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f76e1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45f0157810 0x7f45f0148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:00.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f76e1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f45f0160300 con 0x7f45f00c37b0 2024-09-17T05:21:00.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f76e1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f45f0160180 con 0x7f45f00a8120 2024-09-17T05:21:00.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f76e1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f45f0160480 con 0x7f45f0157810 2024-09-17T05:21:00.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f4c7c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f00c37b0 0x7f45f01ed0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:00.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f4c7c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f00c37b0 0x7f45f01ed0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41648/0 (socket says 172.21.15.150:41648) 2024-09-17T05:21:00.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f4c7c640 1 -- 172.21.15.150:0/2925501425 learned_addr learned my addr 172.21.15.150:0/2925501425 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:00.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f5c7e640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45f0157810 0x7f45f0148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:00.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f547d640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f00a8120 0x7f45f0153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:00.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f4c7c640 1 -- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45f0157810 msgr2=0x7f45f0148df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:00.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f4c7c640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45f0157810 0x7f45f0148df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:00.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f4c7c640 1 -- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f00a8120 msgr2=0x7f45f0153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:00.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f4c7c640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f00a8120 0x7f45f0153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:00.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f4c7c640 1 -- 172.21.15.150:0/2925501425 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f01ec8f0 con 0x7f45f00c37b0 2024-09-17T05:21:00.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f4c7c640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f00c37b0 0x7f45f01ed0d0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f45e001cda0 tx=0x7f45e0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:00.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45e5ffb640 1 -- 172.21.15.150:0/2925501425 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e00b2070 con 0x7f45f00c37b0 2024-09-17T05:21:00.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45e5ffb640 1 -- 172.21.15.150:0/2925501425 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45e00b3040 con 0x7f45f00c37b0 2024-09-17T05:21:00.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f76e1640 1 -- 172.21.15.150:0/2925501425 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45f014fcd0 con 0x7f45f00c37b0 2024-09-17T05:21:00.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45e5ffb640 1 -- 172.21.15.150:0/2925501425 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e0005290 con 0x7f45f00c37b0 2024-09-17T05:21:00.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f76e1640 1 -- 172.21.15.150:0/2925501425 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45f014ef50 con 0x7f45f00c37b0 2024-09-17T05:21:00.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.932+0000 7f45f76e1640 1 -- 172.21.15.150:0/2925501425 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45f0149f40 con 0x7f45f00c37b0 2024-09-17T05:21:00.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.936+0000 7f45e5ffb640 1 -- 172.21.15.150:0/2925501425 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f45e0005430 con 0x7f45f00c37b0 2024-09-17T05:21:00.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.936+0000 7f45e5ffb640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f45d4073230 0x7f45d4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:00.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.936+0000 7f45e5ffb640 1 -- 172.21.15.150:0/2925501425 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f45e0132290 con 0x7f45f00c37b0 2024-09-17T05:21:00.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.936+0000 7f45f547d640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f45d4073230 0x7f45d4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:00.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.936+0000 7f45f547d640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f45d4073230 0x7f45d4075740 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009f20 tx=0x7f45ec0094d0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:00.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:00.940+0000 7f45e5ffb640 1 -- 172.21.15.150:0/2925501425 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f45e00164b0 con 0x7f45f00c37b0 2024-09-17T05:21:01.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.104+0000 7f45f76e1640 1 -- 172.21.15.150:0/2925501425 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f45f0150890 con 0x7f45d4073230 2024-09-17T05:21:01.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.104+0000 7f45e5ffb640 1 -- 172.21.15.150:0/2925501425 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f45f0150890 con 0x7f45d4073230 2024-09-17T05:21:01.104 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 0 to deep-scrub 2024-09-17T05:21:01.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.108+0000 7f45f76e1640 1 -- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f45d4073230 msgr2=0x7f45d4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.108+0000 7f45f76e1640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f45d4073230 0x7f45d4075740 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009f20 tx=0x7f45ec0094d0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.108+0000 7f45f76e1640 1 -- 172.21.15.150:0/2925501425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f00c37b0 msgr2=0x7f45f01ed0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.108+0000 7f45f76e1640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f00c37b0 0x7f45f01ed0d0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f45e001cda0 tx=0x7f45e0004600 comp rx=0 tx=0).stop 2024-09-17T05:21:01.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.108+0000 7f45f76e1640 1 -- 172.21.15.150:0/2925501425 shutdown_connections 2024-09-17T05:21:01.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.108+0000 7f45f76e1640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f45d4073230 0x7f45d4075740 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.108+0000 7f45f76e1640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f45f0157810 0x7f45f0148df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.108+0000 7f45f76e1640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45f00c37b0 0x7f45f01ed0d0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.108+0000 7f45f76e1640 1 --2- 172.21.15.150:0/2925501425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f00a8120 0x7f45f0153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.108+0000 7f45f76e1640 1 -- 172.21.15.150:0/2925501425 >> 172.21.15.150:0/2925501425 conn(0x7f45f00aab60 msgr2=0x7f45f0155570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:01.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.108+0000 7f45f76e1640 1 -- 172.21.15.150:0/2925501425 shutdown_connections 2024-09-17T05:21:01.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.108+0000 7f45f76e1640 1 -- 172.21.15.150:0/2925501425 wait complete. 2024-09-17T05:21:01.127 INFO:tasks.ceph:Scrubbing osd.1 2024-09-17T05:21:01.128 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:21:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 -- 172.21.15.150:0/3281149077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c153410 msgr2=0x7fbf6c15d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 --2- 172.21.15.150:0/3281149077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c153410 0x7fbf6c15d680 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fbf6400ad90 tx=0x7fbf6401db90 comp rx=0 tx=0).stop 2024-09-17T05:21:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 -- 172.21.15.150:0/3281149077 shutdown_connections 2024-09-17T05:21:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 --2- 172.21.15.150:0/3281149077 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c153410 0x7fbf6c15d680 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 --2- 172.21.15.150:0/3281149077 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6c1503b0 0x7fbf6c073e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 -- 172.21.15.150:0/3281149077 >> 172.21.15.150:0/3281149077 conn(0x7fbf6c0c92d0 msgr2=0x7fbf6c0c5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 -- 172.21.15.150:0/3281149077 shutdown_connections 2024-09-17T05:21:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 -- 172.21.15.150:0/3281149077 wait complete. 2024-09-17T05:21:01.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 Processor -- start 2024-09-17T05:21:01.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 -- start start 2024-09-17T05:21:01.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c074380 0x7fbf6c0ac020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6c1503b0 0x7fbf6c0bdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf6c153410 0x7fbf6c0be0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c0c2d30 con 0x7fbf6c074380 2024-09-17T05:21:01.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbf6c0c50e0 con 0x7fbf6c153410 2024-09-17T05:21:01.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf71339640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fbf6c0c2eb0 con 0x7fbf6c1503b0 2024-09-17T05:21:01.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf6affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c074380 0x7fbf6c0ac020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:01.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf6affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c074380 0x7fbf6c0ac020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41668/0 (socket says 172.21.15.150:41668) 2024-09-17T05:21:01.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf6affd640 1 -- 172.21.15.150:0/493287571 learned_addr learned my addr 172.21.15.150:0/493287571 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:01.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf627fc640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6c1503b0 0x7fbf6c0bdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:01.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf6affd640 1 -- 172.21.15.150:0/493287571 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6c1503b0 msgr2=0x7fbf6c0bdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf6affd640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fbf6c1503b0 0x7fbf6c0bdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf6affd640 1 -- 172.21.15.150:0/493287571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf6c153410 msgr2=0x7fbf6c0be0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf6affd640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbf6c153410 0x7fbf6c0be0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.288+0000 7fbf6affd640 1 -- 172.21.15.150:0/493287571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf6c1ee500 con 0x7fbf6c074380 2024-09-17T05:21:01.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf6affd640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c074380 0x7fbf6c0ac020 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00b440 tx=0x7fbf5c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:01.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf63fff640 1 -- 172.21.15.150:0/493287571 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c015020 con 0x7fbf6c074380 2024-09-17T05:21:01.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf63fff640 1 -- 172.21.15.150:0/493287571 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf5c0119a0 con 0x7fbf6c074380 2024-09-17T05:21:01.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf71339640 1 -- 172.21.15.150:0/493287571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf6c0c2020 con 0x7fbf6c074380 2024-09-17T05:21:01.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf63fff640 1 -- 172.21.15.150:0/493287571 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c011df0 con 0x7fbf6c074380 2024-09-17T05:21:01.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf71339640 1 -- 172.21.15.150:0/493287571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf6c0c8340 con 0x7fbf6c074380 2024-09-17T05:21:01.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf71339640 1 -- 172.21.15.150:0/493287571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbf6c1ee9b0 con 0x7fbf6c074380 2024-09-17T05:21:01.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf63fff640 1 -- 172.21.15.150:0/493287571 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fbf5c018020 con 0x7fbf6c074380 2024-09-17T05:21:01.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf63fff640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf4c073200 0x7fbf4c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf63fff640 1 -- 172.21.15.150:0/493287571 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7fbf5c095bc0 con 0x7fbf6c074380 2024-09-17T05:21:01.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf63fff640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] conn(0x7fbf4c07fd20 0x7fbf4c082210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf63fff640 1 -- 172.21.15.150:0/493287571 --> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf5c00fe50 con 0x7fbf4c07fd20 2024-09-17T05:21:01.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf63fff640 1 -- 172.21.15.150:0/493287571 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7fbf5c095fb0 con 0x7fbf6c074380 2024-09-17T05:21:01.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf627fc640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf4c073200 0x7fbf4c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:01.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf6b7fe640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] conn(0x7fbf4c07fd20 0x7fbf4c082210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf6b7fe640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] conn(0x7fbf4c07fd20 0x7fbf4c082210 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.292+0000 7fbf627fc640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf4c073200 0x7fbf4c075710 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c1544a0 tx=0x7fbf54009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.296+0000 7fbf63fff640 1 -- 172.21.15.150:0/493287571 <== osd.1 v2:172.21.15.150:6808/4273227163 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbf5c00fe50 con 0x7fbf4c07fd20 2024-09-17T05:21:01.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf71339640 1 -- 172.21.15.150:0/493287571 --> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbf6c0abca0 con 0x7fbf4c07fd20 2024-09-17T05:21:01.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf63fff640 1 -- 172.21.15.150:0/493287571 <== osd.1 v2:172.21.15.150:6808/4273227163 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbf6c0abca0 con 0x7fbf4c07fd20 2024-09-17T05:21:01.306 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-17T05:21:01.306 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:21:01.306 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-17T05:21:01.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf71339640 1 -- 172.21.15.150:0/493287571 >> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] conn(0x7fbf4c07fd20 msgr2=0x7fbf4c082210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf71339640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.150:6808/4273227163,v1:172.21.15.150:6809/4273227163] conn(0x7fbf4c07fd20 0x7fbf4c082210 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf71339640 1 -- 172.21.15.150:0/493287571 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf4c073200 msgr2=0x7fbf4c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf71339640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fbf4c073200 0x7fbf4c075710 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c1544a0 tx=0x7fbf54009040 comp rx=0 tx=0).stop 2024-09-17T05:21:01.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf71339640 1 -- 172.21.15.150:0/493287571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c074380 msgr2=0x7fbf6c0ac020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf71339640 1 --2- 172.21.15.150:0/493287571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf6c074380 0x7fbf6c0ac020 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00b440 tx=0x7fbf5c00b900 comp rx=0 tx=0).stop 2024-09-17T05:21:01.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf6b7fe640 1 -- 172.21.15.150:0/493287571 reap_dead start 2024-09-17T05:21:01.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf71339640 1 -- 172.21.15.150:0/493287571 shutdown_connections 2024-09-17T05:21:01.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf71339640 1 -- 172.21.15.150:0/493287571 >> 172.21.15.150:0/493287571 conn(0x7fbf6c0c92d0 msgr2=0x7fbf6c0aa210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:01.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf71339640 1 -- 172.21.15.150:0/493287571 shutdown_connections 2024-09-17T05:21:01.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.308+0000 7fbf71339640 1 -- 172.21.15.150:0/493287571 wait complete. 2024-09-17T05:21:01.326 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-17T05:21:01.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.480+0000 7ff0d3519640 1 -- 172.21.15.150:0/2573932998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc073440 msgr2=0x7ff0cc0a7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.480+0000 7ff0d3519640 1 --2- 172.21.15.150:0/2573932998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc073440 0x7ff0cc0a7300 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009c80 tx=0x7ff0c001cd10 comp rx=0 tx=0).stop 2024-09-17T05:21:01.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.480+0000 7ff0d3519640 1 -- 172.21.15.150:0/2573932998 shutdown_connections 2024-09-17T05:21:01.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.480+0000 7ff0d3519640 1 --2- 172.21.15.150:0/2573932998 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0cc074450 0x7ff0cc074860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.480+0000 7ff0d3519640 1 --2- 172.21.15.150:0/2573932998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc073440 0x7ff0cc0a7300 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.480+0000 7ff0d3519640 1 -- 172.21.15.150:0/2573932998 >> 172.21.15.150:0/2573932998 conn(0x7ff0cc0c4a00 msgr2=0x7ff0cc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:01.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.480+0000 7ff0d3519640 1 -- 172.21.15.150:0/2573932998 shutdown_connections 2024-09-17T05:21:01.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.480+0000 7ff0d3519640 1 -- 172.21.15.150:0/2573932998 wait complete. 2024-09-17T05:21:01.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.480+0000 7ff0d3519640 1 Processor -- start 2024-09-17T05:21:01.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.480+0000 7ff0d3519640 1 -- start start 2024-09-17T05:21:01.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d3519640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0cc070e50 0x7ff0cc1478e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d3519640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc073440 0x7ff0cc0a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d3519640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc074450 0x7ff0cc143d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d3519640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff0cc071a20 con 0x7ff0cc073440 2024-09-17T05:21:01.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d3519640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff0cc0adef0 con 0x7ff0cc074450 2024-09-17T05:21:01.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d3519640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff0cc0ad610 con 0x7ff0cc070e50 2024-09-17T05:21:01.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d0ab4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc073440 0x7ff0cc0a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:01.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d0ab4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc073440 0x7ff0cc0a86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41682/0 (socket says 172.21.15.150:41682) 2024-09-17T05:21:01.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d0ab4640 1 -- 172.21.15.150:0/1612775625 learned_addr learned my addr 172.21.15.150:0/1612775625 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:01.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d1ab6640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc074450 0x7ff0cc143d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:01.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d12b5640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0cc070e50 0x7ff0cc1478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:01.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d0ab4640 1 -- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0cc070e50 msgr2=0x7ff0cc1478e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d0ab4640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0cc070e50 0x7ff0cc1478e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d0ab4640 1 -- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc074450 msgr2=0x7ff0cc143d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d0ab4640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc074450 0x7ff0cc143d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d0ab4640 1 -- 172.21.15.150:0/1612775625 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0cc0ab020 con 0x7ff0cc073440 2024-09-17T05:21:01.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d0ab4640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc073440 0x7ff0cc0a86c0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800e740 tx=0x7ff0b800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:01.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0c9ffb640 1 -- 172.21.15.150:0/1612775625 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b80052d0 con 0x7ff0cc073440 2024-09-17T05:21:01.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d3519640 1 -- 172.21.15.150:0/1612775625 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0cc147350 con 0x7ff0cc073440 2024-09-17T05:21:01.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0c9ffb640 1 -- 172.21.15.150:0/1612775625 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff0b800ee50 con 0x7ff0cc073440 2024-09-17T05:21:01.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0c9ffb640 1 -- 172.21.15.150:0/1612775625 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8001b40 con 0x7ff0cc073440 2024-09-17T05:21:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d3519640 1 -- 172.21.15.150:0/1612775625 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0cc146a00 con 0x7ff0cc073440 2024-09-17T05:21:01.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0d3519640 1 -- 172.21.15.150:0/1612775625 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0cc0ada40 con 0x7ff0cc073440 2024-09-17T05:21:01.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.484+0000 7ff0c9ffb640 1 -- 172.21.15.150:0/1612775625 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff0b8003600 con 0x7ff0cc073440 2024-09-17T05:21:01.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.488+0000 7ff0c9ffb640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff0b0073200 0x7ff0b0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.488+0000 7ff0c9ffb640 1 -- 172.21.15.150:0/1612775625 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7ff0b8097d50 con 0x7ff0cc073440 2024-09-17T05:21:01.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.488+0000 7ff0d12b5640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff0b0073200 0x7ff0b0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:01.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.488+0000 7ff0d12b5640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff0b0073200 0x7ff0b0075710 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00af000 tx=0x7ff0c001b550 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:01.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.492+0000 7ff0c9ffb640 1 -- 172.21.15.150:0/1612775625 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff0b8016020 con 0x7ff0cc073440 2024-09-17T05:21:01.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.652+0000 7ff0d3519640 1 -- 172.21.15.150:0/1612775625 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7ff0cc0c60f0 con 0x7ff0b0073200 2024-09-17T05:21:01.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0c9ffb640 1 -- 172.21.15.150:0/1612775625 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff0cc0c60f0 con 0x7ff0b0073200 2024-09-17T05:21:01.653 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 1 to deep-scrub 2024-09-17T05:21:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0d3519640 1 -- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff0b0073200 msgr2=0x7ff0b0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0d3519640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff0b0073200 0x7ff0b0075710 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff0c00af000 tx=0x7ff0c001b550 comp rx=0 tx=0).stop 2024-09-17T05:21:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0d3519640 1 -- 172.21.15.150:0/1612775625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc073440 msgr2=0x7ff0cc0a86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0d3519640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc073440 0x7ff0cc0a86c0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7ff0b800e740 tx=0x7ff0b800ec00 comp rx=0 tx=0).stop 2024-09-17T05:21:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0d3519640 1 -- 172.21.15.150:0/1612775625 shutdown_connections 2024-09-17T05:21:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0d3519640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff0b0073200 0x7ff0b0075710 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0d3519640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0cc074450 0x7ff0cc143d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0d3519640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0cc073440 0x7ff0cc0a86c0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0d3519640 1 --2- 172.21.15.150:0/1612775625 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0cc070e50 0x7ff0cc1478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0d3519640 1 -- 172.21.15.150:0/1612775625 >> 172.21.15.150:0/1612775625 conn(0x7ff0cc0c4a00 msgr2=0x7ff0cc071260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0d3519640 1 -- 172.21.15.150:0/1612775625 shutdown_connections 2024-09-17T05:21:01.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.656+0000 7ff0d3519640 1 -- 172.21.15.150:0/1612775625 wait complete. 2024-09-17T05:21:01.675 INFO:tasks.ceph:Scrubbing osd.2 2024-09-17T05:21:01.676 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:21:01.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.840+0000 7fd953ff4640 1 -- 172.21.15.150:0/1449816240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0c3b80 msgr2=0x7fd94c0c3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.840+0000 7fd953ff4640 1 --2- 172.21.15.150:0/1449816240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0c3b80 0x7fd94c0c3f90 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009840 tx=0x7fd93c01c700 comp rx=0 tx=0).stop 2024-09-17T05:21:01.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.840+0000 7fd953ff4640 1 -- 172.21.15.150:0/1449816240 shutdown_connections 2024-09-17T05:21:01.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.840+0000 7fd953ff4640 1 --2- 172.21.15.150:0/1449816240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0c3b80 0x7fd94c0c3f90 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.840+0000 7fd953ff4640 1 --2- 172.21.15.150:0/1449816240 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd94c0c34d0 0x7fd94c0c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.840+0000 7fd953ff4640 1 -- 172.21.15.150:0/1449816240 >> 172.21.15.150:0/1449816240 conn(0x7fd94c0c7fc0 msgr2=0x7fd94c0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:01.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.840+0000 7fd953ff4640 1 -- 172.21.15.150:0/1449816240 shutdown_connections 2024-09-17T05:21:01.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.840+0000 7fd953ff4640 1 -- 172.21.15.150:0/1449816240 wait complete. 2024-09-17T05:21:01.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd953ff4640 1 Processor -- start 2024-09-17T05:21:01.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd953ff4640 1 -- start start 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd953ff4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd94c0c34d0 0x7fd94c1f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd953ff4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0c3b80 0x7fd94c1efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd953ff4640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd94c0c5cc0 0x7fd94c1516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd953ff4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd94c0713a0 con 0x7fd94c0c3b80 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd953ff4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd94c0c7430 con 0x7fd94c0c34d0 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd953ff4640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd94c0c64e0 con 0x7fd94c0c5cc0 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd95158f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0c3b80 0x7fd94c1efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd95158f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0c3b80 0x7fd94c1efd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41698/0 (socket says 172.21.15.150:41698) 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd95158f640 1 -- 172.21.15.150:0/3744127187 learned_addr learned my addr 172.21.15.150:0/3744127187 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd95158f640 1 -- 172.21.15.150:0/3744127187 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd94c0c5cc0 msgr2=0x7fd94c1516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd95158f640 1 --2- 172.21.15.150:0/3744127187 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd94c0c5cc0 0x7fd94c1516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd95158f640 1 -- 172.21.15.150:0/3744127187 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd94c0c34d0 msgr2=0x7fd94c1f3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd95158f640 1 --2- 172.21.15.150:0/3744127187 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd94c0c34d0 0x7fd94c1f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd95158f640 1 -- 172.21.15.150:0/3744127187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94c0abd90 con 0x7fd94c0c3b80 2024-09-17T05:21:01.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd95158f640 1 --2- 172.21.15.150:0/3744127187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0c3b80 0x7fd94c1efd60 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0980e0 tx=0x7fd93c0a8430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:01.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd9427fc640 1 -- 172.21.15.150:0/3744127187 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c0be070 con 0x7fd94c0c3b80 2024-09-17T05:21:01.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd9427fc640 1 -- 172.21.15.150:0/3744127187 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd93c0a85a0 con 0x7fd94c0c3b80 2024-09-17T05:21:01.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd953ff4640 1 -- 172.21.15.150:0/3744127187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd94c0bdef0 con 0x7fd94c0c3b80 2024-09-17T05:21:01.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd9427fc640 1 -- 172.21.15.150:0/3744127187 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c0a8cd0 con 0x7fd94c0c3b80 2024-09-17T05:21:01.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd953ff4640 1 -- 172.21.15.150:0/3744127187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd94c0bdc10 con 0x7fd94c0c3b80 2024-09-17T05:21:01.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.844+0000 7fd953ff4640 1 -- 172.21.15.150:0/3744127187 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd94c1570f0 con 0x7fd94c0c3b80 2024-09-17T05:21:01.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.848+0000 7fd9427fc640 1 -- 172.21.15.150:0/3744127187 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd93c005040 con 0x7fd94c0c3b80 2024-09-17T05:21:01.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.848+0000 7fd9427fc640 1 --2- 172.21.15.150:0/3744127187 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd930073230 0x7fd930075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.848+0000 7fd9427fc640 1 -- 172.21.15.150:0/3744127187 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7fd93c12fc50 con 0x7fd94c0c3b80 2024-09-17T05:21:01.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.848+0000 7fd9427fc640 1 --2- 172.21.15.150:0/3744127187 >> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] conn(0x7fd93007fcd0 0x7fd9300821c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:01.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.848+0000 7fd9427fc640 1 -- 172.21.15.150:0/3744127187 --> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd93c00b0b0 con 0x7fd93007fcd0 2024-09-17T05:21:01.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.848+0000 7fd9427fc640 1 -- 172.21.15.150:0/3744127187 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7fd93c130040 con 0x7fd94c0c3b80 2024-09-17T05:21:01.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.848+0000 7fd951d90640 1 --2- 172.21.15.150:0/3744127187 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd930073230 0x7fd930075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:01.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.848+0000 7fd952591640 1 --2- 172.21.15.150:0/3744127187 >> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] conn(0x7fd93007fcd0 0x7fd9300821c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:01.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.848+0000 7fd952591640 1 --2- 172.21.15.150:0/3744127187 >> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] conn(0x7fd93007fcd0 0x7fd9300821c0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:01.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.848+0000 7fd951d90640 1 --2- 172.21.15.150:0/3744127187 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd930073230 0x7fd930075740 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd9480058d0 tx=0x7fd948005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:01.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.848+0000 7fd9427fc640 1 -- 172.21.15.150:0/3744127187 <== osd.2 v2:172.21.15.150:6816/1456340777 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd93c00b0b0 con 0x7fd93007fcd0 2024-09-17T05:21:01.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd953ff4640 1 -- 172.21.15.150:0/3744127187 --> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd94c152390 con 0x7fd93007fcd0 2024-09-17T05:21:01.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd9427fc640 1 -- 172.21.15.150:0/3744127187 <== osd.2 v2:172.21.15.150:6816/1456340777 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd94c152390 con 0x7fd93007fcd0 2024-09-17T05:21:01.859 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-17T05:21:01.859 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:21:01.859 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-17T05:21:01.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd953ff4640 1 -- 172.21.15.150:0/3744127187 >> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] conn(0x7fd93007fcd0 msgr2=0x7fd9300821c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd953ff4640 1 --2- 172.21.15.150:0/3744127187 >> [v2:172.21.15.150:6816/1456340777,v1:172.21.15.150:6817/1456340777] conn(0x7fd93007fcd0 0x7fd9300821c0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:01.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd953ff4640 1 -- 172.21.15.150:0/3744127187 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd930073230 msgr2=0x7fd930075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd953ff4640 1 --2- 172.21.15.150:0/3744127187 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd930073230 0x7fd930075740 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd9480058d0 tx=0x7fd948005860 comp rx=0 tx=0).stop 2024-09-17T05:21:01.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd953ff4640 1 -- 172.21.15.150:0/3744127187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0c3b80 msgr2=0x7fd94c1efd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:01.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd953ff4640 1 --2- 172.21.15.150:0/3744127187 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0c3b80 0x7fd94c1efd60 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0980e0 tx=0x7fd93c0a8430 comp rx=0 tx=0).stop 2024-09-17T05:21:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd952591640 1 -- 172.21.15.150:0/3744127187 reap_dead start 2024-09-17T05:21:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd953ff4640 1 -- 172.21.15.150:0/3744127187 shutdown_connections 2024-09-17T05:21:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd953ff4640 1 -- 172.21.15.150:0/3744127187 >> 172.21.15.150:0/3744127187 conn(0x7fd94c0c7fc0 msgr2=0x7fd94c155620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd953ff4640 1 -- 172.21.15.150:0/3744127187 shutdown_connections 2024-09-17T05:21:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:01.860+0000 7fd953ff4640 1 -- 172.21.15.150:0/3744127187 wait complete. 2024-09-17T05:21:01.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-17T05:21:02.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 -- 172.21.15.150:0/3462252104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffacc073440 msgr2=0x7ffacc0a7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 --2- 172.21.15.150:0/3462252104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffacc073440 0x7ffacc0a7300 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009900 tx=0x7ffabc01c850 comp rx=0 tx=0).stop 2024-09-17T05:21:02.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 -- 172.21.15.150:0/3462252104 shutdown_connections 2024-09-17T05:21:02.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 --2- 172.21.15.150:0/3462252104 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffacc070e50 0x7ffacc071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 --2- 172.21.15.150:0/3462252104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffacc073440 0x7ffacc0a7300 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 -- 172.21.15.150:0/3462252104 >> 172.21.15.150:0/3462252104 conn(0x7ffacc0c4a00 msgr2=0x7ffacc0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:02.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 -- 172.21.15.150:0/3462252104 shutdown_connections 2024-09-17T05:21:02.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 -- 172.21.15.150:0/3462252104 wait complete. 2024-09-17T05:21:02.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 Processor -- start 2024-09-17T05:21:02.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 -- start start 2024-09-17T05:21:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffacc070e50 0x7ffacc0ac260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffacc0717a0 0x7ffacc0c1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffacc073440 0x7ffacc149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc0750f0 con 0x7ffacc0717a0 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc074f70 con 0x7ffacc073440 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffad081e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffacc0ad610 con 0x7ffacc070e50 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffac3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffacc0717a0 0x7ffacc0c1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffac3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffacc0717a0 0x7ffacc0c1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41716/0 (socket says 172.21.15.150:41716) 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.032+0000 7ffac3fff640 1 -- 172.21.15.150:0/303149087 learned_addr learned my addr 172.21.15.150:0/303149087 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffacad9d640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffacc073440 0x7ffacc149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffaca59c640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffacc070e50 0x7ffacc0ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffac3fff640 1 -- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffacc070e50 msgr2=0x7ffacc0ac260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffac3fff640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffacc070e50 0x7ffacc0ac260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffac3fff640 1 -- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffacc073440 msgr2=0x7ffacc149820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffac3fff640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffacc073440 0x7ffacc149820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffac3fff640 1 -- 172.21.15.150:0/303149087 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffacc0ac7a0 con 0x7ffacc0717a0 2024-09-17T05:21:02.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffacad9d640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffacc073440 0x7ffacc149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:02.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffaca59c640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffacc070e50 0x7ffacc0ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:02.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffac3fff640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffacc0717a0 0x7ffacc0c1160 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7ffab400b440 tx=0x7ffab400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:02.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffac2ffd640 1 -- 172.21.15.150:0/303149087 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab4015020 con 0x7ffacc0717a0 2024-09-17T05:21:02.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffac2ffd640 1 -- 172.21.15.150:0/303149087 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffab40119a0 con 0x7ffacc0717a0 2024-09-17T05:21:02.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffac2ffd640 1 -- 172.21.15.150:0/303149087 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab4011df0 con 0x7ffacc0717a0 2024-09-17T05:21:02.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffad081e640 1 -- 172.21.15.150:0/303149087 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffacc0c16a0 con 0x7ffacc0717a0 2024-09-17T05:21:02.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffad081e640 1 -- 172.21.15.150:0/303149087 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffacc148170 con 0x7ffacc0717a0 2024-09-17T05:21:02.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffad081e640 1 -- 172.21.15.150:0/303149087 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffacc0adab0 con 0x7ffacc0717a0 2024-09-17T05:21:02.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffac2ffd640 1 -- 172.21.15.150:0/303149087 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ffab4018020 con 0x7ffacc0717a0 2024-09-17T05:21:02.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffac2ffd640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffaac073230 0x7ffaac075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffac2ffd640 1 -- 172.21.15.150:0/303149087 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7ffab4095580 con 0x7ffacc0717a0 2024-09-17T05:21:02.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.036+0000 7ffaca59c640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffaac073230 0x7ffaac075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.040+0000 7ffaca59c640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffaac073230 0x7ffaac075740 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ffabc0af000 tx=0x7ffabc01b090 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:02.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.040+0000 7ffac2ffd640 1 -- 172.21.15.150:0/303149087 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ffab401f020 con 0x7ffacc0717a0 2024-09-17T05:21:02.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.200+0000 7ffad081e640 1 -- 172.21.15.150:0/303149087 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7ffacc0c60f0 con 0x7ffaac073230 2024-09-17T05:21:02.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.200+0000 7ffac2ffd640 1 -- 172.21.15.150:0/303149087 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ffacc0c60f0 con 0x7ffaac073230 2024-09-17T05:21:02.199 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 2 to deep-scrub 2024-09-17T05:21:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.204+0000 7ffad081e640 1 -- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffaac073230 msgr2=0x7ffaac075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.204+0000 7ffad081e640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffaac073230 0x7ffaac075740 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ffabc0af000 tx=0x7ffabc01b090 comp rx=0 tx=0).stop 2024-09-17T05:21:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.204+0000 7ffad081e640 1 -- 172.21.15.150:0/303149087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffacc0717a0 msgr2=0x7ffacc0c1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.204+0000 7ffad081e640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffacc0717a0 0x7ffacc0c1160 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7ffab400b440 tx=0x7ffab400b900 comp rx=0 tx=0).stop 2024-09-17T05:21:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.204+0000 7ffad081e640 1 -- 172.21.15.150:0/303149087 shutdown_connections 2024-09-17T05:21:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.204+0000 7ffad081e640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffaac073230 0x7ffaac075740 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.204+0000 7ffad081e640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffacc073440 0x7ffacc149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.204+0000 7ffad081e640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffacc0717a0 0x7ffacc0c1160 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.204+0000 7ffad081e640 1 --2- 172.21.15.150:0/303149087 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffacc070e50 0x7ffacc0ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.204+0000 7ffad081e640 1 -- 172.21.15.150:0/303149087 >> 172.21.15.150:0/303149087 conn(0x7ffacc0c4a00 msgr2=0x7ffacc152d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.204+0000 7ffad081e640 1 -- 172.21.15.150:0/303149087 shutdown_connections 2024-09-17T05:21:02.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.204+0000 7ffad081e640 1 -- 172.21.15.150:0/303149087 wait complete. 2024-09-17T05:21:02.223 INFO:tasks.ceph:Scrubbing osd.3 2024-09-17T05:21:02.223 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:21:02.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.384+0000 7fc0f762b640 1 -- 172.21.15.150:0/288066330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0f0070be0 msgr2=0x7fc0f0070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.384+0000 7fc0f762b640 1 --2- 172.21.15.150:0/288066330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0f0070be0 0x7fc0f0070ff0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009900 tx=0x7fc0ec01c8e0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.384+0000 7fc0f762b640 1 -- 172.21.15.150:0/288066330 shutdown_connections 2024-09-17T05:21:02.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.384+0000 7fc0f762b640 1 --2- 172.21.15.150:0/288066330 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0f0070be0 0x7fc0f0070ff0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.384+0000 7fc0f762b640 1 --2- 172.21.15.150:0/288066330 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0f00c27c0 0x7fc0f0151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.384+0000 7fc0f762b640 1 -- 172.21.15.150:0/288066330 >> 172.21.15.150:0/288066330 conn(0x7fc0f00c7fc0 msgr2=0x7fc0f00c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:02.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.384+0000 7fc0f762b640 1 -- 172.21.15.150:0/288066330 shutdown_connections 2024-09-17T05:21:02.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.384+0000 7fc0f762b640 1 -- 172.21.15.150:0/288066330 wait complete. 2024-09-17T05:21:02.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f762b640 1 Processor -- start 2024-09-17T05:21:02.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f762b640 1 -- start start 2024-09-17T05:21:02.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f762b640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0f0070be0 0x7fc0f01ed870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f762b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0f00c27c0 0x7fc0f00aa520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f762b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0f0157d40 0x7fc0f0155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f762b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f00a7290 con 0x7fc0f00c27c0 2024-09-17T05:21:02.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f762b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f00a7110 con 0x7fc0f0157d40 2024-09-17T05:21:02.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f762b640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc0f00c22f0 con 0x7fc0f0070be0 2024-09-17T05:21:02.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f4bc6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0f00c27c0 0x7fc0f00aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f4bc6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0f00c27c0 0x7fc0f00aa520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41736/0 (socket says 172.21.15.150:41736) 2024-09-17T05:21:02.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f4bc6640 1 -- 172.21.15.150:0/1630178329 learned_addr learned my addr 172.21.15.150:0/1630178329 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:02.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f53c7640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0f0070be0 0x7fc0f01ed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f4bc6640 1 -- 172.21.15.150:0/1630178329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0f0070be0 msgr2=0x7fc0f01ed870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f4bc6640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc0f0070be0 0x7fc0f01ed870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f4bc6640 1 -- 172.21.15.150:0/1630178329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0f0157d40 msgr2=0x7fc0f0155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f4bc6640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc0f0157d40 0x7fc0f0155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f4bc6640 1 -- 172.21.15.150:0/1630178329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f00a8b10 con 0x7fc0f00c27c0 2024-09-17T05:21:02.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0f4bc6640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0f00c27c0 0x7fc0f00aa520 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0b0040 tx=0x7fc0ec0a5620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:02.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0e5ffb640 1 -- 172.21.15.150:0/1630178329 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0bb070 con 0x7fc0f00c27c0 2024-09-17T05:21:02.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0e5ffb640 1 -- 172.21.15.150:0/1630178329 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0ec0043b0 con 0x7fc0f00c27c0 2024-09-17T05:21:02.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.388+0000 7fc0e5ffb640 1 -- 172.21.15.150:0/1630178329 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec0b2040 con 0x7fc0f00c27c0 2024-09-17T05:21:02.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0f762b640 1 -- 172.21.15.150:0/1630178329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f0149cb0 con 0x7fc0f00c27c0 2024-09-17T05:21:02.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0f762b640 1 -- 172.21.15.150:0/1630178329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f0149800 con 0x7fc0f00c27c0 2024-09-17T05:21:02.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0f762b640 1 -- 172.21.15.150:0/1630178329 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc0f0070ff0 con 0x7fc0f00c27c0 2024-09-17T05:21:02.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0e5ffb640 1 -- 172.21.15.150:0/1630178329 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc0ec005290 con 0x7fc0f00c27c0 2024-09-17T05:21:02.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0e5ffb640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc0d4073200 0x7fc0d4075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0e5ffb640 1 -- 172.21.15.150:0/1630178329 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7fc0ec12f1c0 con 0x7fc0f00c27c0 2024-09-17T05:21:02.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0e5ffb640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] conn(0x7fc0d407fd30 0x7fc0d4082220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0e5ffb640 1 -- 172.21.15.150:0/1630178329 --> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0ec0aee50 con 0x7fc0d407fd30 2024-09-17T05:21:02.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0e5ffb640 1 -- 172.21.15.150:0/1630178329 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7fc0ec12f5b0 con 0x7fc0f00c27c0 2024-09-17T05:21:02.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0f53c7640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc0d4073200 0x7fc0d4075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0f5bc8640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] conn(0x7fc0d407fd30 0x7fc0d4082220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0f5bc8640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] conn(0x7fc0d407fd30 0x7fc0d4082220 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:02.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.392+0000 7fc0f53c7640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc0d4073200 0x7fc0d4075710 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00058d0 tx=0x7fc0e0005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:02.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.396+0000 7fc0e5ffb640 1 -- 172.21.15.150:0/1630178329 <== osd.3 v2:172.21.15.150:6824/158177050 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0ec0aee50 con 0x7fc0d407fd30 2024-09-17T05:21:02.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0f762b640 1 -- 172.21.15.150:0/1630178329 --> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc0f0152650 con 0x7fc0d407fd30 2024-09-17T05:21:02.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0e5ffb640 1 -- 172.21.15.150:0/1630178329 <== osd.3 v2:172.21.15.150:6824/158177050 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc0f0152650 con 0x7fc0d407fd30 2024-09-17T05:21:02.406 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-17T05:21:02.406 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:21:02.406 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-17T05:21:02.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0f762b640 1 -- 172.21.15.150:0/1630178329 >> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] conn(0x7fc0d407fd30 msgr2=0x7fc0d4082220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0f762b640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.150:6824/158177050,v1:172.21.15.150:6825/158177050] conn(0x7fc0d407fd30 0x7fc0d4082220 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0f762b640 1 -- 172.21.15.150:0/1630178329 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc0d4073200 msgr2=0x7fc0d4075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0f762b640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc0d4073200 0x7fc0d4075710 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00058d0 tx=0x7fc0e0005860 comp rx=0 tx=0).stop 2024-09-17T05:21:02.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0f762b640 1 -- 172.21.15.150:0/1630178329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0f00c27c0 msgr2=0x7fc0f00aa520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0f762b640 1 --2- 172.21.15.150:0/1630178329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc0f00c27c0 0x7fc0f00aa520 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0b0040 tx=0x7fc0ec0a5620 comp rx=0 tx=0).stop 2024-09-17T05:21:02.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0f5bc8640 1 -- 172.21.15.150:0/1630178329 reap_dead start 2024-09-17T05:21:02.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0f762b640 1 -- 172.21.15.150:0/1630178329 shutdown_connections 2024-09-17T05:21:02.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0f762b640 1 -- 172.21.15.150:0/1630178329 >> 172.21.15.150:0/1630178329 conn(0x7fc0f00c7fc0 msgr2=0x7fc0f0158150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:02.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0f762b640 1 -- 172.21.15.150:0/1630178329 shutdown_connections 2024-09-17T05:21:02.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.408+0000 7fc0f762b640 1 -- 172.21.15.150:0/1630178329 wait complete. 2024-09-17T05:21:02.426 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-17T05:21:02.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.568+0000 7ff657dac640 1 -- 172.21.15.150:0/3792983500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff63c003a40 msgr2=0x7ff63c003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.568+0000 7ff657dac640 1 --2- 172.21.15.150:0/3792983500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff63c003a40 0x7ff63c003eb0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00c2c0 tx=0x7ff64c01cf40 comp rx=0 tx=0).stop 2024-09-17T05:21:02.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.568+0000 7ff656349640 1 -- 172.21.15.150:0/3792983500 reap_dead start 2024-09-17T05:21:02.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.568+0000 7ff657dac640 1 -- 172.21.15.150:0/3792983500 shutdown_connections 2024-09-17T05:21:02.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.568+0000 7ff657dac640 1 -- 172.21.15.150:0/3792983500 >> 172.21.15.150:0/3792983500 conn(0x7ff6500aab60 msgr2=0x7ff6500aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:02.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.568+0000 7ff657dac640 1 -- 172.21.15.150:0/3792983500 shutdown_connections 2024-09-17T05:21:02.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.568+0000 7ff657dac640 1 -- 172.21.15.150:0/3792983500 wait complete. 2024-09-17T05:21:02.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff657dac640 1 Processor -- start 2024-09-17T05:21:02.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff657dac640 1 -- start start 2024-09-17T05:21:02.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff657dac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff63c003a40 0x7ff6501489a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff657dac640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff65014be90 0x7ff6500c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff657dac640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff65014d450 0x7ff650144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff657dac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff650160aa0 con 0x7ff63c003a40 2024-09-17T05:21:02.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff657dac640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff6500a8120 con 0x7ff65014d450 2024-09-17T05:21:02.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff657dac640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff650160c20 con 0x7ff65014be90 2024-09-17T05:21:02.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff655b48640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff63c003a40 0x7ff6501489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff655b48640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff63c003a40 0x7ff6501489a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41758/0 (socket says 172.21.15.150:41758) 2024-09-17T05:21:02.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff655b48640 1 -- 172.21.15.150:0/614795090 learned_addr learned my addr 172.21.15.150:0/614795090 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:02.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff656349640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff65014d450 0x7ff650144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff655347640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff65014be90 0x7ff6500c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff655b48640 1 -- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff65014be90 msgr2=0x7ff6500c23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff655b48640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff65014be90 0x7ff6500c23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff655b48640 1 -- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff65014d450 msgr2=0x7ff650144030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff655b48640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff65014d450 0x7ff650144030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff655b48640 1 -- 172.21.15.150:0/614795090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff650149160 con 0x7ff63c003a40 2024-09-17T05:21:02.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff655347640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff65014be90 0x7ff6500c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:02.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff656349640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff65014d450 0x7ff650144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:02.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff655b48640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff63c003a40 0x7ff6501489a0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7ff64c01b640 tx=0x7ff64c001cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:02.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff6467fc640 1 -- 172.21.15.150:0/614795090 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c0014c0 con 0x7ff63c003a40 2024-09-17T05:21:02.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff6467fc640 1 -- 172.21.15.150:0/614795090 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff64c001660 con 0x7ff63c003a40 2024-09-17T05:21:02.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff657dac640 1 -- 172.21.15.150:0/614795090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65014a2d0 con 0x7ff63c003a40 2024-09-17T05:21:02.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff6467fc640 1 -- 172.21.15.150:0/614795090 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c006330 con 0x7ff63c003a40 2024-09-17T05:21:02.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.572+0000 7ff657dac640 1 -- 172.21.15.150:0/614795090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff65014c340 con 0x7ff63c003a40 2024-09-17T05:21:02.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.576+0000 7ff657dac640 1 -- 172.21.15.150:0/614795090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6500ae340 con 0x7ff63c003a40 2024-09-17T05:21:02.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.576+0000 7ff6467fc640 1 -- 172.21.15.150:0/614795090 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff64c0b7030 con 0x7ff63c003a40 2024-09-17T05:21:02.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.576+0000 7ff6467fc640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff634073200 0x7ff634075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.576+0000 7ff6467fc640 1 -- 172.21.15.150:0/614795090 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7ff64c12ea80 con 0x7ff63c003a40 2024-09-17T05:21:02.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.576+0000 7ff655347640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff634073200 0x7ff634075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.576+0000 7ff655347640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff634073200 0x7ff634075710 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7ff65014e3f0 tx=0x7ff6400094d0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:02.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.580+0000 7ff6467fc640 1 -- 172.21.15.150:0/614795090 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff64c016cf0 con 0x7ff63c003a40 2024-09-17T05:21:02.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.748+0000 7ff657dac640 1 -- 172.21.15.150:0/614795090 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7ff65014c5e0 con 0x7ff634073200 2024-09-17T05:21:02.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.748+0000 7ff6467fc640 1 -- 172.21.15.150:0/614795090 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff65014c5e0 con 0x7ff634073200 2024-09-17T05:21:02.748 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 3 to deep-scrub 2024-09-17T05:21:02.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.752+0000 7ff657dac640 1 -- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff634073200 msgr2=0x7ff634075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.752+0000 7ff657dac640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff634073200 0x7ff634075710 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7ff65014e3f0 tx=0x7ff6400094d0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.752+0000 7ff657dac640 1 -- 172.21.15.150:0/614795090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff63c003a40 msgr2=0x7ff6501489a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.752+0000 7ff657dac640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff63c003a40 0x7ff6501489a0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7ff64c01b640 tx=0x7ff64c001cc0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.752+0000 7ff657dac640 1 -- 172.21.15.150:0/614795090 shutdown_connections 2024-09-17T05:21:02.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.752+0000 7ff657dac640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff634073200 0x7ff634075710 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.752+0000 7ff657dac640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff65014d450 0x7ff650144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.752+0000 7ff657dac640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff65014be90 0x7ff6500c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.752+0000 7ff657dac640 1 --2- 172.21.15.150:0/614795090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff63c003a40 0x7ff6501489a0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.752+0000 7ff657dac640 1 -- 172.21.15.150:0/614795090 >> 172.21.15.150:0/614795090 conn(0x7ff6500aab60 msgr2=0x7ff650151960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:02.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.752+0000 7ff657dac640 1 -- 172.21.15.150:0/614795090 shutdown_connections 2024-09-17T05:21:02.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.752+0000 7ff657dac640 1 -- 172.21.15.150:0/614795090 wait complete. 2024-09-17T05:21:02.770 INFO:tasks.ceph:Scrubbing osd.4 2024-09-17T05:21:02.771 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:21:02.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.920+0000 7fb22a993640 1 -- 172.21.15.150:0/1084369836 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2240c34d0 msgr2=0x7fb2240c6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.920+0000 7fb22a993640 1 --2- 172.21.15.150:0/1084369836 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2240c34d0 0x7fb2240c6ef0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb21400aa10 tx=0x7fb21401db10 comp rx=0 tx=0).stop 2024-09-17T05:21:02.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.920+0000 7fb22a993640 1 -- 172.21.15.150:0/1084369836 shutdown_connections 2024-09-17T05:21:02.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.920+0000 7fb22a993640 1 --2- 172.21.15.150:0/1084369836 >> v2:172.21.15.176:3300/0 conn(0x7fb2240c3f90 0x7fb2240746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.920+0000 7fb22a993640 1 --2- 172.21.15.150:0/1084369836 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2240c34d0 0x7fb2240c6ef0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.920+0000 7fb22a993640 1 -- 172.21.15.150:0/1084369836 >> 172.21.15.150:0/1084369836 conn(0x7fb2240c7fc0 msgr2=0x7fb2240c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:02.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.920+0000 7fb22a993640 1 -- 172.21.15.150:0/1084369836 shutdown_connections 2024-09-17T05:21:02.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.920+0000 7fb22a993640 1 -- 172.21.15.150:0/1084369836 wait complete. 2024-09-17T05:21:02.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.920+0000 7fb22a993640 1 Processor -- start 2024-09-17T05:21:02.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.920+0000 7fb22a993640 1 -- start start 2024-09-17T05:21:02.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb22a993640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2240c34d0 0x7fb224151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb22a993640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2240c3b80 0x7fb2240c1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb22a993640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2240c3f90 0x7fb2240acaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb22a993640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb2240713a0 con 0x7fb2240c34d0 2024-09-17T05:21:02.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb22a993640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb2240c7430 con 0x7fb2240c3f90 2024-09-17T05:21:02.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb22a993640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fb2240c63d0 con 0x7fb2240c3b80 2024-09-17T05:21:02.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb223fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2240c34d0 0x7fb224151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb223fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2240c34d0 0x7fb224151650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41768/0 (socket says 172.21.15.150:41768) 2024-09-17T05:21:02.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb223fff640 1 -- 172.21.15.150:0/1714002696 learned_addr learned my addr 172.21.15.150:0/1714002696 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:02.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb228f30640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2240c3f90 0x7fb2240acaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb21b7fe640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2240c3b80 0x7fb2240c1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb223fff640 1 -- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2240c3b80 msgr2=0x7fb2240c1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb223fff640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fb2240c3b80 0x7fb2240c1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb223fff640 1 -- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2240c3f90 msgr2=0x7fb2240acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb223fff640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2240c3f90 0x7fb2240acaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb223fff640 1 -- 172.21.15.150:0/1714002696 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb224149720 con 0x7fb2240c34d0 2024-09-17T05:21:02.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb223fff640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2240c34d0 0x7fb224151650 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb2140b1040 tx=0x7fb214005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:02.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb2217fa640 1 -- 172.21.15.150:0/1714002696 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2140b3070 con 0x7fb2240c34d0 2024-09-17T05:21:02.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb2217fa640 1 -- 172.21.15.150:0/1714002696 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb214005530 con 0x7fb2240c34d0 2024-09-17T05:21:02.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb2217fa640 1 -- 172.21.15.150:0/1714002696 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb214004930 con 0x7fb2240c34d0 2024-09-17T05:21:02.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb22a993640 1 -- 172.21.15.150:0/1714002696 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb224152530 con 0x7fb2240c34d0 2024-09-17T05:21:02.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb22a993640 1 -- 172.21.15.150:0/1714002696 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb224151b90 con 0x7fb2240c34d0 2024-09-17T05:21:02.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb22a993640 1 -- 172.21.15.150:0/1714002696 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb2241f0490 con 0x7fb2240c34d0 2024-09-17T05:21:02.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb2217fa640 1 -- 172.21.15.150:0/1714002696 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fb2140be060 con 0x7fb2240c34d0 2024-09-17T05:21:02.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb2217fa640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb204073230 0x7fb204075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb2217fa640 1 -- 172.21.15.150:0/1714002696 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7fb2141301b0 con 0x7fb2240c34d0 2024-09-17T05:21:02.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb2217fa640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] conn(0x7fb20407fd50 0x7fb204082240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:02.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb2217fa640 1 -- 172.21.15.150:0/1714002696 --> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb214001d40 con 0x7fb20407fd50 2024-09-17T05:21:02.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.924+0000 7fb2217fa640 1 -- 172.21.15.150:0/1714002696 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7fb2141305a0 con 0x7fb2240c34d0 2024-09-17T05:21:02.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.928+0000 7fb21b7fe640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb204073230 0x7fb204075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.928+0000 7fb228f30640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] conn(0x7fb20407fd50 0x7fb204082240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:02.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.928+0000 7fb228f30640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] conn(0x7fb20407fd50 0x7fb204082240 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:02.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.928+0000 7fb21b7fe640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb204073230 0x7fb204075740 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb2241f3bf0 tx=0x7fb20c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:02.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.928+0000 7fb2217fa640 1 -- 172.21.15.150:0/1714002696 <== osd.4 v2:172.21.15.176:6826/4247631702 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb214001d40 con 0x7fb20407fd50 2024-09-17T05:21:02.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.940+0000 7fb22a993640 1 -- 172.21.15.150:0/1714002696 --> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb2241570f0 con 0x7fb20407fd50 2024-09-17T05:21:02.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.944+0000 7fb2217fa640 1 -- 172.21.15.150:0/1714002696 <== osd.4 v2:172.21.15.176:6826/4247631702 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb2241570f0 con 0x7fb20407fd50 2024-09-17T05:21:02.941 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-17T05:21:02.941 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:21:02.941 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-17T05:21:02.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.944+0000 7fb22a993640 1 -- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] conn(0x7fb20407fd50 msgr2=0x7fb204082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.944+0000 7fb22a993640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:6826/4247631702,v1:172.21.15.176:6827/4247631702] conn(0x7fb20407fd50 0x7fb204082240 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.944+0000 7fb22a993640 1 -- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb204073230 msgr2=0x7fb204075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.944+0000 7fb22a993640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fb204073230 0x7fb204075740 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb2241f3bf0 tx=0x7fb20c009040 comp rx=0 tx=0).stop 2024-09-17T05:21:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.944+0000 7fb22a993640 1 -- 172.21.15.150:0/1714002696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2240c34d0 msgr2=0x7fb224151650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.944+0000 7fb22a993640 1 --2- 172.21.15.150:0/1714002696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2240c34d0 0x7fb224151650 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb2140b1040 tx=0x7fb214005de0 comp rx=0 tx=0).stop 2024-09-17T05:21:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.944+0000 7fb228f30640 1 -- 172.21.15.150:0/1714002696 reap_dead start 2024-09-17T05:21:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.944+0000 7fb22a993640 1 -- 172.21.15.150:0/1714002696 shutdown_connections 2024-09-17T05:21:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.944+0000 7fb22a993640 1 -- 172.21.15.150:0/1714002696 >> 172.21.15.150:0/1714002696 conn(0x7fb2240c7fc0 msgr2=0x7fb224154b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.944+0000 7fb22a993640 1 -- 172.21.15.150:0/1714002696 shutdown_connections 2024-09-17T05:21:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:02.944+0000 7fb22a993640 1 -- 172.21.15.150:0/1714002696 wait complete. 2024-09-17T05:21:02.962 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-17T05:21:03.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.112+0000 7fc86d388640 1 -- 172.21.15.150:0/201793449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680a9f10 msgr2=0x7fc8680aa320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.112+0000 7fc86d388640 1 --2- 172.21.15.150:0/201793449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680a9f10 0x7fc8680aa320 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fc868073c90 tx=0x7fc85001c8c0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.112+0000 7fc86d388640 1 -- 172.21.15.150:0/201793449 shutdown_connections 2024-09-17T05:21:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.112+0000 7fc86d388640 1 --2- 172.21.15.150:0/201793449 >> v2:172.21.15.176:3300/0 conn(0x7fc8680c3900 0x7fc8680c3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.112+0000 7fc86d388640 1 --2- 172.21.15.150:0/201793449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680a9f10 0x7fc8680aa320 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.112+0000 7fc86d388640 1 --2- 172.21.15.150:0/201793449 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc868075040 0x7fc8680c4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.112+0000 7fc86d388640 1 -- 172.21.15.150:0/201793449 >> 172.21.15.150:0/201793449 conn(0x7fc868070c20 msgr2=0x7fc868071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.112+0000 7fc86d388640 1 -- 172.21.15.150:0/201793449 shutdown_connections 2024-09-17T05:21:03.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.112+0000 7fc86d388640 1 -- 172.21.15.150:0/201793449 wait complete. 2024-09-17T05:21:03.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc86d388640 1 Processor -- start 2024-09-17T05:21:03.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc86d388640 1 -- start start 2024-09-17T05:21:03.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc86d388640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc868075040 0x7fc868159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc86d388640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680a9f10 0x7fc868156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc86d388640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8680c3900 0x7fc8681489f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc86d388640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc868071c60 con 0x7fc8680a9f10 2024-09-17T05:21:03.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc86d388640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc868073390 con 0x7fc868075040 2024-09-17T05:21:03.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc86d388640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fc8681edff0 con 0x7fc8680c3900 2024-09-17T05:21:03.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc8667fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680a9f10 0x7fc868156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc8667fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680a9f10 0x7fc868156360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41792/0 (socket says 172.21.15.150:41792) 2024-09-17T05:21:03.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc8667fc640 1 -- 172.21.15.150:0/580820026 learned_addr learned my addr 172.21.15.150:0/580820026 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:03.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc8677fe640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8680c3900 0x7fc8681489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc866ffd640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc868075040 0x7fc868159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc8667fc640 1 -- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8680c3900 msgr2=0x7fc8681489f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc8667fc640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8680c3900 0x7fc8681489f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc8667fc640 1 -- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc868075040 msgr2=0x7fc868159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc8667fc640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc868075040 0x7fc868159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc8667fc640 1 -- 172.21.15.150:0/580820026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8681f1280 con 0x7fc8680a9f10 2024-09-17T05:21:03.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc866ffd640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc868075040 0x7fc868159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:03.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc8667fc640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680a9f10 0x7fc868156360 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fc8500b0040 tx=0x7fc8500a66f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:03.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc85f7fe640 1 -- 172.21.15.150:0/580820026 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850004180 con 0x7fc8680a9f10 2024-09-17T05:21:03.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc85f7fe640 1 -- 172.21.15.150:0/580820026 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc850004430 con 0x7fc8680a9f10 2024-09-17T05:21:03.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc86d388640 1 -- 172.21.15.150:0/580820026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8680c2cc0 con 0x7fc8680a9f10 2024-09-17T05:21:03.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc85f7fe640 1 -- 172.21.15.150:0/580820026 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8500052b0 con 0x7fc8680a9f10 2024-09-17T05:21:03.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.116+0000 7fc86d388640 1 -- 172.21.15.150:0/580820026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc86815a470 con 0x7fc8680a9f10 2024-09-17T05:21:03.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.120+0000 7fc86d388640 1 -- 172.21.15.150:0/580820026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8680aa320 con 0x7fc8680a9f10 2024-09-17T05:21:03.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.120+0000 7fc85f7fe640 1 -- 172.21.15.150:0/580820026 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fc8500bf030 con 0x7fc8680a9f10 2024-09-17T05:21:03.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.120+0000 7fc85f7fe640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc848073200 0x7fc848075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.120+0000 7fc85f7fe640 1 -- 172.21.15.150:0/580820026 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7fc85012f890 con 0x7fc8680a9f10 2024-09-17T05:21:03.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.120+0000 7fc866ffd640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc848073200 0x7fc848075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.120+0000 7fc866ffd640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc848073200 0x7fc848075710 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc860005fd0 tx=0x7fc860005d40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:03.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.124+0000 7fc85f7fe640 1 -- 172.21.15.150:0/580820026 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc8500c6060 con 0x7fc8680a9f10 2024-09-17T05:21:03.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 -- 172.21.15.150:0/580820026 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fc86806f1c0 con 0x7fc848073200 2024-09-17T05:21:03.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc85f7fe640 1 -- 172.21.15.150:0/580820026 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc86806f1c0 con 0x7fc848073200 2024-09-17T05:21:03.282 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 4 to deep-scrub 2024-09-17T05:21:03.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 -- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc848073200 msgr2=0x7fc848075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc848073200 0x7fc848075710 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc860005fd0 tx=0x7fc860005d40 comp rx=0 tx=0).stop 2024-09-17T05:21:03.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 -- 172.21.15.150:0/580820026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680a9f10 msgr2=0x7fc868156360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680a9f10 0x7fc868156360 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fc8500b0040 tx=0x7fc8500a66f0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 -- 172.21.15.150:0/580820026 shutdown_connections 2024-09-17T05:21:03.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fc848073200 0x7fc848075710 secure :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc860005fd0 tx=0x7fc860005d40 comp rx=0 tx=0).stop 2024-09-17T05:21:03.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fc8680c3900 0x7fc8681489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8680a9f10 0x7fc868156360 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 --2- 172.21.15.150:0/580820026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc868075040 0x7fc868159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 -- 172.21.15.150:0/580820026 >> 172.21.15.150:0/580820026 conn(0x7fc868070c20 msgr2=0x7fc868153930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:03.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 -- 172.21.15.150:0/580820026 shutdown_connections 2024-09-17T05:21:03.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.284+0000 7fc86d388640 1 -- 172.21.15.150:0/580820026 wait complete. 2024-09-17T05:21:03.304 INFO:tasks.ceph:Scrubbing osd.5 2024-09-17T05:21:03.304 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:21:03.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.460+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1482671846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ac0003a40 msgr2=0x7f5ac0003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.460+0000 7f5adfc3c640 1 --2- 172.21.15.150:0/1482671846 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ac0003a40 0x7f5ac0003eb0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0006fd0 tx=0x7f5ad001cf40 comp rx=0 tx=0).stop 2024-09-17T05:21:03.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.460+0000 7f5ade1d9640 1 -- 172.21.15.150:0/1482671846 reap_dead start 2024-09-17T05:21:03.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.460+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1482671846 shutdown_connections 2024-09-17T05:21:03.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.460+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1482671846 >> 172.21.15.150:0/1482671846 conn(0x7f5ad80c7fc0 msgr2=0x7f5ad80c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:03.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1482671846 shutdown_connections 2024-09-17T05:21:03.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1482671846 wait complete. 2024-09-17T05:21:03.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5adfc3c640 1 Processor -- start 2024-09-17T05:21:03.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5adfc3c640 1 -- start start 2024-09-17T05:21:03.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5adfc3c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ac0003a40 0x7f5ad815a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5adfc3c640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ad80ac9e0 0x7f5ad80c13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5adfc3c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ad81501b0 0x7f5ad81505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5adfc3c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad80c6460 con 0x7f5ad81501b0 2024-09-17T05:21:03.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5adfc3c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad80c62e0 con 0x7f5ac0003a40 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5adfc3c640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad80c65e0 con 0x7f5ad80ac9e0 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5ade1d9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ad81501b0 0x7f5ad81505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5ade1d9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ad81501b0 0x7f5ad81505c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41824/0 (socket says 172.21.15.150:41824) 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5ade1d9640 1 -- 172.21.15.150:0/1325206650 learned_addr learned my addr 172.21.15.150:0/1325206650 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5add1d7640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ad80ac9e0 0x7f5ad80c13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5add9d8640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ac0003a40 0x7f5ad815a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5ade1d9640 1 -- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ad80ac9e0 msgr2=0x7f5ad80c13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5ade1d9640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5ad80ac9e0 0x7f5ad80c13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5ade1d9640 1 -- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ac0003a40 msgr2=0x7f5ad815a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5ade1d9640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ac0003a40 0x7f5ad815a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5ade1d9640 1 -- 172.21.15.150:0/1325206650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad81587b0 con 0x7f5ad81501b0 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5add9d8640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5ac0003a40 0x7f5ad815a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:03.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5ade1d9640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ad81501b0 0x7f5ad81505c0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8070cc0 tx=0x7f5ac800fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:03.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5ad67fc640 1 -- 172.21.15.150:0/1325206650 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8004d30 con 0x7f5ad81501b0 2024-09-17T05:21:03.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5ad67fc640 1 -- 172.21.15.150:0/1325206650 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ac800eab0 con 0x7f5ad81501b0 2024-09-17T05:21:03.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1325206650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad8158d30 con 0x7f5ad81501b0 2024-09-17T05:21:03.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5ad67fc640 1 -- 172.21.15.150:0/1325206650 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8005020 con 0x7f5ad81501b0 2024-09-17T05:21:03.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.464+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1325206650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad8158a50 con 0x7f5ad81501b0 2024-09-17T05:21:03.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.468+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1325206650 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5ad80c6be0 con 0x7f5ad81501b0 2024-09-17T05:21:03.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.468+0000 7f5ad67fc640 1 -- 172.21.15.150:0/1325206650 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5ac8019030 con 0x7f5ad81501b0 2024-09-17T05:21:03.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.468+0000 7f5ad67fc640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5ac4073230 0x7f5ac4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.468+0000 7f5ad67fc640 1 -- 172.21.15.150:0/1325206650 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f5ac8096050 con 0x7f5ad81501b0 2024-09-17T05:21:03.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.468+0000 7f5ad67fc640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] conn(0x7f5ac407fd50 0x7f5ac4082240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.468+0000 7f5ad67fc640 1 -- 172.21.15.150:0/1325206650 --> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ac8025e50 con 0x7f5ac407fd50 2024-09-17T05:21:03.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.468+0000 7f5ad67fc640 1 -- 172.21.15.150:0/1325206650 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f5ac8096440 con 0x7f5ad81501b0 2024-09-17T05:21:03.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.468+0000 7f5add9d8640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5ac4073230 0x7f5ac4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.468+0000 7f5add1d7640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] conn(0x7f5ac407fd50 0x7f5ac4082240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.468+0000 7f5add9d8640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5ac4073230 0x7f5ac4075740 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00ab000 tx=0x7f5ad00a7040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:03.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.468+0000 7f5add1d7640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] conn(0x7f5ac407fd50 0x7f5ac4082240 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:03.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.472+0000 7f5ad67fc640 1 -- 172.21.15.150:0/1325206650 <== osd.5 v2:172.21.15.176:6810/2266081607 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5ac8025e50 con 0x7f5ac407fd50 2024-09-17T05:21:03.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1325206650 --> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5ad80ac6a0 con 0x7f5ac407fd50 2024-09-17T05:21:03.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5ad67fc640 1 -- 172.21.15.150:0/1325206650 <== osd.5 v2:172.21.15.176:6810/2266081607 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5ad80ac6a0 con 0x7f5ac407fd50 2024-09-17T05:21:03.483 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-17T05:21:03.483 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:21:03.483 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-17T05:21:03.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] conn(0x7f5ac407fd50 msgr2=0x7f5ac4082240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5adfc3c640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:6810/2266081607,v1:172.21.15.176:6811/2266081607] conn(0x7f5ac407fd50 0x7f5ac4082240 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5ac4073230 msgr2=0x7f5ac4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5adfc3c640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5ac4073230 0x7f5ac4075740 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00ab000 tx=0x7f5ad00a7040 comp rx=0 tx=0).stop 2024-09-17T05:21:03.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1325206650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ad81501b0 msgr2=0x7f5ad81505c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5adfc3c640 1 --2- 172.21.15.150:0/1325206650 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5ad81501b0 0x7f5ad81505c0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8070cc0 tx=0x7f5ac800fe10 comp rx=0 tx=0).stop 2024-09-17T05:21:03.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5ade1d9640 1 -- 172.21.15.150:0/1325206650 reap_dead start 2024-09-17T05:21:03.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1325206650 shutdown_connections 2024-09-17T05:21:03.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1325206650 >> 172.21.15.150:0/1325206650 conn(0x7f5ad80c7fc0 msgr2=0x7f5ad80c41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:03.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1325206650 shutdown_connections 2024-09-17T05:21:03.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.484+0000 7f5adfc3c640 1 -- 172.21.15.150:0/1325206650 wait complete. 2024-09-17T05:21:03.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-17T05:21:03.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.660+0000 7f73595d2640 1 -- 172.21.15.150:0/3627146208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7354154150 msgr2=0x7f73540a8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.660+0000 7f73595d2640 1 --2- 172.21.15.150:0/3627146208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7354154150 0x7f73540a8340 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f734c009930 tx=0x7f734c01c850 comp rx=0 tx=0).stop 2024-09-17T05:21:03.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.660+0000 7f73595d2640 1 -- 172.21.15.150:0/3627146208 shutdown_connections 2024-09-17T05:21:03.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.660+0000 7f73595d2640 1 --2- 172.21.15.150:0/3627146208 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f735406ce50 0x7f735406d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.660+0000 7f73595d2640 1 --2- 172.21.15.150:0/3627146208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7354154150 0x7f73540a8340 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.660+0000 7f73595d2640 1 -- 172.21.15.150:0/3627146208 >> 172.21.15.150:0/3627146208 conn(0x7f73540aab60 msgr2=0x7f73540aaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:03.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.660+0000 7f73595d2640 1 -- 172.21.15.150:0/3627146208 shutdown_connections 2024-09-17T05:21:03.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.660+0000 7f73595d2640 1 -- 172.21.15.150:0/3627146208 wait complete. 2024-09-17T05:21:03.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f73595d2640 1 Processor -- start 2024-09-17T05:21:03.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f73595d2640 1 -- start start 2024-09-17T05:21:03.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f73595d2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f735406ce50 0x7f7354145c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f73595d2640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73540c7140 0x7f73541f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f73595d2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7354154150 0x7f735414f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f73595d2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f73540c7de0 con 0x7f735406ce50 2024-09-17T05:21:03.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f73595d2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f73540c7c60 con 0x7f7354154150 2024-09-17T05:21:03.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f73595d2640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7354163a40 con 0x7f73540c7140 2024-09-17T05:21:03.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f7352ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f735406ce50 0x7f7354145c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f7352ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f735406ce50 0x7f7354145c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41834/0 (socket says 172.21.15.150:41834) 2024-09-17T05:21:03.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f7352ffd640 1 -- 172.21.15.150:0/261803425 learned_addr learned my addr 172.21.15.150:0/261803425 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:03.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f73537fe640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7354154150 0x7f735414f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f73527fc640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73540c7140 0x7f73541f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f7352ffd640 1 -- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73540c7140 msgr2=0x7f73541f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f7352ffd640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73540c7140 0x7f73541f34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f7352ffd640 1 -- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7354154150 msgr2=0x7f735414f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f7352ffd640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7354154150 0x7f735414f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f7352ffd640 1 -- 172.21.15.150:0/261803425 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73541f3a00 con 0x7f735406ce50 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f7352ffd640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f735406ce50 0x7f7354145c60 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f734c01cd30 tx=0x7f734c0a65d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f734b7fe640 1 -- 172.21.15.150:0/261803425 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c0ac070 con 0x7f735406ce50 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f734b7fe640 1 -- 172.21.15.150:0/261803425 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f734c0054d0 con 0x7f735406ce50 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f73595d2640 1 -- 172.21.15.150:0/261803425 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73540c61d0 con 0x7f735406ce50 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f734b7fe640 1 -- 172.21.15.150:0/261803425 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c004290 con 0x7f735406ce50 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.664+0000 7f73595d2640 1 -- 172.21.15.150:0/261803425 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73540a9fb0 con 0x7f735406ce50 2024-09-17T05:21:03.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.668+0000 7f73595d2640 1 -- 172.21.15.150:0/261803425 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f735406d260 con 0x7f735406ce50 2024-09-17T05:21:03.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.668+0000 7f734b7fe640 1 -- 172.21.15.150:0/261803425 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f734c0af070 con 0x7f735406ce50 2024-09-17T05:21:03.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.668+0000 7f734b7fe640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7338073200 0x7f7338075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:03.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.668+0000 7f734b7fe640 1 -- 172.21.15.150:0/261803425 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f734c12f4e0 con 0x7f735406ce50 2024-09-17T05:21:03.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.668+0000 7f73527fc640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7338073200 0x7f7338075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:03.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.668+0000 7f73527fc640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7338073200 0x7f7338075710 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f73541606b0 tx=0x7f734400a390 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:03.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.672+0000 7f734b7fe640 1 -- 172.21.15.150:0/261803425 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f734c0164d0 con 0x7f735406ce50 2024-09-17T05:21:03.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.832+0000 7f73595d2640 1 -- 172.21.15.150:0/261803425 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f7354160700 con 0x7f7338073200 2024-09-17T05:21:03.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.832+0000 7f734b7fe640 1 -- 172.21.15.150:0/261803425 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7354160700 con 0x7f7338073200 2024-09-17T05:21:03.830 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 5 to deep-scrub 2024-09-17T05:21:03.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.836+0000 7f73595d2640 1 -- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7338073200 msgr2=0x7f7338075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.836+0000 7f73595d2640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7338073200 0x7f7338075710 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f73541606b0 tx=0x7f734400a390 comp rx=0 tx=0).stop 2024-09-17T05:21:03.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.836+0000 7f73595d2640 1 -- 172.21.15.150:0/261803425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f735406ce50 msgr2=0x7f7354145c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:03.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.836+0000 7f73595d2640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f735406ce50 0x7f7354145c60 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f734c01cd30 tx=0x7f734c0a65d0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.836+0000 7f73595d2640 1 -- 172.21.15.150:0/261803425 shutdown_connections 2024-09-17T05:21:03.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.836+0000 7f73595d2640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7338073200 0x7f7338075710 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.836+0000 7f73595d2640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7354154150 0x7f735414f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.836+0000 7f73595d2640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f73540c7140 0x7f73541f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.836+0000 7f73595d2640 1 --2- 172.21.15.150:0/261803425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f735406ce50 0x7f7354145c60 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:03.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.836+0000 7f73595d2640 1 -- 172.21.15.150:0/261803425 >> 172.21.15.150:0/261803425 conn(0x7f73540aab60 msgr2=0x7f73540c7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:03.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.836+0000 7f73595d2640 1 -- 172.21.15.150:0/261803425 shutdown_connections 2024-09-17T05:21:03.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:03.836+0000 7f73595d2640 1 -- 172.21.15.150:0/261803425 wait complete. 2024-09-17T05:21:03.853 INFO:tasks.ceph:Scrubbing osd.6 2024-09-17T05:21:03.853 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:21:04.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.004+0000 7f09200f1640 1 -- 172.21.15.150:0/2276504906 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0918070850 msgr2=0x7f09180c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.004+0000 7f09200f1640 1 --2- 172.21.15.150:0/2276504906 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0918070850 0x7f09180c6dd0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0908009870 tx=0x7f090801c700 comp rx=0 tx=0).stop 2024-09-17T05:21:04.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.004+0000 7f09200f1640 1 -- 172.21.15.150:0/2276504906 shutdown_connections 2024-09-17T05:21:04.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.004+0000 7f09200f1640 1 --2- 172.21.15.150:0/2276504906 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0918070850 0x7f09180c6dd0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.004+0000 7f09200f1640 1 --2- 172.21.15.150:0/2276504906 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09180c3ef0 0x7f09181564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.004+0000 7f09200f1640 1 -- 172.21.15.150:0/2276504906 >> 172.21.15.150:0/2276504906 conn(0x7f09180c7fc0 msgr2=0x7f09180c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:04.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.004+0000 7f09200f1640 1 -- 172.21.15.150:0/2276504906 shutdown_connections 2024-09-17T05:21:04.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.004+0000 7f09200f1640 1 -- 172.21.15.150:0/2276504906 wait complete. 2024-09-17T05:21:04.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.004+0000 7f09200f1640 1 Processor -- start 2024-09-17T05:21:04.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.004+0000 7f09200f1640 1 -- start start 2024-09-17T05:21:04.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f09200f1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0918070850 0x7f09180c0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f09200f1640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09180c3ef0 0x7f09180c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f09200f1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09180c7310 0x7f09180ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f09200f1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0918160970 con 0x7f0918070850 2024-09-17T05:21:04.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f09200f1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f09181607f0 con 0x7f09180c7310 2024-09-17T05:21:04.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f09200f1640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f0918160af0 con 0x7f09180c3ef0 2024-09-17T05:21:04.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f091de8d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0918070850 0x7f09180c0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f091de8d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0918070850 0x7f09180c0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41842/0 (socket says 172.21.15.150:41842) 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f091de8d640 1 -- 172.21.15.150:0/1415912219 learned_addr learned my addr 172.21.15.150:0/1415912219 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f091d68c640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09180c3ef0 0x7f09180c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f091e68e640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09180c7310 0x7f09180ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f091de8d640 1 -- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09180c3ef0 msgr2=0x7f09180c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f091de8d640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f09180c3ef0 0x7f09180c5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f091de8d640 1 -- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09180c7310 msgr2=0x7f09180ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f091de8d640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09180c7310 0x7f09180ac8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f091de8d640 1 -- 172.21.15.150:0/1415912219 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f091806a8d0 con 0x7f0918070850 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f091e68e640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09180c7310 0x7f09180ac8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f091de8d640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0918070850 0x7f09180c0ec0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f091400e7d0 tx=0x7f091400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f090e7fc640 1 -- 172.21.15.150:0/1415912219 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091400ccc0 con 0x7f0918070850 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f090e7fc640 1 -- 172.21.15.150:0/1415912219 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0914010430 con 0x7f0918070850 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f09200f1640 1 -- 172.21.15.150:0/1415912219 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09180c1400 con 0x7f0918070850 2024-09-17T05:21:04.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f090e7fc640 1 -- 172.21.15.150:0/1415912219 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914010880 con 0x7f0918070850 2024-09-17T05:21:04.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f09200f1640 1 -- 172.21.15.150:0/1415912219 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09180be920 con 0x7f0918070850 2024-09-17T05:21:04.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f09200f1640 1 -- 172.21.15.150:0/1415912219 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f09181f3c60 con 0x7f0918070850 2024-09-17T05:21:04.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f090e7fc640 1 -- 172.21.15.150:0/1415912219 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f091401f020 con 0x7f0918070850 2024-09-17T05:21:04.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.008+0000 7f090e7fc640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f08fc073200 0x7f08fc075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.012+0000 7f090e7fc640 1 -- 172.21.15.150:0/1415912219 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f0914096520 con 0x7f0918070850 2024-09-17T05:21:04.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.012+0000 7f090e7fc640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] conn(0x7f08fc07fca0 0x7f08fc082190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.012+0000 7f090e7fc640 1 -- 172.21.15.150:0/1415912219 --> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08fc082860 con 0x7f08fc07fca0 2024-09-17T05:21:04.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.012+0000 7f090e7fc640 1 -- 172.21.15.150:0/1415912219 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f0914096910 con 0x7f0918070850 2024-09-17T05:21:04.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.012+0000 7f091d68c640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f08fc073200 0x7f08fc075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.012+0000 7f091e68e640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] conn(0x7f08fc07fca0 0x7f08fc082190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.012+0000 7f091d68c640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f08fc073200 0x7f08fc075710 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f09080af000 tx=0x7f09080b2040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:04.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.012+0000 7f091e68e640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] conn(0x7f08fc07fca0 0x7f08fc082190 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:04.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.012+0000 7f090e7fc640 1 -- 172.21.15.150:0/1415912219 <== osd.6 v2:172.21.15.176:6802/4084217765 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08fc082860 con 0x7f08fc07fca0 2024-09-17T05:21:04.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f09200f1640 1 -- 172.21.15.150:0/1415912219 --> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f09180be710 con 0x7f08fc07fca0 2024-09-17T05:21:04.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f090e7fc640 1 -- 172.21.15.150:0/1415912219 <== osd.6 v2:172.21.15.176:6802/4084217765 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f09180be710 con 0x7f08fc07fca0 2024-09-17T05:21:04.025 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-17T05:21:04.025 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:21:04.025 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-17T05:21:04.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f09200f1640 1 -- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] conn(0x7f08fc07fca0 msgr2=0x7f08fc082190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f09200f1640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:6802/4084217765,v1:172.21.15.176:6803/4084217765] conn(0x7f08fc07fca0 0x7f08fc082190 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f09200f1640 1 -- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f08fc073200 msgr2=0x7f08fc075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f09200f1640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f08fc073200 0x7f08fc075710 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f09080af000 tx=0x7f09080b2040 comp rx=0 tx=0).stop 2024-09-17T05:21:04.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f09200f1640 1 -- 172.21.15.150:0/1415912219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0918070850 msgr2=0x7f09180c0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f09200f1640 1 --2- 172.21.15.150:0/1415912219 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0918070850 0x7f09180c0ec0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f091400e7d0 tx=0x7f091400ec90 comp rx=0 tx=0).stop 2024-09-17T05:21:04.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f091e68e640 1 -- 172.21.15.150:0/1415912219 reap_dead start 2024-09-17T05:21:04.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f09200f1640 1 -- 172.21.15.150:0/1415912219 shutdown_connections 2024-09-17T05:21:04.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f09200f1640 1 -- 172.21.15.150:0/1415912219 >> 172.21.15.150:0/1415912219 conn(0x7f09180c7fc0 msgr2=0x7f0918156c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:04.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f09200f1640 1 -- 172.21.15.150:0/1415912219 shutdown_connections 2024-09-17T05:21:04.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.028+0000 7f09200f1640 1 -- 172.21.15.150:0/1415912219 wait complete. 2024-09-17T05:21:04.044 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-17T05:21:04.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 -- 172.21.15.150:0/1000955721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f926c0c4950 msgr2=0x7f926c0aa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 --2- 172.21.15.150:0/1000955721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f926c0c4950 0x7f926c0aa1d0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f925c009900 tx=0x7f925c01c870 comp rx=0 tx=0).stop 2024-09-17T05:21:04.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 -- 172.21.15.150:0/1000955721 shutdown_connections 2024-09-17T05:21:04.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 --2- 172.21.15.150:0/1000955721 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f926c14cec0 0x7f926c06ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 --2- 172.21.15.150:0/1000955721 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f926c0c4950 0x7f926c0aa1d0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 -- 172.21.15.150:0/1000955721 >> 172.21.15.150:0/1000955721 conn(0x7f926c070b50 msgr2=0x7f926c070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:04.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 -- 172.21.15.150:0/1000955721 shutdown_connections 2024-09-17T05:21:04.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 -- 172.21.15.150:0/1000955721 wait complete. 2024-09-17T05:21:04.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 Processor -- start 2024-09-17T05:21:04.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 -- start start 2024-09-17T05:21:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f926c0aa7b0 0x7f926c14dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f926c0c4950 0x7f926c0c1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f926c14cec0 0x7f926c1f0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f926c06d510 con 0x7f926c0aa7b0 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f926c06d390 con 0x7f926c0c4950 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9272585640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f926c1607c0 con 0x7f926c14cec0 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f926bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f926c0aa7b0 0x7f926c14dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f926bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f926c0aa7b0 0x7f926c14dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41852/0 (socket says 172.21.15.150:41852) 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f926bfff640 1 -- 172.21.15.150:0/2541636117 learned_addr learned my addr 172.21.15.150:0/2541636117 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f9270b22640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f926c14cec0 0x7f926c1f0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f92637fe640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f926c0c4950 0x7f926c0c1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f926bfff640 1 -- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f926c14cec0 msgr2=0x7f926c1f0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f926bfff640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f926c14cec0 0x7f926c1f0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f926bfff640 1 -- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f926c0c4950 msgr2=0x7f926c0c1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f926bfff640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f926c0c4950 0x7f926c0c1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f926bfff640 1 -- 172.21.15.150:0/2541636117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c148a00 con 0x7f926c0aa7b0 2024-09-17T05:21:04.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.188+0000 7f926bfff640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f926c0aa7b0 0x7f926c14dd00 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f925c01cd50 tx=0x7f925c0a6c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:04.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.192+0000 7f92697fa640 1 -- 172.21.15.150:0/2541636117 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c0ac070 con 0x7f926c0aa7b0 2024-09-17T05:21:04.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.192+0000 7f92697fa640 1 -- 172.21.15.150:0/2541636117 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f925c004120 con 0x7f926c0aa7b0 2024-09-17T05:21:04.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.192+0000 7f9272585640 1 -- 172.21.15.150:0/2541636117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926c149000 con 0x7f926c0aa7b0 2024-09-17T05:21:04.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.192+0000 7f92697fa640 1 -- 172.21.15.150:0/2541636117 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c005040 con 0x7f926c0aa7b0 2024-09-17T05:21:04.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.192+0000 7f9272585640 1 -- 172.21.15.150:0/2541636117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f926c0acb10 con 0x7f926c0aa7b0 2024-09-17T05:21:04.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.192+0000 7f9272585640 1 -- 172.21.15.150:0/2541636117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f926c148e20 con 0x7f926c0aa7b0 2024-09-17T05:21:04.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.192+0000 7f92697fa640 1 -- 172.21.15.150:0/2541636117 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f925c0af070 con 0x7f926c0aa7b0 2024-09-17T05:21:04.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.192+0000 7f92697fa640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f924c073200 0x7f924c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.192+0000 7f92697fa640 1 -- 172.21.15.150:0/2541636117 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f925c12f1a0 con 0x7f926c0aa7b0 2024-09-17T05:21:04.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.192+0000 7f92637fe640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f924c073200 0x7f924c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.192+0000 7f92637fe640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f924c073200 0x7f924c075710 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9254006fd0 tx=0x7f9254008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:04.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.196+0000 7f92697fa640 1 -- 172.21.15.150:0/2541636117 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f926c148e20 con 0x7f926c0aa7b0 2024-09-17T05:21:04.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.356+0000 7f9272585640 1 -- 172.21.15.150:0/2541636117 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f926c1efcf0 con 0x7f924c073200 2024-09-17T05:21:04.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.356+0000 7f92697fa640 1 -- 172.21.15.150:0/2541636117 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f925400bda0 con 0x7f924c073200 2024-09-17T05:21:04.355 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 6 to deep-scrub 2024-09-17T05:21:04.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.360+0000 7f9272585640 1 -- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f924c073200 msgr2=0x7f924c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.360+0000 7f9272585640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f924c073200 0x7f924c075710 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9254006fd0 tx=0x7f9254008040 comp rx=0 tx=0).stop 2024-09-17T05:21:04.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.360+0000 7f9272585640 1 -- 172.21.15.150:0/2541636117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f926c0aa7b0 msgr2=0x7f926c14dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.360+0000 7f9272585640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f926c0aa7b0 0x7f926c14dd00 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f925c01cd50 tx=0x7f925c0a6c20 comp rx=0 tx=0).stop 2024-09-17T05:21:04.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.360+0000 7f9272585640 1 -- 172.21.15.150:0/2541636117 shutdown_connections 2024-09-17T05:21:04.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.360+0000 7f9272585640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f924c073200 0x7f924c075710 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.360+0000 7f9272585640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f926c0c4950 0x7f926c0c1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.360+0000 7f9272585640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f926c14cec0 0x7f926c1f0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.360+0000 7f9272585640 1 --2- 172.21.15.150:0/2541636117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f926c0aa7b0 0x7f926c14dd00 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.360+0000 7f9272585640 1 -- 172.21.15.150:0/2541636117 >> 172.21.15.150:0/2541636117 conn(0x7f926c070b50 msgr2=0x7f926c070f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:04.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.360+0000 7f9272585640 1 -- 172.21.15.150:0/2541636117 shutdown_connections 2024-09-17T05:21:04.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.360+0000 7f9272585640 1 -- 172.21.15.150:0/2541636117 wait complete. 2024-09-17T05:21:04.377 INFO:tasks.ceph:Scrubbing osd.7 2024-09-17T05:21:04.377 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-17T05:21:04.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/2369322354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3cc150e60 msgr2=0x7ff3cc0c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 --2- 172.21.15.150:0/2369322354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3cc150e60 0x7ff3cc0c55f0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009920 tx=0x7ff3bc01c7a0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/2369322354 shutdown_connections 2024-09-17T05:21:04.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 --2- 172.21.15.150:0/2369322354 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3cc0c5b30 0x7ff3cc0c38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 --2- 172.21.15.150:0/2369322354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3cc150e60 0x7ff3cc0c55f0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/2369322354 >> 172.21.15.150:0/2369322354 conn(0x7ff3cc0c8eb0 msgr2=0x7ff3cc0c92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:04.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/2369322354 shutdown_connections 2024-09-17T05:21:04.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/2369322354 wait complete. 2024-09-17T05:21:04.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 Processor -- start 2024-09-17T05:21:04.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 -- start start 2024-09-17T05:21:04.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3cc0c5b30 0x7ff3cc14f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3cc0c89f0 0x7ff3cc0bf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3cc150e60 0x7ff3cc1ed880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff3cc0a86f0 con 0x7ff3cc0c5b30 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff3cc0c4e30 con 0x7ff3cc150e60 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d3e0e640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff3cc0a8870 con 0x7ff3cc0c89f0 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d1baa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3cc0c5b30 0x7ff3cc14f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d1baa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3cc0c5b30 0x7ff3cc14f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41860/0 (socket says 172.21.15.150:41860) 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d1baa640 1 -- 172.21.15.150:0/213868043 learned_addr learned my addr 172.21.15.150:0/213868043 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d13a9640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3cc0c89f0 0x7ff3cc0bf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d1baa640 1 -- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3cc0c89f0 msgr2=0x7ff3cc0bf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d1baa640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff3cc0c89f0 0x7ff3cc0bf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d23ab640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3cc150e60 0x7ff3cc1ed880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d1baa640 1 -- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3cc150e60 msgr2=0x7ff3cc1ed880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d1baa640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff3cc150e60 0x7ff3cc1ed880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d1baa640 1 -- 172.21.15.150:0/213868043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3cc0ab030 con 0x7ff3cc0c5b30 2024-09-17T05:21:04.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3d1baa640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3cc0c5b30 0x7ff3cc14f4e0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800b3e0 tx=0x7ff3c800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:04.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.528+0000 7ff3c27fc640 1 -- 172.21.15.150:0/213868043 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8015020 con 0x7ff3cc0c5b30 2024-09-17T05:21:04.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3c27fc640 1 -- 172.21.15.150:0/213868043 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3c8011920 con 0x7ff3cc0c5b30 2024-09-17T05:21:04.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/213868043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3cc0ab310 con 0x7ff3cc0c5b30 2024-09-17T05:21:04.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3c27fc640 1 -- 172.21.15.150:0/213868043 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8011db0 con 0x7ff3cc0c5b30 2024-09-17T05:21:04.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/213868043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3cc0ab8d0 con 0x7ff3cc0c5b30 2024-09-17T05:21:04.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/213868043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff3cc0c38b0 con 0x7ff3cc0c5b30 2024-09-17T05:21:04.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3c27fc640 1 -- 172.21.15.150:0/213868043 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff3c8018020 con 0x7ff3cc0c5b30 2024-09-17T05:21:04.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3c27fc640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff3b0073200 0x7ff3b0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3c27fc640 1 -- 172.21.15.150:0/213868043 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7ff3c8095570 con 0x7ff3cc0c5b30 2024-09-17T05:21:04.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3c27fc640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] conn(0x7ff3b007fd30 0x7ff3b0082220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3c27fc640 1 -- 172.21.15.150:0/213868043 --> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3c800de50 con 0x7ff3b007fd30 2024-09-17T05:21:04.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3c27fc640 1 -- 172.21.15.150:0/213868043 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7ff3c8095960 con 0x7ff3cc0c5b30 2024-09-17T05:21:04.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3d13a9640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff3b0073200 0x7ff3b0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3d23ab640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] conn(0x7ff3b007fd30 0x7ff3b0082220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3d13a9640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff3b0073200 0x7ff3b0075710 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0af000 tx=0x7ff3bc01afe0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:04.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.532+0000 7ff3d23ab640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] conn(0x7ff3b007fd30 0x7ff3b0082220 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:04.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.536+0000 7ff3c27fc640 1 -- 172.21.15.150:0/213868043 <== osd.7 v2:172.21.15.176:6818/621582613 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3c800de50 con 0x7ff3b007fd30 2024-09-17T05:21:04.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/213868043 --> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff3cc071770 con 0x7ff3b007fd30 2024-09-17T05:21:04.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3c27fc640 1 -- 172.21.15.150:0/213868043 <== osd.7 v2:172.21.15.176:6818/621582613 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff3cc071770 con 0x7ff3b007fd30 2024-09-17T05:21:04.547 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-09-17T05:21:04.547 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T05:21:04.547 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-09-17T05:21:04.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] conn(0x7ff3b007fd30 msgr2=0x7ff3b0082220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3d3e0e640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:6818/621582613,v1:172.21.15.176:6819/621582613] conn(0x7ff3b007fd30 0x7ff3b0082220 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff3b0073200 msgr2=0x7ff3b0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3d3e0e640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff3b0073200 0x7ff3b0075710 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0af000 tx=0x7ff3bc01afe0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/213868043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3cc0c5b30 msgr2=0x7ff3cc14f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3d3e0e640 1 --2- 172.21.15.150:0/213868043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3cc0c5b30 0x7ff3cc14f4e0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800b3e0 tx=0x7ff3c800b8a0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3d23ab640 1 -- 172.21.15.150:0/213868043 reap_dead start 2024-09-17T05:21:04.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/213868043 shutdown_connections 2024-09-17T05:21:04.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/213868043 >> 172.21.15.150:0/213868043 conn(0x7ff3cc0c8eb0 msgr2=0x7ff3cc070700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:04.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/213868043 shutdown_connections 2024-09-17T05:21:04.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.548+0000 7ff3d3e0e640 1 -- 172.21.15.150:0/213868043 wait complete. 2024-09-17T05:21:04.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-17T05:21:04.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.716+0000 7fdff4401640 1 -- 172.21.15.150:0/1737616370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfec158170 msgr2=0x7fdfec158580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.716+0000 7fdff4401640 1 --2- 172.21.15.150:0/1737616370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfec158170 0x7fdfec158580 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009900 tx=0x7fdfdc01c880 comp rx=0 tx=0).stop 2024-09-17T05:21:04.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.716+0000 7fdff4401640 1 -- 172.21.15.150:0/1737616370 shutdown_connections 2024-09-17T05:21:04.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.716+0000 7fdff4401640 1 --2- 172.21.15.150:0/1737616370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfec158170 0x7fdfec158580 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.716+0000 7fdff4401640 1 --2- 172.21.15.150:0/1737616370 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfec0c3b40 0x7fdfec157c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.716+0000 7fdff4401640 1 -- 172.21.15.150:0/1737616370 >> 172.21.15.150:0/1737616370 conn(0x7fdfec0c4a00 msgr2=0x7fdfec0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:04.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.716+0000 7fdff4401640 1 -- 172.21.15.150:0/1737616370 shutdown_connections 2024-09-17T05:21:04.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.716+0000 7fdff4401640 1 -- 172.21.15.150:0/1737616370 wait complete. 2024-09-17T05:21:04.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.716+0000 7fdff4401640 1 Processor -- start 2024-09-17T05:21:04.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.716+0000 7fdff4401640 1 -- start start 2024-09-17T05:21:04.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff4401640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfec0c3b40 0x7fdfec147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff4401640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfec151d00 0x7fdfec14e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff4401640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfec158170 0x7fdfec14d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff4401640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdfec0c40d0 con 0x7fdfec158170 2024-09-17T05:21:04.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff4401640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdfec0c3f50 con 0x7fdfec151d00 2024-09-17T05:21:04.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff4401640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdfec161830 con 0x7fdfec0c3b40 2024-09-17T05:21:04.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff299e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfec158170 0x7fdfec14d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff299e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfec158170 0x7fdfec14d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41870/0 (socket says 172.21.15.150:41870) 2024-09-17T05:21:04.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff299e640 1 -- 172.21.15.150:0/3423096302 learned_addr learned my addr 172.21.15.150:0/3423096302 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:04.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff199c640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfec151d00 0x7fdfec14e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff219d640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfec0c3b40 0x7fdfec147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff299e640 1 -- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfec0c3b40 msgr2=0x7fdfec147a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff299e640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfec0c3b40 0x7fdfec147a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff299e640 1 -- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfec151d00 msgr2=0x7fdfec14e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff299e640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfec151d00 0x7fdfec14e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff299e640 1 -- 172.21.15.150:0/3423096302 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfec147fd0 con 0x7fdfec158170 2024-09-17T05:21:04.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff299e640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfec158170 0x7fdfec14d640 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc0b0040 tx=0x7fdfdc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:04.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdfeaffd640 1 -- 172.21.15.150:0/3423096302 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc004230 con 0x7fdfec158170 2024-09-17T05:21:04.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdfeaffd640 1 -- 172.21.15.150:0/3423096302 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdfdc003600 con 0x7fdfec158170 2024-09-17T05:21:04.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff4401640 1 -- 172.21.15.150:0/3423096302 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfec1ede00 con 0x7fdfec158170 2024-09-17T05:21:04.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdfeaffd640 1 -- 172.21.15.150:0/3423096302 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc003a50 con 0x7fdfec158170 2024-09-17T05:21:04.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff4401640 1 -- 172.21.15.150:0/3423096302 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfec1ee3c0 con 0x7fdfec158170 2024-09-17T05:21:04.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdff4401640 1 -- 172.21.15.150:0/3423096302 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfec157c30 con 0x7fdfec158170 2024-09-17T05:21:04.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdfeaffd640 1 -- 172.21.15.150:0/3423096302 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fdfdc0c0030 con 0x7fdfec158170 2024-09-17T05:21:04.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdfeaffd640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfd8073230 0x7fdfd8075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:04.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.720+0000 7fdfeaffd640 1 -- 172.21.15.150:0/3423096302 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7fdfdc12fb00 con 0x7fdfec158170 2024-09-17T05:21:04.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.724+0000 7fdff219d640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfd8073230 0x7fdfd8075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:04.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.724+0000 7fdff219d640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfd8073230 0x7fdfd8075740 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009f20 tx=0x7fdfe40094d0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:04.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.728+0000 7fdfeaffd640 1 -- 172.21.15.150:0/3423096302 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdfdc0c7020 con 0x7fdfec158170 2024-09-17T05:21:04.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.884+0000 7fdff4401640 1 -- 172.21.15.150:0/3423096302 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fdfec0a7730 con 0x7fdfd8073230 2024-09-17T05:21:04.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.884+0000 7fdfeaffd640 1 -- 172.21.15.150:0/3423096302 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdfec0a7730 con 0x7fdfd8073230 2024-09-17T05:21:04.884 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 7 to deep-scrub 2024-09-17T05:21:04.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.888+0000 7fdff4401640 1 -- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfd8073230 msgr2=0x7fdfd8075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.888+0000 7fdff4401640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfd8073230 0x7fdfd8075740 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009f20 tx=0x7fdfe40094d0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.888+0000 7fdff4401640 1 -- 172.21.15.150:0/3423096302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfec158170 msgr2=0x7fdfec14d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:04.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.888+0000 7fdff4401640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfec158170 0x7fdfec14d640 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc0b0040 tx=0x7fdfdc004040 comp rx=0 tx=0).stop 2024-09-17T05:21:04.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.888+0000 7fdff4401640 1 -- 172.21.15.150:0/3423096302 shutdown_connections 2024-09-17T05:21:04.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.888+0000 7fdff4401640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdfd8073230 0x7fdfd8075740 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.888+0000 7fdff4401640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdfec151d00 0x7fdfec14e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.888+0000 7fdff4401640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfec158170 0x7fdfec14d640 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.888+0000 7fdff4401640 1 --2- 172.21.15.150:0/3423096302 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdfec0c3b40 0x7fdfec147a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:04.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.888+0000 7fdff4401640 1 -- 172.21.15.150:0/3423096302 >> 172.21.15.150:0/3423096302 conn(0x7fdfec0c4a00 msgr2=0x7fdfec14f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:04.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.888+0000 7fdff4401640 1 -- 172.21.15.150:0/3423096302 shutdown_connections 2024-09-17T05:21:04.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:04.888+0000 7fdff4401640 1 -- 172.21.15.150:0/3423096302 wait complete. 2024-09-17T05:21:04.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:21:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.052+0000 7f66699fc640 1 -- 172.21.15.150:0/2812937694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6650003a40 msgr2=0x7f6650003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.052+0000 7f66699fc640 1 --2- 172.21.15.150:0/2812937694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6650003a40 0x7f6650003eb0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f66540016f0 tx=0x7f665401c7b0 comp rx=0 tx=0).stop 2024-09-17T05:21:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.052+0000 7f66637fe640 1 -- 172.21.15.150:0/2812937694 reap_dead start 2024-09-17T05:21:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.052+0000 7f66699fc640 1 -- 172.21.15.150:0/2812937694 shutdown_connections 2024-09-17T05:21:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.052+0000 7f66699fc640 1 -- 172.21.15.150:0/2812937694 >> 172.21.15.150:0/2812937694 conn(0x7f66640c4a00 msgr2=0x7f66640c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.052+0000 7f66699fc640 1 -- 172.21.15.150:0/2812937694 shutdown_connections 2024-09-17T05:21:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.052+0000 7f66699fc640 1 -- 172.21.15.150:0/2812937694 wait complete. 2024-09-17T05:21:05.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.052+0000 7f66699fc640 1 Processor -- start 2024-09-17T05:21:05.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.052+0000 7f66699fc640 1 -- start start 2024-09-17T05:21:05.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f66699fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6650003a40 0x7f66641f0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:05.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f66699fc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f666414d8c0 0x7f666414a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f66699fc640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f666414a990 0x7f666414f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f66699fc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f666414d4b0 con 0x7f6650003a40 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f66699fc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f66641576d0 con 0x7f666414d8c0 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f66699fc640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f666414d630 con 0x7f666414a990 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6662ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6650003a40 0x7f66641f0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6662ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6650003a40 0x7f66641f0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:41880/0 (socket says 172.21.15.150:41880) 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6662ffd640 1 -- 172.21.15.150:0/167961441 learned_addr learned my addr 172.21.15.150:0/167961441 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f66637fe640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f666414a990 0x7f666414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f665a7fc640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f666414d8c0 0x7f666414a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6662ffd640 1 -- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f666414a990 msgr2=0x7f666414f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6662ffd640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f666414a990 0x7f666414f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6662ffd640 1 -- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f666414d8c0 msgr2=0x7f666414a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6662ffd640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f666414d8c0 0x7f666414a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:05.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6662ffd640 1 -- 172.21.15.150:0/167961441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6664158aa0 con 0x7f6650003a40 2024-09-17T05:21:05.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f66637fe640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f666414a990 0x7f666414f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:05.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f665a7fc640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f666414d8c0 0x7f666414a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:05.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6662ffd640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6650003a40 0x7f66641f0520 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f6654000c10 tx=0x7f6654003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:05.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6660ff9640 1 -- 172.21.15.150:0/167961441 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66540bb070 con 0x7f6650003a40 2024-09-17T05:21:05.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6660ff9640 1 -- 172.21.15.150:0/167961441 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6654006500 con 0x7f6650003a40 2024-09-17T05:21:05.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f66699fc640 1 -- 172.21.15.150:0/167961441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f666414b9b0 con 0x7f6650003a40 2024-09-17T05:21:05.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6660ff9640 1 -- 172.21.15.150:0/167961441 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66540ae070 con 0x7f6650003a40 2024-09-17T05:21:05.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f66699fc640 1 -- 172.21.15.150:0/167961441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6664150780 con 0x7f6650003a40 2024-09-17T05:21:05.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f66699fc640 1 -- 172.21.15.150:0/167961441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f666414dcd0 con 0x7f6650003a40 2024-09-17T05:21:05.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6660ff9640 1 -- 172.21.15.150:0/167961441 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f66540be070 con 0x7f6650003a40 2024-09-17T05:21:05.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.056+0000 7f6660ff9640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6644073200 0x7f6644075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:05.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.060+0000 7f6660ff9640 1 -- 172.21.15.150:0/167961441 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f665412ed60 con 0x7f6650003a40 2024-09-17T05:21:05.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.060+0000 7f665a7fc640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6644073200 0x7f6644075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:05.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.060+0000 7f665a7fc640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6644073200 0x7f6644075710 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f664c001420 tx=0x7f664c001dd0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:05.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.060+0000 7f6660ff9640 1 -- 172.21.15.150:0/167961441 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6654016510 con 0x7f6650003a40 2024-09-17T05:21:05.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.216+0000 7f66699fc640 1 -- 172.21.15.150:0/167961441 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f666414b4a0 con 0x7f6644073200 2024-09-17T05:21:05.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.240+0000 7f6660ff9640 1 -- 172.21.15.150:0/167961441 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493636 (secure 0 0 0) 0x7f664c00e840 con 0x7f6644073200 2024-09-17T05:21:05.240 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:21:05.244 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T05:21:05.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.244+0000 7f66699fc640 1 -- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6644073200 msgr2=0x7f6644075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:05.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.244+0000 7f66699fc640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6644073200 0x7f6644075710 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f664c001420 tx=0x7f664c001dd0 comp rx=0 tx=0).stop 2024-09-17T05:21:05.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.244+0000 7f66699fc640 1 -- 172.21.15.150:0/167961441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6650003a40 msgr2=0x7f66641f0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:05.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.244+0000 7f66699fc640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6650003a40 0x7f66641f0520 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f6654000c10 tx=0x7f6654003c10 comp rx=0 tx=0).stop 2024-09-17T05:21:05.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.244+0000 7f66699fc640 1 -- 172.21.15.150:0/167961441 shutdown_connections 2024-09-17T05:21:05.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.244+0000 7f66699fc640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f6644073200 0x7f6644075710 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:05.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.244+0000 7f66699fc640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f666414d8c0 0x7f666414a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:05.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.244+0000 7f66699fc640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f666414a990 0x7f666414f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:05.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.244+0000 7f66699fc640 1 --2- 172.21.15.150:0/167961441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6650003a40 0x7f66641f0520 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:05.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.244+0000 7f66699fc640 1 -- 172.21.15.150:0/167961441 >> 172.21.15.150:0/167961441 conn(0x7f66640c4a00 msgr2=0x7f66640c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:05.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.244+0000 7f66699fc640 1 -- 172.21.15.150:0/167961441 shutdown_connections 2024-09-17T05:21:05.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:05.244+0000 7f66699fc640 1 -- 172.21.15.150:0/167961441 wait complete. 2024-09-17T05:21:05.264 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":1013,"stamp":"2024-09-17T05:21:03.885843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37864050,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3107,"num_read_kb":438866,"num_write":2440075,"num_write_kb":36007256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":458355,"ondisk_log_size":458355,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5828384,"kb_used_data":80928,"kb_used_omap":12,"kb_used_meta":5747315,"kb_avail":749146336,"statfs":{"total":773094113280,"available":767125848064,"internally_reserved":0,"allocated":82870272,"data_stored":79467812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":5885251122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4153,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007203"},"pg_stats":[{"pgid":"5.18","version":"68'3479","reported_seq":3591,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.418967+0000","last_change":"2024-09-17T04:49:41.025271+0000","last_active":"2024-09-17T05:20:44.418967+0000","last_peered":"2024-09-17T05:20:44.418967+0000","last_clean":"2024-09-17T05:20:44.418967+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T05:20:44.418967+0000","last_undegraded":"2024-09-17T05:20:44.418967+0000","last_fullsized":"2024-09-17T05:20:44.418967+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3479,"log_dups_size":0,"ondisk_log_size":3479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:39.715602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":12,"num_write":18686,"num_write_kb":287786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"65'1001","reported_seq":1175,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340557+0000","last_change":"2024-09-17T04:49:30.879796+0000","last_active":"2024-09-17T05:20:36.340557+0000","last_peered":"2024-09-17T05:20:36.340557+0000","last_clean":"2024-09-17T05:20:36.340557+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T05:20:36.340557+0000","last_undegraded":"2024-09-17T05:20:36.340557+0000","last_fullsized":"2024-09-17T05:20:36.340557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1001,"log_dups_size":0,"ondisk_log_size":1001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":20774,"num_write":978,"num_write_kb":4679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'4064","reported_seq":4184,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.351711+0000","last_change":"2024-09-17T04:49:31.892213+0000","last_active":"2024-09-17T05:20:36.351711+0000","last_peered":"2024-09-17T05:20:36.351711+0000","last_clean":"2024-09-17T05:20:36.351711+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T05:20:36.351711+0000","last_undegraded":"2024-09-17T05:20:36.351711+0000","last_fullsized":"2024-09-17T05:20:36.351711+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22217,"num_write_kb":281701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"68'3","reported_seq":114,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402753+0000","last_change":"2024-09-17T04:49:40.013210+0000","last_active":"2024-09-17T05:20:43.402753+0000","last_peered":"2024-09-17T05:20:43.402753+0000","last_clean":"2024-09-17T05:20:43.402753+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T05:20:43.402753+0000","last_undegraded":"2024-09-17T05:20:43.402753+0000","last_fullsized":"2024-09-17T05:20:43.402753+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:38.417840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"68'3721","reported_seq":3833,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.413711+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T05:20:43.413711+0000","last_peered":"2024-09-17T05:20:43.413711+0000","last_clean":"2024-09-17T05:20:43.413711+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T05:20:43.413711+0000","last_undegraded":"2024-09-17T05:20:43.413711+0000","last_fullsized":"2024-09-17T05:20:43.413711+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3721,"log_dups_size":0,"ondisk_log_size":3721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":22,"num_write":19557,"num_write_kb":316044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"67'26","reported_seq":148,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352348+0000","last_change":"2024-09-17T04:49:30.901160+0000","last_active":"2024-09-17T05:20:36.352348+0000","last_peered":"2024-09-17T05:20:36.352348+0000","last_clean":"2024-09-17T05:20:36.352348+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T05:20:36.352348+0000","last_undegraded":"2024-09-17T05:20:36.352348+0000","last_fullsized":"2024-09-17T05:20:36.352348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:46.625634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4998","reported_seq":5118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.353392+0000","last_change":"2024-09-17T04:49:31.886787+0000","last_active":"2024-09-17T05:20:36.353392+0000","last_peered":"2024-09-17T05:20:36.353392+0000","last_clean":"2024-09-17T05:20:36.353392+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T05:20:36.353392+0000","last_undegraded":"2024-09-17T05:20:36.353392+0000","last_fullsized":"2024-09-17T05:20:36.353392+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:52.561504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27336,"num_write_kb":388205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'2","reported_seq":113,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362495+0000","last_change":"2024-09-17T04:49:40.004215+0000","last_active":"2024-09-17T05:20:36.362495+0000","last_peered":"2024-09-17T05:20:36.362495+0000","last_clean":"2024-09-17T05:20:36.362495+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T05:20:36.362495+0000","last_undegraded":"2024-09-17T05:20:36.362495+0000","last_fullsized":"2024-09-17T05:20:36.362495+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:51.705020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"68'2765","reported_seq":2876,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.349402+0000","last_change":"2024-09-17T04:49:41.067317+0000","last_active":"2024-09-17T05:20:43.349402+0000","last_peered":"2024-09-17T05:20:43.349402+0000","last_clean":"2024-09-17T05:20:43.349402+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T05:20:43.349402+0000","last_undegraded":"2024-09-17T05:20:43.349402+0000","last_fullsized":"2024-09-17T05:20:43.349402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:37.637477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":14571,"num_write_kb":258996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"68'46","reported_seq":207,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558188+0000","last_change":"2024-09-17T04:49:30.873600+0000","last_active":"2024-09-17T05:20:36.558188+0000","last_peered":"2024-09-17T05:20:36.558188+0000","last_clean":"2024-09-17T05:20:36.558188+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T05:20:36.558188+0000","last_undegraded":"2024-09-17T05:20:36.558188+0000","last_fullsized":"2024-09-17T05:20:36.558188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"65'3140","reported_seq":3259,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.346435+0000","last_change":"2024-09-17T04:49:31.893185+0000","last_active":"2024-09-17T05:20:36.346435+0000","last_peered":"2024-09-17T05:20:36.346435+0000","last_clean":"2024-09-17T05:20:36.346435+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T05:20:36.346435+0000","last_undegraded":"2024-09-17T05:20:36.346435+0000","last_fullsized":"2024-09-17T05:20:36.346435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3140,"log_dups_size":0,"ondisk_log_size":3140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:12.561234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16962,"num_write_kb":252282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"68'751","reported_seq":888,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.388359+0000","last_change":"2024-09-17T04:49:40.019372+0000","last_active":"2024-09-17T05:20:43.388359+0000","last_peered":"2024-09-17T05:20:43.388359+0000","last_clean":"2024-09-17T05:20:43.388359+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T05:20:43.388359+0000","last_undegraded":"2024-09-17T05:20:43.388359+0000","last_fullsized":"2024-09-17T05:20:43.388359+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":75812,"num_write":739,"num_write_kb":4324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"68'2700","reported_seq":2810,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.180944+0000","last_change":"2024-09-17T04:49:41.043531+0000","last_active":"2024-09-17T05:20:43.180944+0000","last_peered":"2024-09-17T05:20:43.180944+0000","last_clean":"2024-09-17T05:20:43.180944+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T05:20:43.180944+0000","last_undegraded":"2024-09-17T05:20:43.180944+0000","last_fullsized":"2024-09-17T05:20:43.180944+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2700,"log_dups_size":0,"ondisk_log_size":2700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":5,"num_write":13438,"num_write_kb":225755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"65'5","reported_seq":166,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559425+0000","last_change":"2024-09-17T04:49:30.874963+0000","last_active":"2024-09-17T05:20:36.559425+0000","last_peered":"2024-09-17T05:20:36.559425+0000","last_clean":"2024-09-17T05:20:36.559425+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T05:20:36.559425+0000","last_undegraded":"2024-09-17T05:20:36.559425+0000","last_fullsized":"2024-09-17T05:20:36.559425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:47:38.844910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"65'4481","reported_seq":4598,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.373776+0000","last_change":"2024-09-17T04:49:31.906566+0000","last_active":"2024-09-17T05:20:36.373776+0000","last_peered":"2024-09-17T05:20:36.373776+0000","last_clean":"2024-09-17T05:20:36.373776+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T05:20:36.373776+0000","last_undegraded":"2024-09-17T05:20:36.373776+0000","last_fullsized":"2024-09-17T05:20:36.373776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:09.311536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24963,"num_write_kb":346034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"68'143","reported_seq":256,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:51.881072+0000","last_change":"2024-09-17T04:49:40.005376+0000","last_active":"2024-09-17T05:20:51.881072+0000","last_peered":"2024-09-17T05:20:51.881072+0000","last_clean":"2024-09-17T05:20:51.881072+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T05:20:51.881072+0000","last_undegraded":"2024-09-17T05:20:51.881072+0000","last_fullsized":"2024-09-17T05:20:51.881072+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:16.854365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1447387,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":383,"num_write":117,"num_write_kb":1479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"68'2942","reported_seq":3054,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.409983+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T05:20:44.409983+0000","last_peered":"2024-09-17T05:20:44.409983+0000","last_clean":"2024-09-17T05:20:44.409983+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T05:20:44.409983+0000","last_undegraded":"2024-09-17T05:20:44.409983+0000","last_fullsized":"2024-09-17T05:20:44.409983+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":16254,"num_write_kb":278162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"65'3968","reported_seq":4087,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362298+0000","last_change":"2024-09-17T04:49:31.895653+0000","last_active":"2024-09-17T05:20:36.362298+0000","last_peered":"2024-09-17T05:20:36.362298+0000","last_clean":"2024-09-17T05:20:36.362298+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T05:20:36.362298+0000","last_undegraded":"2024-09-17T05:20:36.362298+0000","last_fullsized":"2024-09-17T05:20:36.362298+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3968,"log_dups_size":0,"ondisk_log_size":3968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:29.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21939,"num_write_kb":297018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"68'1467","reported_seq":1637,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.367218+0000","last_change":"2024-09-17T04:49:30.881389+0000","last_active":"2024-09-17T05:20:44.367218+0000","last_peered":"2024-09-17T05:20:44.367218+0000","last_clean":"2024-09-17T05:20:44.367218+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T05:20:44.367218+0000","last_undegraded":"2024-09-17T05:20:44.367218+0000","last_fullsized":"2024-09-17T05:20:44.367218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":1467,"log_dups_size":0,"ondisk_log_size":1467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:54.642235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":43333,"num_write":1444,"num_write_kb":4521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'6","reported_seq":138,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.115470+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T05:20:43.115470+0000","last_peered":"2024-09-17T05:20:43.115470+0000","last_clean":"2024-09-17T05:20:43.115470+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T05:20:43.115470+0000","last_undegraded":"2024-09-17T05:20:43.115470+0000","last_fullsized":"2024-09-17T05:20:43.115470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:28.365399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'3236","reported_seq":3349,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.400322+0000","last_change":"2024-09-17T04:49:41.038415+0000","last_active":"2024-09-17T05:20:44.400322+0000","last_peered":"2024-09-17T05:20:44.400322+0000","last_clean":"2024-09-17T05:20:44.400322+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T05:20:44.400322+0000","last_undegraded":"2024-09-17T05:20:44.400322+0000","last_fullsized":"2024-09-17T05:20:44.400322+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3236,"log_dups_size":0,"ondisk_log_size":3236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:17.457429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":17542,"num_write_kb":271055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"65'3303","reported_seq":3421,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.358141+0000","last_change":"2024-09-17T04:49:31.920272+0000","last_active":"2024-09-17T05:20:36.358141+0000","last_peered":"2024-09-17T05:20:36.358141+0000","last_clean":"2024-09-17T05:20:36.358141+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T05:20:36.358141+0000","last_undegraded":"2024-09-17T05:20:36.358141+0000","last_fullsized":"2024-09-17T05:20:36.358141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3303,"log_dups_size":0,"ondisk_log_size":3303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17463,"num_write_kb":238014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"65'3","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.344536+0000","last_change":"2024-09-17T04:49:30.896423+0000","last_active":"2024-09-17T05:20:36.344536+0000","last_peered":"2024-09-17T05:20:36.344536+0000","last_clean":"2024-09-17T05:20:36.344536+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T05:20:36.344536+0000","last_undegraded":"2024-09-17T05:20:36.344536+0000","last_fullsized":"2024-09-17T05:20:36.344536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:51.270360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"68'5","reported_seq":117,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.405924+0000","last_change":"2024-09-17T04:49:40.010923+0000","last_active":"2024-09-17T05:20:43.405924+0000","last_peered":"2024-09-17T05:20:43.405924+0000","last_clean":"2024-09-17T05:20:43.405924+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T05:20:43.405924+0000","last_undegraded":"2024-09-17T05:20:43.405924+0000","last_fullsized":"2024-09-17T05:20:43.405924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:17.424265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"68'2691","reported_seq":2802,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.414897+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T05:20:43.414897+0000","last_peered":"2024-09-17T05:20:43.414897+0000","last_clean":"2024-09-17T05:20:43.414897+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T05:20:43.414897+0000","last_undegraded":"2024-09-17T05:20:43.414897+0000","last_fullsized":"2024-09-17T05:20:43.414897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:06.516432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":14365,"num_write_kb":225564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"65'3640","reported_seq":3759,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.337791+0000","last_change":"2024-09-17T04:49:31.892590+0000","last_active":"2024-09-17T05:20:36.337791+0000","last_peered":"2024-09-17T05:20:36.337791+0000","last_clean":"2024-09-17T05:20:36.337791+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T05:20:36.337791+0000","last_undegraded":"2024-09-17T05:20:36.337791+0000","last_fullsized":"2024-09-17T05:20:36.337791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3640,"log_dups_size":0,"ondisk_log_size":3640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:30.305111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20183,"num_write_kb":277678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"65'2","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340129+0000","last_change":"2024-09-17T04:49:30.859975+0000","last_active":"2024-09-17T05:20:36.340129+0000","last_peered":"2024-09-17T05:20:36.340129+0000","last_clean":"2024-09-17T05:20:36.340129+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T05:20:36.340129+0000","last_undegraded":"2024-09-17T05:20:36.340129+0000","last_fullsized":"2024-09-17T05:20:36.340129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"68'4","reported_seq":136,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.390191+0000","last_change":"2024-09-17T04:49:40.006452+0000","last_active":"2024-09-17T05:20:43.390191+0000","last_peered":"2024-09-17T05:20:43.390191+0000","last_clean":"2024-09-17T05:20:43.390191+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T05:20:43.390191+0000","last_undegraded":"2024-09-17T05:20:43.390191+0000","last_fullsized":"2024-09-17T05:20:43.390191+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:00.010365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"68'3943","reported_seq":4053,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.367267+0000","last_change":"2024-09-17T04:49:41.036658+0000","last_active":"2024-09-17T05:20:43.367267+0000","last_peered":"2024-09-17T05:20:43.367267+0000","last_clean":"2024-09-17T05:20:43.367267+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T05:20:43.367267+0000","last_undegraded":"2024-09-17T05:20:43.367267+0000","last_fullsized":"2024-09-17T05:20:43.367267+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21088,"num_write_kb":359062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"65'3699","reported_seq":3818,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364687+0000","last_change":"2024-09-17T04:49:31.899852+0000","last_active":"2024-09-17T05:20:36.364687+0000","last_peered":"2024-09-17T05:20:36.364687+0000","last_clean":"2024-09-17T05:20:36.364687+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T05:20:36.364687+0000","last_undegraded":"2024-09-17T05:20:36.364687+0000","last_fullsized":"2024-09-17T05:20:36.364687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:07.732884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19661,"num_write_kb":271466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"65'5","reported_seq":134,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:01.675505+0000","last_change":"2024-09-17T05:21:01.675505+0000","last_active":"2024-09-17T05:21:01.675505+0000","last_peered":"2024-09-17T05:21:01.675505+0000","last_clean":"2024-09-17T05:21:01.675505+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T05:21:01.675505+0000","last_undegraded":"2024-09-17T05:21:01.675505+0000","last_fullsized":"2024-09-17T05:21:01.675505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-17T05:21:01.675372+0000","last_deep_scrub":"65'5","last_deep_scrub_stamp":"2024-09-17T05:21:01.675372+0000","last_clean_scrub_stamp":"2024-09-17T05:21:01.675372+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:14.921299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"68'7","reported_seq":159,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.387118+0000","last_change":"2024-09-17T04:49:39.992091+0000","last_active":"2024-09-17T05:20:43.387118+0000","last_peered":"2024-09-17T05:20:43.387118+0000","last_clean":"2024-09-17T05:20:43.387118+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T05:20:43.387118+0000","last_undegraded":"2024-09-17T05:20:43.387118+0000","last_fullsized":"2024-09-17T05:20:43.387118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:18.736763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"68'2532","reported_seq":2644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.411846+0000","last_change":"2024-09-17T04:49:41.050170+0000","last_active":"2024-09-17T05:20:43.411846+0000","last_peered":"2024-09-17T05:20:43.411846+0000","last_clean":"2024-09-17T05:20:43.411846+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T05:20:43.411846+0000","last_undegraded":"2024-09-17T05:20:43.411846+0000","last_fullsized":"2024-09-17T05:20:43.411846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2532,"log_dups_size":0,"ondisk_log_size":2532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13954,"num_write_kb":219876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"65'4598","reported_seq":4718,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339224+0000","last_change":"2024-09-17T04:49:31.892665+0000","last_active":"2024-09-17T05:20:36.339224+0000","last_peered":"2024-09-17T05:20:36.339224+0000","last_clean":"2024-09-17T05:20:36.339224+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T05:20:36.339224+0000","last_undegraded":"2024-09-17T05:20:36.339224+0000","last_fullsized":"2024-09-17T05:20:36.339224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24313,"num_write_kb":335123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"65'9","reported_seq":172,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.367024+0000","last_change":"2024-09-17T04:49:30.888265+0000","last_active":"2024-09-17T05:20:36.367024+0000","last_peered":"2024-09-17T05:20:36.367024+0000","last_clean":"2024-09-17T05:20:36.367024+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T05:20:36.367024+0000","last_undegraded":"2024-09-17T05:20:36.367024+0000","last_fullsized":"2024-09-17T05:20:36.367024+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:44.429507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'50","reported_seq":229,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:48.374529+0000","last_change":"2024-09-17T04:49:39.990739+0000","last_active":"2024-09-17T05:20:48.374529+0000","last_peered":"2024-09-17T05:20:48.374529+0000","last_clean":"2024-09-17T05:20:48.374529+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T05:20:48.374529+0000","last_undegraded":"2024-09-17T05:20:48.374529+0000","last_fullsized":"2024-09-17T05:20:48.374529+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:47.299191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":65,"num_read_kb":114,"num_write":45,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"68'2531","reported_seq":2644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.376480+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T05:20:43.376480+0000","last_peered":"2024-09-17T05:20:43.376480+0000","last_clean":"2024-09-17T05:20:43.376480+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T05:20:43.376480+0000","last_undegraded":"2024-09-17T05:20:43.376480+0000","last_fullsized":"2024-09-17T05:20:43.376480+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2531,"log_dups_size":0,"ondisk_log_size":2531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13881,"num_write_kb":219504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"65'3884","reported_seq":4003,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340033+0000","last_change":"2024-09-17T04:49:31.883157+0000","last_active":"2024-09-17T05:20:36.340033+0000","last_peered":"2024-09-17T05:20:36.340033+0000","last_clean":"2024-09-17T05:20:36.340033+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T05:20:36.340033+0000","last_undegraded":"2024-09-17T05:20:36.340033+0000","last_fullsized":"2024-09-17T05:20:36.340033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21342,"num_write_kb":257935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"68'473","reported_seq":615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.328144+0000","last_change":"2024-09-17T04:49:30.868831+0000","last_active":"2024-09-17T05:20:44.328144+0000","last_peered":"2024-09-17T05:20:44.328144+0000","last_clean":"2024-09-17T05:20:44.328144+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T05:20:44.328144+0000","last_undegraded":"2024-09-17T05:20:44.328144+0000","last_fullsized":"2024-09-17T05:20:44.328144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:32.738912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":57,"num_write":1105,"num_write_kb":359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'11","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410351+0000","last_change":"2024-09-17T04:49:40.000806+0000","last_active":"2024-09-17T05:20:43.410351+0000","last_peered":"2024-09-17T05:20:43.410351+0000","last_clean":"2024-09-17T05:20:43.410351+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T05:20:43.410351+0000","last_undegraded":"2024-09-17T05:20:43.410351+0000","last_fullsized":"2024-09-17T05:20:43.410351+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:51.567269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"68'2691","reported_seq":2803,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404180+0000","last_change":"2024-09-17T04:49:41.042314+0000","last_active":"2024-09-17T05:20:44.404180+0000","last_peered":"2024-09-17T05:20:44.404180+0000","last_clean":"2024-09-17T05:20:44.404180+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T05:20:44.404180+0000","last_undegraded":"2024-09-17T05:20:44.404180+0000","last_fullsized":"2024-09-17T05:20:44.404180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:46:16.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14724,"num_write_kb":243278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"65'4637","reported_seq":4757,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341853+0000","last_change":"2024-09-17T04:49:31.882864+0000","last_active":"2024-09-17T05:20:36.341853+0000","last_peered":"2024-09-17T05:20:36.341853+0000","last_clean":"2024-09-17T05:20:36.341853+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T05:20:36.341853+0000","last_undegraded":"2024-09-17T05:20:36.341853+0000","last_fullsized":"2024-09-17T05:20:36.341853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:06.732488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25111,"num_write_kb":342484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"65'2","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.375220+0000","last_change":"2024-09-17T04:49:30.891197+0000","last_active":"2024-09-17T05:20:36.375220+0000","last_peered":"2024-09-17T05:20:36.375220+0000","last_clean":"2024-09-17T05:20:36.375220+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T05:20:36.375220+0000","last_undegraded":"2024-09-17T05:20:36.375220+0000","last_fullsized":"2024-09-17T05:20:36.375220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:11.550473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'3","reported_seq":117,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393956+0000","last_change":"2024-09-17T04:49:39.995978+0000","last_active":"2024-09-17T05:20:43.393956+0000","last_peered":"2024-09-17T05:20:43.393956+0000","last_clean":"2024-09-17T05:20:43.393956+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T05:20:43.393956+0000","last_undegraded":"2024-09-17T05:20:43.393956+0000","last_fullsized":"2024-09-17T05:20:43.393956+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:44:07.437046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"68'3511","reported_seq":3623,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.341640+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T05:20:43.341640+0000","last_peered":"2024-09-17T05:20:43.341640+0000","last_clean":"2024-09-17T05:20:43.341640+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T05:20:43.341640+0000","last_undegraded":"2024-09-17T05:20:43.341640+0000","last_fullsized":"2024-09-17T05:20:43.341640+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3511,"log_dups_size":0,"ondisk_log_size":3511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19215,"num_write_kb":314590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"65'3495","reported_seq":3615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362299+0000","last_change":"2024-09-17T04:49:31.918669+0000","last_active":"2024-09-17T05:20:36.362299+0000","last_peered":"2024-09-17T05:20:36.362299+0000","last_clean":"2024-09-17T05:20:36.362299+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T05:20:36.362299+0000","last_undegraded":"2024-09-17T05:20:36.362299+0000","last_fullsized":"2024-09-17T05:20:36.362299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:23.964010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18740,"num_write_kb":251346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"65'2","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.351220+0000","last_change":"2024-09-17T04:49:30.902764+0000","last_active":"2024-09-17T05:20:36.351220+0000","last_peered":"2024-09-17T05:20:36.351220+0000","last_clean":"2024-09-17T05:20:36.351220+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T05:20:36.351220+0000","last_undegraded":"2024-09-17T05:20:36.351220+0000","last_fullsized":"2024-09-17T05:20:36.351220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'4","reported_seq":118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.112153+0000","last_change":"2024-09-17T04:49:39.987006+0000","last_active":"2024-09-17T05:20:43.112153+0000","last_peered":"2024-09-17T05:20:43.112153+0000","last_clean":"2024-09-17T05:20:43.112153+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T05:20:43.112153+0000","last_undegraded":"2024-09-17T05:20:43.112153+0000","last_fullsized":"2024-09-17T05:20:43.112153+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:10.119043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"68'2923","reported_seq":3035,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.395059+0000","last_change":"2024-09-17T04:49:41.069177+0000","last_active":"2024-09-17T05:20:43.395059+0000","last_peered":"2024-09-17T05:20:43.395059+0000","last_clean":"2024-09-17T05:20:43.395059+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T05:20:43.395059+0000","last_undegraded":"2024-09-17T05:20:43.395059+0000","last_fullsized":"2024-09-17T05:20:43.395059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:16.855206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":7,"num_write":15908,"num_write_kb":260386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"65'4495","reported_seq":4615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559390+0000","last_change":"2024-09-17T04:49:31.904312+0000","last_active":"2024-09-17T05:20:36.559390+0000","last_peered":"2024-09-17T05:20:36.559390+0000","last_clean":"2024-09-17T05:20:36.559390+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T05:20:36.559390+0000","last_undegraded":"2024-09-17T05:20:36.559390+0000","last_fullsized":"2024-09-17T05:20:36.559390+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:10.624034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24287,"num_write_kb":330898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"65'4","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.365137+0000","last_change":"2024-09-17T04:49:30.890258+0000","last_active":"2024-09-17T05:20:36.365137+0000","last_peered":"2024-09-17T05:20:36.365137+0000","last_clean":"2024-09-17T05:20:36.365137+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T05:20:36.365137+0000","last_undegraded":"2024-09-17T05:20:36.365137+0000","last_fullsized":"2024-09-17T05:20:36.365137+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:56.775377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'2","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.401317+0000","last_change":"2024-09-17T04:49:39.999547+0000","last_active":"2024-09-17T05:20:43.401317+0000","last_peered":"2024-09-17T05:20:43.401317+0000","last_clean":"2024-09-17T05:20:43.401317+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T05:20:43.401317+0000","last_undegraded":"2024-09-17T05:20:43.401317+0000","last_fullsized":"2024-09-17T05:20:43.401317+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:02.718540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"68'3959","reported_seq":4074,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.362853+0000","last_change":"2024-09-17T04:49:41.068537+0000","last_active":"2024-09-17T05:20:43.362853+0000","last_peered":"2024-09-17T05:20:43.362853+0000","last_clean":"2024-09-17T05:20:43.362853+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T05:20:43.362853+0000","last_undegraded":"2024-09-17T05:20:43.362853+0000","last_fullsized":"2024-09-17T05:20:43.362853+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3959,"log_dups_size":0,"ondisk_log_size":3959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:46.695190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":13,"num_write":21335,"num_write_kb":323115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'3351","reported_seq":3471,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.348480+0000","last_change":"2024-09-17T04:49:31.888594+0000","last_active":"2024-09-17T05:20:36.348480+0000","last_peered":"2024-09-17T05:20:36.348480+0000","last_clean":"2024-09-17T05:20:36.348480+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T05:20:36.348480+0000","last_undegraded":"2024-09-17T05:20:36.348480+0000","last_fullsized":"2024-09-17T05:20:36.348480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:05.829248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18427,"num_write_kb":241864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"68'53","reported_seq":243,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.366344+0000","last_change":"2024-09-17T04:49:30.907251+0000","last_active":"2024-09-17T05:20:44.366344+0000","last_peered":"2024-09-17T05:20:44.366344+0000","last_clean":"2024-09-17T05:20:44.366344+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T05:20:44.366344+0000","last_undegraded":"2024-09-17T05:20:44.366344+0000","last_fullsized":"2024-09-17T05:20:44.366344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":116,"num_write":48,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'9","reported_seq":162,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.386330+0000","last_change":"2024-09-17T04:49:40.001304+0000","last_active":"2024-09-17T05:20:44.386330+0000","last_peered":"2024-09-17T05:20:44.386330+0000","last_clean":"2024-09-17T05:20:44.386330+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T05:20:44.386330+0000","last_undegraded":"2024-09-17T05:20:44.386330+0000","last_fullsized":"2024-09-17T05:20:44.386330+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:59.946498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"68'2708","reported_seq":2819,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403990+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T05:20:44.403990+0000","last_peered":"2024-09-17T05:20:44.403990+0000","last_clean":"2024-09-17T05:20:44.403990+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T05:20:44.403990+0000","last_undegraded":"2024-09-17T05:20:44.403990+0000","last_fullsized":"2024-09-17T05:20:44.403990+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14599,"num_write_kb":234639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"65'3456","reported_seq":3576,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339164+0000","last_change":"2024-09-17T04:49:31.903928+0000","last_active":"2024-09-17T05:20:36.339164+0000","last_peered":"2024-09-17T05:20:36.339164+0000","last_clean":"2024-09-17T05:20:36.339164+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T05:20:36.339164+0000","last_undegraded":"2024-09-17T05:20:36.339164+0000","last_fullsized":"2024-09-17T05:20:36.339164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:48.174385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18921,"num_write_kb":260292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"67'13","reported_seq":135,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366577+0000","last_change":"2024-09-17T04:49:30.908522+0000","last_active":"2024-09-17T05:20:36.366577+0000","last_peered":"2024-09-17T05:20:36.366577+0000","last_clean":"2024-09-17T05:20:36.366577+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T05:20:36.366577+0000","last_undegraded":"2024-09-17T05:20:36.366577+0000","last_fullsized":"2024-09-17T05:20:36.366577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:05.575539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"68'473","reported_seq":606,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:47.345918+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T05:20:47.345918+0000","last_peered":"2024-09-17T05:20:47.345918+0000","last_clean":"2024-09-17T05:20:47.345918+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T05:20:47.345918+0000","last_undegraded":"2024-09-17T05:20:47.345918+0000","last_fullsized":"2024-09-17T05:20:47.345918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":54,"num_write":1100,"num_write_kb":354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"68'3699","reported_seq":3812,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.405958+0000","last_change":"2024-09-17T04:49:41.052564+0000","last_active":"2024-09-17T05:20:43.405958+0000","last_peered":"2024-09-17T05:20:43.405958+0000","last_clean":"2024-09-17T05:20:43.405958+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T05:20:43.405958+0000","last_undegraded":"2024-09-17T05:20:43.405958+0000","last_fullsized":"2024-09-17T05:20:43.405958+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:29.374993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":20021,"num_write_kb":309708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"65'3651","reported_seq":3771,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341142+0000","last_change":"2024-09-17T04:49:31.894462+0000","last_active":"2024-09-17T05:20:36.341142+0000","last_peered":"2024-09-17T05:20:36.341142+0000","last_clean":"2024-09-17T05:20:36.341142+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T05:20:36.341142+0000","last_undegraded":"2024-09-17T05:20:36.341142+0000","last_fullsized":"2024-09-17T05:20:36.341142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:31.960892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19517,"num_write_kb":274922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"65'3","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366726+0000","last_change":"2024-09-17T04:49:30.881764+0000","last_active":"2024-09-17T05:20:36.366726+0000","last_peered":"2024-09-17T05:20:36.366726+0000","last_clean":"2024-09-17T05:20:36.366726+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T05:20:36.366726+0000","last_undegraded":"2024-09-17T05:20:36.366726+0000","last_fullsized":"2024-09-17T05:20:36.366726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:56.914361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":218,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.343214+0000","last_change":"2024-09-17T04:49:22.679759+0000","last_active":"2024-09-17T05:20:36.343214+0000","last_peered":"2024-09-17T05:20:36.343214+0000","last_clean":"2024-09-17T05:20:36.343214+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T05:20:36.343214+0000","last_undegraded":"2024-09-17T05:20:36.343214+0000","last_fullsized":"2024-09-17T05:20:36.343214+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:21.657891+0000","last_clean_scrub_stamp":"2024-09-17T04:49:21.657891+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:49.542098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'2","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374244+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T05:20:36.374244+0000","last_peered":"2024-09-17T05:20:36.374244+0000","last_clean":"2024-09-17T05:20:36.374244+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T05:20:36.374244+0000","last_undegraded":"2024-09-17T05:20:36.374244+0000","last_fullsized":"2024-09-17T05:20:36.374244+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:21.202398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"68'2130","reported_seq":2241,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404345+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T05:20:44.404345+0000","last_peered":"2024-09-17T05:20:44.404345+0000","last_clean":"2024-09-17T05:20:44.404345+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T05:20:44.404345+0000","last_undegraded":"2024-09-17T05:20:44.404345+0000","last_fullsized":"2024-09-17T05:20:44.404345+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2130,"log_dups_size":0,"ondisk_log_size":2130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:48.128609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":11413,"num_write_kb":185062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"65'4183","reported_seq":4302,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374483+0000","last_change":"2024-09-17T04:49:31.894321+0000","last_active":"2024-09-17T05:20:36.374483+0000","last_peered":"2024-09-17T05:20:36.374483+0000","last_clean":"2024-09-17T05:20:36.374483+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T05:20:36.374483+0000","last_undegraded":"2024-09-17T05:20:36.374483+0000","last_fullsized":"2024-09-17T05:20:36.374483+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:45.606875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22326,"num_write_kb":302603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"65'5","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.342405+0000","last_change":"2024-09-17T04:49:30.885664+0000","last_active":"2024-09-17T05:20:36.342405+0000","last_peered":"2024-09-17T05:20:36.342405+0000","last_clean":"2024-09-17T05:20:36.342405+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T05:20:36.342405+0000","last_undegraded":"2024-09-17T05:20:36.342405+0000","last_fullsized":"2024-09-17T05:20:36.342405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:52.167394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"68'3","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.376325+0000","last_change":"2024-09-17T04:49:39.990875+0000","last_active":"2024-09-17T05:20:43.376325+0000","last_peered":"2024-09-17T05:20:43.376325+0000","last_clean":"2024-09-17T05:20:43.376325+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T05:20:43.376325+0000","last_undegraded":"2024-09-17T05:20:43.376325+0000","last_fullsized":"2024-09-17T05:20:43.376325+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:07.557383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"68'3285","reported_seq":3397,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.413822+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T05:20:43.413822+0000","last_peered":"2024-09-17T05:20:43.413822+0000","last_clean":"2024-09-17T05:20:43.413822+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T05:20:43.413822+0000","last_undegraded":"2024-09-17T05:20:43.413822+0000","last_fullsized":"2024-09-17T05:20:43.413822+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3285,"log_dups_size":0,"ondisk_log_size":3285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18245,"num_write_kb":282158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"65'4407","reported_seq":4526,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558231+0000","last_change":"2024-09-17T04:49:31.887856+0000","last_active":"2024-09-17T05:20:36.558231+0000","last_peered":"2024-09-17T05:20:36.558231+0000","last_clean":"2024-09-17T05:20:36.558231+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T05:20:36.558231+0000","last_undegraded":"2024-09-17T05:20:36.558231+0000","last_fullsized":"2024-09-17T05:20:36.558231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24558,"num_write_kb":315810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"65'6","reported_seq":148,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352660+0000","last_change":"2024-09-17T04:49:30.900154+0000","last_active":"2024-09-17T05:20:36.352660+0000","last_peered":"2024-09-17T05:20:36.352660+0000","last_clean":"2024-09-17T05:20:36.352660+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T05:20:36.352660+0000","last_undegraded":"2024-09-17T05:20:36.352660+0000","last_fullsized":"2024-09-17T05:20:36.352660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'1445","reported_seq":1601,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410628+0000","last_change":"2024-09-17T04:49:40.020782+0000","last_active":"2024-09-17T05:20:43.410628+0000","last_peered":"2024-09-17T05:20:43.410628+0000","last_clean":"2024-09-17T05:20:43.410628+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T05:20:43.410628+0000","last_undegraded":"2024-09-17T05:20:43.410628+0000","last_fullsized":"2024-09-17T05:20:43.410628+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1445,"log_dups_size":0,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:18.727640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":38239,"num_write":1421,"num_write_kb":4489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'2587","reported_seq":2698,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.177381+0000","last_change":"2024-09-17T04:49:41.031036+0000","last_active":"2024-09-17T05:20:43.177381+0000","last_peered":"2024-09-17T05:20:43.177381+0000","last_clean":"2024-09-17T05:20:43.177381+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T05:20:43.177381+0000","last_undegraded":"2024-09-17T05:20:43.177381+0000","last_fullsized":"2024-09-17T05:20:43.177381+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2587,"log_dups_size":0,"ondisk_log_size":2587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:48.308784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13558,"num_write_kb":238553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"65'4762","reported_seq":4880,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559085+0000","last_change":"2024-09-17T04:49:31.889763+0000","last_active":"2024-09-17T05:20:36.559085+0000","last_peered":"2024-09-17T05:20:36.559085+0000","last_clean":"2024-09-17T05:20:36.559085+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T05:20:36.559085+0000","last_undegraded":"2024-09-17T05:20:36.559085+0000","last_fullsized":"2024-09-17T05:20:36.559085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:39.186462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25049,"num_write_kb":354398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"65'7","reported_seq":170,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340210+0000","last_change":"2024-09-17T04:49:30.903943+0000","last_active":"2024-09-17T05:20:36.340210+0000","last_peered":"2024-09-17T05:20:36.340210+0000","last_clean":"2024-09-17T05:20:36.340210+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T05:20:36.340210+0000","last_undegraded":"2024-09-17T05:20:36.340210+0000","last_fullsized":"2024-09-17T05:20:36.340210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:07.088909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"68'5","reported_seq":118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.116235+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T05:20:43.116235+0000","last_peered":"2024-09-17T05:20:43.116235+0000","last_clean":"2024-09-17T05:20:43.116235+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T05:20:43.116235+0000","last_undegraded":"2024-09-17T05:20:43.116235+0000","last_fullsized":"2024-09-17T05:20:43.116235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"68'2327","reported_seq":2438,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.108349+0000","last_change":"2024-09-17T04:49:41.061127+0000","last_active":"2024-09-17T05:20:43.108349+0000","last_peered":"2024-09-17T05:20:43.108349+0000","last_clean":"2024-09-17T05:20:43.108349+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T05:20:43.108349+0000","last_undegraded":"2024-09-17T05:20:43.108349+0000","last_fullsized":"2024-09-17T05:20:43.108349+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2327,"log_dups_size":0,"ondisk_log_size":2327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:13:58.377257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12892,"num_write_kb":211370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"65'3722","reported_seq":3840,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366989+0000","last_change":"2024-09-17T04:49:31.904298+0000","last_active":"2024-09-17T05:20:36.366989+0000","last_peered":"2024-09-17T05:20:36.366989+0000","last_clean":"2024-09-17T05:20:36.366989+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T05:20:36.366989+0000","last_undegraded":"2024-09-17T05:20:36.366989+0000","last_fullsized":"2024-09-17T05:20:36.366989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3722,"log_dups_size":0,"ondisk_log_size":3722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:14.710512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19724,"num_write_kb":263684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"68'5","reported_seq":154,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:02.705029+0000","last_change":"2024-09-17T05:21:02.705029+0000","last_active":"2024-09-17T05:21:02.705029+0000","last_peered":"2024-09-17T05:21:02.705029+0000","last_clean":"2024-09-17T05:21:02.705029+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T05:21:02.705029+0000","last_undegraded":"2024-09-17T05:21:02.705029+0000","last_fullsized":"2024-09-17T05:21:02.705029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:02.704797+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:02.704797+0000","last_clean_scrub_stamp":"2024-09-17T05:21:02.704797+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:20.480361+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"68'2","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.385249+0000","last_change":"2024-09-17T04:49:40.002909+0000","last_active":"2024-09-17T05:20:43.385249+0000","last_peered":"2024-09-17T05:20:43.385249+0000","last_clean":"2024-09-17T05:20:43.385249+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T05:20:43.385249+0000","last_undegraded":"2024-09-17T05:20:43.385249+0000","last_fullsized":"2024-09-17T05:20:43.385249+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:09.712272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"68'3186","reported_seq":3295,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.381181+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T05:20:43.381181+0000","last_peered":"2024-09-17T05:20:43.381181+0000","last_clean":"2024-09-17T05:20:43.381181+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T05:20:43.381181+0000","last_undegraded":"2024-09-17T05:20:43.381181+0000","last_fullsized":"2024-09-17T05:20:43.381181+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:43.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16763,"num_write_kb":264037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"65'3694","reported_seq":3814,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.362961+0000","last_change":"2024-09-17T04:49:31.912619+0000","last_active":"2024-09-17T05:20:36.362961+0000","last_peered":"2024-09-17T05:20:36.362961+0000","last_clean":"2024-09-17T05:20:36.362961+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T05:20:36.362961+0000","last_undegraded":"2024-09-17T05:20:36.362961+0000","last_fullsized":"2024-09-17T05:20:36.362961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:08.335106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20762,"num_write_kb":284135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"68'15","reported_seq":223,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.342372+0000","last_change":"2024-09-17T04:49:30.890000+0000","last_active":"2024-09-17T05:20:36.342372+0000","last_peered":"2024-09-17T05:20:36.342372+0000","last_clean":"2024-09-17T05:20:36.342372+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T05:20:36.342372+0000","last_undegraded":"2024-09-17T05:20:36.342372+0000","last_fullsized":"2024-09-17T05:20:36.342372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:54.676493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":264,"num_read_kb":264,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"68'3","reported_seq":114,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.394770+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T05:20:43.394770+0000","last_peered":"2024-09-17T05:20:43.394770+0000","last_clean":"2024-09-17T05:20:43.394770+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T05:20:43.394770+0000","last_undegraded":"2024-09-17T05:20:43.394770+0000","last_fullsized":"2024-09-17T05:20:43.394770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:52.364568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"68'3074","reported_seq":3186,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.395538+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T05:20:44.395538+0000","last_peered":"2024-09-17T05:20:44.395538+0000","last_clean":"2024-09-17T05:20:44.395538+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T05:20:44.395538+0000","last_undegraded":"2024-09-17T05:20:44.395538+0000","last_fullsized":"2024-09-17T05:20:44.395538+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:03.537392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":16509,"num_write_kb":287321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'2828","reported_seq":2946,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.342437+0000","last_change":"2024-09-17T04:49:31.891580+0000","last_active":"2024-09-17T05:20:36.342437+0000","last_peered":"2024-09-17T05:20:36.342437+0000","last_clean":"2024-09-17T05:20:36.342437+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T05:20:36.342437+0000","last_undegraded":"2024-09-17T05:20:36.342437+0000","last_fullsized":"2024-09-17T05:20:36.342437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:55.652996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14821,"num_write_kb":206837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"65'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557382+0000","last_change":"2024-09-17T04:49:30.886452+0000","last_active":"2024-09-17T05:20:36.557382+0000","last_peered":"2024-09-17T05:20:36.557382+0000","last_clean":"2024-09-17T05:20:36.557382+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T05:20:36.557382+0000","last_undegraded":"2024-09-17T05:20:36.557382+0000","last_fullsized":"2024-09-17T05:20:36.557382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:13.177298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"68'5","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.388852+0000","last_change":"2024-09-17T04:49:40.017845+0000","last_active":"2024-09-17T05:20:43.388852+0000","last_peered":"2024-09-17T05:20:43.388852+0000","last_clean":"2024-09-17T05:20:43.388852+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T05:20:43.388852+0000","last_undegraded":"2024-09-17T05:20:43.388852+0000","last_fullsized":"2024-09-17T05:20:43.388852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:52.120543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"68'2590","reported_seq":2699,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.324869+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T05:20:43.324869+0000","last_peered":"2024-09-17T05:20:43.324869+0000","last_clean":"2024-09-17T05:20:43.324869+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T05:20:43.324869+0000","last_undegraded":"2024-09-17T05:20:43.324869+0000","last_fullsized":"2024-09-17T05:20:43.324869+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:56.204916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14722,"num_write_kb":243326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'4461","reported_seq":4579,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364760+0000","last_change":"2024-09-17T04:49:31.890068+0000","last_active":"2024-09-17T05:20:36.364760+0000","last_peered":"2024-09-17T05:20:36.364760+0000","last_clean":"2024-09-17T05:20:36.364760+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T05:20:36.364760+0000","last_undegraded":"2024-09-17T05:20:36.364760+0000","last_fullsized":"2024-09-17T05:20:36.364760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4461,"log_dups_size":0,"ondisk_log_size":4461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:32.970566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24667,"num_write_kb":332445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"65'3","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364408+0000","last_change":"2024-09-17T04:49:30.891517+0000","last_active":"2024-09-17T05:20:36.364408+0000","last_peered":"2024-09-17T05:20:36.364408+0000","last_clean":"2024-09-17T05:20:36.364408+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T05:20:36.364408+0000","last_undegraded":"2024-09-17T05:20:36.364408+0000","last_fullsized":"2024-09-17T05:20:36.364408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:45.815544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"68'367","reported_seq":579,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:39.638931+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T05:20:39.638931+0000","last_peered":"2024-09-17T05:20:39.638931+0000","last_clean":"2024-09-17T05:20:39.638931+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T05:20:39.638931+0000","last_undegraded":"2024-09-17T05:20:39.638931+0000","last_fullsized":"2024-09-17T05:20:39.638931+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":180,"num_read_kb":82080,"num_write":364,"num_write_kb":4295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"68'3856","reported_seq":3961,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403743+0000","last_change":"2024-09-17T04:50:12.067788+0000","last_active":"2024-09-17T05:20:44.403743+0000","last_peered":"2024-09-17T05:20:44.403743+0000","last_clean":"2024-09-17T05:20:44.403743+0000","last_became_active":"2024-09-17T04:50:12.066741+0000","last_became_peered":"2024-09-17T04:50:12.066741+0000","last_unstale":"2024-09-17T05:20:44.403743+0000","last_undegraded":"2024-09-17T05:20:44.403743+0000","last_fullsized":"2024-09-17T05:20:44.403743+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3856,"log_dups_size":0,"ondisk_log_size":3856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":20645,"num_write_kb":344954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"65'3521","reported_seq":3639,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.344504+0000","last_change":"2024-09-17T04:49:31.892511+0000","last_active":"2024-09-17T05:20:36.344504+0000","last_peered":"2024-09-17T05:20:36.344504+0000","last_clean":"2024-09-17T05:20:36.344504+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T05:20:36.344504+0000","last_undegraded":"2024-09-17T05:20:36.344504+0000","last_fullsized":"2024-09-17T05:20:36.344504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3521,"log_dups_size":0,"ondisk_log_size":3521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:28.148439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19198,"num_write_kb":253218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"67'10","reported_seq":172,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.358352+0000","last_change":"2024-09-17T04:49:30.894911+0000","last_active":"2024-09-17T05:20:36.358352+0000","last_peered":"2024-09-17T05:20:36.358352+0000","last_clean":"2024-09-17T05:20:36.358352+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T05:20:36.358352+0000","last_undegraded":"2024-09-17T05:20:36.358352+0000","last_fullsized":"2024-09-17T05:20:36.358352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:55.165328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":168,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"68'30","reported_seq":181,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:45.378073+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T05:20:45.378073+0000","last_peered":"2024-09-17T05:20:45.378073+0000","last_clean":"2024-09-17T05:20:45.378073+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T05:20:45.378073+0000","last_undegraded":"2024-09-17T05:20:45.378073+0000","last_fullsized":"2024-09-17T05:20:45.378073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:07.945355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":31,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"68'3085","reported_seq":3195,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.412893+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T05:20:43.412893+0000","last_peered":"2024-09-17T05:20:43.412893+0000","last_clean":"2024-09-17T05:20:43.412893+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T05:20:43.412893+0000","last_undegraded":"2024-09-17T05:20:43.412893+0000","last_fullsized":"2024-09-17T05:20:43.412893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16824,"num_write_kb":288541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"65'3611","reported_seq":3729,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.556980+0000","last_change":"2024-09-17T04:49:31.895116+0000","last_active":"2024-09-17T05:20:36.556980+0000","last_peered":"2024-09-17T05:20:36.556980+0000","last_clean":"2024-09-17T05:20:36.556980+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T05:20:36.556980+0000","last_undegraded":"2024-09-17T05:20:36.556980+0000","last_fullsized":"2024-09-17T05:20:36.556980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3611,"log_dups_size":0,"ondisk_log_size":3611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19975,"num_write_kb":264442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"68'349","reported_seq":575,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.121847+0000","last_change":"2024-09-17T04:49:30.878199+0000","last_active":"2024-09-17T05:20:44.121847+0000","last_peered":"2024-09-17T05:20:44.121847+0000","last_clean":"2024-09-17T05:20:44.121847+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T05:20:44.121847+0000","last_undegraded":"2024-09-17T05:20:44.121847+0000","last_fullsized":"2024-09-17T05:20:44.121847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:54.166615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":190,"num_read_kb":82094,"num_write":345,"num_write_kb":4278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"68'3","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.396510+0000","last_change":"2024-09-17T04:49:39.996403+0000","last_active":"2024-09-17T05:20:44.396510+0000","last_peered":"2024-09-17T05:20:44.396510+0000","last_clean":"2024-09-17T05:20:44.396510+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T05:20:44.396510+0000","last_undegraded":"2024-09-17T05:20:44.396510+0000","last_fullsized":"2024-09-17T05:20:44.396510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:16.447835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"68'2950","reported_seq":3061,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.342183+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T05:20:43.342183+0000","last_peered":"2024-09-17T05:20:43.342183+0000","last_clean":"2024-09-17T05:20:43.342183+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T05:20:43.342183+0000","last_undegraded":"2024-09-17T05:20:43.342183+0000","last_fullsized":"2024-09-17T05:20:43.342183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":16110,"num_write_kb":249022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"65'4529","reported_seq":4647,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.361047+0000","last_change":"2024-09-17T04:49:31.895897+0000","last_active":"2024-09-17T05:20:36.361047+0000","last_peered":"2024-09-17T05:20:36.361047+0000","last_clean":"2024-09-17T05:20:36.361047+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T05:20:36.361047+0000","last_undegraded":"2024-09-17T05:20:36.361047+0000","last_fullsized":"2024-09-17T05:20:36.361047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:51.170384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23581,"num_write_kb":327974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"68'31","reported_seq":214,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.399687+0000","last_change":"2024-09-17T04:49:30.895998+0000","last_active":"2024-09-17T05:20:44.399687+0000","last_peered":"2024-09-17T05:20:44.399687+0000","last_clean":"2024-09-17T05:20:44.399687+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T05:20:44.399687+0000","last_undegraded":"2024-09-17T05:20:44.399687+0000","last_fullsized":"2024-09-17T05:20:44.399687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:39.769223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'8","reported_seq":160,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.415231+0000","last_change":"2024-09-17T04:49:39.990407+0000","last_active":"2024-09-17T05:20:43.415231+0000","last_peered":"2024-09-17T05:20:43.415231+0000","last_clean":"2024-09-17T05:20:43.415231+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T05:20:43.415231+0000","last_undegraded":"2024-09-17T05:20:43.415231+0000","last_fullsized":"2024-09-17T05:20:43.415231+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:40.628810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"68'2811","reported_seq":2924,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.378555+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T05:20:43.378555+0000","last_peered":"2024-09-17T05:20:43.378555+0000","last_clean":"2024-09-17T05:20:43.378555+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T05:20:43.378555+0000","last_undegraded":"2024-09-17T05:20:43.378555+0000","last_fullsized":"2024-09-17T05:20:43.378555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2811,"log_dups_size":0,"ondisk_log_size":2811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15712,"num_write_kb":243392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"65'4109","reported_seq":4227,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.353534+0000","last_change":"2024-09-17T04:49:31.894857+0000","last_active":"2024-09-17T05:20:36.353534+0000","last_peered":"2024-09-17T05:20:36.353534+0000","last_clean":"2024-09-17T05:20:36.353534+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T05:20:36.353534+0000","last_undegraded":"2024-09-17T05:20:36.353534+0000","last_fullsized":"2024-09-17T05:20:36.353534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:37.572868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21556,"num_write_kb":307833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"65'3","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558871+0000","last_change":"2024-09-17T04:49:30.872231+0000","last_active":"2024-09-17T05:20:36.558871+0000","last_peered":"2024-09-17T05:20:36.558871+0000","last_clean":"2024-09-17T05:20:36.558871+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T05:20:36.558871+0000","last_undegraded":"2024-09-17T05:20:36.558871+0000","last_fullsized":"2024-09-17T05:20:36.558871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'15","reported_seq":209,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410016+0000","last_change":"2024-09-17T04:49:39.998907+0000","last_active":"2024-09-17T05:20:43.410016+0000","last_peered":"2024-09-17T05:20:43.410016+0000","last_clean":"2024-09-17T05:20:43.410016+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T05:20:43.410016+0000","last_undegraded":"2024-09-17T05:20:43.410016+0000","last_fullsized":"2024-09-17T05:20:43.410016+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:40.073621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":246,"num_read_kb":246,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"68'2542","reported_seq":2652,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:42.865170+0000","last_change":"2024-09-17T04:49:41.040278+0000","last_active":"2024-09-17T05:20:42.865170+0000","last_peered":"2024-09-17T05:20:42.865170+0000","last_clean":"2024-09-17T05:20:42.865170+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T05:20:42.865170+0000","last_undegraded":"2024-09-17T05:20:42.865170+0000","last_fullsized":"2024-09-17T05:20:42.865170+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2542,"log_dups_size":0,"ondisk_log_size":2542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.895223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13327,"num_write_kb":225533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"65'3736","reported_seq":3854,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.365811+0000","last_change":"2024-09-17T04:49:31.902622+0000","last_active":"2024-09-17T05:20:36.365811+0000","last_peered":"2024-09-17T05:20:36.365811+0000","last_clean":"2024-09-17T05:20:36.365811+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T05:20:36.365811+0000","last_undegraded":"2024-09-17T05:20:36.365811+0000","last_fullsized":"2024-09-17T05:20:36.365811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:10.791861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20513,"num_write_kb":287089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"68'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.363961+0000","last_change":"2024-09-17T04:49:30.879754+0000","last_active":"2024-09-17T05:20:44.363961+0000","last_peered":"2024-09-17T05:20:44.363961+0000","last_clean":"2024-09-17T05:20:44.363961+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T05:20:44.363961+0000","last_undegraded":"2024-09-17T05:20:44.363961+0000","last_fullsized":"2024-09-17T05:20:44.363961+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:36.283892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"68'5","reported_seq":117,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.409527+0000","last_change":"2024-09-17T04:49:40.007638+0000","last_active":"2024-09-17T05:20:43.409527+0000","last_peered":"2024-09-17T05:20:43.409527+0000","last_clean":"2024-09-17T05:20:43.409527+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T05:20:43.409527+0000","last_undegraded":"2024-09-17T05:20:43.409527+0000","last_fullsized":"2024-09-17T05:20:43.409527+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:15.845612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"68'4175","reported_seq":4291,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.426185+0000","last_change":"2024-09-17T04:49:41.050551+0000","last_active":"2024-09-17T05:20:44.426185+0000","last_peered":"2024-09-17T05:20:44.426185+0000","last_clean":"2024-09-17T05:20:44.426185+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T05:20:44.426185+0000","last_undegraded":"2024-09-17T05:20:44.426185+0000","last_fullsized":"2024-09-17T05:20:44.426185+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:11.088230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":40,"num_write":23028,"num_write_kb":362813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"68'25","reported_seq":139,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.384829+0000","last_change":"2024-09-17T04:49:40.008096+0000","last_active":"2024-09-17T05:20:43.384829+0000","last_peered":"2024-09-17T05:20:43.384829+0000","last_clean":"2024-09-17T05:20:43.384829+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T05:20:43.384829+0000","last_undegraded":"2024-09-17T05:20:43.384829+0000","last_fullsized":"2024-09-17T05:20:43.384829+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:37.778756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"65'4339","reported_seq":4456,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.338640+0000","last_change":"2024-09-17T04:49:31.863486+0000","last_active":"2024-09-17T05:20:36.338640+0000","last_peered":"2024-09-17T05:20:36.338640+0000","last_clean":"2024-09-17T05:20:36.338640+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T05:20:36.338640+0000","last_undegraded":"2024-09-17T05:20:36.338640+0000","last_fullsized":"2024-09-17T05:20:36.338640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:16.474105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23134,"num_write_kb":338604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"65'2","reported_seq":122,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366399+0000","last_change":"2024-09-17T04:49:30.884672+0000","last_active":"2024-09-17T05:20:36.366399+0000","last_peered":"2024-09-17T05:20:36.366399+0000","last_clean":"2024-09-17T05:20:36.366399+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T05:20:36.366399+0000","last_undegraded":"2024-09-17T05:20:36.366399+0000","last_fullsized":"2024-09-17T05:20:36.366399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:44.106545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"68'2877","reported_seq":2989,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.395024+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T05:20:43.395024+0000","last_peered":"2024-09-17T05:20:43.395024+0000","last_clean":"2024-09-17T05:20:43.395024+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T05:20:43.395024+0000","last_undegraded":"2024-09-17T05:20:43.395024+0000","last_fullsized":"2024-09-17T05:20:43.395024+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2877,"log_dups_size":0,"ondisk_log_size":2877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15865,"num_write_kb":272557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"68'1000","reported_seq":1162,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.429369+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T05:20:44.429369+0000","last_peered":"2024-09-17T05:20:44.429369+0000","last_clean":"2024-09-17T05:20:44.429369+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T05:20:44.429369+0000","last_undegraded":"2024-09-17T05:20:44.429369+0000","last_fullsized":"2024-09-17T05:20:44.429369+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":9969,"num_write":977,"num_write_kb":4635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"65'4230","reported_seq":4348,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.349210+0000","last_change":"2024-09-17T04:49:31.916278+0000","last_active":"2024-09-17T05:20:36.349210+0000","last_peered":"2024-09-17T05:20:36.349210+0000","last_clean":"2024-09-17T05:20:36.349210+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T05:20:36.349210+0000","last_undegraded":"2024-09-17T05:20:36.349210+0000","last_fullsized":"2024-09-17T05:20:36.349210+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:08.265077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22609,"num_write_kb":319990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"65'3","reported_seq":122,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339504+0000","last_change":"2024-09-17T04:49:30.877590+0000","last_active":"2024-09-17T05:20:36.339504+0000","last_peered":"2024-09-17T05:20:36.339504+0000","last_clean":"2024-09-17T05:20:36.339504+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T05:20:36.339504+0000","last_undegraded":"2024-09-17T05:20:36.339504+0000","last_fullsized":"2024-09-17T05:20:36.339504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:56:14.339809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"68'3526","reported_seq":3630,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393790+0000","last_change":"2024-09-17T04:50:12.072014+0000","last_active":"2024-09-17T05:20:43.393790+0000","last_peered":"2024-09-17T05:20:43.393790+0000","last_clean":"2024-09-17T05:20:43.393790+0000","last_became_active":"2024-09-17T04:50:12.071053+0000","last_became_peered":"2024-09-17T04:50:12.071053+0000","last_unstale":"2024-09-17T05:20:43.393790+0000","last_undegraded":"2024-09-17T05:20:43.393790+0000","last_fullsized":"2024-09-17T05:20:43.393790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20183,"num_write_kb":298078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"68'284","reported_seq":409,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.123738+0000","last_change":"2024-09-17T04:49:30.888069+0000","last_active":"2024-09-17T05:20:44.123738+0000","last_peered":"2024-09-17T05:20:44.123738+0000","last_clean":"2024-09-17T05:20:44.123738+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T05:20:44.123738+0000","last_undegraded":"2024-09-17T05:20:44.123738+0000","last_fullsized":"2024-09-17T05:20:44.123738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:22.943393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2099732,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":4077,"num_write":257,"num_write_kb":2207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'3509","reported_seq":3626,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.339983+0000","last_change":"2024-09-17T04:49:31.888838+0000","last_active":"2024-09-17T05:20:36.339983+0000","last_peered":"2024-09-17T05:20:36.339983+0000","last_clean":"2024-09-17T05:20:36.339983+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T05:20:36.339983+0000","last_undegraded":"2024-09-17T05:20:36.339983+0000","last_fullsized":"2024-09-17T05:20:36.339983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3509,"log_dups_size":0,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18360,"num_write_kb":249727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'5","reported_seq":156,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:39.659577+0000","last_change":"2024-09-17T04:49:40.012318+0000","last_active":"2024-09-17T05:20:39.659577+0000","last_peered":"2024-09-17T05:20:39.659577+0000","last_clean":"2024-09-17T05:20:39.659577+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T05:20:39.659577+0000","last_undegraded":"2024-09-17T05:20:39.659577+0000","last_fullsized":"2024-09-17T05:20:39.659577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:27.230604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"68'3113","reported_seq":3225,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.330899+0000","last_change":"2024-09-17T04:49:41.055263+0000","last_active":"2024-09-17T05:20:43.330899+0000","last_peered":"2024-09-17T05:20:43.330899+0000","last_clean":"2024-09-17T05:20:43.330899+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T05:20:43.330899+0000","last_undegraded":"2024-09-17T05:20:43.330899+0000","last_fullsized":"2024-09-17T05:20:43.330899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":16978,"num_write_kb":256690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"65'657","reported_seq":800,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.349177+0000","last_change":"2024-09-17T04:49:30.901429+0000","last_active":"2024-09-17T05:20:36.349177+0000","last_peered":"2024-09-17T05:20:36.349177+0000","last_clean":"2024-09-17T05:20:36.349177+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T05:20:36.349177+0000","last_undegraded":"2024-09-17T05:20:36.349177+0000","last_fullsized":"2024-09-17T05:20:36.349177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:29.826523+0000","last_clean_scrub_stamp":"2024-09-17T04:49:29.826523+0000","objects_scrubbed":0,"log_size":657,"log_dups_size":0,"ondisk_log_size":657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:40.818659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":79257,"num_write":647,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'3199","reported_seq":3317,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.343729+0000","last_change":"2024-09-17T04:49:31.891724+0000","last_active":"2024-09-17T05:20:36.343729+0000","last_peered":"2024-09-17T05:20:36.343729+0000","last_clean":"2024-09-17T05:20:36.343729+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T05:20:36.343729+0000","last_undegraded":"2024-09-17T05:20:36.343729+0000","last_fullsized":"2024-09-17T05:20:36.343729+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:45.117307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16756,"num_write_kb":239128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"68'43","reported_seq":194,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.403635+0000","last_change":"2024-09-17T04:49:40.018487+0000","last_active":"2024-09-17T05:20:43.403635+0000","last_peered":"2024-09-17T05:20:43.403635+0000","last_clean":"2024-09-17T05:20:43.403635+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T05:20:43.403635+0000","last_undegraded":"2024-09-17T05:20:43.403635+0000","last_fullsized":"2024-09-17T05:20:43.403635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:18:24.586834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"68'3014","reported_seq":3125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402775+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T05:20:43.402775+0000","last_peered":"2024-09-17T05:20:43.402775+0000","last_clean":"2024-09-17T05:20:43.402775+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T05:20:43.402775+0000","last_undegraded":"2024-09-17T05:20:43.402775+0000","last_fullsized":"2024-09-17T05:20:43.402775+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3014,"log_dups_size":0,"ondisk_log_size":3014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16762,"num_write_kb":251577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"65'3751","reported_seq":3871,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364101+0000","last_change":"2024-09-17T04:49:31.909990+0000","last_active":"2024-09-17T05:20:36.364101+0000","last_peered":"2024-09-17T05:20:36.364101+0000","last_clean":"2024-09-17T05:20:36.364101+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T05:20:36.364101+0000","last_undegraded":"2024-09-17T05:20:36.364101+0000","last_fullsized":"2024-09-17T05:20:36.364101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20040,"num_write_kb":277001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"68'3862","reported_seq":3974,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.271131+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T05:20:43.271131+0000","last_peered":"2024-09-17T05:20:43.271131+0000","last_clean":"2024-09-17T05:20:43.271131+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T05:20:43.271131+0000","last_undegraded":"2024-09-17T05:20:43.271131+0000","last_fullsized":"2024-09-17T05:20:43.271131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20235,"num_write_kb":310497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"65'4193","reported_seq":4314,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.356835+0000","last_change":"2024-09-17T04:49:31.886193+0000","last_active":"2024-09-17T05:20:36.356835+0000","last_peered":"2024-09-17T05:20:36.356835+0000","last_clean":"2024-09-17T05:20:36.356835+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T05:20:36.356835+0000","last_undegraded":"2024-09-17T05:20:36.356835+0000","last_fullsized":"2024-09-17T05:20:36.356835+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4193,"log_dups_size":0,"ondisk_log_size":4193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:33.749673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22222,"num_write_kb":306243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"68'3519","reported_seq":3632,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.398772+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T05:20:43.398772+0000","last_peered":"2024-09-17T05:20:43.398772+0000","last_clean":"2024-09-17T05:20:43.398772+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T05:20:43.398772+0000","last_undegraded":"2024-09-17T05:20:43.398772+0000","last_fullsized":"2024-09-17T05:20:43.398772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3519,"log_dups_size":0,"ondisk_log_size":3519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":19548,"num_write_kb":295624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"65'3872","reported_seq":3994,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.360995+0000","last_change":"2024-09-17T04:49:31.921045+0000","last_active":"2024-09-17T05:20:36.360995+0000","last_peered":"2024-09-17T05:20:36.360995+0000","last_clean":"2024-09-17T05:20:36.360995+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T05:20:36.360995+0000","last_undegraded":"2024-09-17T05:20:36.360995+0000","last_fullsized":"2024-09-17T05:20:36.360995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3872,"log_dups_size":0,"ondisk_log_size":3872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:19.117875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20576,"num_write_kb":279241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"68'2938","reported_seq":3050,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.380635+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T05:20:43.380635+0000","last_peered":"2024-09-17T05:20:43.380635+0000","last_clean":"2024-09-17T05:20:43.380635+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T05:20:43.380635+0000","last_undegraded":"2024-09-17T05:20:43.380635+0000","last_fullsized":"2024-09-17T05:20:43.380635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2938,"log_dups_size":0,"ondisk_log_size":2938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":16647,"num_write_kb":271506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"65'3500","reported_seq":3621,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.349042+0000","last_change":"2024-09-17T04:49:31.887492+0000","last_active":"2024-09-17T05:20:36.349042+0000","last_peered":"2024-09-17T05:20:36.349042+0000","last_clean":"2024-09-17T05:20:36.349042+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T05:20:36.349042+0000","last_undegraded":"2024-09-17T05:20:36.349042+0000","last_fullsized":"2024-09-17T05:20:36.349042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:28.515155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18562,"num_write_kb":258704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"68'3339","reported_seq":3453,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.418584+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T05:20:43.418584+0000","last_peered":"2024-09-17T05:20:43.418584+0000","last_clean":"2024-09-17T05:20:43.418584+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T05:20:43.418584+0000","last_undegraded":"2024-09-17T05:20:43.418584+0000","last_fullsized":"2024-09-17T05:20:43.418584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3339,"log_dups_size":0,"ondisk_log_size":3339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17940,"num_write_kb":305481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"65'3819","reported_seq":3939,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341730+0000","last_change":"2024-09-17T04:49:31.895605+0000","last_active":"2024-09-17T05:20:36.341730+0000","last_peered":"2024-09-17T05:20:36.341730+0000","last_clean":"2024-09-17T05:20:36.341730+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T05:20:36.341730+0000","last_undegraded":"2024-09-17T05:20:36.341730+0000","last_fullsized":"2024-09-17T05:20:36.341730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:22:29.073358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21007,"num_write_kb":273012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"68'3374","reported_seq":3493,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.413499+0000","last_change":"2024-09-17T04:50:12.076134+0000","last_active":"2024-09-17T05:20:43.413499+0000","last_peered":"2024-09-17T05:20:43.413499+0000","last_clean":"2024-09-17T05:20:43.413499+0000","last_became_active":"2024-09-17T04:50:12.075350+0000","last_became_peered":"2024-09-17T04:50:12.075350+0000","last_unstale":"2024-09-17T05:20:43.413499+0000","last_undegraded":"2024-09-17T05:20:43.413499+0000","last_fullsized":"2024-09-17T05:20:43.413499+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3374,"log_dups_size":0,"ondisk_log_size":3374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18297,"num_write_kb":270032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"65'4233","reported_seq":4352,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.341464+0000","last_change":"2024-09-17T04:49:31.894484+0000","last_active":"2024-09-17T05:20:36.341464+0000","last_peered":"2024-09-17T05:20:36.341464+0000","last_clean":"2024-09-17T05:20:36.341464+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T05:20:36.341464+0000","last_undegraded":"2024-09-17T05:20:36.341464+0000","last_fullsized":"2024-09-17T05:20:36.341464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:15.734322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23577,"num_write_kb":311694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"68'3382","reported_seq":3494,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.416880+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T05:20:43.416880+0000","last_peered":"2024-09-17T05:20:43.416880+0000","last_clean":"2024-09-17T05:20:43.416880+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T05:20:43.416880+0000","last_undegraded":"2024-09-17T05:20:43.416880+0000","last_fullsized":"2024-09-17T05:20:43.416880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18178,"num_write_kb":293977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"65'3778","reported_seq":3897,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.364369+0000","last_change":"2024-09-17T04:49:31.910280+0000","last_active":"2024-09-17T05:20:36.364369+0000","last_peered":"2024-09-17T05:20:36.364369+0000","last_clean":"2024-09-17T05:20:36.364369+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T05:20:36.364369+0000","last_undegraded":"2024-09-17T05:20:36.364369+0000","last_fullsized":"2024-09-17T05:20:36.364369+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:47.994834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21044,"num_write_kb":281017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"68'2956","reported_seq":3071,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404554+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T05:20:44.404554+0000","last_peered":"2024-09-17T05:20:44.404554+0000","last_clean":"2024-09-17T05:20:44.404554+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T05:20:44.404554+0000","last_undegraded":"2024-09-17T05:20:44.404554+0000","last_fullsized":"2024-09-17T05:20:44.404554+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2956,"log_dups_size":0,"ondisk_log_size":2956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15875,"num_write_kb":264255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"65'4647","reported_seq":4767,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352380+0000","last_change":"2024-09-17T04:49:31.888012+0000","last_active":"2024-09-17T05:20:36.352380+0000","last_peered":"2024-09-17T05:20:36.352380+0000","last_clean":"2024-09-17T05:20:36.352380+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T05:20:36.352380+0000","last_undegraded":"2024-09-17T05:20:36.352380+0000","last_fullsized":"2024-09-17T05:20:36.352380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24699,"num_write_kb":336693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"68'3095","reported_seq":3210,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.396708+0000","last_change":"2024-09-17T04:49:41.045502+0000","last_active":"2024-09-17T05:20:43.396708+0000","last_peered":"2024-09-17T05:20:43.396708+0000","last_clean":"2024-09-17T05:20:43.396708+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T05:20:43.396708+0000","last_undegraded":"2024-09-17T05:20:43.396708+0000","last_fullsized":"2024-09-17T05:20:43.396708+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:19.746356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":36,"num_write":16750,"num_write_kb":272041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"65'4070","reported_seq":4190,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.361587+0000","last_change":"2024-09-17T04:49:31.905414+0000","last_active":"2024-09-17T05:20:36.361587+0000","last_peered":"2024-09-17T05:20:36.361587+0000","last_clean":"2024-09-17T05:20:36.361587+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T05:20:36.361587+0000","last_undegraded":"2024-09-17T05:20:36.361587+0000","last_fullsized":"2024-09-17T05:20:36.361587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4070,"log_dups_size":0,"ondisk_log_size":4070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21906,"num_write_kb":288644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"68'4276","reported_seq":4387,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404966+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T05:20:44.404966+0000","last_peered":"2024-09-17T05:20:44.404966+0000","last_clean":"2024-09-17T05:20:44.404966+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T05:20:44.404966+0000","last_undegraded":"2024-09-17T05:20:44.404966+0000","last_fullsized":"2024-09-17T05:20:44.404966+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22732,"num_write_kb":365385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"65'4174","reported_seq":4292,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.348275+0000","last_change":"2024-09-17T04:49:31.902832+0000","last_active":"2024-09-17T05:20:36.348275+0000","last_peered":"2024-09-17T05:20:36.348275+0000","last_clean":"2024-09-17T05:20:36.348275+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T05:20:36.348275+0000","last_undegraded":"2024-09-17T05:20:36.348275+0000","last_fullsized":"2024-09-17T05:20:36.348275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:48.978456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22870,"num_write_kb":296621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"68'2821","reported_seq":2931,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393189+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T05:20:43.393189+0000","last_peered":"2024-09-17T05:20:43.393189+0000","last_clean":"2024-09-17T05:20:43.393189+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T05:20:43.393189+0000","last_undegraded":"2024-09-17T05:20:43.393189+0000","last_fullsized":"2024-09-17T05:20:43.393189+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2821,"log_dups_size":0,"ondisk_log_size":2821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.356881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15110,"num_write_kb":232606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"65'4280","reported_seq":4399,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.343835+0000","last_change":"2024-09-17T04:49:31.894510+0000","last_active":"2024-09-17T05:20:36.343835+0000","last_peered":"2024-09-17T05:20:36.343835+0000","last_clean":"2024-09-17T05:20:36.343835+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T05:20:36.343835+0000","last_undegraded":"2024-09-17T05:20:36.343835+0000","last_fullsized":"2024-09-17T05:20:36.343835+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4280,"log_dups_size":0,"ondisk_log_size":4280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:57.925122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23007,"num_write_kb":313406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"68'3452","reported_seq":3562,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.303318+0000","last_change":"2024-09-17T04:49:41.038265+0000","last_active":"2024-09-17T05:20:43.303318+0000","last_peered":"2024-09-17T05:20:43.303318+0000","last_clean":"2024-09-17T05:20:43.303318+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T05:20:43.303318+0000","last_undegraded":"2024-09-17T05:20:43.303318+0000","last_fullsized":"2024-09-17T05:20:43.303318+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3452,"log_dups_size":0,"ondisk_log_size":3452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:41.638403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19160,"num_write_kb":306156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"65'4384","reported_seq":4505,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340299+0000","last_change":"2024-09-17T04:49:31.898569+0000","last_active":"2024-09-17T05:20:36.340299+0000","last_peered":"2024-09-17T05:20:36.340299+0000","last_clean":"2024-09-17T05:20:36.340299+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T05:20:36.340299+0000","last_undegraded":"2024-09-17T05:20:36.340299+0000","last_fullsized":"2024-09-17T05:20:36.340299+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:51.897637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23582,"num_write_kb":336581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"68'3472","reported_seq":3589,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.255838+0000","last_change":"2024-09-17T04:50:12.068348+0000","last_active":"2024-09-17T05:20:43.255838+0000","last_peered":"2024-09-17T05:20:43.255838+0000","last_clean":"2024-09-17T05:20:43.255838+0000","last_became_active":"2024-09-17T04:50:12.067942+0000","last_became_peered":"2024-09-17T04:50:12.067942+0000","last_unstale":"2024-09-17T05:20:43.255838+0000","last_undegraded":"2024-09-17T05:20:43.255838+0000","last_fullsized":"2024-09-17T05:20:43.255838+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3472,"log_dups_size":0,"ondisk_log_size":3472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:33.954020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18777,"num_write_kb":292458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"65'3688","reported_seq":3807,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557802+0000","last_change":"2024-09-17T04:49:31.906561+0000","last_active":"2024-09-17T05:20:36.557802+0000","last_peered":"2024-09-17T05:20:36.557802+0000","last_clean":"2024-09-17T05:20:36.557802+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T05:20:36.557802+0000","last_undegraded":"2024-09-17T05:20:36.557802+0000","last_fullsized":"2024-09-17T05:20:36.557802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20358,"num_write_kb":257774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"68'3214","reported_seq":3329,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.416523+0000","last_change":"2024-09-17T04:49:41.038072+0000","last_active":"2024-09-17T05:20:43.416523+0000","last_peered":"2024-09-17T05:20:43.416523+0000","last_clean":"2024-09-17T05:20:43.416523+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T05:20:43.416523+0000","last_undegraded":"2024-09-17T05:20:43.416523+0000","last_fullsized":"2024-09-17T05:20:43.416523+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:13.598340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":17543,"num_write_kb":304029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"65'4068","reported_seq":4186,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374459+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T05:20:36.374459+0000","last_peered":"2024-09-17T05:20:36.374459+0000","last_clean":"2024-09-17T05:20:36.374459+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T05:20:36.374459+0000","last_undegraded":"2024-09-17T05:20:36.374459+0000","last_fullsized":"2024-09-17T05:20:36.374459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:44.007814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21759,"num_write_kb":308395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"68'2551","reported_seq":2663,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402052+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T05:20:43.402052+0000","last_peered":"2024-09-17T05:20:43.402052+0000","last_clean":"2024-09-17T05:20:43.402052+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T05:20:43.402052+0000","last_undegraded":"2024-09-17T05:20:43.402052+0000","last_fullsized":"2024-09-17T05:20:43.402052+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2551,"log_dups_size":0,"ondisk_log_size":2551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13664,"num_write_kb":247232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"65'4369","reported_seq":4488,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.359366+0000","last_change":"2024-09-17T04:49:31.905683+0000","last_active":"2024-09-17T05:20:36.359366+0000","last_peered":"2024-09-17T05:20:36.359366+0000","last_clean":"2024-09-17T05:20:36.359366+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T05:20:36.359366+0000","last_undegraded":"2024-09-17T05:20:36.359366+0000","last_fullsized":"2024-09-17T05:20:36.359366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23171,"num_write_kb":318257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"68'3277","reported_seq":3388,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393579+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T05:20:43.393579+0000","last_peered":"2024-09-17T05:20:43.393579+0000","last_clean":"2024-09-17T05:20:43.393579+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T05:20:43.393579+0000","last_undegraded":"2024-09-17T05:20:43.393579+0000","last_fullsized":"2024-09-17T05:20:43.393579+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:19.387473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18007,"num_write_kb":264733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"65'5432","reported_seq":5550,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366356+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T05:20:36.366356+0000","last_peered":"2024-09-17T05:20:36.366356+0000","last_clean":"2024-09-17T05:20:36.366356+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T05:20:36.366356+0000","last_undegraded":"2024-09-17T05:20:36.366356+0000","last_fullsized":"2024-09-17T05:20:36.366356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29730,"num_write_kb":389609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"68'3135","reported_seq":3246,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.377491+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T05:20:43.377491+0000","last_peered":"2024-09-17T05:20:43.377491+0000","last_clean":"2024-09-17T05:20:43.377491+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T05:20:43.377491+0000","last_undegraded":"2024-09-17T05:20:43.377491+0000","last_fullsized":"2024-09-17T05:20:43.377491+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":16884,"num_write_kb":284681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"65'3941","reported_seq":4059,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.360280+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T05:20:36.360280+0000","last_peered":"2024-09-17T05:20:36.360280+0000","last_clean":"2024-09-17T05:20:36.360280+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T05:20:36.360280+0000","last_undegraded":"2024-09-17T05:20:36.360280+0000","last_fullsized":"2024-09-17T05:20:36.360280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21845,"num_write_kb":304681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"68'3913","reported_seq":4026,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.330331+0000","last_change":"2024-09-17T04:49:41.067207+0000","last_active":"2024-09-17T05:20:43.330331+0000","last_peered":"2024-09-17T05:20:43.330331+0000","last_clean":"2024-09-17T05:20:43.330331+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T05:20:43.330331+0000","last_undegraded":"2024-09-17T05:20:43.330331+0000","last_fullsized":"2024-09-17T05:20:43.330331+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3913,"log_dups_size":0,"ondisk_log_size":3913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:08.703635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":21530,"num_write_kb":364884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"65'3351","reported_seq":3475,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.347528+0000","last_change":"2024-09-17T04:50:12.469782+0000","last_active":"2024-09-17T05:20:36.347528+0000","last_peered":"2024-09-17T05:20:36.347528+0000","last_clean":"2024-09-17T05:20:36.347528+0000","last_became_active":"2024-09-17T04:50:12.469425+0000","last_became_peered":"2024-09-17T04:50:12.469425+0000","last_unstale":"2024-09-17T05:20:36.347528+0000","last_undegraded":"2024-09-17T05:20:36.347528+0000","last_fullsized":"2024-09-17T05:20:36.347528+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:04.098943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18172,"num_write_kb":265354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"68'1954","reported_seq":2066,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.346900+0000","last_change":"2024-09-17T04:49:41.046170+0000","last_active":"2024-09-17T05:20:43.346900+0000","last_peered":"2024-09-17T05:20:43.346900+0000","last_clean":"2024-09-17T05:20:43.346900+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T05:20:43.346900+0000","last_undegraded":"2024-09-17T05:20:43.346900+0000","last_fullsized":"2024-09-17T05:20:43.346900+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":1954,"log_dups_size":0,"ondisk_log_size":1954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:01.683826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":34,"num_write":10478,"num_write_kb":181241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"65'3661","reported_seq":3779,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.344111+0000","last_change":"2024-09-17T04:49:31.895330+0000","last_active":"2024-09-17T05:20:36.344111+0000","last_peered":"2024-09-17T05:20:36.344111+0000","last_clean":"2024-09-17T05:20:36.344111+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T05:20:36.344111+0000","last_undegraded":"2024-09-17T05:20:36.344111+0000","last_fullsized":"2024-09-17T05:20:36.344111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:49.337318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19087,"num_write_kb":273280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"68'2981","reported_seq":3093,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.203029+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T05:20:43.203029+0000","last_peered":"2024-09-17T05:20:43.203029+0000","last_clean":"2024-09-17T05:20:43.203029+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T05:20:43.203029+0000","last_undegraded":"2024-09-17T05:20:43.203029+0000","last_fullsized":"2024-09-17T05:20:43.203029+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:25.574682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":15573,"num_write_kb":271248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"65'3297","reported_seq":3415,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.340784+0000","last_change":"2024-09-17T04:49:31.889717+0000","last_active":"2024-09-17T05:20:36.340784+0000","last_peered":"2024-09-17T05:20:36.340784+0000","last_clean":"2024-09-17T05:20:36.340784+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T05:20:36.340784+0000","last_undegraded":"2024-09-17T05:20:36.340784+0000","last_fullsized":"2024-09-17T05:20:36.340784+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3297,"log_dups_size":0,"ondisk_log_size":3297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:14.385632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17851,"num_write_kb":243868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"68'2872","reported_seq":2983,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403678+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T05:20:44.403678+0000","last_peered":"2024-09-17T05:20:44.403678+0000","last_clean":"2024-09-17T05:20:44.403678+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T05:20:44.403678+0000","last_undegraded":"2024-09-17T05:20:44.403678+0000","last_fullsized":"2024-09-17T05:20:44.403678+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2872,"log_dups_size":0,"ondisk_log_size":2872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15784,"num_write_kb":259984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"65'2574","reported_seq":2692,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.556720+0000","last_change":"2024-09-17T04:49:31.906175+0000","last_active":"2024-09-17T05:20:36.556720+0000","last_peered":"2024-09-17T05:20:36.556720+0000","last_clean":"2024-09-17T05:20:36.556720+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T05:20:36.556720+0000","last_undegraded":"2024-09-17T05:20:36.556720+0000","last_fullsized":"2024-09-17T05:20:36.556720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":2574,"log_dups_size":0,"ondisk_log_size":2574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14007,"num_write_kb":203729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"68'2459","reported_seq":2572,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393891+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T05:20:43.393891+0000","last_peered":"2024-09-17T05:20:43.393891+0000","last_clean":"2024-09-17T05:20:43.393891+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T05:20:43.393891+0000","last_undegraded":"2024-09-17T05:20:43.393891+0000","last_fullsized":"2024-09-17T05:20:43.393891+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2459,"log_dups_size":0,"ondisk_log_size":2459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:44.270214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13572,"num_write_kb":222508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"65'3245","reported_seq":3363,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557014+0000","last_change":"2024-09-17T04:49:31.893111+0000","last_active":"2024-09-17T05:20:36.557014+0000","last_peered":"2024-09-17T05:20:36.557014+0000","last_clean":"2024-09-17T05:20:36.557014+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T05:20:36.557014+0000","last_undegraded":"2024-09-17T05:20:36.557014+0000","last_fullsized":"2024-09-17T05:20:36.557014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3245,"log_dups_size":0,"ondisk_log_size":3245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:05.777568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16903,"num_write_kb":256080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"68'3074","reported_seq":3184,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393584+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T05:20:43.393584+0000","last_peered":"2024-09-17T05:20:43.393584+0000","last_clean":"2024-09-17T05:20:43.393584+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T05:20:43.393584+0000","last_undegraded":"2024-09-17T05:20:43.393584+0000","last_fullsized":"2024-09-17T05:20:43.393584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16752,"num_write_kb":271957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"65'4387","reported_seq":4505,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.557710+0000","last_change":"2024-09-17T04:49:31.903838+0000","last_active":"2024-09-17T05:20:36.557710+0000","last_peered":"2024-09-17T05:20:36.557710+0000","last_clean":"2024-09-17T05:20:36.557710+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T05:20:36.557710+0000","last_undegraded":"2024-09-17T05:20:36.557710+0000","last_fullsized":"2024-09-17T05:20:36.557710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:53.466464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24001,"num_write_kb":317505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"68'2713","reported_seq":2824,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.417489+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T05:20:43.417489+0000","last_peered":"2024-09-17T05:20:43.417489+0000","last_clean":"2024-09-17T05:20:43.417489+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T05:20:43.417489+0000","last_undegraded":"2024-09-17T05:20:43.417489+0000","last_fullsized":"2024-09-17T05:20:43.417489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2713,"log_dups_size":0,"ondisk_log_size":2713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14521,"num_write_kb":262890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"65'4656","reported_seq":4776,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.347596+0000","last_change":"2024-09-17T04:49:31.896041+0000","last_active":"2024-09-17T05:20:36.347596+0000","last_peered":"2024-09-17T05:20:36.347596+0000","last_clean":"2024-09-17T05:20:36.347596+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T05:20:36.347596+0000","last_undegraded":"2024-09-17T05:20:36.347596+0000","last_fullsized":"2024-09-17T05:20:36.347596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:23.954154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25090,"num_write_kb":362900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"68'3588","reported_seq":3702,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.402501+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T05:20:44.402501+0000","last_peered":"2024-09-17T05:20:44.402501+0000","last_clean":"2024-09-17T05:20:44.402501+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T05:20:44.402501+0000","last_undegraded":"2024-09-17T05:20:44.402501+0000","last_fullsized":"2024-09-17T05:20:44.402501+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3588,"log_dups_size":0,"ondisk_log_size":3588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":19474,"num_write_kb":319707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"65'4149","reported_seq":4266,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.347526+0000","last_change":"2024-09-17T04:49:31.906354+0000","last_active":"2024-09-17T05:20:36.347526+0000","last_peered":"2024-09-17T05:20:36.347526+0000","last_clean":"2024-09-17T05:20:36.347526+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T05:20:36.347526+0000","last_undegraded":"2024-09-17T05:20:36.347526+0000","last_fullsized":"2024-09-17T05:20:36.347526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:30.593279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22453,"num_write_kb":298991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"68'3299","reported_seq":3415,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.422884+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T05:20:44.422884+0000","last_peered":"2024-09-17T05:20:44.422884+0000","last_clean":"2024-09-17T05:20:44.422884+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T05:20:44.422884+0000","last_undegraded":"2024-09-17T05:20:44.422884+0000","last_fullsized":"2024-09-17T05:20:44.422884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3299,"log_dups_size":0,"ondisk_log_size":3299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":17863,"num_write_kb":288721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"65'3910","reported_seq":4027,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.555930+0000","last_change":"2024-09-17T04:49:31.894717+0000","last_active":"2024-09-17T05:20:36.555930+0000","last_peered":"2024-09-17T05:20:36.555930+0000","last_clean":"2024-09-17T05:20:36.555930+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T05:20:36.555930+0000","last_undegraded":"2024-09-17T05:20:36.555930+0000","last_fullsized":"2024-09-17T05:20:36.555930+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:21:32.516081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21371,"num_write_kb":294687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"68'2856","reported_seq":2969,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.325969+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T05:20:43.325969+0000","last_peered":"2024-09-17T05:20:43.325969+0000","last_clean":"2024-09-17T05:20:43.325969+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T05:20:43.325969+0000","last_undegraded":"2024-09-17T05:20:43.325969+0000","last_fullsized":"2024-09-17T05:20:43.325969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15916,"num_write_kb":235941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"65'3148","reported_seq":3265,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.373270+0000","last_change":"2024-09-17T04:49:31.902225+0000","last_active":"2024-09-17T05:20:36.373270+0000","last_peered":"2024-09-17T05:20:36.373270+0000","last_clean":"2024-09-17T05:20:36.373270+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T05:20:36.373270+0000","last_undegraded":"2024-09-17T05:20:36.373270+0000","last_fullsized":"2024-09-17T05:20:36.373270+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:39.430447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16748,"num_write_kb":239238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"68'3516","reported_seq":3630,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.409318+0000","last_change":"2024-09-17T04:49:41.041201+0000","last_active":"2024-09-17T05:20:43.409318+0000","last_peered":"2024-09-17T05:20:43.409318+0000","last_clean":"2024-09-17T05:20:43.409318+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T05:20:43.409318+0000","last_undegraded":"2024-09-17T05:20:43.409318+0000","last_fullsized":"2024-09-17T05:20:43.409318+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:02.588786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18785,"num_write_kb":280186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"65'3294","reported_seq":3411,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.558447+0000","last_change":"2024-09-17T04:49:31.889766+0000","last_active":"2024-09-17T05:20:36.558447+0000","last_peered":"2024-09-17T05:20:36.558447+0000","last_clean":"2024-09-17T05:20:36.558447+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T05:20:36.558447+0000","last_undegraded":"2024-09-17T05:20:36.558447+0000","last_fullsized":"2024-09-17T05:20:36.558447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:59.444655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16934,"num_write_kb":260475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"68'2875","reported_seq":2985,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.359568+0000","last_change":"2024-09-17T04:49:41.061092+0000","last_active":"2024-09-17T05:20:43.359568+0000","last_peered":"2024-09-17T05:20:43.359568+0000","last_clean":"2024-09-17T05:20:43.359568+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T05:20:43.359568+0000","last_undegraded":"2024-09-17T05:20:43.359568+0000","last_fullsized":"2024-09-17T05:20:43.359568+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2875,"log_dups_size":0,"ondisk_log_size":2875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:52.714614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15268,"num_write_kb":241386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"65'3071","reported_seq":3192,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.370583+0000","last_change":"2024-09-17T04:49:31.899875+0000","last_active":"2024-09-17T05:20:36.370583+0000","last_peered":"2024-09-17T05:20:36.370583+0000","last_clean":"2024-09-17T05:20:36.370583+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T05:20:36.370583+0000","last_undegraded":"2024-09-17T05:20:36.370583+0000","last_fullsized":"2024-09-17T05:20:36.370583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3071,"log_dups_size":0,"ondisk_log_size":3071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:01.766784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16593,"num_write_kb":218157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"68'2617","reported_seq":2728,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.405858+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T05:20:43.405858+0000","last_peered":"2024-09-17T05:20:43.405858+0000","last_clean":"2024-09-17T05:20:43.405858+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T05:20:43.405858+0000","last_undegraded":"2024-09-17T05:20:43.405858+0000","last_fullsized":"2024-09-17T05:20:43.405858+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3,"num_write":13428,"num_write_kb":250535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"65'3866","reported_seq":3987,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352182+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T05:20:36.352182+0000","last_peered":"2024-09-17T05:20:36.352182+0000","last_clean":"2024-09-17T05:20:36.352182+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T05:20:36.352182+0000","last_undegraded":"2024-09-17T05:20:36.352182+0000","last_fullsized":"2024-09-17T05:20:36.352182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3866,"log_dups_size":0,"ondisk_log_size":3866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20832,"num_write_kb":272078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"68'3056","reported_seq":3165,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.326456+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T05:20:43.326456+0000","last_peered":"2024-09-17T05:20:43.326456+0000","last_clean":"2024-09-17T05:20:43.326456+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T05:20:43.326456+0000","last_undegraded":"2024-09-17T05:20:43.326456+0000","last_fullsized":"2024-09-17T05:20:43.326456+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3056,"log_dups_size":0,"ondisk_log_size":3056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16712,"num_write_kb":275987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"65'4485","reported_seq":4605,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.351407+0000","last_change":"2024-09-17T04:49:31.915119+0000","last_active":"2024-09-17T05:20:36.351407+0000","last_peered":"2024-09-17T05:20:36.351407+0000","last_clean":"2024-09-17T05:20:36.351407+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T05:20:36.351407+0000","last_undegraded":"2024-09-17T05:20:36.351407+0000","last_fullsized":"2024-09-17T05:20:36.351407+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:38:50.465566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23824,"num_write_kb":337279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"68'3229","reported_seq":3338,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.420569+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T05:20:44.420569+0000","last_peered":"2024-09-17T05:20:44.420569+0000","last_clean":"2024-09-17T05:20:44.420569+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T05:20:44.420569+0000","last_undegraded":"2024-09-17T05:20:44.420569+0000","last_fullsized":"2024-09-17T05:20:44.420569+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3229,"log_dups_size":0,"ondisk_log_size":3229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17855,"num_write_kb":268347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"65'3523","reported_seq":3644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.559155+0000","last_change":"2024-09-17T04:49:31.896086+0000","last_active":"2024-09-17T05:20:36.559155+0000","last_peered":"2024-09-17T05:20:36.559155+0000","last_clean":"2024-09-17T05:20:36.559155+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T05:20:36.559155+0000","last_undegraded":"2024-09-17T05:20:36.559155+0000","last_fullsized":"2024-09-17T05:20:36.559155+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3523,"log_dups_size":0,"ondisk_log_size":3523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:08.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19890,"num_write_kb":260121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":321,"num_write":1077532,"num_write_kb":17428719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":198824,"ondisk_log_size":198824,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":18310618,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1391,"num_read_kb":207657,"num_write":4849,"num_write_kb":19832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36683776,"data_stored":36621236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4439,"ondisk_log_size":4439,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1352682,"num_write_kb":18536494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250516,"ondisk_log_size":250516,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":18963064,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1535,"num_read_kb":230833,"num_write":4911,"num_write_kb":20583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37986304,"data_stored":37926128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4504,"ondisk_log_size":4504,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509824,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533064,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":532094,"kb_avail":93838776,"statfs":{"total":96636764160,"available":96090906624,"internally_reserved":0,"allocated":974848,"data_stored":552256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":544864714},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509824,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762124,"kb_used_data":17916,"kb_used_omap":1,"kb_used_meta":744190,"kb_avail":93609716,"statfs":{"total":96636764160,"available":95856349184,"internally_reserved":0,"allocated":18345984,"data_stored":17920368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":762051015},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509824,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756036,"kb_used_data":17268,"kb_used_omap":1,"kb_used_meta":738750,"kb_avail":93615804,"statfs":{"total":96636764160,"available":95862583296,"internally_reserved":0,"allocated":17682432,"data_stored":17246718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756480453},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509824,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755412,"kb_used_data":15300,"kb_used_omap":1,"kb_used_meta":740094,"kb_avail":93616428,"statfs":{"total":96636764160,"available":95863222272,"internally_reserved":0,"allocated":15667200,"data_stored":15236011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":757856711},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509824,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877392,"kb_used_data":14656,"kb_used_omap":1,"kb_used_meta":862718,"kb_avail":93494448,"statfs":{"total":96636764160,"available":95738314752,"internally_reserved":0,"allocated":15007744,"data_stored":14583014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":883423685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509825,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630324,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":628862,"kb_avail":93741516,"statfs":{"total":96636764160,"available":95991312384,"internally_reserved":0,"allocated":1478656,"data_stored":1057972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":643955141},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509825,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":819832,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":816894,"kb_avail":93552008,"statfs":{"total":96636764160,"available":95797256192,"internally_reserved":0,"allocated":2990080,"data_stored":2567342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":836499909},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542528,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694200,"kb_used_data":10472,"kb_used_omap":1,"kb_used_meta":683710,"kb_avail":93677640,"statfs":{"total":96636764160,"available":95925903360,"internally_reserved":0,"allocated":10723328,"data_stored":10304131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":700119494},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2099788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10493952,"data_stored":10488922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4279042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":84770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5726270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4279625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5652480,"data_stored":5642363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:21:05.276 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.276 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.276 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.277 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.277 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.277 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.277 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.278 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.278 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.278 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.278 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.278 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.278 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.278 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.278 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.279 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.279 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.279 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.279 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.279 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.279 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.279 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.279 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.279 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.280 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.280 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.280 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.280 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.280 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.280 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.280 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.280 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.280 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.281 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.281 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.281 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.281 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.281 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.281 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.281 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.281 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.281 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.282 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.282 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.282 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.282 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.282 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.282 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.282 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.282 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.282 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.283 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.283 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.283 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.283 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.283 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.283 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.283 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.283 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.283 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.284 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.284 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.284 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-17T04:49:21.657891+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=21, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.284 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.284 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.284 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.284 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.284 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.284 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.285 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.285 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.285 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.285 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.285 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.285 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.285 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.285 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.286 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.286 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.286 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.286 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.286 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.286 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.286 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.286 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.286 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.287 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.287 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.287 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.287 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.287 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.287 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.287 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.287 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.287 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.287 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.288 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.288 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.288 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.288 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.288 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.288 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.288 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.288 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.289 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.289 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.289 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.289 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.289 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.289 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.289 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.289 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.289 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.290 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.290 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.290 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.290 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.290 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.290 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.290 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.291 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.291 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.291 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.291 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.291 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-09-17T04:49:29.826523+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=29, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.291 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.291 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.291 INFO:tasks.ceph:pgid 5.26 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.291 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.292 INFO:tasks.ceph:pgid 5.27 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.292 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.292 INFO:tasks.ceph:pgid 5.24 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.292 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.292 INFO:tasks.ceph:pgid 5.25 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.292 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.293 INFO:tasks.ceph:pgid 5.22 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.293 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.293 INFO:tasks.ceph:pgid 5.23 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.293 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.293 INFO:tasks.ceph:pgid 5.20 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.293 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.293 INFO:tasks.ceph:pgid 5.21 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.293 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.293 INFO:tasks.ceph:pgid 5.2e last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.294 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.294 INFO:tasks.ceph:pgid 5.2f last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.294 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.294 INFO:tasks.ceph:pgid 5.2c last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.294 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.294 INFO:tasks.ceph:pgid 5.2d last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.294 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.294 INFO:tasks.ceph:pgid 5.2a last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.294 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.295 INFO:tasks.ceph:pgid 5.2b last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.295 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.295 INFO:tasks.ceph:pgid 5.28 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.295 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.295 INFO:tasks.ceph:pgid 5.29 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.295 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.295 INFO:tasks.ceph:pgid 5.36 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.295 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.295 INFO:tasks.ceph:pgid 5.37 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.296 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.296 INFO:tasks.ceph:pgid 5.34 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.296 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.296 INFO:tasks.ceph:pgid 5.35 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.296 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.296 INFO:tasks.ceph:pgid 5.32 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.296 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.296 INFO:tasks.ceph:pgid 5.33 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.297 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.297 INFO:tasks.ceph:pgid 5.30 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.297 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.297 INFO:tasks.ceph:pgid 5.31 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.297 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.297 INFO:tasks.ceph:pgid 5.3e last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.297 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.297 INFO:tasks.ceph:pgid 5.3f last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.297 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.298 INFO:tasks.ceph:pgid 5.3c last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.298 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.298 INFO:tasks.ceph:pgid 5.3d last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.298 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.298 INFO:tasks.ceph:pgid 5.3a last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.298 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.298 INFO:tasks.ceph:pgid 5.3b last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.298 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.298 INFO:tasks.ceph:pgid 5.38 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.299 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.299 INFO:tasks.ceph:pgid 5.39 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.299 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:05.299 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T05:21:07.604 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:21:07.607 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:21:25.300 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:21:25.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.472+0000 7effea011640 1 -- 172.21.15.150:0/575427790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe40750f0 msgr2=0x7effe40c4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:25.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.472+0000 7effea011640 1 --2- 172.21.15.150:0/575427790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe40750f0 0x7effe40c4b30 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7effd8009840 tx=0x7effd801c820 comp rx=0 tx=0).stop 2024-09-17T05:21:25.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.472+0000 7effea011640 1 -- 172.21.15.150:0/575427790 shutdown_connections 2024-09-17T05:21:25.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.472+0000 7effea011640 1 --2- 172.21.15.150:0/575427790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effe40c7e20 0x7effe41553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:25.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.472+0000 7effea011640 1 --2- 172.21.15.150:0/575427790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe40750f0 0x7effe40c4b30 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:25.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.472+0000 7effea011640 1 -- 172.21.15.150:0/575427790 >> 172.21.15.150:0/575427790 conn(0x7effe40aaa80 msgr2=0x7effe40aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:25.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.472+0000 7effea011640 1 -- 172.21.15.150:0/575427790 shutdown_connections 2024-09-17T05:21:25.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.472+0000 7effea011640 1 -- 172.21.15.150:0/575427790 wait complete. 2024-09-17T05:21:25.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effea011640 1 Processor -- start 2024-09-17T05:21:25.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effea011640 1 -- start start 2024-09-17T05:21:25.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effea011640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe4071ca0 0x7effe40ad6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:25.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effea011640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effe40750f0 0x7effe41f0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:25.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effea011640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effe40c7e20 0x7effe414bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:25.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effea011640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7effe4074b30 con 0x7effe4071ca0 2024-09-17T05:21:25.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effea011640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7effe41558f0 con 0x7effe40c7e20 2024-09-17T05:21:25.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effea011640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7effe4074cb0 con 0x7effe40750f0 2024-09-17T05:21:25.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effe37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe4071ca0 0x7effe40ad6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:25.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effe37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe4071ca0 0x7effe40ad6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45256/0 (socket says 172.21.15.150:45256) 2024-09-17T05:21:25.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effe37fe640 1 -- 172.21.15.150:0/3999641096 learned_addr learned my addr 172.21.15.150:0/3999641096 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:25.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effe2ffd640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effe40750f0 0x7effe41f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:25.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effe37fe640 1 -- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effe40750f0 msgr2=0x7effe41f0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:25.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effe3fff640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effe40c7e20 0x7effe414bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:25.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effe37fe640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effe40750f0 0x7effe41f0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:25.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effe37fe640 1 -- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effe40c7e20 msgr2=0x7effe414bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:25.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effe37fe640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effe40c7e20 0x7effe414bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:25.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effe37fe640 1 -- 172.21.15.150:0/3999641096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe40c3200 con 0x7effe4071ca0 2024-09-17T05:21:25.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effe3fff640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effe40c7e20 0x7effe414bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:25.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effe37fe640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe4071ca0 0x7effe40ad6d0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7effd8000c10 tx=0x7effd8005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:25.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effcbfff640 1 -- 172.21.15.150:0/3999641096 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd80b2040 con 0x7effe4071ca0 2024-09-17T05:21:25.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effcbfff640 1 -- 172.21.15.150:0/3999641096 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7effd8004240 con 0x7effe4071ca0 2024-09-17T05:21:25.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effea011640 1 -- 172.21.15.150:0/3999641096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effe414d250 con 0x7effe4071ca0 2024-09-17T05:21:25.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effcbfff640 1 -- 172.21.15.150:0/3999641096 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd8005020 con 0x7effe4071ca0 2024-09-17T05:21:25.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effea011640 1 -- 172.21.15.150:0/3999641096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effe41eff30 con 0x7effe4071ca0 2024-09-17T05:21:25.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.476+0000 7effea011640 1 -- 172.21.15.150:0/3999641096 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effe406a770 con 0x7effe4071ca0 2024-09-17T05:21:25.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.480+0000 7effcbfff640 1 -- 172.21.15.150:0/3999641096 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7effd80051c0 con 0x7effe4071ca0 2024-09-17T05:21:25.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.480+0000 7effcbfff640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7effd0073200 0x7effd0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:25.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.480+0000 7effcbfff640 1 -- 172.21.15.150:0/3999641096 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7effd812ef30 con 0x7effe4071ca0 2024-09-17T05:21:25.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.480+0000 7effe2ffd640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7effd0073200 0x7effd0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:25.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.480+0000 7effe2ffd640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7effd0073200 0x7effd0075710 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7effd40058d0 tx=0x7effd4005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:25.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.484+0000 7effcbfff640 1 -- 172.21.15.150:0/3999641096 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7effd8016410 con 0x7effe4071ca0 2024-09-17T05:21:25.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.656+0000 7effea011640 1 -- 172.21.15.150:0/3999641096 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7effe414c7a0 con 0x7effd0073200 2024-09-17T05:21:25.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.680+0000 7effcbfff640 1 -- 172.21.15.150:0/3999641096 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+494142 (secure 0 0 0) 0x7effd400e940 con 0x7effd0073200 2024-09-17T05:21:25.682 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:21:25.687 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T05:21:25.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.684+0000 7effea011640 1 -- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7effd0073200 msgr2=0x7effd0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:25.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.684+0000 7effea011640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7effd0073200 0x7effd0075710 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7effd40058d0 tx=0x7effd4005860 comp rx=0 tx=0).stop 2024-09-17T05:21:25.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.684+0000 7effea011640 1 -- 172.21.15.150:0/3999641096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe4071ca0 msgr2=0x7effe40ad6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:25.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.684+0000 7effea011640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe4071ca0 0x7effe40ad6d0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7effd8000c10 tx=0x7effd8005db0 comp rx=0 tx=0).stop 2024-09-17T05:21:25.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.684+0000 7effea011640 1 -- 172.21.15.150:0/3999641096 shutdown_connections 2024-09-17T05:21:25.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.684+0000 7effea011640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7effd0073200 0x7effd0075710 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:25.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.684+0000 7effea011640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effe40c7e20 0x7effe414bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:25.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.684+0000 7effea011640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7effe40750f0 0x7effe41f0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:25.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.684+0000 7effea011640 1 --2- 172.21.15.150:0/3999641096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe4071ca0 0x7effe40ad6d0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:25.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.684+0000 7effea011640 1 -- 172.21.15.150:0/3999641096 >> 172.21.15.150:0/3999641096 conn(0x7effe40aaa80 msgr2=0x7effe415e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:25.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.684+0000 7effea011640 1 -- 172.21.15.150:0/3999641096 shutdown_connections 2024-09-17T05:21:25.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:25.684+0000 7effea011640 1 -- 172.21.15.150:0/3999641096 wait complete. 2024-09-17T05:21:25.705 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":1023,"stamp":"2024-09-17T05:21:23.896884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37864050,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3107,"num_read_kb":438866,"num_write":2440075,"num_write_kb":36007256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":458392,"ondisk_log_size":458392,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5836704,"kb_used_data":80928,"kb_used_omap":12,"kb_used_meta":5755635,"kb_avail":749138016,"statfs":{"total":773094113280,"available":767117328384,"internally_reserved":0,"allocated":82870272,"data_stored":79467812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":5893770802},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006872"},"pg_stats":[{"pgid":"5.18","version":"68'3479","reported_seq":3600,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.042279+0000","last_change":"2024-09-17T05:21:10.042098+0000","last_active":"2024-09-17T05:21:10.042279+0000","last_peered":"2024-09-17T05:21:10.042279+0000","last_clean":"2024-09-17T05:21:10.042279+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T05:21:10.042279+0000","last_undegraded":"2024-09-17T05:21:10.042279+0000","last_fullsized":"2024-09-17T05:21:10.042279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3479","last_scrub_stamp":"2024-09-17T05:21:10.041849+0000","last_deep_scrub":"68'3479","last_deep_scrub_stamp":"2024-09-17T05:21:10.041849+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.041849+0000","objects_scrubbed":0,"log_size":3479,"log_dups_size":0,"ondisk_log_size":3479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:08:08.320683+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":12,"num_write":18686,"num_write_kb":287786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"68'1004","reported_seq":1186,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.958535+0000","last_change":"2024-09-17T05:21:07.958535+0000","last_active":"2024-09-17T05:21:07.958535+0000","last_peered":"2024-09-17T05:21:07.958535+0000","last_clean":"2024-09-17T05:21:07.958535+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T05:21:07.958535+0000","last_undegraded":"2024-09-17T05:21:07.958535+0000","last_fullsized":"2024-09-17T05:21:07.958535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1004","last_scrub_stamp":"2024-09-17T05:21:07.958302+0000","last_deep_scrub":"68'1004","last_deep_scrub_stamp":"2024-09-17T05:21:07.958302+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.958302+0000","objects_scrubbed":3,"log_size":1004,"log_dups_size":0,"ondisk_log_size":1004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:58.171045+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":20774,"num_write":978,"num_write_kb":4679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'4064","reported_seq":4192,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.608906+0000","last_change":"2024-09-17T05:21:11.608906+0000","last_active":"2024-09-17T05:21:11.608906+0000","last_peered":"2024-09-17T05:21:11.608906+0000","last_clean":"2024-09-17T05:21:11.608906+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T05:21:11.608906+0000","last_undegraded":"2024-09-17T05:21:11.608906+0000","last_fullsized":"2024-09-17T05:21:11.608906+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4064","last_scrub_stamp":"2024-09-17T05:21:11.608624+0000","last_deep_scrub":"65'4064","last_deep_scrub_stamp":"2024-09-17T05:21:11.608624+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.608624+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:19.389394+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22217,"num_write_kb":281701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"68'3","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.159597+0000","last_change":"2024-09-17T05:21:13.159399+0000","last_active":"2024-09-17T05:21:13.159597+0000","last_peered":"2024-09-17T05:21:13.159597+0000","last_clean":"2024-09-17T05:21:13.159597+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T05:21:13.159597+0000","last_undegraded":"2024-09-17T05:21:13.159597+0000","last_fullsized":"2024-09-17T05:21:13.159597+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-09-17T05:21:13.159185+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2024-09-17T05:21:13.159185+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.159185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:22.788148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"68'3721","reported_seq":3834,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.399621+0000","last_change":"2024-09-17T04:49:41.058430+0000","last_active":"2024-09-17T05:20:44.399621+0000","last_peered":"2024-09-17T05:20:44.399621+0000","last_clean":"2024-09-17T05:20:44.399621+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T05:20:44.399621+0000","last_undegraded":"2024-09-17T05:20:44.399621+0000","last_fullsized":"2024-09-17T05:20:44.399621+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3721,"log_dups_size":0,"ondisk_log_size":3721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":22,"num_write":19557,"num_write_kb":316044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"67'26","reported_seq":156,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.653104+0000","last_change":"2024-09-17T05:21:06.653104+0000","last_active":"2024-09-17T05:21:06.653104+0000","last_peered":"2024-09-17T05:21:06.653104+0000","last_clean":"2024-09-17T05:21:06.653104+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T05:21:06.653104+0000","last_undegraded":"2024-09-17T05:21:06.653104+0000","last_fullsized":"2024-09-17T05:21:06.653104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'26","last_scrub_stamp":"2024-09-17T05:21:06.652875+0000","last_deep_scrub":"67'26","last_deep_scrub_stamp":"2024-09-17T05:21:06.652875+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.652875+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:45.890071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4998","reported_seq":5126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.674317+0000","last_change":"2024-09-17T05:21:13.674317+0000","last_active":"2024-09-17T05:21:13.674317+0000","last_peered":"2024-09-17T05:21:13.674317+0000","last_clean":"2024-09-17T05:21:13.674317+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T05:21:13.674317+0000","last_undegraded":"2024-09-17T05:21:13.674317+0000","last_fullsized":"2024-09-17T05:21:13.674317+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4998","last_scrub_stamp":"2024-09-17T05:21:13.674048+0000","last_deep_scrub":"65'4998","last_deep_scrub_stamp":"2024-09-17T05:21:13.674048+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.674048+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:15:59.587253+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27336,"num_write_kb":388205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'2","reported_seq":122,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.782513+0000","last_change":"2024-09-17T05:21:17.782513+0000","last_active":"2024-09-17T05:21:17.782513+0000","last_peered":"2024-09-17T05:21:17.782513+0000","last_clean":"2024-09-17T05:21:17.782513+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T05:21:17.782513+0000","last_undegraded":"2024-09-17T05:21:17.782513+0000","last_fullsized":"2024-09-17T05:21:17.782513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T05:21:17.782306+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2024-09-17T05:21:17.782306+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.782306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:51.871351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"68'2765","reported_seq":2884,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.043838+0000","last_change":"2024-09-17T05:21:12.043838+0000","last_active":"2024-09-17T05:21:12.043838+0000","last_peered":"2024-09-17T05:21:12.043838+0000","last_clean":"2024-09-17T05:21:12.043838+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T05:21:12.043838+0000","last_undegraded":"2024-09-17T05:21:12.043838+0000","last_fullsized":"2024-09-17T05:21:12.043838+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2765","last_scrub_stamp":"2024-09-17T05:21:12.043445+0000","last_deep_scrub":"68'2765","last_deep_scrub_stamp":"2024-09-17T05:21:12.043445+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.043445+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:25:10.594780+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":14571,"num_write_kb":258996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"68'47","reported_seq":216,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.713427+0000","last_change":"2024-09-17T05:21:06.713427+0000","last_active":"2024-09-17T05:21:06.713427+0000","last_peered":"2024-09-17T05:21:06.713427+0000","last_clean":"2024-09-17T05:21:06.713427+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T05:21:06.713427+0000","last_undegraded":"2024-09-17T05:21:06.713427+0000","last_fullsized":"2024-09-17T05:21:06.713427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'47","last_scrub_stamp":"2024-09-17T05:21:06.713179+0000","last_deep_scrub":"68'47","last_deep_scrub_stamp":"2024-09-17T05:21:06.713179+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.713179+0000","objects_scrubbed":1,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:16.060967+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"65'3140","reported_seq":3267,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.064090+0000","last_change":"2024-09-17T05:21:09.064090+0000","last_active":"2024-09-17T05:21:09.064090+0000","last_peered":"2024-09-17T05:21:09.064090+0000","last_clean":"2024-09-17T05:21:09.064090+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T05:21:09.064090+0000","last_undegraded":"2024-09-17T05:21:09.064090+0000","last_fullsized":"2024-09-17T05:21:09.064090+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3140","last_scrub_stamp":"2024-09-17T05:21:09.063861+0000","last_deep_scrub":"65'3140","last_deep_scrub_stamp":"2024-09-17T05:21:09.063861+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.063861+0000","objects_scrubbed":0,"log_size":3140,"log_dups_size":0,"ondisk_log_size":3140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:54.276035+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16962,"num_write_kb":252282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"68'752","reported_seq":897,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.772498+0000","last_change":"2024-09-17T05:21:14.772498+0000","last_active":"2024-09-17T05:21:14.772498+0000","last_peered":"2024-09-17T05:21:14.772498+0000","last_clean":"2024-09-17T05:21:14.772498+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T05:21:14.772498+0000","last_undegraded":"2024-09-17T05:21:14.772498+0000","last_fullsized":"2024-09-17T05:21:14.772498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'752","last_scrub_stamp":"2024-09-17T05:21:14.772291+0000","last_deep_scrub":"68'752","last_deep_scrub_stamp":"2024-09-17T05:21:14.772291+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.772291+0000","objects_scrubbed":1,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:43.662572+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":75812,"num_write":739,"num_write_kb":4324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"68'2700","reported_seq":2818,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.800307+0000","last_change":"2024-09-17T05:21:16.800307+0000","last_active":"2024-09-17T05:21:16.800307+0000","last_peered":"2024-09-17T05:21:16.800307+0000","last_clean":"2024-09-17T05:21:16.800307+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T05:21:16.800307+0000","last_undegraded":"2024-09-17T05:21:16.800307+0000","last_fullsized":"2024-09-17T05:21:16.800307+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2700","last_scrub_stamp":"2024-09-17T05:21:16.800075+0000","last_deep_scrub":"68'2700","last_deep_scrub_stamp":"2024-09-17T05:21:16.800075+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.800075+0000","objects_scrubbed":0,"log_size":2700,"log_dups_size":0,"ondisk_log_size":2700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:29.326983+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":5,"num_write":13438,"num_write_kb":225755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"68'6","reported_seq":175,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:04.693011+0000","last_change":"2024-09-17T05:21:04.693011+0000","last_active":"2024-09-17T05:21:04.693011+0000","last_peered":"2024-09-17T05:21:04.693011+0000","last_clean":"2024-09-17T05:21:04.693011+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T05:21:04.693011+0000","last_undegraded":"2024-09-17T05:21:04.693011+0000","last_fullsized":"2024-09-17T05:21:04.693011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-17T05:21:04.692899+0000","last_deep_scrub":"68'6","last_deep_scrub_stamp":"2024-09-17T05:21:04.692899+0000","last_clean_scrub_stamp":"2024-09-17T05:21:04.692899+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:43.930095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"65'4481","reported_seq":4606,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.540048+0000","last_change":"2024-09-17T05:21:15.540048+0000","last_active":"2024-09-17T05:21:15.540048+0000","last_peered":"2024-09-17T05:21:15.540048+0000","last_clean":"2024-09-17T05:21:15.540048+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T05:21:15.540048+0000","last_undegraded":"2024-09-17T05:21:15.540048+0000","last_fullsized":"2024-09-17T05:21:15.540048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4481","last_scrub_stamp":"2024-09-17T05:21:15.539814+0000","last_deep_scrub":"65'4481","last_deep_scrub_stamp":"2024-09-17T05:21:15.539814+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.539814+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:36:00.335129+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24963,"num_write_kb":346034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"68'143","reported_seq":264,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.736612+0000","last_change":"2024-09-17T05:21:15.736612+0000","last_active":"2024-09-17T05:21:15.736612+0000","last_peered":"2024-09-17T05:21:15.736612+0000","last_clean":"2024-09-17T05:21:15.736612+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T05:21:15.736612+0000","last_undegraded":"2024-09-17T05:21:15.736612+0000","last_fullsized":"2024-09-17T05:21:15.736612+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'143","last_scrub_stamp":"2024-09-17T05:21:15.736400+0000","last_deep_scrub":"68'143","last_deep_scrub_stamp":"2024-09-17T05:21:15.736400+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.736400+0000","objects_scrubbed":1,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:03.782112+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1447387,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":383,"num_write":117,"num_write_kb":1479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"68'2942","reported_seq":3055,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.417965+0000","last_change":"2024-09-17T04:49:41.036527+0000","last_active":"2024-09-17T05:20:44.417965+0000","last_peered":"2024-09-17T05:20:44.417965+0000","last_clean":"2024-09-17T05:20:44.417965+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T05:20:44.417965+0000","last_undegraded":"2024-09-17T05:20:44.417965+0000","last_fullsized":"2024-09-17T05:20:44.417965+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":16254,"num_write_kb":278162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"65'3968","reported_seq":4095,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.553424+0000","last_change":"2024-09-17T05:21:19.553424+0000","last_active":"2024-09-17T05:21:19.553424+0000","last_peered":"2024-09-17T05:21:19.553424+0000","last_clean":"2024-09-17T05:21:19.553424+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T05:21:19.553424+0000","last_undegraded":"2024-09-17T05:21:19.553424+0000","last_fullsized":"2024-09-17T05:21:19.553424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3968","last_scrub_stamp":"2024-09-17T05:21:19.553025+0000","last_deep_scrub":"65'3968","last_deep_scrub_stamp":"2024-09-17T05:21:19.553025+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.553025+0000","objects_scrubbed":0,"log_size":3968,"log_dups_size":0,"ondisk_log_size":3968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:51.875435+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21939,"num_write_kb":297018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"68'1469","reported_seq":1647,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.571989+0000","last_change":"2024-09-17T05:21:08.571989+0000","last_active":"2024-09-17T05:21:08.571989+0000","last_peered":"2024-09-17T05:21:08.571989+0000","last_clean":"2024-09-17T05:21:08.571989+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T05:21:08.571989+0000","last_undegraded":"2024-09-17T05:21:08.571989+0000","last_fullsized":"2024-09-17T05:21:08.571989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1469","last_scrub_stamp":"2024-09-17T05:21:08.571729+0000","last_deep_scrub":"68'1469","last_deep_scrub_stamp":"2024-09-17T05:21:08.571729+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.571729+0000","objects_scrubbed":2,"log_size":1469,"log_dups_size":0,"ondisk_log_size":1469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:38.358519+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":43333,"num_write":1444,"num_write_kb":4521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'6","reported_seq":139,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.386136+0000","last_change":"2024-09-17T04:49:39.990881+0000","last_active":"2024-09-17T05:20:44.386136+0000","last_peered":"2024-09-17T05:20:44.386136+0000","last_clean":"2024-09-17T05:20:44.386136+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T05:20:44.386136+0000","last_undegraded":"2024-09-17T05:20:44.386136+0000","last_fullsized":"2024-09-17T05:20:44.386136+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'3236","reported_seq":3358,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.262331+0000","last_change":"2024-09-17T05:21:17.262331+0000","last_active":"2024-09-17T05:21:17.262331+0000","last_peered":"2024-09-17T05:21:17.262331+0000","last_clean":"2024-09-17T05:21:17.262331+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T05:21:17.262331+0000","last_undegraded":"2024-09-17T05:21:17.262331+0000","last_fullsized":"2024-09-17T05:21:17.262331+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3236","last_scrub_stamp":"2024-09-17T05:21:17.262125+0000","last_deep_scrub":"68'3236","last_deep_scrub_stamp":"2024-09-17T05:21:17.262125+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.262125+0000","objects_scrubbed":0,"log_size":3236,"log_dups_size":0,"ondisk_log_size":3236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:07:24.547946+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":17542,"num_write_kb":271055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"65'3303","reported_seq":3430,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.208468+0000","last_change":"2024-09-17T05:21:09.208468+0000","last_active":"2024-09-17T05:21:09.208468+0000","last_peered":"2024-09-17T05:21:09.208468+0000","last_clean":"2024-09-17T05:21:09.208468+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T05:21:09.208468+0000","last_undegraded":"2024-09-17T05:21:09.208468+0000","last_fullsized":"2024-09-17T05:21:09.208468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3303","last_scrub_stamp":"2024-09-17T05:21:09.208191+0000","last_deep_scrub":"65'3303","last_deep_scrub_stamp":"2024-09-17T05:21:09.208191+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.208191+0000","objects_scrubbed":0,"log_size":3303,"log_dups_size":0,"ondisk_log_size":3303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.238401+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17463,"num_write_kb":238014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"65'3","reported_seq":132,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.246039+0000","last_change":"2024-09-17T05:21:06.246039+0000","last_active":"2024-09-17T05:21:06.246039+0000","last_peered":"2024-09-17T05:21:06.246039+0000","last_clean":"2024-09-17T05:21:06.246039+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T05:21:06.246039+0000","last_undegraded":"2024-09-17T05:21:06.246039+0000","last_fullsized":"2024-09-17T05:21:06.246039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-09-17T05:21:06.245820+0000","last_deep_scrub":"65'3","last_deep_scrub_stamp":"2024-09-17T05:21:06.245820+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.245820+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:50.693155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"68'5","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.681526+0000","last_change":"2024-09-17T05:21:16.681526+0000","last_active":"2024-09-17T05:21:16.681526+0000","last_peered":"2024-09-17T05:21:16.681526+0000","last_clean":"2024-09-17T05:21:16.681526+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T05:21:16.681526+0000","last_undegraded":"2024-09-17T05:21:16.681526+0000","last_fullsized":"2024-09-17T05:21:16.681526+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:16.681321+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:16.681321+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.681321+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:17:49.939128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"68'2691","reported_seq":2803,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.415310+0000","last_change":"2024-09-17T04:49:41.058889+0000","last_active":"2024-09-17T05:20:43.415310+0000","last_peered":"2024-09-17T05:20:43.415310+0000","last_clean":"2024-09-17T05:20:43.415310+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T05:20:43.415310+0000","last_undegraded":"2024-09-17T05:20:43.415310+0000","last_fullsized":"2024-09-17T05:20:43.415310+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":14365,"num_write_kb":225564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"65'3640","reported_seq":3767,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.847235+0000","last_change":"2024-09-17T05:21:11.847235+0000","last_active":"2024-09-17T05:21:11.847235+0000","last_peered":"2024-09-17T05:21:11.847235+0000","last_clean":"2024-09-17T05:21:11.847235+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T05:21:11.847235+0000","last_undegraded":"2024-09-17T05:21:11.847235+0000","last_fullsized":"2024-09-17T05:21:11.847235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3640","last_scrub_stamp":"2024-09-17T05:21:11.847013+0000","last_deep_scrub":"65'3640","last_deep_scrub_stamp":"2024-09-17T05:21:11.847013+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.847013+0000","objects_scrubbed":0,"log_size":3640,"log_dups_size":0,"ondisk_log_size":3640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:40.737295+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20183,"num_write_kb":277678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"65'2","reported_seq":131,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:04.912716+0000","last_change":"2024-09-17T05:21:04.912716+0000","last_active":"2024-09-17T05:21:04.912716+0000","last_peered":"2024-09-17T05:21:04.912716+0000","last_clean":"2024-09-17T05:21:04.912716+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T05:21:04.912716+0000","last_undegraded":"2024-09-17T05:21:04.912716+0000","last_fullsized":"2024-09-17T05:21:04.912716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-17T05:21:04.912451+0000","last_deep_scrub":"65'2","last_deep_scrub_stamp":"2024-09-17T05:21:04.912451+0000","last_clean_scrub_stamp":"2024-09-17T05:21:04.912451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:39.929521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"68'5","reported_seq":146,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.167300+0000","last_change":"2024-09-17T05:21:10.167098+0000","last_active":"2024-09-17T05:21:10.167300+0000","last_peered":"2024-09-17T05:21:10.167300+0000","last_clean":"2024-09-17T05:21:10.167300+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T05:21:10.167300+0000","last_undegraded":"2024-09-17T05:21:10.167300+0000","last_fullsized":"2024-09-17T05:21:10.167300+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:10.166704+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:10.166704+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.166704+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:21:51.895257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"68'3943","reported_seq":4062,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.108032+0000","last_change":"2024-09-17T05:21:07.107840+0000","last_active":"2024-09-17T05:21:07.108032+0000","last_peered":"2024-09-17T05:21:07.108032+0000","last_clean":"2024-09-17T05:21:07.108032+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T05:21:07.108032+0000","last_undegraded":"2024-09-17T05:21:07.108032+0000","last_fullsized":"2024-09-17T05:21:07.108032+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3943","last_scrub_stamp":"2024-09-17T05:21:07.107602+0000","last_deep_scrub":"68'3943","last_deep_scrub_stamp":"2024-09-17T05:21:07.107602+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.107602+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:04:17.650985+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21088,"num_write_kb":359062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"65'3699","reported_seq":3826,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.489668+0000","last_change":"2024-09-17T05:21:13.489668+0000","last_active":"2024-09-17T05:21:13.489668+0000","last_peered":"2024-09-17T05:21:13.489668+0000","last_clean":"2024-09-17T05:21:13.489668+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T05:21:13.489668+0000","last_undegraded":"2024-09-17T05:21:13.489668+0000","last_fullsized":"2024-09-17T05:21:13.489668+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3699","last_scrub_stamp":"2024-09-17T05:21:13.489313+0000","last_deep_scrub":"65'3699","last_deep_scrub_stamp":"2024-09-17T05:21:13.489313+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.489313+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:16.073785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19661,"num_write_kb":271466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"65'5","reported_seq":134,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:01.675505+0000","last_change":"2024-09-17T05:21:01.675505+0000","last_active":"2024-09-17T05:21:01.675505+0000","last_peered":"2024-09-17T05:21:01.675505+0000","last_clean":"2024-09-17T05:21:01.675505+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T05:21:01.675505+0000","last_undegraded":"2024-09-17T05:21:01.675505+0000","last_fullsized":"2024-09-17T05:21:01.675505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-17T05:21:01.675372+0000","last_deep_scrub":"65'5","last_deep_scrub_stamp":"2024-09-17T05:21:01.675372+0000","last_clean_scrub_stamp":"2024-09-17T05:21:01.675372+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:14.921299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"68'8","reported_seq":169,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.737846+0000","last_change":"2024-09-17T05:21:12.737691+0000","last_active":"2024-09-17T05:21:12.737846+0000","last_peered":"2024-09-17T05:21:12.737846+0000","last_clean":"2024-09-17T05:21:12.737846+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T05:21:12.737846+0000","last_undegraded":"2024-09-17T05:21:12.737846+0000","last_fullsized":"2024-09-17T05:21:12.737846+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-17T05:21:12.737480+0000","last_deep_scrub":"68'8","last_deep_scrub_stamp":"2024-09-17T05:21:12.737480+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.737480+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:04:20.987562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"68'2532","reported_seq":2652,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.822582+0000","last_change":"2024-09-17T05:21:20.822582+0000","last_active":"2024-09-17T05:21:20.822582+0000","last_peered":"2024-09-17T05:21:20.822582+0000","last_clean":"2024-09-17T05:21:20.822582+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T05:21:20.822582+0000","last_undegraded":"2024-09-17T05:21:20.822582+0000","last_fullsized":"2024-09-17T05:21:20.822582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2532","last_scrub_stamp":"2024-09-17T05:21:20.822256+0000","last_deep_scrub":"68'2532","last_deep_scrub_stamp":"2024-09-17T05:21:20.822256+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.822256+0000","objects_scrubbed":0,"log_size":2532,"log_dups_size":0,"ondisk_log_size":2532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:31:07.663177+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13954,"num_write_kb":219876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"65'4598","reported_seq":4726,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.898537+0000","last_change":"2024-09-17T05:21:09.898537+0000","last_active":"2024-09-17T05:21:09.898537+0000","last_peered":"2024-09-17T05:21:09.898537+0000","last_clean":"2024-09-17T05:21:09.898537+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T05:21:09.898537+0000","last_undegraded":"2024-09-17T05:21:09.898537+0000","last_fullsized":"2024-09-17T05:21:09.898537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4598","last_scrub_stamp":"2024-09-17T05:21:09.898190+0000","last_deep_scrub":"65'4598","last_deep_scrub_stamp":"2024-09-17T05:21:09.898190+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.898190+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:04:18.148273+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24313,"num_write_kb":335123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"68'10","reported_seq":181,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.507971+0000","last_change":"2024-09-17T05:21:06.507971+0000","last_active":"2024-09-17T05:21:06.507971+0000","last_peered":"2024-09-17T05:21:06.507971+0000","last_clean":"2024-09-17T05:21:06.507971+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T05:21:06.507971+0000","last_undegraded":"2024-09-17T05:21:06.507971+0000","last_fullsized":"2024-09-17T05:21:06.507971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-17T05:21:06.507757+0000","last_deep_scrub":"68'10","last_deep_scrub_stamp":"2024-09-17T05:21:06.507757+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.507757+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:47.772431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'50","reported_seq":237,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.750545+0000","last_change":"2024-09-17T05:21:10.750545+0000","last_active":"2024-09-17T05:21:10.750545+0000","last_peered":"2024-09-17T05:21:10.750545+0000","last_clean":"2024-09-17T05:21:10.750545+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T05:21:10.750545+0000","last_undegraded":"2024-09-17T05:21:10.750545+0000","last_fullsized":"2024-09-17T05:21:10.750545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'50","last_scrub_stamp":"2024-09-17T05:21:10.750274+0000","last_deep_scrub":"68'50","last_deep_scrub_stamp":"2024-09-17T05:21:10.750274+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.750274+0000","objects_scrubbed":1,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:00.963016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":65,"num_read_kb":114,"num_write":45,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"68'2531","reported_seq":2644,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.376480+0000","last_change":"2024-09-17T04:49:41.066370+0000","last_active":"2024-09-17T05:20:43.376480+0000","last_peered":"2024-09-17T05:20:43.376480+0000","last_clean":"2024-09-17T05:20:43.376480+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T05:20:43.376480+0000","last_undegraded":"2024-09-17T05:20:43.376480+0000","last_fullsized":"2024-09-17T05:20:43.376480+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2531,"log_dups_size":0,"ondisk_log_size":2531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:07.979353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13881,"num_write_kb":219504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"65'3884","reported_seq":4011,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.854850+0000","last_change":"2024-09-17T05:21:10.854850+0000","last_active":"2024-09-17T05:21:10.854850+0000","last_peered":"2024-09-17T05:21:10.854850+0000","last_clean":"2024-09-17T05:21:10.854850+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T05:21:10.854850+0000","last_undegraded":"2024-09-17T05:21:10.854850+0000","last_fullsized":"2024-09-17T05:21:10.854850+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3884","last_scrub_stamp":"2024-09-17T05:21:10.854629+0000","last_deep_scrub":"65'3884","last_deep_scrub_stamp":"2024-09-17T05:21:10.854629+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.854629+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:49:26.336872+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21342,"num_write_kb":257935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"68'473","reported_seq":623,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:03.864229+0000","last_change":"2024-09-17T05:21:03.864229+0000","last_active":"2024-09-17T05:21:03.864229+0000","last_peered":"2024-09-17T05:21:03.864229+0000","last_clean":"2024-09-17T05:21:03.864229+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T05:21:03.864229+0000","last_undegraded":"2024-09-17T05:21:03.864229+0000","last_fullsized":"2024-09-17T05:21:03.864229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'473","last_scrub_stamp":"2024-09-17T05:21:03.864015+0000","last_deep_scrub":"68'473","last_deep_scrub_stamp":"2024-09-17T05:21:03.864015+0000","last_clean_scrub_stamp":"2024-09-17T05:21:03.864015+0000","objects_scrubbed":1,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:47.178085+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":57,"num_write":1105,"num_write_kb":359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'11","reported_seq":133,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.827996+0000","last_change":"2024-09-17T05:21:15.827996+0000","last_active":"2024-09-17T05:21:15.827996+0000","last_peered":"2024-09-17T05:21:15.827996+0000","last_clean":"2024-09-17T05:21:15.827996+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T05:21:15.827996+0000","last_undegraded":"2024-09-17T05:21:15.827996+0000","last_fullsized":"2024-09-17T05:21:15.827996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-17T05:21:15.827718+0000","last_deep_scrub":"68'11","last_deep_scrub_stamp":"2024-09-17T05:21:15.827718+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.827718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:56:26.850794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"68'2691","reported_seq":2811,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:21.674125+0000","last_change":"2024-09-17T05:21:21.674125+0000","last_active":"2024-09-17T05:21:21.674125+0000","last_peered":"2024-09-17T05:21:21.674125+0000","last_clean":"2024-09-17T05:21:21.674125+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T05:21:21.674125+0000","last_undegraded":"2024-09-17T05:21:21.674125+0000","last_fullsized":"2024-09-17T05:21:21.674125+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2691","last_scrub_stamp":"2024-09-17T05:21:21.673911+0000","last_deep_scrub":"68'2691","last_deep_scrub_stamp":"2024-09-17T05:21:21.673911+0000","last_clean_scrub_stamp":"2024-09-17T05:21:21.673911+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:16.011057+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14724,"num_write_kb":243278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"65'4637","reported_seq":4765,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.270658+0000","last_change":"2024-09-17T05:21:07.270658+0000","last_active":"2024-09-17T05:21:07.270658+0000","last_peered":"2024-09-17T05:21:07.270658+0000","last_clean":"2024-09-17T05:21:07.270658+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T05:21:07.270658+0000","last_undegraded":"2024-09-17T05:21:07.270658+0000","last_fullsized":"2024-09-17T05:21:07.270658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4637","last_scrub_stamp":"2024-09-17T05:21:07.270360+0000","last_deep_scrub":"65'4637","last_deep_scrub_stamp":"2024-09-17T05:21:07.270360+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.270360+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:24.143638+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25111,"num_write_kb":342484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"65'2","reported_seq":132,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:04.441505+0000","last_change":"2024-09-17T05:21:04.441505+0000","last_active":"2024-09-17T05:21:04.441505+0000","last_peered":"2024-09-17T05:21:04.441505+0000","last_clean":"2024-09-17T05:21:04.441505+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T05:21:04.441505+0000","last_undegraded":"2024-09-17T05:21:04.441505+0000","last_fullsized":"2024-09-17T05:21:04.441505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-17T05:21:04.441379+0000","last_deep_scrub":"65'2","last_deep_scrub_stamp":"2024-09-17T05:21:04.441379+0000","last_clean_scrub_stamp":"2024-09-17T05:21:04.441379+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:38.530424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'3","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.219105+0000","last_change":"2024-09-17T05:21:15.219105+0000","last_active":"2024-09-17T05:21:15.219105+0000","last_peered":"2024-09-17T05:21:15.219105+0000","last_clean":"2024-09-17T05:21:15.219105+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T05:21:15.219105+0000","last_undegraded":"2024-09-17T05:21:15.219105+0000","last_fullsized":"2024-09-17T05:21:15.219105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-09-17T05:21:15.218897+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2024-09-17T05:21:15.218897+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.218897+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:40:09.056506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"68'3511","reported_seq":3623,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.341640+0000","last_change":"2024-09-17T04:49:41.056924+0000","last_active":"2024-09-17T05:20:43.341640+0000","last_peered":"2024-09-17T05:20:43.341640+0000","last_clean":"2024-09-17T05:20:43.341640+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T05:20:43.341640+0000","last_undegraded":"2024-09-17T05:20:43.341640+0000","last_fullsized":"2024-09-17T05:20:43.341640+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3511,"log_dups_size":0,"ondisk_log_size":3511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:07.759618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19215,"num_write_kb":314590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"65'3495","reported_seq":3623,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:02.679179+0000","last_change":"2024-09-17T05:21:02.679179+0000","last_active":"2024-09-17T05:21:02.679179+0000","last_peered":"2024-09-17T05:21:02.679179+0000","last_clean":"2024-09-17T05:21:02.679179+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T05:21:02.679179+0000","last_undegraded":"2024-09-17T05:21:02.679179+0000","last_fullsized":"2024-09-17T05:21:02.679179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3495","last_scrub_stamp":"2024-09-17T05:21:02.679039+0000","last_deep_scrub":"65'3495","last_deep_scrub_stamp":"2024-09-17T05:21:02.679039+0000","last_clean_scrub_stamp":"2024-09-17T05:21:02.679039+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:09:41.948141+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18740,"num_write_kb":251346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"65'2","reported_seq":132,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.640295+0000","last_change":"2024-09-17T05:21:05.640295+0000","last_active":"2024-09-17T05:21:05.640295+0000","last_peered":"2024-09-17T05:21:05.640295+0000","last_clean":"2024-09-17T05:21:05.640295+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T05:21:05.640295+0000","last_undegraded":"2024-09-17T05:21:05.640295+0000","last_fullsized":"2024-09-17T05:21:05.640295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-17T05:21:05.639951+0000","last_deep_scrub":"65'2","last_deep_scrub_stamp":"2024-09-17T05:21:05.639951+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.639951+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:24.152670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'4","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.253126+0000","last_change":"2024-09-17T05:21:14.253126+0000","last_active":"2024-09-17T05:21:14.253126+0000","last_peered":"2024-09-17T05:21:14.253126+0000","last_clean":"2024-09-17T05:21:14.253126+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T05:21:14.253126+0000","last_undegraded":"2024-09-17T05:21:14.253126+0000","last_fullsized":"2024-09-17T05:21:14.253126+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-17T05:21:14.252925+0000","last_deep_scrub":"68'4","last_deep_scrub_stamp":"2024-09-17T05:21:14.252925+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.252925+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:54.094101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"68'2923","reported_seq":3044,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.233177+0000","last_change":"2024-09-17T05:21:16.233177+0000","last_active":"2024-09-17T05:21:16.233177+0000","last_peered":"2024-09-17T05:21:16.233177+0000","last_clean":"2024-09-17T05:21:16.233177+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T05:21:16.233177+0000","last_undegraded":"2024-09-17T05:21:16.233177+0000","last_fullsized":"2024-09-17T05:21:16.233177+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2923","last_scrub_stamp":"2024-09-17T05:21:16.232932+0000","last_deep_scrub":"68'2923","last_deep_scrub_stamp":"2024-09-17T05:21:16.232932+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.232932+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:23:31.283988+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":7,"num_write":15908,"num_write_kb":260386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"65'4495","reported_seq":4623,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.728414+0000","last_change":"2024-09-17T05:21:07.728414+0000","last_active":"2024-09-17T05:21:07.728414+0000","last_peered":"2024-09-17T05:21:07.728414+0000","last_clean":"2024-09-17T05:21:07.728414+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T05:21:07.728414+0000","last_undegraded":"2024-09-17T05:21:07.728414+0000","last_fullsized":"2024-09-17T05:21:07.728414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4495","last_scrub_stamp":"2024-09-17T05:21:07.728173+0000","last_deep_scrub":"65'4495","last_deep_scrub_stamp":"2024-09-17T05:21:07.728173+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.728173+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:51.042243+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24287,"num_write_kb":330898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"65'4","reported_seq":134,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.467242+0000","last_change":"2024-09-17T05:21:05.467242+0000","last_active":"2024-09-17T05:21:05.467242+0000","last_peered":"2024-09-17T05:21:05.467242+0000","last_clean":"2024-09-17T05:21:05.467242+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T05:21:05.467242+0000","last_undegraded":"2024-09-17T05:21:05.467242+0000","last_fullsized":"2024-09-17T05:21:05.467242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-09-17T05:21:05.467010+0000","last_deep_scrub":"65'4","last_deep_scrub_stamp":"2024-09-17T05:21:05.467010+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.467010+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:56:16.490086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'2","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.198900+0000","last_change":"2024-09-17T05:21:18.198900+0000","last_active":"2024-09-17T05:21:18.198900+0000","last_peered":"2024-09-17T05:21:18.198900+0000","last_clean":"2024-09-17T05:21:18.198900+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T05:21:18.198900+0000","last_undegraded":"2024-09-17T05:21:18.198900+0000","last_fullsized":"2024-09-17T05:21:18.198900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T05:21:18.198640+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2024-09-17T05:21:18.198640+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.198640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:40.743113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"68'3959","reported_seq":4082,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.208937+0000","last_change":"2024-09-17T05:21:15.208937+0000","last_active":"2024-09-17T05:21:15.208937+0000","last_peered":"2024-09-17T05:21:15.208937+0000","last_clean":"2024-09-17T05:21:15.208937+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T05:21:15.208937+0000","last_undegraded":"2024-09-17T05:21:15.208937+0000","last_fullsized":"2024-09-17T05:21:15.208937+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3959","last_scrub_stamp":"2024-09-17T05:21:15.208721+0000","last_deep_scrub":"68'3959","last_deep_scrub_stamp":"2024-09-17T05:21:15.208721+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.208721+0000","objects_scrubbed":0,"log_size":3959,"log_dups_size":0,"ondisk_log_size":3959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:39.405184+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":13,"num_write":21335,"num_write_kb":323115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'3351","reported_seq":3479,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.675116+0000","last_change":"2024-09-17T05:21:07.675116+0000","last_active":"2024-09-17T05:21:07.675116+0000","last_peered":"2024-09-17T05:21:07.675116+0000","last_clean":"2024-09-17T05:21:07.675116+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T05:21:07.675116+0000","last_undegraded":"2024-09-17T05:21:07.675116+0000","last_fullsized":"2024-09-17T05:21:07.675116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3351","last_scrub_stamp":"2024-09-17T05:21:07.674891+0000","last_deep_scrub":"65'3351","last_deep_scrub_stamp":"2024-09-17T05:21:07.674891+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.674891+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:37.974183+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18427,"num_write_kb":241864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"68'53","reported_seq":251,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.172504+0000","last_change":"2024-09-17T05:21:05.172504+0000","last_active":"2024-09-17T05:21:05.172504+0000","last_peered":"2024-09-17T05:21:05.172504+0000","last_clean":"2024-09-17T05:21:05.172504+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T05:21:05.172504+0000","last_undegraded":"2024-09-17T05:21:05.172504+0000","last_fullsized":"2024-09-17T05:21:05.172504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'53","last_scrub_stamp":"2024-09-17T05:21:05.171784+0000","last_deep_scrub":"68'53","last_deep_scrub_stamp":"2024-09-17T05:21:05.171784+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.171784+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:20:05.118143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":116,"num_write":48,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'10","reported_seq":171,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.197679+0000","last_change":"2024-09-17T05:21:19.197679+0000","last_active":"2024-09-17T05:21:19.197679+0000","last_peered":"2024-09-17T05:21:19.197679+0000","last_clean":"2024-09-17T05:21:19.197679+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T05:21:19.197679+0000","last_undegraded":"2024-09-17T05:21:19.197679+0000","last_fullsized":"2024-09-17T05:21:19.197679+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-17T05:21:19.197471+0000","last_deep_scrub":"68'10","last_deep_scrub_stamp":"2024-09-17T05:21:19.197471+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.197471+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:26.619236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"68'2708","reported_seq":2819,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403990+0000","last_change":"2024-09-17T04:49:41.052004+0000","last_active":"2024-09-17T05:20:44.403990+0000","last_peered":"2024-09-17T05:20:44.403990+0000","last_clean":"2024-09-17T05:20:44.403990+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T05:20:44.403990+0000","last_undegraded":"2024-09-17T05:20:44.403990+0000","last_fullsized":"2024-09-17T05:20:44.403990+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:17.266355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14599,"num_write_kb":234639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"65'3456","reported_seq":3584,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.256433+0000","last_change":"2024-09-17T05:21:08.256433+0000","last_active":"2024-09-17T05:21:08.256433+0000","last_peered":"2024-09-17T05:21:08.256433+0000","last_clean":"2024-09-17T05:21:08.256433+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T05:21:08.256433+0000","last_undegraded":"2024-09-17T05:21:08.256433+0000","last_fullsized":"2024-09-17T05:21:08.256433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3456","last_scrub_stamp":"2024-09-17T05:21:08.256215+0000","last_deep_scrub":"65'3456","last_deep_scrub_stamp":"2024-09-17T05:21:08.256215+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.256215+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:05.804588+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18921,"num_write_kb":260292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"67'13","reported_seq":143,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.546066+0000","last_change":"2024-09-17T05:21:07.546066+0000","last_active":"2024-09-17T05:21:07.546066+0000","last_peered":"2024-09-17T05:21:07.546066+0000","last_clean":"2024-09-17T05:21:07.546066+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T05:21:07.546066+0000","last_undegraded":"2024-09-17T05:21:07.546066+0000","last_fullsized":"2024-09-17T05:21:07.546066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-17T05:21:07.545738+0000","last_deep_scrub":"67'13","last_deep_scrub_stamp":"2024-09-17T05:21:07.545738+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.545738+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:29.551757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"68'473","reported_seq":606,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:47.345918+0000","last_change":"2024-09-17T04:49:40.016634+0000","last_active":"2024-09-17T05:20:47.345918+0000","last_peered":"2024-09-17T05:20:47.345918+0000","last_clean":"2024-09-17T05:20:47.345918+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T05:20:47.345918+0000","last_undegraded":"2024-09-17T05:20:47.345918+0000","last_fullsized":"2024-09-17T05:20:47.345918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:11.673328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":54,"num_write":1100,"num_write_kb":354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"68'3699","reported_seq":3820,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.726710+0000","last_change":"2024-09-17T05:21:20.726710+0000","last_active":"2024-09-17T05:21:20.726710+0000","last_peered":"2024-09-17T05:21:20.726710+0000","last_clean":"2024-09-17T05:21:20.726710+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T05:21:20.726710+0000","last_undegraded":"2024-09-17T05:21:20.726710+0000","last_fullsized":"2024-09-17T05:21:20.726710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3699","last_scrub_stamp":"2024-09-17T05:21:20.726343+0000","last_deep_scrub":"68'3699","last_deep_scrub_stamp":"2024-09-17T05:21:20.726343+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.726343+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:00:12.900330+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":20021,"num_write_kb":309708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"65'3651","reported_seq":3779,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.899240+0000","last_change":"2024-09-17T05:21:08.899240+0000","last_active":"2024-09-17T05:21:08.899240+0000","last_peered":"2024-09-17T05:21:08.899240+0000","last_clean":"2024-09-17T05:21:08.899240+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T05:21:08.899240+0000","last_undegraded":"2024-09-17T05:21:08.899240+0000","last_fullsized":"2024-09-17T05:21:08.899240+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3651","last_scrub_stamp":"2024-09-17T05:21:08.899022+0000","last_deep_scrub":"65'3651","last_deep_scrub_stamp":"2024-09-17T05:21:08.899022+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.899022+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:20.889499+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19517,"num_write_kb":274922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"65'3","reported_seq":133,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:03.435826+0000","last_change":"2024-09-17T05:21:03.435826+0000","last_active":"2024-09-17T05:21:03.435826+0000","last_peered":"2024-09-17T05:21:03.435826+0000","last_clean":"2024-09-17T05:21:03.435826+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T05:21:03.435826+0000","last_undegraded":"2024-09-17T05:21:03.435826+0000","last_fullsized":"2024-09-17T05:21:03.435826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-09-17T05:21:03.435568+0000","last_deep_scrub":"65'3","last_deep_scrub_stamp":"2024-09-17T05:21:03.435568+0000","last_clean_scrub_stamp":"2024-09-17T05:21:03.435568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:15.962476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"68'74","reported_seq":228,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.023177+0000","last_change":"2024-09-17T05:21:05.023177+0000","last_active":"2024-09-17T05:21:05.023177+0000","last_peered":"2024-09-17T05:21:05.023177+0000","last_clean":"2024-09-17T05:21:05.023177+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T05:21:05.023177+0000","last_undegraded":"2024-09-17T05:21:05.023177+0000","last_fullsized":"2024-09-17T05:21:05.023177+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'74","last_scrub_stamp":"2024-09-17T05:21:05.022924+0000","last_deep_scrub":"68'74","last_deep_scrub_stamp":"2024-09-17T05:21:05.022924+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.022924+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:13.286994+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'2","reported_seq":116,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.115256+0000","last_change":"2024-09-17T04:49:39.995951+0000","last_active":"2024-09-17T05:20:43.115256+0000","last_peered":"2024-09-17T05:20:43.115256+0000","last_clean":"2024-09-17T05:20:43.115256+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T05:20:43.115256+0000","last_undegraded":"2024-09-17T05:20:43.115256+0000","last_fullsized":"2024-09-17T05:20:43.115256+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"68'2130","reported_seq":2242,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.419272+0000","last_change":"2024-09-17T04:49:41.054950+0000","last_active":"2024-09-17T05:20:44.419272+0000","last_peered":"2024-09-17T05:20:44.419272+0000","last_clean":"2024-09-17T05:20:44.419272+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T05:20:44.419272+0000","last_undegraded":"2024-09-17T05:20:44.419272+0000","last_fullsized":"2024-09-17T05:20:44.419272+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2130,"log_dups_size":0,"ondisk_log_size":2130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":11413,"num_write_kb":185062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"65'4183","reported_seq":4310,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.501868+0000","last_change":"2024-09-17T05:21:14.501868+0000","last_active":"2024-09-17T05:21:14.501868+0000","last_peered":"2024-09-17T05:21:14.501868+0000","last_clean":"2024-09-17T05:21:14.501868+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T05:21:14.501868+0000","last_undegraded":"2024-09-17T05:21:14.501868+0000","last_fullsized":"2024-09-17T05:21:14.501868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4183","last_scrub_stamp":"2024-09-17T05:21:14.501644+0000","last_deep_scrub":"65'4183","last_deep_scrub_stamp":"2024-09-17T05:21:14.501644+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.501644+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:46.619129+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22326,"num_write_kb":302603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"65'5","reported_seq":134,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.248291+0000","last_change":"2024-09-17T05:21:12.248291+0000","last_active":"2024-09-17T05:21:12.248291+0000","last_peered":"2024-09-17T05:21:12.248291+0000","last_clean":"2024-09-17T05:21:12.248291+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T05:21:12.248291+0000","last_undegraded":"2024-09-17T05:21:12.248291+0000","last_fullsized":"2024-09-17T05:21:12.248291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-17T05:21:12.247947+0000","last_deep_scrub":"65'5","last_deep_scrub_stamp":"2024-09-17T05:21:12.247947+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.247947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:37.743922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"68'3","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.782673+0000","last_change":"2024-09-17T05:21:11.782673+0000","last_active":"2024-09-17T05:21:11.782673+0000","last_peered":"2024-09-17T05:21:11.782673+0000","last_clean":"2024-09-17T05:21:11.782673+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T05:21:11.782673+0000","last_undegraded":"2024-09-17T05:21:11.782673+0000","last_fullsized":"2024-09-17T05:21:11.782673+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-09-17T05:21:11.782336+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2024-09-17T05:21:11.782336+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.782336+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:23.772814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"68'3285","reported_seq":3397,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.413822+0000","last_change":"2024-09-17T04:49:41.024278+0000","last_active":"2024-09-17T05:20:43.413822+0000","last_peered":"2024-09-17T05:20:43.413822+0000","last_clean":"2024-09-17T05:20:43.413822+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T05:20:43.413822+0000","last_undegraded":"2024-09-17T05:20:43.413822+0000","last_fullsized":"2024-09-17T05:20:43.413822+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3285,"log_dups_size":0,"ondisk_log_size":3285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:58:12.924609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18245,"num_write_kb":282158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"65'4407","reported_seq":4534,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.778487+0000","last_change":"2024-09-17T05:21:08.778487+0000","last_active":"2024-09-17T05:21:08.778487+0000","last_peered":"2024-09-17T05:21:08.778487+0000","last_clean":"2024-09-17T05:21:08.778487+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T05:21:08.778487+0000","last_undegraded":"2024-09-17T05:21:08.778487+0000","last_fullsized":"2024-09-17T05:21:08.778487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4407","last_scrub_stamp":"2024-09-17T05:21:08.778262+0000","last_deep_scrub":"65'4407","last_deep_scrub_stamp":"2024-09-17T05:21:08.778262+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.778262+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:43.795331+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24558,"num_write_kb":315810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"68'7","reported_seq":157,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.632166+0000","last_change":"2024-09-17T05:21:12.632166+0000","last_active":"2024-09-17T05:21:12.632166+0000","last_peered":"2024-09-17T05:21:12.632166+0000","last_clean":"2024-09-17T05:21:12.632166+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T05:21:12.632166+0000","last_undegraded":"2024-09-17T05:21:12.632166+0000","last_fullsized":"2024-09-17T05:21:12.632166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'7","last_scrub_stamp":"2024-09-17T05:21:12.631953+0000","last_deep_scrub":"68'7","last_deep_scrub_stamp":"2024-09-17T05:21:12.631953+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.631953+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:52:04.393430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'1447","reported_seq":1612,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.697910+0000","last_change":"2024-09-17T05:21:19.697910+0000","last_active":"2024-09-17T05:21:19.697910+0000","last_peered":"2024-09-17T05:21:19.697910+0000","last_clean":"2024-09-17T05:21:19.697910+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T05:21:19.697910+0000","last_undegraded":"2024-09-17T05:21:19.697910+0000","last_fullsized":"2024-09-17T05:21:19.697910+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1447","last_scrub_stamp":"2024-09-17T05:21:19.697700+0000","last_deep_scrub":"68'1447","last_deep_scrub_stamp":"2024-09-17T05:21:19.697700+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.697700+0000","objects_scrubbed":2,"log_size":1447,"log_dups_size":0,"ondisk_log_size":1447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:32.224608+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":38239,"num_write":1421,"num_write_kb":4489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'2587","reported_seq":2706,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.964539+0000","last_change":"2024-09-17T05:21:14.964539+0000","last_active":"2024-09-17T05:21:14.964539+0000","last_peered":"2024-09-17T05:21:14.964539+0000","last_clean":"2024-09-17T05:21:14.964539+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T05:21:14.964539+0000","last_undegraded":"2024-09-17T05:21:14.964539+0000","last_fullsized":"2024-09-17T05:21:14.964539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2587","last_scrub_stamp":"2024-09-17T05:21:14.964208+0000","last_deep_scrub":"68'2587","last_deep_scrub_stamp":"2024-09-17T05:21:14.964208+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.964208+0000","objects_scrubbed":0,"log_size":2587,"log_dups_size":0,"ondisk_log_size":2587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:16.231401+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13558,"num_write_kb":238553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"65'4762","reported_seq":4889,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.698211+0000","last_change":"2024-09-17T05:21:12.698211+0000","last_active":"2024-09-17T05:21:12.698211+0000","last_peered":"2024-09-17T05:21:12.698211+0000","last_clean":"2024-09-17T05:21:12.698211+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T05:21:12.698211+0000","last_undegraded":"2024-09-17T05:21:12.698211+0000","last_fullsized":"2024-09-17T05:21:12.698211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4762","last_scrub_stamp":"2024-09-17T05:21:12.697995+0000","last_deep_scrub":"65'4762","last_deep_scrub_stamp":"2024-09-17T05:21:12.697995+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.697995+0000","objects_scrubbed":0,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:13.662257+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25049,"num_write_kb":354398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"68'8","reported_seq":179,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.914619+0000","last_change":"2024-09-17T05:21:05.914619+0000","last_active":"2024-09-17T05:21:05.914619+0000","last_peered":"2024-09-17T05:21:05.914619+0000","last_clean":"2024-09-17T05:21:05.914619+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T05:21:05.914619+0000","last_undegraded":"2024-09-17T05:21:05.914619+0000","last_fullsized":"2024-09-17T05:21:05.914619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-17T05:21:05.914411+0000","last_deep_scrub":"68'8","last_deep_scrub_stamp":"2024-09-17T05:21:05.914411+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.914411+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:50.560424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"68'5","reported_seq":118,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.116235+0000","last_change":"2024-09-17T04:49:39.990892+0000","last_active":"2024-09-17T05:20:43.116235+0000","last_peered":"2024-09-17T05:20:43.116235+0000","last_clean":"2024-09-17T05:20:43.116235+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T05:20:43.116235+0000","last_undegraded":"2024-09-17T05:20:43.116235+0000","last_fullsized":"2024-09-17T05:20:43.116235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:30:12.588747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"68'2327","reported_seq":2446,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.199619+0000","last_change":"2024-09-17T05:21:17.199619+0000","last_active":"2024-09-17T05:21:17.199619+0000","last_peered":"2024-09-17T05:21:17.199619+0000","last_clean":"2024-09-17T05:21:17.199619+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T05:21:17.199619+0000","last_undegraded":"2024-09-17T05:21:17.199619+0000","last_fullsized":"2024-09-17T05:21:17.199619+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2327","last_scrub_stamp":"2024-09-17T05:21:17.199339+0000","last_deep_scrub":"68'2327","last_deep_scrub_stamp":"2024-09-17T05:21:17.199339+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.199339+0000","objects_scrubbed":0,"log_size":2327,"log_dups_size":0,"ondisk_log_size":2327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:53.006691+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12892,"num_write_kb":211370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"65'3722","reported_seq":3849,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:03.644226+0000","last_change":"2024-09-17T05:21:03.643932+0000","last_active":"2024-09-17T05:21:03.644226+0000","last_peered":"2024-09-17T05:21:03.644226+0000","last_clean":"2024-09-17T05:21:03.644226+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T05:21:03.644226+0000","last_undegraded":"2024-09-17T05:21:03.644226+0000","last_fullsized":"2024-09-17T05:21:03.644226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3722","last_scrub_stamp":"2024-09-17T05:21:03.643638+0000","last_deep_scrub":"65'3722","last_deep_scrub_stamp":"2024-09-17T05:21:03.643638+0000","last_clean_scrub_stamp":"2024-09-17T05:21:03.643638+0000","objects_scrubbed":0,"log_size":3722,"log_dups_size":0,"ondisk_log_size":3722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:49:58.849136+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19724,"num_write_kb":263684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"68'5","reported_seq":155,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:02.705197+0000","last_change":"2024-09-17T05:21:02.705029+0000","last_active":"2024-09-17T05:21:02.705197+0000","last_peered":"2024-09-17T05:21:02.705197+0000","last_clean":"2024-09-17T05:21:02.705197+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T05:21:02.705197+0000","last_undegraded":"2024-09-17T05:21:02.705197+0000","last_fullsized":"2024-09-17T05:21:02.705197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:02.704797+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:02.704797+0000","last_clean_scrub_stamp":"2024-09-17T05:21:02.704797+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:20.480361+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"68'2","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.966186+0000","last_change":"2024-09-17T05:21:13.966186+0000","last_active":"2024-09-17T05:21:13.966186+0000","last_peered":"2024-09-17T05:21:13.966186+0000","last_clean":"2024-09-17T05:21:13.966186+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T05:21:13.966186+0000","last_undegraded":"2024-09-17T05:21:13.966186+0000","last_fullsized":"2024-09-17T05:21:13.966186+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T05:21:13.965981+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2024-09-17T05:21:13.965981+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.965981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:20:33.477072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"68'3186","reported_seq":3296,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.421154+0000","last_change":"2024-09-17T04:49:41.052195+0000","last_active":"2024-09-17T05:20:44.421154+0000","last_peered":"2024-09-17T05:20:44.421154+0000","last_clean":"2024-09-17T05:20:44.421154+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T05:20:44.421154+0000","last_undegraded":"2024-09-17T05:20:44.421154+0000","last_fullsized":"2024-09-17T05:20:44.421154+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16763,"num_write_kb":264037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"65'3694","reported_seq":3822,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.531689+0000","last_change":"2024-09-17T05:21:16.531689+0000","last_active":"2024-09-17T05:21:16.531689+0000","last_peered":"2024-09-17T05:21:16.531689+0000","last_clean":"2024-09-17T05:21:16.531689+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T05:21:16.531689+0000","last_undegraded":"2024-09-17T05:21:16.531689+0000","last_fullsized":"2024-09-17T05:21:16.531689+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3694","last_scrub_stamp":"2024-09-17T05:21:16.531452+0000","last_deep_scrub":"65'3694","last_deep_scrub_stamp":"2024-09-17T05:21:16.531452+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.531452+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:37.863588+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20762,"num_write_kb":284135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"68'19","reported_seq":235,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.923705+0000","last_change":"2024-09-17T05:21:06.923705+0000","last_active":"2024-09-17T05:21:06.923705+0000","last_peered":"2024-09-17T05:21:06.923705+0000","last_clean":"2024-09-17T05:21:06.923705+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T05:21:06.923705+0000","last_undegraded":"2024-09-17T05:21:06.923705+0000","last_fullsized":"2024-09-17T05:21:06.923705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-17T05:21:06.923348+0000","last_deep_scrub":"68'19","last_deep_scrub_stamp":"2024-09-17T05:21:06.923348+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.923348+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:50.769232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":264,"num_read_kb":264,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"68'3","reported_seq":115,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.399465+0000","last_change":"2024-09-17T04:49:40.021798+0000","last_active":"2024-09-17T05:20:44.399465+0000","last_peered":"2024-09-17T05:20:44.399465+0000","last_clean":"2024-09-17T05:20:44.399465+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T05:20:44.399465+0000","last_undegraded":"2024-09-17T05:20:44.399465+0000","last_fullsized":"2024-09-17T05:20:44.399465+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"68'3074","reported_seq":3187,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403314+0000","last_change":"2024-09-17T04:49:41.065288+0000","last_active":"2024-09-17T05:20:44.403314+0000","last_peered":"2024-09-17T05:20:44.403314+0000","last_clean":"2024-09-17T05:20:44.403314+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T05:20:44.403314+0000","last_undegraded":"2024-09-17T05:20:44.403314+0000","last_fullsized":"2024-09-17T05:20:44.403314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":16509,"num_write_kb":287321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'2828","reported_seq":2954,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:03.031642+0000","last_change":"2024-09-17T05:21:03.031642+0000","last_active":"2024-09-17T05:21:03.031642+0000","last_peered":"2024-09-17T05:21:03.031642+0000","last_clean":"2024-09-17T05:21:03.031642+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T05:21:03.031642+0000","last_undegraded":"2024-09-17T05:21:03.031642+0000","last_fullsized":"2024-09-17T05:21:03.031642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2828","last_scrub_stamp":"2024-09-17T05:21:03.031361+0000","last_deep_scrub":"65'2828","last_deep_scrub_stamp":"2024-09-17T05:21:03.031361+0000","last_clean_scrub_stamp":"2024-09-17T05:21:03.031361+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:45:33.333087+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14821,"num_write_kb":206837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"65'5","reported_seq":133,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.726265+0000","last_change":"2024-09-17T05:21:05.726265+0000","last_active":"2024-09-17T05:21:05.726265+0000","last_peered":"2024-09-17T05:21:05.726265+0000","last_clean":"2024-09-17T05:21:05.726265+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T05:21:05.726265+0000","last_undegraded":"2024-09-17T05:21:05.726265+0000","last_fullsized":"2024-09-17T05:21:05.726265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-17T05:21:05.726041+0000","last_deep_scrub":"65'5","last_deep_scrub_stamp":"2024-09-17T05:21:05.726041+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.726041+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:36.025332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"68'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.699238+0000","last_change":"2024-09-17T05:21:20.699238+0000","last_active":"2024-09-17T05:21:20.699238+0000","last_peered":"2024-09-17T05:21:20.699238+0000","last_clean":"2024-09-17T05:21:20.699238+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T05:21:20.699238+0000","last_undegraded":"2024-09-17T05:21:20.699238+0000","last_fullsized":"2024-09-17T05:21:20.699238+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:20.699035+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:20.699035+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.699035+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:54.788080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"68'2590","reported_seq":2700,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.339146+0000","last_change":"2024-09-17T04:49:41.074092+0000","last_active":"2024-09-17T05:20:43.339146+0000","last_peered":"2024-09-17T05:20:43.339146+0000","last_clean":"2024-09-17T05:20:43.339146+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T05:20:43.339146+0000","last_undegraded":"2024-09-17T05:20:43.339146+0000","last_fullsized":"2024-09-17T05:20:43.339146+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14722,"num_write_kb":243326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'4461","reported_seq":4587,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.543088+0000","last_change":"2024-09-17T05:21:12.543088+0000","last_active":"2024-09-17T05:21:12.543088+0000","last_peered":"2024-09-17T05:21:12.543088+0000","last_clean":"2024-09-17T05:21:12.543088+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T05:21:12.543088+0000","last_undegraded":"2024-09-17T05:21:12.543088+0000","last_fullsized":"2024-09-17T05:21:12.543088+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4461","last_scrub_stamp":"2024-09-17T05:21:12.542724+0000","last_deep_scrub":"65'4461","last_deep_scrub_stamp":"2024-09-17T05:21:12.542724+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.542724+0000","objects_scrubbed":0,"log_size":4461,"log_dups_size":0,"ondisk_log_size":4461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:07:02.098906+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24667,"num_write_kb":332445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"65'3","reported_seq":131,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.526247+0000","last_change":"2024-09-17T05:21:09.526247+0000","last_active":"2024-09-17T05:21:09.526247+0000","last_peered":"2024-09-17T05:21:09.526247+0000","last_clean":"2024-09-17T05:21:09.526247+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T05:21:09.526247+0000","last_undegraded":"2024-09-17T05:21:09.526247+0000","last_fullsized":"2024-09-17T05:21:09.526247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-09-17T05:21:09.525904+0000","last_deep_scrub":"65'3","last_deep_scrub_stamp":"2024-09-17T05:21:09.525904+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.525904+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:27.874892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"68'367","reported_seq":579,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:39.638931+0000","last_change":"2024-09-17T04:49:40.004544+0000","last_active":"2024-09-17T05:20:39.638931+0000","last_peered":"2024-09-17T05:20:39.638931+0000","last_clean":"2024-09-17T05:20:39.638931+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T05:20:39.638931+0000","last_undegraded":"2024-09-17T05:20:39.638931+0000","last_fullsized":"2024-09-17T05:20:39.638931+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:56.603277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":180,"num_read_kb":82080,"num_write":364,"num_write_kb":4295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"68'3856","reported_seq":3961,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403743+0000","last_change":"2024-09-17T04:50:12.067788+0000","last_active":"2024-09-17T05:20:44.403743+0000","last_peered":"2024-09-17T05:20:44.403743+0000","last_clean":"2024-09-17T05:20:44.403743+0000","last_became_active":"2024-09-17T04:50:12.066741+0000","last_became_peered":"2024-09-17T04:50:12.066741+0000","last_unstale":"2024-09-17T05:20:44.403743+0000","last_undegraded":"2024-09-17T05:20:44.403743+0000","last_fullsized":"2024-09-17T05:20:44.403743+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3856,"log_dups_size":0,"ondisk_log_size":3856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":20645,"num_write_kb":344954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"65'3521","reported_seq":3647,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.207139+0000","last_change":"2024-09-17T05:21:09.207139+0000","last_active":"2024-09-17T05:21:09.207139+0000","last_peered":"2024-09-17T05:21:09.207139+0000","last_clean":"2024-09-17T05:21:09.207139+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T05:21:09.207139+0000","last_undegraded":"2024-09-17T05:21:09.207139+0000","last_fullsized":"2024-09-17T05:21:09.207139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3521","last_scrub_stamp":"2024-09-17T05:21:09.206770+0000","last_deep_scrub":"65'3521","last_deep_scrub_stamp":"2024-09-17T05:21:09.206770+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.206770+0000","objects_scrubbed":0,"log_size":3521,"log_dups_size":0,"ondisk_log_size":3521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:41.837932+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19198,"num_write_kb":253218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"68'12","reported_seq":182,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.168760+0000","last_change":"2024-09-17T05:21:06.168760+0000","last_active":"2024-09-17T05:21:06.168760+0000","last_peered":"2024-09-17T05:21:06.168760+0000","last_clean":"2024-09-17T05:21:06.168760+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T05:21:06.168760+0000","last_undegraded":"2024-09-17T05:21:06.168760+0000","last_fullsized":"2024-09-17T05:21:06.168760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-17T05:21:06.168428+0000","last_deep_scrub":"68'12","last_deep_scrub_stamp":"2024-09-17T05:21:06.168428+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.168428+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:20:32.765263+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":168,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"68'30","reported_seq":182,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:48.382838+0000","last_change":"2024-09-17T04:49:39.994819+0000","last_active":"2024-09-17T05:20:48.382838+0000","last_peered":"2024-09-17T05:20:48.382838+0000","last_clean":"2024-09-17T05:20:48.382838+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T05:20:48.382838+0000","last_undegraded":"2024-09-17T05:20:48.382838+0000","last_fullsized":"2024-09-17T05:20:48.382838+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":31,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"68'3085","reported_seq":3195,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.412893+0000","last_change":"2024-09-17T04:49:41.051992+0000","last_active":"2024-09-17T05:20:43.412893+0000","last_peered":"2024-09-17T05:20:43.412893+0000","last_clean":"2024-09-17T05:20:43.412893+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T05:20:43.412893+0000","last_undegraded":"2024-09-17T05:20:43.412893+0000","last_fullsized":"2024-09-17T05:20:43.412893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16824,"num_write_kb":288541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"65'3611","reported_seq":3737,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.758081+0000","last_change":"2024-09-17T05:21:09.758081+0000","last_active":"2024-09-17T05:21:09.758081+0000","last_peered":"2024-09-17T05:21:09.758081+0000","last_clean":"2024-09-17T05:21:09.758081+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T05:21:09.758081+0000","last_undegraded":"2024-09-17T05:21:09.758081+0000","last_fullsized":"2024-09-17T05:21:09.758081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3611","last_scrub_stamp":"2024-09-17T05:21:09.757816+0000","last_deep_scrub":"65'3611","last_deep_scrub_stamp":"2024-09-17T05:21:09.757816+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.757816+0000","objects_scrubbed":0,"log_size":3611,"log_dups_size":0,"ondisk_log_size":3611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:17.538586+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19975,"num_write_kb":264442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"68'352","reported_seq":586,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:04.686963+0000","last_change":"2024-09-17T05:21:04.686963+0000","last_active":"2024-09-17T05:21:04.686963+0000","last_peered":"2024-09-17T05:21:04.686963+0000","last_clean":"2024-09-17T05:21:04.686963+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T05:21:04.686963+0000","last_undegraded":"2024-09-17T05:21:04.686963+0000","last_fullsized":"2024-09-17T05:21:04.686963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'352","last_scrub_stamp":"2024-09-17T05:21:04.686835+0000","last_deep_scrub":"68'352","last_deep_scrub_stamp":"2024-09-17T05:21:04.686835+0000","last_clean_scrub_stamp":"2024-09-17T05:21:04.686835+0000","objects_scrubbed":4,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:42.202865+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":190,"num_read_kb":82094,"num_write":345,"num_write_kb":4278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"68'3","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.866522+0000","last_change":"2024-09-17T05:21:17.866522+0000","last_active":"2024-09-17T05:21:17.866522+0000","last_peered":"2024-09-17T05:21:17.866522+0000","last_clean":"2024-09-17T05:21:17.866522+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T05:21:17.866522+0000","last_undegraded":"2024-09-17T05:21:17.866522+0000","last_fullsized":"2024-09-17T05:21:17.866522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-09-17T05:21:17.866319+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2024-09-17T05:21:17.866319+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.866319+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:39.872338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"68'2950","reported_seq":3061,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.342183+0000","last_change":"2024-09-17T04:49:41.030350+0000","last_active":"2024-09-17T05:20:43.342183+0000","last_peered":"2024-09-17T05:20:43.342183+0000","last_clean":"2024-09-17T05:20:43.342183+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T05:20:43.342183+0000","last_undegraded":"2024-09-17T05:20:43.342183+0000","last_fullsized":"2024-09-17T05:20:43.342183+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:51.156215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":16110,"num_write_kb":249022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"65'4529","reported_seq":4655,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.533427+0000","last_change":"2024-09-17T05:21:17.533427+0000","last_active":"2024-09-17T05:21:17.533427+0000","last_peered":"2024-09-17T05:21:17.533427+0000","last_clean":"2024-09-17T05:21:17.533427+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T05:21:17.533427+0000","last_undegraded":"2024-09-17T05:21:17.533427+0000","last_fullsized":"2024-09-17T05:21:17.533427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4529","last_scrub_stamp":"2024-09-17T05:21:17.533062+0000","last_deep_scrub":"65'4529","last_deep_scrub_stamp":"2024-09-17T05:21:17.533062+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.533062+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:12.954223+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23581,"num_write_kb":327974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"68'32","reported_seq":223,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.167606+0000","last_change":"2024-09-17T05:21:07.167606+0000","last_active":"2024-09-17T05:21:07.167606+0000","last_peered":"2024-09-17T05:21:07.167606+0000","last_clean":"2024-09-17T05:21:07.167606+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T05:21:07.167606+0000","last_undegraded":"2024-09-17T05:21:07.167606+0000","last_fullsized":"2024-09-17T05:21:07.167606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'32","last_scrub_stamp":"2024-09-17T05:21:07.167388+0000","last_deep_scrub":"68'32","last_deep_scrub_stamp":"2024-09-17T05:21:07.167388+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.167388+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:03:12.272537+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'10","reported_seq":170,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.729215+0000","last_change":"2024-09-17T05:21:13.729215+0000","last_active":"2024-09-17T05:21:13.729215+0000","last_peered":"2024-09-17T05:21:13.729215+0000","last_clean":"2024-09-17T05:21:13.729215+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T05:21:13.729215+0000","last_undegraded":"2024-09-17T05:21:13.729215+0000","last_fullsized":"2024-09-17T05:21:13.729215+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-17T05:21:13.728980+0000","last_deep_scrub":"68'10","last_deep_scrub_stamp":"2024-09-17T05:21:13.728980+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.728980+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:49:29.211223+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"68'2811","reported_seq":2924,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.378555+0000","last_change":"2024-09-17T04:49:41.052187+0000","last_active":"2024-09-17T05:20:43.378555+0000","last_peered":"2024-09-17T05:20:43.378555+0000","last_clean":"2024-09-17T05:20:43.378555+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T05:20:43.378555+0000","last_undegraded":"2024-09-17T05:20:43.378555+0000","last_fullsized":"2024-09-17T05:20:43.378555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2811,"log_dups_size":0,"ondisk_log_size":2811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:57.076647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15712,"num_write_kb":243392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"65'4109","reported_seq":4235,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.672735+0000","last_change":"2024-09-17T05:21:09.672735+0000","last_active":"2024-09-17T05:21:09.672735+0000","last_peered":"2024-09-17T05:21:09.672735+0000","last_clean":"2024-09-17T05:21:09.672735+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T05:21:09.672735+0000","last_undegraded":"2024-09-17T05:21:09.672735+0000","last_fullsized":"2024-09-17T05:21:09.672735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4109","last_scrub_stamp":"2024-09-17T05:21:09.672433+0000","last_deep_scrub":"65'4109","last_deep_scrub_stamp":"2024-09-17T05:21:09.672433+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.672433+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:53:46.138168+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21556,"num_write_kb":307833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"65'3","reported_seq":131,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:03.722415+0000","last_change":"2024-09-17T05:21:03.722415+0000","last_active":"2024-09-17T05:21:03.722415+0000","last_peered":"2024-09-17T05:21:03.722415+0000","last_clean":"2024-09-17T05:21:03.722415+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T05:21:03.722415+0000","last_undegraded":"2024-09-17T05:21:03.722415+0000","last_fullsized":"2024-09-17T05:21:03.722415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-09-17T05:21:03.722284+0000","last_deep_scrub":"65'3","last_deep_scrub_stamp":"2024-09-17T05:21:03.722284+0000","last_clean_scrub_stamp":"2024-09-17T05:21:03.722284+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:00:13.715589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'19","reported_seq":221,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.126158+0000","last_change":"2024-09-17T05:21:12.126158+0000","last_active":"2024-09-17T05:21:12.126158+0000","last_peered":"2024-09-17T05:21:12.126158+0000","last_clean":"2024-09-17T05:21:12.126158+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T05:21:12.126158+0000","last_undegraded":"2024-09-17T05:21:12.126158+0000","last_fullsized":"2024-09-17T05:21:12.126158+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-17T05:21:12.125935+0000","last_deep_scrub":"68'19","last_deep_scrub_stamp":"2024-09-17T05:21:12.125935+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.125935+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:01.062271+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":246,"num_read_kb":246,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"68'2542","reported_seq":2660,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.794837+0000","last_change":"2024-09-17T05:21:20.794837+0000","last_active":"2024-09-17T05:21:20.794837+0000","last_peered":"2024-09-17T05:21:20.794837+0000","last_clean":"2024-09-17T05:21:20.794837+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T05:21:20.794837+0000","last_undegraded":"2024-09-17T05:21:20.794837+0000","last_fullsized":"2024-09-17T05:21:20.794837+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2542","last_scrub_stamp":"2024-09-17T05:21:20.794493+0000","last_deep_scrub":"68'2542","last_deep_scrub_stamp":"2024-09-17T05:21:20.794493+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.794493+0000","objects_scrubbed":0,"log_size":2542,"log_dups_size":0,"ondisk_log_size":2542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:08:37.076236+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13327,"num_write_kb":225533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"65'3736","reported_seq":3862,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.761625+0000","last_change":"2024-09-17T05:21:07.761625+0000","last_active":"2024-09-17T05:21:07.761625+0000","last_peered":"2024-09-17T05:21:07.761625+0000","last_clean":"2024-09-17T05:21:07.761625+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T05:21:07.761625+0000","last_undegraded":"2024-09-17T05:21:07.761625+0000","last_fullsized":"2024-09-17T05:21:07.761625+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3736","last_scrub_stamp":"2024-09-17T05:21:07.761340+0000","last_deep_scrub":"65'3736","last_deep_scrub_stamp":"2024-09-17T05:21:07.761340+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.761340+0000","objects_scrubbed":0,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:40.011725+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20513,"num_write_kb":287089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"68'5","reported_seq":133,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:01.981394+0000","last_change":"2024-09-17T05:21:01.981394+0000","last_active":"2024-09-17T05:21:01.981394+0000","last_peered":"2024-09-17T05:21:01.981394+0000","last_clean":"2024-09-17T05:21:01.981394+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T05:21:01.981394+0000","last_undegraded":"2024-09-17T05:21:01.981394+0000","last_fullsized":"2024-09-17T05:21:01.981394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:01.981171+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:01.981171+0000","last_clean_scrub_stamp":"2024-09-17T05:21:01.981171+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:53.619421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"68'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.839899+0000","last_change":"2024-09-17T05:21:16.839899+0000","last_active":"2024-09-17T05:21:16.839899+0000","last_peered":"2024-09-17T05:21:16.839899+0000","last_clean":"2024-09-17T05:21:16.839899+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T05:21:16.839899+0000","last_undegraded":"2024-09-17T05:21:16.839899+0000","last_fullsized":"2024-09-17T05:21:16.839899+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:16.839667+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:16.839667+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.839667+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:58.104341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"68'4175","reported_seq":4299,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.788607+0000","last_change":"2024-09-17T05:21:19.788607+0000","last_active":"2024-09-17T05:21:19.788607+0000","last_peered":"2024-09-17T05:21:19.788607+0000","last_clean":"2024-09-17T05:21:19.788607+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T05:21:19.788607+0000","last_undegraded":"2024-09-17T05:21:19.788607+0000","last_fullsized":"2024-09-17T05:21:19.788607+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4175","last_scrub_stamp":"2024-09-17T05:21:19.788380+0000","last_deep_scrub":"68'4175","last_deep_scrub_stamp":"2024-09-17T05:21:19.788380+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.788380+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:01.053054+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":40,"num_write":23028,"num_write_kb":362813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"68'25","reported_seq":147,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.809622+0000","last_change":"2024-09-17T05:21:19.809622+0000","last_active":"2024-09-17T05:21:19.809622+0000","last_peered":"2024-09-17T05:21:19.809622+0000","last_clean":"2024-09-17T05:21:19.809622+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T05:21:19.809622+0000","last_undegraded":"2024-09-17T05:21:19.809622+0000","last_fullsized":"2024-09-17T05:21:19.809622+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'25","last_scrub_stamp":"2024-09-17T05:21:19.809420+0000","last_deep_scrub":"68'25","last_deep_scrub_stamp":"2024-09-17T05:21:19.809420+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.809420+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:38.158408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"65'4339","reported_seq":4465,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.844188+0000","last_change":"2024-09-17T05:21:12.844188+0000","last_active":"2024-09-17T05:21:12.844188+0000","last_peered":"2024-09-17T05:21:12.844188+0000","last_clean":"2024-09-17T05:21:12.844188+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T05:21:12.844188+0000","last_undegraded":"2024-09-17T05:21:12.844188+0000","last_fullsized":"2024-09-17T05:21:12.844188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4339","last_scrub_stamp":"2024-09-17T05:21:12.843934+0000","last_deep_scrub":"65'4339","last_deep_scrub_stamp":"2024-09-17T05:21:12.843934+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.843934+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:00.889647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23134,"num_write_kb":338604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"65'2","reported_seq":130,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.535923+0000","last_change":"2024-09-17T05:21:10.535923+0000","last_active":"2024-09-17T05:21:10.535923+0000","last_peered":"2024-09-17T05:21:10.535923+0000","last_clean":"2024-09-17T05:21:10.535923+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T05:21:10.535923+0000","last_undegraded":"2024-09-17T05:21:10.535923+0000","last_fullsized":"2024-09-17T05:21:10.535923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-17T05:21:10.535695+0000","last_deep_scrub":"65'2","last_deep_scrub_stamp":"2024-09-17T05:21:10.535695+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.535695+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:57.376617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"68'2877","reported_seq":2989,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.395024+0000","last_change":"2024-09-17T04:49:41.037778+0000","last_active":"2024-09-17T05:20:43.395024+0000","last_peered":"2024-09-17T05:20:43.395024+0000","last_clean":"2024-09-17T05:20:43.395024+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T05:20:43.395024+0000","last_undegraded":"2024-09-17T05:20:43.395024+0000","last_fullsized":"2024-09-17T05:20:43.395024+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2877,"log_dups_size":0,"ondisk_log_size":2877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:14.649150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15865,"num_write_kb":272557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"68'1000","reported_seq":1162,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.429369+0000","last_change":"2024-09-17T04:49:40.001195+0000","last_active":"2024-09-17T05:20:44.429369+0000","last_peered":"2024-09-17T05:20:44.429369+0000","last_clean":"2024-09-17T05:20:44.429369+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T05:20:44.429369+0000","last_undegraded":"2024-09-17T05:20:44.429369+0000","last_fullsized":"2024-09-17T05:20:44.429369+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:38.950013+0000","last_clean_scrub_stamp":"2024-09-17T04:49:38.950013+0000","objects_scrubbed":0,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:00.310958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":9969,"num_write":977,"num_write_kb":4635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"65'4230","reported_seq":4356,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.708264+0000","last_change":"2024-09-17T05:21:08.708264+0000","last_active":"2024-09-17T05:21:08.708264+0000","last_peered":"2024-09-17T05:21:08.708264+0000","last_clean":"2024-09-17T05:21:08.708264+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T05:21:08.708264+0000","last_undegraded":"2024-09-17T05:21:08.708264+0000","last_fullsized":"2024-09-17T05:21:08.708264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4230","last_scrub_stamp":"2024-09-17T05:21:08.708027+0000","last_deep_scrub":"65'4230","last_deep_scrub_stamp":"2024-09-17T05:21:08.708027+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.708027+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:45.642058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22609,"num_write_kb":319990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"65'3","reported_seq":131,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.258132+0000","last_change":"2024-09-17T05:21:05.258132+0000","last_active":"2024-09-17T05:21:05.258132+0000","last_peered":"2024-09-17T05:21:05.258132+0000","last_clean":"2024-09-17T05:21:05.258132+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T05:21:05.258132+0000","last_undegraded":"2024-09-17T05:21:05.258132+0000","last_fullsized":"2024-09-17T05:21:05.258132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-09-17T05:21:05.257912+0000","last_deep_scrub":"65'3","last_deep_scrub_stamp":"2024-09-17T05:21:05.257912+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.257912+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:11:50.135225+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"68'3526","reported_seq":3630,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393790+0000","last_change":"2024-09-17T04:50:12.072014+0000","last_active":"2024-09-17T05:20:43.393790+0000","last_peered":"2024-09-17T05:20:43.393790+0000","last_clean":"2024-09-17T05:20:43.393790+0000","last_became_active":"2024-09-17T04:50:12.071053+0000","last_became_peered":"2024-09-17T04:50:12.071053+0000","last_unstale":"2024-09-17T05:20:43.393790+0000","last_undegraded":"2024-09-17T05:20:43.393790+0000","last_fullsized":"2024-09-17T05:20:43.393790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:48.791055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20183,"num_write_kb":298078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"68'284","reported_seq":417,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.583278+0000","last_change":"2024-09-17T05:21:11.583278+0000","last_active":"2024-09-17T05:21:11.583278+0000","last_peered":"2024-09-17T05:21:11.583278+0000","last_clean":"2024-09-17T05:21:11.583278+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T05:21:11.583278+0000","last_undegraded":"2024-09-17T05:21:11.583278+0000","last_fullsized":"2024-09-17T05:21:11.583278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'284","last_scrub_stamp":"2024-09-17T05:21:11.582991+0000","last_deep_scrub":"68'284","last_deep_scrub_stamp":"2024-09-17T05:21:11.582991+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.582991+0000","objects_scrubbed":1,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:27:59.388174+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2099732,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":4077,"num_write":257,"num_write_kb":2207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'3509","reported_seq":3634,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.858178+0000","last_change":"2024-09-17T05:21:13.858178+0000","last_active":"2024-09-17T05:21:13.858178+0000","last_peered":"2024-09-17T05:21:13.858178+0000","last_clean":"2024-09-17T05:21:13.858178+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T05:21:13.858178+0000","last_undegraded":"2024-09-17T05:21:13.858178+0000","last_fullsized":"2024-09-17T05:21:13.858178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3509","last_scrub_stamp":"2024-09-17T05:21:13.857961+0000","last_deep_scrub":"65'3509","last_deep_scrub_stamp":"2024-09-17T05:21:13.857961+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.857961+0000","objects_scrubbed":0,"log_size":3509,"log_dups_size":0,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:26.384869+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18360,"num_write_kb":249727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'6","reported_seq":165,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.852925+0000","last_change":"2024-09-17T05:21:18.852925+0000","last_active":"2024-09-17T05:21:18.852925+0000","last_peered":"2024-09-17T05:21:18.852925+0000","last_clean":"2024-09-17T05:21:18.852925+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T05:21:18.852925+0000","last_undegraded":"2024-09-17T05:21:18.852925+0000","last_fullsized":"2024-09-17T05:21:18.852925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-17T05:21:18.852717+0000","last_deep_scrub":"68'6","last_deep_scrub_stamp":"2024-09-17T05:21:18.852717+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.852717+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:48.639506+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"68'3113","reported_seq":3233,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.797502+0000","last_change":"2024-09-17T05:21:18.797502+0000","last_active":"2024-09-17T05:21:18.797502+0000","last_peered":"2024-09-17T05:21:18.797502+0000","last_clean":"2024-09-17T05:21:18.797502+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T05:21:18.797502+0000","last_undegraded":"2024-09-17T05:21:18.797502+0000","last_fullsized":"2024-09-17T05:21:18.797502+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3113","last_scrub_stamp":"2024-09-17T05:21:18.797283+0000","last_deep_scrub":"68'3113","last_deep_scrub_stamp":"2024-09-17T05:21:18.797283+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.797283+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:56:29.820359+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":16978,"num_write_kb":256690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"68'658","reported_seq":809,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:04.677697+0000","last_change":"2024-09-17T05:21:04.677697+0000","last_active":"2024-09-17T05:21:04.677697+0000","last_peered":"2024-09-17T05:21:04.677697+0000","last_clean":"2024-09-17T05:21:04.677697+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T05:21:04.677697+0000","last_undegraded":"2024-09-17T05:21:04.677697+0000","last_fullsized":"2024-09-17T05:21:04.677697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'658","last_scrub_stamp":"2024-09-17T05:21:04.677438+0000","last_deep_scrub":"68'658","last_deep_scrub_stamp":"2024-09-17T05:21:04.677438+0000","last_clean_scrub_stamp":"2024-09-17T05:21:04.677438+0000","objects_scrubbed":1,"log_size":658,"log_dups_size":0,"ondisk_log_size":658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:18:09.565613+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":79257,"num_write":647,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'3199","reported_seq":3325,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:04.031322+0000","last_change":"2024-09-17T05:21:04.031322+0000","last_active":"2024-09-17T05:21:04.031322+0000","last_peered":"2024-09-17T05:21:04.031322+0000","last_clean":"2024-09-17T05:21:04.031322+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T05:21:04.031322+0000","last_undegraded":"2024-09-17T05:21:04.031322+0000","last_fullsized":"2024-09-17T05:21:04.031322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3199","last_scrub_stamp":"2024-09-17T05:21:04.031235+0000","last_deep_scrub":"65'3199","last_deep_scrub_stamp":"2024-09-17T05:21:04.031235+0000","last_clean_scrub_stamp":"2024-09-17T05:21:04.031235+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:56:53.162460+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16756,"num_write_kb":239128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"68'44","reported_seq":203,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.205031+0000","last_change":"2024-09-17T05:21:14.205031+0000","last_active":"2024-09-17T05:21:14.205031+0000","last_peered":"2024-09-17T05:21:14.205031+0000","last_clean":"2024-09-17T05:21:14.205031+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T05:21:14.205031+0000","last_undegraded":"2024-09-17T05:21:14.205031+0000","last_fullsized":"2024-09-17T05:21:14.205031+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'44","last_scrub_stamp":"2024-09-17T05:21:14.204793+0000","last_deep_scrub":"68'44","last_deep_scrub_stamp":"2024-09-17T05:21:14.204793+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.204793+0000","objects_scrubbed":1,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:50.269927+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"68'3014","reported_seq":3125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402775+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T05:20:43.402775+0000","last_peered":"2024-09-17T05:20:43.402775+0000","last_clean":"2024-09-17T05:20:43.402775+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T05:20:43.402775+0000","last_undegraded":"2024-09-17T05:20:43.402775+0000","last_fullsized":"2024-09-17T05:20:43.402775+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3014,"log_dups_size":0,"ondisk_log_size":3014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:22.956669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16762,"num_write_kb":251577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"65'3751","reported_seq":3879,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.719703+0000","last_change":"2024-09-17T05:21:06.719703+0000","last_active":"2024-09-17T05:21:06.719703+0000","last_peered":"2024-09-17T05:21:06.719703+0000","last_clean":"2024-09-17T05:21:06.719703+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T05:21:06.719703+0000","last_undegraded":"2024-09-17T05:21:06.719703+0000","last_fullsized":"2024-09-17T05:21:06.719703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3751","last_scrub_stamp":"2024-09-17T05:21:06.719435+0000","last_deep_scrub":"65'3751","last_deep_scrub_stamp":"2024-09-17T05:21:06.719435+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.719435+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:50.033505+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20040,"num_write_kb":277001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"68'3862","reported_seq":3974,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.271131+0000","last_change":"2024-09-17T04:49:41.057442+0000","last_active":"2024-09-17T05:20:43.271131+0000","last_peered":"2024-09-17T05:20:43.271131+0000","last_clean":"2024-09-17T05:20:43.271131+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T05:20:43.271131+0000","last_undegraded":"2024-09-17T05:20:43.271131+0000","last_fullsized":"2024-09-17T05:20:43.271131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:30.400182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20235,"num_write_kb":310497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"65'4193","reported_seq":4322,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.170827+0000","last_change":"2024-09-17T05:21:08.170827+0000","last_active":"2024-09-17T05:21:08.170827+0000","last_peered":"2024-09-17T05:21:08.170827+0000","last_clean":"2024-09-17T05:21:08.170827+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T05:21:08.170827+0000","last_undegraded":"2024-09-17T05:21:08.170827+0000","last_fullsized":"2024-09-17T05:21:08.170827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4193","last_scrub_stamp":"2024-09-17T05:21:08.170536+0000","last_deep_scrub":"65'4193","last_deep_scrub_stamp":"2024-09-17T05:21:08.170536+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.170536+0000","objects_scrubbed":0,"log_size":4193,"log_dups_size":0,"ondisk_log_size":4193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:28.895013+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22222,"num_write_kb":306243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"68'3519","reported_seq":3632,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.398772+0000","last_change":"2024-09-17T04:49:41.043352+0000","last_active":"2024-09-17T05:20:43.398772+0000","last_peered":"2024-09-17T05:20:43.398772+0000","last_clean":"2024-09-17T05:20:43.398772+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T05:20:43.398772+0000","last_undegraded":"2024-09-17T05:20:43.398772+0000","last_fullsized":"2024-09-17T05:20:43.398772+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3519,"log_dups_size":0,"ondisk_log_size":3519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":19548,"num_write_kb":295624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"65'3872","reported_seq":4002,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.520552+0000","last_change":"2024-09-17T05:21:20.520552+0000","last_active":"2024-09-17T05:21:20.520552+0000","last_peered":"2024-09-17T05:21:20.520552+0000","last_clean":"2024-09-17T05:21:20.520552+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T05:21:20.520552+0000","last_undegraded":"2024-09-17T05:21:20.520552+0000","last_fullsized":"2024-09-17T05:21:20.520552+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3872","last_scrub_stamp":"2024-09-17T05:21:20.520322+0000","last_deep_scrub":"65'3872","last_deep_scrub_stamp":"2024-09-17T05:21:20.520322+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.520322+0000","objects_scrubbed":0,"log_size":3872,"log_dups_size":0,"ondisk_log_size":3872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:55:14.848751+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20576,"num_write_kb":279241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"68'2938","reported_seq":3050,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.380635+0000","last_change":"2024-09-17T04:49:41.029754+0000","last_active":"2024-09-17T05:20:43.380635+0000","last_peered":"2024-09-17T05:20:43.380635+0000","last_clean":"2024-09-17T05:20:43.380635+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T05:20:43.380635+0000","last_undegraded":"2024-09-17T05:20:43.380635+0000","last_fullsized":"2024-09-17T05:20:43.380635+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2938,"log_dups_size":0,"ondisk_log_size":2938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:01.878911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":16647,"num_write_kb":271506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"65'3500","reported_seq":3629,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.645722+0000","last_change":"2024-09-17T05:21:10.645722+0000","last_active":"2024-09-17T05:21:10.645722+0000","last_peered":"2024-09-17T05:21:10.645722+0000","last_clean":"2024-09-17T05:21:10.645722+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T05:21:10.645722+0000","last_undegraded":"2024-09-17T05:21:10.645722+0000","last_fullsized":"2024-09-17T05:21:10.645722+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3500","last_scrub_stamp":"2024-09-17T05:21:10.645368+0000","last_deep_scrub":"65'3500","last_deep_scrub_stamp":"2024-09-17T05:21:10.645368+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.645368+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:46:40.477420+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18562,"num_write_kb":258704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"68'3339","reported_seq":3453,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.418584+0000","last_change":"2024-09-17T04:49:41.038767+0000","last_active":"2024-09-17T05:20:43.418584+0000","last_peered":"2024-09-17T05:20:43.418584+0000","last_clean":"2024-09-17T05:20:43.418584+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T05:20:43.418584+0000","last_undegraded":"2024-09-17T05:20:43.418584+0000","last_fullsized":"2024-09-17T05:20:43.418584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3339,"log_dups_size":0,"ondisk_log_size":3339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17940,"num_write_kb":305481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"65'3819","reported_seq":3947,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.210452+0000","last_change":"2024-09-17T05:21:16.210452+0000","last_active":"2024-09-17T05:21:16.210452+0000","last_peered":"2024-09-17T05:21:16.210452+0000","last_clean":"2024-09-17T05:21:16.210452+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T05:21:16.210452+0000","last_undegraded":"2024-09-17T05:21:16.210452+0000","last_fullsized":"2024-09-17T05:21:16.210452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3819","last_scrub_stamp":"2024-09-17T05:21:16.210076+0000","last_deep_scrub":"65'3819","last_deep_scrub_stamp":"2024-09-17T05:21:16.210076+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.210076+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:05.948248+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21007,"num_write_kb":273012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"68'3374","reported_seq":3501,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.755517+0000","last_change":"2024-09-17T05:21:19.755517+0000","last_active":"2024-09-17T05:21:19.755517+0000","last_peered":"2024-09-17T05:21:19.755517+0000","last_clean":"2024-09-17T05:21:19.755517+0000","last_became_active":"2024-09-17T04:50:12.075350+0000","last_became_peered":"2024-09-17T04:50:12.075350+0000","last_unstale":"2024-09-17T05:21:19.755517+0000","last_undegraded":"2024-09-17T05:21:19.755517+0000","last_fullsized":"2024-09-17T05:21:19.755517+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3374","last_scrub_stamp":"2024-09-17T05:21:19.755308+0000","last_deep_scrub":"68'3374","last_deep_scrub_stamp":"2024-09-17T05:21:19.755308+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.755308+0000","objects_scrubbed":0,"log_size":3374,"log_dups_size":0,"ondisk_log_size":3374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:59:24.554247+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18297,"num_write_kb":270032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"65'4233","reported_seq":4360,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.199080+0000","last_change":"2024-09-17T05:21:10.199080+0000","last_active":"2024-09-17T05:21:10.199080+0000","last_peered":"2024-09-17T05:21:10.199080+0000","last_clean":"2024-09-17T05:21:10.199080+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T05:21:10.199080+0000","last_undegraded":"2024-09-17T05:21:10.199080+0000","last_fullsized":"2024-09-17T05:21:10.199080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4233","last_scrub_stamp":"2024-09-17T05:21:10.198808+0000","last_deep_scrub":"65'4233","last_deep_scrub_stamp":"2024-09-17T05:21:10.198808+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.198808+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:29:50.002124+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23577,"num_write_kb":311694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"68'3382","reported_seq":3494,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.416880+0000","last_change":"2024-09-17T04:49:41.044885+0000","last_active":"2024-09-17T05:20:43.416880+0000","last_peered":"2024-09-17T05:20:43.416880+0000","last_clean":"2024-09-17T05:20:43.416880+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T05:20:43.416880+0000","last_undegraded":"2024-09-17T05:20:43.416880+0000","last_fullsized":"2024-09-17T05:20:43.416880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:41.618087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18178,"num_write_kb":293977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"65'3778","reported_seq":3905,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.796418+0000","last_change":"2024-09-17T05:21:08.796418+0000","last_active":"2024-09-17T05:21:08.796418+0000","last_peered":"2024-09-17T05:21:08.796418+0000","last_clean":"2024-09-17T05:21:08.796418+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T05:21:08.796418+0000","last_undegraded":"2024-09-17T05:21:08.796418+0000","last_fullsized":"2024-09-17T05:21:08.796418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3778","last_scrub_stamp":"2024-09-17T05:21:08.796155+0000","last_deep_scrub":"65'3778","last_deep_scrub_stamp":"2024-09-17T05:21:08.796155+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.796155+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:53.442168+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21044,"num_write_kb":281017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"68'2956","reported_seq":3071,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404554+0000","last_change":"2024-09-17T04:49:41.050412+0000","last_active":"2024-09-17T05:20:44.404554+0000","last_peered":"2024-09-17T05:20:44.404554+0000","last_clean":"2024-09-17T05:20:44.404554+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T05:20:44.404554+0000","last_undegraded":"2024-09-17T05:20:44.404554+0000","last_fullsized":"2024-09-17T05:20:44.404554+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2956,"log_dups_size":0,"ondisk_log_size":2956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:10.630787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15875,"num_write_kb":264255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"65'4647","reported_seq":4775,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.687332+0000","last_change":"2024-09-17T05:21:14.687332+0000","last_active":"2024-09-17T05:21:14.687332+0000","last_peered":"2024-09-17T05:21:14.687332+0000","last_clean":"2024-09-17T05:21:14.687332+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T05:21:14.687332+0000","last_undegraded":"2024-09-17T05:21:14.687332+0000","last_fullsized":"2024-09-17T05:21:14.687332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4647","last_scrub_stamp":"2024-09-17T05:21:14.687109+0000","last_deep_scrub":"65'4647","last_deep_scrub_stamp":"2024-09-17T05:21:14.687109+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.687109+0000","objects_scrubbed":0,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:04.899852+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24699,"num_write_kb":336693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"68'3095","reported_seq":3218,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.307187+0000","last_change":"2024-09-17T05:21:19.307187+0000","last_active":"2024-09-17T05:21:19.307187+0000","last_peered":"2024-09-17T05:21:19.307187+0000","last_clean":"2024-09-17T05:21:19.307187+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T05:21:19.307187+0000","last_undegraded":"2024-09-17T05:21:19.307187+0000","last_fullsized":"2024-09-17T05:21:19.307187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3095","last_scrub_stamp":"2024-09-17T05:21:19.306844+0000","last_deep_scrub":"68'3095","last_deep_scrub_stamp":"2024-09-17T05:21:19.306844+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.306844+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:32.552770+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":36,"num_write":16750,"num_write_kb":272041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"65'4070","reported_seq":4198,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.546705+0000","last_change":"2024-09-17T05:21:18.546705+0000","last_active":"2024-09-17T05:21:18.546705+0000","last_peered":"2024-09-17T05:21:18.546705+0000","last_clean":"2024-09-17T05:21:18.546705+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T05:21:18.546705+0000","last_undegraded":"2024-09-17T05:21:18.546705+0000","last_fullsized":"2024-09-17T05:21:18.546705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4070","last_scrub_stamp":"2024-09-17T05:21:18.546455+0000","last_deep_scrub":"65'4070","last_deep_scrub_stamp":"2024-09-17T05:21:18.546455+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.546455+0000","objects_scrubbed":0,"log_size":4070,"log_dups_size":0,"ondisk_log_size":4070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:10.791521+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21906,"num_write_kb":288644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"68'4276","reported_seq":4387,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.404966+0000","last_change":"2024-09-17T04:49:41.052001+0000","last_active":"2024-09-17T05:20:44.404966+0000","last_peered":"2024-09-17T05:20:44.404966+0000","last_clean":"2024-09-17T05:20:44.404966+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T05:20:44.404966+0000","last_undegraded":"2024-09-17T05:20:44.404966+0000","last_fullsized":"2024-09-17T05:20:44.404966+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:17.390300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22732,"num_write_kb":365385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"65'4174","reported_seq":4301,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.688484+0000","last_change":"2024-09-17T05:21:17.688484+0000","last_active":"2024-09-17T05:21:17.688484+0000","last_peered":"2024-09-17T05:21:17.688484+0000","last_clean":"2024-09-17T05:21:17.688484+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T05:21:17.688484+0000","last_undegraded":"2024-09-17T05:21:17.688484+0000","last_fullsized":"2024-09-17T05:21:17.688484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4174","last_scrub_stamp":"2024-09-17T05:21:17.688135+0000","last_deep_scrub":"65'4174","last_deep_scrub_stamp":"2024-09-17T05:21:17.688135+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.688135+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:46.578417+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22870,"num_write_kb":296621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"68'2821","reported_seq":2932,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.414070+0000","last_change":"2024-09-17T04:49:41.028478+0000","last_active":"2024-09-17T05:20:44.414070+0000","last_peered":"2024-09-17T05:20:44.414070+0000","last_clean":"2024-09-17T05:20:44.414070+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T05:20:44.414070+0000","last_undegraded":"2024-09-17T05:20:44.414070+0000","last_fullsized":"2024-09-17T05:20:44.414070+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2821,"log_dups_size":0,"ondisk_log_size":2821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15110,"num_write_kb":232606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"65'4280","reported_seq":4407,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.072977+0000","last_change":"2024-09-17T05:21:08.072977+0000","last_active":"2024-09-17T05:21:08.072977+0000","last_peered":"2024-09-17T05:21:08.072977+0000","last_clean":"2024-09-17T05:21:08.072977+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T05:21:08.072977+0000","last_undegraded":"2024-09-17T05:21:08.072977+0000","last_fullsized":"2024-09-17T05:21:08.072977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4280","last_scrub_stamp":"2024-09-17T05:21:08.072718+0000","last_deep_scrub":"65'4280","last_deep_scrub_stamp":"2024-09-17T05:21:08.072718+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.072718+0000","objects_scrubbed":0,"log_size":4280,"log_dups_size":0,"ondisk_log_size":4280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:03:13.177867+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23007,"num_write_kb":313406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"68'3452","reported_seq":3571,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.277348+0000","last_change":"2024-09-17T05:21:18.277177+0000","last_active":"2024-09-17T05:21:18.277348+0000","last_peered":"2024-09-17T05:21:18.277348+0000","last_clean":"2024-09-17T05:21:18.277348+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T05:21:18.277348+0000","last_undegraded":"2024-09-17T05:21:18.277348+0000","last_fullsized":"2024-09-17T05:21:18.277348+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3452","last_scrub_stamp":"2024-09-17T05:21:18.276964+0000","last_deep_scrub":"68'3452","last_deep_scrub_stamp":"2024-09-17T05:21:18.276964+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.276964+0000","objects_scrubbed":0,"log_size":3452,"log_dups_size":0,"ondisk_log_size":3452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:18:56.568305+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19160,"num_write_kb":306156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"65'4384","reported_seq":4513,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.854162+0000","last_change":"2024-09-17T05:21:14.854162+0000","last_active":"2024-09-17T05:21:14.854162+0000","last_peered":"2024-09-17T05:21:14.854162+0000","last_clean":"2024-09-17T05:21:14.854162+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T05:21:14.854162+0000","last_undegraded":"2024-09-17T05:21:14.854162+0000","last_fullsized":"2024-09-17T05:21:14.854162+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4384","last_scrub_stamp":"2024-09-17T05:21:14.853940+0000","last_deep_scrub":"65'4384","last_deep_scrub_stamp":"2024-09-17T05:21:14.853940+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.853940+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:48.942985+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23582,"num_write_kb":336581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"68'3472","reported_seq":3590,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.394078+0000","last_change":"2024-09-17T04:50:12.068348+0000","last_active":"2024-09-17T05:20:43.394078+0000","last_peered":"2024-09-17T05:20:43.394078+0000","last_clean":"2024-09-17T05:20:43.394078+0000","last_became_active":"2024-09-17T04:50:12.067942+0000","last_became_peered":"2024-09-17T04:50:12.067942+0000","last_unstale":"2024-09-17T05:20:43.394078+0000","last_undegraded":"2024-09-17T05:20:43.394078+0000","last_fullsized":"2024-09-17T05:20:43.394078+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3472,"log_dups_size":0,"ondisk_log_size":3472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18777,"num_write_kb":292458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"65'3688","reported_seq":3815,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.757877+0000","last_change":"2024-09-17T05:21:10.757877+0000","last_active":"2024-09-17T05:21:10.757877+0000","last_peered":"2024-09-17T05:21:10.757877+0000","last_clean":"2024-09-17T05:21:10.757877+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T05:21:10.757877+0000","last_undegraded":"2024-09-17T05:21:10.757877+0000","last_fullsized":"2024-09-17T05:21:10.757877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3688","last_scrub_stamp":"2024-09-17T05:21:10.757653+0000","last_deep_scrub":"65'3688","last_deep_scrub_stamp":"2024-09-17T05:21:10.757653+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.757653+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:54.603536+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20358,"num_write_kb":257774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"68'3214","reported_seq":3338,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:22.708445+0000","last_change":"2024-09-17T05:21:22.708445+0000","last_active":"2024-09-17T05:21:22.708445+0000","last_peered":"2024-09-17T05:21:22.708445+0000","last_clean":"2024-09-17T05:21:22.708445+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T05:21:22.708445+0000","last_undegraded":"2024-09-17T05:21:22.708445+0000","last_fullsized":"2024-09-17T05:21:22.708445+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3214","last_scrub_stamp":"2024-09-17T05:21:22.708216+0000","last_deep_scrub":"68'3214","last_deep_scrub_stamp":"2024-09-17T05:21:22.708216+0000","last_clean_scrub_stamp":"2024-09-17T05:21:22.708216+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:08:38.989960+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":17543,"num_write_kb":304029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"65'4068","reported_seq":4187,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.374537+0000","last_change":"2024-09-17T04:49:31.906827+0000","last_active":"2024-09-17T05:20:36.374537+0000","last_peered":"2024-09-17T05:20:36.374537+0000","last_clean":"2024-09-17T05:20:36.374537+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T05:20:36.374537+0000","last_undegraded":"2024-09-17T05:20:36.374537+0000","last_fullsized":"2024-09-17T05:20:36.374537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21759,"num_write_kb":308395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"68'2551","reported_seq":2663,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.402052+0000","last_change":"2024-09-17T04:49:41.067006+0000","last_active":"2024-09-17T05:20:43.402052+0000","last_peered":"2024-09-17T05:20:43.402052+0000","last_clean":"2024-09-17T05:20:43.402052+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T05:20:43.402052+0000","last_undegraded":"2024-09-17T05:20:43.402052+0000","last_fullsized":"2024-09-17T05:20:43.402052+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2551,"log_dups_size":0,"ondisk_log_size":2551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:05.016838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13664,"num_write_kb":247232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"65'4369","reported_seq":4496,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.171692+0000","last_change":"2024-09-17T05:21:11.171692+0000","last_active":"2024-09-17T05:21:11.171692+0000","last_peered":"2024-09-17T05:21:11.171692+0000","last_clean":"2024-09-17T05:21:11.171692+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T05:21:11.171692+0000","last_undegraded":"2024-09-17T05:21:11.171692+0000","last_fullsized":"2024-09-17T05:21:11.171692+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4369","last_scrub_stamp":"2024-09-17T05:21:11.171467+0000","last_deep_scrub":"65'4369","last_deep_scrub_stamp":"2024-09-17T05:21:11.171467+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.171467+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:16:17.551506+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23171,"num_write_kb":318257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"68'3277","reported_seq":3389,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.407229+0000","last_change":"2024-09-17T04:49:41.027025+0000","last_active":"2024-09-17T05:20:43.407229+0000","last_peered":"2024-09-17T05:20:43.407229+0000","last_clean":"2024-09-17T05:20:43.407229+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T05:20:43.407229+0000","last_undegraded":"2024-09-17T05:20:43.407229+0000","last_fullsized":"2024-09-17T05:20:43.407229+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18007,"num_write_kb":264733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"65'5432","reported_seq":5550,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.366356+0000","last_change":"2024-09-17T04:49:31.905893+0000","last_active":"2024-09-17T05:20:36.366356+0000","last_peered":"2024-09-17T05:20:36.366356+0000","last_clean":"2024-09-17T05:20:36.366356+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T05:20:36.366356+0000","last_undegraded":"2024-09-17T05:20:36.366356+0000","last_fullsized":"2024-09-17T05:20:36.366356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:23.511207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29730,"num_write_kb":389609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"68'3135","reported_seq":3247,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.410617+0000","last_change":"2024-09-17T04:49:41.043018+0000","last_active":"2024-09-17T05:20:43.410617+0000","last_peered":"2024-09-17T05:20:43.410617+0000","last_clean":"2024-09-17T05:20:43.410617+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T05:20:43.410617+0000","last_undegraded":"2024-09-17T05:20:43.410617+0000","last_fullsized":"2024-09-17T05:20:43.410617+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":16884,"num_write_kb":284681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"65'3941","reported_seq":4059,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.360280+0000","last_change":"2024-09-17T04:49:31.899027+0000","last_active":"2024-09-17T05:20:36.360280+0000","last_peered":"2024-09-17T05:20:36.360280+0000","last_clean":"2024-09-17T05:20:36.360280+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T05:20:36.360280+0000","last_undegraded":"2024-09-17T05:20:36.360280+0000","last_fullsized":"2024-09-17T05:20:36.360280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:43.454540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21845,"num_write_kb":304681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"68'3913","reported_seq":4035,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.033178+0000","last_change":"2024-09-17T05:21:13.033178+0000","last_active":"2024-09-17T05:21:13.033178+0000","last_peered":"2024-09-17T05:21:13.033178+0000","last_clean":"2024-09-17T05:21:13.033178+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T05:21:13.033178+0000","last_undegraded":"2024-09-17T05:21:13.033178+0000","last_fullsized":"2024-09-17T05:21:13.033178+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3913","last_scrub_stamp":"2024-09-17T05:21:13.032954+0000","last_deep_scrub":"68'3913","last_deep_scrub_stamp":"2024-09-17T05:21:13.032954+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.032954+0000","objects_scrubbed":0,"log_size":3913,"log_dups_size":0,"ondisk_log_size":3913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:43:50.981693+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":21530,"num_write_kb":364884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"65'3351","reported_seq":3483,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.687972+0000","last_change":"2024-09-17T05:21:15.687972+0000","last_active":"2024-09-17T05:21:15.687972+0000","last_peered":"2024-09-17T05:21:15.687972+0000","last_clean":"2024-09-17T05:21:15.687972+0000","last_became_active":"2024-09-17T04:50:12.469425+0000","last_became_peered":"2024-09-17T04:50:12.469425+0000","last_unstale":"2024-09-17T05:21:15.687972+0000","last_undegraded":"2024-09-17T05:21:15.687972+0000","last_fullsized":"2024-09-17T05:21:15.687972+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3351","last_scrub_stamp":"2024-09-17T05:21:15.687734+0000","last_deep_scrub":"65'3351","last_deep_scrub_stamp":"2024-09-17T05:21:15.687734+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.687734+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:27.678211+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18172,"num_write_kb":265354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"68'1954","reported_seq":2075,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.072384+0000","last_change":"2024-09-17T05:21:11.072384+0000","last_active":"2024-09-17T05:21:11.072384+0000","last_peered":"2024-09-17T05:21:11.072384+0000","last_clean":"2024-09-17T05:21:11.072384+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T05:21:11.072384+0000","last_undegraded":"2024-09-17T05:21:11.072384+0000","last_fullsized":"2024-09-17T05:21:11.072384+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1954","last_scrub_stamp":"2024-09-17T05:21:11.072003+0000","last_deep_scrub":"68'1954","last_deep_scrub_stamp":"2024-09-17T05:21:11.072003+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.072003+0000","objects_scrubbed":0,"log_size":1954,"log_dups_size":0,"ondisk_log_size":1954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:10.021708+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":34,"num_write":10478,"num_write_kb":181241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"65'3661","reported_seq":3787,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.237594+0000","last_change":"2024-09-17T05:21:11.237594+0000","last_active":"2024-09-17T05:21:11.237594+0000","last_peered":"2024-09-17T05:21:11.237594+0000","last_clean":"2024-09-17T05:21:11.237594+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T05:21:11.237594+0000","last_undegraded":"2024-09-17T05:21:11.237594+0000","last_fullsized":"2024-09-17T05:21:11.237594+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3661","last_scrub_stamp":"2024-09-17T05:21:11.237338+0000","last_deep_scrub":"65'3661","last_deep_scrub_stamp":"2024-09-17T05:21:11.237338+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.237338+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:42.553058+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19087,"num_write_kb":273280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"68'2981","reported_seq":3094,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.264645+0000","last_change":"2024-09-17T04:49:41.067947+0000","last_active":"2024-09-17T05:20:43.264645+0000","last_peered":"2024-09-17T05:20:43.264645+0000","last_clean":"2024-09-17T05:20:43.264645+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T05:20:43.264645+0000","last_undegraded":"2024-09-17T05:20:43.264645+0000","last_fullsized":"2024-09-17T05:20:43.264645+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":15573,"num_write_kb":271248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"65'3297","reported_seq":3423,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.224870+0000","last_change":"2024-09-17T05:21:13.224870+0000","last_active":"2024-09-17T05:21:13.224870+0000","last_peered":"2024-09-17T05:21:13.224870+0000","last_clean":"2024-09-17T05:21:13.224870+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T05:21:13.224870+0000","last_undegraded":"2024-09-17T05:21:13.224870+0000","last_fullsized":"2024-09-17T05:21:13.224870+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3297","last_scrub_stamp":"2024-09-17T05:21:13.224541+0000","last_deep_scrub":"65'3297","last_deep_scrub_stamp":"2024-09-17T05:21:13.224541+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.224541+0000","objects_scrubbed":0,"log_size":3297,"log_dups_size":0,"ondisk_log_size":3297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:50.433999+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17851,"num_write_kb":243868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"68'2872","reported_seq":2983,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.403678+0000","last_change":"2024-09-17T04:49:41.042018+0000","last_active":"2024-09-17T05:20:44.403678+0000","last_peered":"2024-09-17T05:20:44.403678+0000","last_clean":"2024-09-17T05:20:44.403678+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T05:20:44.403678+0000","last_undegraded":"2024-09-17T05:20:44.403678+0000","last_fullsized":"2024-09-17T05:20:44.403678+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2872,"log_dups_size":0,"ondisk_log_size":2872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:01.320551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15784,"num_write_kb":259984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"65'2574","reported_seq":2700,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.715641+0000","last_change":"2024-09-17T05:21:11.715641+0000","last_active":"2024-09-17T05:21:11.715641+0000","last_peered":"2024-09-17T05:21:11.715641+0000","last_clean":"2024-09-17T05:21:11.715641+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T05:21:11.715641+0000","last_undegraded":"2024-09-17T05:21:11.715641+0000","last_fullsized":"2024-09-17T05:21:11.715641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2574","last_scrub_stamp":"2024-09-17T05:21:11.714891+0000","last_deep_scrub":"65'2574","last_deep_scrub_stamp":"2024-09-17T05:21:11.714891+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.714891+0000","objects_scrubbed":0,"log_size":2574,"log_dups_size":0,"ondisk_log_size":2574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:33:27.672830+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14007,"num_write_kb":203729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"68'2459","reported_seq":2573,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.404946+0000","last_change":"2024-09-17T04:49:41.059456+0000","last_active":"2024-09-17T05:20:43.404946+0000","last_peered":"2024-09-17T05:20:43.404946+0000","last_clean":"2024-09-17T05:20:43.404946+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T05:20:43.404946+0000","last_undegraded":"2024-09-17T05:20:43.404946+0000","last_fullsized":"2024-09-17T05:20:43.404946+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2459,"log_dups_size":0,"ondisk_log_size":2459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13572,"num_write_kb":222508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"65'3245","reported_seq":3371,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.722325+0000","last_change":"2024-09-17T05:21:15.722325+0000","last_active":"2024-09-17T05:21:15.722325+0000","last_peered":"2024-09-17T05:21:15.722325+0000","last_clean":"2024-09-17T05:21:15.722325+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T05:21:15.722325+0000","last_undegraded":"2024-09-17T05:21:15.722325+0000","last_fullsized":"2024-09-17T05:21:15.722325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3245","last_scrub_stamp":"2024-09-17T05:21:15.722102+0000","last_deep_scrub":"65'3245","last_deep_scrub_stamp":"2024-09-17T05:21:15.722102+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.722102+0000","objects_scrubbed":0,"log_size":3245,"log_dups_size":0,"ondisk_log_size":3245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:45.255095+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16903,"num_write_kb":256080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"68'3074","reported_seq":3184,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.393584+0000","last_change":"2024-09-17T04:49:41.042606+0000","last_active":"2024-09-17T05:20:43.393584+0000","last_peered":"2024-09-17T05:20:43.393584+0000","last_clean":"2024-09-17T05:20:43.393584+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T05:20:43.393584+0000","last_undegraded":"2024-09-17T05:20:43.393584+0000","last_fullsized":"2024-09-17T05:20:43.393584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:33.760846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16752,"num_write_kb":271957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"65'4387","reported_seq":4513,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.766348+0000","last_change":"2024-09-17T05:21:14.766348+0000","last_active":"2024-09-17T05:21:14.766348+0000","last_peered":"2024-09-17T05:21:14.766348+0000","last_clean":"2024-09-17T05:21:14.766348+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T05:21:14.766348+0000","last_undegraded":"2024-09-17T05:21:14.766348+0000","last_fullsized":"2024-09-17T05:21:14.766348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4387","last_scrub_stamp":"2024-09-17T05:21:14.765991+0000","last_deep_scrub":"65'4387","last_deep_scrub_stamp":"2024-09-17T05:21:14.765991+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.765991+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:38.768117+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24001,"num_write_kb":317505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"68'2713","reported_seq":2824,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.417489+0000","last_change":"2024-09-17T04:49:41.051809+0000","last_active":"2024-09-17T05:20:43.417489+0000","last_peered":"2024-09-17T05:20:43.417489+0000","last_clean":"2024-09-17T05:20:43.417489+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T05:20:43.417489+0000","last_undegraded":"2024-09-17T05:20:43.417489+0000","last_fullsized":"2024-09-17T05:20:43.417489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2713,"log_dups_size":0,"ondisk_log_size":2713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14521,"num_write_kb":262890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"65'4656","reported_seq":4784,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.066809+0000","last_change":"2024-09-17T05:21:06.066809+0000","last_active":"2024-09-17T05:21:06.066809+0000","last_peered":"2024-09-17T05:21:06.066809+0000","last_clean":"2024-09-17T05:21:06.066809+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T05:21:06.066809+0000","last_undegraded":"2024-09-17T05:21:06.066809+0000","last_fullsized":"2024-09-17T05:21:06.066809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4656","last_scrub_stamp":"2024-09-17T05:21:06.066440+0000","last_deep_scrub":"65'4656","last_deep_scrub_stamp":"2024-09-17T05:21:06.066440+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.066440+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:20:06.012799+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25090,"num_write_kb":362900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"68'3588","reported_seq":3702,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.402501+0000","last_change":"2024-09-17T04:49:41.054754+0000","last_active":"2024-09-17T05:20:44.402501+0000","last_peered":"2024-09-17T05:20:44.402501+0000","last_clean":"2024-09-17T05:20:44.402501+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T05:20:44.402501+0000","last_undegraded":"2024-09-17T05:20:44.402501+0000","last_fullsized":"2024-09-17T05:20:44.402501+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3588,"log_dups_size":0,"ondisk_log_size":3588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:00.149314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":19474,"num_write_kb":319707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"65'4149","reported_seq":4274,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.712006+0000","last_change":"2024-09-17T05:21:16.712006+0000","last_active":"2024-09-17T05:21:16.712006+0000","last_peered":"2024-09-17T05:21:16.712006+0000","last_clean":"2024-09-17T05:21:16.712006+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T05:21:16.712006+0000","last_undegraded":"2024-09-17T05:21:16.712006+0000","last_fullsized":"2024-09-17T05:21:16.712006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4149","last_scrub_stamp":"2024-09-17T05:21:16.711753+0000","last_deep_scrub":"65'4149","last_deep_scrub_stamp":"2024-09-17T05:21:16.711753+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.711753+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:24:32.998820+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22453,"num_write_kb":298991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"68'3299","reported_seq":3415,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.422884+0000","last_change":"2024-09-17T04:49:41.050936+0000","last_active":"2024-09-17T05:20:44.422884+0000","last_peered":"2024-09-17T05:20:44.422884+0000","last_clean":"2024-09-17T05:20:44.422884+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T05:20:44.422884+0000","last_undegraded":"2024-09-17T05:20:44.422884+0000","last_fullsized":"2024-09-17T05:20:44.422884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3299,"log_dups_size":0,"ondisk_log_size":3299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:00.686263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":17863,"num_write_kb":288721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"65'3910","reported_seq":4035,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.693225+0000","last_change":"2024-09-17T05:21:17.693225+0000","last_active":"2024-09-17T05:21:17.693225+0000","last_peered":"2024-09-17T05:21:17.693225+0000","last_clean":"2024-09-17T05:21:17.693225+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T05:21:17.693225+0000","last_undegraded":"2024-09-17T05:21:17.693225+0000","last_fullsized":"2024-09-17T05:21:17.693225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3910","last_scrub_stamp":"2024-09-17T05:21:17.692969+0000","last_deep_scrub":"65'3910","last_deep_scrub_stamp":"2024-09-17T05:21:17.692969+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.692969+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:57:56.576535+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21371,"num_write_kb":294687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"68'2856","reported_seq":2969,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.325969+0000","last_change":"2024-09-17T04:49:41.052121+0000","last_active":"2024-09-17T05:20:43.325969+0000","last_peered":"2024-09-17T05:20:43.325969+0000","last_clean":"2024-09-17T05:20:43.325969+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T05:20:43.325969+0000","last_undegraded":"2024-09-17T05:20:43.325969+0000","last_fullsized":"2024-09-17T05:20:43.325969+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.954948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15916,"num_write_kb":235941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"65'3148","reported_seq":3273,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.711504+0000","last_change":"2024-09-17T05:21:05.711504+0000","last_active":"2024-09-17T05:21:05.711504+0000","last_peered":"2024-09-17T05:21:05.711504+0000","last_clean":"2024-09-17T05:21:05.711504+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T05:21:05.711504+0000","last_undegraded":"2024-09-17T05:21:05.711504+0000","last_fullsized":"2024-09-17T05:21:05.711504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3148","last_scrub_stamp":"2024-09-17T05:21:05.711283+0000","last_deep_scrub":"65'3148","last_deep_scrub_stamp":"2024-09-17T05:21:05.711283+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.711283+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:15.059071+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16748,"num_write_kb":239238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"68'3516","reported_seq":3638,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.997989+0000","last_change":"2024-09-17T05:21:15.997989+0000","last_active":"2024-09-17T05:21:15.997989+0000","last_peered":"2024-09-17T05:21:15.997989+0000","last_clean":"2024-09-17T05:21:15.997989+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T05:21:15.997989+0000","last_undegraded":"2024-09-17T05:21:15.997989+0000","last_fullsized":"2024-09-17T05:21:15.997989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3516","last_scrub_stamp":"2024-09-17T05:21:15.997616+0000","last_deep_scrub":"68'3516","last_deep_scrub_stamp":"2024-09-17T05:21:15.997616+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.997616+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:22.364455+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18785,"num_write_kb":280186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"65'3294","reported_seq":3419,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.733179+0000","last_change":"2024-09-17T05:21:13.733179+0000","last_active":"2024-09-17T05:21:13.733179+0000","last_peered":"2024-09-17T05:21:13.733179+0000","last_clean":"2024-09-17T05:21:13.733179+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T05:21:13.733179+0000","last_undegraded":"2024-09-17T05:21:13.733179+0000","last_fullsized":"2024-09-17T05:21:13.733179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3294","last_scrub_stamp":"2024-09-17T05:21:13.732955+0000","last_deep_scrub":"65'3294","last_deep_scrub_stamp":"2024-09-17T05:21:13.732955+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.732955+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:04:21.983038+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16934,"num_write_kb":260475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"68'2875","reported_seq":2993,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.324899+0000","last_change":"2024-09-17T05:21:20.324899+0000","last_active":"2024-09-17T05:21:20.324899+0000","last_peered":"2024-09-17T05:21:20.324899+0000","last_clean":"2024-09-17T05:21:20.324899+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T05:21:20.324899+0000","last_undegraded":"2024-09-17T05:21:20.324899+0000","last_fullsized":"2024-09-17T05:21:20.324899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2875","last_scrub_stamp":"2024-09-17T05:21:20.324678+0000","last_deep_scrub":"68'2875","last_deep_scrub_stamp":"2024-09-17T05:21:20.324678+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.324678+0000","objects_scrubbed":0,"log_size":2875,"log_dups_size":0,"ondisk_log_size":2875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:38.100241+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15268,"num_write_kb":241386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"65'3071","reported_seq":3200,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.768226+0000","last_change":"2024-09-17T05:21:09.768226+0000","last_active":"2024-09-17T05:21:09.768226+0000","last_peered":"2024-09-17T05:21:09.768226+0000","last_clean":"2024-09-17T05:21:09.768226+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T05:21:09.768226+0000","last_undegraded":"2024-09-17T05:21:09.768226+0000","last_fullsized":"2024-09-17T05:21:09.768226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3071","last_scrub_stamp":"2024-09-17T05:21:09.767968+0000","last_deep_scrub":"65'3071","last_deep_scrub_stamp":"2024-09-17T05:21:09.767968+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.767968+0000","objects_scrubbed":0,"log_size":3071,"log_dups_size":0,"ondisk_log_size":3071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:53.613852+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16593,"num_write_kb":218157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"68'2617","reported_seq":2728,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.405858+0000","last_change":"2024-09-17T04:49:41.034857+0000","last_active":"2024-09-17T05:20:43.405858+0000","last_peered":"2024-09-17T05:20:43.405858+0000","last_clean":"2024-09-17T05:20:43.405858+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T05:20:43.405858+0000","last_undegraded":"2024-09-17T05:20:43.405858+0000","last_fullsized":"2024-09-17T05:20:43.405858+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:46.378102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3,"num_write":13428,"num_write_kb":250535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"65'3866","reported_seq":3987,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:36.352182+0000","last_change":"2024-09-17T04:49:31.882978+0000","last_active":"2024-09-17T05:20:36.352182+0000","last_peered":"2024-09-17T05:20:36.352182+0000","last_clean":"2024-09-17T05:20:36.352182+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T05:20:36.352182+0000","last_undegraded":"2024-09-17T05:20:36.352182+0000","last_fullsized":"2024-09-17T05:20:36.352182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:30.837284+0000","last_clean_scrub_stamp":"2024-09-17T04:49:30.837284+0000","objects_scrubbed":0,"log_size":3866,"log_dups_size":0,"ondisk_log_size":3866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:24.164932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20832,"num_write_kb":272078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"68'3056","reported_seq":3165,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:43.326456+0000","last_change":"2024-09-17T04:49:41.049988+0000","last_active":"2024-09-17T05:20:43.326456+0000","last_peered":"2024-09-17T05:20:43.326456+0000","last_clean":"2024-09-17T05:20:43.326456+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T05:20:43.326456+0000","last_undegraded":"2024-09-17T05:20:43.326456+0000","last_fullsized":"2024-09-17T05:20:43.326456+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3056,"log_dups_size":0,"ondisk_log_size":3056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:22.211991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16712,"num_write_kb":275987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"65'4485","reported_seq":4613,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.674581+0000","last_change":"2024-09-17T05:21:18.674581+0000","last_active":"2024-09-17T05:21:18.674581+0000","last_peered":"2024-09-17T05:21:18.674581+0000","last_clean":"2024-09-17T05:21:18.674581+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T05:21:18.674581+0000","last_undegraded":"2024-09-17T05:21:18.674581+0000","last_fullsized":"2024-09-17T05:21:18.674581+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4485","last_scrub_stamp":"2024-09-17T05:21:18.674357+0000","last_deep_scrub":"65'4485","last_deep_scrub_stamp":"2024-09-17T05:21:18.674357+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.674357+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:06.720069+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23824,"num_write_kb":337279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"68'3229","reported_seq":3338,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:20:44.420569+0000","last_change":"2024-09-17T04:49:41.043749+0000","last_active":"2024-09-17T05:20:44.420569+0000","last_peered":"2024-09-17T05:20:44.420569+0000","last_clean":"2024-09-17T05:20:44.420569+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T05:20:44.420569+0000","last_undegraded":"2024-09-17T05:20:44.420569+0000","last_fullsized":"2024-09-17T05:20:44.420569+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:49:39.959607+0000","last_clean_scrub_stamp":"2024-09-17T04:49:39.959607+0000","objects_scrubbed":0,"log_size":3229,"log_dups_size":0,"ondisk_log_size":3229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:02.371158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17855,"num_write_kb":268347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"65'3523","reported_seq":3652,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.736398+0000","last_change":"2024-09-17T05:21:18.736398+0000","last_active":"2024-09-17T05:21:18.736398+0000","last_peered":"2024-09-17T05:21:18.736398+0000","last_clean":"2024-09-17T05:21:18.736398+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T05:21:18.736398+0000","last_undegraded":"2024-09-17T05:21:18.736398+0000","last_fullsized":"2024-09-17T05:21:18.736398+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3523","last_scrub_stamp":"2024-09-17T05:21:18.736157+0000","last_deep_scrub":"65'3523","last_deep_scrub_stamp":"2024-09-17T05:21:18.736157+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.736157+0000","objects_scrubbed":0,"log_size":3523,"log_dups_size":0,"ondisk_log_size":3523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:52.825202+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19890,"num_write_kb":260121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":321,"num_write":1077532,"num_write_kb":17428719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":198824,"ondisk_log_size":198824,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":18310618,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1391,"num_read_kb":207657,"num_write":4849,"num_write_kb":19832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36683776,"data_stored":36621236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4453,"ondisk_log_size":4453,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1352682,"num_write_kb":18536494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250516,"ondisk_log_size":250516,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":18963064,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1535,"num_read_kb":230833,"num_write":4911,"num_write_kb":20583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37986304,"data_stored":37926128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4525,"ondisk_log_size":4525,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509828,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533064,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":532094,"kb_avail":93838776,"statfs":{"total":96636764160,"available":96090906624,"internally_reserved":0,"allocated":974848,"data_stored":552256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":544864714},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509828,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762124,"kb_used_data":17916,"kb_used_omap":1,"kb_used_meta":744190,"kb_avail":93609716,"statfs":{"total":96636764160,"available":95856349184,"internally_reserved":0,"allocated":18345984,"data_stored":17920368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":762051015},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509828,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760196,"kb_used_data":17268,"kb_used_omap":1,"kb_used_meta":742910,"kb_avail":93611644,"statfs":{"total":96636764160,"available":95858323456,"internally_reserved":0,"allocated":17682432,"data_stored":17246718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":760740293},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509828,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755412,"kb_used_data":15300,"kb_used_omap":1,"kb_used_meta":740094,"kb_avail":93616428,"statfs":{"total":96636764160,"available":95863222272,"internally_reserved":0,"allocated":15667200,"data_stored":15236011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":757856711},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509828,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877392,"kb_used_data":14656,"kb_used_omap":1,"kb_used_meta":862718,"kb_avail":93494448,"statfs":{"total":96636764160,"available":95738314752,"internally_reserved":0,"allocated":15007744,"data_stored":14583014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":883423685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509829,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630324,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":628862,"kb_avail":93741516,"statfs":{"total":96636764160,"available":95991312384,"internally_reserved":0,"allocated":1478656,"data_stored":1057972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":643955141},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509829,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":823992,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":821054,"kb_avail":93547848,"statfs":{"total":96636764160,"available":95792996352,"internally_reserved":0,"allocated":2990080,"data_stored":2567342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840759749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542532,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694200,"kb_used_data":10472,"kb_used_omap":1,"kb_used_meta":683710,"kb_avail":93677640,"statfs":{"total":96636764160,"available":95925903360,"internally_reserved":0,"allocated":10723328,"data_stored":10304131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":700119494},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2099788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10493952,"data_stored":10488922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4279042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":84770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5726270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4279625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5652480,"data_stored":5642363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:21:25.714 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.715 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.715 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.715 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.715 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.716 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.716 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.716 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.716 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.716 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.717 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.717 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.717 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.717 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.717 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.717 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.717 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.718 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.718 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.718 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.718 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.718 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.718 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.718 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-09-17T04:49:38.950013+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=38, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.719 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.719 INFO:tasks.ceph:pgid 5.26 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.719 INFO:tasks.ceph:pgid 5.27 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.719 INFO:tasks.ceph:pgid 5.24 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.720 INFO:tasks.ceph:pgid 5.25 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.720 INFO:tasks.ceph:pgid 5.22 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.720 INFO:tasks.ceph:pgid 5.20 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.720 INFO:tasks.ceph:pgid 5.21 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.720 INFO:tasks.ceph:pgid 5.2f last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.721 INFO:tasks.ceph:pgid 5.2c last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.721 INFO:tasks.ceph:pgid 5.2a last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.721 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.721 INFO:tasks.ceph:pgid 5.28 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.721 INFO:tasks.ceph:pgid 5.29 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.721 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.721 INFO:tasks.ceph:pgid 5.36 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.722 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.722 INFO:tasks.ceph:pgid 5.35 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.722 INFO:tasks.ceph:pgid 5.32 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.722 INFO:tasks.ceph:pgid 5.33 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.722 INFO:tasks.ceph:pgid 5.30 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.722 INFO:tasks.ceph:pgid 5.31 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.722 INFO:tasks.ceph:pgid 5.3e last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.722 INFO:tasks.ceph:pgid 5.3f last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.723 INFO:tasks.ceph:pgid 5.3c last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.723 INFO:tasks.ceph:pgid 5.3b last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.723 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-09-17T04:49:30.837284+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=30, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.723 INFO:tasks.ceph:pgid 5.38 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.723 INFO:tasks.ceph:pgid 5.39 last_scrub_stamp 2024-09-17T04:49:39.959607+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=49, tm_sec=39, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=5, tm_min=20, tm_sec=59, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T05:21:25.723 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T05:21:37.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:21:37.840 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:21:45.724 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T05:21:45.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.895+0000 7f7da508d640 1 -- 172.21.15.150:0/1981956179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8c006470 msgr2=0x7f7d8c00cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:45.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.895+0000 7f7da508d640 1 --2- 172.21.15.150:0/1981956179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8c006470 0x7f7d8c00cb20 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f7d980016c0 tx=0x7f7d9801c790 comp rx=0 tx=0).stop 2024-09-17T05:21:45.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.895+0000 7f7da508d640 1 -- 172.21.15.150:0/1981956179 shutdown_connections 2024-09-17T05:21:45.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.895+0000 7f7da508d640 1 --2- 172.21.15.150:0/1981956179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d8c006470 0x7f7d8c00cb20 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:45.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.895+0000 7f7da508d640 1 -- 172.21.15.150:0/1981956179 >> 172.21.15.150:0/1981956179 conn(0x7f7da00c4a00 msgr2=0x7f7da00c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:45.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.895+0000 7f7da508d640 1 -- 172.21.15.150:0/1981956179 shutdown_connections 2024-09-17T05:21:45.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.895+0000 7f7da508d640 1 -- 172.21.15.150:0/1981956179 wait complete. 2024-09-17T05:21:45.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7da508d640 1 Processor -- start 2024-09-17T05:21:45.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7da508d640 1 -- start start 2024-09-17T05:21:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7da508d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d8c006470 0x7f7da01e0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7da508d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da00abef0 0x7f7da00a85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7da508d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da00ac300 0x7f7da014b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7da508d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7da00c3680 con 0x7f7da00abef0 2024-09-17T05:21:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7da508d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7da006f3e0 con 0x7f7d8c006470 2024-09-17T05:21:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7da508d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f7da00c3800 con 0x7f7da00ac300 2024-09-17T05:21:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d9e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da00abef0 0x7f7da00a85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d9e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da00abef0 0x7f7da00a85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50030/0 (socket says 172.21.15.150:50030) 2024-09-17T05:21:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d9e59c640 1 -- 172.21.15.150:0/1451913734 learned_addr learned my addr 172.21.15.150:0/1451913734 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d9ed9d640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d8c006470 0x7f7da01e0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d9f59e640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da00ac300 0x7f7da014b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d9e59c640 1 -- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da00ac300 msgr2=0x7f7da014b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d9e59c640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da00ac300 0x7f7da014b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d9e59c640 1 -- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d8c006470 msgr2=0x7f7da01e0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:45.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d9e59c640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d8c006470 0x7f7da01e0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:45.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d9e59c640 1 -- 172.21.15.150:0/1451913734 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da01ec8d0 con 0x7f7da00abef0 2024-09-17T05:21:45.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d9e59c640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da00abef0 0x7f7da00a85e0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f7d88011990 tx=0x7f7d88011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:45.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d977fe640 1 -- 172.21.15.150:0/1451913734 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d88017070 con 0x7f7da00abef0 2024-09-17T05:21:45.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d977fe640 1 -- 172.21.15.150:0/1451913734 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d8800fd70 con 0x7f7da00abef0 2024-09-17T05:21:45.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7da508d640 1 -- 172.21.15.150:0/1451913734 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da00a8090 con 0x7f7da00abef0 2024-09-17T05:21:45.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7d977fe640 1 -- 172.21.15.150:0/1451913734 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d880136b0 con 0x7f7da00abef0 2024-09-17T05:21:45.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.899+0000 7f7da508d640 1 -- 172.21.15.150:0/1451913734 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da00a9420 con 0x7f7da00abef0 2024-09-17T05:21:45.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.903+0000 7f7da508d640 1 -- 172.21.15.150:0/1451913734 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7da01501c0 con 0x7f7da00abef0 2024-09-17T05:21:45.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.903+0000 7f7d977fe640 1 -- 172.21.15.150:0/1451913734 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7d88013850 con 0x7f7da00abef0 2024-09-17T05:21:45.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.903+0000 7f7d977fe640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d80073230 0x7f7d80075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:45.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.903+0000 7f7d977fe640 1 -- 172.21.15.150:0/1451913734 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f7d88099450 con 0x7f7da00abef0 2024-09-17T05:21:45.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.903+0000 7f7d9ed9d640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d80073230 0x7f7d80075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:45.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.903+0000 7f7d9ed9d640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d80073230 0x7f7d80075740 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f7d90005e10 tx=0x7f7d900091e0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:45.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:45.911+0000 7f7d977fe640 1 -- 172.21.15.150:0/1451913734 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7d88062d70 con 0x7f7da00abef0 2024-09-17T05:21:46.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.071+0000 7f7da508d640 1 -- 172.21.15.150:0/1451913734 --> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7da00a91d0 con 0x7f7d80073230 2024-09-17T05:21:46.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.095+0000 7f7d977fe640 1 -- 172.21.15.150:0/1451913734 <== mgr.4117 v2:172.21.15.176:6800/882000330 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+495134 (secure 0 0 0) 0x7f7da00a91d0 con 0x7f7d80073230 2024-09-17T05:21:46.101 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:21:46.105 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-09-17T05:21:46.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.099+0000 7f7da508d640 1 -- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d80073230 msgr2=0x7f7d80075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:46.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.099+0000 7f7da508d640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d80073230 0x7f7d80075740 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f7d90005e10 tx=0x7f7d900091e0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.099+0000 7f7da508d640 1 -- 172.21.15.150:0/1451913734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da00abef0 msgr2=0x7f7da00a85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:46.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.099+0000 7f7da508d640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da00abef0 0x7f7da00a85e0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f7d88011990 tx=0x7f7d88011e50 comp rx=0 tx=0).stop 2024-09-17T05:21:46.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.099+0000 7f7da508d640 1 -- 172.21.15.150:0/1451913734 shutdown_connections 2024-09-17T05:21:46.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.099+0000 7f7da508d640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f7d80073230 0x7f7d80075740 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.099+0000 7f7da508d640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f7da00ac300 0x7f7da014b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.099+0000 7f7da508d640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7da00abef0 0x7f7da00a85e0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.099+0000 7f7da508d640 1 --2- 172.21.15.150:0/1451913734 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d8c006470 0x7f7da01e0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.099+0000 7f7da508d640 1 -- 172.21.15.150:0/1451913734 >> 172.21.15.150:0/1451913734 conn(0x7f7da00c4a00 msgr2=0x7f7da00c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:46.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.099+0000 7f7da508d640 1 -- 172.21.15.150:0/1451913734 shutdown_connections 2024-09-17T05:21:46.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.099+0000 7f7da508d640 1 -- 172.21.15.150:0/1451913734 wait complete. 2024-09-17T05:21:46.120 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":1034,"stamp":"2024-09-17T05:21:45.908590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37864050,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3107,"num_read_kb":438866,"num_write":2440075,"num_write_kb":36007256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":458403,"ondisk_log_size":458403,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5836704,"kb_used_data":80928,"kb_used_omap":12,"kb_used_meta":5755635,"kb_avail":749138016,"statfs":{"total":773094113280,"available":767117328384,"internally_reserved":0,"allocated":82870272,"data_stored":79467812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":5893770802},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006083"},"pg_stats":[{"pgid":"5.18","version":"68'3479","reported_seq":3600,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.042279+0000","last_change":"2024-09-17T05:21:10.042098+0000","last_active":"2024-09-17T05:21:10.042279+0000","last_peered":"2024-09-17T05:21:10.042279+0000","last_clean":"2024-09-17T05:21:10.042279+0000","last_became_active":"2024-09-17T04:49:41.024393+0000","last_became_peered":"2024-09-17T04:49:41.024393+0000","last_unstale":"2024-09-17T05:21:10.042279+0000","last_undegraded":"2024-09-17T05:21:10.042279+0000","last_fullsized":"2024-09-17T05:21:10.042279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3479","last_scrub_stamp":"2024-09-17T05:21:10.041849+0000","last_deep_scrub":"68'3479","last_deep_scrub_stamp":"2024-09-17T05:21:10.041849+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.041849+0000","objects_scrubbed":0,"log_size":3479,"log_dups_size":0,"ondisk_log_size":3479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:08:08.320683+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":12,"num_write":18686,"num_write_kb":287786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"68'1004","reported_seq":1186,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.958535+0000","last_change":"2024-09-17T05:21:07.958535+0000","last_active":"2024-09-17T05:21:07.958535+0000","last_peered":"2024-09-17T05:21:07.958535+0000","last_clean":"2024-09-17T05:21:07.958535+0000","last_became_active":"2024-09-17T04:49:30.878913+0000","last_became_peered":"2024-09-17T04:49:30.878913+0000","last_unstale":"2024-09-17T05:21:07.958535+0000","last_undegraded":"2024-09-17T05:21:07.958535+0000","last_fullsized":"2024-09-17T05:21:07.958535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1004","last_scrub_stamp":"2024-09-17T05:21:07.958302+0000","last_deep_scrub":"68'1004","last_deep_scrub_stamp":"2024-09-17T05:21:07.958302+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.958302+0000","objects_scrubbed":3,"log_size":1004,"log_dups_size":0,"ondisk_log_size":1004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:58.171045+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":20774,"num_write":978,"num_write_kb":4679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"65'4064","reported_seq":4193,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.609230+0000","last_change":"2024-09-17T05:21:11.608906+0000","last_active":"2024-09-17T05:21:11.609230+0000","last_peered":"2024-09-17T05:21:11.609230+0000","last_clean":"2024-09-17T05:21:11.609230+0000","last_became_active":"2024-09-17T04:49:31.891381+0000","last_became_peered":"2024-09-17T04:49:31.891381+0000","last_unstale":"2024-09-17T05:21:11.609230+0000","last_undegraded":"2024-09-17T05:21:11.609230+0000","last_fullsized":"2024-09-17T05:21:11.609230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4064","last_scrub_stamp":"2024-09-17T05:21:11.608624+0000","last_deep_scrub":"65'4064","last_deep_scrub_stamp":"2024-09-17T05:21:11.608624+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.608624+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:19.389394+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22217,"num_write_kb":281701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"68'3","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.159597+0000","last_change":"2024-09-17T05:21:13.159399+0000","last_active":"2024-09-17T05:21:13.159597+0000","last_peered":"2024-09-17T05:21:13.159597+0000","last_clean":"2024-09-17T05:21:13.159597+0000","last_became_active":"2024-09-17T04:49:40.012471+0000","last_became_peered":"2024-09-17T04:49:40.012471+0000","last_unstale":"2024-09-17T05:21:13.159597+0000","last_undegraded":"2024-09-17T05:21:13.159597+0000","last_fullsized":"2024-09-17T05:21:13.159597+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-09-17T05:21:13.159185+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2024-09-17T05:21:13.159185+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.159185+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:22.788148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"68'3721","reported_seq":3842,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:24.848407+0000","last_change":"2024-09-17T05:21:24.848407+0000","last_active":"2024-09-17T05:21:24.848407+0000","last_peered":"2024-09-17T05:21:24.848407+0000","last_clean":"2024-09-17T05:21:24.848407+0000","last_became_active":"2024-09-17T04:49:41.050131+0000","last_became_peered":"2024-09-17T04:49:41.050131+0000","last_unstale":"2024-09-17T05:21:24.848407+0000","last_undegraded":"2024-09-17T05:21:24.848407+0000","last_fullsized":"2024-09-17T05:21:24.848407+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3721","last_scrub_stamp":"2024-09-17T05:21:24.848200+0000","last_deep_scrub":"68'3721","last_deep_scrub_stamp":"2024-09-17T05:21:24.848200+0000","last_clean_scrub_stamp":"2024-09-17T05:21:24.848200+0000","objects_scrubbed":0,"log_size":3721,"log_dups_size":0,"ondisk_log_size":3721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:56.965685+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":22,"num_write":19557,"num_write_kb":316044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"67'26","reported_seq":157,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.653302+0000","last_change":"2024-09-17T05:21:06.653104+0000","last_active":"2024-09-17T05:21:06.653302+0000","last_peered":"2024-09-17T05:21:06.653302+0000","last_clean":"2024-09-17T05:21:06.653302+0000","last_became_active":"2024-09-17T04:49:30.899820+0000","last_became_peered":"2024-09-17T04:49:30.899820+0000","last_unstale":"2024-09-17T05:21:06.653302+0000","last_undegraded":"2024-09-17T05:21:06.653302+0000","last_fullsized":"2024-09-17T05:21:06.653302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'26","last_scrub_stamp":"2024-09-17T05:21:06.652875+0000","last_deep_scrub":"67'26","last_deep_scrub_stamp":"2024-09-17T05:21:06.652875+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.652875+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:45.890071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"65'4998","reported_seq":5126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.674317+0000","last_change":"2024-09-17T05:21:13.674317+0000","last_active":"2024-09-17T05:21:13.674317+0000","last_peered":"2024-09-17T05:21:13.674317+0000","last_clean":"2024-09-17T05:21:13.674317+0000","last_became_active":"2024-09-17T04:49:31.884764+0000","last_became_peered":"2024-09-17T04:49:31.884764+0000","last_unstale":"2024-09-17T05:21:13.674317+0000","last_undegraded":"2024-09-17T05:21:13.674317+0000","last_fullsized":"2024-09-17T05:21:13.674317+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4998","last_scrub_stamp":"2024-09-17T05:21:13.674048+0000","last_deep_scrub":"65'4998","last_deep_scrub_stamp":"2024-09-17T05:21:13.674048+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.674048+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:15:59.587253+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27336,"num_write_kb":388205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"68'2","reported_seq":122,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.782513+0000","last_change":"2024-09-17T05:21:17.782513+0000","last_active":"2024-09-17T05:21:17.782513+0000","last_peered":"2024-09-17T05:21:17.782513+0000","last_clean":"2024-09-17T05:21:17.782513+0000","last_became_active":"2024-09-17T04:49:40.003908+0000","last_became_peered":"2024-09-17T04:49:40.003908+0000","last_unstale":"2024-09-17T05:21:17.782513+0000","last_undegraded":"2024-09-17T05:21:17.782513+0000","last_fullsized":"2024-09-17T05:21:17.782513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T05:21:17.782306+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2024-09-17T05:21:17.782306+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.782306+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:51.871351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"68'2765","reported_seq":2884,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.043838+0000","last_change":"2024-09-17T05:21:12.043838+0000","last_active":"2024-09-17T05:21:12.043838+0000","last_peered":"2024-09-17T05:21:12.043838+0000","last_clean":"2024-09-17T05:21:12.043838+0000","last_became_active":"2024-09-17T04:49:41.066842+0000","last_became_peered":"2024-09-17T04:49:41.066842+0000","last_unstale":"2024-09-17T05:21:12.043838+0000","last_undegraded":"2024-09-17T05:21:12.043838+0000","last_fullsized":"2024-09-17T05:21:12.043838+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2765","last_scrub_stamp":"2024-09-17T05:21:12.043445+0000","last_deep_scrub":"68'2765","last_deep_scrub_stamp":"2024-09-17T05:21:12.043445+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.043445+0000","objects_scrubbed":0,"log_size":2765,"log_dups_size":0,"ondisk_log_size":2765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:25:10.594780+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":14571,"num_write_kb":258996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"68'47","reported_seq":216,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.713427+0000","last_change":"2024-09-17T05:21:06.713427+0000","last_active":"2024-09-17T05:21:06.713427+0000","last_peered":"2024-09-17T05:21:06.713427+0000","last_clean":"2024-09-17T05:21:06.713427+0000","last_became_active":"2024-09-17T04:49:30.871438+0000","last_became_peered":"2024-09-17T04:49:30.871438+0000","last_unstale":"2024-09-17T05:21:06.713427+0000","last_undegraded":"2024-09-17T05:21:06.713427+0000","last_fullsized":"2024-09-17T05:21:06.713427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'47","last_scrub_stamp":"2024-09-17T05:21:06.713179+0000","last_deep_scrub":"68'47","last_deep_scrub_stamp":"2024-09-17T05:21:06.713179+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.713179+0000","objects_scrubbed":1,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:16.060967+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"65'3140","reported_seq":3267,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.064090+0000","last_change":"2024-09-17T05:21:09.064090+0000","last_active":"2024-09-17T05:21:09.064090+0000","last_peered":"2024-09-17T05:21:09.064090+0000","last_clean":"2024-09-17T05:21:09.064090+0000","last_became_active":"2024-09-17T04:49:31.892708+0000","last_became_peered":"2024-09-17T04:49:31.892708+0000","last_unstale":"2024-09-17T05:21:09.064090+0000","last_undegraded":"2024-09-17T05:21:09.064090+0000","last_fullsized":"2024-09-17T05:21:09.064090+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3140","last_scrub_stamp":"2024-09-17T05:21:09.063861+0000","last_deep_scrub":"65'3140","last_deep_scrub_stamp":"2024-09-17T05:21:09.063861+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.063861+0000","objects_scrubbed":0,"log_size":3140,"log_dups_size":0,"ondisk_log_size":3140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:54.276035+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16962,"num_write_kb":252282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"68'752","reported_seq":897,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.772498+0000","last_change":"2024-09-17T05:21:14.772498+0000","last_active":"2024-09-17T05:21:14.772498+0000","last_peered":"2024-09-17T05:21:14.772498+0000","last_clean":"2024-09-17T05:21:14.772498+0000","last_became_active":"2024-09-17T04:49:40.019045+0000","last_became_peered":"2024-09-17T04:49:40.019045+0000","last_unstale":"2024-09-17T05:21:14.772498+0000","last_undegraded":"2024-09-17T05:21:14.772498+0000","last_fullsized":"2024-09-17T05:21:14.772498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'752","last_scrub_stamp":"2024-09-17T05:21:14.772291+0000","last_deep_scrub":"68'752","last_deep_scrub_stamp":"2024-09-17T05:21:14.772291+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.772291+0000","objects_scrubbed":1,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:43.662572+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":75812,"num_write":739,"num_write_kb":4324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"68'2700","reported_seq":2818,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.800307+0000","last_change":"2024-09-17T05:21:16.800307+0000","last_active":"2024-09-17T05:21:16.800307+0000","last_peered":"2024-09-17T05:21:16.800307+0000","last_clean":"2024-09-17T05:21:16.800307+0000","last_became_active":"2024-09-17T04:49:41.043079+0000","last_became_peered":"2024-09-17T04:49:41.043079+0000","last_unstale":"2024-09-17T05:21:16.800307+0000","last_undegraded":"2024-09-17T05:21:16.800307+0000","last_fullsized":"2024-09-17T05:21:16.800307+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2700","last_scrub_stamp":"2024-09-17T05:21:16.800075+0000","last_deep_scrub":"68'2700","last_deep_scrub_stamp":"2024-09-17T05:21:16.800075+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.800075+0000","objects_scrubbed":0,"log_size":2700,"log_dups_size":0,"ondisk_log_size":2700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:29.326983+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":5,"num_write":13438,"num_write_kb":225755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"68'6","reported_seq":175,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:04.693011+0000","last_change":"2024-09-17T05:21:04.693011+0000","last_active":"2024-09-17T05:21:04.693011+0000","last_peered":"2024-09-17T05:21:04.693011+0000","last_clean":"2024-09-17T05:21:04.693011+0000","last_became_active":"2024-09-17T04:49:30.871759+0000","last_became_peered":"2024-09-17T04:49:30.871759+0000","last_unstale":"2024-09-17T05:21:04.693011+0000","last_undegraded":"2024-09-17T05:21:04.693011+0000","last_fullsized":"2024-09-17T05:21:04.693011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-17T05:21:04.692899+0000","last_deep_scrub":"68'6","last_deep_scrub_stamp":"2024-09-17T05:21:04.692899+0000","last_clean_scrub_stamp":"2024-09-17T05:21:04.692899+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:43.930095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"65'4481","reported_seq":4606,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.540048+0000","last_change":"2024-09-17T05:21:15.540048+0000","last_active":"2024-09-17T05:21:15.540048+0000","last_peered":"2024-09-17T05:21:15.540048+0000","last_clean":"2024-09-17T05:21:15.540048+0000","last_became_active":"2024-09-17T04:49:31.906254+0000","last_became_peered":"2024-09-17T04:49:31.906254+0000","last_unstale":"2024-09-17T05:21:15.540048+0000","last_undegraded":"2024-09-17T05:21:15.540048+0000","last_fullsized":"2024-09-17T05:21:15.540048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4481","last_scrub_stamp":"2024-09-17T05:21:15.539814+0000","last_deep_scrub":"65'4481","last_deep_scrub_stamp":"2024-09-17T05:21:15.539814+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.539814+0000","objects_scrubbed":0,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:36:00.335129+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24963,"num_write_kb":346034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"68'143","reported_seq":264,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.736612+0000","last_change":"2024-09-17T05:21:15.736612+0000","last_active":"2024-09-17T05:21:15.736612+0000","last_peered":"2024-09-17T05:21:15.736612+0000","last_clean":"2024-09-17T05:21:15.736612+0000","last_became_active":"2024-09-17T04:49:40.005044+0000","last_became_peered":"2024-09-17T04:49:40.005044+0000","last_unstale":"2024-09-17T05:21:15.736612+0000","last_undegraded":"2024-09-17T05:21:15.736612+0000","last_fullsized":"2024-09-17T05:21:15.736612+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'143","last_scrub_stamp":"2024-09-17T05:21:15.736400+0000","last_deep_scrub":"68'143","last_deep_scrub_stamp":"2024-09-17T05:21:15.736400+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.736400+0000","objects_scrubbed":1,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:03.782112+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1447387,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":383,"num_write":117,"num_write_kb":1479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"68'2942","reported_seq":3063,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:22.700584+0000","last_change":"2024-09-17T05:21:22.700584+0000","last_active":"2024-09-17T05:21:22.700584+0000","last_peered":"2024-09-17T05:21:22.700584+0000","last_clean":"2024-09-17T05:21:22.700584+0000","last_became_active":"2024-09-17T04:49:41.035327+0000","last_became_peered":"2024-09-17T04:49:41.035327+0000","last_unstale":"2024-09-17T05:21:22.700584+0000","last_undegraded":"2024-09-17T05:21:22.700584+0000","last_fullsized":"2024-09-17T05:21:22.700584+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2942","last_scrub_stamp":"2024-09-17T05:21:22.700353+0000","last_deep_scrub":"68'2942","last_deep_scrub_stamp":"2024-09-17T05:21:22.700353+0000","last_clean_scrub_stamp":"2024-09-17T05:21:22.700353+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:03.965027+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":16254,"num_write_kb":278162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"65'3968","reported_seq":4095,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.553424+0000","last_change":"2024-09-17T05:21:19.553424+0000","last_active":"2024-09-17T05:21:19.553424+0000","last_peered":"2024-09-17T05:21:19.553424+0000","last_clean":"2024-09-17T05:21:19.553424+0000","last_became_active":"2024-09-17T04:49:31.895205+0000","last_became_peered":"2024-09-17T04:49:31.895205+0000","last_unstale":"2024-09-17T05:21:19.553424+0000","last_undegraded":"2024-09-17T05:21:19.553424+0000","last_fullsized":"2024-09-17T05:21:19.553424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3968","last_scrub_stamp":"2024-09-17T05:21:19.553025+0000","last_deep_scrub":"65'3968","last_deep_scrub_stamp":"2024-09-17T05:21:19.553025+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.553025+0000","objects_scrubbed":0,"log_size":3968,"log_dups_size":0,"ondisk_log_size":3968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:51.875435+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21939,"num_write_kb":297018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"68'1469","reported_seq":1647,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.571989+0000","last_change":"2024-09-17T05:21:08.571989+0000","last_active":"2024-09-17T05:21:08.571989+0000","last_peered":"2024-09-17T05:21:08.571989+0000","last_clean":"2024-09-17T05:21:08.571989+0000","last_became_active":"2024-09-17T04:49:30.880959+0000","last_became_peered":"2024-09-17T04:49:30.880959+0000","last_unstale":"2024-09-17T05:21:08.571989+0000","last_undegraded":"2024-09-17T05:21:08.571989+0000","last_fullsized":"2024-09-17T05:21:08.571989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1469","last_scrub_stamp":"2024-09-17T05:21:08.571729+0000","last_deep_scrub":"68'1469","last_deep_scrub_stamp":"2024-09-17T05:21:08.571729+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.571729+0000","objects_scrubbed":2,"log_size":1469,"log_dups_size":0,"ondisk_log_size":1469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:38.358519+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":43333,"num_write":1444,"num_write_kb":4521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"68'7","reported_seq":148,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:26.412882+0000","last_change":"2024-09-17T05:21:26.412882+0000","last_active":"2024-09-17T05:21:26.412882+0000","last_peered":"2024-09-17T05:21:26.412882+0000","last_clean":"2024-09-17T05:21:26.412882+0000","last_became_active":"2024-09-17T04:49:39.990388+0000","last_became_peered":"2024-09-17T04:49:39.990388+0000","last_unstale":"2024-09-17T05:21:26.412882+0000","last_undegraded":"2024-09-17T05:21:26.412882+0000","last_fullsized":"2024-09-17T05:21:26.412882+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'7","last_scrub_stamp":"2024-09-17T05:21:26.412653+0000","last_deep_scrub":"68'7","last_deep_scrub_stamp":"2024-09-17T05:21:26.412653+0000","last_clean_scrub_stamp":"2024-09-17T05:21:26.412653+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:31:59.370690+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"68'3236","reported_seq":3358,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.262331+0000","last_change":"2024-09-17T05:21:17.262331+0000","last_active":"2024-09-17T05:21:17.262331+0000","last_peered":"2024-09-17T05:21:17.262331+0000","last_clean":"2024-09-17T05:21:17.262331+0000","last_became_active":"2024-09-17T04:49:41.037414+0000","last_became_peered":"2024-09-17T04:49:41.037414+0000","last_unstale":"2024-09-17T05:21:17.262331+0000","last_undegraded":"2024-09-17T05:21:17.262331+0000","last_fullsized":"2024-09-17T05:21:17.262331+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3236","last_scrub_stamp":"2024-09-17T05:21:17.262125+0000","last_deep_scrub":"68'3236","last_deep_scrub_stamp":"2024-09-17T05:21:17.262125+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.262125+0000","objects_scrubbed":0,"log_size":3236,"log_dups_size":0,"ondisk_log_size":3236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:07:24.547946+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":17542,"num_write_kb":271055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"65'3303","reported_seq":3430,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.208468+0000","last_change":"2024-09-17T05:21:09.208468+0000","last_active":"2024-09-17T05:21:09.208468+0000","last_peered":"2024-09-17T05:21:09.208468+0000","last_clean":"2024-09-17T05:21:09.208468+0000","last_became_active":"2024-09-17T04:49:31.917367+0000","last_became_peered":"2024-09-17T04:49:31.917367+0000","last_unstale":"2024-09-17T05:21:09.208468+0000","last_undegraded":"2024-09-17T05:21:09.208468+0000","last_fullsized":"2024-09-17T05:21:09.208468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3303","last_scrub_stamp":"2024-09-17T05:21:09.208191+0000","last_deep_scrub":"65'3303","last_deep_scrub_stamp":"2024-09-17T05:21:09.208191+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.208191+0000","objects_scrubbed":0,"log_size":3303,"log_dups_size":0,"ondisk_log_size":3303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:00.238401+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17463,"num_write_kb":238014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"65'3","reported_seq":132,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.246039+0000","last_change":"2024-09-17T05:21:06.246039+0000","last_active":"2024-09-17T05:21:06.246039+0000","last_peered":"2024-09-17T05:21:06.246039+0000","last_clean":"2024-09-17T05:21:06.246039+0000","last_became_active":"2024-09-17T04:49:30.895512+0000","last_became_peered":"2024-09-17T04:49:30.895512+0000","last_unstale":"2024-09-17T05:21:06.246039+0000","last_undegraded":"2024-09-17T05:21:06.246039+0000","last_fullsized":"2024-09-17T05:21:06.246039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-09-17T05:21:06.245820+0000","last_deep_scrub":"65'3","last_deep_scrub_stamp":"2024-09-17T05:21:06.245820+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.245820+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:50.693155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"68'5","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.681526+0000","last_change":"2024-09-17T05:21:16.681526+0000","last_active":"2024-09-17T05:21:16.681526+0000","last_peered":"2024-09-17T05:21:16.681526+0000","last_clean":"2024-09-17T05:21:16.681526+0000","last_became_active":"2024-09-17T04:49:40.010496+0000","last_became_peered":"2024-09-17T04:49:40.010496+0000","last_unstale":"2024-09-17T05:21:16.681526+0000","last_undegraded":"2024-09-17T05:21:16.681526+0000","last_fullsized":"2024-09-17T05:21:16.681526+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:16.681321+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:16.681321+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.681321+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:17:49.939128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"68'2691","reported_seq":2811,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:30.407474+0000","last_change":"2024-09-17T05:21:30.407474+0000","last_active":"2024-09-17T05:21:30.407474+0000","last_peered":"2024-09-17T05:21:30.407474+0000","last_clean":"2024-09-17T05:21:30.407474+0000","last_became_active":"2024-09-17T04:49:41.058499+0000","last_became_peered":"2024-09-17T04:49:41.058499+0000","last_unstale":"2024-09-17T05:21:30.407474+0000","last_undegraded":"2024-09-17T05:21:30.407474+0000","last_fullsized":"2024-09-17T05:21:30.407474+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2691","last_scrub_stamp":"2024-09-17T05:21:30.407224+0000","last_deep_scrub":"68'2691","last_deep_scrub_stamp":"2024-09-17T05:21:30.407224+0000","last_clean_scrub_stamp":"2024-09-17T05:21:30.407224+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:26:11.285685+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":14365,"num_write_kb":225564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"65'3640","reported_seq":3767,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.847235+0000","last_change":"2024-09-17T05:21:11.847235+0000","last_active":"2024-09-17T05:21:11.847235+0000","last_peered":"2024-09-17T05:21:11.847235+0000","last_clean":"2024-09-17T05:21:11.847235+0000","last_became_active":"2024-09-17T04:49:31.891669+0000","last_became_peered":"2024-09-17T04:49:31.891669+0000","last_unstale":"2024-09-17T05:21:11.847235+0000","last_undegraded":"2024-09-17T05:21:11.847235+0000","last_fullsized":"2024-09-17T05:21:11.847235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3640","last_scrub_stamp":"2024-09-17T05:21:11.847013+0000","last_deep_scrub":"65'3640","last_deep_scrub_stamp":"2024-09-17T05:21:11.847013+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.847013+0000","objects_scrubbed":0,"log_size":3640,"log_dups_size":0,"ondisk_log_size":3640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:40.737295+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20183,"num_write_kb":277678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"65'2","reported_seq":131,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:04.912716+0000","last_change":"2024-09-17T05:21:04.912716+0000","last_active":"2024-09-17T05:21:04.912716+0000","last_peered":"2024-09-17T05:21:04.912716+0000","last_clean":"2024-09-17T05:21:04.912716+0000","last_became_active":"2024-09-17T04:49:30.859076+0000","last_became_peered":"2024-09-17T04:49:30.859076+0000","last_unstale":"2024-09-17T05:21:04.912716+0000","last_undegraded":"2024-09-17T05:21:04.912716+0000","last_fullsized":"2024-09-17T05:21:04.912716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-17T05:21:04.912451+0000","last_deep_scrub":"65'2","last_deep_scrub_stamp":"2024-09-17T05:21:04.912451+0000","last_clean_scrub_stamp":"2024-09-17T05:21:04.912451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:39.929521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"68'5","reported_seq":146,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.167300+0000","last_change":"2024-09-17T05:21:10.167098+0000","last_active":"2024-09-17T05:21:10.167300+0000","last_peered":"2024-09-17T05:21:10.167300+0000","last_clean":"2024-09-17T05:21:10.167300+0000","last_became_active":"2024-09-17T04:49:40.005560+0000","last_became_peered":"2024-09-17T04:49:40.005560+0000","last_unstale":"2024-09-17T05:21:10.167300+0000","last_undegraded":"2024-09-17T05:21:10.167300+0000","last_fullsized":"2024-09-17T05:21:10.167300+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:10.166704+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:10.166704+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.166704+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:21:51.895257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"68'3943","reported_seq":4062,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.108032+0000","last_change":"2024-09-17T05:21:07.107840+0000","last_active":"2024-09-17T05:21:07.108032+0000","last_peered":"2024-09-17T05:21:07.108032+0000","last_clean":"2024-09-17T05:21:07.108032+0000","last_became_active":"2024-09-17T04:49:41.036267+0000","last_became_peered":"2024-09-17T04:49:41.036267+0000","last_unstale":"2024-09-17T05:21:07.108032+0000","last_undegraded":"2024-09-17T05:21:07.108032+0000","last_fullsized":"2024-09-17T05:21:07.108032+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3943","last_scrub_stamp":"2024-09-17T05:21:07.107602+0000","last_deep_scrub":"68'3943","last_deep_scrub_stamp":"2024-09-17T05:21:07.107602+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.107602+0000","objects_scrubbed":0,"log_size":3943,"log_dups_size":0,"ondisk_log_size":3943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:04:17.650985+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21088,"num_write_kb":359062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"65'3699","reported_seq":3826,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.489668+0000","last_change":"2024-09-17T05:21:13.489668+0000","last_active":"2024-09-17T05:21:13.489668+0000","last_peered":"2024-09-17T05:21:13.489668+0000","last_clean":"2024-09-17T05:21:13.489668+0000","last_became_active":"2024-09-17T04:49:31.899490+0000","last_became_peered":"2024-09-17T04:49:31.899490+0000","last_unstale":"2024-09-17T05:21:13.489668+0000","last_undegraded":"2024-09-17T05:21:13.489668+0000","last_fullsized":"2024-09-17T05:21:13.489668+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3699","last_scrub_stamp":"2024-09-17T05:21:13.489313+0000","last_deep_scrub":"65'3699","last_deep_scrub_stamp":"2024-09-17T05:21:13.489313+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.489313+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:16.073785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19661,"num_write_kb":271466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"65'5","reported_seq":134,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:01.675505+0000","last_change":"2024-09-17T05:21:01.675505+0000","last_active":"2024-09-17T05:21:01.675505+0000","last_peered":"2024-09-17T05:21:01.675505+0000","last_clean":"2024-09-17T05:21:01.675505+0000","last_became_active":"2024-09-17T04:49:30.890766+0000","last_became_peered":"2024-09-17T04:49:30.890766+0000","last_unstale":"2024-09-17T05:21:01.675505+0000","last_undegraded":"2024-09-17T05:21:01.675505+0000","last_fullsized":"2024-09-17T05:21:01.675505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-17T05:21:01.675372+0000","last_deep_scrub":"65'5","last_deep_scrub_stamp":"2024-09-17T05:21:01.675372+0000","last_clean_scrub_stamp":"2024-09-17T05:21:01.675372+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:14.921299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"68'8","reported_seq":169,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.737846+0000","last_change":"2024-09-17T05:21:12.737691+0000","last_active":"2024-09-17T05:21:12.737846+0000","last_peered":"2024-09-17T05:21:12.737846+0000","last_clean":"2024-09-17T05:21:12.737846+0000","last_became_active":"2024-09-17T04:49:39.991637+0000","last_became_peered":"2024-09-17T04:49:39.991637+0000","last_unstale":"2024-09-17T05:21:12.737846+0000","last_undegraded":"2024-09-17T05:21:12.737846+0000","last_fullsized":"2024-09-17T05:21:12.737846+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-17T05:21:12.737480+0000","last_deep_scrub":"68'8","last_deep_scrub_stamp":"2024-09-17T05:21:12.737480+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.737480+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:04:20.987562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"68'2532","reported_seq":2652,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.822582+0000","last_change":"2024-09-17T05:21:20.822582+0000","last_active":"2024-09-17T05:21:20.822582+0000","last_peered":"2024-09-17T05:21:20.822582+0000","last_clean":"2024-09-17T05:21:20.822582+0000","last_became_active":"2024-09-17T04:49:41.043787+0000","last_became_peered":"2024-09-17T04:49:41.043787+0000","last_unstale":"2024-09-17T05:21:20.822582+0000","last_undegraded":"2024-09-17T05:21:20.822582+0000","last_fullsized":"2024-09-17T05:21:20.822582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2532","last_scrub_stamp":"2024-09-17T05:21:20.822256+0000","last_deep_scrub":"68'2532","last_deep_scrub_stamp":"2024-09-17T05:21:20.822256+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.822256+0000","objects_scrubbed":0,"log_size":2532,"log_dups_size":0,"ondisk_log_size":2532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:31:07.663177+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13954,"num_write_kb":219876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"65'4598","reported_seq":4726,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.898537+0000","last_change":"2024-09-17T05:21:09.898537+0000","last_active":"2024-09-17T05:21:09.898537+0000","last_peered":"2024-09-17T05:21:09.898537+0000","last_clean":"2024-09-17T05:21:09.898537+0000","last_became_active":"2024-09-17T04:49:31.891798+0000","last_became_peered":"2024-09-17T04:49:31.891798+0000","last_unstale":"2024-09-17T05:21:09.898537+0000","last_undegraded":"2024-09-17T05:21:09.898537+0000","last_fullsized":"2024-09-17T05:21:09.898537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4598","last_scrub_stamp":"2024-09-17T05:21:09.898190+0000","last_deep_scrub":"65'4598","last_deep_scrub_stamp":"2024-09-17T05:21:09.898190+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.898190+0000","objects_scrubbed":0,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:04:18.148273+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24313,"num_write_kb":335123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"68'10","reported_seq":181,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.507971+0000","last_change":"2024-09-17T05:21:06.507971+0000","last_active":"2024-09-17T05:21:06.507971+0000","last_peered":"2024-09-17T05:21:06.507971+0000","last_clean":"2024-09-17T05:21:06.507971+0000","last_became_active":"2024-09-17T04:49:30.887657+0000","last_became_peered":"2024-09-17T04:49:30.887657+0000","last_unstale":"2024-09-17T05:21:06.507971+0000","last_undegraded":"2024-09-17T05:21:06.507971+0000","last_fullsized":"2024-09-17T05:21:06.507971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-17T05:21:06.507757+0000","last_deep_scrub":"68'10","last_deep_scrub_stamp":"2024-09-17T05:21:06.507757+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.507757+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:47.772431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"68'50","reported_seq":237,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.750545+0000","last_change":"2024-09-17T05:21:10.750545+0000","last_active":"2024-09-17T05:21:10.750545+0000","last_peered":"2024-09-17T05:21:10.750545+0000","last_clean":"2024-09-17T05:21:10.750545+0000","last_became_active":"2024-09-17T04:49:39.990210+0000","last_became_peered":"2024-09-17T04:49:39.990210+0000","last_unstale":"2024-09-17T05:21:10.750545+0000","last_undegraded":"2024-09-17T05:21:10.750545+0000","last_fullsized":"2024-09-17T05:21:10.750545+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'50","last_scrub_stamp":"2024-09-17T05:21:10.750274+0000","last_deep_scrub":"68'50","last_deep_scrub_stamp":"2024-09-17T05:21:10.750274+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.750274+0000","objects_scrubbed":1,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:00.963016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":65,"num_read_kb":114,"num_write":45,"num_write_kb":114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"68'2531","reported_seq":2652,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:28.429999+0000","last_change":"2024-09-17T05:21:28.429999+0000","last_active":"2024-09-17T05:21:28.429999+0000","last_peered":"2024-09-17T05:21:28.429999+0000","last_clean":"2024-09-17T05:21:28.429999+0000","last_became_active":"2024-09-17T04:49:41.066135+0000","last_became_peered":"2024-09-17T04:49:41.066135+0000","last_unstale":"2024-09-17T05:21:28.429999+0000","last_undegraded":"2024-09-17T05:21:28.429999+0000","last_fullsized":"2024-09-17T05:21:28.429999+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2531","last_scrub_stamp":"2024-09-17T05:21:28.429731+0000","last_deep_scrub":"68'2531","last_deep_scrub_stamp":"2024-09-17T05:21:28.429731+0000","last_clean_scrub_stamp":"2024-09-17T05:21:28.429731+0000","objects_scrubbed":0,"log_size":2531,"log_dups_size":0,"ondisk_log_size":2531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:29.617040+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13881,"num_write_kb":219504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"65'3884","reported_seq":4011,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.854850+0000","last_change":"2024-09-17T05:21:10.854850+0000","last_active":"2024-09-17T05:21:10.854850+0000","last_peered":"2024-09-17T05:21:10.854850+0000","last_clean":"2024-09-17T05:21:10.854850+0000","last_became_active":"2024-09-17T04:49:31.882377+0000","last_became_peered":"2024-09-17T04:49:31.882377+0000","last_unstale":"2024-09-17T05:21:10.854850+0000","last_undegraded":"2024-09-17T05:21:10.854850+0000","last_fullsized":"2024-09-17T05:21:10.854850+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3884","last_scrub_stamp":"2024-09-17T05:21:10.854629+0000","last_deep_scrub":"65'3884","last_deep_scrub_stamp":"2024-09-17T05:21:10.854629+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.854629+0000","objects_scrubbed":0,"log_size":3884,"log_dups_size":0,"ondisk_log_size":3884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:49:26.336872+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21342,"num_write_kb":257935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"68'473","reported_seq":623,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:03.864229+0000","last_change":"2024-09-17T05:21:03.864229+0000","last_active":"2024-09-17T05:21:03.864229+0000","last_peered":"2024-09-17T05:21:03.864229+0000","last_clean":"2024-09-17T05:21:03.864229+0000","last_became_active":"2024-09-17T04:49:30.867110+0000","last_became_peered":"2024-09-17T04:49:30.867110+0000","last_unstale":"2024-09-17T05:21:03.864229+0000","last_undegraded":"2024-09-17T05:21:03.864229+0000","last_fullsized":"2024-09-17T05:21:03.864229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'473","last_scrub_stamp":"2024-09-17T05:21:03.864015+0000","last_deep_scrub":"68'473","last_deep_scrub_stamp":"2024-09-17T05:21:03.864015+0000","last_clean_scrub_stamp":"2024-09-17T05:21:03.864015+0000","objects_scrubbed":1,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:47.178085+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":57,"num_write":1105,"num_write_kb":359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'11","reported_seq":133,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.827996+0000","last_change":"2024-09-17T05:21:15.827996+0000","last_active":"2024-09-17T05:21:15.827996+0000","last_peered":"2024-09-17T05:21:15.827996+0000","last_clean":"2024-09-17T05:21:15.827996+0000","last_became_active":"2024-09-17T04:49:39.999983+0000","last_became_peered":"2024-09-17T04:49:39.999983+0000","last_unstale":"2024-09-17T05:21:15.827996+0000","last_undegraded":"2024-09-17T05:21:15.827996+0000","last_fullsized":"2024-09-17T05:21:15.827996+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'11","last_scrub_stamp":"2024-09-17T05:21:15.827718+0000","last_deep_scrub":"68'11","last_deep_scrub_stamp":"2024-09-17T05:21:15.827718+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.827718+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:56:26.850794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"68'2691","reported_seq":2811,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:21.674125+0000","last_change":"2024-09-17T05:21:21.674125+0000","last_active":"2024-09-17T05:21:21.674125+0000","last_peered":"2024-09-17T05:21:21.674125+0000","last_clean":"2024-09-17T05:21:21.674125+0000","last_became_active":"2024-09-17T04:49:41.041886+0000","last_became_peered":"2024-09-17T04:49:41.041886+0000","last_unstale":"2024-09-17T05:21:21.674125+0000","last_undegraded":"2024-09-17T05:21:21.674125+0000","last_fullsized":"2024-09-17T05:21:21.674125+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2691","last_scrub_stamp":"2024-09-17T05:21:21.673911+0000","last_deep_scrub":"68'2691","last_deep_scrub_stamp":"2024-09-17T05:21:21.673911+0000","last_clean_scrub_stamp":"2024-09-17T05:21:21.673911+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:21:16.011057+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14724,"num_write_kb":243278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"65'4637","reported_seq":4765,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.270658+0000","last_change":"2024-09-17T05:21:07.270658+0000","last_active":"2024-09-17T05:21:07.270658+0000","last_peered":"2024-09-17T05:21:07.270658+0000","last_clean":"2024-09-17T05:21:07.270658+0000","last_became_active":"2024-09-17T04:49:31.881064+0000","last_became_peered":"2024-09-17T04:49:31.881064+0000","last_unstale":"2024-09-17T05:21:07.270658+0000","last_undegraded":"2024-09-17T05:21:07.270658+0000","last_fullsized":"2024-09-17T05:21:07.270658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4637","last_scrub_stamp":"2024-09-17T05:21:07.270360+0000","last_deep_scrub":"65'4637","last_deep_scrub_stamp":"2024-09-17T05:21:07.270360+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.270360+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:24.143638+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25111,"num_write_kb":342484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"65'2","reported_seq":132,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:04.441505+0000","last_change":"2024-09-17T05:21:04.441505+0000","last_active":"2024-09-17T05:21:04.441505+0000","last_peered":"2024-09-17T05:21:04.441505+0000","last_clean":"2024-09-17T05:21:04.441505+0000","last_became_active":"2024-09-17T04:49:30.890882+0000","last_became_peered":"2024-09-17T04:49:30.890882+0000","last_unstale":"2024-09-17T05:21:04.441505+0000","last_undegraded":"2024-09-17T05:21:04.441505+0000","last_fullsized":"2024-09-17T05:21:04.441505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-17T05:21:04.441379+0000","last_deep_scrub":"65'2","last_deep_scrub_stamp":"2024-09-17T05:21:04.441379+0000","last_clean_scrub_stamp":"2024-09-17T05:21:04.441379+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:38.530424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"68'3","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.219105+0000","last_change":"2024-09-17T05:21:15.219105+0000","last_active":"2024-09-17T05:21:15.219105+0000","last_peered":"2024-09-17T05:21:15.219105+0000","last_clean":"2024-09-17T05:21:15.219105+0000","last_became_active":"2024-09-17T04:49:39.995025+0000","last_became_peered":"2024-09-17T04:49:39.995025+0000","last_unstale":"2024-09-17T05:21:15.219105+0000","last_undegraded":"2024-09-17T05:21:15.219105+0000","last_fullsized":"2024-09-17T05:21:15.219105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-09-17T05:21:15.218897+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2024-09-17T05:21:15.218897+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.218897+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:40:09.056506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"68'3511","reported_seq":3631,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:21.801291+0000","last_change":"2024-09-17T05:21:21.801291+0000","last_active":"2024-09-17T05:21:21.801291+0000","last_peered":"2024-09-17T05:21:21.801291+0000","last_clean":"2024-09-17T05:21:21.801291+0000","last_became_active":"2024-09-17T04:49:41.032735+0000","last_became_peered":"2024-09-17T04:49:41.032735+0000","last_unstale":"2024-09-17T05:21:21.801291+0000","last_undegraded":"2024-09-17T05:21:21.801291+0000","last_fullsized":"2024-09-17T05:21:21.801291+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3511","last_scrub_stamp":"2024-09-17T05:21:21.801082+0000","last_deep_scrub":"68'3511","last_deep_scrub_stamp":"2024-09-17T05:21:21.801082+0000","last_clean_scrub_stamp":"2024-09-17T05:21:21.801082+0000","objects_scrubbed":0,"log_size":3511,"log_dups_size":0,"ondisk_log_size":3511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:28:09.606265+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19215,"num_write_kb":314590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"65'3495","reported_seq":3623,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:02.679179+0000","last_change":"2024-09-17T05:21:02.679179+0000","last_active":"2024-09-17T05:21:02.679179+0000","last_peered":"2024-09-17T05:21:02.679179+0000","last_clean":"2024-09-17T05:21:02.679179+0000","last_became_active":"2024-09-17T04:49:31.918366+0000","last_became_peered":"2024-09-17T04:49:31.918366+0000","last_unstale":"2024-09-17T05:21:02.679179+0000","last_undegraded":"2024-09-17T05:21:02.679179+0000","last_fullsized":"2024-09-17T05:21:02.679179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3495","last_scrub_stamp":"2024-09-17T05:21:02.679039+0000","last_deep_scrub":"65'3495","last_deep_scrub_stamp":"2024-09-17T05:21:02.679039+0000","last_clean_scrub_stamp":"2024-09-17T05:21:02.679039+0000","objects_scrubbed":0,"log_size":3495,"log_dups_size":0,"ondisk_log_size":3495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:09:41.948141+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18740,"num_write_kb":251346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"65'2","reported_seq":132,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.640295+0000","last_change":"2024-09-17T05:21:05.640295+0000","last_active":"2024-09-17T05:21:05.640295+0000","last_peered":"2024-09-17T05:21:05.640295+0000","last_clean":"2024-09-17T05:21:05.640295+0000","last_became_active":"2024-09-17T04:49:30.899517+0000","last_became_peered":"2024-09-17T04:49:30.899517+0000","last_unstale":"2024-09-17T05:21:05.640295+0000","last_undegraded":"2024-09-17T05:21:05.640295+0000","last_fullsized":"2024-09-17T05:21:05.640295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-17T05:21:05.639951+0000","last_deep_scrub":"65'2","last_deep_scrub_stamp":"2024-09-17T05:21:05.639951+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.639951+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:24.152670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'4","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.253126+0000","last_change":"2024-09-17T05:21:14.253126+0000","last_active":"2024-09-17T05:21:14.253126+0000","last_peered":"2024-09-17T05:21:14.253126+0000","last_clean":"2024-09-17T05:21:14.253126+0000","last_became_active":"2024-09-17T04:49:39.986150+0000","last_became_peered":"2024-09-17T04:49:39.986150+0000","last_unstale":"2024-09-17T05:21:14.253126+0000","last_undegraded":"2024-09-17T05:21:14.253126+0000","last_fullsized":"2024-09-17T05:21:14.253126+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4","last_scrub_stamp":"2024-09-17T05:21:14.252925+0000","last_deep_scrub":"68'4","last_deep_scrub_stamp":"2024-09-17T05:21:14.252925+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.252925+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:54.094101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"68'2923","reported_seq":3044,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.233177+0000","last_change":"2024-09-17T05:21:16.233177+0000","last_active":"2024-09-17T05:21:16.233177+0000","last_peered":"2024-09-17T05:21:16.233177+0000","last_clean":"2024-09-17T05:21:16.233177+0000","last_became_active":"2024-09-17T04:49:41.068742+0000","last_became_peered":"2024-09-17T04:49:41.068742+0000","last_unstale":"2024-09-17T05:21:16.233177+0000","last_undegraded":"2024-09-17T05:21:16.233177+0000","last_fullsized":"2024-09-17T05:21:16.233177+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2923","last_scrub_stamp":"2024-09-17T05:21:16.232932+0000","last_deep_scrub":"68'2923","last_deep_scrub_stamp":"2024-09-17T05:21:16.232932+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.232932+0000","objects_scrubbed":0,"log_size":2923,"log_dups_size":0,"ondisk_log_size":2923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:23:31.283988+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":7,"num_write":15908,"num_write_kb":260386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"65'4495","reported_seq":4623,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.728414+0000","last_change":"2024-09-17T05:21:07.728414+0000","last_active":"2024-09-17T05:21:07.728414+0000","last_peered":"2024-09-17T05:21:07.728414+0000","last_clean":"2024-09-17T05:21:07.728414+0000","last_became_active":"2024-09-17T04:49:31.903921+0000","last_became_peered":"2024-09-17T04:49:31.903921+0000","last_unstale":"2024-09-17T05:21:07.728414+0000","last_undegraded":"2024-09-17T05:21:07.728414+0000","last_fullsized":"2024-09-17T05:21:07.728414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4495","last_scrub_stamp":"2024-09-17T05:21:07.728173+0000","last_deep_scrub":"65'4495","last_deep_scrub_stamp":"2024-09-17T05:21:07.728173+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.728173+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:51.042243+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24287,"num_write_kb":330898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"65'4","reported_seq":134,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.467242+0000","last_change":"2024-09-17T05:21:05.467242+0000","last_active":"2024-09-17T05:21:05.467242+0000","last_peered":"2024-09-17T05:21:05.467242+0000","last_clean":"2024-09-17T05:21:05.467242+0000","last_became_active":"2024-09-17T04:49:30.889805+0000","last_became_peered":"2024-09-17T04:49:30.889805+0000","last_unstale":"2024-09-17T05:21:05.467242+0000","last_undegraded":"2024-09-17T05:21:05.467242+0000","last_fullsized":"2024-09-17T05:21:05.467242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-09-17T05:21:05.467010+0000","last_deep_scrub":"65'4","last_deep_scrub_stamp":"2024-09-17T05:21:05.467010+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.467010+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:56:16.490086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'2","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.198900+0000","last_change":"2024-09-17T05:21:18.198900+0000","last_active":"2024-09-17T05:21:18.198900+0000","last_peered":"2024-09-17T05:21:18.198900+0000","last_clean":"2024-09-17T05:21:18.198900+0000","last_became_active":"2024-09-17T04:49:39.998780+0000","last_became_peered":"2024-09-17T04:49:39.998780+0000","last_unstale":"2024-09-17T05:21:18.198900+0000","last_undegraded":"2024-09-17T05:21:18.198900+0000","last_fullsized":"2024-09-17T05:21:18.198900+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T05:21:18.198640+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2024-09-17T05:21:18.198640+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.198640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:40.743113+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"68'3959","reported_seq":4082,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.208937+0000","last_change":"2024-09-17T05:21:15.208937+0000","last_active":"2024-09-17T05:21:15.208937+0000","last_peered":"2024-09-17T05:21:15.208937+0000","last_clean":"2024-09-17T05:21:15.208937+0000","last_became_active":"2024-09-17T04:49:41.067897+0000","last_became_peered":"2024-09-17T04:49:41.067897+0000","last_unstale":"2024-09-17T05:21:15.208937+0000","last_undegraded":"2024-09-17T05:21:15.208937+0000","last_fullsized":"2024-09-17T05:21:15.208937+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3959","last_scrub_stamp":"2024-09-17T05:21:15.208721+0000","last_deep_scrub":"68'3959","last_deep_scrub_stamp":"2024-09-17T05:21:15.208721+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.208721+0000","objects_scrubbed":0,"log_size":3959,"log_dups_size":0,"ondisk_log_size":3959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:39.405184+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":13,"num_write":21335,"num_write_kb":323115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'3351","reported_seq":3480,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.675194+0000","last_change":"2024-09-17T05:21:07.675116+0000","last_active":"2024-09-17T05:21:07.675194+0000","last_peered":"2024-09-17T05:21:07.675194+0000","last_clean":"2024-09-17T05:21:07.675194+0000","last_became_active":"2024-09-17T04:49:31.887396+0000","last_became_peered":"2024-09-17T04:49:31.887396+0000","last_unstale":"2024-09-17T05:21:07.675194+0000","last_undegraded":"2024-09-17T05:21:07.675194+0000","last_fullsized":"2024-09-17T05:21:07.675194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3351","last_scrub_stamp":"2024-09-17T05:21:07.674891+0000","last_deep_scrub":"65'3351","last_deep_scrub_stamp":"2024-09-17T05:21:07.674891+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.674891+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:37.974183+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18427,"num_write_kb":241864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"68'53","reported_seq":251,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.172504+0000","last_change":"2024-09-17T05:21:05.172504+0000","last_active":"2024-09-17T05:21:05.172504+0000","last_peered":"2024-09-17T05:21:05.172504+0000","last_clean":"2024-09-17T05:21:05.172504+0000","last_became_active":"2024-09-17T04:49:30.906142+0000","last_became_peered":"2024-09-17T04:49:30.906142+0000","last_unstale":"2024-09-17T05:21:05.172504+0000","last_undegraded":"2024-09-17T05:21:05.172504+0000","last_fullsized":"2024-09-17T05:21:05.172504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'53","last_scrub_stamp":"2024-09-17T05:21:05.171784+0000","last_deep_scrub":"68'53","last_deep_scrub_stamp":"2024-09-17T05:21:05.171784+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.171784+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:20:05.118143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":116,"num_write":48,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'10","reported_seq":171,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.197679+0000","last_change":"2024-09-17T05:21:19.197679+0000","last_active":"2024-09-17T05:21:19.197679+0000","last_peered":"2024-09-17T05:21:19.197679+0000","last_clean":"2024-09-17T05:21:19.197679+0000","last_became_active":"2024-09-17T04:49:39.994144+0000","last_became_peered":"2024-09-17T04:49:39.994144+0000","last_unstale":"2024-09-17T05:21:19.197679+0000","last_undegraded":"2024-09-17T05:21:19.197679+0000","last_fullsized":"2024-09-17T05:21:19.197679+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-17T05:21:19.197471+0000","last_deep_scrub":"68'10","last_deep_scrub_stamp":"2024-09-17T05:21:19.197471+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.197471+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:26.619236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"68'2708","reported_seq":2827,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:31.410574+0000","last_change":"2024-09-17T05:21:31.410574+0000","last_active":"2024-09-17T05:21:31.410574+0000","last_peered":"2024-09-17T05:21:31.410574+0000","last_clean":"2024-09-17T05:21:31.410574+0000","last_became_active":"2024-09-17T04:49:41.051646+0000","last_became_peered":"2024-09-17T04:49:41.051646+0000","last_unstale":"2024-09-17T05:21:31.410574+0000","last_undegraded":"2024-09-17T05:21:31.410574+0000","last_fullsized":"2024-09-17T05:21:31.410574+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2708","last_scrub_stamp":"2024-09-17T05:21:31.410215+0000","last_deep_scrub":"68'2708","last_deep_scrub_stamp":"2024-09-17T05:21:31.410215+0000","last_clean_scrub_stamp":"2024-09-17T05:21:31.410215+0000","objects_scrubbed":0,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:58.956624+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14599,"num_write_kb":234639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"65'3456","reported_seq":3584,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.256433+0000","last_change":"2024-09-17T05:21:08.256433+0000","last_active":"2024-09-17T05:21:08.256433+0000","last_peered":"2024-09-17T05:21:08.256433+0000","last_clean":"2024-09-17T05:21:08.256433+0000","last_became_active":"2024-09-17T04:49:31.902856+0000","last_became_peered":"2024-09-17T04:49:31.902856+0000","last_unstale":"2024-09-17T05:21:08.256433+0000","last_undegraded":"2024-09-17T05:21:08.256433+0000","last_fullsized":"2024-09-17T05:21:08.256433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3456","last_scrub_stamp":"2024-09-17T05:21:08.256215+0000","last_deep_scrub":"65'3456","last_deep_scrub_stamp":"2024-09-17T05:21:08.256215+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.256215+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:05.804588+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18921,"num_write_kb":260292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"67'13","reported_seq":144,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.546383+0000","last_change":"2024-09-17T05:21:07.546066+0000","last_active":"2024-09-17T05:21:07.546383+0000","last_peered":"2024-09-17T05:21:07.546383+0000","last_clean":"2024-09-17T05:21:07.546383+0000","last_became_active":"2024-09-17T04:49:30.908093+0000","last_became_peered":"2024-09-17T04:49:30.908093+0000","last_unstale":"2024-09-17T05:21:07.546383+0000","last_undegraded":"2024-09-17T05:21:07.546383+0000","last_fullsized":"2024-09-17T05:21:07.546383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'13","last_scrub_stamp":"2024-09-17T05:21:07.545738+0000","last_deep_scrub":"67'13","last_deep_scrub_stamp":"2024-09-17T05:21:07.545738+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.545738+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:29.551757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"68'474","reported_seq":615,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:21.662748+0000","last_change":"2024-09-17T05:21:21.662748+0000","last_active":"2024-09-17T05:21:21.662748+0000","last_peered":"2024-09-17T05:21:21.662748+0000","last_clean":"2024-09-17T05:21:21.662748+0000","last_became_active":"2024-09-17T04:49:40.012458+0000","last_became_peered":"2024-09-17T04:49:40.012458+0000","last_unstale":"2024-09-17T05:21:21.662748+0000","last_undegraded":"2024-09-17T05:21:21.662748+0000","last_fullsized":"2024-09-17T05:21:21.662748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'474","last_scrub_stamp":"2024-09-17T05:21:21.662405+0000","last_deep_scrub":"68'474","last_deep_scrub_stamp":"2024-09-17T05:21:21.662405+0000","last_clean_scrub_stamp":"2024-09-17T05:21:21.662405+0000","objects_scrubbed":1,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:56:32.685481+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":54,"num_write":1100,"num_write_kb":354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"68'3699","reported_seq":3820,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.726710+0000","last_change":"2024-09-17T05:21:20.726710+0000","last_active":"2024-09-17T05:21:20.726710+0000","last_peered":"2024-09-17T05:21:20.726710+0000","last_clean":"2024-09-17T05:21:20.726710+0000","last_became_active":"2024-09-17T04:49:41.052258+0000","last_became_peered":"2024-09-17T04:49:41.052258+0000","last_unstale":"2024-09-17T05:21:20.726710+0000","last_undegraded":"2024-09-17T05:21:20.726710+0000","last_fullsized":"2024-09-17T05:21:20.726710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3699","last_scrub_stamp":"2024-09-17T05:21:20.726343+0000","last_deep_scrub":"68'3699","last_deep_scrub_stamp":"2024-09-17T05:21:20.726343+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.726343+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:00:12.900330+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":20021,"num_write_kb":309708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"65'3651","reported_seq":3779,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.899240+0000","last_change":"2024-09-17T05:21:08.899240+0000","last_active":"2024-09-17T05:21:08.899240+0000","last_peered":"2024-09-17T05:21:08.899240+0000","last_clean":"2024-09-17T05:21:08.899240+0000","last_became_active":"2024-09-17T04:49:31.893605+0000","last_became_peered":"2024-09-17T04:49:31.893605+0000","last_unstale":"2024-09-17T05:21:08.899240+0000","last_undegraded":"2024-09-17T05:21:08.899240+0000","last_fullsized":"2024-09-17T05:21:08.899240+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3651","last_scrub_stamp":"2024-09-17T05:21:08.899022+0000","last_deep_scrub":"65'3651","last_deep_scrub_stamp":"2024-09-17T05:21:08.899022+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.899022+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:20.889499+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19517,"num_write_kb":274922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"65'3","reported_seq":133,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:03.435826+0000","last_change":"2024-09-17T05:21:03.435826+0000","last_active":"2024-09-17T05:21:03.435826+0000","last_peered":"2024-09-17T05:21:03.435826+0000","last_clean":"2024-09-17T05:21:03.435826+0000","last_became_active":"2024-09-17T04:49:30.881407+0000","last_became_peered":"2024-09-17T04:49:30.881407+0000","last_unstale":"2024-09-17T05:21:03.435826+0000","last_undegraded":"2024-09-17T05:21:03.435826+0000","last_fullsized":"2024-09-17T05:21:03.435826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-09-17T05:21:03.435568+0000","last_deep_scrub":"65'3","last_deep_scrub_stamp":"2024-09-17T05:21:03.435568+0000","last_clean_scrub_stamp":"2024-09-17T05:21:03.435568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:15.962476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"68'74","reported_seq":229,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.023466+0000","last_change":"2024-09-17T05:21:05.023177+0000","last_active":"2024-09-17T05:21:05.023466+0000","last_peered":"2024-09-17T05:21:05.023466+0000","last_clean":"2024-09-17T05:21:05.023466+0000","last_became_active":"2024-09-17T04:49:22.679071+0000","last_became_peered":"2024-09-17T04:49:22.679071+0000","last_unstale":"2024-09-17T05:21:05.023466+0000","last_undegraded":"2024-09-17T05:21:05.023466+0000","last_fullsized":"2024-09-17T05:21:05.023466+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'74","last_scrub_stamp":"2024-09-17T05:21:05.022924+0000","last_deep_scrub":"68'74","last_deep_scrub_stamp":"2024-09-17T05:21:05.022924+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.022924+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:13.286994+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"68'2","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:25.395036+0000","last_change":"2024-09-17T05:21:25.395036+0000","last_active":"2024-09-17T05:21:25.395036+0000","last_peered":"2024-09-17T05:21:25.395036+0000","last_clean":"2024-09-17T05:21:25.395036+0000","last_became_active":"2024-09-17T04:49:39.995454+0000","last_became_peered":"2024-09-17T04:49:39.995454+0000","last_unstale":"2024-09-17T05:21:25.395036+0000","last_undegraded":"2024-09-17T05:21:25.395036+0000","last_fullsized":"2024-09-17T05:21:25.395036+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T05:21:25.394937+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2024-09-17T05:21:25.394937+0000","last_clean_scrub_stamp":"2024-09-17T05:21:25.394937+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:56.544385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"68'2130","reported_seq":2250,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:23.664308+0000","last_change":"2024-09-17T05:21:23.664308+0000","last_active":"2024-09-17T05:21:23.664308+0000","last_peered":"2024-09-17T05:21:23.664308+0000","last_clean":"2024-09-17T05:21:23.664308+0000","last_became_active":"2024-09-17T04:49:41.054142+0000","last_became_peered":"2024-09-17T04:49:41.054142+0000","last_unstale":"2024-09-17T05:21:23.664308+0000","last_undegraded":"2024-09-17T05:21:23.664308+0000","last_fullsized":"2024-09-17T05:21:23.664308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2130","last_scrub_stamp":"2024-09-17T05:21:23.664077+0000","last_deep_scrub":"68'2130","last_deep_scrub_stamp":"2024-09-17T05:21:23.664077+0000","last_clean_scrub_stamp":"2024-09-17T05:21:23.664077+0000","objects_scrubbed":0,"log_size":2130,"log_dups_size":0,"ondisk_log_size":2130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:08:39.945821+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":11413,"num_write_kb":185062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"65'4183","reported_seq":4310,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.501868+0000","last_change":"2024-09-17T05:21:14.501868+0000","last_active":"2024-09-17T05:21:14.501868+0000","last_peered":"2024-09-17T05:21:14.501868+0000","last_clean":"2024-09-17T05:21:14.501868+0000","last_became_active":"2024-09-17T04:49:31.893810+0000","last_became_peered":"2024-09-17T04:49:31.893810+0000","last_unstale":"2024-09-17T05:21:14.501868+0000","last_undegraded":"2024-09-17T05:21:14.501868+0000","last_fullsized":"2024-09-17T05:21:14.501868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4183","last_scrub_stamp":"2024-09-17T05:21:14.501644+0000","last_deep_scrub":"65'4183","last_deep_scrub_stamp":"2024-09-17T05:21:14.501644+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.501644+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:46.619129+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22326,"num_write_kb":302603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"65'5","reported_seq":134,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.248291+0000","last_change":"2024-09-17T05:21:12.248291+0000","last_active":"2024-09-17T05:21:12.248291+0000","last_peered":"2024-09-17T05:21:12.248291+0000","last_clean":"2024-09-17T05:21:12.248291+0000","last_became_active":"2024-09-17T04:49:30.883493+0000","last_became_peered":"2024-09-17T04:49:30.883493+0000","last_unstale":"2024-09-17T05:21:12.248291+0000","last_undegraded":"2024-09-17T05:21:12.248291+0000","last_fullsized":"2024-09-17T05:21:12.248291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-17T05:21:12.247947+0000","last_deep_scrub":"65'5","last_deep_scrub_stamp":"2024-09-17T05:21:12.247947+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.247947+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:37.743922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"68'3","reported_seq":124,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.782673+0000","last_change":"2024-09-17T05:21:11.782673+0000","last_active":"2024-09-17T05:21:11.782673+0000","last_peered":"2024-09-17T05:21:11.782673+0000","last_clean":"2024-09-17T05:21:11.782673+0000","last_became_active":"2024-09-17T04:49:39.990426+0000","last_became_peered":"2024-09-17T04:49:39.990426+0000","last_unstale":"2024-09-17T05:21:11.782673+0000","last_undegraded":"2024-09-17T05:21:11.782673+0000","last_fullsized":"2024-09-17T05:21:11.782673+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-09-17T05:21:11.782336+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2024-09-17T05:21:11.782336+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.782336+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:23.772814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"68'3285","reported_seq":3405,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:21.247219+0000","last_change":"2024-09-17T05:21:21.247219+0000","last_active":"2024-09-17T05:21:21.247219+0000","last_peered":"2024-09-17T05:21:21.247219+0000","last_clean":"2024-09-17T05:21:21.247219+0000","last_became_active":"2024-09-17T04:49:41.023339+0000","last_became_peered":"2024-09-17T04:49:41.023339+0000","last_unstale":"2024-09-17T05:21:21.247219+0000","last_undegraded":"2024-09-17T05:21:21.247219+0000","last_fullsized":"2024-09-17T05:21:21.247219+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3285","last_scrub_stamp":"2024-09-17T05:21:21.246882+0000","last_deep_scrub":"68'3285","last_deep_scrub_stamp":"2024-09-17T05:21:21.246882+0000","last_clean_scrub_stamp":"2024-09-17T05:21:21.246882+0000","objects_scrubbed":0,"log_size":3285,"log_dups_size":0,"ondisk_log_size":3285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:36.094827+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18245,"num_write_kb":282158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"65'4407","reported_seq":4534,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.778487+0000","last_change":"2024-09-17T05:21:08.778487+0000","last_active":"2024-09-17T05:21:08.778487+0000","last_peered":"2024-09-17T05:21:08.778487+0000","last_clean":"2024-09-17T05:21:08.778487+0000","last_became_active":"2024-09-17T04:49:31.887274+0000","last_became_peered":"2024-09-17T04:49:31.887274+0000","last_unstale":"2024-09-17T05:21:08.778487+0000","last_undegraded":"2024-09-17T05:21:08.778487+0000","last_fullsized":"2024-09-17T05:21:08.778487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4407","last_scrub_stamp":"2024-09-17T05:21:08.778262+0000","last_deep_scrub":"65'4407","last_deep_scrub_stamp":"2024-09-17T05:21:08.778262+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.778262+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:43.795331+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24558,"num_write_kb":315810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"68'7","reported_seq":157,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.632166+0000","last_change":"2024-09-17T05:21:12.632166+0000","last_active":"2024-09-17T05:21:12.632166+0000","last_peered":"2024-09-17T05:21:12.632166+0000","last_clean":"2024-09-17T05:21:12.632166+0000","last_became_active":"2024-09-17T04:49:30.894651+0000","last_became_peered":"2024-09-17T04:49:30.894651+0000","last_unstale":"2024-09-17T05:21:12.632166+0000","last_undegraded":"2024-09-17T05:21:12.632166+0000","last_fullsized":"2024-09-17T05:21:12.632166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'7","last_scrub_stamp":"2024-09-17T05:21:12.631953+0000","last_deep_scrub":"68'7","last_deep_scrub_stamp":"2024-09-17T05:21:12.631953+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.631953+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:52:04.393430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'1447","reported_seq":1612,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.697910+0000","last_change":"2024-09-17T05:21:19.697910+0000","last_active":"2024-09-17T05:21:19.697910+0000","last_peered":"2024-09-17T05:21:19.697910+0000","last_clean":"2024-09-17T05:21:19.697910+0000","last_became_active":"2024-09-17T04:49:40.017602+0000","last_became_peered":"2024-09-17T04:49:40.017602+0000","last_unstale":"2024-09-17T05:21:19.697910+0000","last_undegraded":"2024-09-17T05:21:19.697910+0000","last_fullsized":"2024-09-17T05:21:19.697910+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1447","last_scrub_stamp":"2024-09-17T05:21:19.697700+0000","last_deep_scrub":"68'1447","last_deep_scrub_stamp":"2024-09-17T05:21:19.697700+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.697700+0000","objects_scrubbed":2,"log_size":1447,"log_dups_size":0,"ondisk_log_size":1447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:32.224608+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":38239,"num_write":1421,"num_write_kb":4489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'2587","reported_seq":2706,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.964539+0000","last_change":"2024-09-17T05:21:14.964539+0000","last_active":"2024-09-17T05:21:14.964539+0000","last_peered":"2024-09-17T05:21:14.964539+0000","last_clean":"2024-09-17T05:21:14.964539+0000","last_became_active":"2024-09-17T04:49:41.030054+0000","last_became_peered":"2024-09-17T04:49:41.030054+0000","last_unstale":"2024-09-17T05:21:14.964539+0000","last_undegraded":"2024-09-17T05:21:14.964539+0000","last_fullsized":"2024-09-17T05:21:14.964539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2587","last_scrub_stamp":"2024-09-17T05:21:14.964208+0000","last_deep_scrub":"68'2587","last_deep_scrub_stamp":"2024-09-17T05:21:14.964208+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.964208+0000","objects_scrubbed":0,"log_size":2587,"log_dups_size":0,"ondisk_log_size":2587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:16.231401+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13558,"num_write_kb":238553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"65'4762","reported_seq":4889,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.698211+0000","last_change":"2024-09-17T05:21:12.698211+0000","last_active":"2024-09-17T05:21:12.698211+0000","last_peered":"2024-09-17T05:21:12.698211+0000","last_clean":"2024-09-17T05:21:12.698211+0000","last_became_active":"2024-09-17T04:49:31.889402+0000","last_became_peered":"2024-09-17T04:49:31.889402+0000","last_unstale":"2024-09-17T05:21:12.698211+0000","last_undegraded":"2024-09-17T05:21:12.698211+0000","last_fullsized":"2024-09-17T05:21:12.698211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4762","last_scrub_stamp":"2024-09-17T05:21:12.697995+0000","last_deep_scrub":"65'4762","last_deep_scrub_stamp":"2024-09-17T05:21:12.697995+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.697995+0000","objects_scrubbed":0,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:13.662257+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25049,"num_write_kb":354398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"68'8","reported_seq":179,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.914619+0000","last_change":"2024-09-17T05:21:05.914619+0000","last_active":"2024-09-17T05:21:05.914619+0000","last_peered":"2024-09-17T05:21:05.914619+0000","last_clean":"2024-09-17T05:21:05.914619+0000","last_became_active":"2024-09-17T04:49:30.903139+0000","last_became_peered":"2024-09-17T04:49:30.903139+0000","last_unstale":"2024-09-17T05:21:05.914619+0000","last_undegraded":"2024-09-17T05:21:05.914619+0000","last_fullsized":"2024-09-17T05:21:05.914619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'8","last_scrub_stamp":"2024-09-17T05:21:05.914411+0000","last_deep_scrub":"68'8","last_deep_scrub_stamp":"2024-09-17T05:21:05.914411+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.914411+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:50.560424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"68'5","reported_seq":126,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:24.415661+0000","last_change":"2024-09-17T05:21:24.415661+0000","last_active":"2024-09-17T05:21:24.415661+0000","last_peered":"2024-09-17T05:21:24.415661+0000","last_clean":"2024-09-17T05:21:24.415661+0000","last_became_active":"2024-09-17T04:49:39.990367+0000","last_became_peered":"2024-09-17T04:49:39.990367+0000","last_unstale":"2024-09-17T05:21:24.415661+0000","last_undegraded":"2024-09-17T05:21:24.415661+0000","last_fullsized":"2024-09-17T05:21:24.415661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:24.415448+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:24.415448+0000","last_clean_scrub_stamp":"2024-09-17T05:21:24.415448+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:23:49.780711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"68'2327","reported_seq":2446,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.199619+0000","last_change":"2024-09-17T05:21:17.199619+0000","last_active":"2024-09-17T05:21:17.199619+0000","last_peered":"2024-09-17T05:21:17.199619+0000","last_clean":"2024-09-17T05:21:17.199619+0000","last_became_active":"2024-09-17T04:49:41.060444+0000","last_became_peered":"2024-09-17T04:49:41.060444+0000","last_unstale":"2024-09-17T05:21:17.199619+0000","last_undegraded":"2024-09-17T05:21:17.199619+0000","last_fullsized":"2024-09-17T05:21:17.199619+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2327","last_scrub_stamp":"2024-09-17T05:21:17.199339+0000","last_deep_scrub":"68'2327","last_deep_scrub_stamp":"2024-09-17T05:21:17.199339+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.199339+0000","objects_scrubbed":0,"log_size":2327,"log_dups_size":0,"ondisk_log_size":2327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:53.006691+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12892,"num_write_kb":211370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"65'3722","reported_seq":3849,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:03.644226+0000","last_change":"2024-09-17T05:21:03.643932+0000","last_active":"2024-09-17T05:21:03.644226+0000","last_peered":"2024-09-17T05:21:03.644226+0000","last_clean":"2024-09-17T05:21:03.644226+0000","last_became_active":"2024-09-17T04:49:31.903922+0000","last_became_peered":"2024-09-17T04:49:31.903922+0000","last_unstale":"2024-09-17T05:21:03.644226+0000","last_undegraded":"2024-09-17T05:21:03.644226+0000","last_fullsized":"2024-09-17T05:21:03.644226+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3722","last_scrub_stamp":"2024-09-17T05:21:03.643638+0000","last_deep_scrub":"65'3722","last_deep_scrub_stamp":"2024-09-17T05:21:03.643638+0000","last_clean_scrub_stamp":"2024-09-17T05:21:03.643638+0000","objects_scrubbed":0,"log_size":3722,"log_dups_size":0,"ondisk_log_size":3722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:49:58.849136+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19724,"num_write_kb":263684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"68'5","reported_seq":155,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:02.705197+0000","last_change":"2024-09-17T05:21:02.705029+0000","last_active":"2024-09-17T05:21:02.705197+0000","last_peered":"2024-09-17T05:21:02.705197+0000","last_clean":"2024-09-17T05:21:02.705197+0000","last_became_active":"2024-09-17T04:49:30.902093+0000","last_became_peered":"2024-09-17T04:49:30.902093+0000","last_unstale":"2024-09-17T05:21:02.705197+0000","last_undegraded":"2024-09-17T05:21:02.705197+0000","last_fullsized":"2024-09-17T05:21:02.705197+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:02.704797+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:02.704797+0000","last_clean_scrub_stamp":"2024-09-17T05:21:02.704797+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:20.480361+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"68'2","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.966186+0000","last_change":"2024-09-17T05:21:13.966186+0000","last_active":"2024-09-17T05:21:13.966186+0000","last_peered":"2024-09-17T05:21:13.966186+0000","last_clean":"2024-09-17T05:21:13.966186+0000","last_became_active":"2024-09-17T04:49:40.002552+0000","last_became_peered":"2024-09-17T04:49:40.002552+0000","last_unstale":"2024-09-17T05:21:13.966186+0000","last_undegraded":"2024-09-17T05:21:13.966186+0000","last_fullsized":"2024-09-17T05:21:13.966186+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-17T05:21:13.965981+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2024-09-17T05:21:13.965981+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.965981+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:20:33.477072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"68'3186","reported_seq":3304,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:22.852177+0000","last_change":"2024-09-17T05:21:22.852177+0000","last_active":"2024-09-17T05:21:22.852177+0000","last_peered":"2024-09-17T05:21:22.852177+0000","last_clean":"2024-09-17T05:21:22.852177+0000","last_became_active":"2024-09-17T04:49:41.041600+0000","last_became_peered":"2024-09-17T04:49:41.041600+0000","last_unstale":"2024-09-17T05:21:22.852177+0000","last_undegraded":"2024-09-17T05:21:22.852177+0000","last_fullsized":"2024-09-17T05:21:22.852177+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3186","last_scrub_stamp":"2024-09-17T05:21:22.851950+0000","last_deep_scrub":"68'3186","last_deep_scrub_stamp":"2024-09-17T05:21:22.851950+0000","last_clean_scrub_stamp":"2024-09-17T05:21:22.851950+0000","objects_scrubbed":0,"log_size":3186,"log_dups_size":0,"ondisk_log_size":3186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:07:12.408132+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16763,"num_write_kb":264037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"65'3694","reported_seq":3822,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.531689+0000","last_change":"2024-09-17T05:21:16.531689+0000","last_active":"2024-09-17T05:21:16.531689+0000","last_peered":"2024-09-17T05:21:16.531689+0000","last_clean":"2024-09-17T05:21:16.531689+0000","last_became_active":"2024-09-17T04:49:31.912224+0000","last_became_peered":"2024-09-17T04:49:31.912224+0000","last_unstale":"2024-09-17T05:21:16.531689+0000","last_undegraded":"2024-09-17T05:21:16.531689+0000","last_fullsized":"2024-09-17T05:21:16.531689+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3694","last_scrub_stamp":"2024-09-17T05:21:16.531452+0000","last_deep_scrub":"65'3694","last_deep_scrub_stamp":"2024-09-17T05:21:16.531452+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.531452+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:37.863588+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20762,"num_write_kb":284135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"68'19","reported_seq":235,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.923705+0000","last_change":"2024-09-17T05:21:06.923705+0000","last_active":"2024-09-17T05:21:06.923705+0000","last_peered":"2024-09-17T05:21:06.923705+0000","last_clean":"2024-09-17T05:21:06.923705+0000","last_became_active":"2024-09-17T04:49:30.889046+0000","last_became_peered":"2024-09-17T04:49:30.889046+0000","last_unstale":"2024-09-17T05:21:06.923705+0000","last_undegraded":"2024-09-17T05:21:06.923705+0000","last_fullsized":"2024-09-17T05:21:06.923705+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-17T05:21:06.923348+0000","last_deep_scrub":"68'19","last_deep_scrub_stamp":"2024-09-17T05:21:06.923348+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.923348+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:50.769232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":264,"num_read_kb":264,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"68'3","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:25.626292+0000","last_change":"2024-09-17T05:21:25.626292+0000","last_active":"2024-09-17T05:21:25.626292+0000","last_peered":"2024-09-17T05:21:25.626292+0000","last_clean":"2024-09-17T05:21:25.626292+0000","last_became_active":"2024-09-17T04:49:40.021166+0000","last_became_peered":"2024-09-17T04:49:40.021166+0000","last_unstale":"2024-09-17T05:21:25.626292+0000","last_undegraded":"2024-09-17T05:21:25.626292+0000","last_fullsized":"2024-09-17T05:21:25.626292+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-09-17T05:21:25.626061+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2024-09-17T05:21:25.626061+0000","last_clean_scrub_stamp":"2024-09-17T05:21:25.626061+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:18:39.258734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"68'3074","reported_seq":3195,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:34.355799+0000","last_change":"2024-09-17T05:21:34.355799+0000","last_active":"2024-09-17T05:21:34.355799+0000","last_peered":"2024-09-17T05:21:34.355799+0000","last_clean":"2024-09-17T05:21:34.355799+0000","last_became_active":"2024-09-17T04:49:41.064373+0000","last_became_peered":"2024-09-17T04:49:41.064373+0000","last_unstale":"2024-09-17T05:21:34.355799+0000","last_undegraded":"2024-09-17T05:21:34.355799+0000","last_fullsized":"2024-09-17T05:21:34.355799+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3074","last_scrub_stamp":"2024-09-17T05:21:34.355435+0000","last_deep_scrub":"68'3074","last_deep_scrub_stamp":"2024-09-17T05:21:34.355435+0000","last_clean_scrub_stamp":"2024-09-17T05:21:34.355435+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:29.254257+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":16509,"num_write_kb":287321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"65'2828","reported_seq":2954,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:03.031642+0000","last_change":"2024-09-17T05:21:03.031642+0000","last_active":"2024-09-17T05:21:03.031642+0000","last_peered":"2024-09-17T05:21:03.031642+0000","last_clean":"2024-09-17T05:21:03.031642+0000","last_became_active":"2024-09-17T04:49:31.891074+0000","last_became_peered":"2024-09-17T04:49:31.891074+0000","last_unstale":"2024-09-17T05:21:03.031642+0000","last_undegraded":"2024-09-17T05:21:03.031642+0000","last_fullsized":"2024-09-17T05:21:03.031642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2828","last_scrub_stamp":"2024-09-17T05:21:03.031361+0000","last_deep_scrub":"65'2828","last_deep_scrub_stamp":"2024-09-17T05:21:03.031361+0000","last_clean_scrub_stamp":"2024-09-17T05:21:03.031361+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:45:33.333087+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14821,"num_write_kb":206837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"65'5","reported_seq":133,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.726265+0000","last_change":"2024-09-17T05:21:05.726265+0000","last_active":"2024-09-17T05:21:05.726265+0000","last_peered":"2024-09-17T05:21:05.726265+0000","last_clean":"2024-09-17T05:21:05.726265+0000","last_became_active":"2024-09-17T04:49:30.886036+0000","last_became_peered":"2024-09-17T04:49:30.886036+0000","last_unstale":"2024-09-17T05:21:05.726265+0000","last_undegraded":"2024-09-17T05:21:05.726265+0000","last_fullsized":"2024-09-17T05:21:05.726265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-09-17T05:21:05.726041+0000","last_deep_scrub":"65'5","last_deep_scrub_stamp":"2024-09-17T05:21:05.726041+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.726041+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:36.025332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"68'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.699238+0000","last_change":"2024-09-17T05:21:20.699238+0000","last_active":"2024-09-17T05:21:20.699238+0000","last_peered":"2024-09-17T05:21:20.699238+0000","last_clean":"2024-09-17T05:21:20.699238+0000","last_became_active":"2024-09-17T04:49:40.017025+0000","last_became_peered":"2024-09-17T04:49:40.017025+0000","last_unstale":"2024-09-17T05:21:20.699238+0000","last_undegraded":"2024-09-17T05:21:20.699238+0000","last_fullsized":"2024-09-17T05:21:20.699238+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:20.699035+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:20.699035+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.699035+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:54.788080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"68'2590","reported_seq":2708,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:32.385276+0000","last_change":"2024-09-17T05:21:32.385276+0000","last_active":"2024-09-17T05:21:32.385276+0000","last_peered":"2024-09-17T05:21:32.385276+0000","last_clean":"2024-09-17T05:21:32.385276+0000","last_became_active":"2024-09-17T04:49:41.073775+0000","last_became_peered":"2024-09-17T04:49:41.073775+0000","last_unstale":"2024-09-17T05:21:32.385276+0000","last_undegraded":"2024-09-17T05:21:32.385276+0000","last_fullsized":"2024-09-17T05:21:32.385276+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2590","last_scrub_stamp":"2024-09-17T05:21:32.385057+0000","last_deep_scrub":"68'2590","last_deep_scrub_stamp":"2024-09-17T05:21:32.385057+0000","last_clean_scrub_stamp":"2024-09-17T05:21:32.385057+0000","objects_scrubbed":0,"log_size":2590,"log_dups_size":0,"ondisk_log_size":2590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:23.520239+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14722,"num_write_kb":243326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"65'4461","reported_seq":4587,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.543088+0000","last_change":"2024-09-17T05:21:12.543088+0000","last_active":"2024-09-17T05:21:12.543088+0000","last_peered":"2024-09-17T05:21:12.543088+0000","last_clean":"2024-09-17T05:21:12.543088+0000","last_became_active":"2024-09-17T04:49:31.889669+0000","last_became_peered":"2024-09-17T04:49:31.889669+0000","last_unstale":"2024-09-17T05:21:12.543088+0000","last_undegraded":"2024-09-17T05:21:12.543088+0000","last_fullsized":"2024-09-17T05:21:12.543088+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4461","last_scrub_stamp":"2024-09-17T05:21:12.542724+0000","last_deep_scrub":"65'4461","last_deep_scrub_stamp":"2024-09-17T05:21:12.542724+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.542724+0000","objects_scrubbed":0,"log_size":4461,"log_dups_size":0,"ondisk_log_size":4461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:07:02.098906+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24667,"num_write_kb":332445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"65'3","reported_seq":131,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.526247+0000","last_change":"2024-09-17T05:21:09.526247+0000","last_active":"2024-09-17T05:21:09.526247+0000","last_peered":"2024-09-17T05:21:09.526247+0000","last_clean":"2024-09-17T05:21:09.526247+0000","last_became_active":"2024-09-17T04:49:30.891171+0000","last_became_peered":"2024-09-17T04:49:30.891171+0000","last_unstale":"2024-09-17T05:21:09.526247+0000","last_undegraded":"2024-09-17T05:21:09.526247+0000","last_fullsized":"2024-09-17T05:21:09.526247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-09-17T05:21:09.525904+0000","last_deep_scrub":"65'3","last_deep_scrub_stamp":"2024-09-17T05:21:09.525904+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.525904+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:27.874892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"68'371","reported_seq":591,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.214729+0000","last_change":"2024-09-17T05:21:20.214729+0000","last_active":"2024-09-17T05:21:20.214729+0000","last_peered":"2024-09-17T05:21:20.214729+0000","last_clean":"2024-09-17T05:21:20.214729+0000","last_became_active":"2024-09-17T04:49:40.003708+0000","last_became_peered":"2024-09-17T04:49:40.003708+0000","last_unstale":"2024-09-17T05:21:20.214729+0000","last_undegraded":"2024-09-17T05:21:20.214729+0000","last_fullsized":"2024-09-17T05:21:20.214729+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'371","last_scrub_stamp":"2024-09-17T05:21:20.214482+0000","last_deep_scrub":"68'371","last_deep_scrub_stamp":"2024-09-17T05:21:20.214482+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.214482+0000","objects_scrubbed":4,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:09:21.303757+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":180,"num_read_kb":82080,"num_write":364,"num_write_kb":4295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"68'3856","reported_seq":3969,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:24.633672+0000","last_change":"2024-09-17T05:21:24.633672+0000","last_active":"2024-09-17T05:21:24.633672+0000","last_peered":"2024-09-17T05:21:24.633672+0000","last_clean":"2024-09-17T05:21:24.633672+0000","last_became_active":"2024-09-17T04:50:12.066741+0000","last_became_peered":"2024-09-17T04:50:12.066741+0000","last_unstale":"2024-09-17T05:21:24.633672+0000","last_undegraded":"2024-09-17T05:21:24.633672+0000","last_fullsized":"2024-09-17T05:21:24.633672+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3856","last_scrub_stamp":"2024-09-17T05:21:24.633451+0000","last_deep_scrub":"68'3856","last_deep_scrub_stamp":"2024-09-17T05:21:24.633451+0000","last_clean_scrub_stamp":"2024-09-17T05:21:24.633451+0000","objects_scrubbed":0,"log_size":3856,"log_dups_size":0,"ondisk_log_size":3856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:54.420240+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":20645,"num_write_kb":344954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"65'3521","reported_seq":3647,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.207139+0000","last_change":"2024-09-17T05:21:09.207139+0000","last_active":"2024-09-17T05:21:09.207139+0000","last_peered":"2024-09-17T05:21:09.207139+0000","last_clean":"2024-09-17T05:21:09.207139+0000","last_became_active":"2024-09-17T04:49:31.883304+0000","last_became_peered":"2024-09-17T04:49:31.883304+0000","last_unstale":"2024-09-17T05:21:09.207139+0000","last_undegraded":"2024-09-17T05:21:09.207139+0000","last_fullsized":"2024-09-17T05:21:09.207139+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3521","last_scrub_stamp":"2024-09-17T05:21:09.206770+0000","last_deep_scrub":"65'3521","last_deep_scrub_stamp":"2024-09-17T05:21:09.206770+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.206770+0000","objects_scrubbed":0,"log_size":3521,"log_dups_size":0,"ondisk_log_size":3521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:41.837932+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19198,"num_write_kb":253218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"68'12","reported_seq":182,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.168760+0000","last_change":"2024-09-17T05:21:06.168760+0000","last_active":"2024-09-17T05:21:06.168760+0000","last_peered":"2024-09-17T05:21:06.168760+0000","last_clean":"2024-09-17T05:21:06.168760+0000","last_became_active":"2024-09-17T04:49:30.893819+0000","last_became_peered":"2024-09-17T04:49:30.893819+0000","last_unstale":"2024-09-17T05:21:06.168760+0000","last_undegraded":"2024-09-17T05:21:06.168760+0000","last_fullsized":"2024-09-17T05:21:06.168760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'12","last_scrub_stamp":"2024-09-17T05:21:06.168428+0000","last_deep_scrub":"68'12","last_deep_scrub_stamp":"2024-09-17T05:21:06.168428+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.168428+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:20:32.765263+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":168,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"68'32","reported_seq":192,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:27.402073+0000","last_change":"2024-09-17T05:21:27.402073+0000","last_active":"2024-09-17T05:21:27.402073+0000","last_peered":"2024-09-17T05:21:27.402073+0000","last_clean":"2024-09-17T05:21:27.402073+0000","last_became_active":"2024-09-17T04:49:39.994379+0000","last_became_peered":"2024-09-17T04:49:39.994379+0000","last_unstale":"2024-09-17T05:21:27.402073+0000","last_undegraded":"2024-09-17T05:21:27.402073+0000","last_fullsized":"2024-09-17T05:21:27.402073+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'32","last_scrub_stamp":"2024-09-17T05:21:27.401852+0000","last_deep_scrub":"68'32","last_deep_scrub_stamp":"2024-09-17T05:21:27.401852+0000","last_clean_scrub_stamp":"2024-09-17T05:21:27.401852+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:54.558158+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":31,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"68'3085","reported_seq":3203,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:23.849815+0000","last_change":"2024-09-17T05:21:23.849815+0000","last_active":"2024-09-17T05:21:23.849815+0000","last_peered":"2024-09-17T05:21:23.849815+0000","last_clean":"2024-09-17T05:21:23.849815+0000","last_became_active":"2024-09-17T04:49:41.040887+0000","last_became_peered":"2024-09-17T04:49:41.040887+0000","last_unstale":"2024-09-17T05:21:23.849815+0000","last_undegraded":"2024-09-17T05:21:23.849815+0000","last_fullsized":"2024-09-17T05:21:23.849815+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3085","last_scrub_stamp":"2024-09-17T05:21:23.849605+0000","last_deep_scrub":"68'3085","last_deep_scrub_stamp":"2024-09-17T05:21:23.849605+0000","last_clean_scrub_stamp":"2024-09-17T05:21:23.849605+0000","objects_scrubbed":0,"log_size":3085,"log_dups_size":0,"ondisk_log_size":3085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:26.434077+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16824,"num_write_kb":288541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"65'3611","reported_seq":3737,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.758081+0000","last_change":"2024-09-17T05:21:09.758081+0000","last_active":"2024-09-17T05:21:09.758081+0000","last_peered":"2024-09-17T05:21:09.758081+0000","last_clean":"2024-09-17T05:21:09.758081+0000","last_became_active":"2024-09-17T04:49:31.892172+0000","last_became_peered":"2024-09-17T04:49:31.892172+0000","last_unstale":"2024-09-17T05:21:09.758081+0000","last_undegraded":"2024-09-17T05:21:09.758081+0000","last_fullsized":"2024-09-17T05:21:09.758081+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3611","last_scrub_stamp":"2024-09-17T05:21:09.757816+0000","last_deep_scrub":"65'3611","last_deep_scrub_stamp":"2024-09-17T05:21:09.757816+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.757816+0000","objects_scrubbed":0,"log_size":3611,"log_dups_size":0,"ondisk_log_size":3611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:17.538586+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19975,"num_write_kb":264442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"68'352","reported_seq":586,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:04.686963+0000","last_change":"2024-09-17T05:21:04.686963+0000","last_active":"2024-09-17T05:21:04.686963+0000","last_peered":"2024-09-17T05:21:04.686963+0000","last_clean":"2024-09-17T05:21:04.686963+0000","last_became_active":"2024-09-17T04:49:30.877773+0000","last_became_peered":"2024-09-17T04:49:30.877773+0000","last_unstale":"2024-09-17T05:21:04.686963+0000","last_undegraded":"2024-09-17T05:21:04.686963+0000","last_fullsized":"2024-09-17T05:21:04.686963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'352","last_scrub_stamp":"2024-09-17T05:21:04.686835+0000","last_deep_scrub":"68'352","last_deep_scrub_stamp":"2024-09-17T05:21:04.686835+0000","last_clean_scrub_stamp":"2024-09-17T05:21:04.686835+0000","objects_scrubbed":4,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:42.202865+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":190,"num_read_kb":82094,"num_write":345,"num_write_kb":4278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"68'3","reported_seq":123,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.866522+0000","last_change":"2024-09-17T05:21:17.866522+0000","last_active":"2024-09-17T05:21:17.866522+0000","last_peered":"2024-09-17T05:21:17.866522+0000","last_clean":"2024-09-17T05:21:17.866522+0000","last_became_active":"2024-09-17T04:49:39.995560+0000","last_became_peered":"2024-09-17T04:49:39.995560+0000","last_unstale":"2024-09-17T05:21:17.866522+0000","last_undegraded":"2024-09-17T05:21:17.866522+0000","last_fullsized":"2024-09-17T05:21:17.866522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2024-09-17T05:21:17.866319+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2024-09-17T05:21:17.866319+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.866319+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:39.872338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"68'2950","reported_seq":3069,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:22.273954+0000","last_change":"2024-09-17T05:21:22.273954+0000","last_active":"2024-09-17T05:21:22.273954+0000","last_peered":"2024-09-17T05:21:22.273954+0000","last_clean":"2024-09-17T05:21:22.273954+0000","last_became_active":"2024-09-17T04:49:41.029557+0000","last_became_peered":"2024-09-17T04:49:41.029557+0000","last_unstale":"2024-09-17T05:21:22.273954+0000","last_undegraded":"2024-09-17T05:21:22.273954+0000","last_fullsized":"2024-09-17T05:21:22.273954+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2950","last_scrub_stamp":"2024-09-17T05:21:22.273719+0000","last_deep_scrub":"68'2950","last_deep_scrub_stamp":"2024-09-17T05:21:22.273719+0000","last_clean_scrub_stamp":"2024-09-17T05:21:22.273719+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:00.694221+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":16110,"num_write_kb":249022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"65'4529","reported_seq":4655,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.533427+0000","last_change":"2024-09-17T05:21:17.533427+0000","last_active":"2024-09-17T05:21:17.533427+0000","last_peered":"2024-09-17T05:21:17.533427+0000","last_clean":"2024-09-17T05:21:17.533427+0000","last_became_active":"2024-09-17T04:49:31.895476+0000","last_became_peered":"2024-09-17T04:49:31.895476+0000","last_unstale":"2024-09-17T05:21:17.533427+0000","last_undegraded":"2024-09-17T05:21:17.533427+0000","last_fullsized":"2024-09-17T05:21:17.533427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4529","last_scrub_stamp":"2024-09-17T05:21:17.533062+0000","last_deep_scrub":"65'4529","last_deep_scrub_stamp":"2024-09-17T05:21:17.533062+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.533062+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:12.954223+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23581,"num_write_kb":327974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"68'32","reported_seq":223,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.167606+0000","last_change":"2024-09-17T05:21:07.167606+0000","last_active":"2024-09-17T05:21:07.167606+0000","last_peered":"2024-09-17T05:21:07.167606+0000","last_clean":"2024-09-17T05:21:07.167606+0000","last_became_active":"2024-09-17T04:49:30.895095+0000","last_became_peered":"2024-09-17T04:49:30.895095+0000","last_unstale":"2024-09-17T05:21:07.167606+0000","last_undegraded":"2024-09-17T05:21:07.167606+0000","last_fullsized":"2024-09-17T05:21:07.167606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'32","last_scrub_stamp":"2024-09-17T05:21:07.167388+0000","last_deep_scrub":"68'32","last_deep_scrub_stamp":"2024-09-17T05:21:07.167388+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.167388+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:03:12.272537+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":147,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"68'10","reported_seq":170,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.729215+0000","last_change":"2024-09-17T05:21:13.729215+0000","last_active":"2024-09-17T05:21:13.729215+0000","last_peered":"2024-09-17T05:21:13.729215+0000","last_clean":"2024-09-17T05:21:13.729215+0000","last_became_active":"2024-09-17T04:49:39.989899+0000","last_became_peered":"2024-09-17T04:49:39.989899+0000","last_unstale":"2024-09-17T05:21:13.729215+0000","last_undegraded":"2024-09-17T05:21:13.729215+0000","last_fullsized":"2024-09-17T05:21:13.729215+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'10","last_scrub_stamp":"2024-09-17T05:21:13.728980+0000","last_deep_scrub":"68'10","last_deep_scrub_stamp":"2024-09-17T05:21:13.728980+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.728980+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:49:29.211223+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":160,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"68'2811","reported_seq":2932,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:33.397158+0000","last_change":"2024-09-17T05:21:33.397158+0000","last_active":"2024-09-17T05:21:33.397158+0000","last_peered":"2024-09-17T05:21:33.397158+0000","last_clean":"2024-09-17T05:21:33.397158+0000","last_became_active":"2024-09-17T04:49:41.051789+0000","last_became_peered":"2024-09-17T04:49:41.051789+0000","last_unstale":"2024-09-17T05:21:33.397158+0000","last_undegraded":"2024-09-17T05:21:33.397158+0000","last_fullsized":"2024-09-17T05:21:33.397158+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2811","last_scrub_stamp":"2024-09-17T05:21:33.396927+0000","last_deep_scrub":"68'2811","last_deep_scrub_stamp":"2024-09-17T05:21:33.396927+0000","last_clean_scrub_stamp":"2024-09-17T05:21:33.396927+0000","objects_scrubbed":0,"log_size":2811,"log_dups_size":0,"ondisk_log_size":2811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:37.576569+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15712,"num_write_kb":243392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"65'4109","reported_seq":4235,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.672735+0000","last_change":"2024-09-17T05:21:09.672735+0000","last_active":"2024-09-17T05:21:09.672735+0000","last_peered":"2024-09-17T05:21:09.672735+0000","last_clean":"2024-09-17T05:21:09.672735+0000","last_became_active":"2024-09-17T04:49:31.894077+0000","last_became_peered":"2024-09-17T04:49:31.894077+0000","last_unstale":"2024-09-17T05:21:09.672735+0000","last_undegraded":"2024-09-17T05:21:09.672735+0000","last_fullsized":"2024-09-17T05:21:09.672735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4109","last_scrub_stamp":"2024-09-17T05:21:09.672433+0000","last_deep_scrub":"65'4109","last_deep_scrub_stamp":"2024-09-17T05:21:09.672433+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.672433+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:53:46.138168+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21556,"num_write_kb":307833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"65'3","reported_seq":131,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:03.722415+0000","last_change":"2024-09-17T05:21:03.722415+0000","last_active":"2024-09-17T05:21:03.722415+0000","last_peered":"2024-09-17T05:21:03.722415+0000","last_clean":"2024-09-17T05:21:03.722415+0000","last_became_active":"2024-09-17T04:49:30.871596+0000","last_became_peered":"2024-09-17T04:49:30.871596+0000","last_unstale":"2024-09-17T05:21:03.722415+0000","last_undegraded":"2024-09-17T05:21:03.722415+0000","last_fullsized":"2024-09-17T05:21:03.722415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-09-17T05:21:03.722284+0000","last_deep_scrub":"65'3","last_deep_scrub_stamp":"2024-09-17T05:21:03.722284+0000","last_clean_scrub_stamp":"2024-09-17T05:21:03.722284+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:00:13.715589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'19","reported_seq":221,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.126158+0000","last_change":"2024-09-17T05:21:12.126158+0000","last_active":"2024-09-17T05:21:12.126158+0000","last_peered":"2024-09-17T05:21:12.126158+0000","last_clean":"2024-09-17T05:21:12.126158+0000","last_became_active":"2024-09-17T04:49:39.997957+0000","last_became_peered":"2024-09-17T04:49:39.997957+0000","last_unstale":"2024-09-17T05:21:12.126158+0000","last_undegraded":"2024-09-17T05:21:12.126158+0000","last_fullsized":"2024-09-17T05:21:12.126158+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'19","last_scrub_stamp":"2024-09-17T05:21:12.125935+0000","last_deep_scrub":"68'19","last_deep_scrub_stamp":"2024-09-17T05:21:12.125935+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.125935+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:01.062271+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":246,"num_read_kb":246,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"68'2542","reported_seq":2660,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.794837+0000","last_change":"2024-09-17T05:21:20.794837+0000","last_active":"2024-09-17T05:21:20.794837+0000","last_peered":"2024-09-17T05:21:20.794837+0000","last_clean":"2024-09-17T05:21:20.794837+0000","last_became_active":"2024-09-17T04:49:41.038498+0000","last_became_peered":"2024-09-17T04:49:41.038498+0000","last_unstale":"2024-09-17T05:21:20.794837+0000","last_undegraded":"2024-09-17T05:21:20.794837+0000","last_fullsized":"2024-09-17T05:21:20.794837+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2542","last_scrub_stamp":"2024-09-17T05:21:20.794493+0000","last_deep_scrub":"68'2542","last_deep_scrub_stamp":"2024-09-17T05:21:20.794493+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.794493+0000","objects_scrubbed":0,"log_size":2542,"log_dups_size":0,"ondisk_log_size":2542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:08:37.076236+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13327,"num_write_kb":225533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"65'3736","reported_seq":3862,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:07.761625+0000","last_change":"2024-09-17T05:21:07.761625+0000","last_active":"2024-09-17T05:21:07.761625+0000","last_peered":"2024-09-17T05:21:07.761625+0000","last_clean":"2024-09-17T05:21:07.761625+0000","last_became_active":"2024-09-17T04:49:31.902242+0000","last_became_peered":"2024-09-17T04:49:31.902242+0000","last_unstale":"2024-09-17T05:21:07.761625+0000","last_undegraded":"2024-09-17T05:21:07.761625+0000","last_fullsized":"2024-09-17T05:21:07.761625+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3736","last_scrub_stamp":"2024-09-17T05:21:07.761340+0000","last_deep_scrub":"65'3736","last_deep_scrub_stamp":"2024-09-17T05:21:07.761340+0000","last_clean_scrub_stamp":"2024-09-17T05:21:07.761340+0000","objects_scrubbed":0,"log_size":3736,"log_dups_size":0,"ondisk_log_size":3736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:40.011725+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20513,"num_write_kb":287089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"68'5","reported_seq":133,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:01.981394+0000","last_change":"2024-09-17T05:21:01.981394+0000","last_active":"2024-09-17T05:21:01.981394+0000","last_peered":"2024-09-17T05:21:01.981394+0000","last_clean":"2024-09-17T05:21:01.981394+0000","last_became_active":"2024-09-17T04:49:30.879515+0000","last_became_peered":"2024-09-17T04:49:30.879515+0000","last_unstale":"2024-09-17T05:21:01.981394+0000","last_undegraded":"2024-09-17T05:21:01.981394+0000","last_fullsized":"2024-09-17T05:21:01.981394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:01.981171+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:01.981171+0000","last_clean_scrub_stamp":"2024-09-17T05:21:01.981171+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:53.619421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"68'5","reported_seq":125,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.839899+0000","last_change":"2024-09-17T05:21:16.839899+0000","last_active":"2024-09-17T05:21:16.839899+0000","last_peered":"2024-09-17T05:21:16.839899+0000","last_clean":"2024-09-17T05:21:16.839899+0000","last_became_active":"2024-09-17T04:49:40.001805+0000","last_became_peered":"2024-09-17T04:49:40.001805+0000","last_unstale":"2024-09-17T05:21:16.839899+0000","last_undegraded":"2024-09-17T05:21:16.839899+0000","last_fullsized":"2024-09-17T05:21:16.839899+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2024-09-17T05:21:16.839667+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2024-09-17T05:21:16.839667+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.839667+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:58.104341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"68'4175","reported_seq":4299,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.788607+0000","last_change":"2024-09-17T05:21:19.788607+0000","last_active":"2024-09-17T05:21:19.788607+0000","last_peered":"2024-09-17T05:21:19.788607+0000","last_clean":"2024-09-17T05:21:19.788607+0000","last_became_active":"2024-09-17T04:49:41.049856+0000","last_became_peered":"2024-09-17T04:49:41.049856+0000","last_unstale":"2024-09-17T05:21:19.788607+0000","last_undegraded":"2024-09-17T05:21:19.788607+0000","last_fullsized":"2024-09-17T05:21:19.788607+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4175","last_scrub_stamp":"2024-09-17T05:21:19.788380+0000","last_deep_scrub":"68'4175","last_deep_scrub_stamp":"2024-09-17T05:21:19.788380+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.788380+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:01.053054+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":40,"num_write":23028,"num_write_kb":362813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"68'25","reported_seq":147,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.809622+0000","last_change":"2024-09-17T05:21:19.809622+0000","last_active":"2024-09-17T05:21:19.809622+0000","last_peered":"2024-09-17T05:21:19.809622+0000","last_clean":"2024-09-17T05:21:19.809622+0000","last_became_active":"2024-09-17T04:49:40.007317+0000","last_became_peered":"2024-09-17T04:49:40.007317+0000","last_unstale":"2024-09-17T05:21:19.809622+0000","last_undegraded":"2024-09-17T05:21:19.809622+0000","last_fullsized":"2024-09-17T05:21:19.809622+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'25","last_scrub_stamp":"2024-09-17T05:21:19.809420+0000","last_deep_scrub":"68'25","last_deep_scrub_stamp":"2024-09-17T05:21:19.809420+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.809420+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:38.158408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"65'4339","reported_seq":4465,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:12.844188+0000","last_change":"2024-09-17T05:21:12.844188+0000","last_active":"2024-09-17T05:21:12.844188+0000","last_peered":"2024-09-17T05:21:12.844188+0000","last_clean":"2024-09-17T05:21:12.844188+0000","last_became_active":"2024-09-17T04:49:31.862513+0000","last_became_peered":"2024-09-17T04:49:31.862513+0000","last_unstale":"2024-09-17T05:21:12.844188+0000","last_undegraded":"2024-09-17T05:21:12.844188+0000","last_fullsized":"2024-09-17T05:21:12.844188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4339","last_scrub_stamp":"2024-09-17T05:21:12.843934+0000","last_deep_scrub":"65'4339","last_deep_scrub_stamp":"2024-09-17T05:21:12.843934+0000","last_clean_scrub_stamp":"2024-09-17T05:21:12.843934+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:00.889647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23134,"num_write_kb":338604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"65'2","reported_seq":130,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.535923+0000","last_change":"2024-09-17T05:21:10.535923+0000","last_active":"2024-09-17T05:21:10.535923+0000","last_peered":"2024-09-17T05:21:10.535923+0000","last_clean":"2024-09-17T05:21:10.535923+0000","last_became_active":"2024-09-17T04:49:30.884311+0000","last_became_peered":"2024-09-17T04:49:30.884311+0000","last_unstale":"2024-09-17T05:21:10.535923+0000","last_undegraded":"2024-09-17T05:21:10.535923+0000","last_fullsized":"2024-09-17T05:21:10.535923+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-09-17T05:21:10.535695+0000","last_deep_scrub":"65'2","last_deep_scrub_stamp":"2024-09-17T05:21:10.535695+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.535695+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:57.376617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"68'2877","reported_seq":2997,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:23.270288+0000","last_change":"2024-09-17T05:21:23.270288+0000","last_active":"2024-09-17T05:21:23.270288+0000","last_peered":"2024-09-17T05:21:23.270288+0000","last_clean":"2024-09-17T05:21:23.270288+0000","last_became_active":"2024-09-17T04:49:41.036858+0000","last_became_peered":"2024-09-17T04:49:41.036858+0000","last_unstale":"2024-09-17T05:21:23.270288+0000","last_undegraded":"2024-09-17T05:21:23.270288+0000","last_fullsized":"2024-09-17T05:21:23.270288+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2877","last_scrub_stamp":"2024-09-17T05:21:23.270069+0000","last_deep_scrub":"68'2877","last_deep_scrub_stamp":"2024-09-17T05:21:23.270069+0000","last_clean_scrub_stamp":"2024-09-17T05:21:23.270069+0000","objects_scrubbed":0,"log_size":2877,"log_dups_size":0,"ondisk_log_size":2877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:52:55.689397+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15865,"num_write_kb":272557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"68'1003","reported_seq":1173,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:29.411436+0000","last_change":"2024-09-17T05:21:29.411436+0000","last_active":"2024-09-17T05:21:29.411436+0000","last_peered":"2024-09-17T05:21:29.411436+0000","last_clean":"2024-09-17T05:21:29.411436+0000","last_became_active":"2024-09-17T04:49:40.000816+0000","last_became_peered":"2024-09-17T04:49:40.000816+0000","last_unstale":"2024-09-17T05:21:29.411436+0000","last_undegraded":"2024-09-17T05:21:29.411436+0000","last_fullsized":"2024-09-17T05:21:29.411436+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1003","last_scrub_stamp":"2024-09-17T05:21:29.411220+0000","last_deep_scrub":"68'1003","last_deep_scrub_stamp":"2024-09-17T05:21:29.411220+0000","last_clean_scrub_stamp":"2024-09-17T05:21:29.411220+0000","objects_scrubbed":3,"log_size":1003,"log_dups_size":0,"ondisk_log_size":1003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:53:18.044635+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":9969,"num_write":977,"num_write_kb":4635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"65'4230","reported_seq":4356,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.708264+0000","last_change":"2024-09-17T05:21:08.708264+0000","last_active":"2024-09-17T05:21:08.708264+0000","last_peered":"2024-09-17T05:21:08.708264+0000","last_clean":"2024-09-17T05:21:08.708264+0000","last_became_active":"2024-09-17T04:49:31.915548+0000","last_became_peered":"2024-09-17T04:49:31.915548+0000","last_unstale":"2024-09-17T05:21:08.708264+0000","last_undegraded":"2024-09-17T05:21:08.708264+0000","last_fullsized":"2024-09-17T05:21:08.708264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4230","last_scrub_stamp":"2024-09-17T05:21:08.708027+0000","last_deep_scrub":"65'4230","last_deep_scrub_stamp":"2024-09-17T05:21:08.708027+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.708027+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:45.642058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22609,"num_write_kb":319990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"65'3","reported_seq":131,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.258132+0000","last_change":"2024-09-17T05:21:05.258132+0000","last_active":"2024-09-17T05:21:05.258132+0000","last_peered":"2024-09-17T05:21:05.258132+0000","last_clean":"2024-09-17T05:21:05.258132+0000","last_became_active":"2024-09-17T04:49:30.876005+0000","last_became_peered":"2024-09-17T04:49:30.876005+0000","last_unstale":"2024-09-17T05:21:05.258132+0000","last_undegraded":"2024-09-17T05:21:05.258132+0000","last_fullsized":"2024-09-17T05:21:05.258132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3","last_scrub_stamp":"2024-09-17T05:21:05.257912+0000","last_deep_scrub":"65'3","last_deep_scrub_stamp":"2024-09-17T05:21:05.257912+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.257912+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:11:50.135225+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"68'3526","reported_seq":3638,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:27.680555+0000","last_change":"2024-09-17T05:21:27.680555+0000","last_active":"2024-09-17T05:21:27.680555+0000","last_peered":"2024-09-17T05:21:27.680555+0000","last_clean":"2024-09-17T05:21:27.680555+0000","last_became_active":"2024-09-17T04:50:12.071053+0000","last_became_peered":"2024-09-17T04:50:12.071053+0000","last_unstale":"2024-09-17T05:21:27.680555+0000","last_undegraded":"2024-09-17T05:21:27.680555+0000","last_fullsized":"2024-09-17T05:21:27.680555+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3526","last_scrub_stamp":"2024-09-17T05:21:27.680149+0000","last_deep_scrub":"68'3526","last_deep_scrub_stamp":"2024-09-17T05:21:27.680149+0000","last_clean_scrub_stamp":"2024-09-17T05:21:27.680149+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:28:15.485332+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20183,"num_write_kb":298078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"68'284","reported_seq":417,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.583278+0000","last_change":"2024-09-17T05:21:11.583278+0000","last_active":"2024-09-17T05:21:11.583278+0000","last_peered":"2024-09-17T05:21:11.583278+0000","last_clean":"2024-09-17T05:21:11.583278+0000","last_became_active":"2024-09-17T04:49:30.887312+0000","last_became_peered":"2024-09-17T04:49:30.887312+0000","last_unstale":"2024-09-17T05:21:11.583278+0000","last_undegraded":"2024-09-17T05:21:11.583278+0000","last_fullsized":"2024-09-17T05:21:11.583278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'284","last_scrub_stamp":"2024-09-17T05:21:11.582991+0000","last_deep_scrub":"68'284","last_deep_scrub_stamp":"2024-09-17T05:21:11.582991+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.582991+0000","objects_scrubbed":1,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:27:59.388174+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2099732,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":4077,"num_write":257,"num_write_kb":2207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"65'3509","reported_seq":3634,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.858178+0000","last_change":"2024-09-17T05:21:13.858178+0000","last_active":"2024-09-17T05:21:13.858178+0000","last_peered":"2024-09-17T05:21:13.858178+0000","last_clean":"2024-09-17T05:21:13.858178+0000","last_became_active":"2024-09-17T04:49:31.888042+0000","last_became_peered":"2024-09-17T04:49:31.888042+0000","last_unstale":"2024-09-17T05:21:13.858178+0000","last_undegraded":"2024-09-17T05:21:13.858178+0000","last_fullsized":"2024-09-17T05:21:13.858178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3509","last_scrub_stamp":"2024-09-17T05:21:13.857961+0000","last_deep_scrub":"65'3509","last_deep_scrub_stamp":"2024-09-17T05:21:13.857961+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.857961+0000","objects_scrubbed":0,"log_size":3509,"log_dups_size":0,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:26.384869+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18360,"num_write_kb":249727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"68'6","reported_seq":165,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.852925+0000","last_change":"2024-09-17T05:21:18.852925+0000","last_active":"2024-09-17T05:21:18.852925+0000","last_peered":"2024-09-17T05:21:18.852925+0000","last_clean":"2024-09-17T05:21:18.852925+0000","last_became_active":"2024-09-17T04:49:40.011602+0000","last_became_peered":"2024-09-17T04:49:40.011602+0000","last_unstale":"2024-09-17T05:21:18.852925+0000","last_undegraded":"2024-09-17T05:21:18.852925+0000","last_fullsized":"2024-09-17T05:21:18.852925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6","last_scrub_stamp":"2024-09-17T05:21:18.852717+0000","last_deep_scrub":"68'6","last_deep_scrub_stamp":"2024-09-17T05:21:18.852717+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.852717+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:48.639506+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"68'3113","reported_seq":3233,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.797502+0000","last_change":"2024-09-17T05:21:18.797502+0000","last_active":"2024-09-17T05:21:18.797502+0000","last_peered":"2024-09-17T05:21:18.797502+0000","last_clean":"2024-09-17T05:21:18.797502+0000","last_became_active":"2024-09-17T04:49:41.054987+0000","last_became_peered":"2024-09-17T04:49:41.054987+0000","last_unstale":"2024-09-17T05:21:18.797502+0000","last_undegraded":"2024-09-17T05:21:18.797502+0000","last_fullsized":"2024-09-17T05:21:18.797502+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3113","last_scrub_stamp":"2024-09-17T05:21:18.797283+0000","last_deep_scrub":"68'3113","last_deep_scrub_stamp":"2024-09-17T05:21:18.797283+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.797283+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:56:29.820359+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":16978,"num_write_kb":256690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"68'658","reported_seq":809,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:04.677697+0000","last_change":"2024-09-17T05:21:04.677697+0000","last_active":"2024-09-17T05:21:04.677697+0000","last_peered":"2024-09-17T05:21:04.677697+0000","last_clean":"2024-09-17T05:21:04.677697+0000","last_became_active":"2024-09-17T04:49:30.900632+0000","last_became_peered":"2024-09-17T04:49:30.900632+0000","last_unstale":"2024-09-17T05:21:04.677697+0000","last_undegraded":"2024-09-17T05:21:04.677697+0000","last_fullsized":"2024-09-17T05:21:04.677697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'658","last_scrub_stamp":"2024-09-17T05:21:04.677438+0000","last_deep_scrub":"68'658","last_deep_scrub_stamp":"2024-09-17T05:21:04.677438+0000","last_clean_scrub_stamp":"2024-09-17T05:21:04.677438+0000","objects_scrubbed":1,"log_size":658,"log_dups_size":0,"ondisk_log_size":658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:18:09.565613+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":79257,"num_write":647,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"65'3199","reported_seq":3325,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:04.031322+0000","last_change":"2024-09-17T05:21:04.031322+0000","last_active":"2024-09-17T05:21:04.031322+0000","last_peered":"2024-09-17T05:21:04.031322+0000","last_clean":"2024-09-17T05:21:04.031322+0000","last_became_active":"2024-09-17T04:49:31.890177+0000","last_became_peered":"2024-09-17T04:49:31.890177+0000","last_unstale":"2024-09-17T05:21:04.031322+0000","last_undegraded":"2024-09-17T05:21:04.031322+0000","last_fullsized":"2024-09-17T05:21:04.031322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3199","last_scrub_stamp":"2024-09-17T05:21:04.031235+0000","last_deep_scrub":"65'3199","last_deep_scrub_stamp":"2024-09-17T05:21:04.031235+0000","last_clean_scrub_stamp":"2024-09-17T05:21:04.031235+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:56:53.162460+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16756,"num_write_kb":239128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"68'44","reported_seq":203,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.205031+0000","last_change":"2024-09-17T05:21:14.205031+0000","last_active":"2024-09-17T05:21:14.205031+0000","last_peered":"2024-09-17T05:21:14.205031+0000","last_clean":"2024-09-17T05:21:14.205031+0000","last_became_active":"2024-09-17T04:49:40.017706+0000","last_became_peered":"2024-09-17T04:49:40.017706+0000","last_unstale":"2024-09-17T05:21:14.205031+0000","last_undegraded":"2024-09-17T05:21:14.205031+0000","last_fullsized":"2024-09-17T05:21:14.205031+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'44","last_scrub_stamp":"2024-09-17T05:21:14.204793+0000","last_deep_scrub":"68'44","last_deep_scrub_stamp":"2024-09-17T05:21:14.204793+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.204793+0000","objects_scrubbed":1,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:50.269927+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"68'3014","reported_seq":3133,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:25.883433+0000","last_change":"2024-09-17T05:21:25.883433+0000","last_active":"2024-09-17T05:21:25.883433+0000","last_peered":"2024-09-17T05:21:25.883433+0000","last_clean":"2024-09-17T05:21:25.883433+0000","last_became_active":"2024-09-17T04:49:41.051013+0000","last_became_peered":"2024-09-17T04:49:41.051013+0000","last_unstale":"2024-09-17T05:21:25.883433+0000","last_undegraded":"2024-09-17T05:21:25.883433+0000","last_fullsized":"2024-09-17T05:21:25.883433+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3014","last_scrub_stamp":"2024-09-17T05:21:25.883222+0000","last_deep_scrub":"68'3014","last_deep_scrub_stamp":"2024-09-17T05:21:25.883222+0000","last_clean_scrub_stamp":"2024-09-17T05:21:25.883222+0000","objects_scrubbed":0,"log_size":3014,"log_dups_size":0,"ondisk_log_size":3014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:36:10.678537+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16762,"num_write_kb":251577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"65'3751","reported_seq":3879,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.719703+0000","last_change":"2024-09-17T05:21:06.719703+0000","last_active":"2024-09-17T05:21:06.719703+0000","last_peered":"2024-09-17T05:21:06.719703+0000","last_clean":"2024-09-17T05:21:06.719703+0000","last_became_active":"2024-09-17T04:49:31.909697+0000","last_became_peered":"2024-09-17T04:49:31.909697+0000","last_unstale":"2024-09-17T05:21:06.719703+0000","last_undegraded":"2024-09-17T05:21:06.719703+0000","last_fullsized":"2024-09-17T05:21:06.719703+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3751","last_scrub_stamp":"2024-09-17T05:21:06.719435+0000","last_deep_scrub":"65'3751","last_deep_scrub_stamp":"2024-09-17T05:21:06.719435+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.719435+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:50.033505+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20040,"num_write_kb":277001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"68'3862","reported_seq":3982,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:26.921122+0000","last_change":"2024-09-17T05:21:26.921122+0000","last_active":"2024-09-17T05:21:26.921122+0000","last_peered":"2024-09-17T05:21:26.921122+0000","last_clean":"2024-09-17T05:21:26.921122+0000","last_became_active":"2024-09-17T04:49:41.042092+0000","last_became_peered":"2024-09-17T04:49:41.042092+0000","last_unstale":"2024-09-17T05:21:26.921122+0000","last_undegraded":"2024-09-17T05:21:26.921122+0000","last_fullsized":"2024-09-17T05:21:26.921122+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3862","last_scrub_stamp":"2024-09-17T05:21:26.920801+0000","last_deep_scrub":"68'3862","last_deep_scrub_stamp":"2024-09-17T05:21:26.920801+0000","last_clean_scrub_stamp":"2024-09-17T05:21:26.920801+0000","objects_scrubbed":0,"log_size":3862,"log_dups_size":0,"ondisk_log_size":3862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:48.252936+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20235,"num_write_kb":310497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"65'4193","reported_seq":4322,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.170827+0000","last_change":"2024-09-17T05:21:08.170827+0000","last_active":"2024-09-17T05:21:08.170827+0000","last_peered":"2024-09-17T05:21:08.170827+0000","last_clean":"2024-09-17T05:21:08.170827+0000","last_became_active":"2024-09-17T04:49:31.885224+0000","last_became_peered":"2024-09-17T04:49:31.885224+0000","last_unstale":"2024-09-17T05:21:08.170827+0000","last_undegraded":"2024-09-17T05:21:08.170827+0000","last_fullsized":"2024-09-17T05:21:08.170827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4193","last_scrub_stamp":"2024-09-17T05:21:08.170536+0000","last_deep_scrub":"65'4193","last_deep_scrub_stamp":"2024-09-17T05:21:08.170536+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.170536+0000","objects_scrubbed":0,"log_size":4193,"log_dups_size":0,"ondisk_log_size":4193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:28.895013+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22222,"num_write_kb":306243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"68'3519","reported_seq":3640,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:22.895975+0000","last_change":"2024-09-17T05:21:22.895975+0000","last_active":"2024-09-17T05:21:22.895975+0000","last_peered":"2024-09-17T05:21:22.895975+0000","last_clean":"2024-09-17T05:21:22.895975+0000","last_became_active":"2024-09-17T04:49:41.042971+0000","last_became_peered":"2024-09-17T04:49:41.042971+0000","last_unstale":"2024-09-17T05:21:22.895975+0000","last_undegraded":"2024-09-17T05:21:22.895975+0000","last_fullsized":"2024-09-17T05:21:22.895975+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3519","last_scrub_stamp":"2024-09-17T05:21:22.895639+0000","last_deep_scrub":"68'3519","last_deep_scrub_stamp":"2024-09-17T05:21:22.895639+0000","last_clean_scrub_stamp":"2024-09-17T05:21:22.895639+0000","objects_scrubbed":0,"log_size":3519,"log_dups_size":0,"ondisk_log_size":3519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:18:36.528312+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":19548,"num_write_kb":295624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"65'3872","reported_seq":4002,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.520552+0000","last_change":"2024-09-17T05:21:20.520552+0000","last_active":"2024-09-17T05:21:20.520552+0000","last_peered":"2024-09-17T05:21:20.520552+0000","last_clean":"2024-09-17T05:21:20.520552+0000","last_became_active":"2024-09-17T04:49:31.920734+0000","last_became_peered":"2024-09-17T04:49:31.920734+0000","last_unstale":"2024-09-17T05:21:20.520552+0000","last_undegraded":"2024-09-17T05:21:20.520552+0000","last_fullsized":"2024-09-17T05:21:20.520552+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3872","last_scrub_stamp":"2024-09-17T05:21:20.520322+0000","last_deep_scrub":"65'3872","last_deep_scrub_stamp":"2024-09-17T05:21:20.520322+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.520322+0000","objects_scrubbed":0,"log_size":3872,"log_dups_size":0,"ondisk_log_size":3872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:55:14.848751+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20576,"num_write_kb":279241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"68'2938","reported_seq":3058,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:28.701114+0000","last_change":"2024-09-17T05:21:28.701114+0000","last_active":"2024-09-17T05:21:28.701114+0000","last_peered":"2024-09-17T05:21:28.701114+0000","last_clean":"2024-09-17T05:21:28.701114+0000","last_became_active":"2024-09-17T04:49:41.028878+0000","last_became_peered":"2024-09-17T04:49:41.028878+0000","last_unstale":"2024-09-17T05:21:28.701114+0000","last_undegraded":"2024-09-17T05:21:28.701114+0000","last_fullsized":"2024-09-17T05:21:28.701114+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2938","last_scrub_stamp":"2024-09-17T05:21:28.700886+0000","last_deep_scrub":"68'2938","last_deep_scrub_stamp":"2024-09-17T05:21:28.700886+0000","last_clean_scrub_stamp":"2024-09-17T05:21:28.700886+0000","objects_scrubbed":0,"log_size":2938,"log_dups_size":0,"ondisk_log_size":2938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:07:18.257068+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":16647,"num_write_kb":271506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"65'3500","reported_seq":3629,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.645722+0000","last_change":"2024-09-17T05:21:10.645722+0000","last_active":"2024-09-17T05:21:10.645722+0000","last_peered":"2024-09-17T05:21:10.645722+0000","last_clean":"2024-09-17T05:21:10.645722+0000","last_became_active":"2024-09-17T04:49:31.886586+0000","last_became_peered":"2024-09-17T04:49:31.886586+0000","last_unstale":"2024-09-17T05:21:10.645722+0000","last_undegraded":"2024-09-17T05:21:10.645722+0000","last_fullsized":"2024-09-17T05:21:10.645722+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3500","last_scrub_stamp":"2024-09-17T05:21:10.645368+0000","last_deep_scrub":"65'3500","last_deep_scrub_stamp":"2024-09-17T05:21:10.645368+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.645368+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:46:40.477420+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18562,"num_write_kb":258704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"68'3339","reported_seq":3461,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:25.697706+0000","last_change":"2024-09-17T05:21:25.697706+0000","last_active":"2024-09-17T05:21:25.697706+0000","last_peered":"2024-09-17T05:21:25.697706+0000","last_clean":"2024-09-17T05:21:25.697706+0000","last_became_active":"2024-09-17T04:49:41.038380+0000","last_became_peered":"2024-09-17T04:49:41.038380+0000","last_unstale":"2024-09-17T05:21:25.697706+0000","last_undegraded":"2024-09-17T05:21:25.697706+0000","last_fullsized":"2024-09-17T05:21:25.697706+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3339","last_scrub_stamp":"2024-09-17T05:21:25.697495+0000","last_deep_scrub":"68'3339","last_deep_scrub_stamp":"2024-09-17T05:21:25.697495+0000","last_clean_scrub_stamp":"2024-09-17T05:21:25.697495+0000","objects_scrubbed":0,"log_size":3339,"log_dups_size":0,"ondisk_log_size":3339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:00.004422+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17940,"num_write_kb":305481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"65'3819","reported_seq":3947,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.210452+0000","last_change":"2024-09-17T05:21:16.210452+0000","last_active":"2024-09-17T05:21:16.210452+0000","last_peered":"2024-09-17T05:21:16.210452+0000","last_clean":"2024-09-17T05:21:16.210452+0000","last_became_active":"2024-09-17T04:49:31.888607+0000","last_became_peered":"2024-09-17T04:49:31.888607+0000","last_unstale":"2024-09-17T05:21:16.210452+0000","last_undegraded":"2024-09-17T05:21:16.210452+0000","last_fullsized":"2024-09-17T05:21:16.210452+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3819","last_scrub_stamp":"2024-09-17T05:21:16.210076+0000","last_deep_scrub":"65'3819","last_deep_scrub_stamp":"2024-09-17T05:21:16.210076+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.210076+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:05.948248+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21007,"num_write_kb":273012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"68'3374","reported_seq":3501,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.755517+0000","last_change":"2024-09-17T05:21:19.755517+0000","last_active":"2024-09-17T05:21:19.755517+0000","last_peered":"2024-09-17T05:21:19.755517+0000","last_clean":"2024-09-17T05:21:19.755517+0000","last_became_active":"2024-09-17T04:50:12.075350+0000","last_became_peered":"2024-09-17T04:50:12.075350+0000","last_unstale":"2024-09-17T05:21:19.755517+0000","last_undegraded":"2024-09-17T05:21:19.755517+0000","last_fullsized":"2024-09-17T05:21:19.755517+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3374","last_scrub_stamp":"2024-09-17T05:21:19.755308+0000","last_deep_scrub":"68'3374","last_deep_scrub_stamp":"2024-09-17T05:21:19.755308+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.755308+0000","objects_scrubbed":0,"log_size":3374,"log_dups_size":0,"ondisk_log_size":3374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:59:24.554247+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":18297,"num_write_kb":270032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"65'4233","reported_seq":4360,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.199080+0000","last_change":"2024-09-17T05:21:10.199080+0000","last_active":"2024-09-17T05:21:10.199080+0000","last_peered":"2024-09-17T05:21:10.199080+0000","last_clean":"2024-09-17T05:21:10.199080+0000","last_became_active":"2024-09-17T04:49:31.893467+0000","last_became_peered":"2024-09-17T04:49:31.893467+0000","last_unstale":"2024-09-17T05:21:10.199080+0000","last_undegraded":"2024-09-17T05:21:10.199080+0000","last_fullsized":"2024-09-17T05:21:10.199080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4233","last_scrub_stamp":"2024-09-17T05:21:10.198808+0000","last_deep_scrub":"65'4233","last_deep_scrub_stamp":"2024-09-17T05:21:10.198808+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.198808+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:29:50.002124+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23577,"num_write_kb":311694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"68'3382","reported_seq":3502,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:36.399176+0000","last_change":"2024-09-17T05:21:36.399176+0000","last_active":"2024-09-17T05:21:36.399176+0000","last_peered":"2024-09-17T05:21:36.399176+0000","last_clean":"2024-09-17T05:21:36.399176+0000","last_became_active":"2024-09-17T04:49:41.044500+0000","last_became_peered":"2024-09-17T04:49:41.044500+0000","last_unstale":"2024-09-17T05:21:36.399176+0000","last_undegraded":"2024-09-17T05:21:36.399176+0000","last_fullsized":"2024-09-17T05:21:36.399176+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3382","last_scrub_stamp":"2024-09-17T05:21:36.398942+0000","last_deep_scrub":"68'3382","last_deep_scrub_stamp":"2024-09-17T05:21:36.398942+0000","last_clean_scrub_stamp":"2024-09-17T05:21:36.398942+0000","objects_scrubbed":0,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:29:00.645629+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18178,"num_write_kb":293977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"65'3778","reported_seq":3905,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.796418+0000","last_change":"2024-09-17T05:21:08.796418+0000","last_active":"2024-09-17T05:21:08.796418+0000","last_peered":"2024-09-17T05:21:08.796418+0000","last_clean":"2024-09-17T05:21:08.796418+0000","last_became_active":"2024-09-17T04:49:31.909971+0000","last_became_peered":"2024-09-17T04:49:31.909971+0000","last_unstale":"2024-09-17T05:21:08.796418+0000","last_undegraded":"2024-09-17T05:21:08.796418+0000","last_fullsized":"2024-09-17T05:21:08.796418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3778","last_scrub_stamp":"2024-09-17T05:21:08.796155+0000","last_deep_scrub":"65'3778","last_deep_scrub_stamp":"2024-09-17T05:21:08.796155+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.796155+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:53.442168+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21044,"num_write_kb":281017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"68'2956","reported_seq":3079,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:24.923795+0000","last_change":"2024-09-17T05:21:24.923795+0000","last_active":"2024-09-17T05:21:24.923795+0000","last_peered":"2024-09-17T05:21:24.923795+0000","last_clean":"2024-09-17T05:21:24.923795+0000","last_became_active":"2024-09-17T04:49:41.044009+0000","last_became_peered":"2024-09-17T04:49:41.044009+0000","last_unstale":"2024-09-17T05:21:24.923795+0000","last_undegraded":"2024-09-17T05:21:24.923795+0000","last_fullsized":"2024-09-17T05:21:24.923795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2956","last_scrub_stamp":"2024-09-17T05:21:24.923549+0000","last_deep_scrub":"68'2956","last_deep_scrub_stamp":"2024-09-17T05:21:24.923549+0000","last_clean_scrub_stamp":"2024-09-17T05:21:24.923549+0000","objects_scrubbed":0,"log_size":2956,"log_dups_size":0,"ondisk_log_size":2956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:28:12.728731+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":15875,"num_write_kb":264255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"65'4647","reported_seq":4775,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.687332+0000","last_change":"2024-09-17T05:21:14.687332+0000","last_active":"2024-09-17T05:21:14.687332+0000","last_peered":"2024-09-17T05:21:14.687332+0000","last_clean":"2024-09-17T05:21:14.687332+0000","last_became_active":"2024-09-17T04:49:31.885888+0000","last_became_peered":"2024-09-17T04:49:31.885888+0000","last_unstale":"2024-09-17T05:21:14.687332+0000","last_undegraded":"2024-09-17T05:21:14.687332+0000","last_fullsized":"2024-09-17T05:21:14.687332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4647","last_scrub_stamp":"2024-09-17T05:21:14.687109+0000","last_deep_scrub":"65'4647","last_deep_scrub_stamp":"2024-09-17T05:21:14.687109+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.687109+0000","objects_scrubbed":0,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:04.899852+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24699,"num_write_kb":336693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"68'3095","reported_seq":3218,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:19.307187+0000","last_change":"2024-09-17T05:21:19.307187+0000","last_active":"2024-09-17T05:21:19.307187+0000","last_peered":"2024-09-17T05:21:19.307187+0000","last_clean":"2024-09-17T05:21:19.307187+0000","last_became_active":"2024-09-17T04:49:41.044671+0000","last_became_peered":"2024-09-17T04:49:41.044671+0000","last_unstale":"2024-09-17T05:21:19.307187+0000","last_undegraded":"2024-09-17T05:21:19.307187+0000","last_fullsized":"2024-09-17T05:21:19.307187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3095","last_scrub_stamp":"2024-09-17T05:21:19.306844+0000","last_deep_scrub":"68'3095","last_deep_scrub_stamp":"2024-09-17T05:21:19.306844+0000","last_clean_scrub_stamp":"2024-09-17T05:21:19.306844+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:32.552770+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":36,"num_write":16750,"num_write_kb":272041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"65'4070","reported_seq":4198,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.546705+0000","last_change":"2024-09-17T05:21:18.546705+0000","last_active":"2024-09-17T05:21:18.546705+0000","last_peered":"2024-09-17T05:21:18.546705+0000","last_clean":"2024-09-17T05:21:18.546705+0000","last_became_active":"2024-09-17T04:49:31.902403+0000","last_became_peered":"2024-09-17T04:49:31.902403+0000","last_unstale":"2024-09-17T05:21:18.546705+0000","last_undegraded":"2024-09-17T05:21:18.546705+0000","last_fullsized":"2024-09-17T05:21:18.546705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4070","last_scrub_stamp":"2024-09-17T05:21:18.546455+0000","last_deep_scrub":"65'4070","last_deep_scrub_stamp":"2024-09-17T05:21:18.546455+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.546455+0000","objects_scrubbed":0,"log_size":4070,"log_dups_size":0,"ondisk_log_size":4070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:10.791521+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21906,"num_write_kb":288644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"68'4276","reported_seq":4395,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:37.394558+0000","last_change":"2024-09-17T05:21:37.394558+0000","last_active":"2024-09-17T05:21:37.394558+0000","last_peered":"2024-09-17T05:21:37.394558+0000","last_clean":"2024-09-17T05:21:37.394558+0000","last_became_active":"2024-09-17T04:49:41.051706+0000","last_became_peered":"2024-09-17T04:49:41.051706+0000","last_unstale":"2024-09-17T05:21:37.394558+0000","last_undegraded":"2024-09-17T05:21:37.394558+0000","last_fullsized":"2024-09-17T05:21:37.394558+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4276","last_scrub_stamp":"2024-09-17T05:21:37.394316+0000","last_deep_scrub":"68'4276","last_deep_scrub_stamp":"2024-09-17T05:21:37.394316+0000","last_clean_scrub_stamp":"2024-09-17T05:21:37.394316+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:19:07.963158+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22732,"num_write_kb":365385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"65'4174","reported_seq":4301,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.688484+0000","last_change":"2024-09-17T05:21:17.688484+0000","last_active":"2024-09-17T05:21:17.688484+0000","last_peered":"2024-09-17T05:21:17.688484+0000","last_clean":"2024-09-17T05:21:17.688484+0000","last_became_active":"2024-09-17T04:49:31.902013+0000","last_became_peered":"2024-09-17T04:49:31.902013+0000","last_unstale":"2024-09-17T05:21:17.688484+0000","last_undegraded":"2024-09-17T05:21:17.688484+0000","last_fullsized":"2024-09-17T05:21:17.688484+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4174","last_scrub_stamp":"2024-09-17T05:21:17.688135+0000","last_deep_scrub":"65'4174","last_deep_scrub_stamp":"2024-09-17T05:21:17.688135+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.688135+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:46.578417+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22870,"num_write_kb":296621,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"68'2821","reported_seq":2940,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:24.299780+0000","last_change":"2024-09-17T05:21:24.299780+0000","last_active":"2024-09-17T05:21:24.299780+0000","last_peered":"2024-09-17T05:21:24.299780+0000","last_clean":"2024-09-17T05:21:24.299780+0000","last_became_active":"2024-09-17T04:49:41.027502+0000","last_became_peered":"2024-09-17T04:49:41.027502+0000","last_unstale":"2024-09-17T05:21:24.299780+0000","last_undegraded":"2024-09-17T05:21:24.299780+0000","last_fullsized":"2024-09-17T05:21:24.299780+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2821","last_scrub_stamp":"2024-09-17T05:21:24.299563+0000","last_deep_scrub":"68'2821","last_deep_scrub_stamp":"2024-09-17T05:21:24.299563+0000","last_clean_scrub_stamp":"2024-09-17T05:21:24.299563+0000","objects_scrubbed":0,"log_size":2821,"log_dups_size":0,"ondisk_log_size":2821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:12.384446+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15110,"num_write_kb":232606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"65'4280","reported_seq":4407,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:08.072977+0000","last_change":"2024-09-17T05:21:08.072977+0000","last_active":"2024-09-17T05:21:08.072977+0000","last_peered":"2024-09-17T05:21:08.072977+0000","last_clean":"2024-09-17T05:21:08.072977+0000","last_became_active":"2024-09-17T04:49:31.892310+0000","last_became_peered":"2024-09-17T04:49:31.892310+0000","last_unstale":"2024-09-17T05:21:08.072977+0000","last_undegraded":"2024-09-17T05:21:08.072977+0000","last_fullsized":"2024-09-17T05:21:08.072977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4280","last_scrub_stamp":"2024-09-17T05:21:08.072718+0000","last_deep_scrub":"65'4280","last_deep_scrub_stamp":"2024-09-17T05:21:08.072718+0000","last_clean_scrub_stamp":"2024-09-17T05:21:08.072718+0000","objects_scrubbed":0,"log_size":4280,"log_dups_size":0,"ondisk_log_size":4280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:03:13.177867+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23007,"num_write_kb":313406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"68'3452","reported_seq":3571,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.277348+0000","last_change":"2024-09-17T05:21:18.277177+0000","last_active":"2024-09-17T05:21:18.277348+0000","last_peered":"2024-09-17T05:21:18.277348+0000","last_clean":"2024-09-17T05:21:18.277348+0000","last_became_active":"2024-09-17T04:49:41.037381+0000","last_became_peered":"2024-09-17T04:49:41.037381+0000","last_unstale":"2024-09-17T05:21:18.277348+0000","last_undegraded":"2024-09-17T05:21:18.277348+0000","last_fullsized":"2024-09-17T05:21:18.277348+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3452","last_scrub_stamp":"2024-09-17T05:21:18.276964+0000","last_deep_scrub":"68'3452","last_deep_scrub_stamp":"2024-09-17T05:21:18.276964+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.276964+0000","objects_scrubbed":0,"log_size":3452,"log_dups_size":0,"ondisk_log_size":3452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:18:56.568305+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19160,"num_write_kb":306156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"65'4384","reported_seq":4513,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.854162+0000","last_change":"2024-09-17T05:21:14.854162+0000","last_active":"2024-09-17T05:21:14.854162+0000","last_peered":"2024-09-17T05:21:14.854162+0000","last_clean":"2024-09-17T05:21:14.854162+0000","last_became_active":"2024-09-17T04:49:31.897168+0000","last_became_peered":"2024-09-17T04:49:31.897168+0000","last_unstale":"2024-09-17T05:21:14.854162+0000","last_undegraded":"2024-09-17T05:21:14.854162+0000","last_fullsized":"2024-09-17T05:21:14.854162+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4384","last_scrub_stamp":"2024-09-17T05:21:14.853940+0000","last_deep_scrub":"65'4384","last_deep_scrub_stamp":"2024-09-17T05:21:14.853940+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.853940+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:48.942985+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23582,"num_write_kb":336581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"68'3472","reported_seq":3598,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:21.847984+0000","last_change":"2024-09-17T05:21:21.847984+0000","last_active":"2024-09-17T05:21:21.847984+0000","last_peered":"2024-09-17T05:21:21.847984+0000","last_clean":"2024-09-17T05:21:21.847984+0000","last_became_active":"2024-09-17T04:50:12.067942+0000","last_became_peered":"2024-09-17T04:50:12.067942+0000","last_unstale":"2024-09-17T05:21:21.847984+0000","last_undegraded":"2024-09-17T05:21:21.847984+0000","last_fullsized":"2024-09-17T05:21:21.847984+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3472","last_scrub_stamp":"2024-09-17T05:21:21.847727+0000","last_deep_scrub":"68'3472","last_deep_scrub_stamp":"2024-09-17T05:21:21.847727+0000","last_clean_scrub_stamp":"2024-09-17T05:21:21.847727+0000","objects_scrubbed":0,"log_size":3472,"log_dups_size":0,"ondisk_log_size":3472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:51.634517+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18777,"num_write_kb":292458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"65'3688","reported_seq":3815,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:10.757877+0000","last_change":"2024-09-17T05:21:10.757877+0000","last_active":"2024-09-17T05:21:10.757877+0000","last_peered":"2024-09-17T05:21:10.757877+0000","last_clean":"2024-09-17T05:21:10.757877+0000","last_became_active":"2024-09-17T04:49:31.905817+0000","last_became_peered":"2024-09-17T04:49:31.905817+0000","last_unstale":"2024-09-17T05:21:10.757877+0000","last_undegraded":"2024-09-17T05:21:10.757877+0000","last_fullsized":"2024-09-17T05:21:10.757877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3688","last_scrub_stamp":"2024-09-17T05:21:10.757653+0000","last_deep_scrub":"65'3688","last_deep_scrub_stamp":"2024-09-17T05:21:10.757653+0000","last_clean_scrub_stamp":"2024-09-17T05:21:10.757653+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:54.603536+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20358,"num_write_kb":257774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"68'3214","reported_seq":3338,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:22.708445+0000","last_change":"2024-09-17T05:21:22.708445+0000","last_active":"2024-09-17T05:21:22.708445+0000","last_peered":"2024-09-17T05:21:22.708445+0000","last_clean":"2024-09-17T05:21:22.708445+0000","last_became_active":"2024-09-17T04:49:41.037495+0000","last_became_peered":"2024-09-17T04:49:41.037495+0000","last_unstale":"2024-09-17T05:21:22.708445+0000","last_undegraded":"2024-09-17T05:21:22.708445+0000","last_fullsized":"2024-09-17T05:21:22.708445+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3214","last_scrub_stamp":"2024-09-17T05:21:22.708216+0000","last_deep_scrub":"68'3214","last_deep_scrub_stamp":"2024-09-17T05:21:22.708216+0000","last_clean_scrub_stamp":"2024-09-17T05:21:22.708216+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:08:38.989960+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":17543,"num_write_kb":304029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"65'4068","reported_seq":4195,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:21.477023+0000","last_change":"2024-09-17T05:21:21.477023+0000","last_active":"2024-09-17T05:21:21.477023+0000","last_peered":"2024-09-17T05:21:21.477023+0000","last_clean":"2024-09-17T05:21:21.477023+0000","last_became_active":"2024-09-17T04:49:31.901987+0000","last_became_peered":"2024-09-17T04:49:31.901987+0000","last_unstale":"2024-09-17T05:21:21.477023+0000","last_undegraded":"2024-09-17T05:21:21.477023+0000","last_fullsized":"2024-09-17T05:21:21.477023+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4068","last_scrub_stamp":"2024-09-17T05:21:21.476668+0000","last_deep_scrub":"65'4068","last_deep_scrub_stamp":"2024-09-17T05:21:21.476668+0000","last_clean_scrub_stamp":"2024-09-17T05:21:21.476668+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:53:48.845542+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21759,"num_write_kb":308395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"68'2551","reported_seq":2672,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:25.911203+0000","last_change":"2024-09-17T05:21:25.911203+0000","last_active":"2024-09-17T05:21:25.911203+0000","last_peered":"2024-09-17T05:21:25.911203+0000","last_clean":"2024-09-17T05:21:25.911203+0000","last_became_active":"2024-09-17T04:49:41.066664+0000","last_became_peered":"2024-09-17T04:49:41.066664+0000","last_unstale":"2024-09-17T05:21:25.911203+0000","last_undegraded":"2024-09-17T05:21:25.911203+0000","last_fullsized":"2024-09-17T05:21:25.911203+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2551","last_scrub_stamp":"2024-09-17T05:21:25.910863+0000","last_deep_scrub":"68'2551","last_deep_scrub_stamp":"2024-09-17T05:21:25.910863+0000","last_clean_scrub_stamp":"2024-09-17T05:21:25.910863+0000","objects_scrubbed":0,"log_size":2551,"log_dups_size":0,"ondisk_log_size":2551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:07:15.467044+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":13664,"num_write_kb":247232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"65'4369","reported_seq":4496,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.171692+0000","last_change":"2024-09-17T05:21:11.171692+0000","last_active":"2024-09-17T05:21:11.171692+0000","last_peered":"2024-09-17T05:21:11.171692+0000","last_clean":"2024-09-17T05:21:11.171692+0000","last_became_active":"2024-09-17T04:49:31.904764+0000","last_became_peered":"2024-09-17T04:49:31.904764+0000","last_unstale":"2024-09-17T05:21:11.171692+0000","last_undegraded":"2024-09-17T05:21:11.171692+0000","last_fullsized":"2024-09-17T05:21:11.171692+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4369","last_scrub_stamp":"2024-09-17T05:21:11.171467+0000","last_deep_scrub":"65'4369","last_deep_scrub_stamp":"2024-09-17T05:21:11.171467+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.171467+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:16:17.551506+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23171,"num_write_kb":318257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"68'3277","reported_seq":3397,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:25.338551+0000","last_change":"2024-09-17T05:21:25.338551+0000","last_active":"2024-09-17T05:21:25.338551+0000","last_peered":"2024-09-17T05:21:25.338551+0000","last_clean":"2024-09-17T05:21:25.338551+0000","last_became_active":"2024-09-17T04:49:41.026100+0000","last_became_peered":"2024-09-17T04:49:41.026100+0000","last_unstale":"2024-09-17T05:21:25.338551+0000","last_undegraded":"2024-09-17T05:21:25.338551+0000","last_fullsized":"2024-09-17T05:21:25.338551+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3277","last_scrub_stamp":"2024-09-17T05:21:25.338328+0000","last_deep_scrub":"68'3277","last_deep_scrub_stamp":"2024-09-17T05:21:25.338328+0000","last_clean_scrub_stamp":"2024-09-17T05:21:25.338328+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:11:35.392718+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18007,"num_write_kb":264733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"65'5432","reported_seq":5558,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:22.499628+0000","last_change":"2024-09-17T05:21:22.499628+0000","last_active":"2024-09-17T05:21:22.499628+0000","last_peered":"2024-09-17T05:21:22.499628+0000","last_clean":"2024-09-17T05:21:22.499628+0000","last_became_active":"2024-09-17T04:49:31.905549+0000","last_became_peered":"2024-09-17T04:49:31.905549+0000","last_unstale":"2024-09-17T05:21:22.499628+0000","last_undegraded":"2024-09-17T05:21:22.499628+0000","last_fullsized":"2024-09-17T05:21:22.499628+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5432","last_scrub_stamp":"2024-09-17T05:21:22.499375+0000","last_deep_scrub":"65'5432","last_deep_scrub_stamp":"2024-09-17T05:21:22.499375+0000","last_clean_scrub_stamp":"2024-09-17T05:21:22.499375+0000","objects_scrubbed":0,"log_size":5432,"log_dups_size":0,"ondisk_log_size":5432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:56:08.217237+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29730,"num_write_kb":389609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"68'3135","reported_seq":3255,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:28.853059+0000","last_change":"2024-09-17T05:21:28.853059+0000","last_active":"2024-09-17T05:21:28.853059+0000","last_peered":"2024-09-17T05:21:28.853059+0000","last_clean":"2024-09-17T05:21:28.853059+0000","last_became_active":"2024-09-17T04:49:41.041211+0000","last_became_peered":"2024-09-17T04:49:41.041211+0000","last_unstale":"2024-09-17T05:21:28.853059+0000","last_undegraded":"2024-09-17T05:21:28.853059+0000","last_fullsized":"2024-09-17T05:21:28.853059+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3135","last_scrub_stamp":"2024-09-17T05:21:28.852845+0000","last_deep_scrub":"68'3135","last_deep_scrub_stamp":"2024-09-17T05:21:28.852845+0000","last_clean_scrub_stamp":"2024-09-17T05:21:28.852845+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:21.097910+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":16884,"num_write_kb":284681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"65'3941","reported_seq":4067,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:23.471761+0000","last_change":"2024-09-17T05:21:23.471761+0000","last_active":"2024-09-17T05:21:23.471761+0000","last_peered":"2024-09-17T05:21:23.471761+0000","last_clean":"2024-09-17T05:21:23.471761+0000","last_became_active":"2024-09-17T04:49:31.898678+0000","last_became_peered":"2024-09-17T04:49:31.898678+0000","last_unstale":"2024-09-17T05:21:23.471761+0000","last_undegraded":"2024-09-17T05:21:23.471761+0000","last_fullsized":"2024-09-17T05:21:23.471761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3941","last_scrub_stamp":"2024-09-17T05:21:23.471413+0000","last_deep_scrub":"65'3941","last_deep_scrub_stamp":"2024-09-17T05:21:23.471413+0000","last_clean_scrub_stamp":"2024-09-17T05:21:23.471413+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:43.496202+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21845,"num_write_kb":304681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"68'3913","reported_seq":4035,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.033178+0000","last_change":"2024-09-17T05:21:13.033178+0000","last_active":"2024-09-17T05:21:13.033178+0000","last_peered":"2024-09-17T05:21:13.033178+0000","last_clean":"2024-09-17T05:21:13.033178+0000","last_became_active":"2024-09-17T04:49:41.066731+0000","last_became_peered":"2024-09-17T04:49:41.066731+0000","last_unstale":"2024-09-17T05:21:13.033178+0000","last_undegraded":"2024-09-17T05:21:13.033178+0000","last_fullsized":"2024-09-17T05:21:13.033178+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3913","last_scrub_stamp":"2024-09-17T05:21:13.032954+0000","last_deep_scrub":"68'3913","last_deep_scrub_stamp":"2024-09-17T05:21:13.032954+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.032954+0000","objects_scrubbed":0,"log_size":3913,"log_dups_size":0,"ondisk_log_size":3913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:43:50.981693+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":21530,"num_write_kb":364884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"65'3351","reported_seq":3483,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.687972+0000","last_change":"2024-09-17T05:21:15.687972+0000","last_active":"2024-09-17T05:21:15.687972+0000","last_peered":"2024-09-17T05:21:15.687972+0000","last_clean":"2024-09-17T05:21:15.687972+0000","last_became_active":"2024-09-17T04:50:12.469425+0000","last_became_peered":"2024-09-17T04:50:12.469425+0000","last_unstale":"2024-09-17T05:21:15.687972+0000","last_undegraded":"2024-09-17T05:21:15.687972+0000","last_fullsized":"2024-09-17T05:21:15.687972+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3351","last_scrub_stamp":"2024-09-17T05:21:15.687734+0000","last_deep_scrub":"65'3351","last_deep_scrub_stamp":"2024-09-17T05:21:15.687734+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.687734+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:27.678211+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18172,"num_write_kb":265354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"68'1954","reported_seq":2075,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.072384+0000","last_change":"2024-09-17T05:21:11.072384+0000","last_active":"2024-09-17T05:21:11.072384+0000","last_peered":"2024-09-17T05:21:11.072384+0000","last_clean":"2024-09-17T05:21:11.072384+0000","last_became_active":"2024-09-17T04:49:41.045797+0000","last_became_peered":"2024-09-17T04:49:41.045797+0000","last_unstale":"2024-09-17T05:21:11.072384+0000","last_undegraded":"2024-09-17T05:21:11.072384+0000","last_fullsized":"2024-09-17T05:21:11.072384+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1954","last_scrub_stamp":"2024-09-17T05:21:11.072003+0000","last_deep_scrub":"68'1954","last_deep_scrub_stamp":"2024-09-17T05:21:11.072003+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.072003+0000","objects_scrubbed":0,"log_size":1954,"log_dups_size":0,"ondisk_log_size":1954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:10.021708+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":34,"num_write":10478,"num_write_kb":181241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"65'3661","reported_seq":3787,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.237594+0000","last_change":"2024-09-17T05:21:11.237594+0000","last_active":"2024-09-17T05:21:11.237594+0000","last_peered":"2024-09-17T05:21:11.237594+0000","last_clean":"2024-09-17T05:21:11.237594+0000","last_became_active":"2024-09-17T04:49:31.890603+0000","last_became_peered":"2024-09-17T04:49:31.890603+0000","last_unstale":"2024-09-17T05:21:11.237594+0000","last_undegraded":"2024-09-17T05:21:11.237594+0000","last_fullsized":"2024-09-17T05:21:11.237594+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3661","last_scrub_stamp":"2024-09-17T05:21:11.237338+0000","last_deep_scrub":"65'3661","last_deep_scrub_stamp":"2024-09-17T05:21:11.237338+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.237338+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:42.553058+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19087,"num_write_kb":273280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"68'2981","reported_seq":3102,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:35.360470+0000","last_change":"2024-09-17T05:21:35.360470+0000","last_active":"2024-09-17T05:21:35.360470+0000","last_peered":"2024-09-17T05:21:35.360470+0000","last_clean":"2024-09-17T05:21:35.360470+0000","last_became_active":"2024-09-17T04:49:41.067057+0000","last_became_peered":"2024-09-17T04:49:41.067057+0000","last_unstale":"2024-09-17T05:21:35.360470+0000","last_undegraded":"2024-09-17T05:21:35.360470+0000","last_fullsized":"2024-09-17T05:21:35.360470+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2981","last_scrub_stamp":"2024-09-17T05:21:35.360087+0000","last_deep_scrub":"68'2981","last_deep_scrub_stamp":"2024-09-17T05:21:35.360087+0000","last_clean_scrub_stamp":"2024-09-17T05:21:35.360087+0000","objects_scrubbed":0,"log_size":2981,"log_dups_size":0,"ondisk_log_size":2981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:59.097994+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":15573,"num_write_kb":271248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"65'3297","reported_seq":3423,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.224870+0000","last_change":"2024-09-17T05:21:13.224870+0000","last_active":"2024-09-17T05:21:13.224870+0000","last_peered":"2024-09-17T05:21:13.224870+0000","last_clean":"2024-09-17T05:21:13.224870+0000","last_became_active":"2024-09-17T04:49:31.888555+0000","last_became_peered":"2024-09-17T04:49:31.888555+0000","last_unstale":"2024-09-17T05:21:13.224870+0000","last_undegraded":"2024-09-17T05:21:13.224870+0000","last_fullsized":"2024-09-17T05:21:13.224870+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3297","last_scrub_stamp":"2024-09-17T05:21:13.224541+0000","last_deep_scrub":"65'3297","last_deep_scrub_stamp":"2024-09-17T05:21:13.224541+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.224541+0000","objects_scrubbed":0,"log_size":3297,"log_dups_size":0,"ondisk_log_size":3297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:50.433999+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17851,"num_write_kb":243868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"68'2872","reported_seq":2991,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:27.893134+0000","last_change":"2024-09-17T05:21:27.893134+0000","last_active":"2024-09-17T05:21:27.893134+0000","last_peered":"2024-09-17T05:21:27.893134+0000","last_clean":"2024-09-17T05:21:27.893134+0000","last_became_active":"2024-09-17T04:49:41.032334+0000","last_became_peered":"2024-09-17T04:49:41.032334+0000","last_unstale":"2024-09-17T05:21:27.893134+0000","last_undegraded":"2024-09-17T05:21:27.893134+0000","last_fullsized":"2024-09-17T05:21:27.893134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2872","last_scrub_stamp":"2024-09-17T05:21:27.892915+0000","last_deep_scrub":"68'2872","last_deep_scrub_stamp":"2024-09-17T05:21:27.892915+0000","last_clean_scrub_stamp":"2024-09-17T05:21:27.892915+0000","objects_scrubbed":0,"log_size":2872,"log_dups_size":0,"ondisk_log_size":2872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:23.314075+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15784,"num_write_kb":259984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"65'2574","reported_seq":2700,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:11.715641+0000","last_change":"2024-09-17T05:21:11.715641+0000","last_active":"2024-09-17T05:21:11.715641+0000","last_peered":"2024-09-17T05:21:11.715641+0000","last_clean":"2024-09-17T05:21:11.715641+0000","last_became_active":"2024-09-17T04:49:31.905346+0000","last_became_peered":"2024-09-17T04:49:31.905346+0000","last_unstale":"2024-09-17T05:21:11.715641+0000","last_undegraded":"2024-09-17T05:21:11.715641+0000","last_fullsized":"2024-09-17T05:21:11.715641+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2574","last_scrub_stamp":"2024-09-17T05:21:11.714891+0000","last_deep_scrub":"65'2574","last_deep_scrub_stamp":"2024-09-17T05:21:11.714891+0000","last_clean_scrub_stamp":"2024-09-17T05:21:11.714891+0000","objects_scrubbed":0,"log_size":2574,"log_dups_size":0,"ondisk_log_size":2574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:33:27.672830+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14007,"num_write_kb":203729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"68'2459","reported_seq":2581,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:26.364277+0000","last_change":"2024-09-17T05:21:26.364277+0000","last_active":"2024-09-17T05:21:26.364277+0000","last_peered":"2024-09-17T05:21:26.364277+0000","last_clean":"2024-09-17T05:21:26.364277+0000","last_became_active":"2024-09-17T04:49:41.058618+0000","last_became_peered":"2024-09-17T04:49:41.058618+0000","last_unstale":"2024-09-17T05:21:26.364277+0000","last_undegraded":"2024-09-17T05:21:26.364277+0000","last_fullsized":"2024-09-17T05:21:26.364277+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2459","last_scrub_stamp":"2024-09-17T05:21:26.364047+0000","last_deep_scrub":"68'2459","last_deep_scrub_stamp":"2024-09-17T05:21:26.364047+0000","last_clean_scrub_stamp":"2024-09-17T05:21:26.364047+0000","objects_scrubbed":0,"log_size":2459,"log_dups_size":0,"ondisk_log_size":2459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:03:44.283882+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13572,"num_write_kb":222508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"65'3245","reported_seq":3371,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.722325+0000","last_change":"2024-09-17T05:21:15.722325+0000","last_active":"2024-09-17T05:21:15.722325+0000","last_peered":"2024-09-17T05:21:15.722325+0000","last_clean":"2024-09-17T05:21:15.722325+0000","last_became_active":"2024-09-17T04:49:31.892710+0000","last_became_peered":"2024-09-17T04:49:31.892710+0000","last_unstale":"2024-09-17T05:21:15.722325+0000","last_undegraded":"2024-09-17T05:21:15.722325+0000","last_fullsized":"2024-09-17T05:21:15.722325+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3245","last_scrub_stamp":"2024-09-17T05:21:15.722102+0000","last_deep_scrub":"65'3245","last_deep_scrub_stamp":"2024-09-17T05:21:15.722102+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.722102+0000","objects_scrubbed":0,"log_size":3245,"log_dups_size":0,"ondisk_log_size":3245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:45.255095+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16903,"num_write_kb":256080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"68'3074","reported_seq":3192,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:24.686359+0000","last_change":"2024-09-17T05:21:24.686359+0000","last_active":"2024-09-17T05:21:24.686359+0000","last_peered":"2024-09-17T05:21:24.686359+0000","last_clean":"2024-09-17T05:21:24.686359+0000","last_became_active":"2024-09-17T04:49:41.042260+0000","last_became_peered":"2024-09-17T04:49:41.042260+0000","last_unstale":"2024-09-17T05:21:24.686359+0000","last_undegraded":"2024-09-17T05:21:24.686359+0000","last_fullsized":"2024-09-17T05:21:24.686359+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3074","last_scrub_stamp":"2024-09-17T05:21:24.686143+0000","last_deep_scrub":"68'3074","last_deep_scrub_stamp":"2024-09-17T05:21:24.686143+0000","last_clean_scrub_stamp":"2024-09-17T05:21:24.686143+0000","objects_scrubbed":0,"log_size":3074,"log_dups_size":0,"ondisk_log_size":3074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:18:38.318816+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16752,"num_write_kb":271957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"65'4387","reported_seq":4513,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:14.766348+0000","last_change":"2024-09-17T05:21:14.766348+0000","last_active":"2024-09-17T05:21:14.766348+0000","last_peered":"2024-09-17T05:21:14.766348+0000","last_clean":"2024-09-17T05:21:14.766348+0000","last_became_active":"2024-09-17T04:49:31.903467+0000","last_became_peered":"2024-09-17T04:49:31.903467+0000","last_unstale":"2024-09-17T05:21:14.766348+0000","last_undegraded":"2024-09-17T05:21:14.766348+0000","last_fullsized":"2024-09-17T05:21:14.766348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4387","last_scrub_stamp":"2024-09-17T05:21:14.765991+0000","last_deep_scrub":"65'4387","last_deep_scrub_stamp":"2024-09-17T05:21:14.765991+0000","last_clean_scrub_stamp":"2024-09-17T05:21:14.765991+0000","objects_scrubbed":0,"log_size":4387,"log_dups_size":0,"ondisk_log_size":4387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:38.768117+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24001,"num_write_kb":317505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"68'2713","reported_seq":2832,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:23.663312+0000","last_change":"2024-09-17T05:21:23.663312+0000","last_active":"2024-09-17T05:21:23.663312+0000","last_peered":"2024-09-17T05:21:23.663312+0000","last_clean":"2024-09-17T05:21:23.663312+0000","last_became_active":"2024-09-17T04:49:41.051483+0000","last_became_peered":"2024-09-17T04:49:41.051483+0000","last_unstale":"2024-09-17T05:21:23.663312+0000","last_undegraded":"2024-09-17T05:21:23.663312+0000","last_fullsized":"2024-09-17T05:21:23.663312+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2713","last_scrub_stamp":"2024-09-17T05:21:23.663099+0000","last_deep_scrub":"68'2713","last_deep_scrub_stamp":"2024-09-17T05:21:23.663099+0000","last_clean_scrub_stamp":"2024-09-17T05:21:23.663099+0000","objects_scrubbed":0,"log_size":2713,"log_dups_size":0,"ondisk_log_size":2713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:36:24.590835+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14521,"num_write_kb":262890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"65'4656","reported_seq":4784,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:06.066809+0000","last_change":"2024-09-17T05:21:06.066809+0000","last_active":"2024-09-17T05:21:06.066809+0000","last_peered":"2024-09-17T05:21:06.066809+0000","last_clean":"2024-09-17T05:21:06.066809+0000","last_became_active":"2024-09-17T04:49:31.895716+0000","last_became_peered":"2024-09-17T04:49:31.895716+0000","last_unstale":"2024-09-17T05:21:06.066809+0000","last_undegraded":"2024-09-17T05:21:06.066809+0000","last_fullsized":"2024-09-17T05:21:06.066809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4656","last_scrub_stamp":"2024-09-17T05:21:06.066440+0000","last_deep_scrub":"65'4656","last_deep_scrub_stamp":"2024-09-17T05:21:06.066440+0000","last_clean_scrub_stamp":"2024-09-17T05:21:06.066440+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:20:06.012799+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25090,"num_write_kb":362900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"68'3588","reported_seq":3710,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:30.673543+0000","last_change":"2024-09-17T05:21:30.673543+0000","last_active":"2024-09-17T05:21:30.673543+0000","last_peered":"2024-09-17T05:21:30.673543+0000","last_clean":"2024-09-17T05:21:30.673543+0000","last_became_active":"2024-09-17T04:49:41.053943+0000","last_became_peered":"2024-09-17T04:49:41.053943+0000","last_unstale":"2024-09-17T05:21:30.673543+0000","last_undegraded":"2024-09-17T05:21:30.673543+0000","last_fullsized":"2024-09-17T05:21:30.673543+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3588","last_scrub_stamp":"2024-09-17T05:21:30.673273+0000","last_deep_scrub":"68'3588","last_deep_scrub_stamp":"2024-09-17T05:21:30.673273+0000","last_clean_scrub_stamp":"2024-09-17T05:21:30.673273+0000","objects_scrubbed":0,"log_size":3588,"log_dups_size":0,"ondisk_log_size":3588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:54:02.790758+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":19474,"num_write_kb":319707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"65'4149","reported_seq":4274,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:16.712006+0000","last_change":"2024-09-17T05:21:16.712006+0000","last_active":"2024-09-17T05:21:16.712006+0000","last_peered":"2024-09-17T05:21:16.712006+0000","last_clean":"2024-09-17T05:21:16.712006+0000","last_became_active":"2024-09-17T04:49:31.902712+0000","last_became_peered":"2024-09-17T04:49:31.902712+0000","last_unstale":"2024-09-17T05:21:16.712006+0000","last_undegraded":"2024-09-17T05:21:16.712006+0000","last_fullsized":"2024-09-17T05:21:16.712006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4149","last_scrub_stamp":"2024-09-17T05:21:16.711753+0000","last_deep_scrub":"65'4149","last_deep_scrub_stamp":"2024-09-17T05:21:16.711753+0000","last_clean_scrub_stamp":"2024-09-17T05:21:16.711753+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:24:32.998820+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22453,"num_write_kb":298991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"68'3299","reported_seq":3423,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:23.916305+0000","last_change":"2024-09-17T05:21:23.916305+0000","last_active":"2024-09-17T05:21:23.916305+0000","last_peered":"2024-09-17T05:21:23.916305+0000","last_clean":"2024-09-17T05:21:23.916305+0000","last_became_active":"2024-09-17T04:49:41.050252+0000","last_became_peered":"2024-09-17T04:49:41.050252+0000","last_unstale":"2024-09-17T05:21:23.916305+0000","last_undegraded":"2024-09-17T05:21:23.916305+0000","last_fullsized":"2024-09-17T05:21:23.916305+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3299","last_scrub_stamp":"2024-09-17T05:21:23.916072+0000","last_deep_scrub":"68'3299","last_deep_scrub_stamp":"2024-09-17T05:21:23.916072+0000","last_clean_scrub_stamp":"2024-09-17T05:21:23.916072+0000","objects_scrubbed":0,"log_size":3299,"log_dups_size":0,"ondisk_log_size":3299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:58.222999+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":17863,"num_write_kb":288721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"65'3910","reported_seq":4035,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:17.693225+0000","last_change":"2024-09-17T05:21:17.693225+0000","last_active":"2024-09-17T05:21:17.693225+0000","last_peered":"2024-09-17T05:21:17.693225+0000","last_clean":"2024-09-17T05:21:17.693225+0000","last_became_active":"2024-09-17T04:49:31.893557+0000","last_became_peered":"2024-09-17T04:49:31.893557+0000","last_unstale":"2024-09-17T05:21:17.693225+0000","last_undegraded":"2024-09-17T05:21:17.693225+0000","last_fullsized":"2024-09-17T05:21:17.693225+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3910","last_scrub_stamp":"2024-09-17T05:21:17.692969+0000","last_deep_scrub":"65'3910","last_deep_scrub_stamp":"2024-09-17T05:21:17.692969+0000","last_clean_scrub_stamp":"2024-09-17T05:21:17.692969+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:57:56.576535+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21371,"num_write_kb":294687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"68'2856","reported_seq":2977,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:29.822618+0000","last_change":"2024-09-17T05:21:29.822618+0000","last_active":"2024-09-17T05:21:29.822618+0000","last_peered":"2024-09-17T05:21:29.822618+0000","last_clean":"2024-09-17T05:21:29.822618+0000","last_became_active":"2024-09-17T04:49:41.051138+0000","last_became_peered":"2024-09-17T04:49:41.051138+0000","last_unstale":"2024-09-17T05:21:29.822618+0000","last_undegraded":"2024-09-17T05:21:29.822618+0000","last_fullsized":"2024-09-17T05:21:29.822618+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2856","last_scrub_stamp":"2024-09-17T05:21:29.822412+0000","last_deep_scrub":"68'2856","last_deep_scrub_stamp":"2024-09-17T05:21:29.822412+0000","last_clean_scrub_stamp":"2024-09-17T05:21:29.822412+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:02.144823+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15916,"num_write_kb":235941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"65'3148","reported_seq":3273,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:05.711504+0000","last_change":"2024-09-17T05:21:05.711504+0000","last_active":"2024-09-17T05:21:05.711504+0000","last_peered":"2024-09-17T05:21:05.711504+0000","last_clean":"2024-09-17T05:21:05.711504+0000","last_became_active":"2024-09-17T04:49:31.898796+0000","last_became_peered":"2024-09-17T04:49:31.898796+0000","last_unstale":"2024-09-17T05:21:05.711504+0000","last_undegraded":"2024-09-17T05:21:05.711504+0000","last_fullsized":"2024-09-17T05:21:05.711504+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3148","last_scrub_stamp":"2024-09-17T05:21:05.711283+0000","last_deep_scrub":"65'3148","last_deep_scrub_stamp":"2024-09-17T05:21:05.711283+0000","last_clean_scrub_stamp":"2024-09-17T05:21:05.711283+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:15.059071+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16748,"num_write_kb":239238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"68'3516","reported_seq":3638,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:15.997989+0000","last_change":"2024-09-17T05:21:15.997989+0000","last_active":"2024-09-17T05:21:15.997989+0000","last_peered":"2024-09-17T05:21:15.997989+0000","last_clean":"2024-09-17T05:21:15.997989+0000","last_became_active":"2024-09-17T04:49:41.040819+0000","last_became_peered":"2024-09-17T04:49:41.040819+0000","last_unstale":"2024-09-17T05:21:15.997989+0000","last_undegraded":"2024-09-17T05:21:15.997989+0000","last_fullsized":"2024-09-17T05:21:15.997989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3516","last_scrub_stamp":"2024-09-17T05:21:15.997616+0000","last_deep_scrub":"68'3516","last_deep_scrub_stamp":"2024-09-17T05:21:15.997616+0000","last_clean_scrub_stamp":"2024-09-17T05:21:15.997616+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:22.364455+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18785,"num_write_kb":280186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"65'3294","reported_seq":3419,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:13.733179+0000","last_change":"2024-09-17T05:21:13.733179+0000","last_active":"2024-09-17T05:21:13.733179+0000","last_peered":"2024-09-17T05:21:13.733179+0000","last_clean":"2024-09-17T05:21:13.733179+0000","last_became_active":"2024-09-17T04:49:31.889338+0000","last_became_peered":"2024-09-17T04:49:31.889338+0000","last_unstale":"2024-09-17T05:21:13.733179+0000","last_undegraded":"2024-09-17T05:21:13.733179+0000","last_fullsized":"2024-09-17T05:21:13.733179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3294","last_scrub_stamp":"2024-09-17T05:21:13.732955+0000","last_deep_scrub":"65'3294","last_deep_scrub_stamp":"2024-09-17T05:21:13.732955+0000","last_clean_scrub_stamp":"2024-09-17T05:21:13.732955+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:04:21.983038+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16934,"num_write_kb":260475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"68'2875","reported_seq":2993,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:20.324899+0000","last_change":"2024-09-17T05:21:20.324899+0000","last_active":"2024-09-17T05:21:20.324899+0000","last_peered":"2024-09-17T05:21:20.324899+0000","last_clean":"2024-09-17T05:21:20.324899+0000","last_became_active":"2024-09-17T04:49:41.060347+0000","last_became_peered":"2024-09-17T04:49:41.060347+0000","last_unstale":"2024-09-17T05:21:20.324899+0000","last_undegraded":"2024-09-17T05:21:20.324899+0000","last_fullsized":"2024-09-17T05:21:20.324899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2875","last_scrub_stamp":"2024-09-17T05:21:20.324678+0000","last_deep_scrub":"68'2875","last_deep_scrub_stamp":"2024-09-17T05:21:20.324678+0000","last_clean_scrub_stamp":"2024-09-17T05:21:20.324678+0000","objects_scrubbed":0,"log_size":2875,"log_dups_size":0,"ondisk_log_size":2875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:49:38.100241+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15268,"num_write_kb":241386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"65'3071","reported_seq":3201,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:09.768523+0000","last_change":"2024-09-17T05:21:09.768226+0000","last_active":"2024-09-17T05:21:09.768523+0000","last_peered":"2024-09-17T05:21:09.768523+0000","last_clean":"2024-09-17T05:21:09.768523+0000","last_became_active":"2024-09-17T04:49:31.898440+0000","last_became_peered":"2024-09-17T04:49:31.898440+0000","last_unstale":"2024-09-17T05:21:09.768523+0000","last_undegraded":"2024-09-17T05:21:09.768523+0000","last_fullsized":"2024-09-17T05:21:09.768523+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3071","last_scrub_stamp":"2024-09-17T05:21:09.767968+0000","last_deep_scrub":"65'3071","last_deep_scrub_stamp":"2024-09-17T05:21:09.767968+0000","last_clean_scrub_stamp":"2024-09-17T05:21:09.767968+0000","objects_scrubbed":0,"log_size":3071,"log_dups_size":0,"ondisk_log_size":3071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:53.613852+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16593,"num_write_kb":218157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"68'2617","reported_seq":2736,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:29.686960+0000","last_change":"2024-09-17T05:21:29.686960+0000","last_active":"2024-09-17T05:21:29.686960+0000","last_peered":"2024-09-17T05:21:29.686960+0000","last_clean":"2024-09-17T05:21:29.686960+0000","last_became_active":"2024-09-17T04:49:41.032962+0000","last_became_peered":"2024-09-17T04:49:41.032962+0000","last_unstale":"2024-09-17T05:21:29.686960+0000","last_undegraded":"2024-09-17T05:21:29.686960+0000","last_fullsized":"2024-09-17T05:21:29.686960+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2617","last_scrub_stamp":"2024-09-17T05:21:29.686700+0000","last_deep_scrub":"68'2617","last_deep_scrub_stamp":"2024-09-17T05:21:29.686700+0000","last_clean_scrub_stamp":"2024-09-17T05:21:29.686700+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:32.271172+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":3,"num_write":13428,"num_write_kb":250535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"65'3866","reported_seq":3995,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:26.636708+0000","last_change":"2024-09-17T05:21:26.636708+0000","last_active":"2024-09-17T05:21:26.636708+0000","last_peered":"2024-09-17T05:21:26.636708+0000","last_clean":"2024-09-17T05:21:26.636708+0000","last_became_active":"2024-09-17T04:49:31.882143+0000","last_became_peered":"2024-09-17T04:49:31.882143+0000","last_unstale":"2024-09-17T05:21:26.636708+0000","last_undegraded":"2024-09-17T05:21:26.636708+0000","last_fullsized":"2024-09-17T05:21:26.636708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3866","last_scrub_stamp":"2024-09-17T05:21:26.636331+0000","last_deep_scrub":"65'3866","last_deep_scrub_stamp":"2024-09-17T05:21:26.636331+0000","last_clean_scrub_stamp":"2024-09-17T05:21:26.636331+0000","objects_scrubbed":0,"log_size":3866,"log_dups_size":0,"ondisk_log_size":3866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:00.943258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20832,"num_write_kb":272078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"68'3056","reported_seq":3173,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:27.953540+0000","last_change":"2024-09-17T05:21:27.953540+0000","last_active":"2024-09-17T05:21:27.953540+0000","last_peered":"2024-09-17T05:21:27.953540+0000","last_clean":"2024-09-17T05:21:27.953540+0000","last_became_active":"2024-09-17T04:49:41.040326+0000","last_became_peered":"2024-09-17T04:49:41.040326+0000","last_unstale":"2024-09-17T05:21:27.953540+0000","last_undegraded":"2024-09-17T05:21:27.953540+0000","last_fullsized":"2024-09-17T05:21:27.953540+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3056","last_scrub_stamp":"2024-09-17T05:21:27.953261+0000","last_deep_scrub":"68'3056","last_deep_scrub_stamp":"2024-09-17T05:21:27.953261+0000","last_clean_scrub_stamp":"2024-09-17T05:21:27.953261+0000","objects_scrubbed":0,"log_size":3056,"log_dups_size":0,"ondisk_log_size":3056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:54:00.070747+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16712,"num_write_kb":275987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"65'4485","reported_seq":4613,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.674581+0000","last_change":"2024-09-17T05:21:18.674581+0000","last_active":"2024-09-17T05:21:18.674581+0000","last_peered":"2024-09-17T05:21:18.674581+0000","last_clean":"2024-09-17T05:21:18.674581+0000","last_became_active":"2024-09-17T04:49:31.914268+0000","last_became_peered":"2024-09-17T04:49:31.914268+0000","last_unstale":"2024-09-17T05:21:18.674581+0000","last_undegraded":"2024-09-17T05:21:18.674581+0000","last_fullsized":"2024-09-17T05:21:18.674581+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4485","last_scrub_stamp":"2024-09-17T05:21:18.674357+0000","last_deep_scrub":"65'4485","last_deep_scrub_stamp":"2024-09-17T05:21:18.674357+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.674357+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:06.720069+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23824,"num_write_kb":337279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"68'3229","reported_seq":3346,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:26.943297+0000","last_change":"2024-09-17T05:21:26.943297+0000","last_active":"2024-09-17T05:21:26.943297+0000","last_peered":"2024-09-17T05:21:26.943297+0000","last_clean":"2024-09-17T05:21:26.943297+0000","last_became_active":"2024-09-17T04:49:41.031725+0000","last_became_peered":"2024-09-17T04:49:41.031725+0000","last_unstale":"2024-09-17T05:21:26.943297+0000","last_undegraded":"2024-09-17T05:21:26.943297+0000","last_fullsized":"2024-09-17T05:21:26.943297+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3229","last_scrub_stamp":"2024-09-17T05:21:26.943079+0000","last_deep_scrub":"68'3229","last_deep_scrub_stamp":"2024-09-17T05:21:26.943079+0000","last_clean_scrub_stamp":"2024-09-17T05:21:26.943079+0000","objects_scrubbed":0,"log_size":3229,"log_dups_size":0,"ondisk_log_size":3229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:29.527551+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17855,"num_write_kb":268347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"65'3523","reported_seq":3652,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T05:21:18.736398+0000","last_change":"2024-09-17T05:21:18.736398+0000","last_active":"2024-09-17T05:21:18.736398+0000","last_peered":"2024-09-17T05:21:18.736398+0000","last_clean":"2024-09-17T05:21:18.736398+0000","last_became_active":"2024-09-17T04:49:31.895711+0000","last_became_peered":"2024-09-17T04:49:31.895711+0000","last_unstale":"2024-09-17T05:21:18.736398+0000","last_undegraded":"2024-09-17T05:21:18.736398+0000","last_fullsized":"2024-09-17T05:21:18.736398+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'3523","last_scrub_stamp":"2024-09-17T05:21:18.736157+0000","last_deep_scrub":"65'3523","last_deep_scrub_stamp":"2024-09-17T05:21:18.736157+0000","last_clean_scrub_stamp":"2024-09-17T05:21:18.736157+0000","objects_scrubbed":0,"log_size":3523,"log_dups_size":0,"ondisk_log_size":3523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:52.825202+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19890,"num_write_kb":260121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":94,"num_read_kb":321,"num_write":1077532,"num_write_kb":17428719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":198824,"ondisk_log_size":198824,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":18310618,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1391,"num_read_kb":207657,"num_write":4849,"num_write_kb":19832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36683776,"data_stored":36621236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4464,"ondisk_log_size":4464,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":1352682,"num_write_kb":18536494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250516,"ondisk_log_size":250516,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":18963064,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1535,"num_read_kb":230833,"num_write":4911,"num_write_kb":20583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37986304,"data_stored":37926128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4525,"ondisk_log_size":4525,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509832,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533064,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":532094,"kb_avail":93838776,"statfs":{"total":96636764160,"available":96090906624,"internally_reserved":0,"allocated":974848,"data_stored":552256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":544864714},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509833,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":762124,"kb_used_data":17916,"kb_used_omap":1,"kb_used_meta":744190,"kb_avail":93609716,"statfs":{"total":96636764160,"available":95856349184,"internally_reserved":0,"allocated":18345984,"data_stored":17920368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":762051015},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509832,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760196,"kb_used_data":17268,"kb_used_omap":1,"kb_used_meta":742910,"kb_avail":93611644,"statfs":{"total":96636764160,"available":95858323456,"internally_reserved":0,"allocated":17682432,"data_stored":17246718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":760740293},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509833,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755412,"kb_used_data":15300,"kb_used_omap":1,"kb_used_meta":740094,"kb_avail":93616428,"statfs":{"total":96636764160,"available":95863222272,"internally_reserved":0,"allocated":15667200,"data_stored":15236011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":757856711},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509833,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877392,"kb_used_data":14656,"kb_used_omap":1,"kb_used_meta":862718,"kb_avail":93494448,"statfs":{"total":96636764160,"available":95738314752,"internally_reserved":0,"allocated":15007744,"data_stored":14583014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":883423685},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509834,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630324,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":628862,"kb_avail":93741516,"statfs":{"total":96636764160,"available":95991312384,"internally_reserved":0,"allocated":1478656,"data_stored":1057972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":643955141},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509833,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":823992,"kb_used_data":2920,"kb_used_omap":1,"kb_used_meta":821054,"kb_avail":93547848,"statfs":{"total":96636764160,"available":95792996352,"internally_reserved":0,"allocated":2990080,"data_stored":2567342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840759749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542537,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694200,"kb_used_data":10472,"kb_used_omap":1,"kb_used_meta":683710,"kb_avail":93677640,"statfs":{"total":96636764160,"available":95925903360,"internally_reserved":0,"allocated":10723328,"data_stored":10304131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":700119494},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2099788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10493952,"data_stored":10488922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":4279042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":84770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5726270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4279625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5652480,"data_stored":5642363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T05:21:46.132 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-17T05:21:46.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.323+0000 7fdd1aa38640 1 -- 172.21.15.150:0/3543344413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140c42e0 msgr2=0x7fdd140c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:46.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.323+0000 7fdd1aa38640 1 --2- 172.21.15.150:0/3543344413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140c42e0 0x7fdd140c6690 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7fdd140ae040 tx=0x7fdd0401dbd0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.323+0000 7fdd1aa38640 1 -- 172.21.15.150:0/3543344413 shutdown_connections 2024-09-17T05:21:46.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.323+0000 7fdd1aa38640 1 --2- 172.21.15.150:0/3543344413 >> v2:172.21.15.176:3300/0 conn(0x7fdd141505a0 0x7fdd14152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.323+0000 7fdd1aa38640 1 --2- 172.21.15.150:0/3543344413 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd140a7350 0x7fdd14150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.323+0000 7fdd1aa38640 1 --2- 172.21.15.150:0/3543344413 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140c42e0 0x7fdd140c6690 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.323+0000 7fdd1aa38640 1 -- 172.21.15.150:0/3543344413 >> 172.21.15.150:0/3543344413 conn(0x7fdd140c4a00 msgr2=0x7fdd140c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:46.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.323+0000 7fdd1aa38640 1 -- 172.21.15.150:0/3543344413 shutdown_connections 2024-09-17T05:21:46.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.323+0000 7fdd1aa38640 1 -- 172.21.15.150:0/3543344413 wait complete. 2024-09-17T05:21:46.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd1aa38640 1 Processor -- start 2024-09-17T05:21:46.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd1aa38640 1 -- start start 2024-09-17T05:21:46.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd1aa38640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd140a7350 0x7fdd1414da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:46.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd1aa38640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140c42e0 0x7fdd140a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:46.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd1aa38640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd141505a0 0x7fdd140a8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:46.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd1aa38640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdd140c6bd0 con 0x7fdd140c42e0 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd1aa38640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd14070b80 con 0x7fdd140a7350 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd1aa38640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fdd140c6d50 con 0x7fdd141505a0 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd137fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140c42e0 0x7fdd140a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd137fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140c42e0 0x7fdd140a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50052/0 (socket says 172.21.15.150:50052) 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd137fe640 1 -- 172.21.15.150:0/1215518517 learned_addr learned my addr 172.21.15.150:0/1215518517 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd18fd5640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd141505a0 0x7fdd140a8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd13fff640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd140a7350 0x7fdd1414da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd137fe640 1 -- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd141505a0 msgr2=0x7fdd140a8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd137fe640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd141505a0 0x7fdd140a8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd137fe640 1 -- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd140a7350 msgr2=0x7fdd1414da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd137fe640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd140a7350 0x7fdd1414da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd137fe640 1 -- 172.21.15.150:0/1215518517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd14154420 con 0x7fdd140c42e0 2024-09-17T05:21:46.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd137fe640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140c42e0 0x7fdd140a8aa0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800c970 tx=0x7fdd0800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:46.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd10ff9640 1 -- 172.21.15.150:0/1215518517 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd08004220 con 0x7fdd140c42e0 2024-09-17T05:21:46.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd10ff9640 1 -- 172.21.15.150:0/1215518517 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd080079b0 con 0x7fdd140c42e0 2024-09-17T05:21:46.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd10ff9640 1 -- 172.21.15.150:0/1215518517 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd080044b0 con 0x7fdd140c42e0 2024-09-17T05:21:46.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd1aa38640 1 -- 172.21.15.150:0/1215518517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd140a84f0 con 0x7fdd140c42e0 2024-09-17T05:21:46.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd1aa38640 1 -- 172.21.15.150:0/1215518517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd140ada10 con 0x7fdd140c42e0 2024-09-17T05:21:46.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.327+0000 7fdd1aa38640 1 -- 172.21.15.150:0/1215518517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd140c6690 con 0x7fdd140c42e0 2024-09-17T05:21:46.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.331+0000 7fdd10ff9640 1 -- 172.21.15.150:0/1215518517 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fdd08005020 con 0x7fdd140c42e0 2024-09-17T05:21:46.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.331+0000 7fdd10ff9640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd00073200 0x7fdd00075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:46.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.331+0000 7fdd10ff9640 1 -- 172.21.15.150:0/1215518517 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7fdd080958e0 con 0x7fdd140c42e0 2024-09-17T05:21:46.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.331+0000 7fdd13fff640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd00073200 0x7fdd00075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:46.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.331+0000 7fdd13fff640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd00073200 0x7fdd00075710 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdd040b0000 tx=0x7fdd0401c410 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:46.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.335+0000 7fdd10ff9640 1 -- 172.21.15.150:0/1215518517 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdd08014030 con 0x7fdd140c42e0 2024-09-17T05:21:46.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.499+0000 7fdd1aa38640 1 -- 172.21.15.150:0/1215518517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fdd14150170 con 0x7fdd140c42e0 2024-09-17T05:21:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd10ff9640 1 -- 172.21.15.150:0/1215518517 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v36)=0 v36) ==== 123+0+0 (secure 0 0 0) 0x7fdd0805f840 con 0x7fdd140c42e0 2024-09-17T05:21:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd10ff9640 1 -- 172.21.15.150:0/1215518517 <== mon.0 v2:172.21.15.150:3300/0 8 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fdd0805ec50 con 0x7fdd140c42e0 2024-09-17T05:21:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd1aa38640 1 -- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd00073200 msgr2=0x7fdd00075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd1aa38640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd00073200 0x7fdd00075710 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdd040b0000 tx=0x7fdd0401c410 comp rx=0 tx=0).stop 2024-09-17T05:21:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd1aa38640 1 -- 172.21.15.150:0/1215518517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140c42e0 msgr2=0x7fdd140a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd1aa38640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140c42e0 0x7fdd140a8aa0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800c970 tx=0x7fdd0800ce30 comp rx=0 tx=0).stop 2024-09-17T05:21:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd1aa38640 1 -- 172.21.15.150:0/1215518517 shutdown_connections 2024-09-17T05:21:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd1aa38640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fdd00073200 0x7fdd00075710 secure :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdd040b0000 tx=0x7fdd0401c410 comp rx=0 tx=0).stop 2024-09-17T05:21:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd1aa38640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fdd141505a0 0x7fdd140a8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd1aa38640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd140c42e0 0x7fdd140a8aa0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd1aa38640 1 --2- 172.21.15.150:0/1215518517 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd140a7350 0x7fdd1414da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd1aa38640 1 -- 172.21.15.150:0/1215518517 >> 172.21.15.150:0/1215518517 conn(0x7fdd140c4a00 msgr2=0x7fdd141e4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd1aa38640 1 -- 172.21.15.150:0/1215518517 shutdown_connections 2024-09-17T05:21:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.507+0000 7fdd1aa38640 1 -- 172.21.15.150:0/1215518517 wait complete. 2024-09-17T05:21:46.528 INFO:tasks.cephfs.filesystem:Destroying file system a and related pools 2024-09-17T05:21:46.528 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:21:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.667+0000 7f9abd960640 1 -- 172.21.15.150:0/2624506019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab80c35b0 msgr2=0x7f9ab80c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:46.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.667+0000 7f9abd960640 1 --2- 172.21.15.150:0/2624506019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab80c35b0 0x7f9ab80c8a80 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80cd080 tx=0x7f9aa001c890 comp rx=0 tx=0).stop 2024-09-17T05:21:46.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.667+0000 7f9abd960640 1 -- 172.21.15.150:0/2624506019 shutdown_connections 2024-09-17T05:21:46.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.667+0000 7f9abd960640 1 --2- 172.21.15.150:0/2624506019 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ab80c8fc0 0x7f9ab80c20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.667+0000 7f9abd960640 1 --2- 172.21.15.150:0/2624506019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab80c35b0 0x7f9ab80c8a80 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.667+0000 7f9abd960640 1 --2- 172.21.15.150:0/2624506019 >> v2:172.21.15.176:3300/0 conn(0x7f9ab81568a0 0x7f9ab80c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.667+0000 7f9abd960640 1 -- 172.21.15.150:0/2624506019 >> 172.21.15.150:0/2624506019 conn(0x7f9ab8074b80 msgr2=0x7f9ab806f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:46.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.667+0000 7f9abd960640 1 -- 172.21.15.150:0/2624506019 shutdown_connections 2024-09-17T05:21:46.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.667+0000 7f9abd960640 1 -- 172.21.15.150:0/2624506019 wait complete. 2024-09-17T05:21:46.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.667+0000 7f9abd960640 1 Processor -- start 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.667+0000 7f9abd960640 1 -- start start 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9abd960640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ab80c35b0 0x7f9ab8146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9abd960640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ab80c8fc0 0x7f9ab8147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9abd960640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab81568a0 0x7f9ab814f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9abd960640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab80c6b70 con 0x7f9ab81568a0 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9abd960640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab80c69f0 con 0x7f9ab80c8fc0 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9abd960640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab80b0fd0 con 0x7f9ab80c35b0 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9ab77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab81568a0 0x7f9ab814f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9ab77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab81568a0 0x7f9ab814f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50076/0 (socket says 172.21.15.150:50076) 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9ab77fe640 1 -- 172.21.15.150:0/1718119512 learned_addr learned my addr 172.21.15.150:0/1718119512 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9ab6ffd640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ab80c35b0 0x7f9ab8146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9ab77fe640 1 -- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ab80c35b0 msgr2=0x7f9ab8146400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:46.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9ab77fe640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ab80c35b0 0x7f9ab8146400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9ab77fe640 1 -- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ab80c8fc0 msgr2=0x7f9ab8147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:46.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9ab77fe640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ab80c8fc0 0x7f9ab8147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9ab77fe640 1 -- 172.21.15.150:0/1718119512 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab8150a10 con 0x7f9ab81568a0 2024-09-17T05:21:46.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9ab77fe640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab81568a0 0x7f9ab814f720 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800d9b0 tx=0x7f9aa800de70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:46.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9aaffff640 1 -- 172.21.15.150:0/1718119512 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8016070 con 0x7f9ab81568a0 2024-09-17T05:21:46.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9aaffff640 1 -- 172.21.15.150:0/1718119512 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f9aa8004e40 con 0x7f9ab81568a0 2024-09-17T05:21:46.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9abd960640 1 -- 172.21.15.150:0/1718119512 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab814e8d0 con 0x7f9ab81568a0 2024-09-17T05:21:46.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9aaffff640 1 -- 172.21.15.150:0/1718119512 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa800bcf0 con 0x7f9ab81568a0 2024-09-17T05:21:46.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9abd960640 1 -- 172.21.15.150:0/1718119512 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab814eb70 con 0x7f9ab81568a0 2024-09-17T05:21:46.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9abd960640 1 -- 172.21.15.150:0/1718119512 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab80b1560 con 0x7f9ab81568a0 2024-09-17T05:21:46.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.671+0000 7f9aaffff640 1 -- 172.21.15.150:0/1718119512 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f9aa8003980 con 0x7f9ab81568a0 2024-09-17T05:21:46.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.675+0000 7f9aaffff640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9a98073230 0x7f9a98075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:46.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.675+0000 7f9aaffff640 1 -- 172.21.15.150:0/1718119512 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f9aa80962b0 con 0x7f9ab81568a0 2024-09-17T05:21:46.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.675+0000 7f9ab6ffd640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9a98073230 0x7f9a98075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:46.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.675+0000 7f9ab6ffd640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9a98073230 0x7f9a98075740 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9ab00058d0 tx=0x7f9ab0005860 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:46.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.679+0000 7f9aaffff640 1 -- 172.21.15.150:0/1718119512 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9aa801f020 con 0x7f9ab81568a0 2024-09-17T05:21:46.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.867+0000 7f9abd960640 1 -- 172.21.15.150:0/1718119512 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ab8157ec0 con 0x7f9ab81568a0 2024-09-17T05:21:46.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.867+0000 7f9aaffff640 1 -- 172.21.15.150:0/1718119512 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) ==== 95+0+5723 (secure 0 0 0) 0x7f9aa8003c70 con 0x7f9ab81568a0 2024-09-17T05:21:46.866 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:21:46.867 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":213,"btime":"2024-09-17T05:20:39:523989+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:20:39.523977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7567":{"gid":7567,"name":"b","rank":0,"incarnation":210,"state":"up:active","state_seq":12,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7567,"qdb_cluster":[7567]},"id":2}]} 2024-09-17T05:21:46.867 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 213 2024-09-17T05:21:46.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.871+0000 7f9abd960640 1 -- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9a98073230 msgr2=0x7f9a98075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:46.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.871+0000 7f9abd960640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9a98073230 0x7f9a98075740 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f9ab00058d0 tx=0x7f9ab0005860 comp rx=0 tx=0).stop 2024-09-17T05:21:46.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.871+0000 7f9abd960640 1 -- 172.21.15.150:0/1718119512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab81568a0 msgr2=0x7f9ab814f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:46.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.871+0000 7f9abd960640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab81568a0 0x7f9ab814f720 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800d9b0 tx=0x7f9aa800de70 comp rx=0 tx=0).stop 2024-09-17T05:21:46.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.871+0000 7f9abd960640 1 -- 172.21.15.150:0/1718119512 shutdown_connections 2024-09-17T05:21:46.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.871+0000 7f9abd960640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f9a98073230 0x7f9a98075740 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.871+0000 7f9abd960640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9ab80c8fc0 0x7f9ab8147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.871+0000 7f9abd960640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ab81568a0 0x7f9ab814f720 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.871+0000 7f9abd960640 1 --2- 172.21.15.150:0/1718119512 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f9ab80c35b0 0x7f9ab8146400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:46.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.871+0000 7f9abd960640 1 -- 172.21.15.150:0/1718119512 >> 172.21.15.150:0/1718119512 conn(0x7f9ab8074b80 msgr2=0x7f9ab8154710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:46.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.871+0000 7f9abd960640 1 -- 172.21.15.150:0/1718119512 shutdown_connections 2024-09-17T05:21:46.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:46.871+0000 7f9abd960640 1 -- 172.21.15.150:0/1718119512 wait complete. 2024-09-17T05:21:46.889 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:21:47.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.039+0000 7f46ba801640 1 -- 172.21.15.150:0/1421305658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b41505a0 msgr2=0x7f46b4152cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.039+0000 7f46ba801640 1 --2- 172.21.15.150:0/1421305658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b41505a0 0x7f46b4152cf0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f46b406ffd0 tx=0x7f46ac01caa0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.039+0000 7f46ba801640 1 -- 172.21.15.150:0/1421305658 shutdown_connections 2024-09-17T05:21:47.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.039+0000 7f46ba801640 1 --2- 172.21.15.150:0/1421305658 >> v2:172.21.15.176:3300/0 conn(0x7f46b40a7350 0x7f46b4150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.039+0000 7f46ba801640 1 --2- 172.21.15.150:0/1421305658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b41505a0 0x7f46b4152cf0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.039+0000 7f46ba801640 1 -- 172.21.15.150:0/1421305658 >> 172.21.15.150:0/1421305658 conn(0x7f46b40c4a00 msgr2=0x7f46b40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:47.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.039+0000 7f46ba801640 1 -- 172.21.15.150:0/1421305658 shutdown_connections 2024-09-17T05:21:47.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.039+0000 7f46ba801640 1 -- 172.21.15.150:0/1421305658 wait complete. 2024-09-17T05:21:47.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46ba801640 1 Processor -- start 2024-09-17T05:21:47.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46ba801640 1 -- start start 2024-09-17T05:21:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46ba801640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b40a7350 0x7f46b41564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46ba801640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46b40c42e0 0x7f46b40aca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:47.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46ba801640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46b41505a0 0x7f46b40ab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:47.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46ba801640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f46b41537c0 con 0x7f46b40a7350 2024-09-17T05:21:47.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46ba801640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f46b4153640 con 0x7f46b40c42e0 2024-09-17T05:21:47.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46ba801640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f46b4160790 con 0x7f46b41505a0 2024-09-17T05:21:47.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b40a7350 0x7f46b41564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:47.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b40a7350 0x7f46b41564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50098/0 (socket says 172.21.15.150:50098) 2024-09-17T05:21:47.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b3fff640 1 -- 172.21.15.150:0/824414117 learned_addr learned my addr 172.21.15.150:0/824414117 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:47.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b37fe640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46b40c42e0 0x7f46b40aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:47.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b8d9e640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46b41505a0 0x7f46b40ab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:47.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b3fff640 1 -- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46b41505a0 msgr2=0x7f46b40ab9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b3fff640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46b41505a0 0x7f46b40ab9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b3fff640 1 -- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46b40c42e0 msgr2=0x7f46b40aca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b3fff640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46b40c42e0 0x7f46b40aca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b3fff640 1 -- 172.21.15.150:0/824414117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b41569f0 con 0x7f46b40a7350 2024-09-17T05:21:47.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b37fe640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46b40c42e0 0x7f46b40aca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:47.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b3fff640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b40a7350 0x7f46b41564b0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f46a400c530 tx=0x7f46a400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:47.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b0ff9640 1 -- 172.21.15.150:0/824414117 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a4016020 con 0x7f46b40a7350 2024-09-17T05:21:47.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b0ff9640 1 -- 172.21.15.150:0/824414117 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f46a4012850 con 0x7f46b40a7350 2024-09-17T05:21:47.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46ba801640 1 -- 172.21.15.150:0/824414117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46b4147d50 con 0x7f46b40a7350 2024-09-17T05:21:47.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b0ff9640 1 -- 172.21.15.150:0/824414117 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a4012cc0 con 0x7f46b40a7350 2024-09-17T05:21:47.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46ba801640 1 -- 172.21.15.150:0/824414117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46b40ad8a0 con 0x7f46b40a7350 2024-09-17T05:21:47.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46ba801640 1 -- 172.21.15.150:0/824414117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46b4150060 con 0x7f46b40a7350 2024-09-17T05:21:47.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.043+0000 7f46b0ff9640 1 -- 172.21.15.150:0/824414117 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f46a4019020 con 0x7f46b40a7350 2024-09-17T05:21:47.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.047+0000 7f46b0ff9640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f46a0073200 0x7f46a0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:47.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.047+0000 7f46b0ff9640 1 -- 172.21.15.150:0/824414117 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f46a40953e0 con 0x7f46b40a7350 2024-09-17T05:21:47.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.047+0000 7f46b37fe640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f46a0073200 0x7f46a0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:47.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.047+0000 7f46b37fe640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f46a0073200 0x7f46a0075710 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f46a8005fd0 tx=0x7f46a8005de0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:47.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.051+0000 7f46b0ff9640 1 -- 172.21.15.150:0/824414117 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f46a4020020 con 0x7f46b40a7350 2024-09-17T05:21:47.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.239+0000 7f46ba801640 1 -- 172.21.15.150:0/824414117 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46b4153c50 con 0x7f46b40a7350 2024-09-17T05:21:47.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46b0ff9640 1 -- 172.21.15.150:0/824414117 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) ==== 95+0+5723 (secure 0 0 0) 0x7f46a405f350 con 0x7f46b40a7350 2024-09-17T05:21:47.241 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:21:47.241 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":213,"btime":"2024-09-17T05:20:39:523989+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:31.253633+0000","modified":"2024-09-17T05:19:54.337480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8199},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8199":{"gid":8199,"name":"c","rank":0,"incarnation":205,"state":"up:active","state_seq":21,"addr":"172.21.15.150:6835/1799698362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1799698362},{"type":"v1","addr":"172.21.15.150:6835","nonce":1799698362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8199,"qdb_cluster":[8199]},"id":1},{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:20:39.523977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7567":{"gid":7567,"name":"b","rank":0,"incarnation":210,"state":"up:active","state_seq":12,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7567,"qdb_cluster":[7567]},"id":2}]} 2024-09-17T05:21:47.241 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 213 2024-09-17T05:21:47.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46ba801640 1 -- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f46a0073200 msgr2=0x7f46a0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46ba801640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f46a0073200 0x7f46a0075710 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f46a8005fd0 tx=0x7f46a8005de0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46ba801640 1 -- 172.21.15.150:0/824414117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b40a7350 msgr2=0x7f46b41564b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46ba801640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b40a7350 0x7f46b41564b0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f46a400c530 tx=0x7f46a400c9f0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46ba801640 1 -- 172.21.15.150:0/824414117 shutdown_connections 2024-09-17T05:21:47.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46ba801640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f46a0073200 0x7f46a0075710 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46ba801640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46b40c42e0 0x7f46b40aca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46ba801640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f46b41505a0 0x7f46b40ab9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46ba801640 1 --2- 172.21.15.150:0/824414117 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46b40a7350 0x7f46b41564b0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46ba801640 1 -- 172.21.15.150:0/824414117 >> 172.21.15.150:0/824414117 conn(0x7f46b40c4a00 msgr2=0x7f46b414abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:47.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46ba801640 1 -- 172.21.15.150:0/824414117 shutdown_connections 2024-09-17T05:21:47.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.243+0000 7f46ba801640 1 -- 172.21.15.150:0/824414117 wait complete. 2024-09-17T05:21:47.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:21:47.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.419+0000 7ff016e6d640 1 -- 172.21.15.150:0/2892555653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0100c42e0 msgr2=0x7ff0100c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.419+0000 7ff016e6d640 1 --2- 172.21.15.150:0/2892555653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0100c42e0 0x7ff0100c6690 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff000009900 tx=0x7ff00001c8e0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.419+0000 7ff016e6d640 1 -- 172.21.15.150:0/2892555653 shutdown_connections 2024-09-17T05:21:47.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.419+0000 7ff016e6d640 1 --2- 172.21.15.150:0/2892555653 >> v2:172.21.15.176:3300/0 conn(0x7ff0100a7350 0x7ff010150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.419+0000 7ff016e6d640 1 --2- 172.21.15.150:0/2892555653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0100c42e0 0x7ff0100c6690 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.419+0000 7ff016e6d640 1 -- 172.21.15.150:0/2892555653 >> 172.21.15.150:0/2892555653 conn(0x7ff0100c4a00 msgr2=0x7ff0100c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:47.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.419+0000 7ff016e6d640 1 -- 172.21.15.150:0/2892555653 shutdown_connections 2024-09-17T05:21:47.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.419+0000 7ff016e6d640 1 -- 172.21.15.150:0/2892555653 wait complete. 2024-09-17T05:21:47.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff016e6d640 1 Processor -- start 2024-09-17T05:21:47.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff016e6d640 1 -- start start 2024-09-17T05:21:47.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff016e6d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0100a7350 0x7ff01014c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:47.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff016e6d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0100c42e0 0x7ff0101f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:47.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff016e6d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0101505a0 0x7ff010146cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:47.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff016e6d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff0100c6d50 con 0x7ff0101505a0 2024-09-17T05:21:47.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff016e6d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff0100c6bd0 con 0x7ff0100a7350 2024-09-17T05:21:47.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff016e6d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff01015fe90 con 0x7ff0100c42e0 2024-09-17T05:21:47.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff01540a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0101505a0 0x7ff010146cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:47.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff01540a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0101505a0 0x7ff010146cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50104/0 (socket says 172.21.15.150:50104) 2024-09-17T05:21:47.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff01540a640 1 -- 172.21.15.150:0/3952381790 learned_addr learned my addr 172.21.15.150:0/3952381790 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:47.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff00ffff640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0100c42e0 0x7ff0101f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:47.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff014c09640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0100a7350 0x7ff01014c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:47.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff01540a640 1 -- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0100c42e0 msgr2=0x7ff0101f1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff01540a640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0100c42e0 0x7ff0101f1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff01540a640 1 -- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0100a7350 msgr2=0x7ff01014c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff01540a640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0100a7350 0x7ff01014c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff01540a640 1 -- 172.21.15.150:0/3952381790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0101543c0 con 0x7ff0101505a0 2024-09-17T05:21:47.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff01540a640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0101505a0 0x7ff010146cb0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7ff00800c760 tx=0x7ff00800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:47.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff00d7fa640 1 -- 172.21.15.150:0/3952381790 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff008003b80 con 0x7ff0101505a0 2024-09-17T05:21:47.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff016e6d640 1 -- 172.21.15.150:0/3952381790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0100ab500 con 0x7ff0101505a0 2024-09-17T05:21:47.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff00d7fa640 1 -- 172.21.15.150:0/3952381790 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7ff008003d20 con 0x7ff0101505a0 2024-09-17T05:21:47.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff00d7fa640 1 -- 172.21.15.150:0/3952381790 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00800aab0 con 0x7ff0101505a0 2024-09-17T05:21:47.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff016e6d640 1 -- 172.21.15.150:0/3952381790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0101eeb10 con 0x7ff0101505a0 2024-09-17T05:21:47.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.423+0000 7ff016e6d640 1 -- 172.21.15.150:0/3952381790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff010150060 con 0x7ff0101505a0 2024-09-17T05:21:47.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.427+0000 7ff00d7fa640 1 -- 172.21.15.150:0/3952381790 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff00800ac50 con 0x7ff0101505a0 2024-09-17T05:21:47.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.427+0000 7ff00d7fa640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feff4073230 0x7feff4075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:47.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.427+0000 7ff00d7fa640 1 -- 172.21.15.150:0/3952381790 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7ff008095340 con 0x7ff0101505a0 2024-09-17T05:21:47.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.427+0000 7ff014c09640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feff4073230 0x7feff4075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:47.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.427+0000 7ff014c09640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feff4073230 0x7feff4075740 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff0000af000 tx=0x7ff000019680 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:47.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.431+0000 7ff00d7fa640 1 -- 172.21.15.150:0/3952381790 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff00805f110 con 0x7ff0101505a0 2024-09-17T05:21:47.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.591+0000 7ff016e6d640 1 -- 172.21.15.150:0/3952381790 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0100a7130 con 0x7ff0101505a0 2024-09-17T05:21:47.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.595+0000 7ff00d7fa640 1 -- 172.21.15.150:0/3952381790 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) ==== 74+0+26103 (secure 0 0 0) 0x7ff00805ea10 con 0x7ff0101505a0 2024-09-17T05:21:47.593 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:21:47.594 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":68,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T05:20:36.328166+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-09-17T04:49:29.488488+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-09-17T04:49:30.232367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.31","mappings":[{"from":0,"to":7}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]},{"pgid":"5.2a","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6833/2857695220":"2024-09-18T05:20:36.328050+0000","172.21.15.150:6832/2857695220":"2024-09-18T05:20:36.328050+0000","172.21.15.176:6835/1690935336":"2024-09-18T05:19:51.316025+0000","172.21.15.176:6834/1690935336":"2024-09-18T05:19:51.316025+0000","172.21.15.150:6834/3297154462":"2024-09-18T05:18:31.294755+0000","172.21.15.176:6837/1903363960":"2024-09-18T05:18:31.294598+0000","172.21.15.176:6836/1903363960":"2024-09-18T05:18:31.294598+0000","172.21.15.150:6835/3297154462":"2024-09-18T05:18:31.294755+0000","172.21.15.176:6835/524846644":"2024-09-18T05:16:53.685445+0000","172.21.15.150:6833/949083177":"2024-09-18T05:16:51.270750+0000","172.21.15.150:6834/178583496":"2024-09-18T05:15:51.246236+0000","172.21.15.176:6836/2338457617":"2024-09-18T05:15:08.260160+0000","172.21.15.150:6833/3933379792":"2024-09-18T05:15:06.233954+0000","172.21.15.150:6835/3179072684":"2024-09-18T05:14:26.220879+0000","172.21.15.150:6833/12982302":"2024-09-18T05:13:46.203493+0000","172.21.15.176:6837/2962113435":"2024-09-18T05:12:21.179693+0000","172.21.15.150:6835/178583496":"2024-09-18T05:15:51.246236+0000","172.21.15.176:6837/713081612":"2024-09-18T04:59:20.944548+0000","172.21.15.150:6834/2081398022":"2024-09-18T05:00:20.961331+0000","172.21.15.150:6835/3079958649":"2024-09-18T04:58:00.922711+0000","172.21.15.150:6834/3079958649":"2024-09-18T04:58:00.922711+0000","172.21.15.150:6832/12982302":"2024-09-18T05:13:46.203493+0000","172.21.15.150:6832/233311351":"2024-09-18T05:04:06.023230+0000","172.21.15.150:6832/949083177":"2024-09-18T05:16:51.270750+0000","172.21.15.150:6833/284564893":"2024-09-18T04:57:40.911134+0000","172.21.15.176:6837/903190072":"2024-09-18T05:04:36.031676+0000","172.21.15.176:6837/2374314256":"2024-09-18T04:56:10.885556+0000","172.21.15.150:6834/3179072684":"2024-09-18T05:14:26.220879+0000","172.21.15.176:6836/3543629780":"2024-09-18T04:58:30.930118+0000","172.21.15.150:6833/4286123257":"2024-09-18T04:55:40.878553+0000","172.21.15.150:6834/480168496":"2024-09-18T04:51:40.821573+0000","172.21.15.176:6836/1045115232":"2024-09-18T04:53:50.852056+0000","172.21.15.176:6837/1045115232":"2024-09-18T04:53:50.852056+0000","172.21.15.150:6835/285376980":"2024-09-18T05:10:51.151028+0000","172.21.15.150:6832/2313391683":"2024-09-18T05:00:45.968345+0000","172.21.15.150:6835/2081398022":"2024-09-18T05:00:20.961331+0000","172.21.15.150:6832/480168496":"2024-09-18T04:51:40.821573+0000","172.21.15.150:6834/151500360":"2024-09-18T04:54:55.866806+0000","172.21.15.150:6833/2313391683":"2024-09-18T05:00:45.968345+0000","172.21.15.150:6833/1275062894":"2024-09-18T04:58:50.936080+0000","172.21.15.150:6832/1275062894":"2024-09-18T04:58:50.936080+0000","172.21.15.150:6832/961213876":"2024-09-18T04:52:55.838337+0000","172.21.15.150:6835/151500360":"2024-09-18T04:54:55.866806+0000","172.21.15.176:6836/2374314256":"2024-09-18T04:56:10.885556+0000","172.21.15.150:6832/1237450980":"2024-09-18T05:06:31.077239+0000","172.21.15.176:6837/2338457617":"2024-09-18T05:15:08.260160+0000","172.21.15.150:6833/961213876":"2024-09-18T04:52:55.838337+0000","172.21.15.150:6832/3532647866":"2024-09-18T04:55:10.871121+0000","172.21.15.176:6836/713081612":"2024-09-18T04:59:20.944548+0000","172.21.15.176:6837/792092400":"2024-09-18T05:09:46.133255+0000","172.21.15.150:6833/2613040311":"2024-09-18T04:51:25.818032+0000","172.21.15.150:6833/3532647866":"2024-09-18T04:55:10.871121+0000","172.21.15.150:6834/2062728083":"2024-09-18T05:03:01.005221+0000","172.21.15.150:6832/523364553":"2024-09-18T05:07:21.092015+0000","172.21.15.150:6833/233311351":"2024-09-18T05:04:06.023230+0000","172.21.15.176:6836/2962113435":"2024-09-18T05:12:21.179693+0000","172.21.15.150:6832/4286123257":"2024-09-18T04:55:40.878553+0000","172.21.15.176:6836/792092400":"2024-09-18T05:09:46.133255+0000","172.21.15.176:6836/702608997":"2024-09-18T05:02:30.998209+0000","172.21.15.150:6832/284564893":"2024-09-18T04:57:40.911134+0000","172.21.15.150:6833/3622725045":"2024-09-18T05:09:06.121144+0000","172.21.15.150:6835/2062728083":"2024-09-18T05:03:01.005221+0000","172.21.15.176:6836/903190072":"2024-09-18T05:04:36.031676+0000","172.21.15.150:6832/1984876738":"2024-09-18T05:05:21.045156+0000","172.21.15.176:6834/524846644":"2024-09-18T05:16:53.685445+0000","172.21.15.150:6833/1984876738":"2024-09-18T05:05:21.045156+0000","172.21.15.150:6832/3933379792":"2024-09-18T05:15:06.233954+0000","172.21.15.150:6834/3807957687":"2024-09-18T05:05:56.055429+0000","172.21.15.150:6835/3807957687":"2024-09-18T05:05:56.055429+0000","172.21.15.176:6837/3940723333":"2024-09-18T05:06:51.083809+0000","172.21.15.150:6833/1237450980":"2024-09-18T05:06:31.077239+0000","172.21.15.176:6837/3543629780":"2024-09-18T04:58:30.930118+0000","172.21.15.176:6837/702608997":"2024-09-18T05:02:30.998209+0000","172.21.15.150:6835/92695211":"2024-09-18T05:08:16.109918+0000","172.21.15.150:6835/2613040311":"2024-09-18T04:51:25.818032+0000","172.21.15.150:6833/523364553":"2024-09-18T05:07:21.092015+0000","172.21.15.150:6834/92695211":"2024-09-18T05:08:16.109918+0000","172.21.15.150:6832/3622725045":"2024-09-18T05:09:06.121144+0000","172.21.15.176:6836/3940723333":"2024-09-18T05:06:51.083809+0000","172.21.15.150:6834/285376980":"2024-09-18T05:10:51.151028+0000","172.21.15.150:6833/2056467678":"2024-09-18T05:12:01.173633+0000","172.21.15.150:6832/2056467678":"2024-09-18T05:12:01.173633+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:21:47.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.595+0000 7ff016e6d640 1 -- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feff4073230 msgr2=0x7feff4075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.595+0000 7ff016e6d640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feff4073230 0x7feff4075740 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff0000af000 tx=0x7ff000019680 comp rx=0 tx=0).stop 2024-09-17T05:21:47.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.595+0000 7ff016e6d640 1 -- 172.21.15.150:0/3952381790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0101505a0 msgr2=0x7ff010146cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.595+0000 7ff016e6d640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0101505a0 0x7ff010146cb0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7ff00800c760 tx=0x7ff00800cc20 comp rx=0 tx=0).stop 2024-09-17T05:21:47.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.595+0000 7ff016e6d640 1 -- 172.21.15.150:0/3952381790 shutdown_connections 2024-09-17T05:21:47.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.595+0000 7ff016e6d640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7feff4073230 0x7feff4075740 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.595+0000 7ff016e6d640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff0101505a0 0x7ff010146cb0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.595+0000 7ff016e6d640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff0100c42e0 0x7ff0101f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.595+0000 7ff016e6d640 1 --2- 172.21.15.150:0/3952381790 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0100a7350 0x7ff01014c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.595+0000 7ff016e6d640 1 -- 172.21.15.150:0/3952381790 >> 172.21.15.150:0/3952381790 conn(0x7ff0100c4a00 msgr2=0x7ff0100c3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.595+0000 7ff016e6d640 1 -- 172.21.15.150:0/3952381790 shutdown_connections 2024-09-17T05:21:47.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.599+0000 7ff016e6d640 1 -- 172.21.15.150:0/3952381790 wait complete. 2024-09-17T05:21:47.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail a 2024-09-17T05:21:47.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.775+0000 7f7530d7d640 1 -- 172.21.15.150:0/3352360436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7518003a10 msgr2=0x7f7518003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.775+0000 7f7530d7d640 1 --2- 172.21.15.150:0/3352360436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7518003a10 0x7f7518003e80 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f751c00bb10 tx=0x7f751c01cd60 comp rx=0 tx=0).stop 2024-09-17T05:21:47.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.775+0000 7f752ad9d640 1 -- 172.21.15.150:0/3352360436 reap_dead start 2024-09-17T05:21:47.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.775+0000 7f7530d7d640 1 -- 172.21.15.150:0/3352360436 shutdown_connections 2024-09-17T05:21:47.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.775+0000 7f7530d7d640 1 -- 172.21.15.150:0/3352360436 >> 172.21.15.150:0/3352360436 conn(0x7f752c0c4a00 msgr2=0x7f752c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:47.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7530d7d640 1 -- 172.21.15.150:0/3352360436 shutdown_connections 2024-09-17T05:21:47.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7530d7d640 1 -- 172.21.15.150:0/3352360436 wait complete. 2024-09-17T05:21:47.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7530d7d640 1 Processor -- start 2024-09-17T05:21:47.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7530d7d640 1 -- start start 2024-09-17T05:21:47.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7530d7d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7518003a10 0x7f752c14f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:47.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7530d7d640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f752c156e40 0x7f752c14dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:47.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7530d7d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c14e190 0x7f752c144260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:47.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7530d7d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f752c150770 con 0x7f752c14e190 2024-09-17T05:21:47.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7530d7d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f752c1508f0 con 0x7f7518003a10 2024-09-17T05:21:47.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7530d7d640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f752c14e970 con 0x7f752c156e40 2024-09-17T05:21:47.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f752ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c14e190 0x7f752c144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:47.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f752ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c14e190 0x7f752c144260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50112/0 (socket says 172.21.15.150:50112) 2024-09-17T05:21:47.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f752ad9d640 1 -- 172.21.15.150:0/2246275474 learned_addr learned my addr 172.21.15.150:0/2246275474 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:47.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f752a59c640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7518003a10 0x7f752c14f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:47.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7523fff640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f752c156e40 0x7f752c14dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:47.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f752ad9d640 1 -- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f752c156e40 msgr2=0x7f752c14dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f752ad9d640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f752c156e40 0x7f752c14dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f752ad9d640 1 -- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7518003a10 msgr2=0x7f752c14f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f752ad9d640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7518003a10 0x7f752c14f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f752ad9d640 1 -- 172.21.15.150:0/2246275474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f752c145050 con 0x7f752c14e190 2024-09-17T05:21:47.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f752ad9d640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c14e190 0x7f752c144260 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f752c0ae070 tx=0x7f7524010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:47.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7522ffd640 1 -- 172.21.15.150:0/2246275474 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7524004b30 con 0x7f752c14e190 2024-09-17T05:21:47.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7522ffd640 1 -- 172.21.15.150:0/2246275474 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f7524015070 con 0x7f752c14e190 2024-09-17T05:21:47.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.779+0000 7f7530d7d640 1 -- 172.21.15.150:0/2246275474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f752c1f3980 con 0x7f752c14e190 2024-09-17T05:21:47.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.783+0000 7f7522ffd640 1 -- 172.21.15.150:0/2246275474 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7524003980 con 0x7f752c14e190 2024-09-17T05:21:47.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.783+0000 7f7530d7d640 1 -- 172.21.15.150:0/2246275474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f752c14d730 con 0x7f752c14e190 2024-09-17T05:21:47.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.783+0000 7f7530d7d640 1 -- 172.21.15.150:0/2246275474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f752c0c6070 con 0x7f752c14e190 2024-09-17T05:21:47.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.783+0000 7f7522ffd640 1 -- 172.21.15.150:0/2246275474 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f7524003b20 con 0x7f752c14e190 2024-09-17T05:21:47.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.783+0000 7f7522ffd640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f750c073200 0x7f750c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:47.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.783+0000 7f7522ffd640 1 -- 172.21.15.150:0/2246275474 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10192+0+0 (secure 0 0 0) 0x7f7524095ae0 con 0x7f752c14e190 2024-09-17T05:21:47.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.783+0000 7f752a59c640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f750c073200 0x7f750c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:47.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.783+0000 7f752a59c640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f750c073200 0x7f750c075710 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f751c0b0000 tx=0x7f751c0a7040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:47.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.787+0000 7f7522ffd640 1 -- 172.21.15.150:0/2246275474 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f752405f930 con 0x7f752c14e190 2024-09-17T05:21:47.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.979+0000 7f7530d7d640 1 -- 172.21.15.150:0/2246275474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "a"} v 0) -- 0x7f752c0a7130 con 0x7f752c14e190 2024-09-17T05:21:47.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.983+0000 7f7522ffd640 1 -- 172.21.15.150:0/2246275474 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "a"}]=0 a marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v214) ==== 151+0+0 (secure 0 0 0) 0x7f752405f230 con 0x7f752c14e190 2024-09-17T05:21:47.980 INFO:teuthology.orchestra.run.smithi150.stderr:a marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T05:21:47.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.987+0000 7f7530d7d640 1 -- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f750c073200 msgr2=0x7f750c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.987+0000 7f7530d7d640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f750c073200 0x7f750c075710 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f751c0b0000 tx=0x7f751c0a7040 comp rx=0 tx=0).stop 2024-09-17T05:21:47.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.987+0000 7f7530d7d640 1 -- 172.21.15.150:0/2246275474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c14e190 msgr2=0x7f752c144260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:47.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.987+0000 7f7530d7d640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c14e190 0x7f752c144260 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f752c0ae070 tx=0x7f7524010f40 comp rx=0 tx=0).stop 2024-09-17T05:21:47.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.987+0000 7f7530d7d640 1 -- 172.21.15.150:0/2246275474 shutdown_connections 2024-09-17T05:21:47.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.987+0000 7f7530d7d640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f750c073200 0x7f750c075710 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.987+0000 7f7530d7d640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f752c14e190 0x7f752c144260 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.987+0000 7f7530d7d640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f752c156e40 0x7f752c14dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.987+0000 7f7530d7d640 1 --2- 172.21.15.150:0/2246275474 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7518003a10 0x7f752c14f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:47.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.987+0000 7f7530d7d640 1 -- 172.21.15.150:0/2246275474 >> 172.21.15.150:0/2246275474 conn(0x7f752c0c4a00 msgr2=0x7f752c0c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:47.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.987+0000 7f7530d7d640 1 -- 172.21.15.150:0/2246275474 shutdown_connections 2024-09-17T05:21:47.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:47.987+0000 7f7530d7d640 1 -- 172.21.15.150:0/2246275474 wait complete. 2024-09-17T05:21:47.990 INFO:tasks.ceph.mds.c.smithi150.stderr: -6106> 2024-09-17T05:19:56.909+0000 7fa135639640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T05:21:48.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm a --yes-i-really-mean-it 2024-09-17T05:21:48.051 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-09-17T05:21:48.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.167+0000 7f5bd9793640 1 -- 172.21.15.150:0/2801803372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bbc004e70 msgr2=0x7f5bbc0052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:48.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.167+0000 7f5bd9793640 1 --2- 172.21.15.150:0/2801803372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bbc004e70 0x7f5bbc0052a0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f5bd40c9cd0 tx=0x7f5bcc01caf0 comp rx=0 tx=0).stop 2024-09-17T05:21:48.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.167+0000 7f5bd37fe640 1 -- 172.21.15.150:0/2801803372 reap_dead start 2024-09-17T05:21:48.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.167+0000 7f5bd9793640 1 -- 172.21.15.150:0/2801803372 shutdown_connections 2024-09-17T05:21:48.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.167+0000 7f5bd9793640 1 -- 172.21.15.150:0/2801803372 >> 172.21.15.150:0/2801803372 conn(0x7f5bd40c9de0 msgr2=0x7f5bd40ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:48.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.167+0000 7f5bd9793640 1 -- 172.21.15.150:0/2801803372 shutdown_connections 2024-09-17T05:21:48.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.167+0000 7f5bd9793640 1 -- 172.21.15.150:0/2801803372 wait complete. 2024-09-17T05:21:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.167+0000 7f5bd9793640 1 Processor -- start 2024-09-17T05:21:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.167+0000 7f5bd9793640 1 -- start start 2024-09-17T05:21:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd9793640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5bbc004e70 0x7f5bd4154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd9793640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5bd415d210 0x7f5bd415c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd9793640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd4156ca0 0x7f5bd414cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd9793640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd40c7ce0 con 0x7f5bd4156ca0 2024-09-17T05:21:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd9793640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd40c5a40 con 0x7f5bd415d210 2024-09-17T05:21:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd9793640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f5bd40c7e60 con 0x7f5bbc004e70 2024-09-17T05:21:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd4156ca0 0x7f5bd414cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd4156ca0 0x7f5bd414cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50136/0 (socket says 172.21.15.150:50136) 2024-09-17T05:21:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd37fe640 1 -- 172.21.15.150:0/2921382155 learned_addr learned my addr 172.21.15.150:0/2921382155 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd27fc640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5bd415d210 0x7f5bd415c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd2ffd640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5bbc004e70 0x7f5bd4154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd37fe640 1 -- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5bbc004e70 msgr2=0x7f5bd4154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd37fe640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5bbc004e70 0x7f5bd4154890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd37fe640 1 -- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5bd415d210 msgr2=0x7f5bd415c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:48.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd37fe640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5bd415d210 0x7f5bd415c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:48.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd37fe640 1 -- 172.21.15.150:0/2921382155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd40aa080 con 0x7f5bd4156ca0 2024-09-17T05:21:48.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd37fe640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd4156ca0 0x7f5bd414cb60 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc006040 tx=0x7f5bcc006e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:48.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bbb7fe640 1 -- 172.21.15.150:0/2921382155 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc006cb0 con 0x7f5bd4156ca0 2024-09-17T05:21:48.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bbb7fe640 1 -- 172.21.15.150:0/2921382155 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f5bcc0a5a50 con 0x7f5bd4156ca0 2024-09-17T05:21:48.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd9793640 1 -- 172.21.15.150:0/2921382155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd415ba20 con 0x7f5bd4156ca0 2024-09-17T05:21:48.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bbb7fe640 1 -- 172.21.15.150:0/2921382155 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc0a5e60 con 0x7f5bd4156ca0 2024-09-17T05:21:48.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd9793640 1 -- 172.21.15.150:0/2921382155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd4154270 con 0x7f5bd4156ca0 2024-09-17T05:21:48.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bd9793640 1 -- 172.21.15.150:0/2921382155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bd40c5330 con 0x7f5bd4156ca0 2024-09-17T05:21:48.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bbb7fe640 1 -- 172.21.15.150:0/2921382155 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f5bcc006070 con 0x7f5bd4156ca0 2024-09-17T05:21:48.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bbb7fe640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5bc0073200 0x7f5bc0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:48.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.171+0000 7f5bbb7fe640 1 -- 172.21.15.150:0/2921382155 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 10278+0+0 (secure 0 0 0) 0x7f5bcc12e9b0 con 0x7f5bd4156ca0 2024-09-17T05:21:48.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.175+0000 7f5bd2ffd640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5bc0073200 0x7f5bc0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:48.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.175+0000 7f5bd2ffd640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5bc0073200 0x7f5bc0075710 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4005fd0 tx=0x7f5bc4005f40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:48.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.179+0000 7f5bbb7fe640 1 -- 172.21.15.150:0/2921382155 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5bcc0b4070 con 0x7f5bd4156ca0 2024-09-17T05:21:48.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.371+0000 7f5bd9793640 1 -- 172.21.15.150:0/2921382155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "a", "yes_i_really_mean_it": true} v 0) -- 0x7f5bd406a7e0 con 0x7f5bd4156ca0 2024-09-17T05:21:48.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.983+0000 7f5bbb7fe640 1 -- 172.21.15.150:0/2921382155 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "a", "yes_i_really_mean_it": true}]=0 v215) ==== 99+0+0 (secure 0 0 0) 0x7f5bcc016850 con 0x7f5bd4156ca0 2024-09-17T05:21:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.987+0000 7f5bd9793640 1 -- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5bc0073200 msgr2=0x7f5bc0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.987+0000 7f5bd9793640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5bc0073200 0x7f5bc0075710 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4005fd0 tx=0x7f5bc4005f40 comp rx=0 tx=0).stop 2024-09-17T05:21:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.987+0000 7f5bd9793640 1 -- 172.21.15.150:0/2921382155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd4156ca0 msgr2=0x7f5bd414cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.987+0000 7f5bd9793640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd4156ca0 0x7f5bd414cb60 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc006040 tx=0x7f5bcc006e80 comp rx=0 tx=0).stop 2024-09-17T05:21:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.987+0000 7f5bd9793640 1 -- 172.21.15.150:0/2921382155 shutdown_connections 2024-09-17T05:21:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.987+0000 7f5bd9793640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f5bc0073200 0x7f5bc0075710 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.987+0000 7f5bd9793640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5bd415d210 0x7f5bd415c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.987+0000 7f5bd9793640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5bd4156ca0 0x7f5bd414cb60 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.987+0000 7f5bd9793640 1 --2- 172.21.15.150:0/2921382155 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f5bbc004e70 0x7f5bd4154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.987+0000 7f5bd9793640 1 -- 172.21.15.150:0/2921382155 >> 172.21.15.150:0/2921382155 conn(0x7f5bd40c9de0 msgr2=0x7f5bd415f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.987+0000 7f5bd9793640 1 -- 172.21.15.150:0/2921382155 shutdown_connections 2024-09-17T05:21:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:48.987+0000 7f5bd9793640 1 -- 172.21.15.150:0/2921382155 wait complete. 2024-09-17T05:21:49.016 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm a_metadata a_metadata --yes-i-really-really-mean-it 2024-09-17T05:21:49.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.231+0000 7fabe46ca640 1 -- 172.21.15.150:0/2166965292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc8004e70 msgr2=0x7fabc80052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:49.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.231+0000 7fabe46ca640 1 --2- 172.21.15.150:0/2166965292 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabc8004e70 0x7fabc80052c0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fabdc0705b0 tx=0x7fabd401caf0 comp rx=0 tx=0).stop 2024-09-17T05:21:49.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.231+0000 7fabe2c67640 1 -- 172.21.15.150:0/2166965292 reap_dead start 2024-09-17T05:21:49.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 -- 172.21.15.150:0/2166965292 shutdown_connections 2024-09-17T05:21:49.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 -- 172.21.15.150:0/2166965292 >> 172.21.15.150:0/2166965292 conn(0x7fabdc0c7fc0 msgr2=0x7fabdc0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:49.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 -- 172.21.15.150:0/2166965292 shutdown_connections 2024-09-17T05:21:49.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 -- 172.21.15.150:0/2166965292 wait complete. 2024-09-17T05:21:49.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 Processor -- start 2024-09-17T05:21:49.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 -- start start 2024-09-17T05:21:49.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabdc0ac560 0x7fabdc0a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:49.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabdc160130 0x7fabdc1f1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:49.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabdc14e040 0x7fabdc14e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:49.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc0c64e0 con 0x7fabdc160130 2024-09-17T05:21:49.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc0713a0 con 0x7fabdc0ac560 2024-09-17T05:21:49.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fabdc1570f0 con 0x7fabdc14e040 2024-09-17T05:21:49.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe1c65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabdc160130 0x7fabdc1f1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:49.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe1c65640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabdc160130 0x7fabdc1f1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50150/0 (socket says 172.21.15.150:50150) 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe1c65640 1 -- 172.21.15.150:0/2964168024 learned_addr learned my addr 172.21.15.150:0/2964168024 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe2c67640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabdc14e040 0x7fabdc14e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe1c65640 1 -- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabdc14e040 msgr2=0x7fabdc14e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe1c65640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabdc14e040 0x7fabdc14e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe2466640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabdc0ac560 0x7fabdc0a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe1c65640 1 -- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabdc0ac560 msgr2=0x7fabdc0a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe1c65640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabdc0ac560 0x7fabdc0a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe1c65640 1 -- 172.21.15.150:0/2964168024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabdc0a9a20 con 0x7fabdc160130 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe1c65640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabdc160130 0x7fabdc1f1510 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fabd80109e0 tx=0x7fabd8010ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabd2ffd640 1 -- 172.21.15.150:0/2964168024 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd800ecb0 con 0x7fabdc160130 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabd2ffd640 1 -- 172.21.15.150:0/2964168024 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fabd8012450 con 0x7fabdc160130 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabe46ca640 1 -- 172.21.15.150:0/2964168024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabdc0a7990 con 0x7fabdc160130 2024-09-17T05:21:49.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.235+0000 7fabd2ffd640 1 -- 172.21.15.150:0/2964168024 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd80128a0 con 0x7fabdc160130 2024-09-17T05:21:49.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.239+0000 7fabe46ca640 1 -- 172.21.15.150:0/2964168024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabdc0a7c40 con 0x7fabdc160130 2024-09-17T05:21:49.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.239+0000 7fabe46ca640 1 -- 172.21.15.150:0/2964168024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabdc0c6ef0 con 0x7fabdc160130 2024-09-17T05:21:49.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.239+0000 7fabd2ffd640 1 -- 172.21.15.150:0/2964168024 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fabd8012a40 con 0x7fabdc160130 2024-09-17T05:21:49.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.239+0000 7fabd2ffd640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabc0073200 0x7fabc0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:49.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.239+0000 7fabd2ffd640 1 -- 172.21.15.150:0/2964168024 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 10278+0+0 (secure 0 0 0) 0x7fabd8096a40 con 0x7fabdc160130 2024-09-17T05:21:49.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.239+0000 7fabe2466640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabc0073200 0x7fabc0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:49.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.239+0000 7fabe2466640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabc0073200 0x7fabc0075710 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fabdc069640 tx=0x7fabcc0010f0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:49.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.243+0000 7fabd2ffd640 1 -- 172.21.15.150:0/2964168024 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fabd80609d0 con 0x7fabdc160130 2024-09-17T05:21:49.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.415+0000 7fabe46ca640 1 -- 172.21.15.150:0/2964168024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "a_metadata", "pool2": "a_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fabdc0aabb0 con 0x7fabdc160130 2024-09-17T05:21:49.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.987+0000 7fabd2ffd640 1 -- 172.21.15.150:0/2964168024 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "a_metadata", "pool2": "a_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'a_metadata' removed v70) ==== 166+0+0 (secure 0 0 0) 0x7fabd80602d0 con 0x7fabdc160130 2024-09-17T05:21:49.987 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'a_metadata' removed 2024-09-17T05:21:49.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.991+0000 7fabe46ca640 1 -- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabc0073200 msgr2=0x7fabc0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:49.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.991+0000 7fabe46ca640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabc0073200 0x7fabc0075710 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fabdc069640 tx=0x7fabcc0010f0 comp rx=0 tx=0).stop 2024-09-17T05:21:49.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.991+0000 7fabe46ca640 1 -- 172.21.15.150:0/2964168024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabdc160130 msgr2=0x7fabdc1f1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:49.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.991+0000 7fabe46ca640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabdc160130 0x7fabdc1f1510 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fabd80109e0 tx=0x7fabd8010ea0 comp rx=0 tx=0).stop 2024-09-17T05:21:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.991+0000 7fabe46ca640 1 -- 172.21.15.150:0/2964168024 shutdown_connections 2024-09-17T05:21:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.991+0000 7fabe46ca640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fabc0073200 0x7fabc0075710 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.991+0000 7fabe46ca640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fabdc14e040 0x7fabdc14e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.991+0000 7fabe46ca640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fabdc160130 0x7fabdc1f1510 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.991+0000 7fabe46ca640 1 --2- 172.21.15.150:0/2964168024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fabdc0ac560 0x7fabdc0a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.991+0000 7fabe46ca640 1 -- 172.21.15.150:0/2964168024 >> 172.21.15.150:0/2964168024 conn(0x7fabdc0c7fc0 msgr2=0x7fabdc152200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.991+0000 7fabe46ca640 1 -- 172.21.15.150:0/2964168024 shutdown_connections 2024-09-17T05:21:49.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:49.991+0000 7fabe46ca640 1 -- 172.21.15.150:0/2964168024 wait complete. 2024-09-17T05:21:50.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm a_data a_data --yes-i-really-really-mean-it 2024-09-17T05:21:50.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 -- 172.21.15.150:0/449348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14f80c3b80 msgr2=0x7f14f80c3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:50.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 --2- 172.21.15.150:0/449348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14f80c3b80 0x7f14f80c3f90 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f14f8071ac0 tx=0x7f14e801c7a0 comp rx=0 tx=0).stop 2024-09-17T05:21:50.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 -- 172.21.15.150:0/449348 shutdown_connections 2024-09-17T05:21:50.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 --2- 172.21.15.150:0/449348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14f80c3b80 0x7f14f80c3f90 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:50.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 --2- 172.21.15.150:0/449348 >> v2:172.21.15.176:3300/0 conn(0x7f14f80c34d0 0x7f14f80c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:50.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 -- 172.21.15.150:0/449348 >> 172.21.15.150:0/449348 conn(0x7f14f80c7fc0 msgr2=0x7f14f80c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:50.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 -- 172.21.15.150:0/449348 shutdown_connections 2024-09-17T05:21:50.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 -- 172.21.15.150:0/449348 wait complete. 2024-09-17T05:21:50.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 Processor -- start 2024-09-17T05:21:50.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 -- start start 2024-09-17T05:21:50.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14f80c34d0 0x7f14f81ee2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:50.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14f80c3b80 0x7f14f8152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:50.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f80c5cc0 0x7f14f80ac2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f14f80c64e0 con 0x7f14f80c3b80 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f14f80713a0 con 0x7f14f80c34d0 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.187+0000 7f1500203640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f14f81570f0 con 0x7f14f80c5cc0 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fd79e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14f80c3b80 0x7f14f8152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fd79e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14f80c3b80 0x7f14f8152170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51314/0 (socket says 172.21.15.150:51314) 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fd79e640 1 -- 172.21.15.150:0/2066504375 learned_addr learned my addr 172.21.15.150:0/2066504375 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fdf9f640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14f80c34d0 0x7f14f81ee2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fe7a0640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f80c5cc0 0x7f14f80ac2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fd79e640 1 -- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f80c5cc0 msgr2=0x7f14f80ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fd79e640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f80c5cc0 0x7f14f80ac2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fd79e640 1 -- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14f80c34d0 msgr2=0x7f14f81ee2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fd79e640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14f80c34d0 0x7f14f81ee2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fd79e640 1 -- 172.21.15.150:0/2066504375 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f81515b0 con 0x7f14f80c3b80 2024-09-17T05:21:50.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fe7a0640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f80c5cc0 0x7f14f80ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:50.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fd79e640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14f80c3b80 0x7f14f8152170 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f14e801cc80 tx=0x7f14e8005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:50.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14ee7fc640 1 -- 172.21.15.150:0/2066504375 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e80043c0 con 0x7f14f80c3b80 2024-09-17T05:21:50.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14ee7fc640 1 -- 172.21.15.150:0/2066504375 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f14e8003600 con 0x7f14f80c3b80 2024-09-17T05:21:50.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f1500203640 1 -- 172.21.15.150:0/2066504375 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f81ed810 con 0x7f14f80c3b80 2024-09-17T05:21:50.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14ee7fc640 1 -- 172.21.15.150:0/2066504375 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e8003a30 con 0x7f14f80c3b80 2024-09-17T05:21:50.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f1500203640 1 -- 172.21.15.150:0/2066504375 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f81edad0 con 0x7f14f80c3b80 2024-09-17T05:21:50.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f1500203640 1 -- 172.21.15.150:0/2066504375 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14f80c6ef0 con 0x7f14f80c3b80 2024-09-17T05:21:50.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14ee7fc640 1 -- 172.21.15.150:0/2066504375 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f14e80a6040 con 0x7f14f80c3b80 2024-09-17T05:21:50.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14ee7fc640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f14dc073230 0x7f14dc075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:50.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14ee7fc640 1 -- 172.21.15.150:0/2066504375 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 9850+0+0 (secure 0 0 0) 0x7f14e812fa80 con 0x7f14f80c3b80 2024-09-17T05:21:50.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.191+0000 7f14fdf9f640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f14dc073230 0x7f14dc075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:50.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.195+0000 7f14fdf9f640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f14dc073230 0x7f14dc075740 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f14f4005fd0 tx=0x7f14f4005d10 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:50.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.195+0000 7f14ee7fc640 1 -- 172.21.15.150:0/2066504375 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f14e80ac030 con 0x7f14f80c3b80 2024-09-17T05:21:50.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.371+0000 7f1500203640 1 -- 172.21.15.150:0/2066504375 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "a_data", "pool2": "a_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f14f814f1b0 con 0x7f14f80c3b80 2024-09-17T05:21:50.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:50.999+0000 7f14ee7fc640 1 -- 172.21.15.150:0/2066504375 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "a_data", "pool2": "a_data", "yes_i_really_really_mean_it": true}]=0 pool 'a_data' removed v71) ==== 154+0+0 (secure 0 0 0) 0x7f14e80164b0 con 0x7f14f80c3b80 2024-09-17T05:21:51.000 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'a_data' removed 2024-09-17T05:21:51.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.007+0000 7f1500203640 1 -- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f14dc073230 msgr2=0x7f14dc075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:51.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.007+0000 7f1500203640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f14dc073230 0x7f14dc075740 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f14f4005fd0 tx=0x7f14f4005d10 comp rx=0 tx=0).stop 2024-09-17T05:21:51.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.007+0000 7f1500203640 1 -- 172.21.15.150:0/2066504375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14f80c3b80 msgr2=0x7f14f8152170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:51.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.007+0000 7f1500203640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14f80c3b80 0x7f14f8152170 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f14e801cc80 tx=0x7f14e8005770 comp rx=0 tx=0).stop 2024-09-17T05:21:51.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.007+0000 7f1500203640 1 -- 172.21.15.150:0/2066504375 shutdown_connections 2024-09-17T05:21:51.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.007+0000 7f1500203640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f14dc073230 0x7f14dc075740 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.007+0000 7f1500203640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f14f80c5cc0 0x7f14f80ac2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.007+0000 7f1500203640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14f80c3b80 0x7f14f8152170 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.007+0000 7f1500203640 1 --2- 172.21.15.150:0/2066504375 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14f80c34d0 0x7f14f81ee2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.007+0000 7f1500203640 1 -- 172.21.15.150:0/2066504375 >> 172.21.15.150:0/2066504375 conn(0x7f14f80c7fc0 msgr2=0x7f14f814aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:51.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.007+0000 7f1500203640 1 -- 172.21.15.150:0/2066504375 shutdown_connections 2024-09-17T05:21:51.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.007+0000 7f1500203640 1 -- 172.21.15.150:0/2066504375 wait complete. 2024-09-17T05:21:51.046 INFO:tasks.cephfs.filesystem:Destroying file system b and related pools 2024-09-17T05:21:51.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:21:51.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.223+0000 7f8cb8f19640 1 -- 172.21.15.150:0/4291541515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40717a0 msgr2=0x7f8cb4074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:51.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.223+0000 7f8cb8f19640 1 --2- 172.21.15.150:0/4291541515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40717a0 0x7f8cb4074450 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4070590 tx=0x7f8cac01c770 comp rx=0 tx=0).stop 2024-09-17T05:21:51.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.223+0000 7f8cb8f19640 1 -- 172.21.15.150:0/4291541515 shutdown_connections 2024-09-17T05:21:51.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.223+0000 7f8cb8f19640 1 --2- 172.21.15.150:0/4291541515 >> v2:172.21.15.176:3300/0 conn(0x7f8cb4070e50 0x7f8cb4071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.223+0000 7f8cb8f19640 1 --2- 172.21.15.150:0/4291541515 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb40717a0 0x7f8cb4074450 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.223+0000 7f8cb8f19640 1 -- 172.21.15.150:0/4291541515 >> 172.21.15.150:0/4291541515 conn(0x7f8cb40c4a00 msgr2=0x7f8cb40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:51.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.223+0000 7f8cb8f19640 1 -- 172.21.15.150:0/4291541515 shutdown_connections 2024-09-17T05:21:51.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.223+0000 7f8cb8f19640 1 -- 172.21.15.150:0/4291541515 wait complete. 2024-09-17T05:21:51.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.223+0000 7f8cb8f19640 1 Processor -- start 2024-09-17T05:21:51.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.223+0000 7f8cb8f19640 1 -- start start 2024-09-17T05:21:51.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb8f19640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb4070e50 0x7f8cb40aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb8f19640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb40717a0 0x7f8cb4157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb8f19640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb4073440 0x7f8cb40c1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb8f19640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb40ad270 con 0x7f8cb4073440 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb8f19640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb40ad0f0 con 0x7f8cb4070e50 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb8f19640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f8cb40ad3f0 con 0x7f8cb40717a0 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb4073440 0x7f8cb40c1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb4073440 0x7f8cb40c1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51322/0 (socket says 172.21.15.150:51322) 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb2d9d640 1 -- 172.21.15.150:0/2383907528 learned_addr learned my addr 172.21.15.150:0/2383907528 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb259c640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb4070e50 0x7f8cb40aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb1d9b640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb40717a0 0x7f8cb4157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb2d9d640 1 -- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb40717a0 msgr2=0x7f8cb4157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb2d9d640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb40717a0 0x7f8cb4157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb2d9d640 1 -- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb4070e50 msgr2=0x7f8cb40aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:51.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb2d9d640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb4070e50 0x7f8cb40aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb2d9d640 1 -- 172.21.15.150:0/2383907528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb41f0dd0 con 0x7f8cb4073440 2024-09-17T05:21:51.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb259c640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb4070e50 0x7f8cb40aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:51.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb1d9b640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb40717a0 0x7f8cb4157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:51.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb2d9d640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb4073440 0x7f8cb40c1160 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f8cac01a3a0 tx=0x7f8cac0a55f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:51.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8caaffd640 1 -- 172.21.15.150:0/2383907528 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac0bb070 con 0x7f8cb4073440 2024-09-17T05:21:51.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8caaffd640 1 -- 172.21.15.150:0/2383907528 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f8cac0a5d00 con 0x7f8cb4073440 2024-09-17T05:21:51.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb8f19640 1 -- 172.21.15.150:0/2383907528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb4157a40 con 0x7f8cb4073440 2024-09-17T05:21:51.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8caaffd640 1 -- 172.21.15.150:0/2383907528 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac0b2040 con 0x7f8cb4073440 2024-09-17T05:21:51.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb8f19640 1 -- 172.21.15.150:0/2383907528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb41f0a60 con 0x7f8cb4073440 2024-09-17T05:21:51.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8cb8f19640 1 -- 172.21.15.150:0/2383907528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cb40ad9d0 con 0x7f8cb4073440 2024-09-17T05:21:51.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8caaffd640 1 -- 172.21.15.150:0/2383907528 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f8cac0be070 con 0x7f8cb4073440 2024-09-17T05:21:51.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8caaffd640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c94073230 0x7f8c94075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:51.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.227+0000 7f8caaffd640 1 -- 172.21.15.150:0/2383907528 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 9417+0+0 (secure 0 0 0) 0x7f8cac12ea20 con 0x7f8cb4073440 2024-09-17T05:21:51.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.231+0000 7f8cb259c640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c94073230 0x7f8c94075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:51.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.231+0000 7f8cb259c640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c94073230 0x7f8c94075740 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8cb41513e0 tx=0x7f8c9c009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:51.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.235+0000 7f8caaffd640 1 -- 172.21.15.150:0/2383907528 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8cac0163f0 con 0x7f8cb4073440 2024-09-17T05:21:51.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.439+0000 7f8cb8f19640 1 -- 172.21.15.150:0/2383907528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cb40ac3d0 con 0x7f8cb4073440 2024-09-17T05:21:51.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.439+0000 7f8caaffd640 1 -- 172.21.15.150:0/2383907528 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) ==== 95+0+4411 (secure 0 0 0) 0x7f8cac0f8c90 con 0x7f8cb4073440 2024-09-17T05:21:51.438 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:21:51.438 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":215,"btime":"2024-09-17T05:21:48:986587+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":8571,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3816691969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3816691969},{"type":"v1","addr":"172.21.15.150:6833","nonce":3816691969}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":215}],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:20:39.523977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7567":{"gid":7567,"name":"b","rank":0,"incarnation":210,"state":"up:active","state_seq":12,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7567,"qdb_cluster":[7567]},"id":2}]} 2024-09-17T05:21:51.439 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 215 2024-09-17T05:21:51.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.443+0000 7f8cb8f19640 1 -- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c94073230 msgr2=0x7f8c94075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:51.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.443+0000 7f8cb8f19640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c94073230 0x7f8c94075740 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f8cb41513e0 tx=0x7f8c9c009040 comp rx=0 tx=0).stop 2024-09-17T05:21:51.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.443+0000 7f8cb8f19640 1 -- 172.21.15.150:0/2383907528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb4073440 msgr2=0x7f8cb40c1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:51.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.443+0000 7f8cb8f19640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb4073440 0x7f8cb40c1160 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f8cac01a3a0 tx=0x7f8cac0a55f0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.443+0000 7f8cb8f19640 1 -- 172.21.15.150:0/2383907528 shutdown_connections 2024-09-17T05:21:51.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.443+0000 7f8cb8f19640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f8c94073230 0x7f8c94075740 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.443+0000 7f8cb8f19640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cb4073440 0x7f8cb40c1160 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.443+0000 7f8cb8f19640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f8cb40717a0 0x7f8cb4157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.443+0000 7f8cb8f19640 1 --2- 172.21.15.150:0/2383907528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8cb4070e50 0x7f8cb40aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.443+0000 7f8cb8f19640 1 -- 172.21.15.150:0/2383907528 >> 172.21.15.150:0/2383907528 conn(0x7f8cb40c4a00 msgr2=0x7f8cb40c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:51.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.443+0000 7f8cb8f19640 1 -- 172.21.15.150:0/2383907528 shutdown_connections 2024-09-17T05:21:51.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.443+0000 7f8cb8f19640 1 -- 172.21.15.150:0/2383907528 wait complete. 2024-09-17T05:21:51.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:21:51.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.623+0000 7ff499153640 1 -- 172.21.15.150:0/1182997343 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4940c42e0 msgr2=0x7ff4940c6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:51.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.623+0000 7ff499153640 1 --2- 172.21.15.150:0/1182997343 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4940c42e0 0x7ff4940c6690 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff484009840 tx=0x7ff48401c7f0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.623+0000 7ff499153640 1 -- 172.21.15.150:0/1182997343 shutdown_connections 2024-09-17T05:21:51.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.623+0000 7ff499153640 1 --2- 172.21.15.150:0/1182997343 >> v2:172.21.15.176:3300/0 conn(0x7ff494152cf0 0x7ff494153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.623+0000 7ff499153640 1 --2- 172.21.15.150:0/1182997343 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff4940c42e0 0x7ff4940c6690 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.623+0000 7ff499153640 1 -- 172.21.15.150:0/1182997343 >> 172.21.15.150:0/1182997343 conn(0x7ff4940c4a00 msgr2=0x7ff4940c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:51.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.623+0000 7ff499153640 1 -- 172.21.15.150:0/1182997343 shutdown_connections 2024-09-17T05:21:51.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.623+0000 7ff499153640 1 -- 172.21.15.150:0/1182997343 wait complete. 2024-09-17T05:21:51.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff499153640 1 Processor -- start 2024-09-17T05:21:51.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff499153640 1 -- start start 2024-09-17T05:21:51.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff499153640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4940a7350 0x7ff494143920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:51.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff499153640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4940c42e0 0x7ff494149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:51.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff499153640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff494152cf0 0x7ff49413df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:51.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff499153640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff4941537c0 con 0x7ff4940a7350 2024-09-17T05:21:51.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff499153640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff494153640 con 0x7ff4940c42e0 2024-09-17T05:21:51.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff499153640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff494150770 con 0x7ff494152cf0 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff492d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4940a7350 0x7ff494143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff492d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4940a7350 0x7ff494143920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51326/0 (socket says 172.21.15.150:51326) 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff492d9d640 1 -- 172.21.15.150:0/3223676296 learned_addr learned my addr 172.21.15.150:0/3223676296 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff49359e640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff494152cf0 0x7ff49413df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff49259c640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4940c42e0 0x7ff494149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff492d9d640 1 -- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff494152cf0 msgr2=0x7ff49413df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff492d9d640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff494152cf0 0x7ff49413df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff492d9d640 1 -- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4940c42e0 msgr2=0x7ff494149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff492d9d640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4940c42e0 0x7ff494149990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff492d9d640 1 -- 172.21.15.150:0/3223676296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4941ee4f0 con 0x7ff4940a7350 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff49259c640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4940c42e0 0x7ff494149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff492d9d640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4940a7350 0x7ff494143920 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7ff484019400 tx=0x7ff4840a5720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff47b7fe640 1 -- 172.21.15.150:0/3223676296 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4840b2040 con 0x7ff4940a7350 2024-09-17T05:21:51.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff47b7fe640 1 -- 172.21.15.150:0/3223676296 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7ff484004200 con 0x7ff4940a7350 2024-09-17T05:21:51.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff499153640 1 -- 172.21.15.150:0/3223676296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4941ee7d0 con 0x7ff4940a7350 2024-09-17T05:21:51.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff47b7fe640 1 -- 172.21.15.150:0/3223676296 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4840050c0 con 0x7ff4940a7350 2024-09-17T05:21:51.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.627+0000 7ff499153640 1 -- 172.21.15.150:0/3223676296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff49414a3f0 con 0x7ff4940a7350 2024-09-17T05:21:51.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.631+0000 7ff499153640 1 -- 172.21.15.150:0/3223676296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4940c6690 con 0x7ff4940a7350 2024-09-17T05:21:51.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.631+0000 7ff47b7fe640 1 -- 172.21.15.150:0/3223676296 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff4840bd030 con 0x7ff4940a7350 2024-09-17T05:21:51.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.631+0000 7ff47b7fe640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff480073200 0x7ff480075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:51.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.631+0000 7ff47b7fe640 1 -- 172.21.15.150:0/3223676296 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 9417+0+0 (secure 0 0 0) 0x7ff48412ecf0 con 0x7ff4940a7350 2024-09-17T05:21:51.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.631+0000 7ff49259c640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff480073200 0x7ff480075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:51.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.635+0000 7ff49259c640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff480073200 0x7ff480075710 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff49414f660 tx=0x7ff488009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:51.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.635+0000 7ff47b7fe640 1 -- 172.21.15.150:0/3223676296 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff484133070 con 0x7ff4940a7350 2024-09-17T05:21:51.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.843+0000 7ff499153640 1 -- 172.21.15.150:0/3223676296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff49414a620 con 0x7ff4940a7350 2024-09-17T05:21:51.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.843+0000 7ff47b7fe640 1 -- 172.21.15.150:0/3223676296 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) ==== 95+0+4411 (secure 0 0 0) 0x7ff4840c4060 con 0x7ff4940a7350 2024-09-17T05:21:51.842 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:21:51.842 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":215,"btime":"2024-09-17T05:21:48:986587+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":8571,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3816691969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3816691969},{"type":"v1","addr":"172.21.15.150:6833","nonce":3816691969}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":215}],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:49:40.378290+0000","modified":"2024-09-17T05:20:39.523977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7567},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7567":{"gid":7567,"name":"b","rank":0,"incarnation":210,"state":"up:active","state_seq":12,"addr":"172.21.15.176:6835/1117422294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1117422294},{"type":"v1","addr":"172.21.15.176:6835","nonce":1117422294}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7567,"qdb_cluster":[7567]},"id":2}]} 2024-09-17T05:21:51.842 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 215 2024-09-17T05:21:51.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.843+0000 7ff499153640 1 -- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff480073200 msgr2=0x7ff480075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:51.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.843+0000 7ff499153640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff480073200 0x7ff480075710 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff49414f660 tx=0x7ff488009040 comp rx=0 tx=0).stop 2024-09-17T05:21:51.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.843+0000 7ff499153640 1 -- 172.21.15.150:0/3223676296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4940a7350 msgr2=0x7ff494143920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:51.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.843+0000 7ff499153640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4940a7350 0x7ff494143920 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7ff484019400 tx=0x7ff4840a5720 comp rx=0 tx=0).stop 2024-09-17T05:21:51.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.847+0000 7ff499153640 1 -- 172.21.15.150:0/3223676296 shutdown_connections 2024-09-17T05:21:51.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.847+0000 7ff499153640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff480073200 0x7ff480075710 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.847+0000 7ff499153640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4940c42e0 0x7ff494149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.847+0000 7ff499153640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff494152cf0 0x7ff49413df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.847+0000 7ff499153640 1 --2- 172.21.15.150:0/3223676296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4940a7350 0x7ff494143920 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:51.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.847+0000 7ff499153640 1 -- 172.21.15.150:0/3223676296 >> 172.21.15.150:0/3223676296 conn(0x7ff4940c4a00 msgr2=0x7ff494147c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:51.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.847+0000 7ff499153640 1 -- 172.21.15.150:0/3223676296 shutdown_connections 2024-09-17T05:21:51.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:51.847+0000 7ff499153640 1 -- 172.21.15.150:0/3223676296 wait complete. 2024-09-17T05:21:51.866 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T05:21:52.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.027+0000 7ffa7c978640 1 -- 172.21.15.150:0/2192243668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa780c7cd0 msgr2=0x7ffa780c80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.027+0000 7ffa7c978640 1 --2- 172.21.15.150:0/2192243668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa780c7cd0 0x7ffa780c80e0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009840 tx=0x7ffa7001c750 comp rx=0 tx=0).stop 2024-09-17T05:21:52.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.027+0000 7ffa7c978640 1 -- 172.21.15.150:0/2192243668 shutdown_connections 2024-09-17T05:21:52.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.027+0000 7ffa7c978640 1 --2- 172.21.15.150:0/2192243668 >> v2:172.21.15.176:3300/0 conn(0x7ffa78072ea0 0x7ffa780732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.027+0000 7ffa7c978640 1 --2- 172.21.15.150:0/2192243668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa780c7cd0 0x7ffa780c80e0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.027+0000 7ffa7c978640 1 -- 172.21.15.150:0/2192243668 >> 172.21.15.150:0/2192243668 conn(0x7ffa780abb00 msgr2=0x7ffa780abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:52.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.027+0000 7ffa7c978640 1 -- 172.21.15.150:0/2192243668 shutdown_connections 2024-09-17T05:21:52.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.027+0000 7ffa7c978640 1 -- 172.21.15.150:0/2192243668 wait complete. 2024-09-17T05:21:52.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7c978640 1 Processor -- start 2024-09-17T05:21:52.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7c978640 1 -- start start 2024-09-17T05:21:52.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7c978640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa78072ea0 0x7ffa780ad420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7c978640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa780c7cd0 0x7ffa7814e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7c978640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa780c97a0 0x7ffa78147740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7c978640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffa78161790 con 0x7ffa78072ea0 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7c978640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffa780c4df0 con 0x7ffa780c97a0 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7c978640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ffa78161910 con 0x7ffa780c7cd0 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa78072ea0 0x7ffa780ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa78072ea0 0x7ffa780ad420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51346/0 (socket says 172.21.15.150:51346) 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7659c640 1 -- 172.21.15.150:0/1621525434 learned_addr learned my addr 172.21.15.150:0/1621525434 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa75d9b640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa780c7cd0 0x7ffa7814e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa76d9d640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa780c97a0 0x7ffa78147740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7659c640 1 -- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa780c7cd0 msgr2=0x7ffa7814e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7659c640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa780c7cd0 0x7ffa7814e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7659c640 1 -- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa780c97a0 msgr2=0x7ffa78147740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7659c640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa780c97a0 0x7ffa78147740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7659c640 1 -- 172.21.15.150:0/1621525434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa780ae990 con 0x7ffa78072ea0 2024-09-17T05:21:52.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa75d9b640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa780c7cd0 0x7ffa7814e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:52.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7659c640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa78072ea0 0x7ffa780ad420 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800cb40 tx=0x7ffa6800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:52.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa6effd640 1 -- 172.21.15.150:0/1621525434 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68016020 con 0x7ffa78072ea0 2024-09-17T05:21:52.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa6effd640 1 -- 172.21.15.150:0/1621525434 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7ffa6800aa10 con 0x7ffa78072ea0 2024-09-17T05:21:52.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7c978640 1 -- 172.21.15.150:0/1621525434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa78148980 con 0x7ffa78072ea0 2024-09-17T05:21:52.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa6effd640 1 -- 172.21.15.150:0/1621525434 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68012420 con 0x7ffa78072ea0 2024-09-17T05:21:52.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7c978640 1 -- 172.21.15.150:0/1621525434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa78147fe0 con 0x7ffa78072ea0 2024-09-17T05:21:52.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.031+0000 7ffa7c978640 1 -- 172.21.15.150:0/1621525434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa78148240 con 0x7ffa78072ea0 2024-09-17T05:21:52.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.035+0000 7ffa6effd640 1 -- 172.21.15.150:0/1621525434 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ffa68019020 con 0x7ffa78072ea0 2024-09-17T05:21:52.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.035+0000 7ffa6effd640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa58073200 0x7ffa58075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:52.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.035+0000 7ffa6effd640 1 -- 172.21.15.150:0/1621525434 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 9417+0+0 (secure 0 0 0) 0x7ffa68095320 con 0x7ffa78072ea0 2024-09-17T05:21:52.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.035+0000 7ffa75d9b640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa58073200 0x7ffa58075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:52.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.035+0000 7ffa75d9b640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa58073200 0x7ffa58075710 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ffa60005fd0 tx=0x7ffa60005de0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:52.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.039+0000 7ffa6effd640 1 -- 172.21.15.150:0/1621525434 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ffa68020020 con 0x7ffa78072ea0 2024-09-17T05:21:52.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.215+0000 7ffa7c978640 1 -- 172.21.15.150:0/1621525434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa780732b0 con 0x7ffa78072ea0 2024-09-17T05:21:52.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.223+0000 7ffa6effd640 1 -- 172.21.15.150:0/1621525434 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+22442 (secure 0 0 0) 0x7ffa6805eec0 con 0x7ffa78072ea0 2024-09-17T05:21:52.225 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:21:52.225 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":71,"fsid":"172984ad-e98f-497a-a83c-2299d42da6ea","created":"2024-09-17T04:49:06.064423+0000","modified":"2024-09-17T05:21:50.993583+0000","last_up_change":"2024-09-17T04:49:20.654299+0000","last_in_change":"2024-09-17T04:49:10.281212+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T04:49:20.992684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-09-17T04:49:38.219844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-09-17T04:49:39.346549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e04420c7-04e9-46b3-93e4-f85feeeeb97a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6801","nonce":437062879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6803","nonce":437062879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6807","nonce":437062879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":437062879},{"type":"v1","addr":"172.21.15.150:6805","nonce":437062879}]},"public_addr":"172.21.15.150:6801/437062879","cluster_addr":"172.21.15.150:6803/437062879","heartbeat_back_addr":"172.21.15.150:6807/437062879","heartbeat_front_addr":"172.21.15.150:6805/437062879","state":["exists","up"]},{"osd":1,"uuid":"4f065472-fc2f-4355-9e30-3ca9a6615ba6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6809","nonce":4273227163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6811","nonce":4273227163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6815","nonce":4273227163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":4273227163},{"type":"v1","addr":"172.21.15.150:6813","nonce":4273227163}]},"public_addr":"172.21.15.150:6809/4273227163","cluster_addr":"172.21.15.150:6811/4273227163","heartbeat_back_addr":"172.21.15.150:6815/4273227163","heartbeat_front_addr":"172.21.15.150:6813/4273227163","state":["exists","up"]},{"osd":2,"uuid":"6fc6848e-0c10-4f16-85f3-48fc378d762e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6817","nonce":1456340777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6818","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6819","nonce":1456340777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6823","nonce":1456340777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":1456340777},{"type":"v1","addr":"172.21.15.150:6821","nonce":1456340777}]},"public_addr":"172.21.15.150:6817/1456340777","cluster_addr":"172.21.15.150:6819/1456340777","heartbeat_back_addr":"172.21.15.150:6823/1456340777","heartbeat_front_addr":"172.21.15.150:6821/1456340777","state":["exists","up"]},{"osd":3,"uuid":"50d85cf9-61f7-4476-9f71-3b5fc940fdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6825","nonce":158177050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6827","nonce":158177050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6831","nonce":158177050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":158177050},{"type":"v1","addr":"172.21.15.150:6829","nonce":158177050}]},"public_addr":"172.21.15.150:6825/158177050","cluster_addr":"172.21.15.150:6827/158177050","heartbeat_back_addr":"172.21.15.150:6831/158177050","heartbeat_front_addr":"172.21.15.150:6829/158177050","state":["exists","up"]},{"osd":4,"uuid":"55c7a772-ac2c-4f68-bba2-cf7cbc9c9569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6827","nonce":4247631702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6829","nonce":4247631702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6832","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6833","nonce":4247631702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":4247631702},{"type":"v1","addr":"172.21.15.176:6831","nonce":4247631702}]},"public_addr":"172.21.15.176:6827/4247631702","cluster_addr":"172.21.15.176:6829/4247631702","heartbeat_back_addr":"172.21.15.176:6833/4247631702","heartbeat_front_addr":"172.21.15.176:6831/4247631702","state":["exists","up"]},{"osd":5,"uuid":"62b9fc78-e4c5-4c20-8391-63c4b411d840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6811","nonce":2266081607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6813","nonce":2266081607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6817","nonce":2266081607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2266081607},{"type":"v1","addr":"172.21.15.176:6815","nonce":2266081607}]},"public_addr":"172.21.15.176:6811/2266081607","cluster_addr":"172.21.15.176:6813/2266081607","heartbeat_back_addr":"172.21.15.176:6817/2266081607","heartbeat_front_addr":"172.21.15.176:6815/2266081607","state":["exists","up"]},{"osd":6,"uuid":"142e4b00-3eee-422a-8274-65396c592493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6803","nonce":4084217765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6805","nonce":4084217765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6809","nonce":4084217765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4084217765},{"type":"v1","addr":"172.21.15.176:6807","nonce":4084217765}]},"public_addr":"172.21.15.176:6803/4084217765","cluster_addr":"172.21.15.176:6805/4084217765","heartbeat_back_addr":"172.21.15.176:6809/4084217765","heartbeat_front_addr":"172.21.15.176:6807/4084217765","state":["exists","up"]},{"osd":7,"uuid":"da214a44-feda-404e-b901-920961e6f786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6819","nonce":621582613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6821","nonce":621582613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6825","nonce":621582613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":621582613},{"type":"v1","addr":"172.21.15.176:6823","nonce":621582613}]},"public_addr":"172.21.15.176:6819/621582613","cluster_addr":"172.21.15.176:6821/621582613","heartbeat_back_addr":"172.21.15.176:6825/621582613","heartbeat_front_addr":"172.21.15.176:6823/621582613","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.872830+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.901557+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.285389+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.568861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.471616+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.003436+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:16.806302+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T04:49:17.630383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.1d","mappings":[{"from":4,"to":5}]},{"pgid":"5.23","mappings":[{"from":4,"to":5}]},{"pgid":"5.2a","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6835/1799698362":"2024-09-18T05:21:47.983780+0000","172.21.15.150:6834/1799698362":"2024-09-18T05:21:47.983780+0000","172.21.15.150:6833/2857695220":"2024-09-18T05:20:36.328050+0000","172.21.15.150:6832/2857695220":"2024-09-18T05:20:36.328050+0000","172.21.15.176:6835/1690935336":"2024-09-18T05:19:51.316025+0000","172.21.15.176:6834/1690935336":"2024-09-18T05:19:51.316025+0000","172.21.15.150:6834/3297154462":"2024-09-18T05:18:31.294755+0000","172.21.15.176:6837/1903363960":"2024-09-18T05:18:31.294598+0000","172.21.15.176:6836/1903363960":"2024-09-18T05:18:31.294598+0000","172.21.15.150:6835/3297154462":"2024-09-18T05:18:31.294755+0000","172.21.15.176:6835/524846644":"2024-09-18T05:16:53.685445+0000","172.21.15.150:6833/949083177":"2024-09-18T05:16:51.270750+0000","172.21.15.150:6834/178583496":"2024-09-18T05:15:51.246236+0000","172.21.15.176:6836/2338457617":"2024-09-18T05:15:08.260160+0000","172.21.15.150:6833/3933379792":"2024-09-18T05:15:06.233954+0000","172.21.15.150:6835/3179072684":"2024-09-18T05:14:26.220879+0000","172.21.15.150:6833/12982302":"2024-09-18T05:13:46.203493+0000","172.21.15.176:6837/2962113435":"2024-09-18T05:12:21.179693+0000","172.21.15.150:6835/178583496":"2024-09-18T05:15:51.246236+0000","172.21.15.176:6837/713081612":"2024-09-18T04:59:20.944548+0000","172.21.15.150:6834/2081398022":"2024-09-18T05:00:20.961331+0000","172.21.15.150:6835/3079958649":"2024-09-18T04:58:00.922711+0000","172.21.15.150:6834/3079958649":"2024-09-18T04:58:00.922711+0000","172.21.15.150:6832/12982302":"2024-09-18T05:13:46.203493+0000","172.21.15.150:6832/233311351":"2024-09-18T05:04:06.023230+0000","172.21.15.150:6832/949083177":"2024-09-18T05:16:51.270750+0000","172.21.15.150:6833/284564893":"2024-09-18T04:57:40.911134+0000","172.21.15.176:6837/903190072":"2024-09-18T05:04:36.031676+0000","172.21.15.176:6837/2374314256":"2024-09-18T04:56:10.885556+0000","172.21.15.150:6834/3179072684":"2024-09-18T05:14:26.220879+0000","172.21.15.176:6836/3543629780":"2024-09-18T04:58:30.930118+0000","172.21.15.150:6833/4286123257":"2024-09-18T04:55:40.878553+0000","172.21.15.150:6834/480168496":"2024-09-18T04:51:40.821573+0000","172.21.15.176:6836/1045115232":"2024-09-18T04:53:50.852056+0000","172.21.15.176:6837/1045115232":"2024-09-18T04:53:50.852056+0000","172.21.15.150:6835/285376980":"2024-09-18T05:10:51.151028+0000","172.21.15.150:6832/2313391683":"2024-09-18T05:00:45.968345+0000","172.21.15.150:6835/2081398022":"2024-09-18T05:00:20.961331+0000","172.21.15.150:6832/480168496":"2024-09-18T04:51:40.821573+0000","172.21.15.150:6834/151500360":"2024-09-18T04:54:55.866806+0000","172.21.15.150:6833/2313391683":"2024-09-18T05:00:45.968345+0000","172.21.15.150:6833/1275062894":"2024-09-18T04:58:50.936080+0000","172.21.15.150:6832/1275062894":"2024-09-18T04:58:50.936080+0000","172.21.15.150:6832/961213876":"2024-09-18T04:52:55.838337+0000","172.21.15.150:6835/151500360":"2024-09-18T04:54:55.866806+0000","172.21.15.176:6836/2374314256":"2024-09-18T04:56:10.885556+0000","172.21.15.150:6832/1237450980":"2024-09-18T05:06:31.077239+0000","172.21.15.176:6837/2338457617":"2024-09-18T05:15:08.260160+0000","172.21.15.150:6833/961213876":"2024-09-18T04:52:55.838337+0000","172.21.15.150:6832/3532647866":"2024-09-18T04:55:10.871121+0000","172.21.15.176:6836/713081612":"2024-09-18T04:59:20.944548+0000","172.21.15.176:6837/792092400":"2024-09-18T05:09:46.133255+0000","172.21.15.150:6833/2613040311":"2024-09-18T04:51:25.818032+0000","172.21.15.150:6833/3532647866":"2024-09-18T04:55:10.871121+0000","172.21.15.150:6834/2062728083":"2024-09-18T05:03:01.005221+0000","172.21.15.150:6832/523364553":"2024-09-18T05:07:21.092015+0000","172.21.15.150:6833/233311351":"2024-09-18T05:04:06.023230+0000","172.21.15.176:6836/2962113435":"2024-09-18T05:12:21.179693+0000","172.21.15.150:6832/4286123257":"2024-09-18T04:55:40.878553+0000","172.21.15.176:6836/792092400":"2024-09-18T05:09:46.133255+0000","172.21.15.176:6836/702608997":"2024-09-18T05:02:30.998209+0000","172.21.15.150:6832/284564893":"2024-09-18T04:57:40.911134+0000","172.21.15.150:6833/3622725045":"2024-09-18T05:09:06.121144+0000","172.21.15.150:6835/2062728083":"2024-09-18T05:03:01.005221+0000","172.21.15.176:6836/903190072":"2024-09-18T05:04:36.031676+0000","172.21.15.150:6832/1984876738":"2024-09-18T05:05:21.045156+0000","172.21.15.176:6834/524846644":"2024-09-18T05:16:53.685445+0000","172.21.15.150:6833/1984876738":"2024-09-18T05:05:21.045156+0000","172.21.15.150:6832/3933379792":"2024-09-18T05:15:06.233954+0000","172.21.15.150:6834/3807957687":"2024-09-18T05:05:56.055429+0000","172.21.15.150:6835/3807957687":"2024-09-18T05:05:56.055429+0000","172.21.15.176:6837/3940723333":"2024-09-18T05:06:51.083809+0000","172.21.15.150:6833/1237450980":"2024-09-18T05:06:31.077239+0000","172.21.15.176:6837/3543629780":"2024-09-18T04:58:30.930118+0000","172.21.15.176:6837/702608997":"2024-09-18T05:02:30.998209+0000","172.21.15.150:6835/92695211":"2024-09-18T05:08:16.109918+0000","172.21.15.150:6835/2613040311":"2024-09-18T04:51:25.818032+0000","172.21.15.150:6833/523364553":"2024-09-18T05:07:21.092015+0000","172.21.15.150:6834/92695211":"2024-09-18T05:08:16.109918+0000","172.21.15.150:6832/3622725045":"2024-09-18T05:09:06.121144+0000","172.21.15.176:6836/3940723333":"2024-09-18T05:06:51.083809+0000","172.21.15.150:6834/285376980":"2024-09-18T05:10:51.151028+0000","172.21.15.150:6833/2056467678":"2024-09-18T05:12:01.173633+0000","172.21.15.150:6832/2056467678":"2024-09-18T05:12:01.173633+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T05:21:52.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.231+0000 7ffa7c978640 1 -- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa58073200 msgr2=0x7ffa58075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.231+0000 7ffa7c978640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa58073200 0x7ffa58075710 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ffa60005fd0 tx=0x7ffa60005de0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.231+0000 7ffa7c978640 1 -- 172.21.15.150:0/1621525434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa78072ea0 msgr2=0x7ffa780ad420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.231+0000 7ffa7c978640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa78072ea0 0x7ffa780ad420 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800cb40 tx=0x7ffa6800cc30 comp rx=0 tx=0).stop 2024-09-17T05:21:52.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.235+0000 7ffa7c978640 1 -- 172.21.15.150:0/1621525434 shutdown_connections 2024-09-17T05:21:52.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.235+0000 7ffa7c978640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ffa58073200 0x7ffa58075710 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.235+0000 7ffa7c978640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa780c97a0 0x7ffa78147740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.235+0000 7ffa7c978640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ffa780c7cd0 0x7ffa7814e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.235+0000 7ffa7c978640 1 --2- 172.21.15.150:0/1621525434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa78072ea0 0x7ffa780ad420 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.235+0000 7ffa7c978640 1 -- 172.21.15.150:0/1621525434 >> 172.21.15.150:0/1621525434 conn(0x7ffa780abb00 msgr2=0x7ffa7815e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:52.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.235+0000 7ffa7c978640 1 -- 172.21.15.150:0/1621525434 shutdown_connections 2024-09-17T05:21:52.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.235+0000 7ffa7c978640 1 -- 172.21.15.150:0/1621525434 wait complete. 2024-09-17T05:21:52.290 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail b 2024-09-17T05:21:52.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.475+0000 7fd20bef6640 1 -- 172.21.15.150:0/1022227738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd20406fab0 msgr2=0x7fd20406fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.475+0000 7fd20bef6640 1 --2- 172.21.15.150:0/1022227738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd20406fab0 0x7fd20406fec0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fd204072ef0 tx=0x7fd1fc01c8c0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.475+0000 7fd20bef6640 1 -- 172.21.15.150:0/1022227738 shutdown_connections 2024-09-17T05:21:52.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.475+0000 7fd20bef6640 1 --2- 172.21.15.150:0/1022227738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd20406fab0 0x7fd20406fec0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.475+0000 7fd20bef6640 1 --2- 172.21.15.150:0/1022227738 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2040c41c0 0x7fd2040ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.475+0000 7fd20bef6640 1 --2- 172.21.15.150:0/1022227738 >> v2:172.21.15.176:3300/0 conn(0x7fd2040c3db0 0x7fd2040ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.475+0000 7fd20bef6640 1 -- 172.21.15.150:0/1022227738 >> 172.21.15.150:0/1022227738 conn(0x7fd2040c4a00 msgr2=0x7fd2040c4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:52.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.475+0000 7fd20bef6640 1 -- 172.21.15.150:0/1022227738 shutdown_connections 2024-09-17T05:21:52.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.475+0000 7fd20bef6640 1 -- 172.21.15.150:0/1022227738 wait complete. 2024-09-17T05:21:52.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd20bef6640 1 Processor -- start 2024-09-17T05:21:52.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd20bef6640 1 -- start start 2024-09-17T05:21:52.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd20bef6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd20406fab0 0x7fd2040a9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:52.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd20bef6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2040c3db0 0x7fd20414e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:52.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd20bef6640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2040c41c0 0x7fd20414ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:52.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd20bef6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd2040a7770 con 0x7fd2040c3db0 2024-09-17T05:21:52.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd20bef6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd2040a75f0 con 0x7fd20406fab0 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd20bef6640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7fd204158480 con 0x7fd2040c41c0 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd209491640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2040c3db0 0x7fd20414e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd209491640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2040c3db0 0x7fd20414e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51356/0 (socket says 172.21.15.150:51356) 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd209491640 1 -- 172.21.15.150:0/2967927641 learned_addr learned my addr 172.21.15.150:0/2967927641 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd209c92640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd20406fab0 0x7fd2040a9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd209491640 1 -- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2040c41c0 msgr2=0x7fd20414ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd209491640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2040c41c0 0x7fd20414ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd209491640 1 -- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd20406fab0 msgr2=0x7fd2040a9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd209491640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd20406fab0 0x7fd2040a9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd209491640 1 -- 172.21.15.150:0/2967927641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd204074090 con 0x7fd2040c3db0 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd209491640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2040c3db0 0x7fd20414e0e0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fd1f800c970 tx=0x7fd1f800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd2027fc640 1 -- 172.21.15.150:0/2967927641 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8007bb0 con 0x7fd2040c3db0 2024-09-17T05:21:52.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd2027fc640 1 -- 172.21.15.150:0/2967927641 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7fd1f800f450 con 0x7fd2040c3db0 2024-09-17T05:21:52.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd20bef6640 1 -- 172.21.15.150:0/2967927641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd204074370 con 0x7fd2040c3db0 2024-09-17T05:21:52.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd2027fc640 1 -- 172.21.15.150:0/2967927641 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8007e60 con 0x7fd2040c3db0 2024-09-17T05:21:52.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.479+0000 7fd20bef6640 1 -- 172.21.15.150:0/2967927641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd204150990 con 0x7fd2040c3db0 2024-09-17T05:21:52.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.483+0000 7fd20bef6640 1 -- 172.21.15.150:0/2967927641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd204158a30 con 0x7fd2040c3db0 2024-09-17T05:21:52.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.483+0000 7fd2027fc640 1 -- 172.21.15.150:0/2967927641 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7fd1f8018400 con 0x7fd2040c3db0 2024-09-17T05:21:52.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.483+0000 7fd2027fc640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd1f0073200 0x7fd1f0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:52.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.483+0000 7fd2027fc640 1 -- 172.21.15.150:0/2967927641 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 9417+0+0 (secure 0 0 0) 0x7fd1f80957d0 con 0x7fd2040c3db0 2024-09-17T05:21:52.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.483+0000 7fd209c92640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd1f0073200 0x7fd1f0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:52.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.483+0000 7fd209c92640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd1f0073200 0x7fd1f0075710 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd2040c2500 tx=0x7fd1f4005f50 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:52.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.487+0000 7fd2027fc640 1 -- 172.21.15.150:0/2967927641 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd1f8014030 con 0x7fd2040c3db0 2024-09-17T05:21:52.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.695+0000 7fd20bef6640 1 -- 172.21.15.150:0/2967927641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "b"} v 0) -- 0x7fd204150fb0 con 0x7fd2040c3db0 2024-09-17T05:21:52.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.699+0000 7fd2027fc640 1 -- 172.21.15.150:0/2967927641 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "b"}]=0 b marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v216) ==== 151+0+0 (secure 0 0 0) 0x7fd1f805fa40 con 0x7fd2040c3db0 2024-09-17T05:21:52.699 INFO:teuthology.orchestra.run.smithi150.stderr:b marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T05:21:52.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.703+0000 7fd20bef6640 1 -- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd1f0073200 msgr2=0x7fd1f0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.703+0000 7fd20bef6640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd1f0073200 0x7fd1f0075710 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd2040c2500 tx=0x7fd1f4005f50 comp rx=0 tx=0).stop 2024-09-17T05:21:52.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.703+0000 7fd20bef6640 1 -- 172.21.15.150:0/2967927641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2040c3db0 msgr2=0x7fd20414e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.703+0000 7fd20bef6640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2040c3db0 0x7fd20414e0e0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fd1f800c970 tx=0x7fd1f800ce30 comp rx=0 tx=0).stop 2024-09-17T05:21:52.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.703+0000 7fd20bef6640 1 -- 172.21.15.150:0/2967927641 shutdown_connections 2024-09-17T05:21:52.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.703+0000 7fd20bef6640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7fd1f0073200 0x7fd1f0075710 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.703+0000 7fd20bef6640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7fd2040c41c0 0x7fd20414ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.703+0000 7fd20bef6640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2040c3db0 0x7fd20414e0e0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.703+0000 7fd20bef6640 1 --2- 172.21.15.150:0/2967927641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd20406fab0 0x7fd2040a9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.703+0000 7fd20bef6640 1 -- 172.21.15.150:0/2967927641 >> 172.21.15.150:0/2967927641 conn(0x7fd2040c4a00 msgr2=0x7fd20406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:52.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.703+0000 7fd20bef6640 1 -- 172.21.15.150:0/2967927641 shutdown_connections 2024-09-17T05:21:52.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.703+0000 7fd20bef6640 1 -- 172.21.15.150:0/2967927641 wait complete. 2024-09-17T05:21:52.724 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm b --yes-i-really-mean-it 2024-09-17T05:21:52.818 INFO:tasks.ceph.mds.b.smithi176.stdout:starting mds.b at 2024-09-17T05:21:52.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.883+0000 7f96b6075640 1 -- 172.21.15.150:0/164179204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b01557b0 msgr2=0x7f96b00c4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.883+0000 7f96b6075640 1 --2- 172.21.15.150:0/164179204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b01557b0 0x7f96b00c4990 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f96b00c9e20 tx=0x7f96a001c770 comp rx=0 tx=0).stop 2024-09-17T05:21:52.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.883+0000 7f96b6075640 1 -- 172.21.15.150:0/164179204 shutdown_connections 2024-09-17T05:21:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.883+0000 7f96b6075640 1 --2- 172.21.15.150:0/164179204 >> v2:172.21.15.176:3300/0 conn(0x7f96b00c7e20 0x7f96b0071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.883+0000 7f96b6075640 1 --2- 172.21.15.150:0/164179204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b01557b0 0x7f96b00c4990 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.883+0000 7f96b6075640 1 -- 172.21.15.150:0/164179204 >> 172.21.15.150:0/164179204 conn(0x7f96b00aaa80 msgr2=0x7f96b00aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.883+0000 7f96b6075640 1 -- 172.21.15.150:0/164179204 shutdown_connections 2024-09-17T05:21:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.883+0000 7f96b6075640 1 -- 172.21.15.150:0/164179204 wait complete. 2024-09-17T05:21:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.883+0000 7f96b6075640 1 Processor -- start 2024-09-17T05:21:52.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.883+0000 7f96b6075640 1 -- start start 2024-09-17T05:21:52.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96b6075640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96b00c7580 0x7f96b0155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:52.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96b6075640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96b00c7e20 0x7f96b015d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:52.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96b6075640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b01557b0 0x7f96b014b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96b6075640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f96b0161a40 con 0x7f96b01557b0 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96b6075640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f96b01618c0 con 0x7f96b00c7e20 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96b6075640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f96b0161bc0 con 0x7f96b00c7580 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96affff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b01557b0 0x7f96b014b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96affff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b01557b0 0x7f96b014b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51376/0 (socket says 172.21.15.150:51376) 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96affff640 1 -- 172.21.15.150:0/3214498783 learned_addr learned my addr 172.21.15.150:0/3214498783 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96af7fe640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96b00c7580 0x7f96b0155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96affff640 1 -- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96b00c7580 msgr2=0x7f96b0155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96affff640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96b00c7580 0x7f96b0155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96aeffd640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96b00c7e20 0x7f96b015d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96affff640 1 -- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96b00c7e20 msgr2=0x7f96b015d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96affff640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96b00c7e20 0x7f96b015d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:52.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96affff640 1 -- 172.21.15.150:0/3214498783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b01f3500 con 0x7f96b01557b0 2024-09-17T05:21:52.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96af7fe640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96b00c7580 0x7f96b0155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:52.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96affff640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b01557b0 0x7f96b014b670 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f96a800e9d0 tx=0x7f96a800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:52.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f9697fff640 1 -- 172.21.15.150:0/3214498783 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8004d30 con 0x7f96b01557b0 2024-09-17T05:21:52.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f9697fff640 1 -- 172.21.15.150:0/3214498783 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f96a800cac0 con 0x7f96b01557b0 2024-09-17T05:21:52.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96b6075640 1 -- 172.21.15.150:0/3214498783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b01f37a0 con 0x7f96b01557b0 2024-09-17T05:21:52.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f9697fff640 1 -- 172.21.15.150:0/3214498783 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8005020 con 0x7f96b01557b0 2024-09-17T05:21:52.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96b6075640 1 -- 172.21.15.150:0/3214498783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b01f3d60 con 0x7f96b01557b0 2024-09-17T05:21:52.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f96b6075640 1 -- 172.21.15.150:0/3214498783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b0071b10 con 0x7f96b01557b0 2024-09-17T05:21:52.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.887+0000 7f9697fff640 1 -- 172.21.15.150:0/3214498783 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f96a80051c0 con 0x7f96b01557b0 2024-09-17T05:21:52.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.891+0000 7f9697fff640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f969c073230 0x7f969c075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:52.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.891+0000 7f9697fff640 1 -- 172.21.15.150:0/3214498783 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 9503+0+0 (secure 0 0 0) 0x7f96a8095890 con 0x7f96b01557b0 2024-09-17T05:21:52.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.891+0000 7f96af7fe640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f969c073230 0x7f969c075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:52.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.891+0000 7f96af7fe640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f969c073230 0x7f969c075740 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f96a00af000 tx=0x7f96a0019510 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:52.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:52.895+0000 7f9697fff640 1 -- 172.21.15.150:0/3214498783 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f96a8016030 con 0x7f96b01557b0 2024-09-17T05:21:53.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.103+0000 7f96b6075640 1 -- 172.21.15.150:0/3214498783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "b", "yes_i_really_mean_it": true} v 0) -- 0x7f96b006a770 con 0x7f96b01557b0 2024-09-17T05:21:53.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.703+0000 7f9697fff640 1 -- 172.21.15.150:0/3214498783 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "b", "yes_i_really_mean_it": true}]=0 v217) ==== 99+0+0 (secure 0 0 0) 0x7f96a805fab0 con 0x7f96b01557b0 2024-09-17T05:21:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.707+0000 7f96b6075640 1 -- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f969c073230 msgr2=0x7f969c075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.707+0000 7f96b6075640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f969c073230 0x7f969c075740 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f96a00af000 tx=0x7f96a0019510 comp rx=0 tx=0).stop 2024-09-17T05:21:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.707+0000 7f96b6075640 1 -- 172.21.15.150:0/3214498783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b01557b0 msgr2=0x7f96b014b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.707+0000 7f96b6075640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b01557b0 0x7f96b014b670 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f96a800e9d0 tx=0x7f96a800ee90 comp rx=0 tx=0).stop 2024-09-17T05:21:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.707+0000 7f96b6075640 1 -- 172.21.15.150:0/3214498783 shutdown_connections 2024-09-17T05:21:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.707+0000 7f96b6075640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f969c073230 0x7f969c075740 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.707+0000 7f96b6075640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96b00c7e20 0x7f96b015d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.707+0000 7f96b6075640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96b01557b0 0x7f96b014b670 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.707+0000 7f96b6075640 1 --2- 172.21.15.150:0/3214498783 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f96b00c7580 0x7f96b0155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.707+0000 7f96b6075640 1 -- 172.21.15.150:0/3214498783 >> 172.21.15.150:0/3214498783 conn(0x7f96b00aaa80 msgr2=0x7f96b00c35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.707+0000 7f96b6075640 1 -- 172.21.15.150:0/3214498783 shutdown_connections 2024-09-17T05:21:53.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.707+0000 7f96b6075640 1 -- 172.21.15.150:0/3214498783 wait complete. 2024-09-17T05:21:53.730 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm b_metadata b_metadata --yes-i-really-really-mean-it 2024-09-17T05:21:53.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.899+0000 7f27e3ca3640 1 -- 172.21.15.150:0/4286754502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27c8003a40 msgr2=0x7f27c8003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:53.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.899+0000 7f27e3ca3640 1 --2- 172.21.15.150:0/4286754502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27c8003a40 0x7f27c8003eb0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f27d80016f0 tx=0x7f27d801c7b0 comp rx=0 tx=0).stop 2024-09-17T05:21:53.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.899+0000 7f27e2240640 1 -- 172.21.15.150:0/4286754502 reap_dead start 2024-09-17T05:21:53.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.899+0000 7f27e3ca3640 1 -- 172.21.15.150:0/4286754502 shutdown_connections 2024-09-17T05:21:53.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.899+0000 7f27e3ca3640 1 -- 172.21.15.150:0/4286754502 >> 172.21.15.150:0/4286754502 conn(0x7f27dc0c7fc0 msgr2=0x7f27dc0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:53.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.899+0000 7f27e3ca3640 1 -- 172.21.15.150:0/4286754502 shutdown_connections 2024-09-17T05:21:53.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.899+0000 7f27e3ca3640 1 -- 172.21.15.150:0/4286754502 wait complete. 2024-09-17T05:21:53.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.899+0000 7f27e3ca3640 1 Processor -- start 2024-09-17T05:21:53.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.899+0000 7f27e3ca3640 1 -- start start 2024-09-17T05:21:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e3ca3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27c8003a40 0x7f27dc157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e3ca3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27dc14c810 0x7f27dc1ef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e3ca3640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27dc149d50 0x7f27dc14a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e3ca3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc0c63d0 con 0x7f27dc14c810 2024-09-17T05:21:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e3ca3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f27dc0713a0 con 0x7f27c8003a40 2024-09-17T05:21:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e3ca3640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f27dc0c6550 con 0x7f27dc149d50 2024-09-17T05:21:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e123e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27dc14c810 0x7f27dc1ef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e123e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27dc14c810 0x7f27dc1ef7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51394/0 (socket says 172.21.15.150:51394) 2024-09-17T05:21:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e123e640 1 -- 172.21.15.150:0/784866528 learned_addr learned my addr 172.21.15.150:0/784866528 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e2240640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27dc149d50 0x7f27dc14a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e1a3f640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27c8003a40 0x7f27dc157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e123e640 1 -- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27dc149d50 msgr2=0x7f27dc14a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:53.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e123e640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27dc149d50 0x7f27dc14a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:53.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e123e640 1 -- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27c8003a40 msgr2=0x7f27dc157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:53.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e123e640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27c8003a40 0x7f27dc157a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:53.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e123e640 1 -- 172.21.15.150:0/784866528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27dc0ad2f0 con 0x7f27dc14c810 2024-09-17T05:21:53.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e1a3f640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27c8003a40 0x7f27dc157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T05:21:53.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e123e640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27dc14c810 0x7f27dc1ef7e0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f27cc001ec0 tx=0x7f27cc00ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:53.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27d27fc640 1 -- 172.21.15.150:0/784866528 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc015070 con 0x7f27dc14c810 2024-09-17T05:21:53.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27d27fc640 1 -- 172.21.15.150:0/784866528 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f27cc0109a0 con 0x7f27dc14c810 2024-09-17T05:21:53.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e3ca3640 1 -- 172.21.15.150:0/784866528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27dc14a8b0 con 0x7f27dc14c810 2024-09-17T05:21:53.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27d27fc640 1 -- 172.21.15.150:0/784866528 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc010e10 con 0x7f27dc14c810 2024-09-17T05:21:53.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e3ca3640 1 -- 172.21.15.150:0/784866528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27dc14ad60 con 0x7f27dc14c810 2024-09-17T05:21:53.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27e3ca3640 1 -- 172.21.15.150:0/784866528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27dc14b300 con 0x7f27dc14c810 2024-09-17T05:21:53.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27d27fc640 1 -- 172.21.15.150:0/784866528 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f27cc018070 con 0x7f27dc14c810 2024-09-17T05:21:53.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.903+0000 7f27d27fc640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f27c0073200 0x7f27c0075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:53.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.907+0000 7f27d27fc640 1 -- 172.21.15.150:0/784866528 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 9503+0+0 (secure 0 0 0) 0x7f27cc095520 con 0x7f27dc14c810 2024-09-17T05:21:53.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.907+0000 7f27e1a3f640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f27c0073200 0x7f27c0075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:53.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.907+0000 7f27e1a3f640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f27c0073200 0x7f27c0075710 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f27d80016c0 tx=0x7f27c8005840 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:53.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:53.911+0000 7f27d27fc640 1 -- 172.21.15.150:0/784866528 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f27cc05f6c0 con 0x7f27dc14c810 2024-09-17T05:21:54.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:54.083+0000 7f27e3ca3640 1 -- 172.21.15.150:0/784866528 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "b_metadata", "pool2": "b_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f27dc14c420 con 0x7f27dc14c810 2024-09-17T05:21:55.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.027+0000 7f27d27fc640 1 -- 172.21.15.150:0/784866528 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "b_metadata", "pool2": "b_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'b_metadata' removed v73) ==== 166+0+0 (secure 0 0 0) 0x7f27dc14c420 con 0x7f27dc14c810 2024-09-17T05:21:55.025 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'b_metadata' removed 2024-09-17T05:21:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.031+0000 7f27e3ca3640 1 -- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f27c0073200 msgr2=0x7f27c0075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.031+0000 7f27e3ca3640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f27c0073200 0x7f27c0075710 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f27d80016c0 tx=0x7f27c8005840 comp rx=0 tx=0).stop 2024-09-17T05:21:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.031+0000 7f27e3ca3640 1 -- 172.21.15.150:0/784866528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27dc14c810 msgr2=0x7f27dc1ef7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.031+0000 7f27e3ca3640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27dc14c810 0x7f27dc1ef7e0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f27cc001ec0 tx=0x7f27cc00ea20 comp rx=0 tx=0).stop 2024-09-17T05:21:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.031+0000 7f27e3ca3640 1 -- 172.21.15.150:0/784866528 shutdown_connections 2024-09-17T05:21:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.031+0000 7f27e3ca3640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f27c0073200 0x7f27c0075710 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.031+0000 7f27e3ca3640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f27dc149d50 0x7f27dc14a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.031+0000 7f27e3ca3640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27dc14c810 0x7f27dc1ef7e0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.031+0000 7f27e3ca3640 1 --2- 172.21.15.150:0/784866528 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27c8003a40 0x7f27dc157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.031+0000 7f27e3ca3640 1 -- 172.21.15.150:0/784866528 >> 172.21.15.150:0/784866528 conn(0x7f27dc0c7fc0 msgr2=0x7f27dc0c8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:55.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.031+0000 7f27e3ca3640 1 -- 172.21.15.150:0/784866528 shutdown_connections 2024-09-17T05:21:55.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.031+0000 7f27e3ca3640 1 -- 172.21.15.150:0/784866528 wait complete. 2024-09-17T05:21:55.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm b_data b_data --yes-i-really-really-mean-it 2024-09-17T05:21:55.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.215+0000 7f4c3ee59640 1 -- 172.21.15.150:0/263596653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c380c1490 msgr2=0x7f4c380c8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:55.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.215+0000 7f4c3ee59640 1 --2- 172.21.15.150:0/263596653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c380c1490 0x7f4c380c8d80 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f4c3808dd40 tx=0x7f4c3001cd80 comp rx=0 tx=0).stop 2024-09-17T05:21:55.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.215+0000 7f4c3ee59640 1 -- 172.21.15.150:0/263596653 shutdown_connections 2024-09-17T05:21:55.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.215+0000 7f4c3ee59640 1 --2- 172.21.15.150:0/263596653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c380c1490 0x7f4c380c8d80 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:55.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.215+0000 7f4c3ee59640 1 --2- 172.21.15.150:0/263596653 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c380c1080 0x7f4c380c8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:55.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.215+0000 7f4c3ee59640 1 --2- 172.21.15.150:0/263596653 >> v2:172.21.15.176:3300/0 conn(0x7f4c38150980 0x7f4c380c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:55.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.215+0000 7f4c3ee59640 1 -- 172.21.15.150:0/263596653 >> 172.21.15.150:0/263596653 conn(0x7f4c3806f7d0 msgr2=0x7f4c38074580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:55.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.215+0000 7f4c3ee59640 1 -- 172.21.15.150:0/263596653 shutdown_connections 2024-09-17T05:21:55.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.215+0000 7f4c3ee59640 1 -- 172.21.15.150:0/263596653 wait complete. 2024-09-17T05:21:55.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3ee59640 1 Processor -- start 2024-09-17T05:21:55.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3ee59640 1 -- start start 2024-09-17T05:21:55.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3ee59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c380c1080 0x7f4c380cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:55.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3ee59640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c380c1490 0x7f4c3814b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:55.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3ee59640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c38150980 0x7f4c381ef840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:55.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3ee59640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4c380c7f40 con 0x7f4c380c1080 2024-09-17T05:21:55.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3ee59640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4c380c7dc0 con 0x7f4c380c1490 2024-09-17T05:21:55.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3ee59640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7f4c38075210 con 0x7f4c38150980 2024-09-17T05:21:55.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3cbf5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c380c1080 0x7f4c380cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:55.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3cbf5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c380c1080 0x7f4c380cbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51418/0 (socket says 172.21.15.150:51418) 2024-09-17T05:21:55.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3cbf5640 1 -- 172.21.15.150:0/324619780 learned_addr learned my addr 172.21.15.150:0/324619780 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:55.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3d3f6640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c38150980 0x7f4c381ef840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:55.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c37fff640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c380c1490 0x7f4c3814b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:55.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3cbf5640 1 -- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c38150980 msgr2=0x7f4c381ef840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:55.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3cbf5640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c38150980 0x7f4c381ef840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:55.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3cbf5640 1 -- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c380c1490 msgr2=0x7f4c3814b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:55.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3cbf5640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c380c1490 0x7f4c3814b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:55.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3cbf5640 1 -- 172.21.15.150:0/324619780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c381f0360 con 0x7f4c380c1080 2024-09-17T05:21:55.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3cbf5640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c380c1080 0x7f4c380cbc80 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c00c970 tx=0x7f4c2c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:55.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c357fa640 1 -- 172.21.15.150:0/324619780 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c004220 con 0x7f4c380c1080 2024-09-17T05:21:55.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c357fa640 1 -- 172.21.15.150:0/324619780 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f4c2c007950 con 0x7f4c380c1080 2024-09-17T05:21:55.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3ee59640 1 -- 172.21.15.150:0/324619780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c381f0600 con 0x7f4c380c1080 2024-09-17T05:21:55.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c357fa640 1 -- 172.21.15.150:0/324619780 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c007d60 con 0x7f4c380c1080 2024-09-17T05:21:55.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.219+0000 7f4c3ee59640 1 -- 172.21.15.150:0/324619780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c381538b0 con 0x7f4c380c1080 2024-09-17T05:21:55.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.223+0000 7f4c3ee59640 1 -- 172.21.15.150:0/324619780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c380c76f0 con 0x7f4c380c1080 2024-09-17T05:21:55.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.223+0000 7f4c357fa640 1 -- 172.21.15.150:0/324619780 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7f4c2c003600 con 0x7f4c380c1080 2024-09-17T05:21:55.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.223+0000 7f4c357fa640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4c24073230 0x7f4c24075740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:55.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.223+0000 7f4c357fa640 1 -- 172.21.15.150:0/324619780 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 9075+0+0 (secure 0 0 0) 0x7f4c2c094fa0 con 0x7f4c380c1080 2024-09-17T05:21:55.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.223+0000 7f4c37fff640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4c24073230 0x7f4c24075740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:55.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.223+0000 7f4c37fff640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4c24073230 0x7f4c24075740 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4c38160cf0 tx=0x7f4c28005d00 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:55.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.227+0000 7f4c357fa640 1 -- 172.21.15.150:0/324619780 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4c2c014030 con 0x7f4c380c1080 2024-09-17T05:21:55.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:55.391+0000 7f4c3ee59640 1 -- 172.21.15.150:0/324619780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "b_data", "pool2": "b_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4c381571b0 con 0x7f4c380c1080 2024-09-17T05:21:56.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.035+0000 7f4c357fa640 1 -- 172.21.15.150:0/324619780 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "b_data", "pool2": "b_data", "yes_i_really_really_mean_it": true}]=0 pool 'b_data' removed v74) ==== 154+0+0 (secure 0 0 0) 0x7f4c2c05f2e0 con 0x7f4c380c1080 2024-09-17T05:21:56.035 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'b_data' removed 2024-09-17T05:21:56.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.039+0000 7f4c3ee59640 1 -- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4c24073230 msgr2=0x7f4c24075740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:56.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.039+0000 7f4c3ee59640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4c24073230 0x7f4c24075740 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4c38160cf0 tx=0x7f4c28005d00 comp rx=0 tx=0).stop 2024-09-17T05:21:56.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.039+0000 7f4c3ee59640 1 -- 172.21.15.150:0/324619780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c380c1080 msgr2=0x7f4c380cbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:56.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.039+0000 7f4c3ee59640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c380c1080 0x7f4c380cbc80 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c00c970 tx=0x7f4c2c00ce30 comp rx=0 tx=0).stop 2024-09-17T05:21:56.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.039+0000 7f4c3ee59640 1 -- 172.21.15.150:0/324619780 shutdown_connections 2024-09-17T05:21:56.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.039+0000 7f4c3ee59640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7f4c24073230 0x7f4c24075740 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.039+0000 7f4c3ee59640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c380c1490 0x7f4c3814b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.039+0000 7f4c3ee59640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7f4c38150980 0x7f4c381ef840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.039+0000 7f4c3ee59640 1 --2- 172.21.15.150:0/324619780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c380c1080 0x7f4c380cbc80 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.039+0000 7f4c3ee59640 1 -- 172.21.15.150:0/324619780 >> 172.21.15.150:0/324619780 conn(0x7f4c3806f7d0 msgr2=0x7f4c3814fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:56.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.039+0000 7f4c3ee59640 1 -- 172.21.15.150:0/324619780 shutdown_connections 2024-09-17T05:21:56.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.039+0000 7f4c3ee59640 1 -- 172.21.15.150:0/324619780 wait complete. 2024-09-17T05:21:56.072 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-17T05:21:56.072 ERROR:tasks.ceph.mds.a:tried to stop a non-running daemon 2024-09-17T05:21:56.072 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T05:21:56.072 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:21:56.095 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T05:21:56.095 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T05:21:56.095 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:21:56.127 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T05:21:56.127 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T05:21:56.127 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:21:56.190 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T05:21:56.190 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-17T05:21:56.190 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-17T05:21:56.190 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:21:56.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T05:21:56.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 -- 172.21.15.150:0/2607031935 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff224157ca0 msgr2=0x7ff2241580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:56.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 --2- 172.21.15.150:0/2607031935 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff224157ca0 0x7ff2241580b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009c80 tx=0x7ff21c01cc80 comp rx=0 tx=0).stop 2024-09-17T05:21:56.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 -- 172.21.15.150:0/2607031935 shutdown_connections 2024-09-17T05:21:56.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 --2- 172.21.15.150:0/2607031935 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff224157ca0 0x7ff2241580b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 --2- 172.21.15.150:0/2607031935 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff2240ad7f0 0x7ff2240c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 -- 172.21.15.150:0/2607031935 >> 172.21.15.150:0/2607031935 conn(0x7ff2240c9de0 msgr2=0x7ff2240ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:56.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 -- 172.21.15.150:0/2607031935 shutdown_connections 2024-09-17T05:21:56.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 -- 172.21.15.150:0/2607031935 wait complete. 2024-09-17T05:21:56.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 Processor -- start 2024-09-17T05:21:56.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 -- start start 2024-09-17T05:21:56.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2240ad7f0 0x7ff22414b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:56.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 --2- >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff224151e10 0x7ff22414d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:56.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff224157ca0 0x7ff2241447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:56.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff224158770 con 0x7ff2240ad7f0 2024-09-17T05:21:56.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff2241585f0 con 0x7ff224157ca0 2024-09-17T05:21:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22bc99640 1 -- --> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] -- mon_getmap magic: 0 -- 0x7ff2240c7ce0 con 0x7ff224151e10 2024-09-17T05:21:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff229a35640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2240ad7f0 0x7ff22414b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff229a35640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2240ad7f0 0x7ff22414b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51428/0 (socket says 172.21.15.150:51428) 2024-09-17T05:21:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff229a35640 1 -- 172.21.15.150:0/1518711782 learned_addr learned my addr 172.21.15.150:0/1518711782 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-09-17T05:21:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22a236640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff224157ca0 0x7ff2241447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff229234640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff224151e10 0x7ff22414d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff229a35640 1 -- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff224151e10 msgr2=0x7ff22414d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff229a35640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff224151e10 0x7ff22414d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff229a35640 1 -- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff224157ca0 msgr2=0x7ff2241447c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff229a35640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff224157ca0 0x7ff2241447c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff229a35640 1 -- 172.21.15.150:0/1518711782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22414a9a0 con 0x7ff2240ad7f0 2024-09-17T05:21:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.743+0000 7ff22a236640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff224157ca0 0x7ff2241447c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T05:21:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.747+0000 7ff229a35640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2240ad7f0 0x7ff22414b160 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7ff22000ea20 tx=0x7ff22000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.747+0000 7ff2167fc640 1 -- 172.21.15.150:0/1518711782 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22000cd80 con 0x7ff2240ad7f0 2024-09-17T05:21:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.747+0000 7ff2167fc640 1 -- 172.21.15.150:0/1518711782 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7ff220010430 con 0x7ff2240ad7f0 2024-09-17T05:21:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.747+0000 7ff2167fc640 1 -- 172.21.15.150:0/1518711782 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2200108a0 con 0x7ff2240ad7f0 2024-09-17T05:21:56.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.747+0000 7ff22bc99640 1 -- 172.21.15.150:0/1518711782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff224148fa0 con 0x7ff2240ad7f0 2024-09-17T05:21:56.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.747+0000 7ff22bc99640 1 -- 172.21.15.150:0/1518711782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2241edc10 con 0x7ff2240ad7f0 2024-09-17T05:21:56.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.747+0000 7ff22bc99640 1 -- 172.21.15.150:0/1518711782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2241ede70 con 0x7ff2240ad7f0 2024-09-17T05:21:56.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.747+0000 7ff2167fc640 1 -- 172.21.15.150:0/1518711782 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 97234+0+0 (secure 0 0 0) 0x7ff220010a40 con 0x7ff2240ad7f0 2024-09-17T05:21:56.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.747+0000 7ff2167fc640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff208073200 0x7ff208075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T05:21:56.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.747+0000 7ff2167fc640 1 -- 172.21.15.150:0/1518711782 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 8555+0+0 (secure 0 0 0) 0x7ff220095990 con 0x7ff2240ad7f0 2024-09-17T05:21:56.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.747+0000 7ff229234640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff208073200 0x7ff208075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T05:21:56.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.747+0000 7ff229234640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff208073200 0x7ff208075710 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff2240ab060 tx=0x7ff218009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T05:21:56.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.751+0000 7ff2167fc640 1 -- 172.21.15.150:0/1518711782 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff22005fe50 con 0x7ff2240ad7f0 2024-09-17T05:21:56.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.959+0000 7ff22bc99640 1 -- 172.21.15.150:0/1518711782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff224158250 con 0x7ff2240ad7f0 2024-09-17T05:21:56.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.959+0000 7ff2167fc640 1 -- 172.21.15.150:0/1518711782 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) ==== 95+0+3099 (secure 0 0 0) 0x7ff22005f750 con 0x7ff2240ad7f0 2024-09-17T05:21:56.959 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-09-17T05:21:56.959 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":217,"btime":"2024-09-17T05:21:53:705419+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6837/2843535555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6836","nonce":2843535555},{"type":"v1","addr":"172.21.15.176:6837","nonce":2843535555}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":203},{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.176:6835/1802708095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6834","nonce":1802708095},{"type":"v1","addr":"172.21.15.176:6835","nonce":1802708095}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":217},{"gid":8571,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6833/3816691969","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3816691969},{"type":"v1","addr":"172.21.15.150:6833","nonce":3816691969}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":215}],"filesystems":[]} 2024-09-17T05:21:56.960 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 217 2024-09-17T05:21:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.963+0000 7ff22bc99640 1 -- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff208073200 msgr2=0x7ff208075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.963+0000 7ff22bc99640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff208073200 0x7ff208075710 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff2240ab060 tx=0x7ff218009040 comp rx=0 tx=0).stop 2024-09-17T05:21:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.963+0000 7ff22bc99640 1 -- 172.21.15.150:0/1518711782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2240ad7f0 msgr2=0x7ff22414b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T05:21:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.963+0000 7ff22bc99640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2240ad7f0 0x7ff22414b160 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7ff22000ea20 tx=0x7ff22000eee0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.963+0000 7ff22bc99640 1 -- 172.21.15.150:0/1518711782 shutdown_connections 2024-09-17T05:21:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.963+0000 7ff22bc99640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:6800/882000330,v1:172.21.15.176:6801/882000330] conn(0x7ff208073200 0x7ff208075710 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.963+0000 7ff22bc99640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff224157ca0 0x7ff2241447c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.963+0000 7ff22bc99640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.176:3301/0,v1:172.21.15.176:6790/0] conn(0x7ff224151e10 0x7ff22414d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.963+0000 7ff22bc99640 1 --2- 172.21.15.150:0/1518711782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2240ad7f0 0x7ff22414b160 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T05:21:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.963+0000 7ff22bc99640 1 -- 172.21.15.150:0/1518711782 >> 172.21.15.150:0/1518711782 conn(0x7ff2240c9de0 msgr2=0x7ff2240c6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T05:21:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.963+0000 7ff22bc99640 1 -- 172.21.15.150:0/1518711782 shutdown_connections 2024-09-17T05:21:56.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-09-17T05:21:56.963+0000 7ff22bc99640 1 -- 172.21.15.150:0/1518711782 wait complete. 2024-09-17T05:22:02.292 INFO:tasks.ceph.osd.0:Stopped 2024-09-17T05:22:02.293 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-17T05:22:02.293 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:22:02.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-17T05:22:08.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:22:08.118 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:22:08.395 INFO:tasks.ceph.osd.1:Stopped 2024-09-17T05:22:08.396 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-17T05:22:08.396 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:22:08.483 INFO:tasks.ceph.osd.2:Stopped 2024-09-17T05:22:08.483 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-17T05:22:08.483 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:22:08.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-17T05:22:08.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-17T05:22:14.585 INFO:tasks.ceph.osd.3:Stopped 2024-09-17T05:22:14.586 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-17T05:22:14.586 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:22:14.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-17T05:22:14.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-17T05:22:20.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-17T05:22:20.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-17T05:22:20.688 INFO:tasks.ceph.osd.4:Stopped 2024-09-17T05:22:20.688 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-17T05:22:20.688 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:22:26.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-09-17T05:22:26.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-09-17T05:22:26.790 INFO:tasks.ceph.osd.5:Stopped 2024-09-17T05:22:26.791 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-17T05:22:26.791 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:22:26.878 INFO:tasks.ceph.osd.6:Stopped 2024-09-17T05:22:26.878 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-17T05:22:26.878 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:22:31.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-09-17T05:22:31.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-09-17T05:22:32.980 INFO:tasks.ceph.osd.7:Stopped 2024-09-17T05:22:32.981 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-17T05:22:32.982 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-17T05:22:32.982 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:22:33.017 INFO:tasks.ceph.mgr.y:Stopped 2024-09-17T05:22:33.017 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-17T05:22:33.017 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:22:33.049 INFO:tasks.ceph.mgr.x:Stopped 2024-09-17T05:22:33.049 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-17T05:22:33.049 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T05:22:33.050 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:22:33.080 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T05:22:33.080 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T05:22:33.080 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:22:33.115 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T05:22:33.115 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T05:22:33.115 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T05:22:33.175 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T05:22:33.175 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-17T05:22:33.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | egrep -v 'Replacing daemon mds' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-17T05:22:33.311 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-17T05:22:33.311 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-17T05:22:33.570 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-17T05:22:33.570 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-17T05:22:33.801 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-17T05:22:33.801 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-17T05:22:34.036 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi150.front.sepia.ceph.com 2024-09-17T05:22:34.036 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-17T05:22:34.299 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-17T05:22:34.299 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-17T05:22:34.531 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-17T05:22:34.531 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-17T05:22:34.716 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-17T05:22:34.716 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-17T05:22:34.882 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-17T05:22:34.883 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-17T05:22:35.040 INFO:tasks.ceph:Archiving mon data... 2024-09-17T05:22:35.041 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553/data/mon.a.tgz 2024-09-17T05:22:35.042 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-09-17T05:22:35.064 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.zcLmtf4wCL 2024-09-17T05:22:35.064 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.zcLmtf4wCL 2024-09-17T05:22:35.236 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.zcLmtf4wCL 2024-09-17T05:22:35.333 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.zcLmtf4wCL is 495KB 2024-09-17T05:22:35.398 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.zcLmtf4wCL 2024-09-17T05:22:35.403 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553/data/mon.b.tgz 2024-09-17T05:22:35.403 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-09-17T05:22:35.418 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.shNmCLarDt 2024-09-17T05:22:35.418 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.shNmCLarDt 2024-09-17T05:22:35.592 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.shNmCLarDt 2024-09-17T05:22:35.692 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.shNmCLarDt is 510KB 2024-09-17T05:22:35.753 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.shNmCLarDt 2024-09-17T05:22:35.759 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553/data/mon.c.tgz 2024-09-17T05:22:35.759 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-09-17T05:22:35.808 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.pzNtM52QJB 2024-09-17T05:22:35.808 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.pzNtM52QJB 2024-09-17T05:22:35.971 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.pzNtM52QJB 2024-09-17T05:22:36.035 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.pzNtM52QJB is 510KB 2024-09-17T05:22:36.097 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.pzNtM52QJB 2024-09-17T05:22:36.102 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-17T05:22:36.102 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T05:22:36.105 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T05:22:36.182 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T05:22:36.219 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T05:22:36.219 INFO:tasks.ceph:Archiving crash dumps... 2024-09-17T05:22:36.221 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553/remote/smithi150/crash 2024-09-17T05:22:36.222 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T05:22:36.237 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553/remote/smithi176/crash 2024-09-17T05:22:36.237 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T05:22:36.276 INFO:tasks.ceph:Shutting down logrotate 2024-09-17T05:22:36.276 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:22:36.281 DEBUG:teuthology.orchestra.run.smithi176:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T05:22:36.360 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-17T05:22:36.373 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-17T05:22:36.414 INFO:tasks.ceph:Compressing logs... 2024-09-17T05:22:36.415 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T05:22:36.417 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T05:22:36.435 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51153.log 2024-09-17T05:22:36.436 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42928.log 2024-09-17T05:22:36.436 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50722.log 2024-09-17T05:22:36.436 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51153.log.gz 2024-09-17T05:22:36.437 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37636.log 2024-09-17T05:22:36.438 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82767.log 2024-09-17T05:22:36.438 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.50722.log.gz 2024-09-17T05:22:36.438 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46239.log 2024-09-17T05:22:36.439 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37636.log: gzip/var/log/ceph/ceph-client.admin.82767.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37740.log 2024-09-17T05:22:36.439 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82767.log.gz 2024-09-17T05:22:36.439 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45618.log 2024-09-17T05:22:36.439 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46239.log.gz 2024-09-17T05:22:36.439 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47111.log 2024-09-17T05:22:36.439 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50519.log 2024-09-17T05:22:36.441 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45618.log: /var/log/ceph/ceph-client.admin.47111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83045.log 2024-09-17T05:22:36.442 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37740.log.gz 2024-09-17T05:22:36.442 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50519.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47111.log.gz 2024-09-17T05:22:36.442 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37584.log 2024-09-17T05:22:36.442 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50519.log.gz 2024-09-17T05:22:36.442 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48887.log 2024-09-17T05:22:36.442 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83045.log.gz 2024-09-17T05:22:36.442 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47868.log 2024-09-17T05:22:36.442 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44765.log 2024-09-17T05:22:36.442 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48887.log.gz 2024-09-17T05:22:36.442 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38190.log 2024-09-17T05:22:36.447 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44765.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47868.log.gz 2024-09-17T05:22:36.447 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82967.log 2024-09-17T05:22:36.447 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45386.log 2024-09-17T05:22:36.447 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38190.log: /var/log/ceph/ceph-client.admin.82967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74308.log 2024-09-17T05:22:36.447 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82967.log.gz 2024-09-17T05:22:36.447 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52151.log 2024-09-17T05:22:36.447 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42928.log.gz 2024-09-17T05:22:36.447 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37195.log 2024-09-17T05:22:36.447 INFO:teuthology.orchestra.run.smithi150.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.74308.log.gz 2024-09-17T05:22:36.447 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45200.log 2024-09-17T05:22:36.447 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52151.log.gz 2024-09-17T05:22:36.448 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48421.log 2024-09-17T05:22:36.448 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50064.log 2024-09-17T05:22:36.448 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50748.log 2024-09-17T05:22:36.448 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48421.log.gz 2024-09-17T05:22:36.448 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28911.log 2024-09-17T05:22:36.448 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50064.log.gz 2024-09-17T05:22:36.448 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35641.log 2024-09-17T05:22:36.448 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50748.log.gz 2024-09-17T05:22:36.448 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28911.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-17T05:22:36.448 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49647.log 2024-09-17T05:22:36.448 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35615.log 2024-09-17T05:22:36.478 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74467.log 2024-09-17T05:22:36.478 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49647.log.gz 2024-09-17T05:22:36.478 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50571.log 2024-09-17T05:22:36.478 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48511.log 2024-09-17T05:22:36.478 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74467.log.gz 2024-09-17T05:22:36.478 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51062.log 2024-09-17T05:22:36.478 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50571.log.gz 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43888.log 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48511.log.gz 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46011.log 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51062.log.gz 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37886.log 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43700.log 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46011.log.gz 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46848.log 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36628.log 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35810.log 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46848.log.gz 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35944.log 2024-09-17T05:22:36.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48293.log 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38190.log.gz 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47288.log 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46092.log 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48293.log.gz 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50958.log 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47288.log.gz 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81493.log 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46092.log.gz 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48188.log 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50958.log.gz 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81493.log: gzip -5 --verbose -- 87.2% /var/log/ceph/ceph-client.admin.74173.log -- replaced with /var/log/ceph/ceph-client.admin.45618.log.gz 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81493.log.gz 2024-09-17T05:22:36.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37787.log 2024-09-17T05:22:36.481 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37195.log.gz 2024-09-17T05:22:36.481 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48188.log.gz/var/log/ceph/ceph-client.admin.74173.log: 2024-09-17T05:22:36.481 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35340.log 2024-09-17T05:22:36.481 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74173.log.gz 2024-09-17T05:22:36.481 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37787.log: 78.8% -- replaced with /var/log/ceph/ceph-client.admin.28911.log.gz 2024-09-17T05:22:36.481 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53242.log 2024-09-17T05:22:36.487 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49368.log 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35429.log 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49368.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53242.log.gz 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42186.log 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49368.log.gz 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46934.log 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46639.log 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46980.log 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46934.log.gz 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46639.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46639.log.gz 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42515.log 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46980.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43700.log.gz 2024-09-17T05:22:36.488 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36422.log 2024-09-17T05:22:36.489 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45200.log.gz 2024-09-17T05:22:36.489 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.0.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46980.log.gz 2024-09-17T05:22:36.489 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38095.log 2024-09-17T05:22:36.489 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35340.log.gz 2024-09-17T05:22:36.489 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49591.log 2024-09-17T05:22:36.489 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50365.log 2024-09-17T05:22:36.489 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48115.log 2024-09-17T05:22:36.489 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49591.log.gz 2024-09-17T05:22:36.489 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51179.log 2024-09-17T05:22:36.489 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50365.log.gz 2024-09-17T05:22:36.489 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47389.log 2024-09-17T05:22:36.489 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48115.log: /var/log/ceph/ceph-client.admin.51179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82336.log 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48115.log.gz 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51179.log.gz 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42545.log 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44765.log.gz 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47389.log.gz 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83542.log 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.82336.log.gz 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51565.log 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46874.log 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83542.log.gz 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47085.log 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51565.log.gz 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47258.log 2024-09-17T05:22:36.490 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46874.log.gz 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48459.log 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47085.log.gz 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38014.log 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47258.log.gz 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74282.log 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43888.log.gz 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48459.log.gz 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79583.log 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44501.log 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74282.log.gz 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.79583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83383.log 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.79583.log.gz 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47415.log 2024-09-17T05:22:36.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49342.log 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.83383.log.gz 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35485.log 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47415.log.gz 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45438.log 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36422.log.gz 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49342.log.gz 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48686.log 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36628.log.gz 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41345.log 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41535.log 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48686.log.gz 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49528.log 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9%/var/log/ceph/ceph-client.admin.41535.log: -- replaced with /var/log/ceph/ceph-client.admin.37787.log.gz 2024-09-17T05:22:36.492 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67114.log 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44501.log.gz 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36229.log 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49528.log.gz 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41535.log.gz 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.67114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82032.log 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67114.log.gz 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83253.log 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50932.log 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82032.log.gz 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83568.log 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83253.log.gz 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50932.log: -- /var/log/ceph/ceph-client.admin.49170.log 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83568.log: gzip 87.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.50932.log.gz -- 2024-09-17T05:22:36.493 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.42575.log 2024-09-17T05:22:36.494 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83568.log.gz 2024-09-17T05:22:36.494 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81519.log 2024-09-17T05:22:36.494 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49170.log.gz 2024-09-17T05:22:36.494 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41405.log 2024-09-17T05:22:36.494 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75147.log 2024-09-17T05:22:36.494 INFO:teuthology.orchestra.run.smithi150.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.81519.log.gz 2024-09-17T05:22:36.494 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83019.log 2024-09-17T05:22:36.494 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.75147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74360.log 2024-09-17T05:22:36.494 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83019.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.75147.log.gz 2024-09-17T05:22:36.494 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66270.log 2024-09-17T05:22:36.496 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83019.log.gz 2024-09-17T05:22:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83771.log 2024-09-17T05:22:36.497 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74360.log.gz 2024-09-17T05:22:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.66270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36100.log 2024-09-17T05:22:36.497 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6%/var/log/ceph/ceph-client.admin.83771.log: -- replaced with /var/log/ceph/ceph-client.admin.66270.log.gz 2024-09-17T05:22:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46480.log 2024-09-17T05:22:36.497 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83771.log.gz 2024-09-17T05:22:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48562.log 2024-09-17T05:22:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44631.log 2024-09-17T05:22:36.497 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36100.log.gz 2024-09-17T05:22:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48562.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46480.log.gz 2024-09-17T05:22:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49882.log 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48562.log.gz 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49617.log 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36627.log 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49882.log.gz 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49617.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42515.log.gz 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81676.log 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49617.log.gz 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44631.log.gz 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81931.log 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36627.log: /var/log/ceph/ceph-client.admin.81676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80048.log 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3%/var/log/ceph/ceph-client.admin.81931.log: -- replaced with /var/log/ceph/ceph-client.admin.81676.log.gz 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71353.log 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81931.log.gz 2024-09-17T05:22:36.498 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.80048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46265.log 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80048.log.gz 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35866.log 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.71353.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36627.log.gz 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46183.log 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71353.log.gz 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46265.log.gz 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50261.log 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29507.log 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46183.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38095.log.gz 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37402.log 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46183.log.gz 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50261.log.gz 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.29507.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.70446.log 2024-09-17T05:22:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48507.log 2024-09-17T05:22:36.500 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.70446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44657.log 2024-09-17T05:22:36.500 INFO:teuthology.orchestra.run.smithi150.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.70446.log.gz 2024-09-17T05:22:36.500 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48009.log 2024-09-17T05:22:36.500 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48507.log.gz 2024-09-17T05:22:36.500 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.44657.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.77171.log 2024-09-17T05:22:36.509 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-17T05:22:36.509 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-17T05:22:36.509 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28994.log 2024-09-17T05:22:36.509 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34616.log 2024-09-17T05:22:36.509 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-17T05:22:36.509 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28994.log: /var/log/ceph/ceph-client.1.34616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29049.log 2024-09-17T05:22:36.509 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29146.log 2024-09-17T05:22:36.509 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5/var/log/ceph/ceph-client.admin.29049.log: --verbose -- /var/log/ceph/ceph-client.admin.29025.log 2024-09-17T05:22:36.509 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29146.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-17T05:22:36.509 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29025.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T05:22:36.510 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28970.log 2024-09-17T05:22:36.510 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-17T05:22:36.510 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29122.log 2024-09-17T05:22:36.510 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29097.log 2024-09-17T05:22:36.510 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29122.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T05:22:36.510 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29073.log 2024-09-17T05:22:36.510 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-09-17T05:22:36.510 INFO:teuthology.orchestra.run.smithi176.stderr:gzip/var/log/ceph/ceph-client.admin.29073.log: -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-09-17T05:22:36.510 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-17T05:22:36.510 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-17T05:22:36.518 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.4.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.29146.log.gz 2024-09-17T05:22:36.532 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.28994.log.gz 2024-09-17T05:22:36.532 INFO:teuthology.orchestra.run.smithi176.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.29025.log.gz 2024-09-17T05:22:36.532 INFO:teuthology.orchestra.run.smithi176.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.29049.log.gz 2024-09-17T05:22:36.532 INFO:teuthology.orchestra.run.smithi176.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.29073.log.gz 2024-09-17T05:22:36.532 INFO:teuthology.orchestra.run.smithi176.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.29097.log.gz 2024-09-17T05:22:36.532 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.29122.log.gz 2024-09-17T05:22:36.550 INFO:teuthology.orchestra.run.smithi176.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.28970.log.gz 2024-09-17T05:22:36.575 INFO:teuthology.orchestra.run.smithi176.stderr: 94.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T05:22:36.647 INFO:teuthology.orchestra.run.smithi176.stderr: 93.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T05:22:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37454.log 2024-09-17T05:22:36.677 INFO:teuthology.orchestra.run.smithi150.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.48009.log.gz 2024-09-17T05:22:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.77171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48267.log 2024-09-17T05:22:36.677 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.77171.log.gz 2024-09-17T05:22:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.37454.log: /var/log/ceph/ceph-client.admin.70548.log 2024-09-17T05:22:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44605.log 2024-09-17T05:22:36.677 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48267.log.gz 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.70548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83823.log 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70548.log.gz 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41204.log 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44605.log: /var/log/ceph/ceph-client.admin.83823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80143.log 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83823.log.gz 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82448.log 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.80143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46421.log 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38149.log 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80143.log.gz 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82448.log.gz 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76885.log 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46421.log.gz 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83227.log 2024-09-17T05:22:36.678 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.76885.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41405.log.gz 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51233.log 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76885.log.gz 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42545.log.gz 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29017.log 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83227.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42575.log.gz 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46343.log 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83227.log.gz 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51233.log.gz 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46665.log 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44657.log.gz 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45024.log 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46343.log.gz 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42987.log 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46665.log.gz 2024-09-17T05:22:36.679 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50855.log 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44605.log.gz 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49830.log 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82087.log 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38149.log.gz 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49830.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50855.log.gz 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81415.log 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.29017.log.gz 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45024.log.gz 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42987.log.gz 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49830.log.gz 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29037.log 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82087.log.gz 2024-09-17T05:22:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50035.log 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81415.log.gz 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79886.log 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44972.log 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50035.log.gz 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.29037.log.gzgzip 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36229.log.gz 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.79886.log: /var/log/ceph/ceph-client.admin.44972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83875.log 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79886.log.gz 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35840.log 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83745.log 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83875.log.gz 2024-09-17T05:22:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79916.log 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74250.log 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83745.log.gz 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.79916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37532.log 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44972.log.gz 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74250.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79916.log.gz 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81650.log 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74250.log.gz 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41509.log 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41924.log 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.81650.log.gz 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29081.log 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80113.log 2024-09-17T05:22:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44709.log 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.80113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45592.log 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80113.log.gz 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34746.log 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.45592.log: /var/log/ceph/ceph-client.admin.28875.log 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.34746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83692.log 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43310.log 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35429.log.gz 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35485.log.gz 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44275.log 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83692.log.gz 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39038.log 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82310.log 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41924.log.gz 2024-09-17T05:22:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43598.log 2024-09-17T05:22:36.684 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82420.log 2024-09-17T05:22:36.684 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82310.log.gz 2024-09-17T05:22:36.684 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28863.log 2024-09-17T05:22:36.684 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82368.log 2024-09-17T05:22:36.684 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82420.log.gz 2024-09-17T05:22:36.684 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35589.log 2024-09-17T05:22:36.684 INFO:teuthology.orchestra.run.smithi150.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.29081.log.gz 2024-09-17T05:22:36.684 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45592.log.gz 2024-09-17T05:22:36.684 INFO:teuthology.orchestra.run.smithi150.stderr: 75.4% -- replaced with /var/log/ceph/ceph-client.admin.28875.log.gz 2024-09-17T05:22:36.684 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34746.log.gz 2024-09-17T05:22:36.684 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46729.log 2024-09-17T05:22:36.684 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82368.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43310.log.gz 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82368.log.gz 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44275.log.gz 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39038.log.gz 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr: 75.4% -- replaced with /var/log/ceph/ceph-client.admin.28863.log.gz 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74534.log 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43598.log.gz 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28899.log 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46729.log: /var/log/ceph/ceph-client.admin.35589.log: /var/log/ceph/ceph-client.admin.74534.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46729.log.gz 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74534.log.gz 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47744.log 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mon.a.log: 75.8% -- replaced with /var/log/ceph/ceph-client.admin.28899.log.gz 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45672.log 2024-09-17T05:22:36.685 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50404.log 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47744.log.gz 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45226.log 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42754.log 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45672.log.gz 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50404.log.gz 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29531.log 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83279.log 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45226.log.gz 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44791.log 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37114.log 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.83279.log.gz/var/log/ceph/ceph-client.admin.44791.log: 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42728.log 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46038.log 2024-09-17T05:22:36.686 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47059.log 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.29507.log.gz 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48634.log 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46038.log: /var/log/ceph/ceph-client.admin.47059.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46038.log.gz 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43754.log 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48634.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.29531.log.gz 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47059.log.gz 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35695.log 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48634.log.gz 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44791.log.gz 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29046.log 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43059.log 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37114.log.gz 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35192.log 2024-09-17T05:22:36.687 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35403.log 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45334.log 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35192.log: /var/log/ceph/ceph-client.admin.35403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52311.log 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36074.log 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74830.log 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52311.log.gz 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29089.log 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36074.log: /var/log/ceph/ceph-client.admin.74830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81624.log 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74830.log.gz 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73940.log 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38014.log.gz 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48360.log 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41204.log.gz 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81624.log.gz 2024-09-17T05:22:36.688 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.73940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29676.log 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41345.log.gz 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42186.log.gz 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48360.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73940.log.gz 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44445.log 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48360.log.gz 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44709.log.gz 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43942.log 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45386.log.gz 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82219.log 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45438.log.gz 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48035.log 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80139.log 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82219.log.gz 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41977.log 2024-09-17T05:22:36.689 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42728.log.gz 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48035.log.gz 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.80139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82141.log 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41977.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80139.log.gz 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36428.log 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43754.log.gz 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44098.log 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82141.log.gz 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45750.log 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5/var/log/ceph/ceph-client.admin.44098.log: --verbose -- /var/log/ceph/ceph-client.admin.46291.log 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38040.log 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45142.log 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46291.log.gz 2024-09-17T05:22:36.690 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41587.log 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48966.log 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45142.log: /var/log/ceph/ceph-client.admin.41587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35892.log 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50545.log 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48966.log.gz 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45843.log 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73812.log 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50545.log.gz 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82271.log 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45843.log: /var/log/ceph/ceph-client.admin.73812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48241.log 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73812.log.gz 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83849.log 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.82271.log.gz 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75192.log 2024-09-17T05:22:36.691 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48241.log: /var/log/ceph/ceph-client.admin.83849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83149.log 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48241.log.gz 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.75192.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.83849.log.gz 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50984.log 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.75192.log.gz 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37987.log 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.83149.log.gz 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42149.log 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37987.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50984.log.gz 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74389.log 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36438.log 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43165.log 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7%/var/log/ceph/ceph-client.admin.36438.log: -- replaced with /var/log/ceph/ceph-client.admin.74389.log.gz 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35368.log 2024-09-17T05:22:36.692 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74508.log 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35723.log 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48162.log 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74508.log: /var/log/ceph/ceph-client.admin.35723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83071.log 87.3% 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74508.log.gz 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37140.log 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48162.log.gz 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43676.log 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83071.log.gz 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74682.log 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80193.log 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71203.log 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74682.log.gz 2024-09-17T05:22:36.693 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.80193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54375.log 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80193.log.gz 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.71203.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.29046.log.gz 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47230.log 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71203.log.gz 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48638.log 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54375.log: /var/log/ceph/ceph-client.admin.47230.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54375.log.gz 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53858.log 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47230.log.gz 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46213.log 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48638.log.gz 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.53858.log: /var/log/ceph/ceph-client.admin.42108.log 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46213.log: 87.4% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53858.log.gz 2024-09-17T05:22:36.694 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43059.log.gz 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36227.log 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35192.log.gz 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42108.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46213.log.gz 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29085.log 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% 87.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.36074.log.gz -- replaced with /var/log/ceph/ceph-client.admin.35403.log.gz 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr: -5 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29088.log 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36227.log: /var/log/ceph/ceph-client.admin.29085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82474.log 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81441.log 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42190.log 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.82474.log.gz 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35667.log 2024-09-17T05:22:36.695 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81441.log.gz 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45360.log 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45873.log 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45334.log.gz 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44393.log 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.45873.log: /var/log/ceph/ceph-client.admin.50597.log 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48214.log 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48803.log 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50597.log.gz 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80074.log 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48214.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.29089.log.gz 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44445.log.gz 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44946.log 2024-09-17T05:22:36.696 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48214.log.gz 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.80074.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48803.log.gz 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43942.log.gz 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74415.log 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.80074.log.gz 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35996.log 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41691.log 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35996.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74415.log.gz 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37480.log 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41691.log: gzip 87.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.36428.log.gz -- 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.47654.log 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37480.log: gzip -5 --verbose 87.4% -- -- replaced with /var/log/ceph/ceph-client.admin.44098.log.gz /var/log/ceph/ceph-client.admin.71229.log 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-17T05:22:36.697 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.47654.log: /var/log/ceph/ceph-client.admin.41434.log 2024-09-17T05:22:36.698 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47654.log.gz 2024-09-17T05:22:36.698 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.71229.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35368.log.gz 2024-09-17T05:22:36.700 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41587.log.gz 2024-09-17T05:22:36.700 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28923.log 2024-09-17T05:22:36.700 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43676.log.gz 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.71229.log.gz 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42448.log 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36438.log.gz 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45750.log.gz 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28923.log: /var/log/ceph/ceph-client.admin.42448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41483.log 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36232.log 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35996.log.gz 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44301.log 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.37140.log.gz 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29070.log 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43083.log 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43916.log 2024-09-17T05:22:36.701 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.43083.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.80022.log 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38040.log.gz 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44820.log 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.80022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36230.log 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80022.log.gz 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29074.log 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44820.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.29676.log.gz 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44069.log 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35589.log.gz 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49038.log 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29074.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35615.log.gz 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82993.log 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35641.log.gz 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49142.log 2024-09-17T05:22:36.702 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35667.log.gz 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49038.log: /var/log/ceph/ceph-client.admin.82993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82394.log 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49038.log.gz 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35695.log.gz 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35723.log.gz 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49142.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.82993.log.gz 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83797.log 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49142.log.gz 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69521.log 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.82394.log.gz 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36022.log 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83797.log: /var/log/ceph/ceph-client.admin.69521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36152.log 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83797.log.gz 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69521.log.gz 2024-09-17T05:22:36.703 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36620.log 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36022.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42190.log.gz 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46118.log 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44739.log 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45698.log 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46118.log.gz 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36854.log 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40100.log 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37506.log 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44144.log 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42958.log 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66686.log 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80943.log 2024-09-17T05:22:36.704 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.66686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47357.log 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45360.log.gz 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66686.log.gz 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.80943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44170.log 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.80943.log.gz 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43165.log.gz 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83438.log 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47357.log: /var/log/ceph/ceph-client.admin.44170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51259.log 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47357.log.gz 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45282.log 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.83438.log.gz 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50800.log 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51259.log.gz 2024-09-17T05:22:36.705 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79970.log 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76666.log 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50800.log.gz 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.79970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49778.log 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79970.log.gz 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.76666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76911.log 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.76666.log.gz 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82605.log 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49778.log.gz 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.76911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79996.log 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41434.log.gz 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76911.log.gz 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49554.log 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82605.log.gz 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr: 81.2% -- replaced with /var/log/ceph/ceph-client.admin.28923.log.gz 2024-09-17T05:22:36.706 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.79996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41814.log 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49554.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.79996.log.gz 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29087.log 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41814.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49554.log.gz 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41730.log 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41665.log 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71482.log 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41665.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.29085.log.gz 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37636.log.gz 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.71482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81871.log 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71482.log.gz 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.29074.log.gz 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36984.log 2024-09-17T05:22:36.707 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.1.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42448.log.gz 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82527.log 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6%/var/log/ceph/ceph-client.admin.36984.log: -- replaced with /var/log/ceph/ceph-client.admin.81871.log.gz 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44820.log.gz 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50826.log 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36232.log.gz 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42422.log 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36227.log.gz 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82527.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36230.log.gz 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50826.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.82527.log.gz 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49012.log 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44069.log.gz 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44144.log.gz 2024-09-17T05:22:36.708 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50826.log.gz 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81389.log 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45873.log.gz 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.37323.log 87.2% 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36620.log.gz 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49012.log.gz 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47163.log 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.36854.log.gz 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81389.log.gz 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44301.log.gz 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.0.37323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36231.log 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47032.log 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47163.log.gz 2024-09-17T05:22:36.709 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36932.log 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44393.log.gz 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45282.log.gz 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47032.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.24013.log 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47032.log.gz 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74334.log 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.tmp-client.admin.24013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45566.log 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.24013.log.gz 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81572.log 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4%/var/log/ceph/ceph-client.admin.45566.log: -- replaced with /var/log/ceph/ceph-client.admin.74334.log.gz 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45985.log 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74441.log 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81572.log.gz 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37428.log 2024-09-17T05:22:36.710 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45985.log.gz 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36420.log 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74441.log.gz 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40228.log 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43284.log 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82915.log 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.43284.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42835.log 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36022.log.gz 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49856.log 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.82915.log.gz 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50488.log 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65828.log 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49856.log: /var/log/ceph/ceph-client.admin.50488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76978.log 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49856.log.gz 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50488.log.gz 2024-09-17T05:22:36.711 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.65828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45776.log 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65828.log.gz 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41230.log 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.76978.log: /var/log/ceph/ceph-client.admin.45776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37373.log 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76978.log.gz 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47315.log 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36984.log.gz 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43572.log 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46703.log 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43572.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47315.log.gz 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45115.log 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71297.log 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46703.log.gz 2024-09-17T05:22:36.712 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35286.log 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.71297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42030.log 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71297.log.gz 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44249.log 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50207.log 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71256.log 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65244.log 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50207.log.gz 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.71256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79846.log 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41814.log.gz 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71256.log.gz 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.65244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49752.log 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43083.log.gz 2024-09-17T05:22:36.713 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.65244.log.gz 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50005.log 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.79846.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.79846.log.gz 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44475.log 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50005.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49752.log.gz 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81793.log 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50005.log.gz 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49973.log 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42861.log 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81793.log.gz 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82500.log 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49973.log.gz 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76937.log 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43916.log.gz 2024-09-17T05:22:36.714 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42958.log.gz 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82500.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42835.log.gzgzip 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29071.log 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.82500.log.gz 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.76937.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43284.log.gz 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35511.log 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.29087.log.gz 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29071.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76937.log.gz 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45698.log.gz 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51317.log/var/log/ceph/ceph-client.admin.35511.log: 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36152.log.gz 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35970.log 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46506.log 2024-09-17T05:22:36.715 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51317.log.gz 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82579.log 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35970.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36231.log.gz 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44170.log.gz 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81545.log 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44249.log.gz 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46506.log.gz 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47972.log 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35286.log.gz 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82579.log.gz 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36648.log 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37373.log.gz 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35511.log.gz 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81545.log.gz 2024-09-17T05:22:36.716 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28826.log 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36420.log.gz 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47972.log.gz 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43572.log.gz 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40442.log 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42240.log 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28826.log.gz 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49723.log 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42034.log 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46532.log 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49723.log.gz 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42676.log 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51205.log 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46532.log.gz 2024-09-17T05:22:36.717 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42370.log 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.51205.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37610.log 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82737.log 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51205.log.gz 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70952.log 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41840.log 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.82737.log.gz 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41256.log 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.70952.log: /var/log/ceph/ceph-client.admin.41840.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70952.log.gzgzip 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68182.log 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42344.log 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.68182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83901.log 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42344.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.68182.log.gz 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45050.log 2024-09-17T05:22:36.718 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81903.log 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.83901.log.gz 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.45050.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36421.log 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81903.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36421.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81903.log.gz 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47519.log 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36639.log 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43440.log 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37584.log.gz 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36639.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47519.log.gz 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41758.log 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44683.log 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83097.log 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45817.log 2024-09-17T05:22:36.719 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43650.log 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45817.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.83097.log.gzgzip 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80722.log 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38545.log 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.80722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83331.log 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80722.log.gz 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83620.log 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48397.log 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83620.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83331.log.gz 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47493.log 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83620.log.gz 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37714.log 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48397.log.gz 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.47493.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43826.log 2024-09-17T05:22:36.720 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37714.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47493.log.gz 2024-09-17T05:22:36.721 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36219.log 2024-09-17T05:22:36.721 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43492.log 2024-09-17T05:22:36.721 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T05:22:36.721 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36126.log 2024-09-17T05:22:36.721 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47790.log 2024-09-17T05:22:36.721 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43340.log 2024-09-17T05:22:36.721 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41146.log 2024-09-17T05:22:36.721 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47790.log.gz 2024-09-17T05:22:36.721 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54636.log 2024-09-17T05:22:36.721 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45646.log 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71125.log 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54636.log.gz 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47467.log 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.71125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49196.log 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.71125.log.gz 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47545.log 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49196.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47467.log.gz 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39331.log 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35810.log.gz 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35840.log.gz 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35866.log.gz/var/log/ceph/ceph-client.admin.47545.log: 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36958.log 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49196.log.gz 2024-09-17T05:22:36.722 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47545.log.gz 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49090.log 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50432.log 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35784.log 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35892.log.gz 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50432.log: gzip 87.0% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.35944.log.gz /var/log/ceph/ceph-client.admin.35260.log 87.6% 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49090.log.gz 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50432.log.gz 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41951.log 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37402.log.gz 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42082.log 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35970.log.gz 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.41951.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38217.log 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37428.log.gz 2024-09-17T05:22:36.723 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35563.log 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42082.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37454.log.gz 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77004.log 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37506.log.gz 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37480.log.gz 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82825.log 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37532.log.gz 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.77004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42621.log 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37610.log.gz 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.77004.log.gz 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35455.log 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82825.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37714.log.gz 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50110.log 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82825.log.gz 2024-09-17T05:22:36.724 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49225.log 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41178.log 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50110.log.gz 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82941.log 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49225.log.gz 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47597.log 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41178.log: /var/log/ceph/ceph-client.admin.82941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50462.log 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36958.log.gz 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36932.log.gz 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.82941.log.gz 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82683.log 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47597.log.gz 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49064.log 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82683.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50462.log.gz 2024-09-17T05:22:36.725 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37935.log 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82683.log.gz 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.49064.log: /var/log/ceph/ceph-client.admin.46395.log 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41230.log.gz 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49064.log.gz 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43382.log 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41256.log.gz 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45465.log 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42240.log.gz 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42676.log.gz 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43382.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46395.log.gz 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44505.log 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44683.log.gz 2024-09-17T05:22:36.726 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81363.log 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.29071.log.gz 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81986.log 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40442.log.gz 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43492.log.gz 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79944.log 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.81363.log.gz 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36126.log.gz 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36219.log.gz 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38069.log 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.79944.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81986.log.gz 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51757.log 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35260.log.gz 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79944.log.gz 2024-09-17T05:22:36.727 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36048.log 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43650.log.gz 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37062.log 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51757.log.gz 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36440.log 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42783.log 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36647.log 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47571.log 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82061.log 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28820.log 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47571.log.gz 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48585.log 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82061.log.gz 2024-09-17T05:22:36.728 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43139.log 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48738.log 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48585.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35455.log.gz 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36640.log 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48585.log.gz 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53721.log 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48738.log.gz 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43624.log 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50339.log 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53721.log.gz 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46811.log 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82657.log 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50339.log.gz 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36226.log 2024-09-17T05:22:36.729 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46811.log.gz 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51036.log 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82657.log.gz 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67703.log 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36421.log.gz 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81767.log 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51036.log.gz 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.67703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29080.log 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81767.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67703.log.gz 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46785.log 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81767.log.gz 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45520.log 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71901.log 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46785.log.gz 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44894.log 2024-09-17T05:22:36.730 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.71901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39513.log 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71901.log.gz 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36440.log.gz 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74203.log 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.39513.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36639.log.gz 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35312.log 87.6% 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36640.log.gz 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36648.log.gz 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36647.log.gz 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76833.log 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41951.log.gz 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44505.log.gz 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.74203.log.gz 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47137.log 2024-09-17T05:22:36.731 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.76833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40648.log 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76833.log.gz 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35918.log 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47137.log.gz 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49280.log 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44998.log 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47816.log 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49280.log.gz 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44328.log 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81819.log 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45170.log 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47816.log.gz 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41371.log 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81819.log.gz 2024-09-17T05:22:36.732 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45170.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49804.log 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29084.log 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49804.log.gz 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36906.log 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46317.log 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45412.log 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43191.log 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46317.log.gz 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29073.log 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50696.log 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29073.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41561.log 2024-09-17T05:22:36.733 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50696.log.gz 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42396.log 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51088.log 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38829.log 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42003.log 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51088.log.gz 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44196.log 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29086.log 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47894.log 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37012.log 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46369.log 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47894.log.gz 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81957.log 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77815.log 2024-09-17T05:22:36.734 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81957.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46369.log.gz 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49251.log 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.77815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48857.log 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81957.log.gz 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77815.log.gz 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48712.log 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49251.log.gz 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.48857.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49677.log 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48712.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46066.log 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48857.log.gz 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48712.log.gz 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80271.log 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46066.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49677.log.gz 2024-09-17T05:22:36.735 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65373.log 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46066.log.gz 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.80271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82167.log 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80271.log.gz 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.65373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82245.log 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82167.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65373.log.gz 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45491.log 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82167.log.gz 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83409.log 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82245.log.gz 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83666.log 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43728.log 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83409.log.gz 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42902.log 2024-09-17T05:22:36.736 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.83666.log.gz 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83719.log 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43728.log: /var/log/ceph/ceph-client.admin.42902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48065.log 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47680.log 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83719.log.gz 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47191.log 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48065.log.gz 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49502.log 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47191.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47680.log.gz 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50774.log 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.47191.log.gz 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83490.log 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49502.log.gz 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51010.log 2024-09-17T05:22:36.737 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83490.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50774.log.gz 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44868.log 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.83490.log.gz 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41872.log 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.51010.log.gz 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42489.log 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81598.log 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49424.log 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83305.log 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81598.log.gz 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44553.log 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49424.log.gz 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42702.log 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44553.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83305.log.gz 2024-09-17T05:22:36.738 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38121.log 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr: 95.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50670.log 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41613.log 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.38121.log: /var/log/ceph/ceph-client.admin.50670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47628.log 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50670.log.gz 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45308.log 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48447.log 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45308.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47628.log.gz 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49908.log 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43033.log 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48447.log.gz/var/log/ceph/ceph-client.admin.49908.log: 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36880.log 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49908.log.gz 2024-09-17T05:22:36.739 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49936.log 2024-09-17T05:22:36.740 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29083.log 2024-09-17T05:22:36.740 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81467.log 2024-09-17T05:22:36.740 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79697.log 2024-09-17T05:22:36.740 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49936.log.gz 2024-09-17T05:22:36.740 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83201.log 2024-09-17T05:22:36.740 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3%/var/log/ceph/ceph-client.admin.79697.log: -- replaced with /var/log/ceph/ceph-client.admin.81467.log.gz 2024-09-17T05:22:36.740 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50287.log 2024-09-17T05:22:36.740 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79697.log.gz 2024-09-17T05:22:36.740 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50181.log 2024-09-17T05:22:36.740 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83201.log.gz 2024-09-17T05:22:36.740 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82711.log 2024-09-17T05:22:36.740 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50287.log.gz 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82193.log 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50181.log.gz 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35537.log 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82711.log.gz 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28993.log 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82193.log.gz 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37813.log 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83516.log 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49116.log 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37688.log 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.83516.log.gz 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49476.log 2024-09-17T05:22:36.741 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49116.log.gz 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50155.log 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82794.log 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49476.log.gz 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37088.log 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50155.log.gz 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41282.log 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82794.log.gz 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47006.log 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44946.log.gz 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47920.log 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47441.log 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47006.log.gz 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44920.log 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47920.log.gz 2024-09-17T05:22:36.742 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83357.log 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47441.log.gz 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45076.log 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52694.log 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83357.log.gz 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71177.log 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43970.log 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52694.log.gz 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.71177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76535.log 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71177.log.gz 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45724.log 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.76535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51291.log 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.76535.log.gz 2024-09-17T05:22:36.743 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51343.log 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46157.log 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51291.log.gz 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46755.log 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.51343.log.gz 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36439.log 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46157.log.gz 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50313.log 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46755.log.gz 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45959.log 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43783.log 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50313.log.gz 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80573.log 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43783.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45959.log.gz 2024-09-17T05:22:36.744 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45115.log.gz 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45142.log.gz 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45776.log.gz 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.80573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46448.log 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.80573.log.gz 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45817.log.gz 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81845.log 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45843.log.gz 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42298.log 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46448.log: /var/log/ceph/ceph-client.admin.81845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74654.log 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46448.log.gz 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81845.log.gz 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42298.log: /var/log/ceph/ceph-client.admin.42650.log 2024-09-17T05:22:36.745 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45646.log.gz 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35784.log.gz 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50233.log 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74654.log.gz 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47842.log 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42809.log 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50233.log.gz 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48764.log 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35918.log.gz 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47842.log.gz 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35563.log.gz 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36215.log 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42809.log: /var/log/ceph/ceph-client.admin.48764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49396.log 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48764.log.gz 2024-09-17T05:22:36.746 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80197.log 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45256.log 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49396.log.gz 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36048.log.gz 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.80197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82113.log 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80197.log.gz 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82553.log 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35159.log 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82113.log.gz 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41898.log 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82553.log.gz 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37961.log 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41788.log 2024-09-17T05:22:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43520.log 2024-09-17T05:22:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44419.log 2024-09-17T05:22:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46561.log 2024-09-17T05:22:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41308.log 2024-09-17T05:22:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36425.log 2024-09-17T05:22:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41308.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46561.log.gz 2024-09-17T05:22:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37558.log 2024-09-17T05:22:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43864.log 2024-09-17T05:22:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36425.log: /var/log/ceph/ceph-client.admin.37558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71323.log 2024-09-17T05:22:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43221.log 2024-09-17T05:22:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36178.log 2024-09-17T05:22:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.71323.log: /var/log/ceph/ceph-client.admin.43221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83123.log 2024-09-17T05:22:36.748 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71323.log.gz 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42244.log 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50903.log 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42244.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83123.log.gz 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44039.log 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35758.log 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50903.log: /var/log/ceph/ceph-client.admin.44039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28935.log 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50903.log.gz 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.35758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43546.log 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29584.log 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83466.log 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.29584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51114.log 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76859.log 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37987.log.gz 2024-09-17T05:22:36.749 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38545.log.gz 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.83466.log.gz 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39331.log.gz 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39513.log.gz 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38217.log.gz 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82851.log 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40100.log.gz 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40228.log.gz 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51114.log.gz 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41665.log.gz 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41730.log.gz/var/log/ceph/ceph-client.admin.82851.log: 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41758.log.gz 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41691.log.gz 2024-09-17T05:22:36.750 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.82631.log 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.76859.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82851.log.gz 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74560.log 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76859.log.gz 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41977.log.gz 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42030.log.gz 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82631.log: gzip -5 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82631.log.gz 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42034.log.gz -- replaced with /var/log/ceph/ceph-client.admin.42082.log.gz 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.74560.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42108.log.gz 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74560.log.gz 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr: -- /var/log/ceph/ceph-client.admin.67535.log 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47946.log 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.67535.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42149.log.gz 2024-09-17T05:22:36.751 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42422.log.gz 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67535.log.gz 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83175.log 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42754.log.gz 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81728.log 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42783.log.gz 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47946.log.gz 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36828.log 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42861.log.gz 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83175.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43826.log.gz 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81728.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44328.log.gz 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37662.log 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.83175.log.gz 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81728.log.gz 2024-09-17T05:22:36.752 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44739.log.gz 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36828.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45050.log.gz 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41639.log 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41060.log 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44222.log 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49450.log 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37062.log.gz 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44222.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37088.log.gz 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36638.log 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.37012.log.gz 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28887.log 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49450.log.gz 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.36638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48829.log 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41178.log.gz 2024-09-17T05:22:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.28887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45902.log 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42621.log.gz 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42650.log.gz 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42244.log.gz 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44557.log 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48829.log.gz 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43247.log 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44557.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45902.log.gz 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46904.log 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43111.log 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44013.log 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5%/var/log/ceph/ceph-client.admin.43111.log: -- replaced with /var/log/ceph/ceph-client.admin.46904.log.gz 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42134.log 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83594.log 2024-09-17T05:22:36.754 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44367.log 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.83594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43466.log 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.83594.log.gz 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82889.log 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81702.log 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.82889.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.82889.log.gz 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.81702.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81702.log.gz 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44920.log.gz 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45465.log.gz 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42702.log.gz 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41898.log.gz 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42344.log.gz 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42298.log.gz 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42370.log.gz 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42396.log.gz 2024-09-17T05:22:36.755 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43139.log.gz 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42809.log.gz 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% 78.9% -- replaced with /var/log/ceph/ceph-client.admin.43440.log.gz -- replaced with /var/log/ceph/ceph-client.admin.28820.log.gz 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.29080.log.gz 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.29086.log.gz 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40648.log.gz 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43546.log.gz -- replaced with /var/log/ceph/ceph-client.admin.43520.log.gz 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.36215.log.gz 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36226.log.gz 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41146.log.gz 87.8% 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44039.log.gz 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44222.log.gz 2024-09-17T05:22:36.756 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45076.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44998.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35312.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41282.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41613.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43191.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43624.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43728.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45170.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36638.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.36828.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36880.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36906.log.gz 2024-09-17T05:22:36.757 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38121.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44013.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43970.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44419.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45256.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45308.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37961.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38829.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41639.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43783.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37558.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37688.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37662.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35159.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35758.log.gz 2024-09-17T05:22:36.758 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41371.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41840.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41872.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42902.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43033.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41561.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 78.0% -- replaced with /var/log/ceph/ceph-client.admin.28887.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43247.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28993.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.28935.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.29084.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.29083.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.29073.log.gz 2024-09-17T05:22:36.759 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42489.log.gz 2024-09-17T05:22:36.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-09-17T05:22:36.762 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36178.log.gz 2024-09-17T05:22:36.762 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41060.log.gz 2024-09-17T05:22:36.762 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44196.log.gz 2024-09-17T05:22:36.762 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43221.log.gz 2024-09-17T05:22:36.762 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35537.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36439.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37813.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44553.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44557.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42003.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42134.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45412.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41788.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43111.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36425.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44367.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45724.log.gz 2024-09-17T05:22:36.763 INFO:teuthology.orchestra.run.smithi150.stderr: 86.9% 86.8% -- replaced with /var/log/ceph/ceph-client.admin.29070.log.gz -- replaced with /var/log/ceph/ceph-client.admin.29088.log.gz 2024-09-17T05:22:36.764 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-17T05:22:36.764 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43864.log.gz 2024-09-17T05:22:36.764 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41308.log.gz 2024-09-17T05:22:36.764 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43466.log.gz 2024-09-17T05:22:36.764 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44868.log.gz 2024-09-17T05:22:36.764 INFO:teuthology.orchestra.run.smithi150.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.29584.log.gz 2024-09-17T05:22:36.860 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38069.log.gz 2024-09-17T05:22:36.861 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43382.log.gz 2024-09-17T05:22:36.861 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43340.log.gz 2024-09-17T05:22:36.861 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44894.log.gz 2024-09-17T05:22:37.167 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37886.log.gz 2024-09-17T05:22:37.168 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37935.log.gz 2024-09-17T05:22:37.193 INFO:teuthology.orchestra.run.smithi150.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41483.log.gz 2024-09-17T05:22:37.193 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41509.log.gz 2024-09-17T05:22:37.195 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45491.log.gz 2024-09-17T05:22:37.195 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45520.log.gz 2024-09-17T05:22:37.195 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45566.log.gz 2024-09-17T05:22:37.263 INFO:teuthology.orchestra.run.smithi150.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T05:22:38.616 INFO:teuthology.orchestra.run.smithi176.stderr: 94.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-09-17T05:22:38.644 INFO:teuthology.orchestra.run.smithi176.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-17T05:22:40.755 INFO:teuthology.orchestra.run.smithi176.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-17T05:22:41.764 INFO:teuthology.orchestra.run.smithi176.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-17T05:22:44.981 INFO:teuthology.orchestra.run.smithi150.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-17T05:22:57.714 INFO:teuthology.orchestra.run.smithi176.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-17T05:22:58.961 INFO:teuthology.orchestra.run.smithi150.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-17T05:23:01.169 INFO:teuthology.orchestra.run.smithi176.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-17T05:23:01.668 INFO:teuthology.orchestra.run.smithi176.stderr: 94.1% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-09-17T05:23:04.381 INFO:teuthology.orchestra.run.smithi150.stderr: 94.4% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-09-17T05:23:06.447 INFO:teuthology.orchestra.run.smithi176.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-17T05:23:10.662 INFO:teuthology.orchestra.run.smithi150.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-17T05:23:10.963 INFO:teuthology.orchestra.run.smithi176.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-17T05:23:11.833 INFO:teuthology.orchestra.run.smithi150.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-17T05:23:13.178 INFO:teuthology.orchestra.run.smithi150.stderr: 94.3% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-09-17T05:23:20.200 INFO:teuthology.orchestra.run.smithi150.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-17T05:25:09.854 INFO:teuthology.orchestra.run.smithi176.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.1.34616.log.gz 2024-09-17T05:25:09.856 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-17T05:25:09.856 INFO:teuthology.orchestra.run.smithi176.stderr:real 2m33.388s 2024-09-17T05:25:09.856 INFO:teuthology.orchestra.run.smithi176.stderr:user 3m35.007s 2024-09-17T05:25:09.856 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m9.950s 2024-09-17T05:25:29.294 INFO:teuthology.orchestra.run.smithi150.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.37323.log.gz 2024-09-17T05:25:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-09-17T05:25:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:real 2m52.875s 2024-09-17T05:25:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:user 4m2.523s 2024-09-17T05:25:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m11.508s 2024-09-17T05:25:29.296 INFO:tasks.ceph:Archiving logs... 2024-09-17T05:25:29.297 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553/remote/smithi150/log 2024-09-17T05:25:29.298 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T05:25:54.727 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553/remote/smithi176/log 2024-09-17T05:25:54.728 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T05:26:15.986 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T05:26:16.006 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T05:26:16.006 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T05:26:16.009 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T05:26:16.078 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T05:26:16.087 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T05:26:16.088 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T05:26:16.090 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T05:26:16.126 INFO:teuthology.orchestra.run.smithi150.stdout: remote refid st t when poll reach delay offset jitter 2024-09-17T05:26:16.126 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================== 2024-09-17T05:26:16.126 INFO:teuthology.orchestra.run.smithi150.stdout:*hv01.front.sepi 67.205.162.81 3 u 63 128 377 0.151 -5.567 1.449 2024-09-17T05:26:16.127 INFO:teuthology.orchestra.run.smithi150.stdout:+hv02.front.sepi 63.231.80.2 3 u 62 128 377 0.149 -4.549 1.491 2024-09-17T05:26:16.127 INFO:teuthology.orchestra.run.smithi150.stdout:+hv03.front.sepi 74.6.168.72 3 u 52 128 377 0.161 -1.032 2.186 2024-09-17T05:26:16.127 INFO:teuthology.orchestra.run.smithi150.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-17T05:26:16.139 INFO:teuthology.orchestra.run.smithi176.stdout: remote refid st t when poll reach delay offset jitter 2024-09-17T05:26:16.140 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================== 2024-09-17T05:26:16.140 INFO:teuthology.orchestra.run.smithi176.stdout:+hv01.front.sepi 67.205.162.81 3 u 123 128 377 0.146 -2.147 0.206 2024-09-17T05:26:16.140 INFO:teuthology.orchestra.run.smithi176.stdout:*hv02.front.sepi 63.231.80.2 3 u 124 128 377 0.146 -1.140 0.179 2024-09-17T05:26:16.140 INFO:teuthology.orchestra.run.smithi176.stdout:+hv03.front.sepi 74.6.168.72 3 u 68 128 377 0.149 +1.537 0.070 2024-09-17T05:26:16.140 INFO:teuthology.orchestra.run.smithi176.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-17T05:26:16.141 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T05:26:16.149 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T05:26:16.150 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T05:26:16.158 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T05:26:16.166 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T05:26:16.174 INFO:teuthology.task.internal:Duration was 2886.818929 seconds 2024-09-17T05:26:16.174 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T05:26:16.183 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T05:26:16.183 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T05:26:16.185 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T05:26:16.359 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T05:26:16.359 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-09-17T05:26:16.359 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T05:26:16.370 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-09-17T05:26:16.370 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T05:26:16.381 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T05:26:16.381 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T05:26:16.413 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T05:26:16.475 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T05:26:16.476 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T05:26:16.478 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T05:26:16.702 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T05:26:16.714 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T05:26:16.714 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T05:26:16.736 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T05:26:16.760 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T05:26:16.769 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T05:26:16.781 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T05:26:16.798 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-09-17T05:26:16.830 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-09-17T05:26:16.855 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T05:26:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T05:26:16.867 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T05:26:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T05:26:16.903 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T05:26:16.912 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T05:26:16.914 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553/remote/smithi150 2024-09-17T05:26:16.914 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T05:26:16.936 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908553/remote/smithi176 2024-09-17T05:26:16.936 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T05:26:16.967 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T05:26:16.967 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T05:26:16.977 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T05:26:17.012 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T05:26:17.022 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T05:26:17.022 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T05:26:17.032 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T05:26:17.032 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T05:26:17.034 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T05:26:17.038 INFO:teuthology.orchestra.run.smithi150.stdout: 658625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 05:26 /home/ubuntu/cephtest 2024-09-17T05:26:17.060 INFO:teuthology.orchestra.run.smithi176.stdout: 658615 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 05:26 /home/ubuntu/cephtest 2024-09-17T05:26:17.061 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T05:26:17.071 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T05:26:17.093 INFO:teuthology.run:Summary data: description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} duration: 2886.818929195404 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-09-17T05:26:17.093 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T05:26:17.169 INFO:teuthology.run:pass